Scientists are using a Duke research misconduct case to draw attention to what they say is an increasing problem in some fields of scientific research—experiments that are not easily reproducible.
The researchers, largely biostatisticians, say that data in fields such as genomics have become so complex and unwieldy that they are difficult to interpret without sophisticated computer programs. But when scientists do not provide the data or programs they use to draw the conclusions they publish in scientific journals, it can be difficult or impossible for other researchers to verify their work.
“There has been a growing chorus for making data and software available widely because of the complexity of the data we are relying upon,” said Scott Zeger, vice provost for research at Johns Hopkins University and a professor of biostatistics.
Zeger is also a member of Scientists for Reproducible Research, a new group that advocates for stronger standards of reproducibility. The group, which formed this summer in the wake of research misconduct allegations against Duke cancer researcher Dr. Anil Potti, recently submitted a letter to the journal Nature, calling for scholarly journals to help ensure that the research they publish is reproducible.
“Journals should demand that authors submit sufficient detail for the independent assessment of their paper’s conclusions,” the authors wrote in the letter, which was published in the Sept. 23 issue of Nature.
The letter, signed by 47 members of Scientists for Reproducible Research, recommends that journals require scientists to provide their data, software and codes used to analyze the data and descriptions of other methods of analysis used in the research. The letter says that following these principles will help ensure that published research is reproducible and therefore valid.
Zeger said providing more information in journals will allow other scientists to more easily verify published research and help prevent errors.
“You can do some kicking of the scientific tires to see if the findings are robust,” he said. “At a minimum, you need to be able to reproduce the findings.”
Two biostatisticians who have identified errors in some of Potti’s research said that their efforts to verify his results were hindered by the absence of data or software scripts from some of his published research.
Kevin Coombes said he and fellow University of Texas MD Anderson Cancer Center biostatistician Keith Baggerly spent more than 1,500 hours trying to reconstruct portions of Potti’s research. They ultimately published work pointing out errors in Potti’s research on the use of genomics to target cancer treatments.
Coombes said that as the findings of complex research in fields such as genomics are used more frequently to inform the ways in which doctors treat patients, it becomes even more important to ensure that they are accurate.
He pointed out that delays in verifying Potti’s research meant that some cancer patients in clinical trials were assigned to cancer treatments based on genomic information that may not have predictive value.
Enrollment in clinical trials based on Potti’s research was suspended this summer. Huntington Willard, director of Duke’s Institute for Genome Sciences and Policy, noted that all patients in clinical trials based on Potti’s research received “standard of care” cancer treatment.
In an interview before the letter’s publication, Willard said he thought that discussions about the need for a better set of standards in genomic research would have occurred even without the allegations against Potti.
“It’s the normal development of clinically relevant research as it moves toward clinical practice,” he said.
Several biostatisticians who are members of Scientists for Reproducible Research said members of their field have been trying to draw attention for some time to the issue of research that is difficult to replicate. The allegations against Potti have helped highlight their cause and galvanized biostatisticians, they said.
“People came together because they felt a mistake was being made,” Zeger said. “All were of a mind that reproducibility was essential.”
He said he is optimistic that more stringent standards of reproducibility will be adopted in the future. Scientists will become more suspicious of journal articles that do not include the information necessary to reproduce the findings, he predicted.
“What will happen over time is people whose research is not reproducible will be less influential,” Zeger said. “If I can’t reproduce your findings, then I’ll discount them.”
Get The Chronicle straight to your inbox
Signup for our weekly newsletter. Cancel at any time.