Improving quality and reproducibility of research

by Alexey Bersenev on March 9, 2016 · 0 comments

in open science

Post to Twitter Send Gmail Post to LinkedIn

I’ve written about reproducibility in stem cell research in the past few years. Reproducibility problem is well recognized now, but it is not going away, especially in such rapidly evolving and competitive field as stem cell research. We all remember STAP scandal, which was unfolding due to lack of reproducibility.

There are some initiatives and proposals to improve reproducibility in life sciences. One of the most interesting projects is Validation by Science Exchange. New reproducibility projects and proposals for improving quality of research are continue developing. Recently, new channel Preclinical Reproducibility and Robustness was launched by online publishing platform F1000Research.

Behind of new reproducibility project is biotech giant Amgen. In 2012 Amgen declaired that most (47 of 53) high profile academic cancer papers cannot be reproduced by industry folks. F1000Research provides a rapid way to get your reproducibility report out for review by your peers after posting. Conventional publishing system does not allow easy rapid correction of errors and publication of failed replication efforts. F1000 could be an ideal platform for industry if they want to be transparent about their results of academic studies replication. About new project from the channel’s editorial:

We write to promote an additional effort to improve scientific standards–one aimed at strengthening the self-correcting nature of science through the widespread, rapid publication of the failures (as well as the successes) of attempts to reproduce published scientific findings.

Industry can enhance the self-correcting nature of science by organizing a robust effort to publish both its non-confirming and its confirming results, while simultaneously encouraging publication of such validation experiments by academic scientists.

So far, Amgen has published 3 failed reports. Monya Baker of Nature wrote about these reports:

The three studies that Amgen has posted deliberately do not make a detailed comparison of their results to previous papers, says Kamb. “We don’t want to make strong conclusions that someone else’s work is wrong with a capital W,” he says.

“We believe that interested scientists can look at our methods and results and draw their own conclusions,” Kamb says. Amgen researchers did not contact the original authors when they conducted their studies, he says, but future postings could be collaborative.

Though it is very unusual move for the industry, it is great initiative, which deserves our attention! It is not clear for now if this channel will gain traction and attract other industry folks:

The F1000 initiative is useful, but previous efforts have tried and failed to encourage the reporting of replications and negative results, cautions John Ioannidis, who studies scientific robustness at California’s Stanford University. That is because, in general, the scientific community undervalues such work, he says.

I hope, this initiative will work well!

A couple of other potential solutions for improvement of research quality were recently discussed – auditing of scientific manuscripts and quality assurance. Unlike auditing of publications, I found the concept of quality assurance (QA) in research more interesting and realistic. This is another Nature’s piece by Monya Baker that I’d highly recommend to read for every researcher! It nicely outlines all potential benefits of introduction quality assurance in research lab, but also highlights probable misunderstanding and resistance to this approach:

Any scientist adopting a QA system has to wager that the up-front hassle will pay off in the future. “It is very difficult to get people to check and annotate everything, because they think it is nonsense,” says Carmen Navarro-Aragay, head of the University of Barcelona quality team that worked with Cirera. “They realize the value only when they get results that they do not understand and find that the answer is lurking somewhere in their notebooks.”

Scientists need to take the lead on which QA elements they incorporate, says Melissa Eitzen, director of regulatory operations at the University of Texas Medical Branch in Galveston. “You want to give them tips that they can take or not take,” she says. “If they choose it, they’ll do it. If you tell them they have to do it, that’s a struggle.”

I think, some degree of quality assurance will be beneficial for some lab and for some projects within one lab. I think, it is especially important if lab performs human translational research with potential move to clinical trials. There are some regulatory recommendation and guidance for researchers about QA:

In 2001, the World Health Organization published guidelines for QA in basic research. And in 2006, the British Association of Research Quality Assurance (now simply the RQA) in Ipswich issued guidelines for basic biomedical research. But few academic researchers know that these standards exist (Davies certainly didn’t back in 2007).

So, if you do translational stem cell/ cell therapy research, try to apply QA principles and see if it will improve the quality and lab operations. What do you think about potential value of QA for research?

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: