Reproducing results of past experiments is a way that science advances. Reproducible experiments allow the sciences to build from prior research. Presently, there is a crisis of reproducible results in the sciences. Without reproducible results, scientific data is reduced to hearsay – this reduces forward progress.
2016 findings in the journal Nature indicate that reproducing the results of experimentation comes hard for many. In the chemical sciences alone, 60 percent of chemical scientists find non-reproducibility of their own data to be an issue. So, what is reproducibility – and is there a way scientists can fix this troubling issue?
What Does ‘Reproducible Research’ Mean?
To get to the bottom of these research problems, we should define the term reproducible – and place it in an appropriate context. Let’s take the definition of ‘reproduce’ from the American Heritage Dictionary of the English Language:
“1. To produce again or anew; re-create: The lab failed to reproduce their original results. The movie reproduces life in the 1950s.
2. To produce a copy, imitation, or representation of: reproduce the sounds of a live concert in a recording.
3. To generate (offspring) by sexual or asexual means.
4. To bring (an event in one’s memory, for example) to mind again; recall.“
Examining the definition, you notice it covers a lot of ground: from lab work to info-graphics to offspring. However, the dominant aspect of the definition in science, at least, is re-creating results from prior research.
2016 Survey in the Journal, Nature
We find the problems of reproducibility in multiple issues, such as poor experimental design, a lack of productive mentorship, an incomplete understanding of experimental statistics, and bad analyses, to name a few. Those are some of the answers given by more than 1500 of scientists cited in a 2016 issue of Nature.
Reproducibility is an issue across all science disciplines, and there are a few common denominators.
Scientists who were cited in the Nature article listed measures that could improve this reproducibility crisis. Ninety percent of them listed three items: 1) more robust experimental design 2) better statistical analysis 3) improved mentorship.
These three items center around better educating students – the new research scientists. While new researchers learn the ropes in graduate school, undergraduate research may need to be strengthened. A stronger grounding in research fundamentals would make already overburdened graduate programs produce stronger research candidates.
This writer believes that all undergraduates would benefit from understanding the meaning of the discovery of novel results. Having personally spent a couple of years as an undergraduate researcher, I gained valuable understanding of lab independence but not necessarily the hard-fought for discoveries that senior researchers fight for.
While certain undergraduate programs require students perform a semester or two of research, in the opinion of this chemist, all science disciplines would benefit a mandatory requirement of one year of original undergraduate practical research.
Although the American Chemical Society does not strictly require undergraduate research for accreditation, many institutions have their undergraduate students participate in some kind of research prior to graduation. The American Chemical Society states:
“At the undergraduate level, research is self-directed work under the guidance and supervision of a mentor/advisor ― usually a university professor. A gradual transition towards independence is encouraged as a student gains confidence and is able to work with minor supervision.”
As can be seen from the statement, emphasis is placed upon independent lab work, and not so much understanding the meaning of scientific discovery. While much is gleaned from independent thought– much more is gained from understanding how statistical analyses explain discovery. That may be the weak spot in the chain.
Improving Reproducibility: Three Major Factors?
Above, you’ll find listed the three major factors for reproducibility, as identified by the majority of scientists surveyed by Nature. Let’s take a closer look at each of the three:
- Improved Robustness in Experimentation Design
Researchers perform experiments to explain and explore the workings of nature. Reproducibility comes into play after repetitive analyses yield results that can be confirmed independently of the original research by others. In essence, experimentation becomes robust when it is easily repeated across the sub-disciplines of that branch of science.
Robust experimentation design causes little or no confusion to other experimenters, whether it is the experienced researcher or the novice. Robust design is, in essence, portable – or can be performed by any researcher across different laboratories. Presently, researchers report discoveries within their immediate field, to a specialized group of readers. The research design may not be easily understandable within undergraduate circles, or even for inexperienced graduate students.
Improving design robustness allows other researchers, not in the same immediate specialty as the researcher, to reproduce methods and associated discoveries. Standardizing the experimentation that led to the discovery is a potential answer. Because standardizing puts the experiment in terms that are common to both experts and novice, it also removes uncertainties in interpretation of experimentation. It allows others to easily follow the procedure. This may be viewed as a way to eliminate confusion for other scientists, and improve reproducibility.
- Improved Statistical Analyses
In some experimental sciences, such as chemistry or physics, undergraduate colleges teach rudimentary statistical analyses within the curriculum. This standard of experimentation teaches students to analyze how errors occur in standard measurement processes affecting any their results. Results that are standard, in many instances, do not stretch creative processes of scientific discovery. While in Doctoral research programs, statistical analyses are performed to evaluate the credibility of discoveries – advancing frontiers of knowledge.
By contrasting hard science undergraduate curricula with hard science Ph.D. curricula, one comes away with the understanding that coursework may need to be added. Courses that, for example, teach the student to evaluate the credibility of discoveries based on statistical analysis could further improve reproducibility.
- Improved Mentoring
The purpose of mentoring graduate students in scientific research is introduction to the methods of discovery. Possessing a curious mind and scientific skills from undergraduate school are two aspects necessary for learning to be a professional researcher. Mentoring in graduate school is meant to bring the novice researcher up to professional speed. However, in the end, good mentoring shows students how to attack research problems. It gives the student the means to feel confident enough to perform professionally. Good mentoring also gives the student proper direction to know where to look to instruct themselves to understand how to use statistical analyses.
Mentoring students addresses immeasurable parameters. Parameters that deal with interpersonal relations between mentor, mentee, and professionalism. While better mentoring processes could alleviate how undergraduates and Ph.D. students approach statistical analyses and standardizing experimentation, not all Ph.D. programs are created equally. Some students do receive appropriate levels of mentoring that would allow them to thrive in doctoral programs. All too often, the attitude of many established research programs is for students to discover their way (as a right of passage).
The (Chemical) Research Paradigm
While the focus of this article is chemical research, the reproducibility crisis exists in all research sub-disciplines.Reproducibility of research is the key to unlock the door to future research successes. Research at a Ph.D. level is designed to push back the darkness of ignorance within a field or discipline. It is meant to create and discover new avenues of knowledge– something that not all experimenters approaching a specialized field know how to attain.