NOT-OD-15-103: Enhancing Reproducibility through Rigor and Transparency

Enhancing Reproducibility through Rigor and Transparency

Notice Number: NOT-OD-15-103

Key Dates
Release Date: June 9, 2015

Issued by
National Institutes of Health (NIH)

Purpose

The National Institutes of Health (NIH) Office of Extramural Research (OER) plans to clarify and revise application instructions and review criteria to enhance reproducibility of research findings through increased scientific rigor and transparency. These updates, pending approval by the White House Office of Management and Budget (OMB), will take effect for applications submitted for the January 25, 2016, due date and beyond.

Background

NIH’s mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability. Key to the successful application of that knowledge toward health outcomes is scientific rigor in conducting biomedical research. One of NIH’s four stated goals is to exemplify and promote the highest level of scientific integrity, public accountability, and social responsibility in the conduct of science (see http://www.nih.gov/about/mission.htm).

Recent attention on the inability to reproduce or extend research findings reported in some peer-reviewed literature led the NIH Director and Principal Deputy Director to publish a commentary on possible causes and potential NIH actions (http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586). NIH successfully piloted some of these actions in specific funding opportunities over the past year. The results of these pilots have informed the revised instructions to address NIH expectations about the rigor and transparency in NIH supported research.

Newly revised grant application instructions will: clarify long-standing expectations to ensure that NIH is funding the best and most rigorous science; highlight the need for applicants to describe details that may have been previously overlooked; highlight the need for reviewers to consider such details in their reviews through revised review criteria; and minimize additional burden. These new instructions and revised review criteria will focus on four areas deemed important for enhancing rigor and transparency: 1) the scientific premise of the proposed research, 2) rigorous experimental design for robust and unbiased results, 3) consideration of relevant biological variables, and 4) authentication of key biological and/or chemical resources.

Commentaries by NIH leadership have highlighted these four areas as requiring attention (see also http://www.nature.com/news/policy-nih-to-balance-sex-in-cell-and-animal-studies-1.15195 and http://www.sciencemag.org/content/346/6216/1452.full.pdf). While much of the attention on reproducibility has centered on preclinical research, the basic principles and areas of focus apply to the full spectrum of biomedical research - from basic to clinical. Investigators will need to consider how all four areas apply to their proposed research. Likewise, reviewers will assess whether these areas have been appropriately addressed by the applicant through the revised peer-review criteria.

Scientific Premise

All research builds upon prior research, which may include observations, preliminary data, or published literature. The scientific premise for an application is the research that is used to form the basis for the proposed research question; NIH has always strived to fund projects that are based on a strong foundation. Moving forward, NIH expects applicants to describe the general strengths and weaknesses of the prior research being cited by the investigator as crucial to support the application. It is expected that this consideration of general strengths and weaknesses could include attention to the rigor of the previous experimental designs, as well as the incorporation of relevant biological variables and authentication of key resources. For example, basing one's proposed research on previous publications that lacked statistical power, were not blinded, lacked detail on the sex of animals or authentication of cell lines would be considered a weakness of the application if it does not identify these weaknesses and propose ways to improve going forward. Likewise, conclusions drawn from prior research that used a small sample size may not adequately support the next phase of research, such as moving to a higher species of animals or to humans. See Landis, et al., 2012 (http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html) for a description of the value of scientific premise and core reporting standards important for evaluating the premise.

It is important to note that some funding opportunities have required an evaluation of the scientific premise in applications for some time. The new application instructions and revised review criteria will ensure that all areas of research emphasize the principle that meritorious science is built on a rigorous foundation.

Rigorous Experimental Design

Scientific rigor is the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. This includes full transparency in reporting experimental details so that others may reproduce and extend the findings.
NIH expects applicants to describe how they will achieve robust and unbiased results when describing the experimental design and proposed methods. Robust results are obtained using methods designed to avoid bias and can be reproduced under well-controlled and reported experimental conditions.

A number of NIH Institutes have published guidelines for reporting elements of rigor in experimental design to be included in applications considered for funding, namely: National Institute of Neurological Disorders and Stroke (NOT-NS-11-023), National Institute of Mental Health (NOT-MH-14-004) and National Institute on Drug Abuse (NOT-DA-14-007). Similar guidelines have been adopted by journals for preclinical research across many disciplines; see http://www.nih.gov/about/reporting-preclinical-research.htm and http://www.nih.gov/about/endorsing-jounals.htm. Many of these principles can be adopted or adapted in all areas of biomedical research.

Consideration of Sex and Other Relevant Biological Variables

Biological variables, such as sex, age, weight, and underlying health conditions, are often critical factors affecting health or disease. In particular, sex is a biological variable that is frequently ignored in animal study designs and analyses, leading to an incomplete understanding of potential sex-based differences in basic biological function, disease processes and treatment response. NIH expects that sex as a biological variable will be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Strong justification from the scientific literature, preliminary data or other relevant considerations must be provided for applications proposing to study only one sex. Please refer to NOT-OD-15-102 for further consideration of NIH expectations about sex as a biological variable.

Similarly, investigators should consider other biological variables, as appropriate, in the design and analyses of their proposed studies. Research plans and findings should clearly indicate which biological variables are tested or controlled. Clear justification should be provided for exclusion of variables that may be relevant but are not considered in the research plan. For example, studies using young adult animals should clearly describe their study population and not generalize findings to juvenile or aged animals.

Authentication of Key Biological and/or Chemical Resources

The quality of the resources used to conduct research is critical to the ability to reproduce the results. NIH expects that key biological and/or chemical resources will be regularly authenticated to ensure their identity and validity for use in the proposed studies. Key biological and/or chemical resources may or may not be generated with NIH funds and: 1) may differ from laboratory to laboratory or over time; 2) may have qualities and/or qualifications that could influence the research data; and 3) are integral to the proposed research. These include, but are not limited to, cell lines, specialty chemicals, antibodies and other biologics.

There are not clear consensus guidelines on authenticating different types of resources, and NIH encourages the research community to inform the development of such guidelines. In the absence of clear guidelines, researchers should transparently report on what they have done to authenticate key resources, so that consensus can emerge.

Implementation

Pending OMB approval, the new revised grant application instructions will be incorporated into the SF424 (Research and Related) Application Guide and funding opportunity announcements in Fall of 2015, for application submission for the January 25, 2016 due date and beyond. Applicants will be instructed to include their consideration of scientific premise, rigorous experimental design, and consideration of sex and other relevant biological variables in the Research Strategy section. Page limits for this section will not change. Reviewers will be asked to evaluate scientific premise as part of Significance, and rigorous experimental design and consideration of sex and other biological variables as part of the Approach criteria. As such, evaluation of these three areas will be included in the assessment of overall impact. Authentication of Key Resources will be incorporated as a new attachment under the Other Research Plan Sections and reviewers will be asked to comment on the plan but not consider it when scoring overall impact.

More detailed information about the specifics of implementation will become available this fall.

A new site focused on what these reproducibility efforts mean for NIH applicants and grantees may be found at https://grants.nih.gov/reproducibility. The associated Frequently Asked Questions (FAQ; https://grants.nih.gov/reproducibility/faqs.htm), will continue to be updated.

Inquiries

Please direct all inquiries to:

Email: [email protected]