The rigor and transparency policy applies to most research grant and mentored career development award applications submitted for the January 25, 2016 due date and beyond. Specific exceptions are listed in the research (NOT-OD-16-011) and career development (Ks except K12 and KL2; NOT-OD-16-012) notices and in the activity codes list for rigor and transparency.
Research grant and mentored career development award applications submitted for due dates on or after January 25, 2016 must address the rigor and transparency requirements outlined in the application instructions. Beginning at the end of March, NIH will be rolling out updated application forms (FORMS-D) to be used for due dates of May 25 and beyond. Pay close attention to the new form instructions, as the placement of some of the rigor information will have changed.
Research Performance Progress Reports (RPPR) submitted January 25, 2016 or later will be expected to emphasize rigorous approaches taken to ensure robust and unbiased results. Rigor should be addressed in the RPPR for any grant that funds research. Please refer to the RPPR instruction guide for more details. Institutional training grants do not need to address rigor at this time. Details will be provided well ahead of due dates for implementation
For more information on pre-award and post-award policy and forms updates, please refer to guide notices NOT-OD-16-004 and NOT-OD-16-005.
Scientific premise, scientific rigor, and relevant biological variables such as sex should be addressed within the Research Strategy of research applications, as these elements are integral to the research plan. Since scientific premise will be reviewed and scored as part of the Significance review criterion for research grant applications, applicants should address premise as part of their corresponding Significance section in the Research Strategy. Scientific rigor and relevant biological variables will be reviewed and scored as part of the Approach review criterion.
For mentored career development award applications, all three areas (scientific premise, scientific rigor, and relevant biological variables such as sex) should be addressed in the Research Strategy and all three areas will be reviewed as part of the Research Plan.
Authentication of key resources will be addressed in a separate attachment and will not be scored.
The new rigor and transparency updates apply to most research and career development opportunities, including those published prior to the policy with additional due dates on or after January 25, 2016. Existing FOAs for which rigor and transparency apply were updated with links to the policy in November of 2015. FOAs published on or after October 2015 include the updated rigor and transparency language, if applicable for the activity code.
Yes. The updated review language will apply to all competing applications submitted under applicable activities, regardless of whether the submission is a new application (an "-01"), a resubmission (an "A1"), or a renewal (a "Type 2").
The NIH expects that rigor and transparency can be included within the existing page limits for the Research Strategy. Brief plans describing the authentication of key biological and/or chemical resources should be addressed in a separate attachment. Unless otherwise specified in the relevant FOA, the page limitations described in the SF424 Application Guide and the Table of Page Limits are applicable. Submission of an application that does not comply with the page limit requirements may lead to rejection of the application during agency validation. Since not all applications will need an authentication plan attachment based on the science proposed, applications submitted without the authentication attachment will proceed to review.
Please refer to NOT-OD-07-018 and the specific FOA for guidance on what information may be provided as appendices. Applicants are prohibited from using the Appendix to circumvent page limitations in any section of the application for which a page limit applies.
Reviewers are directed to consider the strengths and weaknesses associated with each of the review criteria and to weigh them appropriately for the specific application assigned to them. Thus, the NIH policy on rigor and transparency will contribute to the relevant criterion scores and the overall impact score of each application, with the exception of Authentication of Key Biological and/or Chemical Resources.
For research grant applications, reviewers will be instructed to evaluate scientific premise as part of the significance criterion, and to evaluate scientific rigor and relevant biological variables such as sex as part of the approach criterion, as well as in their assessment of overall impact.
Reviewers will comment on the proposed Key Biological and/or Chemical Resource Authentication Plan as an additional review consideration, but should not consider it in the overall impact score.
Details on the updated application instructions, progress reporting, and review language can be found in the following guide notices:
Rigor and transparency in research grant applications (NOT-OD-16-011)
Rigor and transparency in career development awards (NOT-OD-16-012)
Investigators are expected to report on approaches taken to ensure robust and unbiased results in their progress report as well as in their submitted manuscripts. A set of guidelines for reporting preclinical research in publications was developed during an NIH sponsored joint workshop with the Nature Publishing Group and Science in June 2014 on the issue of reproducibility and rigor of research findings. Journal editors representing over 30 basic/preclinical science journals agreed on a set of principles and guidelines for reporting preclinical research to facilitate these goals and over 135 journals have endorsed the principles.
Investigators will be directed to emphasize the approaches taken to ensure robust and unbiased results, including any developments affecting the proposed experimental design, methodology, analysis and interpretation (Section B. – Accomplishments) in the NIH Research Performance Progress Report (RPPR). If sufficient information is not provided in the progress report, program officials may request the additional information needed to assess progress.
Program Officials have several options in cases where an investigator encounters challenges in meeting their research objectives. A clear explanation of the difficulties, as well as the actions the investigator has taken to overcome them, should be provided in the progress report. POs may respond with a range of measures from offering suggestions for alternative approaches, requesting an interim progress report, or issuing a no-cost extension to delay the noncompeting renewal until the difficulties are resolved. In extreme circumstances, where is it clear that the research objectives cannot be achieved, it may be appropriate to end support for the project earlier than planned.
Several funding opportunities have been announced that incorporated some of the elements discussed in the original commentaries by Landis et al. 2012 and Collins and Tabak, 2014. The purpose of these pilots was to ensure that the rigor and transparency policy implemented by NIH is feasible. The pilots that were conducted verified that it is feasible for applicants to address scientific premise and to describe the scientific rigor in their proposed research within existing page limits. Some reviewers who participated in reviews of pilot FOAs responded to a survey, and most reported that the research strategies in the applications they reviewed were supported by rigorous experimental design and methodological approaches.
NIH is planning a process evaluation to assess how applicants address the policy on rigor and transparency in their grant applications and how reviewers take the policy into consideration in their reviews.
Scientific premise concerns the quality and strength of the research used to form the basis for the proposed research question. NIH expects applicants to describe the general strengths and weaknesses of the prior research being cited by the applicant as crucial to support the application. The NIH expects this consideration of general strengths and weaknesses to include attention to the rigor of the previous experimental designs, as well as relevant biological variables and authentication of key resources.
The scientific premise will be reviewed as part of the Significance criterion for research grant applications. Instructions for significance already include consideration of the importance of the problem, critical barriers to progress, how the proposed project will improve scientific knowledge, and how the field will change if the aims are achieved. Scientific premise includes a retrospective consideration of the foundation for the application, rather than a prospective analysis should the aims be achieved.
All prior research cited by an investigator as crucial to support their application should be assessed for strengths and weaknesses, regardless of whether it is the investigator’s own preliminary data (published or unpublished) or published data from others. Any weaknesses or gaps in rigor, or reporting on rigor, should be acknowledged, along with a plan to address those gaps going forward.
The instructions for scientific premise (under Significance, for research grant applications) relate to the scientific rigor of the foundation of knowledge upon which the research questions are posed, including preliminary data if they are provided; in contrast, the instructions for scientific rigor (under Approach, for research grant applications) relate to the rigor of the proposed research. If preliminary data are not provided in an application, a critical assessment of the scientific literature that supports and/or contradicts the research question(s) can be provided.
Scientific rigor is the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. This includes full transparency in reporting experimental details so that others may reproduce and extend the findings. Investigators should apply the elements of rigor that are appropriate for their science.
Robust results are obtained with solid, well-controlled experiments capable of being reproduced under well-controlled conditions using reported experimental details. Applicants should consider methods to reduce bias, such as having multiple individuals recording assessments, defining terminology in advance, using independent, blinded assessors, etc. "Robust" and "unbiased" results are goals, not absolute standards to be met, and may vary across scientific disciplines.
Applicants are encouraged to include a succinct description of the experimental design and methods with enough detail to assure the reviewers that the necessary elements of rigor will be addressed. As per the instructions in the SF424, all applications must be self-contained within specified page limits. Reviewers are under no obligation to review and/or read published methods cited in an application. Similarly, unless otherwise specified in a solicitation, URLs may not be used to provide information necessary to the review. Moreover, reviewers are cautioned that they should not directly access an internet site as it could compromise their anonymity.
Innovative research involves a greater level of risk because of the novelty of the research questions, but innovative research can still be carried out in a rigorous manner. The risk associated with the research can be identified and managed by considering the scientific premise, identifying the factors that are unknown, and incorporating strategies to reduce bias and ensure the methods are designed to generate robust results appropriate for the stage of research. Exploratory research may be able to accommodate more risk than clinical research. Strategies to mitigate risk should increase commensurate with research type or stage, such as moving from preclinical into clinical research. Innovative research projects are expected to generate data that is reproducible and provides a foundation for future studies.
NIH-supported research projects should be designed to achieve robust and unbiased results. All research projects can be designed, described, and conducted rigorously, whether the research is driven by a hypothesis or directed by a goal such as technology development. Similarly, research projects that are highly innovative can be designed, described, and conducted rigorously.
The review criteria for rigor and transparency generally apply to all research grant applications; however special review considerations may be indicated in the FOA for some activities, such as research resource grants.
As with information typically included in grant applications, one cannot present every detail, yet there are ways to succinctly state what is planned. For example: "10 males and 10 females will be randomized to blinded treatment and control groups, giving 80% power to detect a treatment effect size of 65% compared to a baseline response of 5% at a significance level of 0.05."
A number of NIH institutes and centers have issued more detailed guidelines in specific funding opportunities or for an area of funding, such as all preclinical research. For examples of guidance that may be helpful for you to consider as you develop your application, see NINDS Guidance, NOT-MH-14-004, and NOT-DA-14-007. For examples of past funding opportunities, see PAR-13-023 (R21) and RFA-NR-15-001 (R01).
The updated review language directs peer reviewers to assess the scientific rigor of the experimental design, including the appropriateness of any justification for the sample size selection, under the Approach review criterion for research grant applications. The updated review language formalizes NIH's expectations that statistical power be addressed. Program staff also may, at the IC's discretion, recommend changes to experimental designs, for example, to establish conformity with adopted best practices for the research discipline.
Appropriate strategies that consider sex as a biological variable (SABV) depend on the context of the research question, existing knowledge about male and female biology and behavior in a given area of research, available methodology, and other factors. NIH review committees will evaluate consideration of sex as a biological variable as part of their assessment of the research strategy for applications for research involving vertebrate animals and humans. An SABV decision tree for reviewers is now availableand is also useful for applicants.
Accounting for sex as a biological variable in applications for NIH-funded research could be reflected in:
relevant review of available literature on the influence of biological sex
formulation of research questions
incorporating both males and females into studies
articulating strong justification for a single-sex study
consideration of the influence of sex in study design
stratified randomization of males and females into experimental conditions
characterization of study results for males and females
examination of treatment or toxicity effects for each sex separately
consideration of the influence of sex in the interpretation of study results
appropriate generalization of research findings
Note that this is not an exhaustive list. Please see Clayton, 2015 for additional information on what it means to consider sex as a biological variable.
Applicants should consider the biological variables that are relevant to the experimental design of the study. The choice of animal model or human population to be included will vary with the scientific topic of the proposed research. For example, sex, age, weight, and underlying health conditions are biological variables that may affect outcome and should be considered where applicable.
In particular, sex is a biological variable that has been frequently ignored in animal study designs and analyses, leading to an incomplete understanding of potential sex-based differences in basic biological function, disease processes, and treatment responses. NIH expects that sex as a biological variable (SABV) will be considered in research designs, analyses, and reporting of vertebrate animal and human studies. Strong justification from the scientific literature, preliminary data, or other relevant considerations must be provided for applications proposing to study only one sex.
Applicants are instructed to discuss their consideration of sex as a biological variable for vertebrate animal and human studies. Sex should be accounted for when using primary cells, tissue explants, or other samples taken directly from the animal or human. In instances where cells or tissues will be implanted into a host (e.g. patient-derived xenografts), the sex of both the sample and the host should be considered and reported. A strong scientific justification must be provided for applications proposing the use of a single sex.
It is important that researchers using established cell lines consider the possible role of sex in their research. However, NIH recognizes current challenges to the authentication of the sex of established cell lines. At this time, NIH is working to enhance strategies and techniques to address these challenges. Although established cell lines are not explicitly addressed in this notice, NIH encourages researchers to consider sex as a biological variable and be transparent in reporting the sex of established cell lines, when known, as well as any SABV-related study results.
When existing data sets are used for research, there may be limitations in the data available which thereby influence the types of questions that can be asked along with the generalizability of the research. If a single-sex study is proposed because of limitations in existing data sets, grantees must provide strong justification for the use of such data. NIH review committees will evaluate consideration of sex as a biological variable as part of their assessment of the research strategy for applications proposing research involving vertebrate animals and humans and this will be reflected in reviewers’ scores. Prospective studies should be designed in a rigorous manner such that proper attention to biological variables, such as sex, is included in the plan.
NIH expects that sex as a biological variable will be factored into research designs, analyses, and reporting in vertebrate animal and human studies. NOT-OD-15-103 further communicates that a strong justification must be provided for applications proposing to study only one sex. NOT-OD-16-011 describes implementing rigor and transparency in NIH and AHRQ research grant applications and NOT-OD-16-012 describes implementation for mentored career development award applications.
Unless there is strong justification for a single-sex study, NIH expects that both males and females will be included in research involving vertebrate animals and humans. The research question at hand, along with existing knowledge in a given field, should guide the ways in which males and females are studied and how sex as a biological variable is considered. When studies incorporate both males and females, researchers should disaggregate this data, whether the study was statistically powered to detect sex differences or not. Reporting study data in a disaggregated manner enables its use for further study; for example, reporting descriptive statistics for males and females separately can provide useful information that would contribute to our understanding of male and female biology. Studies that control for sex in multivariate analyses should also report sex-specific results; controlling for sex tacitly implies that sex is a potentially influential variable.
The NIH is mandated by law (Public Health Service Act sec. 492B, 42 U.S.C. sec. 289a-2) and NIH policy to ensure the inclusion of women, minorities, and children in clinical research. The goal is to ensure that individuals are included in clinical research in a manner that is appropriate to the scientific question under study.
The new instructions for applicants and review language introduced in October 2015 expand upon the NIH inclusion policy in crucial ways. NIH expects sex as a biological variable to be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Appropriate analysis and transparent reporting of data by sex may enhance the scientific rigor, transparency, and applicability of biomedical research. Strong justification from the scientific literature, preliminary data, or other relevant considerations must be provided for applications proposing to study only one sex. Please refer to NOT-OD-15-102 for further consideration of NIH expectations about sex as a biological variable.
Similarly, investigators should consider other biological variables, as appropriate, in the design and analyses of their proposed studies. Research plans and findings should clearly indicate which biological variables are tested or controlled. Clear justification should be provided for exclusion of variables that may be relevant but are not considered in the research plan. For example, for studies involving young adult animals the study population should be clearly described and not generalize findings to juvenile or aged animals.
Addressing the influence of sex in biomedical research with animals does not necessarily imply an increase in costs. Rather, well-designed research either tests or controls for variables that might influence outcomes, and sex is one such variable among many that must be considered to obtain valid results. As always, each applicant must assess the characteristics of the test system, analytic approaches, and other factors in determining his or her research design. Budgets requested should be relative to the scope of the proposed research.
Supporting optimal and rigorous research design is the best investment NIH can make. Considering the possible role of sex early in the research continuum may save money by revealing differences or similarities that need to be taken into consideration in subsequent phases of study. Moreover, by making more sex-specific data available, investigators can more readily build on a stronger body of knowledge, which will likely enhance the efficiency of future research.
For details on NIH application updates concerning the Vertebrate Animals Section, please view guide notice NOT-OD-16-006.
Applicants must provide strong justification to study only one sex. Scarce animal resources may be considered adequate justification, based on the evidence of scarcity. Acceptable justifications may also include the study of sex-specific conditions or phenomena (e.g., ovarian or prostate cancer), or investigations in which the study of one sex is scientifically appropriate. The absence of evidence regarding sex differences in an area of research does not constitute strong justification to study only one sex. Cost also is not a consideration in determining whether both sexes are to be included in experiments.
Applicants should provide a justification that the species are appropriate for the proposed research in the vertebrate animals section. The rationale for the number of subjects planned for study should now be explained in the Research Strategy section, for research grant applications.
The PHS Policy on Humane Care and Use of Laboratory Animals, the Animal Welfare Act Regulations, and the Guide for the Care and Use of Laboratory Animals require animal use protocols to include a rationale for the number of animals to be used and require that the number proposed be limited to the minimum necessary to obtain valid results. It is the IACUC’s responsibility to review and confirm that a sound, objective, and logical reason has been provided for the number of animals proposed. If a PI has appropriately considered sex as a biological variable relevant to the study design, this is consistent with the federal requirement to use the minimum number, and acceptable to the IACUC.
Justification of the choice of sex(es) proposed in animal study protocols is not required by the PHS Policy on Humane Care and Use of Laboratory Animals, the Animal Welfare Act Regulations, or the Guide for the Care and Use of Laboratory Animals. Many IACUCs require identification of the sex of animals for facility management purposes, and some IACUCs ask for justification for studies proposing one sex. As stated in NOT-OD-15-102, accounting for sex as a biological variable should begin with the development of the research question and study design and will become part of review criteria for Approach during peer review, for research grant applications.
In September 2014, NIH released an RFI (NOT-OD-14-128) to gather input from the research community and other interested stakeholders regarding the consideration of SABV in biomedical research. Feedback from professional societies, patient advocacy groups, and individual scientists reflected a variety of scientific perspectives from basic, clinical, and translational areas of inquiry (see the full report here). The vast majority of respondents agreed that consideration of SABV is an issue affecting the reproducibility, rigor, and/or generalizability of research findings. Despite agreement that considering SABV is good science, scientists and other stakeholders were also concerned about practical matters, such as cost, as well as constraints on methodological and experimental design. More than half of respondents suggested that NIH could offer tangible resources to help with SABV policy implementation. The NIH Office of Research on Women’s Health and the NIH Office of Extramural Research have taken into account this input, and have been developing SABV training resources and scientific tools such as courses, workshops, and online resources to help applicants, reviewers, and NIH program staff.
The quality of resources used to conduct research is critical to the ability to reproduce the results.
Key resources refer to established resources that will be used in the proposed research.
Key biological and/or chemical resources include, but are not limited to, cell lines, specialty chemicals, antibodies and other biologics. Key biological and/or chemical resources may or may not have been generated with NIH funds and:
may differ from laboratory to laboratory or over time;
may have qualities and/or qualifications that could influence the research data; and
are integral to the proposed research.
Standard laboratory reagents that are not expected to vary do not need to be included in the plan. Examples are buffers and other common biologicals or chemicals.
Depending on the research study, biological samples may be considered key biological resources that need to be authenticated if they are an established resource, particularly if the investigator received the samples from an outside source.
Each investigator will have to determine which resources used in their research fit these criteria and are therefore key to the proposed research.
Applicants should briefly describe the methods they plan to use to authenticate key resources based on their scientific experience and judgment, referencing relevant standards where applicable. Guidelines and approaches will vary depending on the reagent/resource and the experimental context in which it will be used.
The NIH encourages the scientific community to establish consensus standards and best practices that can be cited by applicants. In addition, applicants are encouraged to discuss their research strategy with a program officer in the appropriate scientific area to learn whether additional resources are available on the NIH web site or elsewhere that can be used to guide their plans for resource authentication.
If key resources have been purchased or obtained from an outside source that provided data on prior authentication, the investigator is still expected to provide their own authentication plans for these key resources.
Applicants proposing to use established key biological and/or chemical resources are expected to include an authentication plan in the "Authentication of Key Biological and/or Chemical Resources" attachment, even if the key resources were purchased or obtained from an outside source that provided data on prior authentication. The authentication plan must include only a description of the methods proposed to authenticate key resources prior to use and at regular intervals, if appropriate. The plan should be no more than one page. Key resources and the methods for authentication will vary by research field.
For example, applicants proposing to use cell lines should describe the method they plan to use to verify the identity and purity of the lines, which might include short tandem repeat (STR) profiling and mycoplasma testing. Applicants proposing to use chemicals that are key to the research should describe the method used to validate the chemical, which might include liquid or gas chromatography or mass spectrometry. Applicants proposing to use genetically modified animals or cells should describe the method used to confirm the genome modification, which might include PCR amplification or Southern blot. When published consensus standards exist, these may be cited in this section as the procedure(s) that will be used for validation.
Authentication data should not be included in the plan.
See FAQ #1 for the definition of key biological and/or chemical resources.
The "Authentication of Key Biological and/or Chemical Resources" attachment should include only the authentication plans for established key biological and/or chemical resources. Do not include plans for the authentication of data sets, databases, machinery, or electronics in the attachment.
If the Research Strategy does not propose use of key biological and/or chemical resources, the Authentication Plan attachment may include a brief statement indicating that no key biological and/or chemical resources will be used in the activities proposed in the application.
Applicants proposing to generate a new key biological and/or chemical resource (such as a new human cancer cell line) should describe development and authentication of the resource in the Approach section of the Research Strategy.
See FAQ #1 for the definition of key biological and/or chemical resources.
The new application instructions and review language on authentication of key biological and/or chemical resources are intended for applications proposing use of established research resources that should be authenticated prior to and during use. For applications proposing to establish a new biological resource, such as a cell line, animal model, antibody or probe, the research activities to be conducted in pursuit of the resource development, including plans for validating the resource, should be described in the Research Strategy section of the application and will be subject to scoring.
For applications proposing to collect primary cells for short-term culture as part of the proposed research, the research activities to be conducted, including plans for authenticating the identity of the cells, should be described in the Research Strategy section of the application and will be subject to scoring. If the primary cells or cultures derived from fresh primary cells are being obtained from another laboratory, an authentication plan should be provided.
After scoring of the application is complete, Scientific Review Groups (SRGs) will comment on the plans for resource authentication in a manner consistent with the scientific goals of the research. Any concerns raised about the adequacy of the plans for resource authentication should be resolved by the program official before the application/proposal is funded.
In cases where the application will not be funded, applicants may contact the NIH program official assigned to the application to discuss potential solutions to address inadequate authentication plans prior to resubmitting the application.