NIH All About Grants Podcast

Rigor and Reproducibility Transcript

December 2018


Announcer:                     From the National Institutes of Health in Bethesda, Maryland, this is All About Grants. 

David Kosub:                 Hello and welcome to another edition of NIH’s All About Grants podcast.  I’m your host David Kosub with the NIH’s Office of Extramural Research.  We recognize that to address public health concerns, science must be empirical, replicable, and rigorous.  We know that the biomedical and behavioral research community considers these factors too when preparing, performing, and publishing their research.  In January 2016, NIH established a policy to continue strengthening these precepts of rigor and reproducibility in the research that we support.  And that’s what brings us here today.  We have with us Dr. Patricia Valdez.  She is NIH’s Extramural Research Integrity Officer and will be discussing what applicants should be considering as it relates to the policy and grant applications, as well as what a reviewer may be thinking about when scoring those applications, and what to do post award.  Thank you for being with us. 


Dr. Patricia Valdez:        My pleasure.


David Kosub:                 All right, Patricia.  Let’s begin by hearing what led NIH to establishing the Enhancing Reproducibility Through Rigor and Transparency policy back in January 2016.


Dr. Patricia Valdez:        Right.  So, reproducibility has been in the media for several years now.  I think it was around 2014 there were these large publications that indicated that pharmaceutical companies could not reproduce a lot of the published data.  And so of course that’s a concern because pharmaceutical companies would like to translate the research to make drugs.  So that was obviously a concern.  At the NIH of course, we’re always concerned that we are funding the most rigorous science.  We want to make sure that we’re building on a solid foundation of scientific research to, you know, really have the ability to further translate that research into being beneficial to human health.  Now along with these issues of not being able to reproduce the published data, there are other concerns that, you know, the community was expressing at the time and still are expressing.  So that includes this problem of misidentified cell lines.  So many, many cell lines have been used over the years, actually for decades.  People thought they were a particular type of cancer cell line, but they were actually a different type of cancer cell line.  A lot of them were actually HeLa cells.  So that’s an issue.  So, we want to make sure that people are aware that, you know, they should pay attention to these types of things.  And another concern was the lack of information that we were seeing on both sexes in biomedical research.  Certain fields tend to choose, you know, one sex over the other.  For instance, neuroscience, they use a lot of male rats.  In immunology, they use a lot of female mice.  And so, what we really wanted to do is to make sure that we were capturing information on both sexes.  So that was another concern.  And then of course, you know, we wanted to make sure that our applicants were providing enough information in their applications so that reviewers could actually judge the rigor of the science that they were proposing.


David Kosub:                 Okay.  So, what exactly is spelled out in the policy, the nuts and bolts, if you will?  How do these principles aim to enhance the rigor of NIH supported science?


Dr. Patricia Valdez:        Okay.  So, there’s four elements to the policy enhancing reproducibility.  The first element we call rigor of the prior research, formerly scientific premise.  And this really aims to make sure that, again, we’re building on the, you know, solid foundation.  So we ask applicants to talk about the rigor of the prior research that they’re using to support their proposal.  So we want to make sure that research is rigorous or if they identify any problems that then they’re going to go ahead and address those issues.  The second element is scientific rigor itself.  So this really refers to experimental design.  In this case, you know, we’re asking applicants to make sure that they’re including enough information on whether the study is blinded, whether it’s randomized, how did they estimate sample size, how did they determine the effect size, those types of things.  The third element is called consideration of biological variables such as sex.  So in this case, we all know that, you know, health and weight and other biological variables can actually influence the outcome of experiments.  The fourth element is called authentication of key biological and/or chemical resources.  And this is really we want you to validate your resources, give us the plan for how you’re going to make sure that the cell line that you’re working with is actually, you know, the cell line that you think it is.  How do you know the specificity of your antibodies?  How do you know that the mice are of the correct genotype?  That type of thing.  So those are the four elements of the policy.


David Kosub:                 Great.  And for what research areas at NIH does this policy apply?


Dr. Patricia Valdez:        It applies to most research grant applications.  And that includes SBIR applications and it also includes Mentored Career Development Award applications. 


David Kosub:                 Great.  Can you tell us a bit more about how this policy is addressed in the application instructions?  In other words, where should researchers put this information as it relates to the key elements of the policy in their grant application?  And it would also be great if you could speak to this as it relates to the NIH guide notice published in September 2018, clarifying what is meant by rigor of prior research.


Dr. Patricia Valdez:        Right.  So this policy rolled out in 2016.  At that time the first element was referred to, it’s still referred to technically, as scientific premise.  And what we were finding is that applicants and reviewers were not getting the essence of what we were actually hoping.  The word premise is usually, people think of it as maybe it’s the hypothesis of the study.  But what we were really asking applicants to focus on was the rigor of the prior research.  And so, you know, starting with applications that come in for due dates on January 25, 2019 and beyond, now we’ve changed the term scientific premise.  Now we’re using the term rigor of the prior research.  So rigor of the prior research, formerly scientific premise, it’s still addressed under “significance.”  And in addition we’re asking that applicants also include their plans to address any of the weaknesses that they identify under “approach.”  So that means that rigor of the prior research is going to be appearing both under “significance” and under “approach.”  Scientific rigor itself, again that’s what we refer to for experimental design, that’s going to be addressed under the “approach.”  And again that’s where you want to include information on sample sizes, blinding randomization, analysis plans, those types of things.  Consideration of relevant biological variables, the third element, that’s also addressed under “approach.”  And then we have the fourth element, which is the authentication plan.  Now, that one comes under a separate unscored attachment.  We expect it to be just one page and it’s only about your plans for authentication.  We don’t want to see data there, really just the plans.  That’s all. 


David Kosub:                 Great.  Building on some of this, our research community would probably appreciate hearing a bit more about what resources NIH may have as it relates to enhancing rigor of one’s grant application.  Can you speak to some of these and where they may be found?


Dr. Patricia Valdez:        Sure.  Our reproducibility site.  We have a lot of really good, really good resources for applicants and reviewers also.  Right now we do have a lot of FAQs.  So if you have any questions, just check that out.  There’s SABV, or sex As a biological variable, resources there.  In a few weeks we’re going to be posting some examples of authentication plans.  Right now there’s some examples of scientific rigor in applications.  So you can find that there.  We’re also going to be including other resources, including something called the experimental design assistant that can help applicants with designing and calculating sample sizes for animal experiments.  So it is very useful resources and we’re going to be adding more resources there in the coming weeks. 


David Kosub:                 Great.  And now let’s move to the peer review process.  When a reviewer is evaluating an application’s scientific merit, how do they consider rigor and transparency as part of the overall impact score of the application? 


Dr. Patricia Valdez:        So the first three elements—so that’s again rigor of the prior research, scientific rigor, and consideration of biological variables—those three contribute to score.  Reviewers are asked to specifically look and score and base their score on, you know, how the applicant has described the rigor of the prior research, how they’re going to address the weaknesses.  We ask the reviewers to evaluate the experimental design and the analysis plans.  You know, have they presented robust strategies?  And then we also ask them to look at how the applicant has considered relevant biological variables.  We ask that they make sure that there’s a justification there if the applicant has decided to study only one sex.  Now for authentication—that’s not scored—does receive some brief comments from the reviewers.  It’s either acceptable or unacceptable.  And so the reviewers will comment and give a brief comment on the plans.  If the reviewers say that there’s a problem with the authentication plan, that doesn’t knock the application out of the water.  What will happen then if the application is otherwise meritorious, the program official can work with the PI to come up with an acceptable authentication plan at that point.


David Kosub:                 Great.  Now after the peer review process and you’ve gotten the grant, what should an awardee know to ensure that they comply with the annual reporting requirements outlined in the policy?


Dr. Patricia Valdez:        So in the RPPR, the progress report, awardees are asked to emphasize the approaches that they have taken in the past year or that they plan to take if there’s an additional year on the project to achieve robust and unbiased results.  So at that point the applicant can talk, you know, in detail about how they’re really, you know, emphasizing rigor in their studies.


David Kosub:                 Great.  And finally, from the administrative side here at NIH, what is a program officer looking at on the progress reports as it relates to this policy?  And are there times when they may reach out to an investigator to address rigor?


Dr. Patricia Valdez:        Yeah.  So when they review the progress report, they might decide that maybe the sample size, for instance, in the study was not adequate.  And so they may reach out to the investigators about their concerns about those types of things.  Also, for instance, if in the application they propose to study both sexes but in the progress report there’s only one sex that’s reported then the program official can go back to the PI and ask further questions.


David Kosub:                 Fantastic.  Thank you very much, Patricia.  We greatly appreciate your time on this.  And as a reminder, please do visit the NIH’s rigor and reproducibility website where you can find a wealth of information from application instructions to reviewer guidance, FAQs, training modules, infographics, and so much more.  So please check back regularly for the latest and greatest.  You can also look at the National Institute of General Medical Science’s website.  They have a great clearinghouse where you can find some additional information such as training modules on enhancing data reproducibility, experimental designs and authenticating cell lines.  This has been David Kosub with the NIH’s All About Grants.