NIH All About Grants Podcast

Discussing Not Discussed

Announcer: From the National Institutes of Health in Bethesda, Maryland, this is All About Grants.

David Kosub: Hello and welcome to another edition of NIH's All About Grants podcast. I'm your host David Kosub with the NIH's Office of Extramural Research. And today, we're going to be touching on a topic that's probably near and dear to a lot of people's hearts and that's you've gone through this whole process of applying, you've submitted your application, it went in for review, but it wasn't discussed. What do we actually do with that? How do you take that information and move forward? And today, we're going to be discussing Not Discussed, and I'm glad to say we have with us Dr. Gene Carstea. He is a Review Branch Chief within the NIH�s Center for Scientific Review, and welcome to the show, Gene.

Gene Carstea: Hey David, thanks for having me. I'm excited to be here. You know, I always welcome the chance to talk about peer review and be able to peel back the curtain on what happens in study section. So thanks again.

David Kosub: Sure. Well, let's look behind that curtain. Let's start with a setting the stage question, just can you give us a brief overview of a study section where some applications, you know, may be discussed and some may not be.

Gene Carstea: Sure. Well, you know, I always find it helpful to take one step back and paint the big picture of NIH peer review, and in this case, how discussed and not discussed applications are a part of our process. So for- everyone knows, for over a century, the NIH has been supporting meritorious research with the ambitious goal of advancing basic science, translational science, as well as clinical practices in science. And we know that it's through study sections that we identify, the NIH identifies meritorious research. So to give you an idea of the scope, at the CSR, we have about 200 chartered study sections covering all the scientific and health topics that NIH supports. We review on the order of about 60,000 applications a year, so it's no slouch there. In each study section, they're designed to evaluate somewhere between 70 and 90 applications. And we do that three times a year. I know that's a lot of numbers but this is in part a numbers game. So for each application evaluated, the study section will generate an advisory report, an advisory report that we call summary statements, which touches upon really two overall evaluative themes. One is the importance of the research goals, and if accomplished, what's the potential for high impact for this research? We call that "Should it be done." And then it's assessing the likelihood that the research goals can even be accomplished, and we refer to that as "Can it be done." So reviewers will combine these two evaluative outcomes and generate an overall impact score. So back to the topic of today. How do we generate a discussed and a not discussed list? Again, it's a numbers game. So for 70 to 90 applications, which rise to any one study section, the SRO makes review assignments. And in a typical study section, we have three reviewers assigned to each of these applications. The reviewers will have between four and six weeks to generate their critiques, including a preliminary overall impact score. About a week before the study section meeting, the SRO will rank the applications based on the average of the three preliminary overall impact scores. And the top 50% of the applications will be discussed, leaving the bottom 50% to be on that dreaded not discussed list.

David Kosub: Should it be done and can it be done is like the motto I should start living my life by. But jumping to that numbers game, that 50% number, is anything special about that number?

Gene Carstea: So generally speaking, the top scoring applications are the ones that are seriously considered for support by the funding institutes. The top 10, 20, and sometimes even more percentage are considered to be awarded by the funding institute. So since CSR does not directly award research grants, our evaluations are provided to the institutes so that they can make these award decisions. Since it would be impractical and actually quite restrictive, both in time and energy, to discuss all applications in a study section meeting, we try to rigorously discuss, you know, two and a half to up to five times the number of applications that would even be considered for awarding by the funding institutes. And so for that reason, we discuss 50% of the applications. And this is a number that has been discussed between CSR and the funding institutes that deem to be a reasonable amount to have discussed in the meeting.

David Kosub: But it's not like a hard and fast rule, like there may be some flexibility or some differences in different study sections in that 50%.

Gene Carstea: No, it�s a general standard. The 50% is the general standard amongst study section meetings, but, you know, there is exceptions. So, you know, if after lining up the applications by their overall impact score and seeing a number of tied applications with the same score, that kind of straddle that 50% line. Study sections may actually discuss more than 50% of the applications that are there, particularly, you know, if these are deemed as competitive applications or potentially competitive. Also, if an application's average score places it beneath that 50% line, it may still be discussed during a study section meeting, if a reviewer in the study section, and this could be either an assigned reviewer or an unassigned reviewer, thinks that any given application could be competitive, even though it's sitting below the line, it can be identified and it could be called up for review. It's called sometimes rescuing an application. So we see it as not getting caught up in the process and we can't forget why we're here. We're here to identify applications which have the potential to significantly advance both national and global science.

David Kosub: Rescue, that reminds me of the old Aretha Franklin Song of "Rescue Me. Take my application in your arms." [LAUGHS]. I'm sorry, I couldn't resist. So get to like some nuts and bolts type question, is there anything different in the review process for application that's not discussed compared to say one that is?

Gene Carstea: Well, you know, it's in that not discussed list, and the simple answer is if the score placed it in the bottom half. So the real question is what makes an application get that lower score? So we come back to the evaluation of "should it be done" and "can it be done," and for "should it be done," for any application, you know, any one reviewer may ask, you know, will this contribute to a better understanding of basic foundational science? Or will it impact the way we diagnose illness or establish effective healthcare programs? Or will it provide insight into human disease? Or will it lead to a better understanding of human health, or fill a clinical or a scientific gap? So these are the things that we consider in terms of importance of the topic. Then we rely on the experts to provide their informed judgment whether an investigator can accomplish these goals through the experimental approach that they propose. So it's the combination of these two basic standards which results in scores that may place it, you know, either above or below that 50% threshold.

David Kosub: I see. So if someone does have a not discussed application, but they have some like questions, who should they contact?

Gene Carstea: Oh, it's definitely the program officers. Before the meeting and talking about the venue of an application being assigned, you talked to the SRO or you talked to the chief of the review branch. After the meeting, the applicant should wait for their summary statement, and when they get their summary statement, they should be able to discuss it with their assigned program officer.

David Kosub: And say, like, someone, you know, has some questions that, you know, maybe they feel that their application which was not discussed, you know, actually should have been discussed. You know, does someone have any recourse to kind of say, "Hey, I would really like this application to be considered?"

Gene Carstea: Well, I guess you may be asking, you know, if they feel like it's been treated fairly during the review process in a study section, and fairness is actually at the top of our list. Every application needs to be treated fairly and equitably discussed or not discussed. So if an applicant feels that the procedural norms and the guidelines were not completely followed, they should be able to contact their program officer and go over their summary statement to go over all the reasons why it may have not been discussed. The PO can answer these questions and help them out in understanding that. If they still feel that it was not properly evaluated, the NIH has a robust appeal process and they can consider challenging what the study section has done. There's four main categories, evidence of bias, conflict of interest, lack of appropriate expertise, as well as factual errors. It's important that these individuals understand that the official letter of appeal is sent to the program officer and it's handled within the funding institute. It's also important to note that in addition to the standard appeal process, CSR also provides a reporting avenue on our side as well. If they feel that there was bias associated with this review or it was somehow otherwise unfair, applicants can contact the CSR directly, and what they do is they contact our Chief Diversity Officer. We try to make it easy for them. So the contact information is under every email signature of members of CSR, and it's also located on the description page of every study section. All the allegations of bias or otherwise unfair review are investigated by CSR's senior management. And if we agree that there was some issues associated with that review, it will be re-reviewed that round for consideration during that council round. And it thus circumvents the need for an appeal. And obviously if we don't agree with them, the appeal process can go on as planned.

David Kosub: Glad to hear there's definitely some, you know, avenues that folks can consider. You know, I totally appreciate this hearing more about the discussing not discussed. Before we head out, I always like to leave the opportunity for our guests to kind of mention anything that they think the audience should hear again or should learn more about. Is there anything you'd like to leave our audience with about, you know, not discussed topic?

Gene Carstea: Well, you know, the NIH receives a ton of applications. Many of them, I would say most of them are quite strong. So if your application is not discussed, it does not imply that it's a weak application. You know, you want to try to understand- try to understand the criticisms that came out of the review. Was it important? Was it significant? Was it the approach that was the problem? Was it your message? Was it not crystal clear to the reviewers? So we say revise and resubmit. Applications that have been ND-ed, not discussed, during one cycle, have come back and have scored well enough to be awarded in subsequent meetings. So it's the mission of the NIH to lead the global scientific effort to advance health and health related science. And the NIH relies upon the strength of the community because we're all partners in this together.

David Kosub: Here here. Well, Gene, thank you for taking the time to discuss Not Discussed with us. All this information has been very helpful. And for those listening, if you would like some more information, please don't hesitate to check out the information that's available on the NIH Office of Extramural Research, our grants site, as well as information that's available through the NIH's Center for Scientific Reviews website. There's a lot of information, so please check it out. This has been David Kosub with NIH�s All About Grants. Thank you.