NIH All About Grants Podcast
Inclusion Plans: Application Review and Post-Award
>> From the National Institutes of Health in Bethesda, Maryland. This is all about grants.
[ Music Playing ]
>> Kosub: Hello and welcome to another virtual edition of NIH's All About Grants Podcast. I'm your host, David Kosub with the NIH Office of Extramural Research. And we are back for part two of our conversation on inclusion plans. In our earlier conversation, we discussed applications and what you should be thinking about as it relates to that. In this conversation, we're going to be looking at the review process and post award. And I am glad to say that we have with us Dawn Corbett. She serves as the NIH inclusion policy officer here at the Office of Extramural Research, NIH. And I welcome you to the show.
>> Corbett: Thanks, David.
>> Kosub: So jumping in from the application, how do I actually know, How does someone actually know that the inclusion plan that they put in there is scientifically appropriate?
>> Corbett: So that's a good question because in your inclusion plan, you not only need to describe the population that you're including, but you also need to think about whether or not it's actually appropriate. And you need to explain why we don't have, you know, hard and fast rules about how something is scientifically appropriate. But a good place to start is usually considering the prevalence or incidence of the population that actually has the condition. And I might ask myself if my study population looks different than the people with the condition, can I explain why? And you should explain why if it whether or not it looks different. Inclusions are explained based on a scientific or ethical reason. So, for example, if you're doing Alzheimer's research, you may not be including children in your study, and that's okay. But you should state that don't assume that the reviewers are going to know something they may not be familiar with the area of science that you're talking about. So you make sure the exclusion is based on a scientific or ethical reason. So what does that look like? For example, maybe the condition does not occur in a specific group like the case I mentioned with children in Alzheimer's disease. You know, maybe an age specific study is warranted or perhaps there's safety or ethical reasons for not including certain groups. So you'll want to think about that and make sure that that's explained in your inclusion plan. You also want to think about the analyses you're doing. So if you're doing any subgroup analyses, you want to make sure that those are adequately powered. And then finally, you want to make sure it's realistic. So it may be, you know, technically, scientifically appropriate, but if it's not something that you can actually do, then it would not be appropriate. So you should provide some evidence about how you're predicting your study population as well, not just what it should look like, but you should get into the details in your inclusion plan about why you think your population will look like it will based on your recruitment method. So you may include information about, you know, people who've historically participated in your study and your institution. If you're recruiting at certain clinics, for example, you may talk about the demographics of those clinics and think about who's actually likely to participate in your study. So those are the things you want to think about in terms of whether or not your plan is scientifically appropriate. And then you also want to think about, you know, how are you going to continue to assess recruitment and retention and make modifications, which we'll talk about a little bit later in the podcast.
>> Kosub: So you mentioned that reviewers in your answer, they're going to be looking at these plans to assess their appropriateness as well based on those factors as well. Can you tell us more about like how they're looking at it from their standpoint?
>> Sure. So reviewers are going to be looking we have guidelines for the review of inclusion on the basis of sex or gender, race, ethnicity and age in clinical research. Those are publicly available on our website. And they detail the things that peer review is going to consider as they're looking at your application. So they're going to be looking first is a complete did you provide all the information that you were supposed to provide? So plans for inclusion of women, racial, ethnic minorities and individuals across the lifespan. Sometimes people miss one or more of those. And did you include the table so that they have enough information in they're going to be looking at, is it scientifically appropriate? So that gets back to what we just talked about. They're going to be trying to understand why you picked the study population, and did you provide enough information for them to understand that? And then the last thing, as I touched on briefly, is it realistic? So, you know, have you provided enough information so that they understand that you actually can recruit and retain the population that you said you would?
>> Kosub: And what about like when these reviewers come back with, you know, their responses on, you know, on your application and there's some, you know, information as it relates to your inclusion plans. Perhaps the application did not get funded, but there's some advice on how to improve it. Like what? What kind of steps would you suggest to recommend, you know, an applicant take when responding to these reviewers on their inclusion plans?
>> Corbett: I should mention that in your summary statement there maybe there will there will be some information about occlusion if you're doing clinical research. And included in that are a gender minority and age codes, which are going to appear on your summary statement in addition to your comments. So how you respond depends on a couple of things. The first is I would look at your gender minority codes if reviewers find those acceptable or unacceptable. Peer review actually has to go through the plans and assign them a code based on the demographics of your population. They'll be a number, for example, one through five, which will explain the demographics of your population. And then it gets an area code which says, Is this something we find unacceptable, acceptable or unacceptable? if you have a U code? The application cannot be funded until this is resolved. So in the case of a U code, each IC has their own processes for resolving these. And typically that's going to involve discussion with your program officer about revising this plan. So they will likely be reaching out to you during the just in time process and you'll need to work with them to try to make this plan acceptable. Now, in some cases, you may not actually get an unacceptable code, but peer review may include a comment about inclusion that is not necessarily a bar to funding, but the process is going to be similar that your program officer may reach out to you and ask you to address this for your concerns. One thing to keep in mind is at this point, some adjustments are more difficult. So example, you know, changing the budget or the scope of your application is going to be difficult at this point in the process. So it really is best to address any potential issues before you submit your application. But if you do end up with an unacceptable code or there are comments, it doesn't necessarily mean the application can't be funded, but you will need to address those concerns prior to award.
>> Kosub: Okay. Great. That that's definitely helpful information. And if I may, switching kind of going from the application and post and review to potentially getting the award and then doing once progress reporting on the actual project itself and the inclusion of the participants in the study. How do these inclusion plans, I guess, relate to progress reporting? Do you have to talk about each step you're making along the way?
>> Corbett: Yes, you do. So in each progress report, which is going to be at least annual, you're going to need to provide updated enrollment information. So this will be provided on the inclusion enrollment report, the provided updated and cumulative enrollment. And then if you're doing an defined Phase three clinical trial, you'll also need to talk about the status of any analysis that you're doing by sex or gender, race and ethnicity. In addition, if you're doing, for example, delayed onset studies, these may be studies that you described in your application, but you did not have the full information available for them because they may be starting a little later. For example, if there's any of those studies that you did not provide plans for in the application, you would need to provide those in the progress report as well.
>> Kosub: Thanks for that and I'll plug another couple of podcasts in this one. We had some on valid analysis, stratified analysis and also on delayed onset stuff. So check out those podcasts too. I can always plug it when I can. So participant enrollment is not going the way you kind of hoped it was planned to go. How do you address that in your progress reports? And, you know, how are we looking at it from the NIH side?
>> Corbett: Yeah. You know, I briefly touched on earlier, it's really important to be continually assessing the enrollment and making sure that it's on target, not just your overall enrollment, but your enrollment by sex or gender, race, ethnicity and age. And detecting those problems early, your P.O. is going to need to assess that in your progress report. So when you submit that each year, we'll take a look at it and see if the enrollment is appropriate and on target. If it's not likely, we'll be reaching out to you to talk about how you might make some modification to be back on target, figure out what's going on. It's always better to reach out to your P.O. proactively about these things. So if you are having challenges, you may be able to provide you resources. They have experience with studies, not meeting enrollment goals, and they can give you some suggestions about what to do. So please don't hesitate to reach out to them. But if you've not and your PO contacts you after you submit your progress report, they may ask you, for example, for corrective action plan. They may ask you to provide an interim report, you know, so that you can do more frequent check ins about your enrollment. in more severe cases when the issues are severe or they are something that have not been corrected over time, this can also result in additional enforcement action by NIH in accordance with our public health service policies. So could ultimately end up with orderly phase out suspension, termination or withholding of support.
>> Kosub: Great. It's all very helpful information. And before we close out, Dawn, you have any final thoughts as it relates to review and or progress report or post award for inclusion as it relates to inclusion plans, perhaps beyond what you've mentioned already?
>> Corbett: Sure. I mean, one thing that I will say is that today we're talking about inclusion plans, but it's really important to consider inclusion throughout your entire application. So while certain things are going to be in your inclusion plan, think about inclusion throughout your study design. Are your inclusion or exclusion criteria too narrow? Do your outcomes reflect the concerns of participants? Do the risks of excluding certain groups outweigh their benefits? If you keep a diverse study populations in mind throughout the application process, it'll really help you to have a more successful study.
>> Kosub: Well, that's great. Thank you very much, Dawn. Truly appreciate this opportunity to hear more about inclusion plans during the review process and post award. If you want more information, as Don mentioned, check out the website for inclusion policies at the grants site, the NIH.gov site. There's a wealth of information there. And with that this has been David Kosub with the NIH's All about grants. Thank you.