Part 1. Overview Information

Key Dates

The application due date for this NOFO is New Date:December 13, 2024

No late applications will be accepted for this NOFO. All applications are due by 5:00 PM local time of applicant organization. 

Applicants are encouraged to apply early to allow adequate time to make any corrections to errors found in the application during the submission process by the due date.

Not Applicable

Generally, four months after receipt date

It is critical that applicants follow the Multi-Project (M) Instructions in the SF424 (R&R) Application Guide, except where instructed to do otherwise (in this NOFO or in a Notice from the NIH Guide for Grants and Contracts). Conformance to all requirements (both in the Application Guide and the NOFO) is required and strictly enforced. Applicants must read and follow all application instructions in the Application Guide as well as any program-specific instructions noted in Section IV and follow the AHRQ Grants Policy and Guidance found on the AHRQ website at http://www.ahrq.gov/funding/policies/nofoguidance/index.html.

When the program-specific instructions deviate from those in the Application Guide, follow the program-specific instructions.

Applications that do not comply with these instructions may be delayed or not accepted for review.

There are several options available to submit your application through Grants.gov to NIH and Department of Health and Human Services partners. You must use one of these submission options to access the application forms for this opportunity.

1. Use the NIH ASSIST system to prepare, submit and track your application online.
2. Use an institutional system-to-system (S2S) solution to prepare and submit your application to Grants.gov and eRA Commons to track your application. Check with your institutional officials regarding availability.
   
   



Table of Contents
Part 1. Overview Information
Key Dates
Part 2. Full Text of Announcement
Section I. Notice of Funding Opportunity Description
Section II. Award Information
Section III. Eligibility Information
Section IV. Application and Submission Information
Section V. Application Review Information
Section VI. Award Administration Information
Section VII. Agency Contacts
Section VIII. Other Information

Part 2. Full Text of Announcement

Section I. Notice of Funding Opportunity Description

This NOFO invites applications for a National Evaluation Center (NEC) to conduct an independent multimethod, rapid-cycle formative and summative program evaluation of AHRQ's Healthcare Extension Service. The NEC will: 1) develop data, measures, and methods to assess the range, reach, effectiveness, and equity of healthcare extension service models proposed and implemented by cooperative recipients; and 2) conduct an independent, multimethod rapid-cycle formative and summative program evaluation of the Healthcare Extension Service, and the technical assistance and other services provided by the National Coordinating Center (NCC). The goal of the NEC evaluation is to provide a detailed understanding of how healthcare extension services vary across Cooperative recipients, assess the range, reach, effectiveness, and equity of these models, identify contextual factors associated with recipient success, and document the barriers and facilitators to the delivery of healthcare extension services. As an integral part of the Healthcare Extension Service, the NEC shares rapid-cycle evaluation results and findings with AHRQ, the NCC, cooperative recipients, and the public.

The NEC will consist of three cores: 1) Data and Measurement, 2) Program Monitoring and Feedback, and 3) Program Assessment and Dissemination.

AHRQ’s Healthcare Extension Service will include the NEC, the State-based Healthcare Extension Cooperatives to Accelerate Implementation of Actionable Knowledge into Practice (U19) (referred to as Cooperatives) and the National Coordinating Center (NCC) for AHRQ's Healthcare Extension Service: State-based Solutions to Healthcare Improvement (U54).

This is one of three related NOFOs. Applicants are strongly encouraged to review the State-based Healthcare Extension Cooperatives to Accelerate Implementation of Actionable Knowledge into Practice (U19) NOFO and the National Coordinating Center (NCC) for AHRQ's Healthcare Extension Service: State-based Solutions to Healthcare Improvement (U54) NOFO to prepare applications for this NOFO.

Background:

It can take a long time for evidence-based, patient-centered improvements to be implemented, and there can be tremendous variation within and across healthcare delivery organizations and states in the quality and consistency of evidence-based care delivery. As seen in AHRQ’s National Healthcare Quality and Disparities Reports, the United States often struggles to deliver high-quality, evidence-based care, particularly for people with lower incomes and from certain racial and ethnic backgrounds who receive their care from safety net healthcare delivery organizations.

Barriers to the dissemination and implementation of PCOR evidence into healthcare delivery include: (1) policies and payment structures that do not align with implementing evidence-based care; (2) health information technology that does not produce real-time information that can be used for quality improvement or make data sharing across healthcare settings easy; (3) workforce issues such as shortages, inefficiencies, and lack of training; (4) limited coordination and technical support for practice transformation; and (5) limited connections between healthcare delivery organizations and other sectors, such as public health and community organizations. These challenges occur across the healthcare delivery system and significantly affect more resource-challenged safety net healthcare delivery organizations.

A promising path to accelerate the dissemination and implementation of patient-centered healthcare delivery improvements is to build relationships and implementation capacity within and across organizations. At the state level, there needs to be coordination and alignment of improvement efforts through a systematic process inclusive of affected stakeholders, including but not limited to Medicaid agencies, managed care organizations, safety net healthcare delivery organizations, tribal healthcare organizations, clinicians and staff, patients, and community-based organizations. On a more local level, improvements also include delivering targeted support to local healthcare delivery organizations, improving healthcare delivery processes, and providing training, education, and other types of assistance to organizations, clinicians, and staff.

AHRQ's Healthcare Extension Service, modeled after the U.S. Department of Agriculture Cooperative Extension System and incorporating principles from learning health systems, proposes to help healthcare policy, payment, and delivery organizations overcome implementation barriers and improve the quality of healthcare services for medically underserved people through state- or system- and local-level, evidence-based improvements.

AHRQ's Healthcare Extension Service is aligned with AHRQ's mission to produce evidence that improves healthcare so that it is safer, higher quality, more accessible, equitable, and affordable, and to work within the U.S. Department of Health and Human Services and with other partners to make sure that the evidence is understood and used in care improvement. It is aligned with AHRQ’s Patient Centered Outcomes Research Trust Fund Strategic Framework’s mission, goals, vision, and priorities. This includes early, ongoing, meaningful stakeholder engagement, addressing and measuring impacts on health equity, and supporting a wide range of healthcare delivery organizations.

The NEC will provide a detailed understanding of how healthcare extension services vary across recipients, assess the range, reach, effectiveness, and equity of these models, identify contextual factors associated with success, and document the barriers and facilitators to delivering healthcare extension services. 

Key Terms for this NOFO:

Logic Model: A graphic depiction of how an initiative or program is intended to work. A logic model documents the planned inputs and program activities along with the expected outputs and outcomes of those activities. A theory of change or theory of action describes the relationships between inputs, activities, and outputs and outcomes.

Multimethod Evaluation: Uses both qualitative and quantitative data collection and analytic methods to develop a rich and rigorous understanding of program implementation, process, and impact.

Rapid Cycle Evaluation: Provides formative evaluation information, allowing for real-time feedback in a timely and iterative manner for continuous program and implementation improvement, together with summative evaluation information.

Objectives and Scope:

This NOFO requires the NEC to conduct a multimethod, rapid-cycle program evaluation to determine the extent to which the Healthcare Extension Service, as implemented by the various cooperative recipients, accelerates the implementation of evidence-based improvements in healthcare policy, payment, and practice and reduces healthcare disparities, especially among Medicaid, uninsured, and other underserved populations.

National Evaluation Center Components: Applicants must demonstrate an understanding of the interrelationships between the Cores' activities and between the Cores and the other components of the Healthcare Extension Service (i.e., the cooperatives, and the NCC).   

The NEC will include the following components with the following responsibilities:

Data and Measurement Core: The Data and Measurement Core will be responsible for identifying or developing data, measures, and methods for documenting cooperative activities, including but not limited to documentation of implementation strategies, surveys, and other data collection instruments (such as interview and observation guides) for assessing implementation, process, and impacts on advancing health care equity in public health and healthcare delivery; measures and methods for documenting NCC activities; and secondary data sources (such as federal administrative data, claims data, and other data). The Data and Measurement Core will work with cooperatives to refine their initiative logic models, assess the evaluability of cooperative plans, and identify opportunities for measure harmonization and cross-collaborative evaluations. The Data and Measurement Core will provide technical expertise and leadership to the NCC-convened data and measurement workgroup. The Data and Measurement Core will identify and develop common measures and data collection approaches for assessing the implementation process and content for use by the cooperatives.

Program Monitoring and Feedback Core: The Program Monitoring and Feedback Core will work with NCC staff to obtain recipient self-monitoring data to inform quarterly rapid-cycle evaluation feedback to inform ongoing program improvements. The Program Monitoring and Feedback Core will be responsible for ensuring regular collection of data monitoring NCC activities. The Program Monitoring and Feedback Core will be responsible for conducting site visits to collect detailed implementation data to inform rapid-cycle evaluation reports and other feedback to support ongoing program improvements in cooperatives' functioning, progress, and success. For example, these reports and feedback may provide information on common challenges encountered by the cooperatives for the NCC to address. These site visits will be coordinated with NCC site visits.

Program Assessment and Dissemination Core: The Program Assessment and Dissemination Core will be responsible for identifying or developing multimethod rapid-cycle approaches to analyzing implementation, process, and impact data and for summarizing these data in formative and summative assessments. The Program Assessment and Dissemination Core will provide annual formative reports and other feedback to AHRQ, the NCC, and the cooperatives, as well as a summative assessment of the Healthcare Extension Service. The Program Assessment and Dissemination Core will be responsible for developing and implementing a plan that maximizes public access to data sets developed or compiled under this NOFO. See AHRQ's Public Access Policy for additional guidance: https://www.ahrq.gov/funding/policies/publicaccess/index.html

The NEC must:

Propose an approach for evaluation measures, methods, and strategies for gaining a detailed understanding of 1) how Healthcare Extension Services vary across Cooperative recipients; 2) the contextual factors associated with recipient success such as administrative, geographic, or organizational factors; and 3) the barriers and facilitators to Healthcare Extension service delivery.

Propose a rigorous approach for conducting multimethod, rapid-cycle formative, and summative program evaluations of the Healthcare Extension Service program to document and assess variation in the structure and success of the various cooperatives and how the activities of the NCC support the success of these cooperatives.    

The approach must include:

Using and refining the overall program and cooperative-level logic models to guide Extension Service - and initiative-level program evaluation efforts and to identify potential cross-recipient evaluation and learning opportunities.

Using primary and secondary data sources (such as federal and state administrative data, claims data, and other data, as well as cooperative recipient-collected data where appropriate) to formatively and summatively assess the Healthcare Extension program.

An implementation evaluation focused on the contextual and implementation factors associated with initial and ongoing cooperative development. The implementation evaluation must incorporate: 1) data collected by Cooperative recipients using measures determined by the NEC, including but not limited to the health system and practice environment and context within and across recipients, health system and practice-level team structures, electronic health record features relevant to evidence-informed quality improvements, health system and practice ownership patterns, health information exchange, and processes for monitoring work process changes; and 2) primary and secondary data collected by the NEC.A process evaluation focused on how cooperatives organize their work and deliver services, including but not limited to multistakeholder engagement, payment alignment efforts, workforce training, extension service delivery, implementation strategies, selection of improvement topics, funds distribution, and engagement with the NCC.

An impact evaluation of the Healthcare Extension Service focused on public health impacts and impacts on healthcare delivery, including but not limited to changes in work processes, workforce composition, patient, family, provider, and community experience of care that enhances trust in the healthcare system; health equity, patient-centeredness, and healthcare quality.

A process and effectiveness evaluation of NCC services supporting and aligning activities across the cooperatives.

Propose site visits (at least annually) to collect detailed implementation data to inform rapid-cycle evaluation reports and other feedback to support ongoing program improvements, including cooperatives' functioning, progress, and success.

Provide detailed plans for assessments and dissemination, including working with the NCC to obtain and use Cooperatives' monitoring data to inform quarterly rapid-cycle feedback to inform ongoing program improvement; providing formative reports and other feedback to the NCC and the Cooperatives; and providing a final summative assessment of the Healthcare Extension Service.  

Collaborate with the NCC to support the Healthcare Extension Service by participating in evaluation-focused learning network meetings and providing evaluation and program monitoring technical expertise to guide and align recipient self-monitoring and reporting.

Include detailed plans for disseminating formative and summative program evaluation findings and maximizing public access to data sets developed or compiled under this NOFO.

Overview of NOFO Requirements:

Applicants must describe their plans for the NEC core structure and how this structure will support the accomplishment of the evaluation objectives. They must also provide a detailed research plan following the instructions in Section IV below. 

See Section VIII. Other Information for award authorities and regulations.

Investigators proposing clinical trials may refer to the Research Methods Resources website for information about developing statistical methods and study designs.

Section II. Award Information

These projects are being funded pursuant to 42 USC 299b-37 which authorizes AHRQ to make these awards to support the agency's dissemination of comparative clinical effectiveness research findings.

All applications submitted and AHRQ grants made in response to this NOFO are subject to 45 CFR Part 75 (Uniform Administrative Requirements, Cost Principles and Audit Requirements for HHS Awards; https://www.ecfr.gov/cgi-bin/text-idx?node=pt45.1.75), the HHS Grants Policy Statement (see https://www.ahrq.gov/funding/policies/hhspolicy/index.html), and the terms and conditions set forth in the Notice of Award.

Section III. Eligibility Information

1. Eligible Applicants

Higher Education Institutions

• Public/State Controlled Institutions of Higher Education
• Private Institutions of Higher Education

The following types of Higher Education Institutions are always encouraged to apply for AHRQ support as Public or Private Institutions of Higher Education:

• Hispanic-serving Institutions
• Historically Black Colleges and Universities (HBCUs)
• Tribally Controlled Colleges and Universities (TCCUs)
• Alaska Native and Native Hawaiian Serving Institutions
• Asian American Native American Pacific Islander Serving Institutions (AANAPISIs)

Nonprofits Other Than Institutions of Higher Education

• Nonprofits with 501(c)(3) IRS Status (Other than Institutions of Higher Education)
• Nonprofits without 501(c)(3) IRS Status (Other than Institutions of Higher Education)

For-Profit Organizations

• Small Businesses
• For-Profit Organizations (Other than Small Businesses)

Local Governments

• State Governments
• County Governments
• City or Township Governments
• Special District Governments
• Indian/Native American Tribal Governments (Federally Recognized)
• Indian/Native American Tribal Governments (Other than Federally Recognized)

Federal Governments

• Eligible Agencies of the Federal Government
• U.S. Territory or Possession

Other

• Independent School Districts
• Public Housing Authorities/Indian Housing Authorities
• Native American Tribal Organizations (other than Federally recognized tribal governments)
• Faith-based or Community-based Organizations
• Regional Organizations

HHS grants policy requires that the grant recipient perform a substantive role in the conduct of the planned project or program activity and not merely serve as a conduit of funds to another party or parties. If consortium/contractual activities represent a significant portion of the overall project, the applicant must justify why the applicant organization, rather than the party(s) performing this portion of the overall project, should be the recipient and what substantive role the applicant organization will play. Justification can be provided in the Specific Aims or Research Strategy section of the PHS398 Research Plan Component sections of the SF424 (R&R) application.  There is no budget allocation guideline for determining substantial involvement; determination of substantial involvement is based on a review of the primary project activities for which grant support is provided and the organization(s) that will be performing those activities  

Foreign components, as defined in the HHS Grants Policy Statement via https://www.hhs.gov/sites/default/files/grants/grants/policies-regulations/hhsgps107.pdf may participate in projects as member of consortia or as subcontractors only.

Non-domestic (non-U.S.) Entities (Foreign Organization) are not eligible to apply.

Non-domestic (non-U.S.) components of U.S. Organizations are not eligible to apply.

Applicant organizations

Applicant organizations must complete and maintain the following registrations as described in the How to Apply- Application Guide to be eligible to apply for or receive an award. All registrations must be completed prior to the application being submitted. Registration can take 6 weeks or more, so applicants should begin the registration process as soon as possible.

• System for Award Management (SAM) – Applicants must complete and maintain an active registration, which requires renewal at least annually. The renewal process may require as much time as the initial registration. SAM registration includes the assignment of a Commercial and Government Entity (CAGE) Code for domestic organizations which have not already been assigned a CAGE Code.
  • NATO Commercial and Government Entity (NCAGE) Code – Foreign organizations must obtain an NCAGE code (in lieu of a CAGE code) in order to register in SAM.
  • Unique Entity Identifier (UEI) - A UEI is issued as part of the SAM.gov registration process. The same UEI must be used for all registrations, as well as on the grant application.
• eRA Commons - Once the unique organization identifier is established, organizations can register with eRA Commons in tandem with completing their Grants.gov registration; all registrations must be in place by time of submission. eRA Commons requires organizations to identify at least one Signing Official (SO) and at least one Program Director/Principal Investigator (PD/PI) account in order to submit an application.
• Grants.gov – Applicants must have an active SAM registration in order to complete the Grants.gov registration.

Program Directors/Principal Investigators (PD(s)/PI(s))

All PD(s)/PI(s) must have an eRA Commons account.  PD(s)/PI(s) should work with their organizational officials to either create a new account or to affiliate their existing account with the applicant organization in eRA Commons. If the PD/PI is also the organizational Signing Official, they must have two distinct eRA Commons accounts, one for each role. Obtaining an eRA Commons account can take up to 2 weeks.

Any individual(s) with the skills, knowledge, and resources necessary to carry out the proposed research as the Program Director(s)/Principal Investigator(s) (PD(s)/PI(s)) is invited to work with his/her organization to develop an application for support.  Individuals from diverse backgrounds, including underrepresented racial and ethnic groups, individuals with disabilities, and women are always encouraged to apply for AHRQ support. A minimum total effort for each Overall PD/PI or for each MPI is 20% effort for all functions served on the U19; effort cannot be reduced below this level during the entire project period.

For institutions/organizations proposing multiple PDs/PIs, visit the Multiple Program Director/Principal Investigator Policy and submission details in the Senior/Key Person Profile (Expanded) Component of the SF424 (R&R) Application Guide. The AHRQ multiple PDs/PIs policy can be found at https://grants.nih.gov/grants/guide/notice-files/NOT-HS-16-018.html.  

2. Cost Sharing

This NOFO does not require cost sharing.

While there is no cost sharing requirement included in this NOFO, AHRQ welcomes applicant institutions, including any collaborating institutions, to devote resources to this effort. An indication of institutional support from the applicant and its collaborators indicates a greater potential of success and sustainability of the project. Examples of institutional support would include: donated equipment and space, institutional funded staff time and effort, or other resource investments. Applicant institutions should indicate institutional support by outlining the specific contributions to the project and providing assurances that their organization and any collaborators are committed to providing these funds and resources to the project. This information can be included at the end of the budget justification section of the application, but institutional support dollars are not to be shown/included in the detailed budget request.

3. Additional Information on Eligibility

Number of Applications

Applicant organizations may submit more than one application, provided that each application is scientifically distinct.

AHRQ will not accept duplicate or highly overlapping applications under review at the same time. 

Section IV. Application and Submission Information

1. Requesting an Application Package

The application forms package specific to this opportunity must be accessed through ASSIST or an institutional system-to-system solution. A button to apply using ASSIST is available in Part 1 of this NOFO. See the administrative office for instructions if planning to use an institutional system-to-system solution.

2. Content and Form of Application Submission

It is critical that applicants follow the Multi-Project (M) Instructions in the How to Apply - Application Guide, except where instructed in this notice of funding opportunity to do otherwise and where instructions in the How to Apply - Application Guide are directly related to the Grants.gov downloadable forms currently used with most AHRQ opportunities. Conformance to the requirements in the How to Apply - Application Guide is required and strictly enforced. Applications that are out of compliance with these instructions may be delayed or not accepted for review.

Letter of Intent

Although a letter of intent is not required, is not binding, and does not enter into the review of a subsequent application, the information that it contains allows AHRQ staff to estimate the potential review workload and plan the review. 

 By the date listed in Part 1. Overview Information, prospective applicants are asked to submit a letter of intent that includes the following information:

• Descriptive title of proposed activity
• Name(s), address(es), and telephone number(s) of the PD(s)/PI(s)
• Names of other key personnel
• Participating institution(s)
• Number and title of this NOFO

The letter of intent should be sent to:

Jesse Crosson, PhD
Center for Financing, Access, and Cost Trends 
Agency for Healthcare Research and Quality
Email: AHRQ_HES@ahrq.hhs.gov

Page Limitations

All page limitations described in the How to Apply- Application Guide and the Table of Page Limits must be followed.

Instructions for the Submission of Multi-Component Applications

The following section supplements the instructions found in How to Apply- Application Guide and should be used for preparing a multi-component application.

The application should consist of the following components:

• Overall: required
• Data and Measurement Core: required, minimum/maximum of 1
• Program Monitoring and Feedback Core: required, minimum/maximum of 1
• Program Assessment and Dissemination Core: required, minimum/maximum of 1

Pre-application Technical Assistance (TA) Webinar

AHRQ encourages applicants to check the following webpage for resources for this NOFO:  AHRQ’s Healthcare Extension Service Initiative | Agency for Healthcare Research and Quality.  AHRQ encourages applicants to take advantage of any TA webinars that may be held by AHRQ program staff. Participation is optional and is not required to apply. Information about a TA webinar will be published through the NIH Guide for Grants and Contracts and posted on the AHRQ website.

Overall Component

When preparing the application, use Component Type ‘Overall’.

All instructions in the How to Apply - Application Guide must be followed, with the following additional instructions, as noted.

SF424(R&R) Cover (Overall)

Complete entire form.

PHS 398 Cover Page Supplement (Overall)

Follow standard instructions.

Research & Related Other Project Information (Overall)

Follow standard instructions.

Project/Performance Site Locations (Overall)

Enter primary site only.

A summary of Project/Performance Sites in the Overall section of the assembled application image in eRA Commons compiled from data collected in the other components will be generated upon submission.

Research and Related Senior/Key Person Profile (Overall)

Include only the Project Director/Principal Investigator (PD/PI) and any multi-PDs/PIs (if applicable to this NOFO) for the entire application.

A summary of Senior/Key Persons followed by their Biographical Sketches in the Overall section of the assembled application image in eRA Commons will be generated upon submission.

Budget (Overall)

The only budget information included in the Overall component is the Estimated Project Funding section of the SF424 (R&R) Cover.  A minimum total effort for each Overall PD/PI or for each MPI is 20% effort for all functions served on the U19; effort cannot be reduced below this level during the entire project period.

Travel: Applicants must budget for up to two key personnel to travel to the Washington, DC area once each year to meet with AHRQ staff, other recipients, and any invited guests for a two-day meeting. 

A budget summary in the Overall section of the assembled application image in eRA Commons compiled from detailed budget data collected in the other components will be generated upon submission.

PHS 398 Research Plan (Overall)

Specific Aims:

List the aims of the proposed U19 National Evaluation Center application. Summarize the expected outcome(s) of the U19 as a whole, including the impact that the results of the proposed project(s) will have on the Healthcare Extension Service. Briefly describe how the U19 is structured and how the Core(s) fit into the overall goals alone and together.

Research Strategy:

This section should summarize the structure and overall strategy of the National Evaluation Center. The Research Strategy should be organized into the subsections below and address the following areas: overall significance; overall innovation; overall approach; overall investigators; and overall environment. Specifics of the Cores should be described separately.

The Research Strategy must consist of the subsections A-E defined below:

• Subsection A. NEC Overview: Present a concise overall vision and plan for the proposed NEC. Describe why the proposed U19 is important in the context of the known challenges for the development of the Healthcare Extension Service program, the issues affecting the speed and success of dissemination and implementation of PCOR into practice, and how the Healthcare Extension Service and the NEC can contribute to improved understanding of these issues; the organization of safety net healthcare delivery and the challenges faced in this environment; and AHRQ’s goals for the Healthcare Extension Service program. Focus on the plans for the funding period of the NEC and highlight how the NEC approach will support ongoing learning and improvement within the Healthcare Extension Service, expand knowledge related to dissemination and implementation of PCOR evidence into practice, improve understanding of safety net healthcare delivery, and create resources to improve public understanding of improvement and care in these settings. Include plans to coordinate with ongoing evaluations of other federal and state efforts to improve care in the safety net.
• Subsection B. NEC Organization and Integration: Describe the organizational structure of the NEC. Explain how the components of the NEC, including key personnel, will interact to form a multi-disciplinary team focused on rapid-cycle evaluation of the Healthcare Extension Service. State the overall milestones of the NEC that are anticipated in the funding period. Describe how the NEC will collaborate with the U54 National Coordinating Center and the U19 state Cooperatives.
• Subsection C. Research Plan: Detail a comprehensive, multimethod, rapid-cycle program evaluation plan that includes regular, near real-time, formative, and summative reporting (e.g., evaluation briefs) to the NCC and cooperatives on cooperative processes and their effectiveness. The plan must include program monitoring and primary and secondary data while accounting for challenges in synthesizing across the cooperatives, given the likely heterogeneity of healthcare extension services delivered and the expected variation in quality improvement focus across recipients. The plan must include:
  • An implementation evaluation of the contextual and implementation factors associated with cooperative development and extension service delivery.
  • A process evaluation of how cooperatives organize their work and deliver services, including but not limited to multistakeholder collaboration (e.g., payment alignment efforts, workforce development, internal learning processes, coordination with federal initiatives and programs, process for developing health priorities); extension service delivery (e.g., service type, implementation strategies used, workforce characteristics), and local evaluation activities.
  • A process and effectiveness evaluation of NCC services and activities, including but not limited to technical assistance services, learning networks, and measurement and methods coordination.
  • An impact evaluation of the Healthcare Extension Service program including the public health impacts as well as impacts on healthcare delivery including but not limited to changes in work processes, workforce composition, policy, payment, patient and family experiences of care, and healthcare quality. This plan should consider the likely variation in the focus of state cooperative improvement initiatives.The comprehensive plan should demonstrate consideration of issues related to healthcare delivery in the safety net across all aspects of the planned evaluation and address the following research questions:
• Implementation:
  • What are the barriers and facilitators to initial and ongoing state cooperative development and extension service delivery?
  • What contextual factors are associated with cooperative development? How does context vary within and across recipients?
  • How do cooperatives tailor and adapt their extension services? What implementation strategies and extension services do cooperatives use and at what dosage?
• Process:
  • How do cooperatives select improvement topics? How do cooperatives use data on health disparities to inform this selection?
  • How do cooperatives identify partners? How do cooperatives engage with community and other partners over time?
  • What care settings and providers are selected for extension service support? How is this determined?
  • How do cooperatives engage with and perceive the value of services and functions of the NCC?
  • How do cooperatives identify workforce training needs and meet those needs for a diverse workforce?
• Impact:
  • To what extent are work process changes implemented to support evidence uptake? What are the common and divergent patterns of these changes across cooperative recipients? What are the key drivers of the observed changes?
  • What is the effect of cooperative activities on patient, family, provider, and community experience of care that enhances trust in safety net healthcare delivery?
  • To what extent do the cooperatives improve care delivery in the safety net?

Given the rapid-cycle nature of the evaluation and potential modification of the program over time in response to formative evaluation feedback, the application must describe a plan for maintaining flexibility and allowing for the adaptation of the evaluation approach to continue meeting the aims and objectives of the project over time.

The research plan should include a timeline of the proposed activities including how the activities conducted by each Core fit together. Each of the Core applications should provide a more detailed timeline of the specific activities of the Core.

• Subsection D. Investigators: Succinctly describe specific expertise of the Overall PDs/PIs and Core Leaders that makes them appropriate for their associated roles on the NEC team, without duplicating information in biosketches. For the Overall PD(s)/PI(s), describe prior experience leading multimethod, rapid-cycle program evaluation research endeavors. Describe how this selected team has collaborated in the past to accomplish rapid-cycle evaluation including implementation and process evaluations as well as procuring and using secondary data including federal data resources. Describe how the scientific, clinical, and technical expertise of the NEC team is complementary and will strengthen the Healthcare Extension Service program.
• Subsection E. Environment: Summarize the unique aspects of the participating organizations that are important to the success of the NEC. Describe how the scientific and technical resources of the NEC team will strengthen the Healthcare Extension Service program, without duplicating information in the facilities and resources sections.

Letters of Support:

Provide letters of support from the Institution(s) and partners that are included in the U19 National Evaluation Center that specify the nature of the collaborations and the roles of each partner. Applications must include letter(s) of support from an institutional official endorsing the proposed NEC and that describes the available institutional resources that will support the research. Letters from investigators who will serve as consultants or collaborators on the project, but with no measurable efforts, should also be included. Do not include letters from investigators who will have committed efforts in the application. 

Resource Sharing Plan:
Individuals are required to comply with the instructions for the Resource Sharing Plans as provided in the How to Apply - Application Guide.

Data Management Plan should NOT be submitted under Other Plans attachment, it is to be submitted under Resource Sharing plan as per AHRQ DMP policy.

Submission of a data management plan is required. AHRQ applicants are reminded to refer to NOT-HS-20-011: The Agency for Healthcare Research and Quality Data Management Plan Policy (https:/grants.nih.gov/grants/guide/notice-/NOT-HS-20-011.html for additional information on how to incorporate their data management plan into the resource sharing plan.

Appendix: Only limited Appendix materials are allowed. Follow all instructions for the Appendix as described in the How to Apply - Application Guide.

PHS Human Subjects and Clinical Trials Information (Overall)

When involving human subjects research, clinical research, clinical research and clinical trials follow all instructions for the PHS Human Subjects and Clinical Trials Information form in the How to Apply - Application Guide, with the following additional instructions:

If you answered “Yes” to the question “Are Human Subjects Involved?” on the R&R Other Project Information form, there must be at least one human subjects study record using the Study Record: PHS Human Subjects and Clinical Trials Information form or a Delayed Onset Study record within the application. The study record(s) must be included in the component(s) where the work is being done, unless the same study spans multiple components. To avoid the creation of duplicate study records, a single study record with sufficient information for all involved components must be included in the Overall component when the same study spans multiple components.

Study Record: PHS Human Subjects and Clinical Trials Information

All instructions in the How to Apply - Application Guide must be followed.

Delayed Onset Study

Note: Delayed onset does NOT apply to a study that can be described but will not start immediately (i.e., delayed start). All instructions in the How to Apply- Application Guide must be followed.

PHS Assignment Request Form (Overall)

All instructions in the How to Apply- Application Guide must be followed.

Data and Measurement Core

When preparing your application, use Component Type ‘Measurement.’

All instructions in the SF424 (R&R) Application Guide must be followed, with the following additional instructions, as noted.

SF424 (R&R) Cover (Data and Measurement Core)

Complete only the following fields:

• Applicant Information
• Type of Applicant (optional)
• Descriptive Title of Applicant’s Project
• Proposed Project Start/Ending Dates

PHS 398 Cover Page Supplement (Data and Measurement Core)

Research & Related Other Project Information (Data and Measurement Core)

Human Subjects: Answer only the ‘Are Human Subjects Involved?’ and 'Is the Project Exempt from Federal regulations?’ questions.

Project Narrative:  Do not complete. Note: ASSIST screens will show an asterisk for this attachment indicating it is required. However, eRA systems only enforce this requirement in the Overall component and applications will not receive an error if omitted in other components.

Project /Performance Site Location(s) (Data and Measurement Core)

List all performance sites that apply to the specific component.

Note: The Project Performance Site form allows up to 300 sites, prior to using additional attachment for additional entries.

Research & Related Senior/Key Person Profile (Data and Measurement Core)

• In the Project Director/Principal Investigator section of the form, use Project Role of ‘Other’ with Category of ‘Core Lead’ and provide a valid eRA Commons ID in the Credential field.
• In the additional Senior/Key Profiles section, list Senior/Key persons that are working in the component.
• Include a single Biographical Sketch for each Senior/Key person listed in the application regardless of the number of components in which they participate. When a Senior/Key person is listed in multiple components, the Biographical Sketch can be included in any one component.

If more than 100 Senior/Key persons are included in a component, the Additional Senior Key Person attachments should be used.  

Budget (Data and Measurement Core)

Budget forms appropriate for the specific component will be included in the application package.

Note: The R&R Budget form included in many of the component types allows for up to 100 Senior/Key Persons in section A and 100 Equipment Items in section C prior to using attachments for additional entries. All other SF424 (R&R) instructions apply.

PHS 398 Research Plan (Data and Measurement Core)

Specific Aims:  Succinctly describe the specific objectives and goals of the Data and Measurement Core. List the overall activities and services of the Data and Measurement Core. In addition, describe the Core's relationship to the NEC's goals and how the Core will support the goals of the NEC.   

Research Strategy:  The Data and Measurement Core has three main functions: 1) identifying or developing data, measures, and methods for documenting state cooperative and NCC activities; 2) providing technical expertise in support of cooperative-level evaluation efforts; and 3) supporting cooperative-level data collection efforts focused on the common measures identified by the Core for documenting implementation context, strategy, and impacts.

Data and Measurement approach:

• Describe the plans for leading work with the NCC and the cooperative recipients on aligning data collection efforts while minimizing burden on those providing data. Applicants should plan for submission of Paperwork Reduction Act clearance packages.
• Describe the approach for providing technical and methodological expertise. Applicants should plan for: ongoing participation in the NCC-convened data and measurement workgroup; providing technical and methodological expertise to this workgroup; and providing technical and methodological expertise to cooperative research partners in support of their work to develop and provide actionable data to cooperative members. In addition, the applicant should describe an approach for working with cooperative recipients to refine initial cooperative- and individual initiative-level logic models, identifying common measures relating to implementation and context across recipients, assessing evaluability, and where possible facilitating aggregation and synthesis of findings across local evaluations.
• Describe how the proposed activities and services of the Data and Measurement Core will ensure the accomplishment of the overall goals, objectives, and milestones of the NEC.
• Describe the role(s) of the Core Lead and key participants without duplicating information provided in the biosketches or the Budget Justification.
• Detail plans for data standardization, data collection, quality assurance, and data management. Applicants may assume that they will be provided with standardized data from recipients on key measures identified by the NEC awardee along with monitoring data from each cooperative on topics including but not limited to implementation context, strategies, and participant characteristics. As cooperatives will be required to cooperate and coordinate on data collection and reporting, the application should not include funds to compensate cooperatives for data collection and reporting activities.
• Describe the proposed processes and measures for documenting cooperative activities including but not limited to documentation of implementation strategies for the development of data resources for improved understanding of healthcare extension models.
• Describe proposed surveys and other data collection instruments (such as interview and observation guides) for assessing implementation and process. Include details on the anticipated frequency of data collection, target respondents, and approaches for determining adequate response rates.
• Describe the proposed measures and methods for documenting NCC activities.
• Describe the proposed methods for determining impacts on public health and healthcare delivery, including use of secondary data sources including but not limited to:  federal administrative data, medical claims data, Medicaid enrollment data, T-MSIS annual provider file, and Medicaid managed care data sources

Provide a detailed timeline of the Data and Measurement Core activities and how these relate to the overall NEC timeline. 

Letters of Support: Submit only with Overall Component

Resource Sharing Plan:

Individuals are required to comply with the instructions for the Resource Sharing Plans as provided in the SF424 (R&R) Application Guide, with the following modification:

Data Management Plan should NOT be submitted under Other Plans attachment, it is to be submitted under Resource Sharing plan as per AHRQ DMP policy.

Submission of a data management plan is required. AHRQ applicants are reminded to refer to NOT-HS-20-011: The Agency for Healthcare Research and Quality Data Management Plan Policy (https:/grants.nih.gov/grants/guide/notice-/NOT-HS-20-011.html for additional information on how to incorporate their data management plan into the resource sharing plan.

PHS Human Subjects and Clinical Trials Information (Data and Measurement Core)

When involving human subjects research, clinical research, and clinical trials follow all instructions for the PHS Human Subjects and Clinical Trials Information form in the SF424 (R&R) Application Guide, with the following additional instructions:

If you answered “Yes” to the question “Are Human Subjects Involved?” on the R&R Other Project Information form, you must include at least one human subjects study record using the Study Record: PHS Human Subjects and Clinical Trials Information form or a Delayed Onset Study record.

Study Record: PHS Human Subjects and Clinical Trials Information

All instructions in the SF424 (R&R) Application Guide must be followed

Delayed Onset Study

Note: Delayed onset does NOT apply to a study that can be described but will not start immediately (i.e., delayed start).All instructions in the SF424 (R&R) Application Guide must be followed

Program Monitoring and Feedback Core

When preparing your application, use Component Type ‘Monitoring.’

All instructions in the SF424 (R&R) Application Guide must be followed, with the following additional instructions, as noted.

SF424 (R&R) Cover (Program Monitoring and Feedback Core)

Complete only the following fields:

• Applicant Information
• Type of Applicant (optional)
• Descriptive Title of Applicant’s Project
• Proposed Project Start/Ending Dates

PHS 398 Cover Page Supplement (Program Monitoring and Feedback Core)

Research & Related Other Project Information (Program Monitoring and Feedback Core)

Human Subjects: Answer only the ‘Are Human Subjects Involved?’ and 'Is the Project Exempt from Federal regulations?’ questions.

Project Narrative:  Do not complete.

Project /Performance Site Location(s) (Program Monitoring and Feedback Core)

List all performance sites that apply to the specific component.

Note: The Project Performance Site form allows up to 300 sites, prior to using additional attachment for additional entries.

Research & Related Senior/Key Person Profile (Program Monitoring and Feedback Core)

• In the Project Director/Principal Investigator section of the form, use Project Role of ‘Other’ with Category of ‘Core Lead’ and provide a valid eRA Commons ID in the Credential field.
• In the additional Senior/Key Profiles section, list Senior/Key persons that are working in the component.
• Include a single Biographical Sketch for each Senior/Key person listed in the application regardless of the number of components in which they participate. When a Senior/Key person is listed in multiple components, the Biographical Sketch can be included in any one component.

If more than 100 Senior/Key persons are included in a component, the Additional Senior Key Person attachments should be used.   

Budget (Program Monitoring and Feedback Core)

Budget forms appropriate for the specific component will be included in the application package.

Note: The R&R Budget form included in many of the component types allows for up to 100 Senior/Key Persons in section A and 100 Equipment Items in section C prior to using attachments for additional entries. All other SF424 (R&R) instructions apply.

PHS 398 Research Plan (Program Monitoring and Feedback Core)

Specific Aims: Succinctly describe the specific objectives and goals of the Program Monitoring and Feedback Core. List the overall activities and services of the Program Monitoring and Feedback Core. In addition, describe the Core's relationship to the NEC's goals and how the Core will support the goals of the NEC.

Research Strategy: The function of the Program Monitoring and Feedback Core is to collect implementation and process data from cooperatives and the NCC and to provide near real-time feedback to inform rapid-cycle evaluation and ongoing program improvement. These data will also inform regular formative and summative reports created by the Program Assessment and Dissemination Core.

Program Monitoring and Feedback approach:

• Describe plans for program monitoring and data collection to support NEC goals and objectives. Applicants should describe an approach to conducting cooperative recipient site visits (including the number and frequency of these visits) and for coordinating the cadence and timing of these site visits with NCC and AHRQ staff to minimize burden on recipients while ensuring the rigorous collection of implementation and process data. Site visits should be conducted at least annually.
• Describe how program monitoring data (both from site visits and from recipient self-monitoring) will inform rapid-cycle evaluation and ongoing program improvement. Include proposed methods for rapid data analysis, quality assurance processes, and at least quarterly feedback of data including site visit and self-monitoring data to the NCC, cooperatives, and AHRQ to support ongoing improvement efforts.
• Describe how the proposed activities and services of the Program Monitoring and Feedback Core will ensure the accomplishment of the overall goals, objectives, and milestones of the NEC.
• Describe the role(s) of the Core Lead and key participants without duplicating information provided in the Biosketches or the Budget Justification.

Provide a detailed timeline of the Program Monitoring and Feedback Core activities and how these relate to the overall NEC timeline. 

Letters of Support: Submit only with the Overall Component.

Resource Sharing Plan:

Individuals are required to comply with the instructions for the Resource Sharing Plans as provided in the SF424 (R&R) Application Guide, with the following modification:

Data Management Plan should NOT be submitted under Other Plans attachment, it is to be submitted under Resource Sharing plan as per AHRQ DMP policy.

Submission of a data management plan is required. AHRQ applicants are reminded to refer to NOT-HS-20-011: The Agency for Healthcare Research and Quality Data Management Plan Policy (https:/grants.nih.gov/grants/guide/notice-/NOT-HS-20-011.html for additional information on how to incorporate their data management plan into the resource sharing plan.

PHS Human Subjects and Clinical Trials Information (Program Monitoring and Feedback Core)

When involving human subjects research, clinical research, and clinical trials follow all instructions for the PHS Human Subjects and Clinical Trials Information form in the SF424 (R&R) Application Guide, with the following additional instructions:

If you answered “Yes” to the question “Are Human Subjects Involved?” on the R&R Other Project Information form, you must include at least one human subjects study record using the Study Record: PHS Human Subjects and Clinical Trials Information form or a Delayed Onset Study record.

Study Record: PHS Human Subjects and Clinical Trials Information

All instructions in the SF424 (R&R) Application Guide must be followed

Delayed Onset Study

Note: Delayed onset does NOT apply to a study that can be described but will not start immediately (i.e., delayed start).All instructions in the SF424 (R&R) Application Guide must be followed

Program Assessment and Dissemination Core

When preparing your application, use Component Type ‘Assessment.’

All instructions in the SF424 (R&R) Application Guide must be followed, with the following additional instructions, as noted.

SF424 (R&R) Cover (Program Assessment and Dissemination Core)

Complete only the following fields:

• Applicant Information
• Type of Applicant (optional)
• Descriptive Title of Applicant’s Project
• Proposed Project Start/Ending Dates

PHS 398 Cover Page Supplement (Program Assessment and Dissemination Core)

Research & Related Other Project Information (Program Assessment and Dissemination Core)

Human Subjects: Answer only the ‘Are Human Subjects Involved?’ and 'Is the Project Exempt from Federal regulations?’ questions.

Project Narrative:  Do not complete. Note: ASSIST screens will show an asterisk for this attachment indicating it is required. However, eRA systems only enforce this requirement in the Overall component and applications will not receive an error if omitted in other components.

Project /Performance Site Location(s) (Program Assessment and Dissemination Core)

List all performance sites that apply to the specific component.

Note: The Project Performance Site form allows up to 300 sites, prior to using additional attachment for additional entries.

Research & Related Senior/Key Person Profile (Program Assessment and Dissemination Core)

• In the Project Director/Principal Investigator section of the form, use Project Role of ‘Other’ with Category of ‘Core Lead’ and provide a valid eRA Commons ID in the Credential field.
• In the additional Senior/Key Profiles section, list Senior/Key persons that are working in the component.
• Include a single Biographical Sketch for each Senior/Key person listed in the application regardless of the number of components in which they participate. When a Senior/Key person is listed in multiple components, the Biographical Sketch can be included in any one component.
• If more than 100 Senior/Key persons are included in a component, the Additional Senior Key Person attachments should be used.   

Budget (Program Assessment and Dissemination Core)

Budget forms appropriate for the specific component will be included in the application package.

Note: The R&R Budget form included in many of the component types allows for up to 100 Senior/Key Persons in section A and 100 Equipment Items in section C prior to using attachments for additional entries. All other SF424 (R&R) instructions apply.

PHS 398 Research Plan (Program Assessment and Dissemination Core)

Specific Aims: Succinctly describe the specific objectives and goals of the Program Assessment and Dissemination Core. List the overall activities and services of the Program Assessment and Dissemination Core. In addition, describe the Core's relationship to the NEC's goals and how the Core will support the goals of the NEC.   

Research Strategy:  The functions of the Program Assessment and Dissemination Core are to: 1) rigorously analyze data from multiple sources to assess program implementation, process, and impact; and 2) provide and disseminate regular reports to the NCC, cooperatives, AHRQ staff, and the public summarizing formative and summative assessments of implementation, process, and impact of the Healthcare Extension Service.

Program Assessment and Dissemination approach:

• Describe plans for identifying or developing multimethod rapid-cycle approaches to analyzing implementation, process, and impact data for formative and summative assessments. Include description of analytic frameworks. Incorporate rigorous qualitative methodologies such as grounded theory or phenomenology to understand how patients, clinicians, and families experience and engage with healthcare systems.
• Describe data collection methods (e.g., survey design, interviews, site visits, secondary data, ethnography, rapid ethnographic analysis) and management, including quality assurance processes, and how these different methods and sources contribute to rapid-cycle evaluation of complex multi-site evaluation efforts.
• Describe how the Program Assessment and Dissemination Core will provide formative and summative assessments. Include the cadence, proposed schedule, and potential content of these assessments.
• Describe the dissemination plans (including proposed schedule and potential content) along with quality assurance processes for both formative and summative reports.
• Applicants should include plans for attendance and participation at annual recipient meetings throughout the funding period.
• Provide a detailed plan for dissemination of evaluation findings and reports of self-monitoring data to cooperative, NCC, AHRQ, and external audiences.
• Provide plans for creation of public use datasets.
• Describe how the proposed activities and services of the Program Assessment and Dissemination Core will ensure the accomplishment of the overall goals, objectives, and milestones of the NEC.
• Describe the role(s) of the Core Lead and key participants without duplicating information provided in the Biosketches or the Budget Justification.

Provide a detailed timeline of the Program Assessment and Dissemination Core activities and how these relate to the overall NEC timeline. The timeline should account for the timing of related activities in other Cores including but not limited to expectations: regarding Paperwork Reduction Act package approvals; receipt of data from Cores, cooperatives, and secondary data sources; and AHRQ review of reports. 

Letters of Support: Submit only with Overall Component.

Resource Sharing Plan:

Individuals are required to comply with the instructions for the Resource Sharing Plans as provided in the SF424 (R&R) Application Guide, with the following modification:

Data Management Plan should NOT be submitted under Other Plans attachment, it is to be submitted under Resource Sharing plan as per AHRQ DMP policy.

Submission of a data management plan is required. AHRQ applicants are reminded to refer to NOT-HS-20-011: The Agency for Healthcare Research and Quality Data Management Plan Policy (https:/grants.nih.gov/grants/guide/notice-/NOT-HS-20-011.html for additional information on how to incorporate their data management plan into the resource sharing plan.

PHS Human Subjects and Clinical Trials Information (Program Assessment and Dissemination Core)

When involving human subjects research, clinical research, and clinical trials follow all instructions for the PHS Human Subjects and Clinical Trials Information form in the SF424 (R&R) Application Guide, with the following additional instructions:

If you answered “Yes” to the question “Are Human Subjects Involved?” on the R&R Other Project Information form, you must include at least one human subjects study record using the Study Record: PHS Human Subjects and Clinical Trials Information form or a Delayed Onset Study record.

Study Record: PHS Human Subjects and Clinical Trials Information

All instructions in the SF424 (R&R) Application Guide must be followed

Delayed Onset Study

Note: Delayed onset does NOT apply to a study that can be described but will not start immediately (i.e., delayed start).All instructions in the SF424 (R&R) Application Guide must be followed. 

3. Unique Entity Identifier and System for Award Management (SAM)

See Part 2. Section III.1 for information regarding the requirement for obtaining a unique entity identifier and for completing and maintaining active registrations in System for Award Management (SAM), NATO Commercial and Government Entity (NCAGE) Code (if applicable), eRA Commons, and Grants.gov.

4. Submission Dates and Times

Part I. contains information about Key Dates and times. Applicants are encouraged to submit applications before the due date to ensure they have time to make any application corrections that might be necessary for successful submission. When a submission date falls on a weekend or Federal holiday, the application deadline is automatically extended to the next business day.

Organizations must submit applications to Grants.gov (the online portal to find and apply for grants across all Federal agencies) using ASSIST or other electronic submission systems. Applicants must then complete the submission process by tracking the status of the application in the eRA Commons, NIH’s electronic system for grants administration. AHRQ and Grants.gov systems check the application against many of the application instructions upon submission. Errors must be corrected and a changed/corrected application must be submitted to Grants.gov on or before the application due date and time. If a Changed/Corrected application is submitted after the deadline, the application will be considered late. Applications that miss the due date and time are subjected to the NIH Grants Policy Statement Section 2.3.9.2 Electronically Submitted Applications.

Applicants are responsible for viewing their application before the due date in the eRA Commons to ensure accurate and successful submission.

Information on the submission process and a definition of on-time submission are provided in How to Apply- Application Guide.

5. Intergovernmental Review (E.O. 12372)

This initiative is not subject to intergovernmental review.

6. Funding Restrictions

For efficient grant administration, AHRQ grant administration procedures will be used and conducted in accordance with the terms and conditions, cost principles, and other considerations described in the HHS Grants Policy Statement. The Grants Policy Statement can be found at http://www.ahrq.gov/funding/grants/policies/hhspolicy.html.

Grant funds may NOT be used for the purchase of food of any kind (refreshments, meals, etc.). AHRQ will not award funds to support the cost of food.

These awards will not be made under expanded authorities; therefore, pre-award costs are not allowable

NOTE: No Cost Extensions will not be allowed.   

7. Other Submission Requirements and Information

Applications must be submitted electronically following the instructions described in the SF424 (R&R) Application Guide.  Paper applications will not be accepted.                                                     

Applicants must complete all required registrations before the application due date. Section III. Eligibility Information contains information about registration.

For assistance with your electronic application or for more information on the electronic submission process, visit How to Apply – Application Guide. If you encounter a system issue beyond your control that threatens your ability to complete the submission process on-time, you must follow the Dealing with System Issues guidance. For assistance with application submission, contact the Application Submission Contacts in Section VII.

Important reminders:

All PD(s)/PI(s) and component Project Leads must include their eRA Commons ID in the Credential field of the Senior/Key Person Profile form. Failure to register in the Commons and to include a valid PD/PI Commons ID in the credential field will prevent the successful submission of an electronic application to AHRQ.

The applicant organization must ensure that the unique entity identifier provided on the application is the same identifier used in the organization’s profile in the eRA Commons and for the System for Award Management. Additional information may be found in the SF424 (R&R) Application Guide.

See more tips for avoiding common errors.

The applicant should pay particular attention to the SF424 (R&R) application guide instructions (https://grants.nih.gov/grants/how-to-apply-application-guide/forms-f/general-forms-f.pdf) concerning letters of support. Letters of support, recommendation, or affirmation from any entity or individual not directly participating in the project should not be included.

Upon receipt, applications will be evaluated for completeness and compliance with application instructions by the Center for Scientific Review, and for responsiveness by AHRQ. Applications that are incomplete or non-compliant will not be accepted.

Institutional Review Board (IRB) approval of human subjects is not required prior to peer review of an application (see https://grants.nih.gov/grants/guide/notice-files/NOT-HS-00-003.html). However, initiation of IRB review, if necessary or applicable, is strongly encouraged to assure timely commencement of research.

Although there is no immediate acknowledgment of the receipt of an application, applicants are generally notified of the review and funding assignment within eight (8) weeks.

Please be sure that you observe the total cost, project period, and page number limitations specified above for this NOFO. Application processing may be delayed or the application may be rejected if it does not comply with these requirements

Post Submission Materials

Applicants are required to follow the instructions for post-submission materials, as described in the policy. Any instructions provided here are in addition to the instructions in the policy.

Priority Populations

AHRQ is committed to the inclusion of priority populations in health services research. The overall portfolio of health services research that AHRQ conducts and supports shall include the populations specifically named in AHRQs authorizing legislation: inner city; rural; low income; minority; women; children; elderly; and those with special health care needs, including those who have disabilities, need chronic care, or need end-of-life health care. 42 U.S.C. 299(c)(1). AHRQ also includes in its definition of priority populations those groups identified in Section 2(a) of Executive Order 13985 as members of underserved communities: Black, Latino, and Indigenous and Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of religious minorities; lesbian, gay, bisexual, transgender, and queer (LGBTQ+) persons; persons with disabilities; persons who live in rural areas; and persons otherwise adversely affected by persistent poverty or inequality.

AHRQ will broadly implement this inclusion policy across the research that AHRQ supports and conducts so that the portfolio of research is inclusive of all populations. AHRQ intends that these populations be included in studies such that the research design explicitly allows conduct of valid analyses. The policy applies to all grant applications. Investigators should review the document entitled, AHRQ Policy on the Inclusion of Priority Populations, which is available at https://grants.nih.gov/grants/guide/notice-files/NOT-HS-21-015.html. Applicants under this NOFO must consider and discuss including priority populations in research design as specified in this Notice.

Public Access to AHRQ-Funded Scientific Publications

Investigators should review the document titled AHRQ Announces new Policy for Public Access to AHRQ-Funded Scientific Publications, which is available at (http://grants.nih.gov/grants/guide/notice-files/NOT-HS-16-008.html). For all research arising from AHRQ support, this policy requires that AHRQ-funded authors submit an electronic version of the author's final peer-reviewed accepted manuscript to the National Library of Medicine's PubMed Central (PMC) to be made publicly available within 12 months of the publishers date of publication.

AHRQ Data Management Plan Policy

Investigators should review the document titled AHRQ Data Management Plan (DMP) Policy, which is available at (https://grants.nih.gov/grants/guide/notice-files/NOT-HS-20-011.html). This policy requires applicants for AHRQ new/competing grants and research contracts to include a DMP for managing, storing and disseminating the primary data, samples, physical collections, and other supporting materials created or gathered in the course of research funded by AHRQ, or state why data management is not possible, as a component of their grant application or research contract application.

Plan for Sharing Research Data

The precise content of the data-sharing plan will vary, depending on the data being collected and how the investigator is planning to share the data. Applicants who are planning to share data should describe briefly the expected schedule for data sharing; the format of the final dataset; the documentation to be provided; whether or not any analytic tools also will be provided; whether or not a data-sharing agreement will be required and, if so, a brief description of such an agreement (including the criteria for deciding who can receive the data and whether or not any conditions will be placed on their use); and the mode of data sharing (e.g., under its own auspices by mailing a disk or posting data on its institutional or personal website or through a data archive or enclave). Investigators choosing to share under their own auspices may wish to enter into a data-sharing agreement. References to data sharing may also be appropriate in other sections of the application.

The reasonableness of the data-sharing plan or the rationale for not sharing research data will be assessed by the reviewers. However, reviewers will not factor the proposed data-sharing plan into the determination of scientific merit or the priority score.

Data Confidentiality

The AHRQ confidentiality statute, 42 USC 299c-3(c), requires that information that is obtained in the course of AHRQ supported activities and that identifies individuals or establishments be used only for the purpose for which it was supplied. Information that is obtained in the course of AHRQ-supported activities and that identifies an individual may be published or released only with the consent of the individual who supplied the information or is described in it. There are civil monetary penalties for violation of the confidentiality provision of the AHRQ statute. 42 USC 299c-3(d). In the Human Subjects section of the application, applicants must describe procedures for ensuring the confidentiality of the identifying information to be collected (see NOT-HS-18-012: Confidentiality in AHRQ-Supported Research). The description of the procedures should include a discussion of who will be permitted access to this information, both raw data and machine-readable files, and how personal identifiers and other identifying or identifiable data will be restricted and safeguarded. Identifiable patient health information collected by recipient under this NOFO will also be obtained and managed in accordance with the HIPAA Privacy Rule, 45 CFR Parts 160 and 164.

The recipient should ensure that computer systems containing confidential data have a level and scope of security that equals or exceeds that established by the HIPAA Security Rules if applicable (see HIPAA website in prior paragraph) and that established by the Office of Management and Budget (OMB) in OMB Circular No. A-130, Appendix III - Security of Federal Automated Information Systems. The applicability and intended means of applying these confidentiality and security standards to subcontractors and vendors, if any, should be addressed in the application.

Sharing Research Resources: Rights in Data

Unless otherwise provided in grant awards, AHRQ recipients may copyright, or seek patents for, as appropriate, final and interim products and materials developed in whole or in part with AHRQ support, including, but not limited to, methodological tools, measures, software with documentation, literature searches, and analyses. Such copyrights and patents are subject to a royalty-free, non-exclusive, and irrevocable AHRQ license to reproduce, publish, use, or disseminate for any purpose consistent with AHRQs statutory responsibilities and to authorize others to do so for any purpose consistent with AHRQ’s statutory responsibilities. In accordance with its legislative dissemination mandate, AHRQ purposes may include, subject to statutory confidentiality protections, making project materials, databases, results, and algorithms available for verification or replication by other researchers. In addition, subject to AHRQ budget constraints, final products may be made available to the health care community and the public by AHRQ or its agents if such distribution would significantly increase access to a product and thereby produce substantial or valuable public health benefits. Ordinarily, to accomplish distribution, AHRQ publicizes research findings but relies on recipients to publish research results in peer-reviewed journals and to market grant-supported products. AHRQ requests that recipients notify the Office of Communications (OC) when an AHRQ-funded research article has been accepted for publication in a peer-reviewed journal. Researchers should submit manuscripts that have been accepted for publication in a peer-reviewed journal to JournalPublishing@ahrq.hhs.gov at least four to six weeks in advance of the journal’s expected publication date.

Regulations applicable to AHRQ recipients concerning intangible rights and copyright can be found at 45 CFR 75.322.

Section V. Application Review Information

1. Criteria

The mission of AHRQ is to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable, and to work with the U.S. Department of Health and Human Services (HHS) and other partners to make sure that the evidence is understood and used.

AHRQ's priority areas of focus are detailed here: https://www.ahrq.gov/funding/policies/nofoguidance/index.html.

As part of this mission, applications are submitted to AHRQ to support health services research which are evaluated for scientific and technical merit through the AHRQ peer review system.

Applications that are complete and responsive to the NOFO will be evaluated for scientific and technical merit by an appropriate objective group convened in accordance with standard AHRQ peer-review procedures that are described in 42 CFR Part 67, Subpart A. Incomplete and/or non-responsive applications or applications not following instructions given in this NOFO will not be reviewed. Only the review criteria described below will be considered in the review process.

Merit Review Criteria: Merit Review Criteria, as described below, will be considered in the review process.

A proposed Clinical Trial application may include study design, methods, and intervention that are not by themselves innovative but address important questions or unmet needs. Additionally, the results of the clinical trial may indicate that further clinical development of the intervention is unwarranted or lead to new avenues of scientific investigation.

Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the project proposed).

Reviewers will consider each of the review criteria below in the determination of scientific merit and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Significance

Does the project address an important problem or a critical barrier to progress in the field? Is the prior research that serves as the key support for the proposed project rigorous? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

• How will the proposed rapid-cycle evaluation produce information useful for ongoing improvement of the Healthcare Extension Service program?
• How will the proposed evaluation provide information useful to others seeking to spread the Healthcare Extension Service approach and disseminate and implement PCOR evidence into practice? 

In addition, for applications involving clinical trials

Are the scientific rationale and need for a clinical trial to test the proposed hypothesis or intervention well supported by preliminary data, clinical and/or preclinical studies, or information in the literature or knowledge of biological mechanisms? For trials focusing on clinical or public health endpoints, is this clinical trial necessary for testing the safety, efficacy or effectiveness of an intervention that could lead to a change in clinical practice, community behaviors or health care policy? 

Investigator(s)

Are the PD(s)/PI(s), collaborators, and other researchers well suited to the project? If Early Stage Investigators or those in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

• Are the PD(s)/PI(s), Core Leaders, collaborators, and other researchers appropriately trained and well qualified for the project?
• Do the investigators have expertise in procuring and using secondary data such as Medicaid data sources including but not limited to enrollment, administrative data, Transformed Medicaid Statistical Information System (T-MSIS) annual provider file, claims, and Medicaid managed care data sources?
• Do the investigators have experience conducting formative multimethod rapid-cycle evaluations and experience with collecting actionable and targeted data and using the information collected to provide quick and credible feedback for adaptive and responsive development of ongoing program activities?
• Do the investigators have technical expertise on measurement and data collection for documenting contextual and implementation factors to identify and address state- and system-level issues that affect implementation?
• Do the investigators and other team members have appropriate skills and experience for their roles in complex, collaborative project settings involving coordination across multiple, diverse sites and organizations, community, and state partners?
• Are the total hours described for all key personnel appropriate to the level of effort required for the performance of their responsibilities? Does the applicant include staff with sufficient dedicated time for collaboration and coordination with the NCC?
• Are the Core Leads appropriately trained and qualified for their roles and are they sufficiently engaged for conducting the required work and collaborating across the NEC?
• Does the applicant demonstrate sufficient technical and methodological expertise needed to lead the NCC-convened data and measurement workgroup?
• Does the applicant ensure that the proposed investigative team has no overlap regarding investigators and direct cost funding with any State Cooperative or National Coordinating Center application.

In addition, for applications involving clinical trials

With regard to the proposed leadership for the project, do the PD/PI(s) and key personnel have the expertise, experience, and ability to organize, manage and implement the proposed clinical trial and meet milestones and timelines? Do they have appropriate expertise in study coordination, data management and statistics? For a multicenter trial, is the organizational structure appropriate and does the application identify a core of potential center investigators and staffing for a coordinating center?

Innovation

• Are the concepts, approaches, methodologies, measurement instruments, data collection techniques, and analysis plans rigorous and follow accepted scientific practices?
• Does the evaluation benefit from the use of state-of-the-art methods in creating a robust, multimethod program evaluation that is likely to produce results that are meaningful and valid?   

In addition, for applications involving clinical trials

Does the design/research plan include innovative elements, as appropriate, that enhance its sensitivity, potential for information or potential to advance scientific knowledge or clinical practice?

Approach

Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Have the investigators included plans to address weaknesses in the rigor of prior research that serves as the key support for the proposed project? Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? Have the investigators presented adequate plans to address relevant biological variables, such as sex, for studies in human subjects?

If the project involves human subjects and/or clinical research, are the plans to address:

1) the protection of human subjects from research risks, and
2) inclusion (or exclusion) of individuals on the basis of sex/gender, race, and ethnicity, as well as the inclusion or exclusion of individuals of all ages (including children and older adults), justified in terms of the scientific goals and research strategy proposed?

• To what extent does the proposed approach rigorously fulfill the requirement for production of an independent multimethod rapid-cycle formative and summative evaluation of the Healthcare Extension Service?
• To what extent does the proposed approach support ongoing feedback for continuous program improvement, including allowing for rapid identification of opportunities for correction and improvement and timely action on that information?
• Does the application provide a rigorous and thorough approach to both qualitative and quantitative measurement and data collection?

In addition, for applications involving clinical trials

Does the application adequately address the following, if applicable

Study Design

Is the study design justified and appropriate to address primary and secondary outcome variable(s)/endpoints that will be clear, informative and relevant to the hypothesis being tested? Is the scientific rationale/premise of the study based on previously well-designed preclinical and/or clinical research? Given the methods used to assign participants and deliver interventions, is the study design adequately powered to answer the research question(s), test the proposed hypothesis/hypotheses, and provide interpretable results? Is the trial appropriately designed to conduct the research efficiently? Are the study populations (size, gender, age, demographic group), proposed intervention arms/dose, and duration of the trial, appropriate and well justified?

Are potential ethical issues adequately addressed? Is the process for obtaining informed consent or assent appropriate? Is the eligible population available? Are the plans for recruitment outreach, enrollment, retention, handling dropouts, missed visits, and losses to follow-up appropriate to ensure robust data collection? Are the planned recruitment timelines feasible and is the plan to monitor accrual adequate? Has the need for randomization (or not), masking (if appropriate), controls, and inclusion/exclusion criteria been addressed? Are differences addressed, if applicable, in the intervention effect due to sex/gender and race/ethnicity?

Are the plans to standardize, assure quality of, and monitor adherence to, the trial protocol and data collection or distribution guidelines appropriate? Is there a plan to obtain required study agent(s)? Does the application propose to use existing available resources, as applicable?

Data Management and Statistical Analysis

Are planned analyses and statistical approach appropriate for the proposed study design and methods used to assign participants and deliver interventions? Are the procedures for data management and quality control of data adequate at clinical site(s) or at center laboratories, as applicable? Have the methods for standardization of procedures for data management to assess the effect of the intervention and quality control been addressed? Is there a plan to complete data analysis within the proposed period of the award?

Environment

Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

Overall Impact - Data and Measurement

Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the project proposed).

Scored Review Criteria - Data and Measurement

Reviewers will consider each of the review criteria below in the determination of scientific merit and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Approach

Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Have investigators included plans to address weaknesses in the rigor of prior research that serves as the key support for the proposed project? Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?  Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility, and will particularly risky aspects be managed? Have the investigators presented adequate plans to address relevant biological variables, such as sex, for human subjects?

• Are the plans for data standardization, data collection, quality assurance, and data management rigorous and appropriate for the proposed approach?
• Does the applicant demonstrate an understanding of the use of logic models to guide program evaluation?
• Are the proposed processes and measures for documenting cooperative activities likely to improve understanding of the Healthcare Extension Service as delivered?
• Are the proposed data collection instruments and approaches appropriate for rigorously meeting the objectives of the NEC?

Overall Impact - Program Monitoring and Feedback

Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the project proposed).

Scored Review Criteria - Program Monitoring and Feedback

Reviewers will consider each of the review criteria below in the determination of scientific merit and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Approach

Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Have investigators included plans to address weaknesses in the rigor of prior research that serves as the key support for the proposed project? Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?  Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? Have the investigators presented adequate plans to address relevant biological variables, such as sex, for human subjects?

• Is the proposed approach to site visit data collection appropriately designed to collect rigorous implementation and process data while minimizing burden on state cooperative recipients? Does this approach rely on appropriate frameworks to guide data collection?
• Are the proposed methods for rapid data analysis, quality assurance processes, and feedback of site visit and self-monitoring data to the NCC, cooperatives, and AHRQ effectively designed to support ongoing program improvement efforts?

Overall Impact - Program Assessment and Dissemination

Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the project proposed).

Scored Review Criteria - Program Assessment and Dissemination

Reviewers will consider each of the review criteria below in the determination of scientific merit and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Approach

Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Have investigators included plans to address weaknesses in the rigor of prior research that serves as the key support for the proposed project? Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?  Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? Have the investigators presented adequate plans to address relevant biological variables, such as sex, for human subjects?

• Is the approach to rapid-cycle analysis of data for formative and summative assessments rigorous and informed by appropriate analytic frameworks?
• Are the proposed data collection methods appropriate to rapid-cycle evaluation of complex multi-site projects?
• Is the proposed dissemination approach appropriate for the timely provision of formative and summative assessments?
• Do the plans for dissemination of findings effectively inform both internal and external audiences about key findings and results of the project? Is the plan for creation and distribution of public use datasets appropriately informed by data use policies?

In addition, for applications involving clinical trials

If proposed, are the administrative, data coordinating, enrollment and laboratory/testing centers, appropriate for the trial proposed?

Does the application adequately address the capability and ability to conduct the trial at the proposed site(s) or centers? Are the plans to add or drop enrollment centers, as needed, appropriate?

If international site(s) is/are proposed, does the application adequately address the complexity of executing the clinical trial?

If multi-sites/centers, is there evidence of the ability of the individual site or center to: (1) enroll the proposed numbers; (2) adhere to the protocol; (3) collect and transmit data in an accurate and timely fashion; and, (4) operate within the proposed organizational structure?

Additional Review Criteria - Overall

As applicable for the project proposed, reviewers will evaluate the following additional items while determining scientific and technical merit, and in providing an overall impact score, but will not give separate scores for these items.

Data Management Plan

The reviewers will comment on whether the Data Management Plan is reasonable.   

Inclusion of Priority Populations

Peer reviewers will assess the adequacy of plans to address the needs of AHRQ priority populations.

Peer reviewers must include their assessment of the proposed inclusion plan for priority populations in evaluating the overall scientific and technical merit of the application and assigning the impact score.

In evaluating the overall impact of the application, the review groups will:

• Evaluate the application for the presence or absence of the inclusion plan based on the proposed research objectives.
• Evaluate the adequacy of the proposed plan for the inclusion of priority populations.
• Evaluate the proposed justification for the exclusion of priority populations when a requirement for inclusion is described as inappropriate with respect to the purpose of the research.
• Evaluate the plans for outreach and recruitment of study participants, including priority populations, where appropriate.
• Evaluate the proposed plan for study design, execution, and outcome assessments so that study results will be relevant to one or more priority populations, where appropriate.
• Assess the plan as being acceptable or unacceptable with regard to the appropriateness of the inclusion or exclusion of priority populations in the proposed research.

Degree of Responsiveness

Reviewers will assess how well the application addresses the purpose and objectives of this NOFO. How responsive is the application to the special eligibility criteria, including the project requirements, noted in the NOFO?

Budget and Period of Support

The committee will evaluate whether the proposed budget is reasonable, and whether the requested period of support is appropriate in relation to the proposed research.

Resubmissions

Not applicable

Renewals

Not applicable

Revisions

Not applicable

Additional Review Considerations - Overall

As applicable for the project proposed, reviewers will consider each of the following items, but will not give scores for these items, and should not consider them in providing an overall impact score.

Applications from Foreign Organizations

Not Applicable

2. Review and Selection Process

Applications that are complete and responsive to the NOFO will be evaluated for scientific and technical merit by an appropriate Scientific Review Group convened in accordance with standard AHRQ peer review procedures that are described in 42 CFR Part 67, Subpart A. Incomplete and/or non-responsive applications or applications not following instructions given in this NOFO will not be reviewed.

As part of the scientific peer review, all applications:

• May undergo a selection process in which only those applications deemed to have the highest scientific and technical merit (generally the top half of applications under review) will be discussed and assigned an overall impact score.
• Will receive a written critique.

Applications may undergo a selection process in which only those applications deemed to have the highest scientific and technical merit (generally the top half of applications under review) will be discussed and assigned an overall impact score.

The following will be considered in making funding decisions:

• Scientific and technical merit of the proposed project as determined by scientific peer review.
• Availability of funds.
• Responsiveness to goals and objectives of the NOFO.
• Relevance and fit within AHRQ research priorities, as well as overall programmatic and geographic balance of the proposed project to program priorities.

3. Anticipated Announcement and Award Dates

After the peer review of the application is completed, the PD/PI will be able to access their Summary Statement (written critique) via the eRA Commons. Refer to Part 1 for dates for peer review, advisory council review, and earliest start date.

Information regarding the disposition of applications is available in the HHS Grants Policy Statement (see https://www.ahrq.gov/funding/policies/hhspolicy/index.html).

Section VI. Award Administration Information

1. Award Notices

If the application is under consideration for funding, AHRQ Division of Grants Management staff will request "Just-In-Time" information from the applicant. Just-In-Time information generally consists of information on other support, any additional information necessary to address administrative and budgetary issues, and certification of IRB approval of the project's proposed use of human subjects. For details regarding IRB approval, applicants may refer to the "AHRQ Revised Policy for Institutional Review Board (IRB) Review of Human Subjects Protocols in Grant Applications" (https://grants.nih.gov/grants/guide/notice-files/not-hs-00-003.html). Applicant should also be aware of the AHRQ policy for use of single IRB for cooperative research, 45 CFR 46.114 (b)” https://grants.nih.gov/grants/guide/notice-files/NOT-HS-20-005.html".

If all administrative and programmatic issues are resolved, a formal notification in the form of a Notice of Award (NoA) will be provided to the applicant organization for successful applications. The NoA signed by the AHRQ grants management officer is the authorizing document and will be sent via email to the e-mail address designated by the recipient organization during the eRA Commons registration process.

Recipients must comply with any funding restrictions described in Section IV.6. Funding Restrictions.

Selection of an application for award is not an authorization to begin performance. Any costs incurred before receipt of the NoA are at the recipient's risk. These costs may be reimbursed only to the extent considered allowable pre-award costs.

Any application awarded in response to this NOFO will be subject to the UEI, SAM Registration, and Transparency Act requirements.

Individual awards are based on the application submitted to, and as approved by, the AHRQ and are subject to the AHRQ-specific terms and conditions identified in the NoA.

2. Administrative and National Policy Requirements

All AHRQ grant and cooperative agreement awards are subject to HHS’s “Uniform Administrative Requirements, Cost Principles, and Audit Requirements for HHS Awards,” (http://www.ecfr.gov/cgi-bin/text-idx?node=pt45.1.75, and the HHS Grants Policy Statement (see http://www.ahrq.gov/funding/policies/hhspolicy/index.html), and the terms and conditions set forth in the Notice of Award.

All applicants will be subject to a term and condition that applies the terms of 48 CFR section 3.908 to the award and requires that recipients inform their employees in writing of employee whistleblower rights and protections under 41 U.S.C. 4712 in the predominant native language of the workforce.

As necessary, additional Terms and Conditions will be incorporated into the NoA.

If you are successful and receive a Notice of Award, in accepting the award, you agree that the award and any activities thereunder are subject to all provisions of 45 CFR Part 75, currently in effect or implemented during the period of the award, other Department regulations and policies in effect at the time of the award, and applicable statutory provisions.

If you receive an award, HHS may terminate it if any of the conditions in 2 CFR 200.340(a)(1)-(4) are met. No other termination conditions apply.

There is a prohibition on certain telecommunications and video surveillance services or equipment that became effective on or after August 13, 2020.

If you receive an award, you must follow all applicable nondiscrimination laws. You agree to this when you register in SAM.gov. You must also submit an Assurance of Compliance (HHS-690). To learn more, see the HHS Office for Civil Rights website.

 Below are available HHS resources on Health Literacy:

• HHS Health.gov: Health Literate Care Model
• AHRQ: Health Literacy Universal Precautions Toolkit

For additional guidance regarding how the provisions apply to AHRQ grant programs, please contact the Scientific/Research Contact that is identified in Section VII under Agency Contacts of this NOFO.

In accordance with the statutory provisions contained in Section 872 of the Duncan Hunter National Defense Authorization Act of Fiscal Year 2009 (Public Law 110-417), AHRQ awards will be subject to the Federal Recipient Performance and Integrity Information System (FAPIIS) requirements. FAPIIS requires Federal award making officials to review and consider information about an applicant in the designated integrity and performance system (currently FAPIIS) prior to making an award. An applicant, at its option, may review information in the designated integrity and performance systems accessible through FAPIIS and comment on any information about itself that a Federal agency previously entered and is currently in FAPIIS. The Federal awarding agency will consider any comments by the applicant, in addition to other information in FAPIIS, in making a judgement about the applicant’s integrity, business ethics, and record of performance under Federal awards when completing the review of risk posed by applicants as described in 45 CFR Part 75.205 “Federal awarding agency review of risk posed by applicants.” This provision will apply to all AHRQ grants and cooperative agreements except fellowships.

The following special terms of award are in addition to otherwise applicable OMB administrative guidelines, DHHS grant administration regulations at 45 CFR Parts 75, and other DHHS, PHS, and AHRQ grant administration policy statements. AHRQ will use these procedures in evaluating and administering this cooperative agreement.

The administrative and funding instrument used for this program is the cooperative agreement. A cooperative agreement is an "assistance" mechanism (rather than an "acquisition” mechanism), in which substantial AHRQ programmatic involvement with the recipient is anticipated during the performance of the activities. Under a cooperative agreement, the AHRQ purpose is to support and stimulate the recipients' activities by involvement in and otherwise working jointly with the award recipient in a partnership role; it is not to assume direction, prime responsibility, or a dominant role in the activities. Consistent with this concept, the dominant role and prime responsibility resides with the grant recipient and the PD(s)/PI(s) for the project as a whole, although specific tasks and activities may be shared between the recipient and AHRQ as described in the NOFO. Cooperative activities are intended to strengthen the individual recipient activities through the facilitation of data sharing, data access and communications.

All cooperative activities that include significant government involvement will require prior approval by AHRQ.

Activities conducted under this award that involve the collection of information e.g., conducting surveys or requesting responses to uniform questions from nine or more persons, establishments or other entities, with certain exceptions, are currently required to be cleared by OMB under the Paperwork Reduction Act (PRA). (44 USC 3501-3521). PRA review and approval is required in cooperative agreements if AHRQ has significant input or control over the data collection activity. Submissions for clearance under the PRA are through AHRQ and HHS. Therefore, impacted recipients should include the time in their proposed timelines to develop materials and receive necessary clearances. It typically takes at least 6 months from date of initial submission to AHRQ and sometimes much longer if submissions are incomplete or the justification for the proposed data collection plans are questioned during the clearance process. Information collection that requires PRA clearance may be obtained. Detailed information on the PRA can be found at Paperwork Reduction Act (PRA) of 1995 | Guidance Portal (hhs.gov).

 Program Director/Principal Investigator (PD/PI) Responsibilities:

The PD(s)/PI(s) will have the primary responsibility for conducting research in accordance with the terms and conditions of the Notice of Award and cooperating with other key parties, including the AHRQ Program Official, the NCC and the State Cooperative recipients. This includes but is not limited to the following:

• Determine approaches, designing, and setting project milestones and implementing the project plan for the NEC.
• Share issues with AHRQ related to evaluating the Healthcare Extension Service program.  
• Work collaboratively with the NCC and the Cooperatives.
• Participate in group activities, including program-wide meetings and communications.
• Collaborate with Cooperatives and NCC in the publication and dissemination of program results and the eventual release to the scientific and healthcare communities of methods, tools, results, and other resources.
• The PD/PI will participate in monthly virtual meetings with the program official and other AHRQ personnel as appropriate. The PI will attend an annual two-day meeting with AHRQ and other recipients, which is held in the Washington, D.C. area, unless otherwise determined.
• Provide key personnel for participation in the NCC-convened learning network meetings focused on evaluation.
• In addition to the annual progress report, the recipient is required to submit quarterly progress reports to AHRQ. The recipient is required to submit a program logic model within the first six months of award. Instructions on reporting requirements will be provided at the time of award.

AHRQ Responsibilities:

AHRQ staff have substantial programmatic involvement during the period of the cooperative agreement that is above and beyond the normal stewardship role in awards, as described below:

Cooperative activities are intended to strengthen individual projects and generate collaboration across projects. Specific cooperative aims and activities in carrying out individual projects may be shared among grant recipients and AHRQ program officials. AHRQ staff will provide advice and support to recipients and will assist in cooperative work of the project beyond the usual program stewardship for grants.

AHRQ will:

• Work with the NEC Director/PI(s) to ensure the objectives of the program are being met.
• Provide the NEC technical assistance, advice, and coordination; however, the role of the AHRQ staff will be to facilitate and not to direct activities.
• Coordinate between the NEC, State-based Healthcare Extension Services Cooperatives recipients, and NCC recipient when appropriate.
• Monitor NEC progress towards meeting project milestones and adherence to the strategic goals of the program; through a variety of means such as regular communications with NEC recipient and review of project implementation plans.
• Review and provide input on written documents, including information and materials for the activities conducted, prior to submission for publication or public dissemination.
• Work closely with grant recipients and other federal partners to ensure dissemination of products.
• Participate with the award recipient in the dissemination of project findings, best practices, and lessons learned from the project.
• Serve as resources for, or interfacing with, other ongoing AHRQ activities that may be relevant to the activities in the NEC to avoid duplication and facilitate collaboration and communication in overlapping areas.
• Additionally, AHRQ program staff will be responsible for the normal scientific and programmatic stewardship of the award and will be named in the award notice.

Collaborative Responsibilities

Applicants must commit to collaborating with AHRQ, other recipients and other recipients as described in Section l. The awardee and AHRQ will also participate in periodic meetings including monthly teleconferences and annual in-person meetings.

For details regarding annual progress report submission, refer to https://www.ahrq.gov/funding/grant-mgmt/noncomp.html. If instructions on the AHRQ website are different from the RPPR Instructions, follow the instructions on the AHRQ website. Annual progress reports are due 3 months before the start date of the next budget period of the award. Recipients are required to submit expenditure data on the Federal Financial Report (FFR; SF 425) annually.

Expenditure data is to be reported on the Federal Financial Report (FFR; SF 425). AHRQ requires annual financial expenditure reports for ALL grant programs as described in the HHS Grants Policy Statement (https://www.ahrq.gov/funding/policies/hhspolicy/index.html). AHRQ implementation of the FFR retains a financial reporting period that coincides with the budget period of a particular project. However, the due date for annual FFRs is 90 days after the end of the calendar quarter in which the budget period ends. For example, if the budget period ends 4/30/2023, the annual FFR is due 9/30/2023 (90 days after the end of the calendar quarter of 6/30/2023).

A final Progress Report, final Federal Financial Report, and Final Invention Statement are required when an award ends. All final reports are due within 120 days of the project period end date. For further details regarding grant closeout requirements, refer to http://www.ahrq.gov/funding/grant-mgmt/index.html#Closeout.

The Federal Funding Accountability and Transparency Act of 2006 (Transparency Act), includes a requirement for recipients of Federal grants to report information about first-tier subawards and executive compensation under Federal assistance awards issued in FY2011 or later. All recipients of applicable AHRQ grants and cooperative agreements are required to report to the Federal Subaward Reporting System (FSRS) available at www.fsrs.gov on all subawards over the threshold. See the 2 CFR Part 170 (https://www.ecfr.gov/current/title-2/subtitle-A/chapter-I/part-170) for additional information on this reporting requirement.

In accordance with the regulatory requirements provided at 45 CFR 75.113 and 2 CFR Part 200.113 and Appendix XII to 45 CFR Part 75 and 2 CFR Part 200, recipients that have currently active Federal grants, cooperative agreements, and procurement contracts from all Federal awarding agencies with a cumulative total value greater than $10,000,000 for any period of time during the period of performance of a Federal award, must report and maintain the currency of information reported in the System for Award Management (SAM) about civil, criminal, and administrative proceedings in connection with the award or performance of a Federal award that reached final disposition within the most recent five-year period. The recipient must also make semiannual disclosures regarding such proceedings. Proceedings information will be made publicly available in the designated integrity and performance system (currently FAPIIS). This is a statutory requirement under section 872 of Public Law 110-417, as amended (41 U.S.C. 2313). As required by section 3010 of Public Law 111-212, all information posted in the designated integrity and performance system on or after April 15, 2011, except past performance reviews required for Federal procurement contracts, will be publicly available. Full reporting requirements and procedures are found in Appendix XII to 45 CFR Part 75 and 2 CFR Part 200 – Award Term and Condition for Recipient Integrity and Performance Matters.

Recipient performance will be measured based on success in the following Program goals:

Data and Measurement:

Performance measures:

• How many NCC-convened work group meetings were attended?
• How many Cooperatives were engaged with logic model refinement?
• Timeline: report annually.

Monitoring and Feedback:

Performance measures:

• How many site visits were conducted? What were the goals of these site visits?
• What program monitoring measures were identified? How often were these data collected?
• How many formative reports were delivered? Which audiences were engaged?
• Timeline: report annually.

Assessment and Dissemination:

• How many evaluation reports were provided?
• How many dissemination activities were conducted? Which audiences were engaged?
• How many public access datasets were produced? How were these disseminated?
• Timeline: report annually.

Section VII. Agency Contacts

We encourage inquiries concerning this funding opportunity and welcome the opportunity to answer questions from potential applicants.

eRA Service Desk (Questions regarding ASSIST, eRA Commons, application errors and warnings, documenting system problems that threaten submission by the due date, and post-submission issues)

Finding Help Online: https://www.era.nih.gov/need-help  (preferred method of contact)
Telephone: 301-402-7469 or 866-504-9552 (Toll Free)

Grants.gov Customer Support (Questions regarding Grants.gov registration and Workspace)
Contact Center Telephone: 800-518-4726
Email: support@grants.gov

Jesse Crosson, PhD
Center for Financing, Access, and Cost Trends
Agency for Healthcare Research and Quality 
Email: AHRQ_HES@ahrq.hhs.gov

Examine your eRA Commons account for review assignment and contact information (information appears two weeks after the submission due date).

Xavier Bogle, PhD
Scientific Review Officer
Division of Scientific Review 
Office of Extramural Research, Education, and Priority Populations 
Agency for Healthcare Research and Quality 
Email: DSR@ahrq.hhs.gov

Janene Dyson
Office of Management Services
Division of Grants Management
Agency for Healthcare Research and Quality
Telephone: 301-427-1491
Email: janene.dyson@ahrq.hhs.gov

Section VIII. Other Information

Recently issued AHRQ policy notices may affect your application submission. A full list of policy notices published by AHRQ is provided in the NIH Guide for Grants and Contracts. Notices can also be found at AHRQ Grants Policy Notices (https://www.ahrq.gov/funding/policies/hhspolicy/index.html).

This program is described in the Assistance Listings (formerly called the Catalog of Federal Domestic Assistance) at https://sam.gov/content/home and is not subject to the intergovernmental review requirements of Executive Order 12372 or Health Systems Agency review.

Awards are made under the authority of 42 USC 299 et seq., and in accordance with 45 CFR Part 75 and other referenced applicable statutes and regulations. All awards are subject to the terms and conditions, cost principles, and other considerations described in the HHS Grants Policy Statement. The HHS Grants Policy Statement can be found at http://www.ahrq.gov/funding/policies/hhspolicy/index.html