It is critical that applicants follow the instructions in the SF424 (R&R) Application Guide, except where instructed to do otherwise (in this FOA or in a Notice from the NIH Guide for Grants and Contracts). Conformance to all requirements (both in the Application Guide and the FOA) is required and strictly enforced. Applicants must read and follow all application instructions in the Application Guide as well as any program-specific instructions noted in Section IV, and follow the AHRQ Grants Policy and Guidance found on the AHRQ website at http://www.ahrq.gov/funding/policies/foaguidance/index.html. When the program-specific instructions deviate from those in the Application Guide, follow the program-specific instructions. Applications that do not comply with these instructions may be delayed or not accepted for review.

Part 1. Overview Information
Part 2. Full Text of the Announcement
Section I. Funding Opportunity Description
Section II. Award Information
Section III. Eligibility Information
Section IV. Application and Submission Information
Section V. Application Review Information
Section VI. Award Administration Information
Section VII. Agency Contacts
Section VIII. Other Information

Purpose and Background Information

The growing availability of clinical and health data coupled with an emerging cadre of advanced, sophisticated statistical approaches developed by health services researchers can facilitate a paradigm shift in health care delivery that can rapidly improve the quality and safety of care. The Learning Health System is a model in which both institutional and clinical practice partners are actively engaged and participate in the generation, adoption, and application of evidence. The Learning Health System (LHS) is a new and rapidly evolving conceptual approach in which science (generated from health services research (HSR), patient-centered outcomes research (PCOR), and clinical research), informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the care process, in which stakeholders (i.e., providers, patients and families) are active participants in all elements, and new knowledge is captured as an integral by-product of the care experience (IOM, 2012). A Learning Health System Researcher is an individual who is embedded within a health system and collaborates with its stakeholders to produce novel insights and evidence that can be rapidly implemented to improve the outcomes of individual and populations and health system performance.

Current HSR/PCOR and clinical research training programs are focused largely on supporting knowledge generation (Forrest et al, BMC HSR; 2009) but do not specifically support the conduct of research within Learning Healthcare Systems (LHSs) that take advantage of clinical data resources available there. Moreover, current training programs do not support the growing need for researchers trained and prepared to conduct rapid and continuous learning and translation of findings into patient-centered care. AHRQ, in collaboration with experts drawn from HSR, PCOR and LHS fields, developed a new framework and set of core competencies (see www.ahrq.gov/LHStrainingcompetencies) for training the next generation of HSR/PCOR professionals to become learning health system researchers. The new competencies build on prior foundational HSR/PCOR competencies but address the unique training needs of investigators leading efforts within a LHS to transform care delivery. The competencies also underscore the value of including patients and other stakeholders in systems and comparative effectiveness research studies by ensuring that end users have a hand in shaping the questions and outcomes to assure their relevance.

The purpose of this K12 Institutional Mentored Career Development Program is to train clinical and research scientists to have the skills to support and lead efforts to apply PCOR methods and conduct PCOR research in a LHS and to facilitate rapid implementation of evidence that will improve quality of care and patient outcomes.

Please note that once selected to the program, individuals to be trained under this K12 will be referred to as scholars. Throughout this FOA these individuals are also sometimes referred to as "trainees," "participants," or "candidates."

The Agency for Healthcare Research and Quality (AHRQ), in partnership with the Patient-Centered Outcomes Research Institute (PCORI), invites applications that propose innovative career development programs to build a critical and sustainable LHS research workforce. Programs are expected to prepare newly-trained clinician and research scientists for independent research careers and equip scholars to advance the field through their training and scholarship as the next generation of Learning Health System Researchers.

The program objectives are to:

• Develop and implement a training program that includes both didactic and experiential learning and embeds the scholars at the interface of research, informatics, and clinical operations within learning health systems;
• Identify, recruit, and train clinician and research scientists who are committed to conducting PCOR in health care settings that generate new evidence to facilitate rapid implementation that will improve quality of care and patient outcomes;
• Establish Centers of Excellence in Learning Health System Research Training focusing on the application and mastery of the newly developed core LHS researcher competencies; and
• Support a learning collaborative across funded Centers of Excellence in LHS Research Training to promote cross institutional scholar-mentor interactions, cooperation on multi-site projects, dissemination of project findings, methodological advances, and the development of a shared curriculum.

This institutional K12 training program will establish Centers of Excellence in LHS Researcher Training and focus on the application and mastery of the newly developed core LHS researcher competencies (see www.ahrq.gov/LHStrainingcompetencies) while also building on existing training infrastructure. One of the benefits of an institutional program is that it allows scholars to regularly interact and learn from one another as they pursue separate projects. Mentors based at a common institution will be able to pool their knowledge to develop the didactic curriculum and learn from one another as they share their experiences in helping scholars to navigate their way through successful projects. This approach will produce a cadre of LHS researchers, LHS mentors, and rapid learning projects that can accelerate advances in knowledge, health system performance, and patient outcomes. The benefit of supporting multiple institutions is that it broadens the pool of potential expertise beyond what is typically available at a single institution and it expands the capacity to more rapidly grow the field of individuals capable of successfully working at the interface of clinical operations, informatics and patient-centered outcomes research.

Applicants must be academic institutions with a track record of excellence in training clinical research scientists to work within health care delivery systems or from healthcare delivery systems with a track record of excellence in training researchers and conducting system-based research aimed at improving clinical treatment and/or health care delivery. Applicants must establish collaborative relationships with at least one health system. Systems that lead or participate in a PCORnet Clinical Data Research Network (CDRN) could discuss whether and how clinical data could be linked to the common data model format. Affiliation with PCORnet is not a requirement. Similarly, non-academic health systems may apply but would be expected to partner with an academic institution in providing formal, educational training needs. Both institutional and system-based applicants must describe these collaborative relationships in their applications. All applicants will be expected to recruit and train candidates from beyond their own institution (who are affiliated with other health systems or institutions, including PCORnet-affiliated sites), in order to provide experiential research opportunities for candidates associated with a variety of institutions. All applicants must demonstrate mentoring support from both research and operational leaders.

Each applicant institution must:

• Demonstrate a focus on conducting PCOR within health systems that health systems can rapidly implement to improve quality of care and patient outcomes.
• Demonstrate knowledge and strategy for involving and engaging patients in research and for integrating health equity into the LHS Training program and health systems research.
• Use AHRQ-sponsored LHS training competencies or identify competencies their program will deploy, with justification.
• Develop and implement a training program that includes both didactic and experiential learning and embeds scholars at the interface of research, informatics, and clinical operations within learning health systems.
• Have a strong and high quality infrastructure to support a combination of didactic and experiential training including mentorship and learning opportunities provided within both academic and learning health system settings allowing for the application and mastery of the knowledge and skills outlined across the LHS researcher core competencies. The entire program should be designed to rigorously monitor and provide a well-defined structure to ensure progress for all scholars.
• Encourage collaboration with a member(s) of PCORnet in order to link researchers, patient communities, clinicians, and health systems in research partnerships that leverage large volumes of health data while also ensuring that research focuses on the questions and outcomes that matter most to patients and those who care for them.
• Include research projects designed with LHS partners and conducted within a LHS. Projects should focus on systematically generating evidence, systematically adopting evidence and/or systematically applying evidence in order to expand the health system’s ability to support population health management and/or personalized health care.
• Include an evaluation with clear impact metrics linked to improvements in health system performance, going beyond preparing manuscripts for publication.
• Demonstrate evidence of institutional support (direct or in-kind) from health systems in which scholars would conduct their research.
• Incorporate the PCORI Methodology Standards in the LHS training curriculum.

Scholar appointments must be based on continuous, full-time, 12-month support and are typically 2-3 years in duration. A scholar may not enter the program if there will not be sufficient time remaining in the original project period of the project to allow a minimum of 2 years of training (e.g. year 4 of a 5 year project). Applicants, both institutional and health system-based, should recruit candidates from across network partners thereby ensuring no more than 50% of scholars come from the applicant institution or system. Applicants should demonstrate a long-term commitment to scholars (e.g., after the scholar appointment ends, applicant organizations are encouraged to continue to provide protected time for researchers to facilitate transition to independence).

Both AHRQ and PCORI place a strong emphasis on engagement of stakeholders, particularly patients, in research that they support. Applications must address how patient and other stakeholder engagement will be addressed in the training plan.

Learning Collaborative

The Learning Collaborative will be facilitated by AHRQ in collaboration with PCORI and will focus on improving understanding of best practices for training in a LHS and will be performed through a collaborative model (i.e. within and between institutions). Further, the collaborative will serve as a national forum to promote cross institutional scholar-mentor interactions, cooperation on multi-site projects, dissemination of project findings, methodological advances, and the development of a shared curriculum that can in time be delivered online. The online shared curriculum should result in a comprehensive and efficient training model within the participating institutions and expand the reach of the program to other health systems.

Institutions will be required to participate in the Learning Collaborative. The Learning Collaborative will have monthly web-based meetings and will also meet in-person annually. Applicants must plan and budget for up to four team members (PI(s)/PD(s) and mentors) to travel to the Washington, DC area annually for each year of the grant to participate in a one-day program directors meeting to discuss and provide periodic evaluation of all aspects of the K12 LHS training program and the learning collaborative activities.

Research Career Training Objectives:

42 U.S.C. 299b-37(e) authorizes AHRQ to establish a grant program that provides for the training of researchers in the methods used to conduct patient-centered outcomes research. These awards will be funded by PCORI under an agreement pursuant to 42 U.S.C. 299b-37(g).

This FOA invites applications in which PCOR is embedded in the health system to facilitate rapid implementation. AHRQ is interested in applications that address PCOR within learning health systems using descriptive, surveillance, intervention, and comparative effectiveness research to better understand variation in practice and outcomes. The intent is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision-makers. Research career training objectives must focus the content on health care delivery within the United States.

42 U.S.C. 299b-37(e) authorizes AHRQ to establish a grant program that provides for the training of researchers in the methods used to conduct patient-centered outcomes research. These awards will be funded together by PCORI under an agreement pursuant to 42 U.S.C. 299b-37(g).

All applications submitted and AHRQ grants made in response to this FOA are subject to 45 CFR Part 75 (Uniform Administrative Requirements, Cost Principles and Audit Requirements for HHS Awards; http://www.ecfr.gov/cgi-bin/text-idx?node=pt45.1.75), the HHS Grants Policy Statement (see http://www.ahrq.gov/funding/policies/hhspolicy/index.html), and the terms and conditions set forth in the Notice of Award.

Higher Education Institutions

• Public/State Controlled Institutions of Higher Education
• Private Institutions of Higher Education

The following types of Higher Education Institutions are always encouraged to apply for AHRQ support as Public or Private Institutions of Higher Education:

• Hispanic-serving Institutions
• Historically Black Colleges and Universities (HBCUs)
• Tribally Controlled Colleges and Universities (TCCUs)
• Alaska Native and Native Hawaiian Serving Institutions
• Asian American Native American Pacific Islander Serving Institutions (AANAPISIs)

AHRQ’s authorizing legislation does not allow for-profit organizations to be eligible to lead applications under this research mechanism. For-profit organizations may participate in projects as members of consortia or as subcontractors only. Because the purpose of this program is to improve healthcare in the United States, foreign institutions may participate in projects as members of consortia or as subcontractors only. Applications submitted by for-profit organizations or foreign institutions will not be reviewed. Organizations described in section 501(c) 4 of the Internal Revenue Code that engage in lobbying are not eligible.

HHS grants policy requires that the grant recipient perform a substantive role in the conduct of the planned project or program activity and not merely serve as a conduit of funds to another party or parties. If consortium/contractual activities represent a significant portion of the overall project, the applicant must justify why the applicant organization, rather than the party(s) performing this portion of the overall project, should be the grantee and what substantive role the applicant organization will play. Justification can be provided in the Specific Aims or Research Strategy section of the PHS398 Research Plan Component sections of the SF424 (R&R) application. There is no budget allocation guideline for determining substantial involvement; determination of substantial involvement is based on a review of the primary project activities for which grant support is provided and the organization(s) that will be performing those activities.

Non-domestic (non-U.S.) Entities (Foreign Institutions) are not eligible to apply.
Non-domestic (non-U.S.) components of U.S. Organizations are not eligible to apply.

Applicant Organizations

Applicant organizations must complete and maintain the following registrations as described in the SF 424 (R&R) Application Guide to be eligible to apply for or receive an award. All registrations must be completed prior to the application being submitted. Registration can take 6 weeks or more, so applicants should begin the registration process as soon as possible. Dun and Bradstreet Universal Numbering System (DUNS) - All registrations require that applicants be issued a DUNS number. After obtaining a DUNS number, applicants can begin both SAM and eRA Commons registrations. The same DUNS number must be used for all registrations, as well as on the grant application.

• System for Award Management (https://www.sam.gov/portal/SAM/#1) (formerly CCR) Applicants must complete and maintain an active registration, which requires renewal at least annually. The renewal process may require as much time as the initial registration. SAM registration includes the assignment of a Commercial and Government Entity (CAGE) Code for domestic organizations which have not already been assigned a CAGE Code.
• eRA Commons - Applicants must have an active DUNS number and SAM registration in order to complete the eRA Commons registration. Organizations can register with the eRA Commons as they are working through their SAM or Grants.gov registration. eRA Commons requires organizations to identify at least one Signing Official (SO) and at least one Program Director/Principal Investigator (PD/PI) account in order to submit an application.
• Grants.gov Applicants must have an active DUNS number and SAM registration in order to complete the Grants.gov registration.

Program Directors/Principal Investigators (PD(s)/PI(s))

All PD(s)/PI(s) must have an eRA Commons account. PD(s)/PI(s) should work with their organizational officials to either create a new account or to affiliate their existing account with the applicant organization in eRA Commons. If the PD/PI is also the organizational Signing Official, they must have two distinct eRA Commons accounts, one for each role. Obtaining an eRA Commons account can take up to 2 weeks.

Any individual(s) with the skills, knowledge, and resources necessary to carry out the proposed research training program as the Training Program Director/Principal Investigator (Training PD/PI) is invited to work with his/her organization to develop an application for support. Individuals from underrepresented racial and ethnic groups as well as individuals with disabilities are always encouraged to apply for AHRQ support.

The PD(s)/PI(s) must have experience in the design, implementation and management of related research training programs for developing clinical investigators, and must be able to demonstrate a superior record of preparation of post-doctoral researchers who conduct clinical comparative effectiveness research and have demonstrated a successful record of involving health system stakeholder input and partners in the conduct of research that has influenced health care delivery beyond the bed-side. The PD/PI must be able to demonstrate access to institutional or other funds for a scholar to complete his /her research project in the event of a gap in his or her mentor’s research funding.

For institutions/organizations proposing multiple PDs/PIs, visit the Multiple Program Director/Principal Investigator Policy and submission details in the Senior/Key Person Profile (Expanded) Component of the SF 424 (R&R) Application Guide. See also AHRQ's NOT-HS-16-018 related to Multiple PD/PI via https://grants.nih.gov/grants/guide/notice-files/NOT-HS-16-018.html.

The PD(s)/PI(s) should be an established investigator(s) in the scientific area in which the application is targeted and capable of providing both administrative and scientific leadership to the development and implementation of the proposed program.

The PD(s)/PI(s) will be responsible for the selection, appointment, and mentoring of scholars to the approved research training program, and for the overall direction, management, administration, and evaluation of the program. The PD(s)/PI(s) is/are responsible for the submission of their selected candidates to obtain AHRQ prior approval before appointment. The PD(s)/PI(s) will be expected to monitor and assess the program and submit all documents and reports as required. The PD(s)/PI(s) has (have) responsibility for the day to day administration of the program. The PD(s)/PI(s) will be expected to work closely with other PD(s)/PI(s) across the funded LHS K12 programs to design, plan, coordinate, and establish the learning collaborative to achieve the aforementioned goals in the Scope of the Program.

It is expected that the PD(s)/PI(s) will each devote not less than 10% and no more than 20% full-time effort (4 to 8 hours per week) annually to this K12 program.

This RFA does not require cost sharing.

While there is no cost sharing requirement included in this RFA, AHRQ welcomes applicant institutions, including any collaborating institutions, to devote resources to this effort. An indication of institutional support from the applicant and its collaborators indicates a greater potential of success and sustainability of the project. Examples of institutional support would include: donated equipment and space, institutional funded staff time and effort, or other resource investments. Applicant institutions should indicate institutional support by outlining the specific contributions to the project and providing assurances that their organization and any collaborators are committed to providing these funds and resources to the project. This information can be included at the end of the budget justification section of the application, but institutional support dollars are not to be shown/included in the detailed budget request.

Applicant organizations may submit more than one application, provided that each application is programmatically distinct.

AHRQ will not accept duplicate or highly overlapping applications under review at the same time. This means that AHRQ will not accept:

• A new (A0) application that is submitted before issuance of the summary statement from the review of an overlapping new (A0) or resubmission (A1) application.
• A resubmission (A1) application that is submitted before issuance of the summary statement from the review of the previous new (A0) application.
• An application that has substantial overlap with another application pending appeal of initial peer review.

See also NOT-HS-16-018 related to Multiple PD/PI via https://grants.nih.gov/grants/guide/notice-files/NOT-HS-16-018.html.

Program faculty who serve as mentors should have strong records as researchers, including recent publications and successful competition for research support in the area of the proposed research training program. Program faculty should also have a record of research training, including successful, former trainees who have established productive careers relevant to the AHRQ mission.

Researchers from diverse backgrounds, including racial and ethnic minorities, persons with disabilities, and women are encouraged to participate as mentors. Mentors must have research expertise and experience relevant to the proposed program. Mentors must be committed to continue their involvement throughout the total period of the award.

Each scholar appointed to the program must have a primary mentor. All proposed mentors must: (1) be recognized as accomplished investigators in the area of comparative effectiveness research in Learning Health Systems, and/or clinical epidemiology; (2) be actively involved in comparative effectiveness research; (3) have a successful record of providing the type of career development and training activities required under this award; and (4) have a successful record of engendering his or her scholars research careers as evidenced by scholars research productivity, their ability to obtain/participate in a continuing role in independent research, and their ability to work with health system stakeholders in a responsive fashion that has an impact on health care delivery.

Program mentors should include established research and clinician scientists with various skill sets needed and extensive experience to conduct research in a LHS. For example, mentors specialized in system science will bring expertise in understanding how health systems operate and how to apply systems theory to research and implementation. Scholars will be able to learn from informatics experts how to use information systems to improve patient outcomes and health system performance. Mentors with skills in patient engagement can facilitate patient engagement in all aspects of the research experience. Scholars may also be able to learn from patients who participate in patient engagement activities. Reimbursement of patient participation is allowable under this grant (see http://www.pcori.org/sites/default/files/PCORI-Compensation-Framework-for-Engaged-Research-Partners.pdf or PCORI's compensation framework) and, if requested, is to be budgeted as part of the scholar's $25,000 in "Other Expenses" for research-development support costs. Program mentors should have direct involvement in the development and operation of the program. It is anticipated that mentors will assist the PD(s)/PI(s) in the design and evaluation of the didactic coursework and experiential training curriculum needed for a successful program. Mentors are anticipated to propose potential research projects designed with LHS partners and conducted within LHSs. Mentors are expected to actively assist PD(s)/PI(s) in the development and implementation of the learning collaborative across the funded K12 Centers of Excellence.

Once selected to the program, candidates/participants will be referred to as scholars. The research experience of the candidates selected for support under this award should resemble those supported by the individual Mentored Clinical Scientist Development Award (K08), the Mentored Research Scientist Development Award (K01), the Mentored Patient-Oriented Career Development Award (K23), or postdoctoral trainees and fellows appointed to F32 or T32 grants or eligible for K99 grants. Scholars to be supported by the institutional career development program must be at the career level for which the planned program is intended and appointed for a two- or three-year period. The latest entry time for any new Scholar appointment is anticipated to be year 4 of the grant. It is the PI(s)/PD(s)'s responsibility to ensure each candidate's fulfillment of the selection criteria when recruiting Scholars to the career development program. The PD(s)/PI(s) is/are responsible for the submission of their select candidates to obtain AHRQ prior approval before appointment.

It is recognized that not all candidates will be named upon submission of the application or start of the grant. To the extent that individuals and their salary levels and research development costs are known at the time of application, they should be individually included and justified. In lieu of this level of specificity, applicants may provide a more general justification for these costs within the guidelines of allowable components. As much detail as possible, however, is necessary in order to provide reviewers with a sense of anticipated needs. For example, what are average salary and tuition needs, average mentoring needs, average statistical support and computing needs, average insurance needs, average travel needs, etc.? If these needs vary by the types of scholars anticipated, what is the anticipated mix of the components required for how many of each type of scholar requested in the budget? All of these estimates should be based on the institution’s anticipated pool of candidates.

Candidates must be U.S. citizens or non-citizen nationals, or individuals lawfully admitted for permanent residence, and who have earned a clinical or research doctorate, including, but not limited to: Ph.D., Sc.D., Dr.P.H., M.D., D.O., D.C., N.D., D.D.S., D.M.D., D.N.S., D.V.M., O.D., D.P.M., Eng.D., D.P.T., O.T.D., Pharm.D., D.S.W., Psy.D., or equivalent doctoral degree, at the time of appointment. As noted above, these candidates must demonstrate commitment to conducting research within learning health systems focusing on rapid and continuous learning and translation of findings into patient-centered care. There are no specific requirements that scholars need to be postdoctoral fellows or junior faculty or that persons of differing ranks (e.g., fellow vs. junior faculty) must be provided the same level of salary support.

Candidates must be able to commit a minimum of nine person-months (75% of full-time professional effort) conducting research career development activities associated with this award for a two or three year period. The remaining three months (25% effort) can be divided among other research, clinical, and teaching activities but only if these activities are consistent with the goals of the Award, i.e., the candidate’s development into an independent investigator.

At the time of placement on the grant, candidate must have a full-time appointment at their respective institution. It is encouraged that there is a balanced number of appointments with no more than half of the scholars originating from the applicant institution.

Additional details on citizenship, training period, and aggregate duration of support are available in the HHS Grants Policy Statement at https://www.ahrq.gov/funding/policies/hhspolicy/index.html.

All scholars appointed to this grant can move on to individual mentored career development support grants upon completion of their two- or three-year appointment on the K12 grant. Any scholar appointed to an AHRQ-supported K12 program or an AHRQ-supported individual mentored career development may not exceed a total of five years of AHRQ mentored career development support.

Individuals are eligible for placement as a scholar if they have been, or currently are the PD/PI of an NIH or AHRQ R36, F31, or F32 grant, or a Federal or non-Federal award that duplicates the provisions or research goals of these grants, and otherwise meet all other eligibility criteria. The scholar will have to terminate the existing award before being appointed to the K12. Individuals are also eligible to be scholar candidates if they have been, or currently are the PD/PI of an NIH or AHRQ R03 or R21 grant, or a Federal or non-Federal award that duplicates the provisions or research goals of an R03 or R21 grant.

Individuals are NOT eligible to be scholar candidates if they:

• Have simultaneously submitted or have an application pending peer review for any other Federal career development award, or a research project grant (R01). However, individuals may concurrently submit an application for an AHRQ or NIH Small Grant (R03) or Exploratory/Developmental Grants (R21);
• Have simultaneously submitted or have an application pending peer review for any non-Federal research grant, contract, or cooperative agreement over $100,000 direct costs per year;
• Have been or are currently a PD/PI on any other Federal mentored career development awards;
• Have been or are currently supported on an institutional K12 grant or KL2 (or similar grant);
• Have been or are currently a PD/PI on a Federal research grant (such as R01, R29, P01) or subproject leaders on Program Project (P01) and Center Grants (P50); and/or
• Have been or are currently a PD/PI on peer-reviewed non-Federal research grants, contracts or cooperative agreements over $100,000 direct costs per year.

Note that K12 Scholars should be strongly encouraged to apply for independent research grant support (e.g., R01, R21, R03) by the end of their final year of appointment to the K12 grant.

Scholars are NOT allowed to reduce their level of effort below 75% over the course of the award.

Buttons to access the online ASSIST system or to download application forms are available in Part 1 of this FOA. See your administrative office for instructions if you plan to use an institutional system-to-system solution.

It is critical that applicants follow the Training (T) Instructions in the SF424 (R&R) Application Guide including Supplemental Grant Application Instructions except where instructed in this funding opportunity announcement to do otherwise. Conformance to the requirements in the Application Guide is required and strictly enforced. Applications that are out of compliance with these instructions may be delayed or not accepted for review.

For information on Application Submission and Receipt, visit Frequently Asked Questions Application Guide, Electronic Submission of Grant Applications.

Letter of Intent

Although a letter of intent is not required, is not binding, and does not enter into the review of a subsequent application, the information that it contains allows AHRQ staff to estimate the potential review workload and plan the review.

By the date listed in Part 1. Overview Information, prospective applicants are asked to submit a letter of intent that includes the following information:

• Number and title of this funding opportunity
• Descriptive title of proposed activity
• Name(s), address(es), and telephone number(s) of the PD(s)/PI(s)
• Names of other key personnel
• Participating institution(s)

The letter of intent should be emailed to:

C. Jean Hsieh, PhD, OT
Telephone: 301-427-1534
Email: Chinghui.Hsieh@ahrq.hhs.gov

AHRQ will convene a technical assistance conference call to provide potential applicants with background information and respond to questions about the preparation of an application in response to this RFA.

The conference call will take place on Wednesday, October 11, 2017 from 1:00 p.m. to 2:00 p.m. Eastern time. Call in information will be posted under the Announcements section on the AHRQ training webpage (www.ahrq.gov/training) approximately 5 days before the call. The conference call is open to any individual or organization. Participation is not a prerequisite to submitting an application.

Participants are encouraged to submit questions via email prior to the conference call. Please submit questions to trainingta@ahrq.hhs.gov by Friday, October 6, 2017, 5:00 p.m. Eastern time. Questions of a similar topic and nature may be grouped at the sole discretion of AHRQ staff. Notes from the conference call will be posted on the AHRQ web site.

All page limitations described in the SF424 (R&R) Application Guide and the Table of Page Limits must be followed.

Instructions for Application Submission

The following section supplements the instructions found in the SF424 (R&R) Application Guide and should be used for preparing an application to this FOA.

Follow all instructions provided in the SF424 (R&R) Application Guide.

Follow all instructions provided in the SF424 (R&R) Application.

Follow all instructions provided in the SF424 (R&R) Application, with the following additional modifications:

Substitute the term scholars for all references to trainees in the SF424 (R&R) Application Guide, and substitute the term career development for all references to training in the SF424 (R&R) Application Guide.

Project Summary/Abstract. Provide an abstract of the entire application. Include the objectives, rationale and design of the research career development program, as well as key activities in the training plan. Indicate the planned duration of appointments, the projected number of scholars including their levels (i.e., postdoctoral, junior faculty), and intended scholar outcomes.

Other Attachments. An Advisory Committee is not a required component of this career development program. However, if an Advisory Committee is intended, provide a plan for the appointment of an Advisory Committee to monitor progress of the career development program. The composition, roles, responsibilities, and desired expertise of committee members, frequency of committee meetings, and other relevant information should be included. Describe how the Advisory Committee will evaluate the overall effectiveness of the program. Proposed Advisory Committee members should be named in the application if they have been invited to participate at the time the application is submitted.

The filename provided for each Other Attachment will be the name used for the bookmark in the electronic application in eRA Commons.

Follow all instructions provided in the SF424 (R&R) Application.

Follow all instructions provided in the SF424 (R&R) Application.

Follow all instructions provided in the SF424 (R&R) Application Guide

Follow all instructions provided in the SF424 (R&R) Application Guide with the following additional modifications:

• The R&R Budget Component must be used instead of the Training Budget Component.
• Include all personnel other than the Training PD(s)/PI(s) in the Other Personnel section, including clerical and administrative staff.
• Do not complete the section on Participant/Trainee Support Costs.
• Use the Travel section to submit costs for PD(s)/PI(s) and mentor travel to an annual program directors meeting with AHRQ and PCORI .
• Label one of the blank lines in the Other Direct Costs section as "Other Expenses" and enter up to $25,000 per scholar in a lump-sum to be used for research development support. Describe these costs fully in the budget justification.

The PHS 398 Research Training Program Plan Form is comprised of the following sections:

Training Program Faculty, Trainees, and Training Record Other Training Program Sections

Appendix

Follow all instructions provided in the SF424 (R&R) Application Guide with the following additional modifications:

Particular attention must be given to the required Training Data Tables. Applicants should summarize, in the body of the application, key data from the tables that highlight the characteristics of the applicant pool, faculty mentors, the educational and career outcomes of participants, and other factors that contribute to the overall environment of the program.

Training Program

Program Plan

Program Administration.

The sponsoring institution must assure support for the proposed program. Appropriate institutional commitment to the program includes the provision of adequate staff, facilities, and educational resources that can contribute to the planned program.

The applicant institution must have a strong and high quality research program in PCOR and training in PCOR methods and must have the requisite faculty, staff, potential scholars and facilities on site to conduct the proposed institutional program. In many cases, it is anticipated that the proposed program will complement other ongoing career development programs occurring at the applicant institution and that a substantial number of program faculty will have active research projects in which participating scholars may gain relevant experiences consistent with their research interests and goals.

Institutions with existing career development programs must explain what distinguishes this program from the others, how their programs will synergize with one another, if applicable, and make it clear that the pool of faculty, potential scholars, and resources are robust enough to support additional programs. If a program administrator position is planned, a description of the scientific expertise, leadership, and administrative capabilities essential to coordinate a program for developing investigators must be included in the application.

The plans for administering the overall program, including the proposed strategy and structure for monitoring the program should be described. Details about the mentoring team, solidifying the infrastructure needed in preparation for successful Scholar training, curriculum planning, including implementation of the LHS competencies, and finalizing the didactic and experiential training component and syllabus should also be described.

The plan and structure for Program Administration should be described in accordance with the nature of the partnership between an institution and a health system, as well as a multiple PDs/PIs model, if applicable, for the purpose of the Award.

A multiple PI/PD model is allowed. The multiple-PI model is intended to supplement, and not to replace, the traditional single PI model. The overarching goal is to encourage collaboration among equals when that is the most appropriate way to the proposed program. Each of the listed PIs designated by the grantee institution will be expected to share responsibility for directing the project or activity. As in current applications, peer reviewers will consider whether the designated PIs have appropriate training and experience to carry out the proposed program activities.

To facilitate communication with AHRQ, the institution will be asked to select a Contact PI at the time of application. The Contact PI will be responsible for relaying communications between all of the PIs and the AHRQ. Being named Contact PI will not confer any special authority or responsibility for the project. The applicant should describe in a Leadership Plan the governance and organizational structure of the leadership team, including communication plans, process for making decisions on program direction, and procedures for resolving conflicts.

Proposed Training.

Provide an overview of the proposed program: Describe the immediate and long-term objectives of the program, including activities that will be used to ensure that the objectives of the program are met.

Include information about planned courses, curricula, seminars, workshops, or tutorials that will be incorporated into the training program and mentored research experiences and activities (including core competencies to be conveyed, the method they will be conveyed, and how they will be assessed). Address how the AHRQ sponsored LHS competencies will be incorporated in the training program or provide a justification if other competencies will be used. Include information on how PCORI Methodology Standards and Curriculum (including updates as available) will be incorporated in the training curriculum. Section 937(e) of the Public Health Service Act authorizes AHRQ to establish a grant program that provides for the training of researchers in comparative effectiveness methods and states that, "[a]t a minimum, such training shall be in methods that meet the methodological standards adopted [by PCORI] under 1181(d)(9) of the Social Security Act."

Address how patient centeredness, patient engagement, health disparities, and health equity will be incorporated in the training plans and ideally operationalized into scholars' research projects. A specific strategy for incorporating patient centeredness and health disparities and health equity should be articulated.

Institutions with existing career development programs must explain what distinguishes this program from the others, how the programs will synergize, if applicable, and justify that the faculty, pool of potential scholars, and resources are robust enough to support additional programs. The description must include planned strategies to be used to ensure that the objectives are met.

The PD(s)/PI(s) must also describe past research career development activities/experiences including those of mentors, documenting the success of former trainees in establishing independent productive scientific careers.

Plans should also address training required for professional development, communication of research, and translation of research into practice and/or policy. Specific mentoring plans should be included. The plan should also include descriptions of any proposed rotations/practicum in a LHS, which will be made available for scholars in academic-based programs. These are not required, but are highly encouraged for applicants and their partnering health systems. They should be designed to take advantage of LHS career development experiences not available at, but complementary to, those at the applicant institution. Documentation of commitment on the part of the stakeholder group should be included in the application, but does not count toward the research training plan page limitation.

Describe an evaluation plan to review and determine the effectiveness of the program. This should include plans to obtain feedback from current and former scholars to help identify weaknesses and to provide suggestions for program improvements, as well as plans for assessing scholars career development and progression, including metrics such as degree completion (if applicable), publications, and subsequent positions. Specified evaluation metrics should be tied to the goals of the program. Evaluation results should be included in the Final Progress Report.

Describe in general terms the pool of potential candidates, including information about the types of prior clinical and research training and the expected career level required for the program. Describe plans to recruit candidates and explain how these plans will be implemented (see also section on Recruitment and Retention Plan to Enhance Diversity). Describe the evaluation criteria to be used in the selection of scholars. Provide brief summaries of career development plans that the program will employ. The application must contain a description of how career development plans will be tailored to the needs of the prospective candidates, taking into account their past experiences and competences. The plan should guarantee existence of an adequate group of highly qualified potential candidates available to embark on a two- to three- year commitment to career development in PCOR that is responsive to stakeholder needs.

The application must include a statement from the applicant institution describing the commitment to the planned program. The institution must assure that essential time will be allowed for the PD(s)/PI(s), other faculty and mentors, and that the PD(s)/PI(s) will require that institutions selected by scholars for mentored career development research activities will provide written assurance of protected time for, as well as essential time for faculty involved in mentoring. Include documentation of an institution's commitment to the proposed research career development program's goals, and a description of support, financial or otherwise, to be provided to the program. The support could include space, equipment, release time for the PD/PI and participating faculty, support for additional scholars in the program, or any other creative ways to improve and enhance the growth of the research career development program. Applications can also include a statement of support by systems leaders indicating knowledge, awareness, and support for patient engagement, patient-centeredness, and support and precedent for reaching out to patients within their systems.

Individuals are required to comply with the instructions for Recruitment and Retention Plan to Enhance Diversity as provided in Chapter 8 of the SF424 (R&R) Application Guide.

Standardized data tables for institutional training and career development programs (e.g., T32 and K12 programs) are located at https://grants.nih.gov/grants/forms/data-tables.htm. The data tables are designed to enable consistent entry and review of data for training program faculty and scholars. The information provided in these data tables is not counted toward the 25 page limitation for the main body of the application (Items 2.2-2.4 in the SF424 instructions). Applicants are required to include only Tables 2, 4, 5B and 6B. Please read the Introduction to the Data Tables linked on the above web site before starting to prepare the tables. The data tables should be numbered consecutively and titled as shown, even if some are not used. Indicate by table number and title those tables that are intentionally omitted. The information included in the data tables should be used to present a detailed documentation of the proposed K12 Program.

Tables 5B and 6B should include both postdoctoral fellows as well as junior faculty supported through individual or institutional mentored career development grants. Publications of past scholars should only include work conducted during their period of training, regardless of when the publication actually appeared, may be listed in Table 6B. Applicants may provide explanatory notes at the bottom of the standardized data tables.

Additional tables may be included as part of the research training program plan. However, such tables will count toward the 25-page limit.

Individuals are required to comply with the instructions for Plan for Instruction in the Responsible Conduct of Research as provided in the SF424 (R&R) Application Guide.

Particular attention must be given to the required Training Data Tables. Add Information above about other content related to the PHS 398 Research Training Program Plan. Include only information that is different from SF424 (and not otherwise described above the list of sections of the Research Training Program Plan is provided in the bulleted list above, for your reference).

Do not use the Appendix to circumvent page limits. Follow all instructions for the Appendix as described in the SF424 (R&R) Application Guide. See NIH/NIOSH/AHRQ New Policy Appendix Guide located at NOT-OD-16-129 via https://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-129.html

All instructions in the SF424 (R&R) Application Guide must be followed.

See Part 1. Section III.1 for information regarding the requirement for obtaining a unique entity identifier and for completing and maintaining active registrations in System for Award Management (SAM), NATO Commercial and Government Entity (NCAGE) Code (if applicable), eRA Commons, and Grants.gov

Part I. Overview Information contains information about Key Dates and times. Applicants are encouraged to submit applications before the due date to ensure they have time to make any application corrections that might be necessary for successful submission. When a submission date falls on a weekend or Federal Holiday the application deadline is automatically extended to the next business day.

Organizations must submit applications to Grants.gov (the online portal to find and apply for grants across all Federal agencies). Applicants must then complete the submission process by tracking the status of the application in the eRA Commons, NIH’s electronic system for grants administration. AHRQ and Grants.gov systems check the application against many of the application instructions upon submission. Errors must be corrected and a changed/corrected application must be submitted to Grants.gov on or before the application due date and time. If a Changed/Corrected application is submitted after the deadline, the application will be considered late.

Applicants are responsible for viewing their application before the due date in the eRA Commons to ensure accurate and successful submission.

Information on the submission process and a definition of on-time submission are provided in the SF424 (R&R) Application Guide.

SF424 (R&R) Application Guide.

This initiative is not subject to intergovernmental review.

For efficient grant administration, AHRQ grant administration procedures will be used and conducted in accordance with the terms and conditions, cost principles, and other considerations described in the HHS Grants Policy Statement. The Grants Policy Statement can be found at http://www.hhs.gov/sites/default/files/grants/grants/policies-regulations/hhsgps107.pdf.

Pre-award costs are not allowed without AHRQ prior approval.

The incurrence of pre-award costs in anticipation of a competing or non-competing award imposes no obligation on AHRQ either to make the award or to increase the amount of the approved budget if an award is made for less than the amount anticipated and is inadequate to cover the pre-award costs incurred. AHRQ expects the grantee to be fully aware that pre-award costs result in borrowing against future support and that such borrowing must not impair the grantee's ability to accomplish the project objectives in the approved time frame or in any way adversely affect the conduct of the project.

Applications must be submitted electronically following the instructions described in the SF424 (R&R) Application Guide. Paper applications will not be accepted.

Applicants must complete all required registrations before the application due date. Section III. Eligibility Information contains information about registration.

For assistance with your electronic application or for more information on the electronic submission process, visit Applying Electronically. If you encounter a system issue beyond your control that threatens your ability to complete the submission process on-time, you must follow the Guidelines for Applicants Experiencing System Issues. For assistance with application submission, contact the Application Submission Contacts in Section VII.

Important reminders:

All PD(s)/PI(s) must include their eRA Commons ID in the Credential field of the Senior/Key Person Profile Component of the SF424(R&R) Application Package. Failure to register in the Commons and to include a valid PD/PI Commons ID in the credential field will prevent the successful submission of an electronic application to NIH.

The applicant organization must ensure that the DUNS number it provides on the application is the same number used in the organization’s profile in the eRA Commons and for the System for Award Management (SAM). Additional information may be found in the SF424 (R&R) Application Guide.

See more tips for avoiding common errors.

Upon receipt, applications will be evaluated for completeness by the Center for Scientific Review, NIH and for responsiveness by AHRQ. Applications that are incomplete will not be reviewed.

In order to expedite the review, applicants are requested to notify the AHRQ Referral Office by email Gerald.Calderone@ahrq.hhs.gov when the application has been submitted. Please include the RFA number and title, PD/PI name, and title of the application.

Institutional Review Board (IRB) approval of human subjects is not required prior to peer review of an application (see https://grants.nih.gov/grants/guide/notice-files/NOT-HS-00-003.html). However, initiation of IRB review, if necessary or applicable, is strongly encouraged to assure timely commencement of research.

Although there is no immediate acknowledgement of the receipt of an application, applicants are generally notified of the review and funding assignment within eight (8) weeks.

Applicants are required to follow the instructions for post-submission materials, as described in the following policy: NOT-OD-16-130

Priority Populations

AHRQ supports research, evaluations, and demonstration projects concerning the delivery of health care in inner-city and rural areas (including frontier areas), and health care for priority populations. Priority populations include low income groups, minority groups, women, children, the elderly, and individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care. 42 USC 299(c). Women and members of minority groups are included in all AHRQ-supported research projects involving human subjects, unless a clear and compelling rationale and justification is provided that inclusion is inappropriate, e.g. because of the lack of connection between the study and the health of women or particular minorities. Investigators should review the document entitled, AHRQ Policy on the Inclusion of Priority Populations, which is available at https://grants.nih.gov/grants/guide/notice-files/NOT-HS-03-010.html. Applicants under this FOA should consider and discuss including priority populations in the research design as specified in this Notice.

Public Access to AHRQ-Funded Scientific Publications

Investigators should review the document titled AHRQ Announces new Policy for Public Access to AHRQ-Funded Scientific Publications which is available at (http://grants.nih.gov/grants/guide/notice-files/NOT-HS-16-008.html). For all research arising from AHRQ support, this policy requires that AHRQ-funded authors submit an electronic version of the author’s final peer-reviewed accepted manuscript to the National Library of Medicine's PubMed Central (PMC) to be made publicly available within 12 months of the publisher’s date of publication.

The precise content of the data-sharing plan will vary, depending on the data being collected and how the investigator is planning to share the data. Applicants who are planning to share data should describe briefly the expected schedule for data sharing; the format of the final dataset; the documentation to be provided; whether or not any analytic tools also will be provided; whether or not a data-sharing agreement will be required and, if so, a brief description of such an agreement (including the criteria for deciding who can receive the data and whether or not any conditions will be placed on their use); and the mode of data sharing (e.g., under its own auspices by mailing a disk or posting data on its institutional or personal website or through a data archive or enclave). Investigators choosing to share under their own auspices may wish to enter into a data-sharing agreement. References to data sharing may also be appropriate in other sections of the application.

The reasonableness of the data sharing plan or the rationale for not sharing research data will be assessed by the reviewers. However, reviewers will not factor the proposed data sharing plan into the determination of scientific merit or the priority score.

The AHRQ confidentiality statute, 42 USC 299c-3(c), requires that information that is obtained in the course of AHRQ supported activities and that identifies individuals or establishments be used only for the purpose for which it was supplied. Information that is obtained in the course of AHRQ-supported activities and that identifies an individual may be published or released only with the consent of the individual who supplied the information or is described in it. There are civil monetary penalties for violation of the confidentiality provision of the AHRQ statute. 42 USC 299c-3(d). In the Human Subjects section of the application, applicants must describe procedures for ensuring the confidentiality of the identifying information to be collected. The description of the procedures should include a discussion of who will be permitted access to this information, both raw data and machine readable files, and how personal identifiers and other identifying or identifiable data will be restricted and safeguarded. Identifiable patient health information collected by grantees under this RFA will also be obtained and managed in accordance with the HIPAA Privacy Rule, 45 CFR Parts 160 and 164.

The grantee should ensure that computer systems containing confidential data have a level and scope of security that equals or exceeds that established by the HIPAA Security Rules if applicable (see HIPAA website in prior paragraph) and that established by the Office of Management and Budget (OMB) in OMB Circular No. A-130, Appendix III - Security of Federal Automated Information Systems. The applicability and intended means of applying these confidentiality and security standards to subcontractors and vendors, if any, should be addressed in the application.

Unless otherwise provided in grant awards, AHRQ grantees may copyright, or seek patents for, as appropriate, final and interim products and materials developed in whole or in part with AHRQ support, including, but not limited to, methodological tools, measures, software with documentation, literature searches, and analyses. Such copyrights and patents are subject to a royalty-free, non-exclusive, and irrevocable AHRQ license to reproduce, publish, use or disseminate for any purpose consistent with AHRQ s statutory responsibilities and to authorize others to do so for any purpose consistent with AHRQ’s statutory responsibilities. In accordance with its legislative dissemination mandate, AHRQ purposes may include, subject to statutory confidentiality protections, making project materials, databases, results, and algorithms available for verification or replication by other researchers. In addition, subject to AHRQ budget constraints, final products may be made available to the health care community and the public by AHRQ or its agents if such distribution would significantly increase access to a product and thereby produce substantial or valuable public health benefits. Ordinarily, to accomplish distribution, AHRQ publicizes research findings but relies on grantees to publish research results in peer-reviewed journals and to market grant-supported products. AHRQ requests that grantees notify the Office of Communications (OC) when an AHRQ-funded research article has been accepted for publication in a peer-reviewed journal. Researchers should submit manuscripts that have been accepted for publication in a peer-reviewed journal to JournalPublishing@ahrq.hhs.gov at least four to six weeks in advance of the journal’s expected publication date.

Regulations applicable to AHRQ grantees concerning intangible rights and copyright can be found at 45 CFR 75.322.

The mission of AHRQ is to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable, and to work with the U.S. Department of Health and Human Services (HHS) and other partners to make sure that the evidence is understood and used.

AHRQ's priority areas of focus are:

• Improve health care quality by accelerating implementation of patient-centered outcomes research (PCOR).
• Make health care safer.
• Increase accessibility by evaluating coverage expansion.
• Improve health care affordability, efficiency, and cost transparency.

As part of this mission, applications submitted to AHRQ to support health services research are evaluated for scientific and technical merit through the AHRQ peer review system.

Applications that are complete and responsive to the FOA will be evaluated for scientific and technical merit by an appropriate peer review group convened in accordance with standard AHRQ peer review procedures that are described in 42 CFR Part 67, Subpart A. Incomplete and/or non-responsive applications or applications not following instructions given in this FOA will not be reviewed

As part of the initial merit review, all applications will:

• Undergo a selection process in which only those applications deemed to have the highest scientific merit will be discussed and assigned a priority score;
• Receive a written critique.

Administrative Criteria: Upon receipt, applications will be evaluated for completeness by the Center for Scientific Review, NIH, and for responsiveness by AHRQ.

Merit Review Criteria: Merit Review Criteria, as described below, will be considered in the review process.

Reviewers will provide an overall impact score to reflect their assessment of the likelihood that the proposed training program will prepare individuals for successful, productive scientific research careers and thereby exert a sustained influence on the research field(s) involved, in consideration of the following review criteria and additional review criteria (as applicable for the project proposed).

Reviewers will consider each of the review criteria below in the determination of the merit of the training program, and give a separate score for each. When applicable, the reviewers will consider relevant questions in the context of proposed short-term training. An application does not need to be strong in all categories to be judged likely to have major scientific impact.

Does the proposed program clearly outline a plan to recruit and develop well-qualified junior investigators for successful careers as researchers to conduct research in learning health systems? Is there evidence of an adequate pool of potential scholars who could benefit from receiving career development support? Are the content and duration of any proposed didactic, training-related, and research-related activities of the program appropriate and aligned with CER methods and PCOR? Are appropriate timelines indicated for career progression and transition to independence? Does the institutional environment (e.g., research facilities and other relevant resources) in which the program will be conducted contribute to the probability of success? Does the proposed career development program benefit from unique features of the scientific environment, subject populations, or employ useful collaborative arrangements? Is the institutional commitment to the proposed program appropriate? If multiple sites are participating, is this adequately justified in terms of the career development and research experiences provided? Is there sufficient assurance that the required effort of the PD/PI, mentors and scholars will be devoted directly to the career development activities? Does the program have a rigorous evaluation plan to assess the quality and effectiveness of the career development program? Additionally, for non-academic institutions, is there evidence of effective partnership with an academic institution to ensure robust formal educational training?

• Do the PD(s)/PI(s) have the experience to develop, direct and administer the proposed program?
• Does the leadership team bring complementary and integrated expertise to the program? Is there evidence that an appropriate 10%-20% level of effort will be devoted by the program leadership to ensure program objectives?
• Are the research qualifications, scientific stature, previous leadership and mentoring experience, and track record(s) appropriate for the proposed career development program?
• Are the PD(s)/PI(s) currently engaged in research relevant to the scientific area of the proposed program?

For applications designating multiple PDs/PIs:

• Is a strong justification provided that the multiple PD/PI leadership approach will benefit the career development program and the scholars?
• Is a strong and compelling leadership approach evident, including the designated roles and responsibilities, governance, and organizational structure consistent with and justified by the aims of the career development program and the complementary expertise of the PDs/PIs?

• Do the mentors have strong records as researchers, including recent publications and successful competition for research support in areas directly related to the proposed research training program?
• Do the mentors have appropriate expertise and experience, as well as track records of past mentoring and training?
• Are the quality and extent of the mentors roles in providing guidance and scientific advice to the scholars acceptable? Are the mentors currently engaged in relevant research?

Is a recruitment plan proposed with strategies likely to attract high quality scholar candidates? Are there well-defined and well justified recruitment and selection strategies? Is there evidence of a sufficiently large, competitive scholar pool to warrant the proposed size of the career development program? Is there an appropriate plan for the content, the phasing, and the proposed duration of the career development plan for achieving scientific independence for the prospective candidates? What is the likelihood that the career development plan will contribute significantly to the scientific development of the scholars? Does the plan for selection of the scholars include all of the eligibility criteria stated in the RFA?

Is there evidence of a successful past training record of the PD(s)/PI(s) and mentors, including the success of former scholars in seeking independent support and establishing productive scientific careers? Does the program have a rigorous evaluation plan to assess the quality and effectiveness of the training?

As applicable for the project proposed, reviewers will evaluate the following additional items while determining scientific and technical merit, and in providing an overall impact score, but will not give separate scores for these items.

Patient-Centeredness

Does the training plan address the importance and incorporate patient-centeredness throughout the curriculum? Does the application focus on improving patient-centered outcomes, selected in consultation with patients and other key stakeholders, emphasizing outcomes that can close the evidence gap on issues important to patients and other stakeholders?

Stakeholder and Patient Engagement

Does the application describe how patients and stakeholders will be incorporated and engaged in the curriculum as part of the training plan? Are stakeholder and patient roles and functions articulated in protocols for the structure and governance for research and training activities?

Health Disparities and Health Equity

Does the application describe how health disparities and health equity will be addressed and incorporated in the training plan and curriculum, and into specific research projects?

Collaboration with members of PCORnet (as relevant)

Does the application describe partnerships with PCORnet researchers, patient communities, clinicians, and health systems that are already in place and may be leveraged for the LHS research projects within the training program?

Incorporation of PCORI Methodology Standards

Does the application describe how the PCORI Methodology Standards and Curriculum (including updates as available) are incorporated in the training curriculum?

Inclusion of LHS Competencies

Does the application describe how the AHRQ sponsored LHS competencies will be incorporated in the training curriculum? Or, if other competencies are used, is a justification provided?

For research that involves human subjects but does not involve one of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate the justification for involvement of human subjects and the proposed protections from research risk relating to their participation according to the following five review criteria: 1) risk to subjects, 2) adequacy of protection against risks, 3) potential benefits to the subjects and others, 4) importance of the knowledge to be gained, and 5) data and safety monitoring for clinical trials.

For research that involves human subjects and meets the criteria for one or more of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate: 1) the justification for the exemption, 2) human subjects involvement and characteristics, and 3) sources of materials. For additional information on review of the Human Subjects section, please refer to the Human Subjects Protection and Inclusion Guidelines.

Reviewers will assess how well the application addresses the purpose and objectives of this FOA. How responsive is the application to the special eligibility criteria, including the project requirements, noted in the FOA?

The committee will evaluate whether the proposed budget is reasonable, and whether the requested period of support is appropriate in relation to the proposed research.

Reviewers will assess the adequacy of plans to address the needs of both genders, racial and ethnic minorities (and subgroups). Reviewers will assess the adequacy of attention to AHRQ priority populations (see above discussion on Priority Populations in section IV.7 Other Submission Requirements ).

Not Applicable

Recruitment Plan to Enhance Diversity

Peer reviewers will separately evaluate the recruitment plan to enhance diversity after the overall score has been determined. Reviewers will examine the strategies to be used in the recruitment of individuals from underrepresented groups. The plan will be rated as ACCEPTABLE or UNACCEPTABLE, and the consensus of the review committee will be included in an administrative note in the summary statement.

Training in the Responsible Conduct of Research

All applications for support under this FOA must include a plan to fulfill HHS requirements for instruction in the Responsible Conduct of Research (RCR). Taking into account the specific characteristics of the career development program, the level of scholar experience, and the particular circumstances of the scholars, the reviewers will evaluate the adequacy of the proposed RCR training in relation to the following five required components: 1) Format - Does the plan satisfactorily address the format of instruction, e.g. lectures, coursework and/or real-time discussion groups, including face-to-face interaction? (A plan involving only on-line instruction is not acceptable.); 2) Subject Matter Does the plan include a sufficiently broad selection of subject matter, such as conflict of interest, authorship, data management, human subjects and animal use, laboratory safety, research misconduct, research ethics? 3) Faculty Participation - Does the plan adequately describe how faculty will participate in the instruction? 4) Duration of Instruction - Does the plan meet the minimum requirements for RCR, i.e., at least eight contact hours of instruction? 5) Frequency of Instruction Does the plan meet the minimum requirements for RCR, i.e., at least once during each career stage (undergraduate, post-baccalaureate, predoctoral, postdoctoral, and faculty levels) and at a frequency of no less than once every four years?

Plans and past record will be rated as ACCEPTABLE or UNACCEPTABLE, and the summary statement will provide the consensus of the review committee.

Applications will be evaluated for scientific and technical merit by an appropriate Scientific Review Group in accordance with the stated review criteria. Assignment to a Scientific Review Group will be shown in the eRA Commons.

As part of the scientific peer review, all applications:

• May undergo a selection process in which only those applications deemed to have the highest scientific and technical merit (generally the top half of applications under review) will be discussed and assigned an overall impact score.
• Will receive a written critique.

Applications will compete for available funds with all other recommended applications submitted in response to this FOA.

The following will be considered in making funding decisions:

• Scientific and technical merit of the proposed project as determined by scientific peer review.
• Availability of funds.
• Responsiveness to goals and objectives of the FOA.
• Relevance and fit within AHRQ research priorities, as well as overall programmatic and geographic balance of the proposed project to program priorities.

After the peer review of the application is completed, the PD/PI will be able to access his or her Summary Statement (written critique) via the eRA Commons.

Information regarding the disposition of applications is available in the HHS Grants Policy Statement (see https://www.ahrq.gov/funding/policies/hhspolicy/index.html).

If the application is under consideration for funding, AHRQ will request "Just-In-Time" information from the applicant. Just-In-Time information generally consists of information on other support, any additional information necessary to address administrative and budgetary issues, and certification of IRB approval of the project's proposed use of human subjects. For details regarding IRB approval, applicants may refer to the "AHRQ Revised Policy for Institutional Review Board (IRB) Review of Human Subjects Protocols in Grant Applications" (https://grants.nih.gov/grants/guide/notice-files/not-hs-00-003.html).

If all administrative and programmatic issues are resolved, a formal notification in the form of a Notice of Award (NoA) will be provided to the applicant organization for successful applications. The NoA signed by the AHRQ grants management officer is the authorizing document and will be sent via email to the e-mail address designated by the recipient organization during the eRA Commons registration process.

Awardees must comply with any funding restrictions described in Section IV.6. Funding Restrictions. Selection of an application for award is not an authorization to begin performance. Any costs incurred before receipt of the NoA are at the recipient's risk. These costs may be reimbursed only to the extent considered allowable pre-award costs.

Any application awarded in response to this FOA will be subject to the DUNS, SAM Registration, and Transparency Act requirements as noted on the AHRQ web site at https://www.ahrq.gov/funding/grant-mgmt/index.html.

All AHRQ grant and cooperative agreement awards are subject to OMB's Uniform Grant Guidance, HHS’s Uniform Administrative Requirements, Cost Principles, and Audit Requirements for HHS Awards, (http://www.ecfr.gov/cgi-bin/text-idx?node=pt45.1.75), and the HHS Grants Policy Statement (see http://www.ahrq.gov/funding/policies/hhspolicy/index.html), and the terms and conditions set forth in the Notice of Award. As necessary, additional Terms and Conditions will be incorporated into the award statement.

A standard term and condition of award will be included in the Notice of Award (NOA) that states: in any grant-related activity in which family, marital, or household consideration are, by statute or regulation, relevant for purposes of determining beneficiary eligibility or participation, grantees must treat same-sex spouses, marriages, and households on the same terms as opposite-sex spouses, marriages, and households, respectively. By same-sex spouses, HHS means individuals of the same sex who have entered into marriages that are valid in the jurisdiction where performed, including any of the 50 states, the District of Columbia, or a U.S. territory or in a foreign country, regardless of whether or not the couple resides in a jurisdiction that recognizes same-sex marriage. By same-sex marriages, HHS means marriages between two individuals validly entered into in the jurisdiction where performed, including any of the 50 states, the District of Columbia, or a U.S. territory or in a foreign country, regardless of whether or not the couple resides in a jurisdiction that recognizes same-sex marriage. By marriage, HHS does not mean registered domestic partnerships, civil unions or similar formal relationships recognized under the law of the jurisdiction of celebration as something other than a marriage.

All applicants will be subject to a term and condition that applies the terms of 48 CFR section 3.908 to the award, and requires that grantees inform their employees in writing of employee whistleblower rights and protections under 41U.S.C. 4712 in the predominant native language of the workforce.

Recipients of federal financial assistance (FFA) from HHS must administer their programs in compliance with federal civil rights law. This means that recipients of HHS funds must ensure equal access to their programs without regard to a person’s race, color, national origin, disability, age and, in some circumstances, sex and religion. This includes ensuring your programs are accessible to persons with limited English proficiency. HHS recognizes that research projects are often limited in scope for many reasons that are nondiscriminatory, such as the principal investigator’s scientific interest, funding limitations, recruitment requirements, and other considerations. Thus, criteria in research protocols that target or exclude certain populations are warranted where nondiscriminatory justifications establish that such criteria are appropriate with respect to the health or safety of the subjects, the scientific study design, or the purpose of the research.

For additional guidance regarding how the provisions apply to AHRQ grant programs, please contact the Scientific/Research Contact that is identified in Section VII under Agency Contacts of this FOA. HHS provides general guidance to recipients of FFA on meeting their legal obligation to take reasonable steps to provide meaningful access to their programs by persons with limited English proficiency. Please see https://www.hhs.gov/civil-rights/for-individuals/special-topics/limited-english-proficiency/fact-sheet-guidance/index.html. The HHS Office for Civil Rights also provides guidance on complying with civil rights laws enforced by HHS. Please see http://www.hhs.gov/ocr/civilrights/understanding/section1557/index.html; and https://www.hhs.gov/civil-rights/index.html Recipients of FFA also have specific legal obligations for serving qualified individuals with disabilities. Please see https://www.hhs.gov/civil-rights/for-individuals/disability/. l. Please contact the HHS Office for Civil Rights for more information about obligations and prohibitions under federal civil rights laws at https://www.hhs.gov/ocr/about-us/contact-us/index.html o or call 1-800-368-1019 or TDD 1-800-537-7697. Also note it is an HHS Departmental goal to ensure access to quality, culturally competent care, including long-term services and supports, for vulnerable populations. For further guidance on providing culturally and linguistically appropriate services, recipients should review the National Standards for Culturally and Linguistically Appropriate Services in Health and Health Care at http://minorityhealth.hhs.gov/omh/browse.aspx?lvl=2&lvlid=53.

In accordance with the statutory provisions contained in Section 872 of the Duncan Hunter National Defense Authorization Act of Fiscal Year 2009 (Public Law 110-417), AHRQ awards will be subject to the Federal Awardee Performance and Integrity Information System (FAPIIS) requirements. FAPIIS requires Federal award making officials to review and consider information about an applicant in the designated integrity and performance system (currently FAPIIS) prior to making an award. An applicant, at its option, may review information in the designated integrity and performance systems accessible through FAPIIS and comment on any information about itself that a Federal agency previously entered and is currently in FAPIIS. The Federal awarding agency will consider any comments by the applicant, in addition to other information in FAPIIS, in making a judgment about the applicant s integrity, business ethics, and record of performance under Federal awards when completing the review of risk posed by applicants as described in 45 CFR Part 75.205 Federal awarding agency review of risk posed by applicants. This provision will apply to all NIH grants and cooperative agreements except fellowships.

When multiple years are involved, awardees will be required to submit the Research Performance Progress Report (RPPR) annually unless specified otherwise in the terms of the award. Grantees are required to submit expenditure data on the Federal Financial Report (FFR; SF 425) annually.

For details regarding annual progress report submission, refer to https://www.ahrq.gov/funding/grant-mgmt/noncomp.html. If instructions on the AHRQ website are different from the RPPR Instructions, follow the instructions on the AHRQ website. Annual progress reports are due 3 months before the anniversary of the award.

Expenditure data is to be reported on the Federal Financial Report (FFR; SF 425). AHRQ requires annual financial expenditure reports for ALL grant programs as described in the HHS Grants Policy Statement (https://www.ahrq.gov/funding/policies/hhspolicy/index.html). AHRQ implementation of the FFR retains a financial reporting period that coincides with the budget period of a particular project. However, the due date for annual FFRs is 90 days after the end of the calendar quarter in which the budget period ends. For example, if the budget period ends 4/30/2018, the annual FFR is due 9/30/2018 (90 days after the end of the calendar quarter of 6/30/2018).

A final Progress Report, final Federal Financial Report, and Final Invention Statement are required when an award ends. All final reports are due within 90 days of the project period end date. For further details regarding grant closeout requirements, refer https://www.ahrq.gov/funding/grant-mgmt/closeout.html.

The Federal Funding Accountability and Transparency Act of 2006 (Transparency Act), includes a requirement for awardees of Federal grants to report information about first-tier subawards and executive compensation under Federal assistance awards issued in FY2011 or later. All awardees of applicable AHRQ grants and cooperative agreements are required to report to the Federal Subaward Reporting System (FSRS) available at www.fsrs.gov on all subawards over $25,000. See the HHS Grants Policy Statement (https://www.ahrq.gov/funding/policies/hhspolicy/index.html) for additional information on this reporting requirement.

In accordance with the regulatory requirements provided at 45 CFR 75.113 and Appendix XII to 45 CFR Part 75, recipients that have currently active Federal grants, cooperative agreements, and procurement contracts from all Federal awarding agencies with a cumulative total value greater than $10,000,000 for any period of time during the period of performance of a Federal award, must report and maintain the currency of information reported in the System for Award Management (SAM) about civil, criminal, and administrative proceedings in connection with the award or performance of a Federal award that reached final disposition within the most recent five-year period. The recipient must also make semiannual disclosures regarding such proceedings. Proceedings information will be made publicly available in the designated integrity and performance system (currently FAPIIS). This is a statutory requirement under section 872 of Public Law 110-417, as amended (41 U.S.C. 2313). As required by section 3010 of Public Law 111-212, all information posted in the designated integrity and performance system on or after April 15, 2011, except past performance reviews required for Federal procurement contracts, will be publicly available. Full reporting requirements and procedures are found in Appendix XII to 45 CFR Part 75 Award Term and Conditions for Recipient Integrity and Performance Matters.

• The institution must submit a completed Statement of Appointment (PHS Form 2271) for each scholar appointed or reappointed to the grant. Grantees must submit the PHS 2271 data electronically using the xTrain system. More information on xTrain is available at xTrain (eRA Commons). An appointment or reappointment may begin any time during the budget period, but not before the budget period start date of the grant year.
• A notarized statement verifying possession of permanent residency documentation must be submitted with the Statement of Appointment (PHS Form 2271). Individuals with a Conditional Permanent Resident status must first meet full (non-conditional) Permanent Residency requirements before receiving support.

Termination Notice: Within 30 days of the end of the total support period, the institution must submit a Termination Notice (PHS Form 416-7) via xTrain for each Scholar.

In carrying out its stewardship of human resource-related programs, the AHRQ may request information essential to an assessment of the effectiveness of this program from databases and from participants themselves. Participants may be contacted after the completion of this award for periodic updates on various aspects of their employment history, authorship of scientific publications, support from independent research grants or contracts, honors and awards, professional activities, and other information helpful in evaluating the impact of the program.

The overall evaluation of the program will be based on metrics that will include, but are not limited to, the following:

For programs involving postdoctorates and early career investigators:

• Subsequent participation in a formal research training or career development program relevant to the proposed career development program
• Subsequent participation in research or employment in a LHS leading effort in PCOR to transform delivery of care
• Authorship of scientific publications in the field of learning health system research
• Subsequent independent research grant support from NIH, AHRQ, or another source

We encourage inquiries concerning this funding opportunity and welcome the opportunity to answer questions from potential applicants.

eRA Commons Help Desk (Questions regarding eRA Commons registration, submitting and tracking an application, documenting system problems that threaten submission by the due date, post submission issues):
Finding Help Online: http://grants.nih.gov/support/ (preferred method of contact)
Telephone: 301-402-7469 or 866-504-9552 (Toll Free)

Grants.gov Customer Support (Questions regarding Grants.gov registration and submission, downloading forms and application packages)
Contact Center Telephone: 800-518-4726
Email: support@grants.gov

Grants Info (Questions regarding application instructions and process, finding NIH grant resources)
Email: GrantsInfo@nih.gov (preferred method of contact)
Telephone: 301-945-7573

C. Jean Hsieh, PhD, OT
Agency for Healthcare Research and Quality
Telephone: 301-427-1534
Email: Chinghui.Hsieh@ahrq.hhs.gov

Nghia Vo, M.D.
Agency for Healthcare Research and Quality
Telephone: 301-427-1191
Email: Nghia.Vo@ahrq.hhs.gov

Ms. Galen Gregor
Agency for Healthcare Research and Quality
Telephone: 301-427-1457
Email: Galen.Gregor@ahrq,hhs.gov

Recently issued AHRQ policy notices may affect your application submission. A full list of policy notices published by AHRQ is provided in the NIH Guide for Grants and Contracts. Notices can also be found at AHRQ Grants Policy Notices. All awards are subject to the terms and conditions, cost principles, and other considerations described in the HHS Grants Policy Statement.

This program is described in the Catalog of Federal Domestic Assistance at http://www.cfda.gov/ and is not subject to the intergovernmental review requirements of Executive Order 12372 or Health Systems Agency review. Awards are made under the authority of 42 USC 299 et seq., and in accordance with 45 CFR Part 75 and other referenced applicable statutes and regulations. All awards are subject to the terms and conditions, cost principles, and other considerations described in the HHS Grants Policy Statement. The HHS Grants Policy Statement can be found at https://www.ahrq.gov/funding/policies/hhspolicy/index.html.