Summary

At the National Institutes of Health (NIH), reform is coming, but many questions remain about what a more effective NIH should look like. A Center for NIH Innovation should routinely roll out new ideas for how to structure science funding, how to do peer review, and how to find the best "high-risk" research, and it should then run experiments on those ideas.

Problem

It is time for serious NIH reform. All of the relevant congressional committees issued major white papers in 2024 detailing ideas for NIH reform, and incoming NIH director Jay Bhattacharya has indicated strong interest in reform. Even establishment figures have written that NIH's "research enterprise has become sclerotic, cautious, focused on doing what it has always done and withdrawing from clinical research," and that NIH "needs to shake off its doldrums and embrace the very traits that are essential in generating outstanding science—creativity, persistence, and courage."

But we know surprisingly little about the best science funding approaches. Open questions include:

• Is it better to give more funding to the person, not the project?
• Are 10-year grants more effective than 3- or 4-year grants?
• Is peer review the best system for grant allocation?
• Should we "red-team" scientific fields in which funded grants all seem to follow one particular theory?
• Should NIH program officers have more discretion to overrule or even bypass peer review?
• What's the optimal method of scientific training and education? Top athletes, musicians, and CEOs almost always have personal coaches; is there a way that we could provide "personal trainers" to top scientists so as to broaden their impact?
• What would happen if we demanded that all scientific grants publicly report at least one "failure?" Would we encourage more risk-taking and/or truth-telling?
• What's the best way of funding "high-risk, high-reward" research?
• What's the best way of recruiting top talent to biomedicine, given the long and uncertain pathway to an NIH-funded career?

Although many reform ideas have been proposed, there is little rigorous evidence as to what works and when. For example, the Canadian government recently commissioned an international expert panel on scientific funding, led by former Princeton President Shirley Tilghman. The panel found that across the world: "Many funding practices seem promising but are not substantiated by rigorous evaluations." As another major report pointed out, "research funders... rarely systematically experiment with different ways to design and run their funding programmes. As a result, research funders are missing out on opportunities to achieve their goals in a cost-effective way, and to further accelerate the progress of science."

In short, we don't know very much. That's why we need to "turn the scientific method on ourselves." We hand out billions of dollars every year to scientists who are expected to follow rigorous principles of observation and experimentation, but we do not use those same principles to study how we hand out the money in the first place.

The most important priority for NIH should therefore be to start a more deliberate process of internal experimentation and evaluation. That way, we will learn from many reform ideas (which ones to keep, which ones to discard, and which ones to modify), thus leading to improved scientific innovation over time.

Running randomized experiments within government, however, can be seen as high-risk. The last thing any NIH official wants is to be called in front of Congress to answer complaints from famous scientists upset about how funding was handed out. If we want NIH to experiment with new approaches to peer review, etc., policymakers should empower a specific team with the responsibility, mandate, and budget to do so.

Solution

Executive

HHS should establish a Center for NIH Innovation (CNI), preferably by following the statutory process of invoking the Scientific Management Review Board (see 42 U.S.C. § 281) and producing a report with the appropriate notice to Congress. Alternatively, Congress could create CNI directly through authorizing legislation or through appropriations. The current statutory framework limits the number of Institutes and Centers to a total of 27: see 42 U.S.C. § 281(d). Congress would either have to raise this number to 28, or else HHS/Congress could agree on eliminating another Institute/Center (IC) (most people would suggest the National Center for Complementary and Integrative Health).

CNI's mission would be to develop pilot experiments as to how NIH hands out funds, evaluate the results, and promote more widespread adoption of successful programs. Several key features would ensure CNI's success:

• CNI should be funded directly by Congress as an independent Center or as a line item within the Director's Office, and should be guaranteed funding on at least a 3-5 year time scale so that it can take up longer-term experiments.
• CNI should be able to require the participation of other NIH ICs in ongoing experiments. For example, if CNI wants to imitate the National Science Foundation in performing an experiment with peer review, the Center for Scientific Review should be required to participate in that experiment in good faith.
• CNI should be empowered to waive statutory requirements that otherwise apply to NIH, such as the requirement that all grants be approved by a majority of peer reviewers (see 42 U.S.C. § 289a-1). Waiver authority is important to ensure that CNI doesn't constantly need to return to Congress to get special permission to engage in an innovative experiment, such as allowing program officers to use a "golden ticket" or a limited lottery.
• Finally, CNI should be empowered to take approaches that work, and scale them up into NIH-wide policies, so that the rest of NIH benefits from experimental learnings.

Justification

The idea of regular experimentation, evaluation, and feedback loops has become popular throughout government over the past decade or more. Several other federal agencies have been creating offices to engage in regular experimentation and testing, including the Center for Medicare and Medicaid Innovation, the Office of Investor Research at the SEC and its POSITIER initiative, the Office of Healthcare Innovation and Learning at Veterans Affairs, and the Office of Evaluation Sciences at the Government Services Administration (which grew out of the White House Social and Behavioral Sciences Team). Collectively, these offices and centers have launched well over 150 experiments within government, including everything from a cardiovascular disease risk reduction tool to improving a suicide prevention hotline to changing how we pay for end-stage kidney disease.

While experiments with science funding are rare to date, NIH did run an experiment several years ago on whether blinding peer reviewers to an applicant's identity would reduce racial disparities. More recently, the National Science Foundation (NSF) in partnership with the Institute for Progress has launched an experiment with the so-called "golden ticket" approach, in which a grant can get funded if one peer reviewer loves it (and deploys a metaphorical "golden ticket") even if the other reviewers dislike it. The idea is that some truly breakthrough ideas go unappreciated at first, and we might find more such ideas if we looked for cases where one reviewer saw the potential. That said, NSF's pilot experiment is being conducted on a private basis, and the results may not be public. NIH should be required to make all results public, absent a very compelling reason (such as a serious risk of compromising patient privacy). But NSF's willingness to try different peer review approaches is hopefully the first of many experiments that will improve the effectiveness of science funding in spurring American innovation.

Appendix

Sample Legislative Text:

Section 401 of the Public Health Service Act (42 U.S.C. § 281) is amended by inserting the following new subsection (g) after subsection (f) (with the current subsections (g) and (h) being renamed (h) and (i) respectively):

"(g) Center for NIH Innovation.

(1) In General—There is hereby created a Center for NIH Innovation ("CNI") at the National Institutes of Health to carry out the duties described in this section, with the overall goal of accelerating the pace of biomedical advancement. The purpose of CNI is to work with other NIH Institutes and Centers to test new ways of sourcing, reviewing, and funding grants and contracts; measure the impact of pilot projects and experiments; and scale up innovations that are successful.

(2) Deadline—The Secretary of HHS shall ensure that CNI launches and is able to carry out its statutory mission by [date].

(3) Organization—The CNI Director shall be appointed for up to two 5-year terms by the NIH Director. The Office of Evaluation, Performance and Reporting and the Office of Portfolio Analysis are hereby consolidated with CNI and shall report to the CNI Director.

(4) Consultation with Advisory Council—CNI shall create an Advisory Council with 5 or more representatives from other NIH Institutes/Centers, 5 or more representatives from universities, and 3 or more researchers with expertise in meta-science. This advisory council shall meet at least twice each calendar year, and shall provide CNI with expert advice on ideas for experimentation and evaluation of NIH's processes. CNI shall also make an email address available to the public for suggesting other ideas and relevant submissions shall be considered by CNI and its Advisory Council at the biannual meetings.

(5) Selection of Ideas to Be Tested—CNI shall select ideas to test by gathering evidence as to promising methods for improving science funding from other national science funders, the advisory council, the academic literature, or other submissions of ideas.

(6) Evaluation—

(A) Where possible, CNI shall attempt to use randomization or cutoff-based methods to pilot new science funding models and to determine their effects.

(B) CNI shall evaluate the success of alternative scientific funding models by a diversity of outcomes, including qualitative evidence, citations, patents, prominence of new discoveries, and other signals of scientific achievement. CNI shall additionally explore funding external meta-scientific work to determine how best to measure the outcomes of scientific funding, and whether various short-term outcomes are indicative of longer-term outcomes.

(C) The results of any pilot, experiment, or other evaluation shall be made fully public and transparent (absent good cause, such as patient privacy).

(7) Waiver Authority-CNI may waive the requirements of the Public Health Service Act as regards peer review or any other issue as may be necessary for purposes of carrying out its mission with respect to testing new models of NIH funding;

(8) Limitations on Review—There shall be no administrative or judicial review of –

(a) The selection of NIH funding models for testing or expansion under this section;

(b) The selection of organizations, sites, or participants to test those models; or,

(c) The elements, parameters, scope, and duration of such funding models.

(9) Partnership With Other NIH Institutes and Centers—Other NIH Institutes and Centers shall work under CNI's leadership to test new ideas for performing peer review, sourcing scientific ideas, funding innovative research, and the like. If an Institute or Center does not wish to participate in a CNI-led study or evaluation, its objection may be overruled by a majority vote of the Advisory Council described in subsection (4).

(10)Partnership With Outside Expertise–CNI should regularly work with outside scholars to share NIH's internal data on proposals and peer review scores, and to partner with them on pilots and experiments. CNI should also investigate a partnership with the Office of Evaluation Sciences (OES) at the Government Services Administration, and should report back to Congress within one year of enactment on a possible partnership with OES.