Follow That Cell
National Institutes of Health (NIH)
Summary
The NIH Single Cell Analysis Program (SCAP) is searching for novel, robust methods for analysis of individual cells that can serve as the basis for predicting alterations in cell behavior and function over time. The ultimate goal is to develop new tools and methods that allow time-dependent measurements at the single cell level in a complex tissue environment to assess functional changes, provide information on the health status of a given cell and help guide diagnosis and therapeutic treatments related to human disease states. The NIH believes the Follow that Cell Challenge will stimulate investment from both public and private sectors in single-cell-analysis research and product development, which, in turn, could lead to the development of more sensitive, robust and cost-effective assay approaches, reagents, tools and devices for basic research and clinical diagnosis. Through this $500,000 prize competition, the NIH Single Cell Analysis Program aims to spur the development of innovative solutions that allow the research community to "follow that cell."
The Follow that Cell Phase 1 winners and finalists pose for a photograph. (Photo courtesy of NIH)
This challenge was structured in two linked phases. Phase 1, which required a written solution describing a novel method for analyzing dynamic states of individual cells that can serve as the basis for predicting alterations in cell behavior and function over time, was initiated on Aug. 15, 2014 with prize winners and finalists announced on March 16, 2015. Phase 1 prize winners and finalists were invited to attend the third annual Single Cell Analysis Investigators meeting on April 20-21, 2015, at which they presented posters on their theoretical solutions.
Phase 2, a reduction to practice where solvers will execute their proposed solutions described in Phase 1 and produce single cell data in support of a significant biological or clinical question by measuring changes in a single cell over time, was open only to Phase 1 prize winners and finalists. Phase 2 submissions are due by March 30, 2017, and winners will be announced on July 31, 2017.
InnoCentive, Inc., supported the development, hosting and marketing of the challenge under contract with NIH.
Results
Phase 1 of the Follow that Cell Challenge is complete, and of the 33 eligible submissions received, 16 were selected as Phase 1 finalists, with cash prizes and travel reimbursement awarded to five solvers totaling $100,000. These finalists were from a variety of academic disciplines, including biology, physics, bioengineering, pharmacy, computer sciences, neurosciences and others. Two of the finalists are postdoctoral fellows, six are assistant professors, two are associate professors and six are tenured professors. Six of the finalists had never received an NIH research grant prior to the challenge; three had received only one grant. Submissions were received from institutions across all regions of the United States.
Each of the submissions from the Phase 1 finalists described potential advances that, if successful in Phase 2, would significantly impact research efforts. The top five (prize-winning) submissions offer highly innovative solutions that show particular promise and originality. All proposed solutions from the Phase 1 finalists are posted on the NIH Single Cell Analysis Program website with an executive summary of each solution.
NIH practice currently does not allow the use of federal grant or contract money to support development of solutions. Organizers did follow up with the Phase 2 solvers and found that some have secured non-federal funding to support Phase 2; however, other Phase 2 solvers have not yet provided this information. The organizers do not expect the 16 finalists to convene again, but the teams are mostly still intact and working on Phase 2. NIH has requested six-month status updates and have encouraged the solvers to bring any issues or problems to the attention of the NIH scientific team.
Phase 2 started in March 2015 and will finish in 2017. Up to two prizes may be awarded from the total pool of $400,000 for Phase 2, and depending on the scientific assessments by the Technical Evaluation Panel, the judges will determine how to allocate the prize pool.
Areas of Excellence
Area of Excellence #1: "Determine If a Prize Is Appropriate"
Between 2011 and 2012, the NIH issued requests for information (RFIs) and held two workshops asking the research community how best to accelerate research in single cell analysis and to identify key challenges in the area of single cell research. In response to this public input, the NIH Single Cell Analysis Program initiated several programs, including studies to evaluate cellular heterogeneity using transcriptional profiling of single cells through cooperative research grants, projects to develop exceptionally innovative tools and technologies for single cell analysis through exploratory/developmental grants and efforts in accelerating the integration and translation of technologies to characterize biological processes at the single cell level through research project grants. While these efforts are already advancing the field of single cell analysis, a recurring theme from the RFIs and workshops was the need to conduct multiple tests on a single cell over time.
At this point, NIH wanted to know if there were other ways to stimulate efforts from a broader and more diverse community, and ultimately it was decided to run a challenge to provide another avenue for seeking innovation. Through this challenge mechanism NIH was looking for a way to stimulate fresh ideas and engage individuals or teams from different backgrounds, perhaps teaming with biologists, to propose new and creative ways to measure cells in a population over time.
Area of Excellence #2: "Estimate Budget and Resources"
The NIH Single Cell Analysis Program determined, through its budget development process, to set aside $500,000 and the necessary administrative costs to develop and announce a challenge. SCAP decided that the award needed to be robust enough to attract a large pool of solvers. As with many cross-organization initiatives, the majority of the work performed by core staff is in addition to their regular full-time duties, and the cost of staff time is borne by their home organization.
The Follow that Cell Challenge was the first purely scientific challenge run by NIH, and the amount of time and effort from staff and experts needed to define the problem, develop evaluation criteria and address all administrative aspects of eligibility, intellectual property and timeline was substantially more than initially anticipated. Additionally, the initial cost of the support contract increased, due to the decision to structure the Follow that Cell Challenge as a two-phased challenge and required InnoCentive, the contractor, to host two separate phases, adding an additional two-and-a-half years to the timeline.
The Follow that Cell Challenge was led by the coordinators for the Single Cell Analysis Program Working Group and administratively managed by the National Institute of Mental Health (NIMH) Challenge Manager on behalf of the NIH Common Fund. Additional individual scientific support was provided by another scientific leader in the Working Group and a scientific program analyst. InnoCentive, Inc., supported the development, hosting and marketing of the challenge under contract with NIH.
A number of other individuals and groups were also critical to the development and execution of the Follow that Cell Challenge, including the NIH Challenge Manager, contracting specialists, NIH Federal Register coordinator, senior attorneys at the HHS Office of General Counsel, other senior professional staff at NIMH and NIH and clerical support staff. NIH leadership was involved in approval and judging, including the Co-Chairs of the Single Cell Analysis Working Group; the Director of the Division of Program Coordination, Planning, and Strategic Initiatives; and the NIH Principal Deputy Director. Several groups provided advice and assistance including the SCAP Coordinators, SCAP Challenge Working Group, Subject Matter Experts, SCAP Working Group, Technical Evaluation Panel, Program Evaluators, and the Communications Team. Overall, a total of 80 individuals, or about 2.7 full-time equivalents, contributed to the Follow that Cell Challenge development and the execution of Phase 1.
Additional direct costs included the amount of the prize pools and the contractual cost to InnoCentive, from January 2014 through July 2017, to support development, hosting, marketing, screening and prize payments for the challenge.
Area of Excellence #3: "Develop Submission Requirements"
The submission requirements and evaluation criteria for both phases of the Follow that Cell Challenge were announced in one Federal Register Notice. Specific instructions for Phase 1 submissions included a description of the proposed solution with page-length limitations, font size and margins; a required executive summary; background information; a state-of-the-art statement; a feasibility assessment; and a biographical sketch. These were needed for thoroughness of solution and also for the evaluators and judges to have uniformity and thereby ensure fairness in the evaluation. Phase 2, a reduction to practice, also requires a detailed project description, in addition to the execution of the solution and information on the quality and significance of the data. The Federal Register Notice stated that additional submission requirements would be available to Phase 2 solvers no later than 30 days after the Phase 1 finalists were announced. NIH held a webinar on April 10, 2015 for all eligible Phase 2 solvers to go over the requirements as posted in the Federal Register Notice. The slides from the webinar were posted on the NIH website. Additionally, the NIH recognizes that some solvers may encounter challenges or opportunities in Phase 2 that were not anticipated in the theoretical Phase 1, so Phase 2 submissions allow for documenting and explaining any changes from the original design in Phase 1. Lastly, NIH has asked Phase 2 solvers to submit a one-page status update at six-month intervals; however, since this status report was not part of the Federal Register Notice, it cannot be a requirement for Phase 2.
Submissions for Phase 1 of the challenge were evaluated using specific criteria for:
- time course measurements;
- predictability;
- cellular environment;
- significance; and
- adaptability plus feasibility, throughput and data content.
Phase 2 will be evaluated using the same criteria. Additionally, the Federal Register Notice stated that as part of the evaluation process for Phase 2, the technical evaluation panel may request a demonstration of the technology.
Area of Excellence #4: "Complete Reporting"
The NIH Follow that Cell Challenge has submitted two reports to be included in the Office of Science and Technology Policy (OSTP) annual report on the use of the America COMPETES Act prize authority by agencies. One in fiscal year 2014 when the challenge was announced and the second in fiscal year 2015 after the completion of Phase 1. A final annual report will be due upon completion of Phase 2 for the fiscal year 2017 annual report. Reports are submitted through HHS to OSTP.
HHS requires a report five days after the selection of prize awardees and has provided a template. NIH suggests that challenge managers begin work on the template when the challenge is being developed and announced. Completing the template as you work through the challenge makes the final report much easier. Additionally, HHS compiles annual individual reports from challenges launched, continuing or completed in a given fiscal year. The information collected is used both for the synthesized report sent to OSTP and for internal strategy on how to provide future support. HHS requests informal updates a year after a challenge ends to better understand impacts on both the internal challenge sponsor and on the problem area.
NIH practice currently does not allow the use of federal grant or contract money to support development of solutions, so follow up with Phase 2 solvers was done to determine if they were able to secure non-federal funding support for Phase 2. Some Phase 2 solvers have secured non-federal funding for support of Phase 2, but other Phase 2 solvers have not provided this information. This information was included in the OSTP report.
Challenge Type
Scientific
Challenges seek to encourage innovation and solicit creative ideas. Much of the biomedical research supported by the federal government is through research grants and contracts, but challenges are a way to solicit additional new ideas from individuals who may or may not be familiar with the NIH. Even before the NIH Big Think in 2010, scientists had been discussing the need for new ideas in single cell research and specifically the importance of being able to pinpoint changes in a single cell within a population of cells. The Follow that Cell Challenge was designed to stimulate creative ideas from a broad audience with varied expertise, different backgrounds and different career stages. The solvers embraced this mechanism of seeking solutions to a tough problem and offered the following comments to describe their experience:
- The challenge generated an increased level of excitement
- They had fun putting together the proposal
- It pushed our thinking
- The proposal was easier without having to think about budget and personnel justification, and they were able to devote all the thinking to the science
- A challenge offers a novel approach by presenting the need and asking for solutions without constraints
- It was open to everyone, not just NIH researchers or funded researchers
- It generated creative ideas
- Solvers can submit proposals without strict NIH or NSF guidelines
- They enjoyed working on the solution
One lesson learned: in order to secure government-use intellectual property (IP) rights, as outlined in the rules of the Follow that Cell Challenge, agreements between solvers, employers and the NIH needed to be developed and signed. While the Phase 1 prize winners and finalists were announced on the schedule outlined in the Federal Register Notice, the actual prize payments to solvers could not be made until the agreements were signed. Agreements took one month to nearly one full year after the announcements to get signed, and prize winners were anxious about getting the promised prize money. All Phase 2 solvers were notified of the NIH requirement for a signed IP agreement, which should allow Phase 2 prizes to be awarded in a timely manner. NIH and HHS staff now plan to attend various meetings of research administrators to educate and explain challenges to individuals from the administrative side of the organization.
Legal Authority
America COMPETES Act