No Petri Dish
Centers for Disease Control and Prevention (CDC)
Summary
Surveillance of infectious diseases is an integral part of public health prevention and control programs. A number of these programs have successfully implemented networks to detect and investigate outbreaks and to implement effective, targeted controls. Examples of these networks include PulseNet, the National Tuberculosis Surveillance System and the Active Bacterial Core Surveillance program, among others. Each of these programs relies on the culture and testing of patient isolates at community and clinical laboratories, and the subsequent submission of these cultured isolates to state and local public health laboratories for detailed analysis.
A new generation of commercial diagnostic tests are now entering the marketplace that offer physicians faster results, more types of information than were previously available in some cases, and that can test for broad groups of pathogens based on patient symptoms and presentation. Unfortunately, unlike culture-based methods, these new tests do not result in isolates being available for public health purposes, and it is expected that this lack of isolates will make it increasingly difficult or impossible to detect and investigate outbreaks or other important public health trends. In order to maintain and improve important public health surveillance activities, new laboratory methods are needed that do not depend on isolates or cultures.
The "No Petri Dish" Challenge asked an open-ended question and was broken into two phases to evaluate submissions. (File Photo)
The Shiga-toxin-producing Escherichia coli (STEC) represents a difficult challenge. STEC is similar in most respects to commensal E. coli (i.e., strains of E. coli normally found in the human gut). Tracking the pathogen therefore requires reliably identifying a variable marker that can be used for subtyping and to link these data to other markers that can be used for identification and characterization of the pathogen, all within a "dirty" and complex sample matrix (i.e., a stool sample). The challenge was to "describe a novel or innovative method to strain-type and characterize pathogenic organisms, such as STEC, directly from a complex clinical sample, without the need for culture or culture-based amplification."
Contestants submitted their solutions between Sept. 1, 2014 and Nov. 30, 2014. The winner was notified on Dec. 15, 2014. An award of $200,000 went to the winning entrant. Only one winner was selected.
This challenge asked an open-ended question, and thus, the challenge organizers expected a wide range of proposed solutions. The competition was consequently split into two phases. In the first phase, all qualified submissions that were received during the challenge period were submitted to an expert review by a panel of CDC and external subject matter experts. In the second phase, the leading proposal from the technical review was invited to participate in a timed challenge, using a blinded set of synthetic datasets that had been designed by the technical review panel. This two-phased approach allowed the No Petri Dish Challenge to consider a wide range of potential technical solutions, while enabling the reviewers to tailor validation and verification of performance claims using expert-curated data.
Results
The "No Petri Dish" Diagnostic Test Challenge attracted considerable interest from both academia and industry through social media and email inquiries, and received some creative submissions and feedback during the contest's open period. Eight submissions were received from seven project teams. Two submissions were ineligible for the challenge due to citizenship restrictions under the America COMPETES Act. The remaining six submissions from academic centers and nonprofit organizations presented a wide range of solutions.
The technical review panel selected the solution that was submitted by OneCodex, a YCombinator-backed startup that had developed an elegant, cloud-based metagenomic analytical platform. OneCodex already had been developing their software platform at the time of the challenge, and they were able to incorporate many new and useful features as a result of the challenge requirements and subsequent CDC feedback. The OneCodex team was then contacted and invited to participate in a second and final challenge phase before a final award determination was made. In this second phase, CDC subject matter experts generated five synthetic datasets that approximated whole-sample metagenomic data from human stool with differing levels of different reference strains of STEC. The OneCodex team was given 24 hours to analyze the samples on its platform, determine what strains (if any) were present in each sample and provide whatever additional details and characteristics (e.g., antimicrobial resistance genotype) it could extract from the samples. While the OneCodex team did not submit a perfect solution for all samples (all samples were challenging; several were deliberately impossible), its solution impressed the review committee, as did the quality and clarity of the report they generated.
The OneCodex team was awarded the grand prize of $200,000 for this challenge on Jan. 15, 2015.
Areas of Excellence
Area of Excellence #1: "Design the Challenge"
This challenge posed a very open-ended and difficult question, and the CDC challenge team recognized from the outset that a wide range of potential solutions could be proposed. As a result, the challenge was carefully designed to include two phases: an initial technical and scientific review by an expert panel and a proof-of-viability phase, which would have been adjusted according to the finalists from the technical review. In the case of the No Petri Dish challenge, the finalist proposed a bioinformatic analysis solution, and as such, the technical review panel developed a blinded set of simulated data from samples to validate the finalist's solution and verify its claims of performance. Had one or more laboratory-based solutions been proposed, the review panel was also prepared to distribute a similarly blinded set of simulated human clinical specimens. Upon submission of a successful result to the timed Phase 2 of the challenge, a single grand prize award was made to the winning team.
In retrospect, this design and incentive structure was ideal for the challenge. The No Petri Dish Challenge was essentially a single award, winner-takes-all challenge with two phases. Because we asked a very open-ended question and expected a wide range of proposed solutions, we knew we would need a second phase to adequately assess the finalists. In future challenges, depending on the nature and scope, the challenge team may consider adjusting the value of the overall prize pool and including prizes for finalists and runners up.
Area of Excellence #2: "Create an Implementation Plan"
The implementation plan for this challenge was developed through ongoing consultation between the Division of Foodborne, Waterborne, and Enteric Diseases (DFWED), the Division of Healthcare Quality Promotion (DHQP), and the Office of Advanced Molecular Detection (OAMD). The challenge team developed the challenge question and evaluation criteria, drafted all project documents and web content and coordinated the technical review.
This collaboration on the design and implementation of the challenge was important, because while these different stakeholder groups all had shared interest in metagenomics and culture-independent diagnostics, their specific objectives and priorities differed. In drafting the implementation plan for this challenge, the project team was able to align objectives and evaluation criteria, resulting in improvements to the design of the challenge and a higher level of engagement from CDC stakeholders. In retrospect, dedicated support personnel or contract support would have helped this challenge run more smoothly, since the CDC core challenge team was small and burdened with other responsibilities. Even so, this was a successful challenge, with a very successful outcome.
Area of Excellence #3: "Manage Solutions"
Following the challenge competition, CDC subject matter experts from the Division of Foodborne, Waterborne and Environmental Diseases (DFWED) and the Office of Advanced Molecular Detection (OAMD) worked with the winning team to develop and refine several of the new platform features that were developed as a result of the challenge, to improve reference database curation and to assess the utility of the software for the analysis of a variety of primary clinical and environmental samples.
CDC and OneCodex are continuing to collaborate on the development of software tools to analyze genomic sequencing data from complex metagenomic samples. While its target customer base is clinical diagnostics, the challenge competition encouraged OneCodex to develop a number of features that will be of direct benefit to public health applications, and CDC is confident that the challenge competition has secured lasting interest and commitment from OneCodex's leadership and scientific team.
The most significant barrier to long-term engagement with this solution is not scientific or technical, but rather current limitations of federal cloud-computing policy. The OneCodex platform, which runs as a Software as a Service (SaaS) offering on top of Amazon Web Services, does not have the necessary FedRAMP approvals for routine standalone use. Consequently, CDC programs must continue to work with the enterprise information security office to determine how to authorize access to its software and other similar tools. In the meantime, CDC has been collaborating with the company and has conducted pilot tests on its platform using carefully selected non-sensitive and de-identified or simulated data.
Challenge Type
Scientific
This challenge was a success on several important levels. First, it resulted in the identification of a software solution that met an important and emerging public health need and established an important precedent for engagement, innovation and collaboration between public health and the biotech startup community. Second, the competition elicited significant interest around the public health challenges posed by new non-culture-based clinical diagnostic assays and has helped to foster academic research around potential solutions. And third, it has helped to establish challenge competitions and public-private partnerships as key drivers for innovation at the agency.
Legal Authority
America COMPETES Act