Reference Data Challenge

National Institute of Standards and Technology (NIST)

Summary

The Reference Data Challenge was a call to action for app developers to help improve the way NIST shares scientific reference data. Scientists and engineers need data—from the atomic weight of carbon and the structure of benzene to the most precise value for the speed of light. High quality physical and chemical reference data help researchers design experiments, build better products, solve health and environmental problems and even study the stars. NIST provides some of the most accurate and comprehensive datasets in the world, known as standard reference data (SRD). In this challenge, entrants created an app that used one (or more) of six popular SRD. A panel of judges—including internet pioneer Vint Cerf and the Department of Commerce Chief Data Officer Ian Kalin—selected the winning apps based on the apps' potential impact, creativity and innovation, implementation and use of SRD.

The challenge ran over a two-month period and was hosted on the Devpost website. A total of $45,000 in prizes was offered, with first place receiving $30,000, second place $10,000 and third place $5,000.

Results

As a result of the challenge, 25 new apps were created using NIST reference data, helping to initiate a modernization of NIST's publically accessible data. Over 130 participants registered on the site, building interest in NIST reference data. The contest also stimulated the growth of at least one new company: the first place winner, Meru Apps LLC, intends to develop products that will help laboratory scientists use data more effectively.

The top prize of $30,000 went to Kris Reyes from Meru Apps in Princeton, N.J. His app, Meru Lab Reference, allows users to quickly access NIST chemical species data with the tap of a near-field communication (NFC) tag, a smart chip that is able to store digital information and share it with a smartphone. The second place prize of $10,000 went to college students Zachary Ratliff from Waco, Texas, and Daniel Graham from Danville, Ky., for their Lab Pal app that is a "go-to" tool for scientists and engineers. The third place award of $5,000 went to a team from MetroStar Systems in Reston, Va., whose app, ChemBook, provides multimedia information about chemical elements from NIST and other sources. Honorable mention awards (no cash prize) went to Andy Hall's SciCalc9000 app, a scientific calculator that integrates NIST SRD, and Annie Hui and Neil Wood of R-Star Technology Solutions, whose Thermocouple Calibrator app is a handy tool for converting between voltage and temperature. This challenge kicked off a bigger NIST open data commitment to improve the accessibility of SRD for app developers and other users of this valuable scientific data.

Areas of Excellence

Area of Excellence #1: "Estimate Budget and Resources"

NIST personnel were involved in the design and execution of the Reference Data Challenge. In addition to a significant amount of the challenge manager's (a NIST employee) time, staff in the NIST Office of Information and Systems Management (OISM) were employed to generate machine readable (JSON) data files. NIST research and technical staff were consulted to help identify and prepare the data. NIST's Office of the Chief Counsel also dedicated time for scoping and finalizing the challenge rules and associated documentation per America COMPETES Act requirements. The challenge was hosted on a website provided by Devpost (formerly Challengepost). The prize funds, challenge website hosting and OISM time were billed to an account established for the purpose of the challenge. Other staff time was provided in kind or as part of established responsibilities. In total, executing the challenge used approximately one-fifth FTE (challenge manager) and two FTEs at 10 percent effort, and $37,500 for IT services (data preparation and challenge website).

Area of Excellence #2: "Develop Terms and Conditions"

The NIST challenge manager coordinated closely with NIST's Office of the Chief Counsel to develop a comprehensive set of rules that provided clarity to participants about the requirements and expectations for their engagement in the challenge, while maintaining compliance with requirements stipulated under the America COMPETES Act authority. The terms and conditions (or rules, in this case) were developed with the goals of the challenge in mind: to increase awareness of important NIST scientific data resources and to stimulate innovative uses of NIST data as part of a larger open data commitment. Below are some aspects to NIST's development of the challenge rules, which were posted in a notice in the Federal Register.

• Eligibility: This competition was open to all individuals over the age of 18 that are residents of one of the 50 United States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, the Commonwealth of the Northern Mariana Islands or American Samoa; and to for-profit or nonprofit corporations, institutions or other validly formed legal entities organized or incorporated in, and which maintain a primary place of business in, any of the preceding jurisdictions. An individual, whether participating singly or with a group, was required to be a citizen or permanent resident of the United States.
• Intellectual Property: NIST did not make any claim to ownership of entries, consistent with the challenge goal of stimulating innovative app solutions by individuals outside of NIST.
• Minimum Criteria: The rules provided a list of minimum criteria for consideration for a prize. While some of these were stipulated by the legal authority used (e.g., no use of NIST or Department of Commerce logos), additional criteria were added to support robust, complete apps. For example, the integrity and acknowledgement of NIST data was an important aspect of apps resulting from this challenge. Therefore, one criterion stipulated that the app include a statement that "This product uses data provided by the National Institute of Standards and Technology (NIST) but is not endorsed or certified by NIST," and that the NIST standard reference data numbers be displayed prominently.
• Freely Available Data: NIST required that each submission use at least one of six required datasets posted on the NIST website and on data.gov. The rules allowed for the addition of other freely available datasets that enhanced the usefulness of the app for other users. In this competition, there were no requirements for open source solutions.

Area of Excellence #3: "Pay Winners"

Upon determination of the three winners by the judging panel, the challenge manager immediately informed the winners that they were being considered for a prize. To claim the prize, they were required to complete a winner verification form, which verified their eligibility and confirmed they were qualified to participate in the challenge. Upon receipt of this completed form, NIST initiated payment processing in a timely manner in line with the public announcement of the winners.

Challenge Type

Software

Technology

This app challenge required solvers to use one (or more) of six valuable NIST scientific datasets, helping to promote awareness and use of NIST reference data. The first place winner started a company and is developing his app for wide use by scientists and engineers.

Legal Authority

America COMPETES Act (15 U.S.C. § 3719)

Challenge Website

nistdata.devpost.com