Real-Time Crime Forecasting Challenge Posting

The Real-Time Crime Forecasting Challenge seeks to harness the advances in data science to address the challenges of crime and justice. It encourages data scientists across all scientific disciplines to foster innovation in forecasting methods. The goal is to develop algorithms that advance place-based crime forecasting through the use of data from one police jurisdiction.

On this page find:

1. Overview
2. How to Enter
3. Important Dates
4. Judges
5. Judging Criteria
6. Prizes
7. Other Rules and Conditions
8. Prize Disbursement and Challenge Winners
9. Contact Information
10. Data for Download

I. Overview

The National Institute of Justice's (NIJ) Real-Time Crime Forecasting Challenge (the Challenge) hopes to provide researchers and the federal government with a better understanding of the potential for crime forecasting in America. This Challenge will offer a comprehensive comparative analysis between current "off-the-shelf" crime forecasting products used by many police departments and more innovative forecasting methods used by other scientific disciplines.

This Challenge is issued pursuant to 28 U.S.C. 530C.

As an interdisciplinary research arm of the U.S. Department of Justice, NIJ invests in scientific research across disciplines to serve the needs of the criminal justice community. NIJ recognizes that rapid advances in data sciences are being used to forecast consumer behavior, detect medical anomalies, and provide informatics about product consumers. These advances have been made by students, professors, scientists, corporations, and individuals across the spectrum of scientific disciplines, including biology, cognitive behavioral research, economics, and statistics. NIJ seeks innovative crime forecasting methods through the participation of data scientists from a wide range of scientific fields. By taking innovative ideas from various branches of science, NIJ hopes to inspire researchers and thinkers to discover ways that their own scientific pursuits might be used to help solve some of the criminal justice community's most vexing problems.

With this Challenge, NIJ aims to:

1. Encourage "nontraditional" crime forecasting researchers to compete against more "traditional" crime forecasting researchers.
2. Compare available crime forecasting methods.
3. Improve place-based crime forecasting.

Accordingly, the Challenge will have three categories of contestants: students; individuals/small businesses; and large businesses. (See Section VI.) NIJ will evaluate all entries for both effectiveness and efficiency. (See Section V.)

This Challenge will be based on the locations listed in calls-for-service (CFS) records provided by the Portland Police Bureau (PPB) for the period of March 1, 2012 through February 28, 2017. (See Section X below for all available CFS data.) NIJ will initially release data for the period of March 1, 2012 through July 31, 2016, and NIJ will then release updated PPB's CFS data over a six-month period. (See Section III for the data release schedule.) During the final week of the six-month data rollout, contestants will submit forecasts of where the largest concentrations of crimes will occur within the PPB jurisdiction. The four crime categories are: all calls-for-service; burglary (residential and commercial); street crime; and motor vehicle theft. (See Table 1 for category definitions.) Contestants may submit forecasts for all or some of these categories. Crime forecasts can be submitted for each crime category for periods of one week, two weeks, one month, two months, and three months.

Contestants should be aware that other entities may make other data available through free or fee-based services (e.g., cloud and data sharing sites) that may or may not also be useful in developing their algorithms. Contestants are permitted, but not required, to use any other data sets or services.

After the PPB CFS data are collected for March 1, 2017, through May 31, 2017, NIJ will compare the Challenge entries to the actual data for each crime category and each designated time period, and will determine the effectiveness and efficiency index values. The most effective and the most efficient entries for each crime category, time period, and contestant type will be declared the Challenge winners. (See Section VI.) Contestants may win for more than one crime category and time period.

The dates for predictions in each category are: 

• One week: March 1-7
• Two weeks: March 1-14
• One month: March 1-31
• Two months: March 1-April 30
• Three months: March 1-May 31

NIJ may award a total Challenge prize of up to $1,200,000.

II. How to Enter

Entries must be submitted through the Office of Justice Programs’ Grants Management System (GMS). Registration and entry are free. 

The Challenge submission period is from 12:00 a.m. (ET) February 22, 2017 through 11:59 p.m. (ET) February 28, 2017. 

Contestants must submit entries under the appropriate contestant type by selecting the applicable solicitation within GMS. Contestant types are:

• Student: Enrolled as a full-time student in high school or as a full-time, degree-seeking student in an undergraduate program (associate or bachelor’s degree). 
• Small Team/Business: A team comprised of 1-20 individuals, or a small business with less than 21 employees. Teams should enter using a team name and provide a list of all individuals on the team. A small business should enter using the name of its business.
• Large Business: A business with more than 20 employees. A large business should enter using the name of its business.

Note that Student and Small Team/Business contestants may enter the Large Business category, but Large Businesses may not enter the Student or Small Team/Business categories.

Individuals may participate in only one Challenge entry.

A Challenge entry is defined as: forecasts for one or more of the CFS categories (4) for any of the five time periods (a maximum of 20 total forecasts). Note that a contestant is not required to provide a forecast for every CFS category and time period.

Update to Table 1

Table 1 was updated on February 8, 2017.[1] Changes include:

• Burglary: Removed code BURG. 
• Street Crime: Removed codes ASSLT, DIST, SHOOT, TAB. Added codes THRETP and THRETW.
• Theft of Autos: Removed code VEHST.

Submission Format and Naming

The following clarification was added on December 9, 2016.

Forecasts must be submitted as a .zip file containing up to four folders (one for each crime type). Each of those four folders may contain up to five folders inside (one for each time period). Each of those files must contain a shapefile, which includes all of the files for that forecast.

View a sample submission prepared by NIJ (zip, 19.2 MB).

File names and structure must adhere to the following conventions:

Naming convention:

• Zip file: Use the name of the team, Business, or last name (for student entries).
• Crime type folders: "ACFS" for all calls-for-service, "Burg" for burglary, "SC" for street crime, "TOA" for theft of auto.
• Timeframe folders: 1WK, 2WK, 1MO, 2MO, or 3MO
• Shapefiles: NAME_CRIME_TIME
  Example: TEAMNIJ_BURG_2MO

Zip file structure:

• Zip file
  • Folder for each crime type.
    • Folder for each timeframe
      • Shapefile

III. Important Dates

Data release dates (see "Data" below for downloads):

• On September 1, 2016, initial release of PPB CFS data for 03/01/12-07/31/16
• On September 7, 2016, release of PPB CFS data for 08/01/16-08/31/16
• On October 7, 2016, release of PPB CFS data for 09/01/16-09/30/16
• On November 7, 2016, release of PPB CFS data for 10/01/16-10/31/16
• On December 7, 2016, release of PPB CFS data for 11/01/16-11/30/16
• On January 7, 2017, release of PPB CFS data for 12/01/16-12/31/16
• On February 7, 2017 release of PPB CFS data for 01/01/17-01/31/17
• On February 17, 2017 release of PPB CFS data for 02/01/17-02/14/17
• On February 24, 2017 release of PPB CFS data for 02/15/17-02/21/17
• On February 27, 2017 release of PPB CFS data for 02/22/17-02/26/17
• On February 28, 2017 release of PPB CFS data for 02/27/17 and any CFS data for 02/28/17 that are available at time of last upload (3 p.m. eastern time).
  • Note: No data for 2/28/2017 will be posted. 
• On March 6, 2017 release of any PPB CFS data for 02/27/17-02/28/17 not yet released
• On June 7th, 2017 release of PPB CFS data for 03/01/17-05/31/17

Submission and selection dates:

• Submission period begins: February 22, 2017
• Submission period ends: February 28, 2017
• All Winners Announced: June 30, 2017 (anticipated)

IV. Judges

NIJ will score entries according to the judging criteria described in Section V. The Director of NIJ or their designee will make the final award determination. If the Director of NIJ or their designee determines that no entry is deserving of an award, no prizes will be awarded.

V. Judging Criteria

NIJ will use the following criteria to score Challenge entries: [2]

(A) Prediction Accuracy Index (PAI)

• The PAI will measure the effectiveness of the forecasts with the following equation:
  
• Where n equals the number of crimes that occur in the forecasted area, N equals the total number of crimes, a equals the forecasted area, and A equals the area of the entire study area.

(B) Prediction Efficiency Index* (PEI*)

• The PEI* will measure the efficiency of the forecast with the following equation:
  
• Where PEI* equals the maximum obtainable PAI value for the amount of area forecasted, a. As such:
  
• Where n* equals the maximum obtainable n for the amount of area forecasted, a.

See Example Calculation of Prediction Accuracy Index and Prediction Efficiency Index (pdf , 2 pages).

Macros Use to Determine Eligibility and Scores

This section was added July 27, 2017

In response to questions regarding how submissions were judged, we have posted the following files containing the Visual Basic macros used to determine eligibility and scores for each submission:

• The Runner Macro was used to do the initial simple setup and call the other two macros.
• The Join Macro joined each submission to the actual data for each crime type and period.
• The Calculations Macro obtained the needed data points from the joined shapefiles created above and inserted them into an excel document.

The excel document created by the Calculations Macro was conditionally formatted to indicate if values (e.g., max grid cell size, a, and A) were in compliance with the requirements of the Challenge and to trigger manual review of the shapefiles to examine the cause of the violation.

VI. Prizes

A total of $1,200,000 is available in prizes and will be divided among the three contestant types as follows:

• Large Business Contestants: 40 prizes of $15,000 each for a total prize of $600,000.
• Small Team/Business Contestants: 40 prizes of $10,000 each for a total prize of $400,000.
• Student Contestants: 40 prizes of $5,000 each for a total prize of $200,000.

Table 3 uses the Student category as an example of the prize structure. The Challenge will use the same prize structure for the other contestant categories however, with the amounts, $10,000 and $15,000, respectively, per prize.

VII. Other Rules and Conditions

A. Submission Period

The Challenge Submission Period ends on February 28, 2017. Entries submitted after the designated Submission Periods will be disqualified and will not be reviewed.

B. Eligibility

The Challenge is open to: (1) individual residents of the 50 United States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, and American Samoa[3] who are at least 13 years old at the time of entry; (2) teams of eligible individuals; and (3) corporations or other legal entities (e.g., partnerships or nonprofit organizations) that are domiciled in any jurisdiction specified in (1). Entries by contestants under the age of 18 must include the co-signature of the contestant’s parent or legal guardian. Contestants may submit or participate in the submission of only one entry. Employees of NIJ and individuals or entities listed on the Federal Excluded Parties list (available from SAM.gov) are not eligible to participate. Employees of the Federal Government should consult with the Ethics Officer at their place of employment prior to submitting an entry for this Challenge. The Challenge is subject to all applicable federal laws and regulations. Submission of an entry constitutes a contestant’s full and unconditional agreement to all applicable rules and conditions. Eligibility for the prize award(s) is contingent upon fulfilling all requirements set forth herein.

C. General Warranties and Conditions

• Release of Liability: By entering the Challenge, each contestant agrees to: (a) comply with and be bound by all applicable rules and conditions, and the decisions of NIJ, which are binding and final in all matters relating to this Challenge; (b) release and hold harmless NIJ and any other organizations responsible for sponsoring, fulfilling, administering, advertising or promoting the Challenge, and all of their respective past and present officers, directors, employees, agents and representatives (collectively, the “Released Parties”) from and against any and all claims, expenses, and liability arising out of or relating to the contestant’s entry or participation in the Challenge, and/or the contestant’s acceptance, use, or misuse of the prize or recognition.
  The Released Parties are not responsible for: (a) any incorrect or inaccurate information, whether caused by contestants, printing errors or by any of the equipment or programming associated with, or utilized in, the Challenge; (b) technical failures of any kind, including, but not limited to malfunctions, interruptions, or disconnections in phone lines or network hardware or software; (c) unauthorized human intervention in any part of the entry process or the Challenge; (d) technical or human error which may occur in the administration of the Challenge or the processing of entries; or (e) any injury or damage to persons or property which may be caused, directly or indirectly, in whole or in part, from contestant's participation in the Challenge or receipt or use or misuse of any prize. If for any reason a contestant's entry is confirmed to have been erroneously deleted, lost, or otherwise destroyed or corrupted, contestant's sole remedy is to submit another entry in the Challenge so long as the resubmitted entry is prior to the deadline for submissions.
• Termination and Disqualification: NIJ reserves the authority to cancel, suspend, and/or modify the Challenge, or any part of it, if any fraud, technical failures, or any other factor beyond NIJ’s reasonable control impairs the integrity or proper functioning of the Challenge, as determined by NIJ in its sole discretion. NIJ reserves the authority to disqualify any contestant it believes to be tampering with the entry process or the operation of the Challenge or to be acting in violation of any applicable rule or condition. Any attempt by any person to undermine the legitimate operation of the Challenge may be a violation of criminal and civil law, and, should such an attempt be made, NIJ reserves the authority to seek damages from any such person to the fullest extent permitted by law. NIJ’s failure to enforce any term of any applicable rule or condition shall not constitute a waiver of that term.
• Intellectual Property: By entering the Challenge, each contestant warrants that (a) he or she is the author and/or authorized owner of the entry; (b) that the entry is wholly original with the contestant (or is an improved version of an existing solution that the contestant is legally authorized to enter in the Challenge); (c) that the submitted entry does not infringe any copyright, patent, or any other rights of any third party; and (d) that the contestant has the legal authority to assign and transfer to NIJ all necessary rights and interest (past, present, and future) under copyright and other intellectual property law, for all material included in the Challenge proposal that may be held by the contestant and/or the legal holder of those rights. Each contestant agrees to hold the Released Parties harmless for any infringement of copyright, trademark, patent, and/or other real or intellectual property right, which may be caused, directly or indirectly, in whole or in part, from contestant’s participation in the Challenge.
• Publicity: By entering the Challenge, each contestant consents, as applicable, to NIJ’s use of his/her/its name, likeness, photograph, voice, and/or opinions, and disclosure of his/her/its hometown and State for promotional purposes in any media, worldwide, without further payment or consideration.
• Privacy: Personal and contact information submitted through https://grants.ojp.usdoj.gov is not collected for commercial or marketing purposes. Information submitted throughout the Challenge will be used only to communicate with contestants regarding entries and/or the Challenge.
• Compliance With Law: By entering the Challenge, each contestant guarantees that the entry complies with all federal and state laws and regulations.

VIII. Prize Disbursement and Challenge Winners

A. Prize Disbursement and Requirements

Prize winners must comport with all applicable laws and regulations regarding prize receipt and disbursement. For example, NIJ is not responsible for withholding any applicable taxes from the award.

B. Specific Disqualification Rule

If the announced winner(s) of the Challenge prize is found to be ineligible or is disqualified for any reason listed under “Other Rules and Conditions: Eligibility,” NIJ may make the award to the next runner(s) up, as previously determined by the NIJ Director.

C. Rights Retained by Contestants and Challenge Winners

1. All legal rights in any materials or products submitted in entering the Challenge are retained by the contestant and/or the legal holder of those rights. Entry in the Challenge constitutes
   express authorization for NIJ staff to review and analyze any and all aspects of submitted entries.
2. Upon acceptance of any Challenge prizes, the winning contestant(s) agrees to allow NIJ to post on NIJ’s website the shapefile and all associated files needed to map the winning shapefile. This will allow others to verify the scores of winning submissions.

IX. Contact Information

Starting February 17 and through the remainder of the Challenge, we ask that all questions be submitted through the discussion section on Challenge.gov. 

For technical questions about the application process, first read through "Section II. How to Enter" and the detailed registration and submission instructions. If you still have questions, contact the OJP Grants Management System Help Desk — open 7 a.m. to 9 p.m. ET — at 1-888-549-9901.

X. Data for Download

Portland Police Districts (zip, 114 KB)

Calls-for-service data:

• March-May, 2017 (zip, 7.83 MB) — posted June 9, 2017 
• February 28, 2017 (zip, 57.5 KB) — posted February 28, 2017
• February 27, 2017 (zip, 56 KB) — posted February 28, 2017
• February 22-26, 2017 (zip, 231 KB) — posted February 27, 2017
• February 15-21, 2017 (zip, 309 KB) — posted February 23, 2017
• February 1-14, 2017 (zip, 589 KB) — posted February 17, 2017
• January 2017 (zip, 1.27 MB) — posted February 7, 2017
• December 2016 (zip, 1.18 MB) — posted January 7, 2016
• November 2016 (zip, 1.24 MB) — posted December 7, 2016
• October 2016 (zip, 1.3 MB) — posted November 7, 2016
• September 2016 (zip, 1.36 MB) — posted October 7, 2016
• August 2016 (zip, 1.44 MB) — posted September 7, 2016
• 2016, January through July (zip, 8.17 MB) — posted September 1, 2016
• 2015, full year (zip, 13.8 MB) — posted September 1, 2016
• 2014, full year (zip, 12.6 MB) — posted September 1, 2016
• 2013, full year (zip, 11.7 MB) — posted September 1, 2016
• 2012, March through December (zip, 9.94 MB) — posted September 1, 2016