U.S. flag

An official website of the United States government, Department of Justice.

Real-Time Crime Forecasting Challenge Posting

Crime Forecasting Challenge logo

The Real-Time Crime Forecasting Challenge seeks to harness the advances in data science to address the challenges of crime and justice. It encourages data scientists across all scientific disciplines to foster innovation in forecasting methods. The goal is to develop algorithms that advance place-based crime forecasting through the use of data from one police jurisdiction.

On this page find:

  1. Overview
  2. How to Enter
  3. Important Dates
  4. Judges
  5. Judging Criteria
  6. Prizes
  7. Other Rules and Conditions
  8. Prize Disbursement and Challenge Winners
  9. Contact Information
  10. Data for Download

I. Overview

The National Institute of Justice's (NIJ) Real-Time Crime Forecasting Challenge (the Challenge) hopes to provide researchers and the federal government with a better understanding of the potential for crime forecasting in America. This Challenge will offer a comprehensive comparative analysis between current "off-the-shelf" crime forecasting products used by many police departments and more innovative forecasting methods used by other scientific disciplines.

This Challenge is issued pursuant to 28 U.S.C. 530C.

As an interdisciplinary research arm of the U.S. Department of Justice, NIJ invests in scientific research across disciplines to serve the needs of the criminal justice community. NIJ recognizes that rapid advances in data sciences are being used to forecast consumer behavior, detect medical anomalies, and provide informatics about product consumers. These advances have been made by students, professors, scientists, corporations, and individuals across the spectrum of scientific disciplines, including biology, cognitive behavioral research, economics, and statistics. NIJ seeks innovative crime forecasting methods through the participation of data scientists from a wide range of scientific fields. By taking innovative ideas from various branches of science, NIJ hopes to inspire researchers and thinkers to discover ways that their own scientific pursuits might be used to help solve some of the criminal justice community's most vexing problems.

With this Challenge, NIJ aims to:

  1. Encourage "nontraditional" crime forecasting researchers to compete against more "traditional" crime forecasting researchers.
  2. Compare available crime forecasting methods.
  3. Improve place-based crime forecasting.

Accordingly, the Challenge will have three categories of contestants: students; individuals/small businesses; and large businesses. (See Section VI.) NIJ will evaluate all entries for both effectiveness and efficiency. (See Section V.)

This Challenge will be based on the locations listed in calls-for-service (CFS) records provided by the Portland Police Bureau (PPB) for the period of March 1, 2012 through February 28, 2017. (See Section X below for all available CFS data.) NIJ will initially release data for the period of March 1, 2012 through July 31, 2016, and NIJ will then release updated PPB's CFS data over a six-month period. (See Section III for the data release schedule.) During the final week of the six-month data rollout, contestants will submit forecasts of where the largest concentrations of crimes will occur within the PPB jurisdiction. The four crime categories are: all calls-for-service; burglary (residential and commercial); street crime; and motor vehicle theft. (See Table 1 for category definitions.) Contestants may submit forecasts for all or some of these categories. Crime forecasts can be submitted for each crime category for periods of one week, two weeks, one month, two months, and three months.

Contestants should be aware that other entities may make other data available through free or fee-based services (e.g., cloud and data sharing sites) that may or may not also be useful in developing their algorithms. Contestants are permitted, but not required, to use any other data sets or services.

After the PPB CFS data are collected for March 1, 2017, through May 31, 2017, NIJ will compare the Challenge entries to the actual data for each crime category and each designated time period, and will determine the effectiveness and efficiency index values. The most effective and the most efficient entries for each crime category, time period, and contestant type will be declared the Challenge winners. (See Section VI.) Contestants may win for more than one crime category and time period.

The dates for predictions in each category are: 

  • One week: March 1-7
  • Two weeks: March 1-14
  • One month: March 1-31
  • Two months: March 1-April 30
  • Three months: March 1-May 31

NIJ may award a total Challenge prize of up to $1,200,000.

II. How to Enter

Entries must be submitted through the Office of Justice Programs’ Grants Management System (GMS). Registration and entry are free. 

The Challenge submission period is from 12:00 a.m. (ET) February 22, 2017 through 11:59 p.m. (ET) February 28, 2017. 

Contestants must submit entries under the appropriate contestant type by selecting the applicable solicitation within GMS. Contestant types are:

  • Student: Enrolled as a full-time student in high school or as a full-time, degree-seeking student in an undergraduate program (associate or bachelor’s degree). 
  • Small Team/Business: A team comprised of 1-20 individuals, or a small business with less than 21 employees. Teams should enter using a team name and provide a list of all individuals on the team. A small business should enter using the name of its business.
  • Large Business: A business with more than 20 employees. A large business should enter using the name of its business.

Note that Student and Small Team/Business contestants may enter the Large Business category, but Large Businesses may not enter the Student or Small Team/Business categories.

Individuals may participate in only one Challenge entry.

A Challenge entry is defined as: forecasts for one or more of the CFS categories (4) for any of the five time periods (a maximum of 20 total forecasts). Note that a contestant is not required to provide a forecast for every CFS category and time period.

Update to Table 1

Table 1 was updated on February 8, 2017.[1] Changes include:

  • Burglary: Removed code BURG. 
  • Street Crime: Removed codes ASSLT, DIST, SHOOT, TAB. Added codes THRETP and THRETW.
  • Theft of Autos: Removed code VEHST.
Table 1: Crime Category Definitions – CFS Code and Translation
CFS Category Code Translation
Burglary
BURGP BURGLARY – PRIORITY *H
PROWLP PROWLER
Street Crime
ASSLTP ASSAULT –PRIORITY
Note: This code initially was listed erroneously as "ASSLTT"
ASSLTW ASSAULT WITH WEAPON *H
DISTP DISTURBANCE – PRIORITY
DISTW DISTURBANCE – WITH WEAPON *H
GANG GANG RELATED
ROBP ROBBERY – PRIORITY *H
ROBW ROBBERY – WITH WEAPON *H
SHOOTW SHOOTING – WITH WEAPON *H
SHOTS SHOTS FIRED
STABW STABBING WITH WEAPON *H
THRETP THREAT - PRIORITY
THRETW THREAT - WITH WEAPON *H
VICE VICE-DRUGS, LIQUOR, PROSTITUTION, GAMBLING
Theft of Auto
RSTLN ROLLING STOLEN *H
VEHREC VEHICLE RECOVERED
VEHSTP VEHICLE STOLEN – PRIORITY
All CFS This category includes all CFS including those in the above categories.

Exclusions:

  • The following CFS are excluded from the data set: administrative, arson, crisis, death, domestic disturbance/violence, juvenile offenses, kidnapping, k9 explosive sweep, missing person, rape, restraining order, sex offense, stalking, and suicide.
  • The following addresses are also excluded: 737 SE 106th Ave (East Precinct), 449 NE Emerson St (North Precinct), 2801 N Gantenbein Ave (Legacy Emanuel Hospital), 4804 NE Glisan St (Providence Hospital), 1111 SW 2nd Ave (Central Precinct), 10300 SE Main St (Adventist Medical Center), 1014 NW 22nd Ave (Legacy Good Samaritan Medical Center), 41 NE Grand Ave (Detox Center), 3181 SW Sam Jackson Park Rd (OHSU Hospital), 10123 SE Market St (Adventist Medical Center), 12240 NE Glisan (Multnomah County Sheriff’s Office), and 3303 SW Bond Ave (OHSU Hospital).
On February 17, 2017, NIJ updated the list of required files.

On January 26, 2017, NIJ clarified that the variables for hotspot and area must be named "hotspot" and "area."
Table 2: Requirements for Entries
Requirement Description of Requirement
Required files .dbf
.prj
.shp
.shx
Optional files* .sbn 
.sbx
Projection of files NAD_1983_HARN_StatePlane_Oregon_North_FIPS_3601_Feet_Intl
Required variables
  • Unique ID for each cell.
  • A binary variable (1 – hot spot, 0 – not) that must be named "hotspot."
  • Area for each cell measured in square feet to 4 decimal places. This variable must be named "area."
Cell shape Any shape
Individual cell area** 62,500 ft2 – 360,000 ft2
Minimum cell height 125ft (this clarification was added on December 9, 2016)
Total forecasted area 0.25 mi2 – .75 mi2

*If possible, we ask that you include the .sbn and .sbx files but submissions without them will be accepted and scored.

**Cells forming the outer boundary of the study area (Portland Police Districts, see file in Section X.) must be trimmed so that the total area of all cells equals 147.71 square miles (+/-0.02 square miles). The area of each interior cell must be equal to one another.

Submission Format and Naming

The following clarification was added on December 9, 2016.

Forecasts must be submitted as a .zip file containing up to four folders (one for each crime type). Each of those four folders may contain up to five folders inside (one for each time period). Each of those files must contain a shapefile, which includes all of the files for that forecast.

View a sample submission prepared by NIJ (zip, 19.2 MB).

File names and structure must adhere to the following conventions:

Naming convention:

  • Zip file: Use the name of the team, Business, or last name (for student entries).
  • Crime type folders: "ACFS" for all calls-for-service, "Burg" for burglary, "SC" for street crime, "TOA" for theft of auto.
  • Timeframe folders: 1WK, 2WK, 1MO, 2MO, or 3MO
  • Shapefiles: NAME_CRIME_TIME
    Example: TEAMNIJ_BURG_2MO

Zip file structure:

  • Zip file
    • Folder for each crime type.
      • Folder for each timeframe
        • Shapefile

III. Important Dates

Data release dates (see "Data" below for downloads):

  • On September 1, 2016, initial release of PPB CFS data for 03/01/12-07/31/16
  • On September 7, 2016, release of PPB CFS data for 08/01/16-08/31/16
  • On October 7, 2016, release of PPB CFS data for 09/01/16-09/30/16
  • On November 7, 2016, release of PPB CFS data for 10/01/16-10/31/16
  • On December 7, 2016, release of PPB CFS data for 11/01/16-11/30/16
  • On January 7, 2017, release of PPB CFS data for 12/01/16-12/31/16
  • On February 7, 2017 release of PPB CFS data for 01/01/17-01/31/17
  • On February 17, 2017 release of PPB CFS data for 02/01/17-02/14/17
  • On February 24, 2017 release of PPB CFS data for 02/15/17-02/21/17
  • On February 27, 2017 release of PPB CFS data for 02/22/17-02/26/17
  • On February 28, 2017 release of PPB CFS data for 02/27/17 and any CFS data for 02/28/17 that are available at time of last upload (3 p.m. eastern time).
    • Note: No data for 2/28/2017 will be posted. 
  • On March 6, 2017 release of any PPB CFS data for 02/27/17-02/28/17 not yet released
  • On June 7th, 2017 release of PPB CFS data for 03/01/17-05/31/17

Submission and selection dates:

  • Submission period begins: February 22, 2017
  • Submission period ends: February 28, 2017
  • All Winners Announced: June 30, 2017 (anticipated)

IV. Judges

NIJ will score entries according to the judging criteria described in Section V. The Director of NIJ or their designee will make the final award determination. If the Director of NIJ or their designee determines that no entry is deserving of an award, no prizes will be awarded.

V. Judging Criteria

NIJ will use the following criteria to score Challenge entries: [2]

(A) Prediction Accuracy Index (PAI)

  • The PAI will measure the effectiveness of the forecasts with the following equation:
    PAI equals the quotien of n divided by N divided by the quotient of a divided by A
  • Where n equals the number of crimes that occur in the forecasted area, N equals the total number of crimes, a equals the forecasted area, and A equals the area of the entire study area.

(B) Prediction Efficiency Index* (PEI*)

  • The PEI* will measure the efficiency of the forecast with the following equation:
    PEI* equals PAI divided by PAI*
  • Where PEI* equals the maximum obtainable PAI value for the amount of area forecasted, a. As such:
    PEI* equals n divided by n*
  • Where n* equals the maximum obtainable n for the amount of area forecasted, a.

See Example Calculation of Prediction Accuracy Index and Prediction Efficiency Index (pdf , 2 pages).

Macros Use to Determine Eligibility and Scores

This section was added July 27, 2017

In response to questions regarding how submissions were judged, we have posted the following files containing the Visual Basic macros used to determine eligibility and scores for each submission:

  • The Runner Macro was used to do the initial simple setup and call the other two macros.
  • The Join Macro joined each submission to the actual data for each crime type and period.
  • The Calculations Macro obtained the needed data points from the joined shapefiles created above and inserted them into an excel document.

The excel document created by the Calculations Macro was conditionally formatted to indicate if values (e.g., max grid cell size, a, and A) were in compliance with the requirements of the Challenge and to trigger manual review of the shapefiles to examine the cause of the violation.

VI. Prizes

A total of $1,200,000 is available in prizes and will be divided among the three contestant types as follows:

  • Large Business Contestants: 40 prizes of $15,000 each for a total prize of $600,000.
  • Small Team/Business Contestants: 40 prizes of $10,000 each for a total prize of $400,000.
  • Student Contestants: 40 prizes of $5,000 each for a total prize of $200,000.

Table 3 uses the Student category as an example of the prize structure. The Challenge will use the same prize structure for the other contestant categories however, with the amounts, $10,000 and $15,000, respectively, per prize.

Table 3: Challenge Prizes
    1 Week
3/1-3/7
2 Week
3/1-3/14
1 Month
3/1-3/31
2 Month
3/1-4/30
3 Month
3/1-5/31
Burglary Effectiveness $5,000 $5,000 $5,000 $5,000 $5,000
Efficiency $5,000 $5,000 $5,000 $5,000 $5,000
Motor Vehicle Theft Effectiveness $5,000 $5,000 $5,000 $5,000 $5,000
Efficiency $5,000 $5,000 $5,000 $5,000 $5,000
Street Crime Effectiveness $5,000 $5,000 $5,000 $5,000 $5,000
Efficiency $5,000 $5,000 $5,000 $5,000 $5,000
All Calls-for-Service Effectiveness $5,000 $5,000 $5,000 $5,000 $5,000
Efficiency $5,000 $5,000 $5,000 $5,000 $5,000

All prizes will be awarded at the end of the competition. Any contestant who wins multiple prizes will receive one prize payment for the total amount won.

Contestants entering as a team must submit a “Team Roster” that lists the individual team members. The Team Roster must indicate the proportional amount of any prize to be paid to each member. Each team member must sign the Team Roster next to his or her name. For example:

John Q. Public 20 percent (Signature)________.

This allows teams to designate proportional prizes to their members. Team entries that do not submit a Team Roster signed by each member of the team will be disqualified. All prizes are subject to the availability of appropriated funds and will be awarded at the discretion of the NIJ Director.

VII. Other Rules and Conditions

A. Submission Period

The Challenge Submission Period ends on February 28, 2017. Entries submitted after the designated Submission Periods will be disqualified and will not be reviewed.

B. Eligibility

The Challenge is open to: (1) individual residents of the 50 United States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, and American Samoa[3] who are at least 13 years old at the time of entry; (2) teams of eligible individuals; and (3) corporations or other legal entities (e.g., partnerships or nonprofit organizations) that are domiciled in any jurisdiction specified in (1). Entries by contestants under the age of 18 must include the co-signature of the contestant’s parent or legal guardian. Contestants may submit or participate in the submission of only one entry. Employees of NIJ and individuals or entities listed on the Federal Excluded Parties list (available from SAM.gov) are not eligible to participate. Employees of the Federal Government should consult with the Ethics Officer at their place of employment prior to submitting an entry for this Challenge. The Challenge is subject to all applicable federal laws and regulations. Submission of an entry constitutes a contestant’s full and unconditional agreement to all applicable rules and conditions. Eligibility for the prize award(s) is contingent upon fulfilling all requirements set forth herein.

C. General Warranties and Conditions

  • Release of Liability: By entering the Challenge, each contestant agrees to: (a) comply with and be bound by all applicable rules and conditions, and the decisions of NIJ, which are binding and final in all matters relating to this Challenge; (b) release and hold harmless NIJ and any other organizations responsible for sponsoring, fulfilling, administering, advertising or promoting the Challenge, and all of their respective past and present officers, directors, employees, agents and representatives (collectively, the “Released Parties”) from and against any and all claims, expenses, and liability arising out of or relating to the contestant’s entry or participation in the Challenge, and/or the contestant’s acceptance, use, or misuse of the prize or recognition.
    The Released Parties are not responsible for: (a) any incorrect or inaccurate information, whether caused by contestants, printing errors or by any of the equipment or programming associated with, or utilized in, the Challenge; (b) technical failures of any kind, including, but not limited to malfunctions, interruptions, or disconnections in phone lines or network hardware or software; (c) unauthorized human intervention in any part of the entry process or the Challenge; (d) technical or human error which may occur in the administration of the Challenge or the processing of entries; or (e) any injury or damage to persons or property which may be caused, directly or indirectly, in whole or in part, from contestant's participation in the Challenge or receipt or use or misuse of any prize. If for any reason a contestant's entry is confirmed to have been erroneously deleted, lost, or otherwise destroyed or corrupted, contestant's sole remedy is to submit another entry in the Challenge so long as the resubmitted entry is prior to the deadline for submissions.
  • Termination and Disqualification: NIJ reserves the authority to cancel, suspend, and/or modify the Challenge, or any part of it, if any fraud, technical failures, or any other factor beyond NIJ’s reasonable control impairs the integrity or proper functioning of the Challenge, as determined by NIJ in its sole discretion. NIJ reserves the authority to disqualify any contestant it believes to be tampering with the entry process or the operation of the Challenge or to be acting in violation of any applicable rule or condition. Any attempt by any person to undermine the legitimate operation of the Challenge may be a violation of criminal and civil law, and, should such an attempt be made, NIJ reserves the authority to seek damages from any such person to the fullest extent permitted by law. NIJ’s failure to enforce any term of any applicable rule or condition shall not constitute a waiver of that term.
  • Intellectual Property: By entering the Challenge, each contestant warrants that (a) he or she is the author and/or authorized owner of the entry; (b) that the entry is wholly original with the contestant (or is an improved version of an existing solution that the contestant is legally authorized to enter in the Challenge); (c) that the submitted entry does not infringe any copyright, patent, or any other rights of any third party; and (d) that the contestant has the legal authority to assign and transfer to NIJ all necessary rights and interest (past, present, and future) under copyright and other intellectual property law, for all material included in the Challenge proposal that may be held by the contestant and/or the legal holder of those rights. Each contestant agrees to hold the Released Parties harmless for any infringement of copyright, trademark, patent, and/or other real or intellectual property right, which may be caused, directly or indirectly, in whole or in part, from contestant’s participation in the Challenge.
  • Publicity: By entering the Challenge, each contestant consents, as applicable, to NIJ’s use of his/her/its name, likeness, photograph, voice, and/or opinions, and disclosure of his/her/its hometown and State for promotional purposes in any media, worldwide, without further payment or consideration.
  • Privacy: Personal and contact information submitted through https://grants.ojp.usdoj.gov is not collected for commercial or marketing purposes. Information submitted throughout the Challenge will be used only to communicate with contestants regarding entries and/or the Challenge.
  • Compliance With Law: By entering the Challenge, each contestant guarantees that the entry complies with all federal and state laws and regulations.

VIII. Prize Disbursement and Challenge Winners

A. Prize Disbursement and Requirements

Prize winners must comport with all applicable laws and regulations regarding prize receipt and disbursement. For example, NIJ is not responsible for withholding any applicable taxes from the award.

B. Specific Disqualification Rule

If the announced winner(s) of the Challenge prize is found to be ineligible or is disqualified for any reason listed under “Other Rules and Conditions: Eligibility,” NIJ may make the award to the next runner(s) up, as previously determined by the NIJ Director.

C. Rights Retained by Contestants and Challenge Winners

  1. All legal rights in any materials or products submitted in entering the Challenge are retained by the contestant and/or the legal holder of those rights. Entry in the Challenge constitutes
    express authorization for NIJ staff to review and analyze any and all aspects of submitted entries.
  2. Upon acceptance of any Challenge prizes, the winning contestant(s) agrees to allow NIJ to post on NIJ’s website the shapefile and all associated files needed to map the winning shapefile. This will allow others to verify the scores of winning submissions.

IX. Contact Information

Starting February 17 and through the remainder of the Challenge, we ask that all questions be submitted through the discussion section on Challenge.gov. 

For technical questions about the application process, first read through "Section II. How to Enter" and the detailed registration and submission instructions. If you still have questions, contact the OJP Grants Management System Help Desk — open 7 a.m. to 9 p.m. ET — at 1-888-549-9901.

X. Data for Download

Portland Police Districts (zip, 114 KB)

Calls-for-service data:

Notes

[note 1] All codes that include “Cold” in the translation were originally to be included in the corresponding crime type category. However, after discussions with the Portland Police Bureau, NIJ determined that it was not advantageous to include those codes in forecasts of future crimes. Those codes have been re-categorized within the shapefile as “Other.” Applicants may use the cold case locations to aide in their forecasts but “Cold” calls will not be included in the scoring calculations for the individual categories.

The codes THRETP and THRETW were omitted in error and should be included in the “Street Crimes” category.

[note 2] The PAI was originally proposed to the field by Chainey, Tompson, and Uhlig (2008). They sought to define a measure for testing forecasting accuracy. They said it measures "the hit rate against the areas where crimes are predicted to occur with respect to the size of the study area;" the actual equation is given with the Challenge. The criminology field (specifically those working on forecasting and prediction) have principally relied on this measure since its inception.

The PEI* (and PEI) were proposed by Hunt (2016) as a complimentary measure to the PAI. He sought to define a measure for testing forecasting efficiency. It is meant to measure how well a forecast does compared to how well it could have done (post hoc). It was only recently introduced, however it is the only known plausible alternative at this time to compliment the PAI. The equation is provided in the Challenge for how it is measured.

References:

  • Chainey, S., Thompson, L., & Uhligh, S. (2008). The Utility of Hotspot Mapping for Predicting Spatial Patterns of Crime. Security(21), 4-28.
  • Hunt, J. (2016). Do Crime Hot Spots Move? Exploring the Effects of the Modifiable Areal Unit Problem and Modifiable Temporal Unit Problem on Crime Hot Spot Stability. Archived with ProQuest Dissertations & Theses.

[note 3] A resident is defined as a citizen or resident non-citizen of the United States. A resident non-citizen, for purposes of this Challenge, is someone who meets either the green card test or the substantial presence test as defined by the Internal Revenue Service (IRS)

Date Created: August 1, 2017