Real-Time Crime Forecasting Challenge Webinar

Speaking in this video:

• Howard Spivak, M.D., Principle Deputy Director
• Joel Hunt, Senior Computer Scientist
• Mary Jo Giovacchini, National Criminal Justice Reference Service

MARY JO GIOVACCHINI: Good afternoon, everyone, and welcome to today's webinar, Crime Forecasting Challenge, hosted by the National Institute of Justice. At this time, I would like to introduce your presenters today, Dr. Howard Spivak, Principal Deputy Director at the National Institute of Justice, and Joel Hunt, Senior Computer Scientist at the National Institute of Justice.

HOWARD SPIVAK: Good afternoon, everybody. I welcome you to this webinar. It's relating to a project we're very excited about. I plan on talking with you very briefly and then passing things on to Joel to discuss the meat of this initiative.

But I want to start just by giving a quick overview of the National Institute of Justice, since I suspect at least some of you, if not many of you, are unfamiliar with us and what we do. The National Institute of Justice, more affectionately known as NIJ, is the research, development, and evaluation agency of the US Department of Justice. So, we actually do research to advance, practice, and policy in criminal justice within the Justice Department. We're dedicated to improving the knowledge and understanding of crime and justice issues through science. And we invest in scientific research across a spectrum of disciplines to serve the needs of the criminal justice community, both those who practice in criminal justice as well as those who are affected by the criminal justice system itself. So, it's within this context that we have initiated this challenge around an extremely important area related to policing and the effectiveness of the criminal justice system in preventing or reducing crime. So, on that note, I'd like to pass things on to Joel Hunt, who will be going into the details of the challenge and then be answering most of your questions. Thank you.

JOEL HUNT: Thank you, Howard. So, I'm going to begin with a little bit of a background on data science research at the National Institute of Justice. Lately, there's been a major focus on the analysis and visualization of crime data. It's been much more recent role, though. This has origins back in the '80s for us. The major role in the implementation of computer-based crime mapping analysis in American Policing has been a longstanding area of research for us. We've been looking at geospatial strategies and predictive algorithms, much more specifically since--in the last probably six, seven years. The timeline of our data science activity, like I said, it began a lot in the '80s when we started working with the University of Illinois at Chicago to explore mapping crime in the context of community policing, looking at things like hotspot policing in the '80s. It's a -- this was done some -- the Minneapolis research by Sherman, and things like that. Let's see. Sorry.

In the 1990s, we initiated the Drug Market Analysis Program. This is -- was what -- was one of our efforts at looking at forecasting drug usage patterns in the United States. This led to things like Drug Use Forecasting Program and later on the Arrestee Drug Use--Drug Abuse Monitoring Program.

In 1997, we funded the development of CrimeStat, which is a piece of free software. We're now on version 4.2 I believe. That's freely available on NIJ's website [http://www.nij.gov/topics/technology/maps/pages/crimestat.aspx]. This is software that can do advanced spatial and temporal analysis of any type of spatial data. It's titled CrimeStat but you can use any type of data in it.

In 1997, we also established the Crime Mapping Research Center, later known as the Mapping and Analysis for Public Safety Program. Through this program, we spent a lot of time creating and conducting training on mapping and analysis of crime, specifically for crime analysts and police departments.

In 2009, we began the exploration of the potential of crime prediction and forecasting. This is when we began some of the work in what became predictive policing. It started with a meeting (pdf, 20 pages), where Chief Bratton and many other experts in the field met with NIJ. [ 20 pages] And it's kind of when the term predictive policing started to be coined and used. Out of that, we funded research such as the crime for--Predictive Policing at Shreveport, Louisiana, Chicago.

We've done risk terrain modeling work in six or seven different cities now. We've been doing near repeat phenomenon work with the group from Temple. So, we've really been doing a lot.

And a lot of this has now led us to--starting to think about what other sciences have forecasting abilities, and how can we start to leverage some of the forecasting abilities of these other sciences to increase the effectiveness and efficiency of crime forecasting. And that's directly what led to this challenge.

Our goals here, obviously, to advance the place-based crime forecasting abilities that--both the effectiveness and the efficiency of these algorithms or tools.

We're looking at engaging scientists from--not traditionally engaged--that have not been traditionally engaged by a lot of our research. Typically, we work a lot with criminal justice faculty from sociology faculty, things like that. What we're really hoping here is to keep them engaged but also start to bring in economists, and biostatisticians, and medical anomaly detection, things like that. Things that have had a lot longer standing in history and development of their algorithms. And because of that, we're thinking that if we're able to leverage those advancements that we will be able to jumpstart crime forecasting to a much further place than if we were to just stick with our traditional criminal justice researchers. We're really hoping to encourage this to be as widespread of a challenge as possible, so that we can get an independent, comprehensive, comparative analysis of all types of algorithm, techniques, and/or products.

So, the study--the challenge itself is going to be based on calls-for-service out of Portland Police Bureau in Oregon. They are providing calls-for-service data at the address level for Mach 2012 through February 2017 [/funding/Pages/fy16-crime-forecasting-challenge.aspx#data] to train your algorithms, to start creating models, things like that. We've already done our first two data releases, which will put you through March 2012 through August 26--yeah, August 2016. We're about ready to release September tomorrow. And then next month, around this time, we will do that. I'll get more to the actual data release towards the end of it.

What we're asking for is for contestants to forecast--sorry. So, what will happen is once the last week of February 2017 occurs, we will open up our Grants Management System for contestants to actually enter the competition.

And what the competition is, is to ask contestants to forecast what will happen the first week of March; the first two weeks of March; all of March; March and April; March, April, and May. And then we will wait for that timeframe to elapse so that we can gather the real data and actually evaluate the forecasts that are submitted to us against what really happened. At the bottom of this slide, people will see that there's a link to the webpage. That is where the data can be found along with all the rules and stipulations for the challenge.

We should note that contestants may use other data sets or services that they believe will help them in this process. Be creative. Use whatever you like. We will not ask for the intellectual property associated with this. We're just merely asking for the actual maps showing the forecasted areas of where the contestant thinks the hotspot of crime will be.

We've only excluded a few types of addresses and calls-for-services. The addresses are typically where police stations are because that's not where the crimes actually occurred. The calls-for-services that have been removed are typically because that--there's a victim information associated with those types of calls-for-service. Let's see here.

All right. So, the four crime areas that we're looking at, forecasts we conducted in, are burglary, which is residential and commercial, and the reason we have them combined here is that Portland is a primarily a mixed land usage area. So, they don't typically desegregate residential from commercial until much further along in the process of the detectives actually doing--looking at the crime. So, in order for efficiency and us being able to release the data in a timely manner, we left it as burglary. The other crimes we're looking at are street crime, motor vehicle theft, and all calls-for-service. There is a list on the actual challenge page that says exactly what calls-for-service are associated with each of these broad categories. And again, we're looking at forecast being done for any or all of the five time periods, and again, that's the first week of March; first two weeks of March; all of March; March and April; March, April, and May.

So, this means that there could be a total of 20 forecasts. It's five for each of the crime categories, if you want to enter all of them.

The actual--what you'll actually be submitting will be a .zip folder containing the shapefiles that meet all of the requirements in table two of the challenge. There is a screenshot of that table here, but those are the actual requirements. I mean, you must make sure that you have all of the files necessary for us to actually map it out, that is in the right projection, that you include a binary variable that says which cells you're forecasting are going to be the cells you think will have the highest crime counts for those different time periods. It should be noted that you can use different cells and cell sizes between the predictions that--and what I mean by that is you can use a square grid that's a certain size for burglary two-week, that doesn't mean it has to be the same cell size or shape for burglary one-month. You can change between the forecasts.

Again, all entries must be submitted to the Office of Justice Program Grants Management System. On the challenge page, there's a document a link that will help walk you through how you get entered into the system, so it's easier, come that last week of February, to actually submit. Registration and entry are completely free. And again, the link is at the bottom for the webpage which has the instructions on how to do this.

We'll be using two different criteria to judge the submissions. We're going to use the Prediction Accuracy Index as our measure of effectiveness. This was initially proposed to the field by Chaney and his colleagues in roughly 2008, 2009. And the concept behind this is that it--it's essentially, what percentage of the crimes did you forecast divided by what percentage of the land area did you forecast.

The measure for efficiency is the Predictive Efficiency Index*. The PEI* is actually a measure of what is the PAI you obtained divided by what is the maximum PAI you could have obtained for that amount of area had you done an ad hoc forecast. I.e. what--how well could you have done with perfect knowledge of the data.

There will be up to a hundred and twenty prizes.

Oh, sorry. I forgot one thing. At this point, you guys will probably be seeing that a little poll question is going to be popping up probably in the next few seconds. It's going to be asking what category of the challenge you're interested in potentially applying under.

And so what we have is a hundred and twenty prizes that are going to be awarded, forty for each of three categories that you can apply under.

The first one is the Student category. It's $5,000 per prize, and it's considered and enrolled full-time student in either high school or full-time degree seeking student in an undergraduate program. There's the Small Teams/Small Business, which is $10,000 per award, which is teams comprised of one to twenty individuals or a small business with less than twenty-one employees. So, if you're an individual that's not a student, you can still apply. You just have to apply under a small team. Or if you want, you can even go up to the next higher category, which is the Large Businesses category which is $15,000 per award, which is any business with more than 20 employees or anyone who would like to apply into this higher category at--for the chance of the greater prize money. If you do apply under large business, that means you are excluded from receiving the prizes at the lower categories. All prizes will be awarded at the end of the competition, where I figure--we should have the results, we're thinking by probably late June. At which point, contestants will be notified, and any contestants winning multiple prizes will receive one single payment for the total amount. Any contestants that win just one prize will just get also just one single award. This is kind of a breakdown of how the prize structure works. This is showing the students. It would be the same for the Small Teams/Small Business or Large Business, just change the amount in each box accordingly from 5,000 to 10,000, 15,000. So, you can see that this is how the 40 prizes are structured for each category.

Challenge Prizes

		1 Week 3/1-3/7	2 Week 3/1-3/14	1 Month 3/1-3/31	2 Month 3/1-4/30	3 Month 3/1-5/21
Burglary	Effectiveness	$5,000	$5,000	$5,000	$5,000	$5,000
	Efficiency	$5,000	$5,000	$5,000	$5,000	$5,000
Motor Vehicle Theft	Effectiveness	$5,000	$5,000	$5,000	$5,000	$5,000
	Efficiency	$5,000	$5,000	$5,000	$5,000	$5,000
Street Crime	Effectiveness	$5,000	$5,000	$5,000	$5,000	$5,000
	Efficiency	$5,000	$5,000	$5,000	$5,000	$5,000
All Calls-for-Service	Effectiveness	$5,000	$5,000	$5,000	$5,000	$5,000
	Efficiency	$5,000	$5,000	$5,000	$5,000	$5,000

MARY JO GIOVACCHINI: You have a little less than a minute left to do the polling. If you're interested, go ahead and submit your answers. At this time, the poll will close in about 40 seconds.

JOEL HUNT: So eligibility. This challenge is open to residents of the United States and its territories who are 13 years old at the time of entry. Entries by contestants who are under the age of 18, it should say must include the co-signature of the contestant's parents or legal guardian. There are very few exclusions to who may not participate. It's employees of NIJ and individuals or entities listed on the Federal Excluded Parties list are ineligible to participate. Employees of other Federal Agencies, if you are considering applying, you should consult with your Ethics Officer concerning your eligibility.

Again, these were the key date on some we're referencing earlier. As I said, the first two key dates have already happened, which was the two initial data releases for the calls-for-service. Tomorrow will be the third day to release, which should be all of the August calls-for-service data coming--being available--or sorry, all of September's data being--coming available. Most of these are just release dates for the data, but other important--want to note is February 22, 2017 is when the submission period begins for when GMS will actually be opened. And you're able to start uploading your forecast in the .zip folder. The other important date is then February 28th by 11:59 P.M. Eastern Time, that submission period ends. So any submissions that have not been entered at that point, will not be considered nor they even able to be uploaded. On--and then again, we're hoping to have our winners announced by June 30th, which is probably the one that most people are most excited about and concerned about.

I'm going to now start going over some of the Q & As that have been already typed in, and any use that--any of that come in during this time.

MARY JO GIOVACCHINI: Yes, go ahead and continue to submit your questions at this time. And we do have a few already. So please be patient with us, we are going to work through them beginning with the first one that was received, and we will hopefully get to you. If we do run out of time, which I don't anticipate at this point, and we are unable to answer your question, we will somehow get that answer either posted to the challenge page or to the NIJ FAQ page related to the challenge.

So our first question today is, do cells need to be the same shape, size within a forecast?

JOEL HUNT: So they must be the same size within a forecast. We are--currently, the language in the solicitation indicates vaguely that they should be the same shape. We are going to reconsider this. However, at this point, the determination at this point is that they should be the same shape and size. Again, if we do change that, it will be posted and updated to the Q & A, so if you do register for the updates, you would be notified that if we do make that change.

MARY JO GIOVACCHINI: A similar question, can a cell vary in size within an individual prediction?

JOEL HUNT: They cannot change--they've--the only cells that will not be the same exact total area are those that need to be trimmed due to jurisdictional boundaries or lake boundaries. I believe there's a large lake in--or is a park in northeast, but otherwise, all those interior cells should be the same shape and size at this point.

MARY JO GIOVACCHINI: Would it be allowed to publish findings of data after the challenge in an academic venue?

JOEL HUNT: So this is an interesting one. Currently, our agreement with Portland Police Bureau is that this data is made available for the purposes of this challenge. We are reaching back out to Portland Police Bureau to make--see if they will be okay with people using this data for alternative means. Whether it's for educational purposes, training purposes, things like that. We will update the website with their response or--and I'll leave it at that for now. We'll wait for their response.

MARY JO GIOVACCHINI: And our next question. Does a university qualify as a small team or large business?

JOEL HUNT: It depends on how people are--how the team is formed. If--you may have a bunch of individuals from the same university or even multiple universities form a small team where they are using their own free time to work. If they are working and want to apply under the university's name and DUNS number and all of that, then their university is over 20 people and they would be considered a large business.

MARY JO GIOVACCHINI: Question about PAI and related to Table Two, do we need to--do we need to design our own cells to forecast over? As long as they meet the minimum/maximum area amount, they could be any shape no matter how strange?

JOEL HUNT: Well, the big thing here to remember is that all the cells need to be the same shape and size and they must cover the entire jurisdiction of Portland. So there are requirements in Table Two that say that the total area of all the cells must equal--I believe it's somewhere around a hundred and forty-seven square miles. It's in Table Two the exact number. And that the total area of the forecasted cells, the ones that are being indicated to most likely have the highest concentration of crime also need to be between a quarter and three quarters of a square mile. So--and the odds of using completely odd-shaped cells that are going to be the same area and can perfectly cover the police jurisdiction is unlikely. However, if there is a shape that works, you are allowed to use it.

MARY JO GIOVACCHINI: I am pursuing a master's degree. Can a graduate student enter in the small team category?

JOEL HUNT: A graduate student can enter in either the small team, small--or the large business if they want. You can always move up a category. So if you want, as a graduate student, be a team of one, you can be a small team of one.

MARY JO GIOVACCHINI: Similar question. What is the category for a group of graduate students from a university in the US?

JOEL HUNT: So that depends on whether or not you want to use your university's DUNS number and information like that. In which case, you would probably fall under a large business because you're using the university's category or you guys can form a team of individuals and as a small team, if you do that, then all participants on your team must be US residents and meet all the other requirements that are outlined in the legal part of the challenge.

MARY JO GIOVACCHINI: In the formula of PEI*, how is n*--how will n be decided?

JOEL HUNT: So small “n: is the number of crimes that are in the cells that you actually forecasted for that time period. Large “N” is the total number of crimes during that time period for that crime type.

MARY JO GIOVACCHINI: Our academic group that is current--that currently is working on a similar project, a group of five people, we would like to apply, but we would like to know how as a university affiliated member under the--which category we apply, small team or large business?

JOEL HUNT: Yeah, this goes back to the same type of answer. It depends on whether or not you want to represent your university or you're forming your own small team and doing it on your own time. And the thing to remember with small teams is you must--when you enter and apply, provide a list of who all the individuals are on the team along with what percentage of any potential prize money will go to each of those individuals.

MARY JO GIOVACCHINI: Can one person submit both to a small team and a large business category? If not, how can we choose which to submit to if we are eligible for multiple categories?

JOEL HUNT: You may only submit into one category. And it's whichever one you would like to submit to. I mean, obviously, as you go into some of the larger businesses ones, there will be different levels of competition, but there are also different levels of prize money. So it ends up being which one you--if you are eligible for multiple ones, it's which one you would like to apply to.

MARY JO GIOVACCHINI: Does solicitation indicate any shape, not that they need to be the same shape?

JOEL HUNT: And currently, we're looking at clarifying that language at the current moment because it appears as though the word same had at some point became omitted from the solicitation. However, we are going to clarify that language. But it--at this point, it should be the same shape.

MARY JO GIOVACCHINI: Can an international graduate student participate or students?

JOEL HUNT: So--well, graduate students are never going to be categorized as students because the student category is for high school and undergraduate students. When working with international students, it may fall down--it may depend on what type of visa you are here on and whether or not you're considered a resident. So if you have specific one--a specific scenario, I would probably recommend emailing the NCJRS, email address that's provided with your specific situation because there's far too many to just give a yes/no answer.

Following is an updated answer to queries about residency and eligibility.

Q: For purposes of this Challenge, what is the definition of a “resident” of the 50 United States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, the Northern Mariana Islands, and American Samoa?

A: A resident is defined as a citizen or resident alien of the United States. A resident alien, for purposes of this Challenge, is someone who meets either the green card test or the substantial presence test as defined by the Internal Revenue Service (IRS)

MARY JO GIOVACCHINI: And actually, the next slide does provide the NCJRS email address as well as other contact information. So we'll leave this slide up for the remainder of the Q & A’s and it will also be a part of the posted transcript.

Do you guy--do you guys can't--do you guys provide one output file expected as example?

JOEL HUNT: We could--so if all you're asking for is what a potential output file would look like or what a single submission file would look like, we could do that. However, we're trying not to bias it by trying to use a specific cell size or shape. A--an output should be a group of cells, whatever shape you decide, hexagon, pentagon, squares, rectangles, what have you, that's sometimes called a fishnet overlay. It should be that cells that have completely overlaid the entire jurisdiction of Portland Police Bureau, with all the variables that are generated during that. There's normally an X min, Y min type variable that's auto generated when you create these overlays. And all you're doing then is just a binary yes/no variable inside that cell--inside that shapefiles saying yes or no if that cell is forecasted to be a hotspot as we call them or a cell likely to receive or experience a high call for service volume for the time period.

MARY JO GIOVACCHINI: Will you provide some example of shapefile? Any recommendation on the tools to read or create a shapefile?

JOEL HUNT: So it depends on what--I--I'm most familiar with working in the Esri ArcGIS environment. If you're working on version nine or old--any of the version nines, there's a program called HAF's tools that can create the square cells--square cells for you. In version 10 and newer, I think it's already built into the Esri ArcGIS system. It's in one of the toolboxes to create an overlay. Again, all it would be is a--I mean, we could provide one but all of this is going to be--it's just a bunch of squares that overall form the shape of the Portland Police jurisdiction.

MARY JO GIOVACCHINI: The participation is only possible from within the United States?

JOEL HUNT: Or the territories. Mostly that is because the GMS system will block external applications from coming in and the restrictions on the eligibility in terms of being a US resident.

MARY JO GIOVACCHINI: What do you mean by US resident?

JOEL HUNT: Again, this is one of those, if there are specific questions due to, like, J-1 Visas or things like that where there is--I know some J-1 Visas here supply Social Security numbers, but you may not be--in some cases, you're considered residents and for other things, you're not. It would be best, again, to email grants at NI--NCJRS.gov with your specific situation so that the legal team can look at it and say whether or not you're eligible to receive the type of funding that we have.

MARY JO GIOVACCHINI: For the list of addresses to exclude, do you mean we should exclude them from the shapefile system?

JOEL HUNT: We've already excluded them from the dataset so that you do not have to worry about that. We were just purely bringing attention too that some calls for service have been removed from the dataset.

MARY JO GIOVACCHINI: How is the number of crimes per cell submitted in the shapefile? Will that be the name of the cell or some other attribute?

JOEL HUNT: So, you're not going to be forecasting how many crimes you think happen in each of these cells. You--what you're forecasting is whether or not you think a cell is likely to receive or experience a high volume of calls for service. So you--what you're trying to forecast is the--which cells you think will have the highest call for service in the--for that time period. So it's just a binary variable yes/no if you want that one to be considered as a high-crime forecasted cell.

MARY JO GIOVACCHINI: International students on--or research engineer in a graduate school, can we participate in this challenge without a green card or citizenship?

JOEL HUNT: Again, it all falls down onto residence, whether or not you are a US resident. And if you have questions about that, it would be best to email the email address provided with your specific situation.

MARY JO GIOVACCHINI: Clarification about hotspot variable. I am--I am assuming that the hotspot needs to be calculated over the total forecasted time period. What if my cell size is smaller than the forecasted hotspot?

JOEL HUNT: I'm not fully sure what is meant by that, but I--I'll try to clarify this a little bit more. So, you're going to take the Portland Police Bureau jurisdiction and you're going to overlay whatever cell, shape, and size that you want as long as it meets all the requirements. What you're going to then do is select multiple cells that you think are forecast--oh, let me clarify that. You will then indicate which cells you forecast to have the highest--likely to have the highest calls-for-service volume during that next time period that you're forecasting for. So, all of these, you're going to be forecasting multiple cells. You're going to say, depending on what cell, shape, size, it could be, these 40 cells are the ones you are forecasting to be the highest calls-for-service volume.

MARY JO GIOVACCHINI: Can we submit a sample output file to validate that our predictions are in a correct format?

JOEL HUNT: At this time, that is not an option. However, I can look into it. And again, if so, we will update the FAQ page with a response to that.

MARY JO GOIVACCHINI: How is the small N in the formula PEI to be determined?

JOEL HUNT: Again, the small “n” is the--so what happens is, hypothetical situation that you forecast 30 cells throughout the jurisdiction to be hotspots. What we will then do is take the real data that occurred and see how many crimes occur in those actual 30 cells that you have indicated. And that is what your small “n” is.

MARY JO GOIVACCHINI: This is similar to the other questions that we've already received, but can an F-1 Student Visa holder who is legally enrolled in a university in the USA participate in the challenge?

JOEL HUNT: It's the same answer again. Please, with your specific situation, please email the grants at NCJRS.gov. I'm unfortunately not a J.D., so I'm not sure how all the--that legal part works.

MARY JO GOIVACCHINI: Aren't you worried that allowing commercial datasets will unnecessarily privilege those who merely possess more data over actual methodology?

JOEL HUNT: That is always a challenge. However, since we do not ask for the intellectual property. We would never know what datas used anyway so all we are trying to do is let people know that it is an option if people would like to do so.

MARY JO GOIVACCHINI: Can we publish on the methods we use to win the competition? It goes back to an earlier question.

JOEL HUNT: Yeah. Yeah. It goes back the same one. If you're talking purely about your methodology, then that--you're free to do with that as you like. You maintain all intellectual property rights to it. If you're trying to post results, again, we're going to be reaching back out to Portland Police Bureau to see if they can just give a complete clearance or not or if it's going to be a case-by-case basis for people using the data for other purposes.

MARY JO GOIVACCHINI: How is the small “n” star (n*) in the efficiency formula determined?

JOEL HUNT: So, if we go back to the prior example and we say that you forecast that 30 different cells are the ones that you forecast to be high crime areas, small “n” is how many crimes occurred in those 30 cells. Small N star (n*) is what are the--what is the sum of the 30 highest cells. Not just the ones you forecasted but looking at all the cells, if you were to--selected the 30 best cells, how many could you have forecasted?

MARY JO GOIVACCHINI: Will the final judgment be based simply on PEI star (PEI*) or is it a combination of PAI and PEI star (PEI*)?

JOEL HUNT: So, I'm going to quick back to a different slide here. So, looking at this, the PAI is what we're using for the effectiveness. So you'll see that the--there are going to be 20 prizes for PAI in each category and 20 prizes for PEI star (PEI*) which is the efficiency measure.

MARY JO GOIVACCHINI: At this time, we do not have any further questions in the queue. We will give you a few minutes to see if you have any additional questions. In the meantime, we're going to put up the information slide again. If you have any questions, you can go to the Challenge.gov in the FAQ section. We do recommend that you sign up and be notified via email when FAQs are posted. You may also contact the National Criminal Justice Reference Service at the 800 number or the email address listed. If you have additional questions, they will work with Joel to get an answer back to you. In addition, the--this presentation, the webinar, and the slides will be posted and you will receive an email with a direct link to those posted material in approximately 10 days, give or take.

We do have another question. If our predictions are good, will our methods be potentially used by police departments to help them in the future?

JOEL HUNT: So, at the conclusion of this, we are going to be releasing the information on the top winners in each group, the top applications in each group. At that point, if police departments would like to reach out to these individual--students, small teams, small businesses, large businesses to potentially work with them, there is that potential. NIJ will not be taking any of the intellectual property, so it's not that we will be able to disseminate it to police departments to use. US applicants are--to be a contestant, you must have intellectual property that you're submitting and you will retain that. So if you would like to--if you do well and would like to work with police departments, you are encouraged to do so.

MARY JO GOIVACCHINI: How is hot defined? By comparing with itself, say last week or last month, or simply sorting all cells overall and listing those with more crimes in the whole Portland area?

JOEL HUNT: Yeah, so the term "hotspot" is always an arbitrary term. Hotspot is always defined as X number of cells that have the highest calls-for-service category. And where that threshold cut is, is going to change every time period. What is a hotspot in time one, say 20 calls for service may not be a hotspot in time two even if it's 20 again because there may be even more cells that are now greater than 20. So it's just an arbitrary cut point and that cut point is somewhat defined by the total area that you must forecast to, that quarter square mile to three quarter square mile. So, it--you're essentially, when you're applying, going to be defining where that threshold is. If you forecast 30 cells, well then, we're going to compare it to 30 cells.

MARY JO GOIVACCHINI: Are there any other awards associated with this competition? Speaking opportunities, etcetera?

JOEL HUNT: At this time, no. However, we're looking at some potential opportunities for some of the winners to potentially either do presentations at conferences such as International Association of Chiefs of Police or International Association of Crime Analysts, there are some considerations and talks going on with the Global City Teams Challenge work, things like that, where they have a professional day and things like that. But at this point, we're still trying to iron out these additional potential incentives.

MARY JO GOIVACCHINI: Has NIJ funded any projects relevant to this challenge? And if so, are final reports from those efforts available?

JOEL HUNT: So, we have funded things like Rutgers and the Risk Terrain Modeling, it's a 2012 award. I'm not sure where their final report is at in terms of the archival process. However, if you do a simple Google web search, their webpage has already archived on their own website so that would be an easy place to go. The Predictive Policing out of Shreveport, which was evaluated by Rand, that report should already be out and I believe archived. There are a few other links actually on the NIJ webpage. There's a Predictive Policing webpage on NIJ.gov that will link you to some of the work that's been funded. And if there are final reports, they should be linked too in there also.

MARY JO GOIVACCHINI: What do--what does "Others" category mean in the data? E.g. 2012, simply other than the three, auto theft, street burglary, where can I find a full list of subcategories?

JOEL HUNT: So there are only going to be the four categories. They'll be the burglary, street crime, motor vehicle theft, and then other--and the other goes towards the all calls for service forecast. All calls for--calls for service forecast is for all four of those subcategories then. So when you're doing all calls for service, don't omit motor vehicle theft or burglary. They should be included in the all calls for service.

MARY JO GOIVACCHINI: At this time, we do not have any further questions. I'll give it a few seconds here to see if anything else come up. All right. If there's nothing further, then that will conclude today's webinar. On behalf of the National Institute of Justice, I'd like to thank you for joining us in this webinar. We hope you found it helpful, and have a great day.

JOEL HUNT: Thank you everyone. It's appreciated.

MARY JO GOIVACCHINI: We do have one other question.

JOEL HUNT: One question. Can--then we're good to go.

MARY JO GOIVACCHINI: Can they point at 75 square miles take cells from different neighborhoods?

JOEL HUNT: Yeah. These are not--these can be as disjointed a cells as needed. They do not have to be congruent cells.

MARY JO GOIVACCHINI: All right.

JOEL HUNT: All right.

MARY JO GOIVACCHINI: At this time, that will conclude our webinar. Again, thank you very much and have a wonderful day.

JOEL HUNT: Thank you.