Face recognition technology uses artificial intelligence to identify images of people captured by a camera or appearing on a webpage. The technology has evolved from a promising research niche to a profoundly impactful criminal justice resource.
As the power and precision of face recognition technology have advanced, so too has the federal role in cultivating and assessing face recognition algorithm development, and in helping to harvest viable end products for criminal justice, national defense, homeland security, and intelligence uses, as well as for various commercial applications.
For nearly a quarter century, the National Institute of Justice (NIJ) has assumed a central role in moving facial analysis algorithms from a hit-or-miss tool set to a widely deployed, rapidly advancing array of products where, according to the National Institute of Standards and Technology (NIST), 25 developers had face algorithms in 2018 that outperformed the best algorithm reported just four years earlier.
NIJ, as the research arm of the U.S. Department of Justice (DOJ), has directly supported building new face algorithms as well as creating programs to evaluate and compare face algorithms’ effectiveness. A third NIJ role has been as a close, ongoing partner with other federal entities representing a range of national interests and research priorities, to help shape the scope of face analysis tool development by identifying high-value applications and spotting potential risks and benefits.
Face recognition technology is a potent, practical application of artificial intelligence. Its reliable range is no longer confined to well-lit, head-on mugshots snapped at short range. Rather, face technology can now detect images captured from a distance of hundreds of meters, sometimes in poor light or at an off-angle, or where the subject’s face is partially masked or otherwise obscured. New-generation face algorithms tied to face image datasets and powered by ultrafast image processors can compare a face captured by a camera, or posted on the internet, to millions of stored face images in a database and produce, with high confidence, a list of potential matches to images of known persons.
Representative areas of NIJ engagement are described below under “NIJ’s Historical Role in Face Algorithm Research and Development.” These include:
- Early use of face recognition to control and monitor access to local correctional facilities.
- Development of high-powered, neural network face algorithms, followed by a groundbreaking “apples to apples” comparative evaluation of newer neural network algorithms and older face recognition algorithms in widespread use (see accompanying story).
- Evolution of face recognition of individuals from a distance.
- Long-standing support for NIST’s periodic Face Recognition Vendor Test, which has evolved into the singular national platform for periodically identifying the best face algorithms available. In evaluating face algorithms submitted by their developers and posting the results, NIST gives invaluable recognition to the winners, while identifying best-in-field products for government and private sector users.
With the growth in face algorithm competency has come public concern over the technology’s capacity to infringe on the individual rights of those being observed. In June 2019 testimony before the House Committee on Oversight and Reform, an FBI official noted the constant need to strike a balance between the technology’s probing power and privacy rights. “Facial recognition is a tool that, if used properly, can greatly enhance law enforcement capabilities and improve public safety, but if used carelessly and improperly, may negatively impact privacy and civil liberties,” noted Kimberly J. Del Greco, deputy assistant director for the FBI’s Criminal Justice Information Services Division, in written testimony. FBI policy requires that its face recognition tools be used only for leads, requiring follow-up corroboration, and that human examiners check all identifications made by machines, Del Greco added.
Policy questions aside, rapidly maturing face recognition technology has already been integrated into critical roles in intelligence, defense, criminal justice, homeland security, air traffic security, and other priority use areas.
Key Recent Developments in Face Recognition
Seven seminal developments mainly in the past decade have stoked the face recognition growth engine, according to preeminent authority (and NIJ grantee) Anil Jain, a Michigan State University computer science professor. They are:
- Availability of face images in mass quantities, largely due to the advent of social media, as individuals equipped with inexpensive but high-quality cameras upload images to the internet by the billions. Jain explained that with the proliferation of unrestricted internet images, “[w]e can crawl the web and get millions of faces.” (Facebook alone had nearly 2.4 billion monthly active users as of March 2019.) The recent face algorithm refinement surge is also a function of the free, public availability of massive image datasets from government and industry sources, Jain noted.
- Crowd sourcing of face identification, with major face recognition interests retaining, at modest cost, thousands of workers from around the world to assist with tagging of faces and facial features, greatly enhancing the technology’s impact. Crowdsourcing “has become a tremendous asset to the face recognition community,” Jain said.
- Deep convolutional neural network algorithms, self-training machine learning algorithms that mimic the mammalian brain and are particularly suited to the pattern-recognition capability essential to face recognition. (See discussion of deep neural networks in accompanying article). NIST credits the emergence of deep neural networks with a dramatic leap in face algorithm proficiency between 2013 and 2018. “The massive reduction in error rates over the last five years stem[s] from wholesale replacement of the old algorithms with those based on (deep) convolutional neural networks … This constitutes a revolution rather than the evolution that defined the period 2010-2013,” according to a report by NIST staff on the agency’s 2018 face algorithm competition.
- The NIST Face Recognition Vendor Test, which now attracts a significant number of face algorithm makers seeking national validation and a reputation for excellence. The NIST competition has created an efficient feedback loop:
- Government entities help drive research and development of face algorithms.
- Universities and commercial entities develop new algorithms and voluntarily submit them to NIST for testing.
- NIST tests the submitted algorithms and identifies the best performers.
- Government (and other) users reap the benefits of the best face algorithms identified by NIST.
- Hardware advancements, allowing the formidable data-handling prowess of new-generation graphic processing units to work hand in hand with deep neural networks. This allows a comparison between an object-face image and millions of known images in a matter of seconds.
- A large government investment in face algorithm development, particularly in the defense, homeland security, intelligence, and criminal investigation realms.
- Janus, a project of the Intelligence Advanced Research Projects Activity (IARPA) within the Office of the Director of National Intelligence, designed to measure and validate techniques with the potential to improve recognition of unconstrained images — meaning images “in the wild,” reflecting a range of real-world imaging conditions. IARPA launched Janus to address perceived inadequacies in face recognition algorithms.
Those transformative, relatively recent developments rest upon foundational face recognition work by NIJ and others stretching back more than two decades.
NIJ’s Historical Role in Face Algorithm Research and Development
NIJ began funding face detection and recognition research in 1996. The research agency supported development of technology, as well as standards used in field demonstrations, and helped transition technology to local practitioner agencies.
Interagency Cooperation
Justice Biometrics Cooperative
An early cooperative effort was anchored in, but not confined to, DOJ. The Justice Biometric Cooperative (JBC) was founded in 2003 as a focal point and clearinghouse for information on biometric research, technologies, and applications. The founder of the cooperative, DOJ Chief Science Advisor Vahid Majidi, Ph.D., said at the time that although component entities were free to pursue their own biometric activities, “[w]orking together will accelerate advanced solutions.” NIJ also served as the primary DOJ representative on a biometrics committee of the InterNational Committee for Information Technology Standards (INCITS), playing a central role in the JBC. The goal of INCITS’s M1 Committee was the development of better, standards-based biometric security solutions for homeland defense and other purposes. One of five approved standards produced by the M1 Committee was ANSI/INCITS 385-2004: Face Recognition Format for Data Interchange. During the same period, NIJ, the FBI, and other agencies were cosponsoring the Face Recognition Vendor Test Challenge at NIST.
Subcommittee on Biometrics and Identity Management of the White House National Science and Technology Council
NIJ was a key participant in the development and promulgation of a National Science and Technology Council biometrics subcommittee working group involving a number of federal agencies.
Informal Interagency Working Group
More recently, an informal working group of key agencies engaged in face detection and recognition has been highly active, with regular meetings and information-sharing efforts. Participant agencies in addition to NIJ have included the U.S. Departments of Defense, Homeland Security, Justice, and Commerce.
Pittsburgh Pattern Recognition
Federal investment also led to development of nonproprietary face algorithms that became a form of community property for agency use. A leading face algorithm, Pittsburgh Pattern Recognition (PittPatt), developed in 2004 by Carnegie Mellon University with federal funds, was tweaked differently by intelligence and law enforcement agencies to suit their varying needs. Federal agencies’ collective commitment to building better face algorithms reflected recognition that the state of the art for accuracy of face matching was, at the time, “well down on the curve in terms of not doing well,” said William Ford, a senior science advisor at NIJ. Subsequent progress along the curve has been substantial, with massive acceleration since the 2013 advent of convolutional neural network algorithms. Patrick Grother, a NIST computer scientist and an author of that agency’s report on the 2018 Vendor Test, said the rapid advance of machine learning tools has effectively revolutionized the industry. “The implication [of the fact] that error rates have fallen this far is that end users will need to update their technology,” Grother said in a NIST publication.[1]
Research and Development
The following is a representative sampling, from dozens of past NIJ grants totaling millions of dollars, of advancing research and development of face detection and recognition technology. At this time, NIJ is not issuing new funding for face algorithm work, but several NIJ grants remain active, and NIJ continues to serve as an essential subject-matter collaborator with other federal offices on face algorithm development and evaluation.
| Title | Awardee and Source | Description |
|---|---|---|
| Face Recognition and Intelligent Software Development for Internet Exploration for Child Pornography and Exploitation | Analytic Services, Inc., 1998 grant, 2004 report. | Development of face recognition technology to identify subjects on the internet, identify and locate missing and exploited children, and fight child pornography on the web. |
| Pilot Project | Prince George's County Correctional Center, field application begun in 2002. | Use of face recognition to control routine staff access to, and egress from, a correctional facility. In an article describing the project at the time, the center's chief of security, Lt. Col. Carl Crumbacker, said the technology was an improvement over human security checks. "With 300-plus employees, it's very hard to have an officer identify each one." He added, "Logging employees as they enter and exit the facility allows officers to find out who is in the facility in case of emergency." |
| Recognition of Non-Cooperative Individuals at a Distance Using 3-D Face Biometrics | University of Southern California, 2006 grant, 2008 report | An early 3D face modeling program found that video data could yield a 3D face shape for successful face recognition at a distance of 3 to 9 meters. The researchers cautioned, however, "The performance of 3D-3D recognition with the currently generated models does not quite match that of 2D-2D." The effective visual range of face recognition technology has greatly expanded in ensuing years, with effective recognition at distances of 300 meters or more, depending on environmental conditions. |
| TACIDS: Tactical Identification System Using Facial Recognition | , The Automated Regional Justice Information System, 2007 grant, 2012 report | A consortium of 82 local, state, and federal law enforcement agencies received a smartphone-based tool that enabled officers in the field to identify individuals who had been stopped or arrested, but who were not carrying, or did not produce, an ID. Identification was made by comparing suspect photos taken on the scene to databases holding nearly 1.4 million booking photos and mugshots. |
| Matching Forensic Sketches to Large Face Image Databases | Michigan State University, 2011 grant, 2015 report | Primarily intended as a case-lead generating product, an algorithm was developed to optimize the potential of facial features in sketches of suspects drawn by law enforcement sketch artists, for automated face recognition applications. |
| Practitioner Centric Video Analytics | GE, 2013 grant, 2015 report | Using 3D video analytics, GE developed a working prototype trained to recognize certain common behavior patterns that could be predictors or indicators of criminal activities. 3D technology is not found across the spectrum of face recognition applications; 2D is more common. But for particular uses, said Ford, "3D has great promise. It is much farther advanced than it was 10 years ago. It used to require a special camera — now you can use a smartphone." Ford noted that NIJ funded the Amber View project by the West Virginia High Technology Consortium Foundation in which 3D cameras were used to take and store photos of children for reference by law enforcement in the event of a suspected abduction. By 2006, the program was operational, with annual school photos of children routinely uploaded into the Amber View system for reference if needed. |
| Janus Program | NIJ Memorandum of Understanding with IARPA | NIJ and IARPA share insights and information to help guide Janus activities and determine the applicability of Janus work to NIJ face recognition activities. See item 7 in "Key Recent Developments in Face Recognition" above, for a description of the Janus program. |
| A Simultaneous Low Resolution and Off-Pose Angle Face Algorithm as Investigative Lead Generative Tool for Law Enforcement | Carnegie Mellon University, 2013 grant, 2017 report | In a project of high interest to law enforcement and homeland security, Carnegie Mellon developed a new algorithm proficient at recognizing faces in poor lighting conditions, or at an angle with less than the full face in view. Reliable recognition of faces photographed at off angles is vital to agencies such as the Transportation Security Administration, which relies on ceiling- and wall-mounted cameras to identify passing travelers in real time. |
| Matching Forensic Sketches to Large Face Image Databases | Michigan State University, 2011 grant, 2015 report | Primarily intended as a lead-generating product, an algorithm was developed to optimize the potential of automated face recognition using sketches drawn by law enforcement sketch artists. |
| Design and Implementation of Forensic Facial Identification Experts Test | University of Texas, Dallas, 2015 grant (ongoing) | Development of a test to assess the proficiency of forensic face examiners across a broad range of facial identification tasks, and to compare their abilities with and without the assistance of deep learning face algorithms. |
Going Forward: Managing Higher Degrees of Difficulty
The latest NIST Face Recognition Vendor Test reflected significant progress in the technology across the board. In the 2018 challenge, top algorithms experienced a failure (i.e., false positive or false negative match) on NIST-provided data inputs only 0.2% of the time, compared with a 4% failure rate in 2014 — an improvement by a factor of 20. Leading face recognition authority Anil Jain cautioned, however, that the field has far to go in improving face recognition performance. Achieving ever better accuracy will continue to pose a serious challenge, even with the worldwide proliferation of security cameras — including an estimated 200 million CCTV cameras installed by the Chinese government — generating images from a distance, from bad angles, and in poor light, he said.
Looking ahead, Jain said that research priorities are likely to include developing defenses to secure sensitive government face image databases from cyberattacks; combating attempts to breach secure systems that use face recognition for access; and establishing standards to decide, for example, who owns an individual’s biometric data, and when and under what circumstances biometric data should be removed from a government database.