Former NYPD inspector Joseph Courtesis:
“We create less bias when we use facial recognition algorithms in our work.”
You also mentioned link analysis as an important aspect of connecting disparate sources of data, even ones outside law enforcement databases. Can you give us an example of how you used it in your day-to-day work?
I am basing this example on what could happen. Imagine you and I were members of a criminal organisation but when interviewed by the police, you claim we never met. How could the police uncover our hidden connection?
The key is to follow the trail of data by finding connections across large datasets that would not typically cross-pollinate. For example, link analysis tools can identify how the only phonecall made by you, during an arrest, could match a phone number found in a public database that belongs to me.
“We were aware of the technological limitations and did not want to scale facial recognition to the entire department. Rather, we created a subunit dedicated to facial recognition searches, which made it possible to govern the whole process under a comprehensive policy.”
It all started with something as mundane as grocery shopping. When I swiped my supermarket membership card, I did not realise that my phone number became intertwined with the store’s database. And what do supermarkets do? They sell this data. This finding leads the police investigator to ask you the defining question: “Why did you use your one phonecall to call Joe if you claim to be strangers?”
You see, no human being would be able to find that connection, but with the link analysis, police investigators can ask the system to search through disparate databases and uncover these connections in seconds.
The use of biometrics in criminal investigations
How widespread is the use of biometrics in police forces?
It is growing every day. Fingerprints are still the gold standard in identification. DNA testing has developed so significantly that we can now get DNA results in about two hours. Facial recognition is evolving so quickly that the accuracy of these algorithms is almost equal to that of fingerprint algorithms.
Can you walk us through the process of applying facial recognition to a piece of evidence from a crime scene?
Let’s say a bank got robbed. In that bank, there was a camera that captured the perpetrator committing the crime. The police upload the image into the technology and run it through a database or a repository of images – typically mugshots.
The system then produces a gallery or a list of the most likely candidates. For example, out of ten million photos, it will produce the top ten or the top hundred candidates. You decide how large you want that list or gallery to be. Some agencies rank it by a threshold setting, others by the top ten or top three. It is all part of your personal policy.
Now, with an algorithm accuracy of 99%, if the perpetrator is already in the database, they are likely to pop up first on that list. However, you still have to go through the list and analyse the candidates thoroughly. Most of the time, it will be easy to spot who the perpetrator is not. You will go through the list mumbling “That’s not him, that’s not him…”
However, when the perpetrator is not already in the database, the algorithm will still produce a candidate list of similar-looking people. For this reason, all facial recognition leads have to go through a thorough vetting process. During this process, the goal is to “rule out” the lead and try to prove the algorithm wrong by deploying human review and investigative follow-up.
The investigative follow-up will either rule the lead out or produce corroborating evidence that helps establish probable cause to arrest the suspect. For example, you may look at the lead subject’s open-source social media and find incriminating images or posts. On the other hand, you may find images that support a strong alibi such as a picture of them in Disneyland from the same day the crime was committed.
So, for police officers and investigators, facial recognition is one step in a multi-step image identification process and is never the sole factor in producing probable cause to arrest.
I speak for myself and not for other agencies, but I did not even show my investigators the percentage accuracy of each match in the list of candidates because I did not want it to influence their decision. I would rather they just look at each candidate as if they are probably not the subject.
Even though the algorithm is a fantastic tool for developing leads, it is not perfect. You have to keep in mind these are just leads. The important distinction is that you are more likely to make a mistake identifying a perpetrator without this technology than with it.
“Even though facial recognition is a fantastic tool for developing leads, these are just leads. The important distinction is that you are more likely to make a mistake identifying a perpetrator without this technology than with it.”
Can you give examples where facial recognition has been very helpful in a criminal investigation?
It is extremely helpful in criminal investigations captured by video surveillance and cameras. Imagine a burglary, which is one of the most difficult crimes to solve. Somebody breaks into your house while you are not home and when you come back you see a broken window and that your things are gone. There are no witnesses, and a lot of the time, no evidence.
Typically, the only way we solved these crimes was through some sort of forensics – the perpetrator may have left a fingerprint or DNA behind. Today, most people have their own camera systems that might have captured the perpetrator committing the crime, giving us significant leads to follow up on. This has increased the number of leads available to us, resulting in improved clearance rates for these types of crimes.
Is biometrics used ethically in law enforcement?
You now advise about the ethical use of biometrics, especially facial recognition. What exactly are you advocating for?
Well, since facial recognition is one of the newer biometric technologies used by law enforcement, it comes with some limitations we need to tackle. It is still not as good as fingerprints or DNA. That is why I am working on setting expectations for law enforcement departments that want to use it in their investigations.
They need to understand that facial recognition does not produce probable cause to arrest, but merely a lead in the investigation. I am also educating about the strengths and weaknesses of this technology and helping to create a policy about its ethical use that is not only transparent but also enforceable.
Let us stay with this topic for a bit. When might the use of facial recognition in criminal investigations be problematic?
To put it into perspective – law enforcement has been identifying suspects through images long before there was an algorithm around. We just were not very good at it and therefore could not produce strong leads with it.
Facial recognition improved our efforts but, just like adding any other technology to an already existing process, you must think policy before technology. I admit we probably could have been quicker in establishing our policy regarding facial recognition because without understanding the weaknesses of the technology, we cannot ensure that, for example, the facial recognition lead is not too suggestive.
Remember that this is a strong algorithm that looks through millions of images to find the best match for the probe image you provided. However, before acting on the recommended lead, it is important to thoroughly evaluate and validate it. There is a decent amount of vetting that needs to go into this process that people might not understand if they do not understand the technology.
Can you also say why or how using it in a controversial way could affect human rights? For example, using it to identify protesters at a rally?
This also falls into the lack of a comprehensive policy issue. And to begin with, policy needs to be consistent with the way you conduct law enforcement investigations.
Law enforcement should not use facial recognition to identify people exercising their rights to free speech and peaceful protest. While peaceful protesting is typically protected, there is a distinction when it crosses the line into criminal activities, such as assaulting police officers or damaging property. In such cases, the use of technology in law enforcement investigations becomes appropriate, if it remains within the legal boundaries.
“The fact that facial recognition cannot be used in ways that violate human rights has to be clearly stated in the policy. While it may seem like common sense, it is important to have this written down to ensure responsible use.”
The policy should clearly state that facial recognition technology cannot be used in ways that violate rights or cause community unrest. While it may seem like common sense, it is important to have this written down to avoid any problems or misinterpretations and ensure responsible use.
How do you go about creating these policies with law enforcement?
It must be clear from the policy when to not use facial recognition, or when it is expected to be used. For example, I would advise to never use facial recognition as the sole source of identification, nor would I use it with live video feeds.
When should you use it? Well, I suggest you use it when you have reasonable suspicion that someone has committed or is about to commit a crime. Also use it when trying to identify a witness, aid somebody who cannot identify themselves or verify an identification.
The harmful myths connected to facial recognition
Another big concern of the public is bias. It is believed that face recognition algorithms could disproportionately harm black people, as the technology has higher error rates for people with dark skin.
This statement is misleading as well. The truth is that there are hundreds, if not thousands, of facial recognition algorithms being tested by the National Institute of Standards of Technology (NIST), so to say that the technology overall has a higher error rate in people with darker skin just does not hold.
Yes, there may be some algorithms that had higher error rates for people of colour, I do not deny that. But we should not label the entire industry as biased based on some algorithms that may not have performed well.
On top of that, the introduction of convolutional neural networks and deep learning, which are nowadays embedded in facial recognition algorithms, has improved them at a rate that exceeds Moore’s Law, which states that technology doubles in progress every two years.
“There may be some algorithms that had higher error rates for people of colour. But we should not label the entire industry as biased based on some algorithms that may not have performed well.”
Algorithms today are in some cases performing with 99.88% accuracy across all demographics – it is pretty incredible. However, the discussion about algorithms often comes down to talking about inferior algorithms tested 7–8 years ago.
They occur when a facial recognition algorithm produces a match between two individuals who bear a striking resemblance to each other, but are not the same person. In other words, the algorithm incorrectly identifies someone as a match, when in fact they are just a doppelganger or look-alike of the person being searched for. This can occur due to similarities in facial features.
We can mitigate the chances of an algorithm creating a doppelganger match by:
- Robust Training Data
- Unique Feature Extraction Techniques
- Algorithm Refinement
- Confidence Thresholds
- Human Verification
So the algorithms deployed today are far superior to the ones created in the past.
You may call me crazy, but for me, algorithms today may be too accurate. When you are developing a lead, you never want the lead to be too suggestive. Society wants a perfect solution for lead development, but leads are not supposed to be perfect – if they were, they would not be called a lead.
Our aim is to create an effective process to generate leads, and these algorithms are far superior to humans in creating leads from images. Actually, talking about bias, I think it would be more biased to not use these algorithms than to use them.
Do you see any potential danger related to your comment about the algorithms being too perfect for creating leads?
Yes, honestly, I do think that we are possibly entering into a different problem that we did not anticipate. The newer algorithms are more likely to produce a doppelganger match from a large dataset than the older algorithms.
Also, if the algorithms are 99.88% accurate, are we aware enough to find that small percentage where they are wrong? However, I do not want to scare anybody. We do not have to move away from facial recognition, because there are steps that can mitigate these issues.
I recently wrote a resolution that recommends an investigative checklist or best practices for vetting a lead generated by facial recognition technology. I submitted the resolution to the International Association of Chiefs of Police for review.
“Leads are not supposed to be perfect. Our aim is to create an effective process to generate leads, and these algorithms are far superior to humans in creating leads from images. Actually, talking about bias, I think it would be more biased to not use these algorithms than to use them.”
Finding a balance between safety, civil rights and case integrity
How can law enforcement benefit from face recognition while minimising the risk to individual privacy and other issues such as the integrity of the criminal investigation?
This is where the policy comes in. When developing the policy, it is crucial to find a balance between public safety, civil rights and the integrity of a case. It is a delicate balance because we need to protect the ongoing cases while being transparent with the community.
The policy aims to find the right rules for using facial recognition so that it promotes trust, accountability and transparency. By finding this balance, we can address concerns and make sure the technology is used responsibly.
“When developing the policy, it is crucial to find a balance between public safety, civil rights and the integrity of a case. It is a delicate balance because we need to protect the ongoing cases while being transparent with the community.”
Can you provide an example of how policy balances these three important roles?
There are currently three identifiable ways of using facial recognition. The first one is post-investigatory facial recognition, where an officer has an image of the perpetrator from a crime scene and their task is to identify them.
The second one is called mobile ID verification. Imagine an officer pulls you over for a minor traffic violation, and you’ve forgotten your ID at home. The officer asks for your permission to take a picture of you and identify you through a mobile ID verification system. It is an opt-in, therefore the officer is not violating anybody’s civil rights.
The third one is live facial recognition, where the technology is connected to video surveillance equipment and is identifying people in public as they go by. This is the one that the public worries about the most, and the most difficult one with which to ensure individual privacy and civil rights. In this case, we need more scrutiny in our policy. I never deployed live facial recognition and I cannot think of anybody in the U.S. that deployed it.
There are, however, agencies from abroad that deploy it. They usually put a warrant on this type of facial recognition. So, if they wanted to use the technology, they would have to get permission from a higher authority and then deploy it. In the case of an emergency, they could deploy it without permission but then have just 24 hours after the incident to explain why they deployed it.
“I have no objections to regulations on this technology because I do not want to live in a country where my face is being captured against my will and put into some database.”
To ensure privacy and protect individual civil liberties, it is important to actively develop a comprehensive policy for each use case, identifying the prohibited uses and the acceptable-use scenarios.
Do you think that using this technology for 24 hours without a warrant is a good policy, or could it be improved somehow?
If I were going to deploy live facial recognition, I would think that it is a rather good policy and the only way of appropriately governing the police and minimising the risk of violating people’s rights and civil liberties.
To give you a perfect example of a use case, imagine you are at a state fair and there is a children’s rides section. By putting up a temporary surveillance system in that section with a sign that says This area is monitored by facial recognition, you could monitor the presence of people who are on the sex offender monitoring list, and thus have an order to stay away from children.
For me, this is an example of a use case the public would have no problem with. I can imagine the people who are on the sex offender monitor list could have a problem with it, but the public should be fine.
Are there threats from facial recognition to the police as well? For example, when used by crime gangs?
There are credible threats, especially from companies that may issue facial recognition technology outside of a law enforcement setting and use it to search against social media databases. This creates a dangerous situation for undercover officers who work their whole careers pretending not to be law enforcement. This type of technology could expose their connection to law enforcement and put them and their families’ lives in danger.
In light of these potential dangers, I would highly suggest that we take a second look at our undercover program and consider suspending or revising it before it is not too late.
Is there some way policy could help to tackle this problem?
I sadly do not have an answer to this because how do you put a policy on a private company and make sure you can trust them? I would like to sit down with some subject matter experts and discuss it in a lot more detail. It is definitely the next topic I am looking to explore.
What is your future outlook on biometric data in criminal investigations from both ethical and technological points of view?
Moving forward, we are going to see many more comprehensive policies being developed. Eventually, each U.S. state will have a policy that governs the way law enforcement uses the technology, and at that point, the federal government will recapitulate all those policies into one government-regulated policy.
I have no objections to regulations on this technology because I do not want to live in a country where my face is being captured against my will and put into some database. I am not looking for that. I do not think anybody is.
AUTHOR: Kristína Zrnčíková
ILLUSTRATIONS: Matej Mihályi