AI researcher Martin Tamajka: “If we are to trust AI in courtrooms, it needs to justify its decisions.”
Martin, can you tell us what explainable AI is and how we can imagine it at work? Is it essentially a “ChatGPT” that can give you resources and reasoning?
First of all, let’s look at the ordinary “black box AI”. In this model you feed it data and train it for a specific task, for example: “tell me whether there is a dog or a fish in the picture”. We can see the input (the data you trained it with) and the output (the prediction), but we don’t exactly know what happened in between to generate the prediction.
Of course, we know the principle: in artificial neural networks, hundreds of thousands or even millions of nodes add or multiply the individual pixels to find structures characteristic of an image of a dog or a fish. In fact, they act as if they subsequently filtered the image – first, they detect simple structures like edges or blobs of different colours, and later they combine these structures into more abstract ones until they reach the final prediction.
However, we don’t know exactly what calculations and node combinations have led to that specific response – even though we can understand how multiplication and addition work, there are so many of them in this decision-making process that humans are not able to mentally contain them.
And this is where explainable AI could step in and show us its decision process?
Yes, explainable AI can show us what influences its decision-making. For example, in the dog/fish-image scenario, explainable AI could highlight the sets of pixels that have influenced its decisions the most. So, if the picture had a rectangular shape and a lot of water, it’s most likely a fish. Whereas if the picture had three big black blobs and big ears, it’s most likely a dog.
In this way, we could potentially verify whether the prediction of an AI system was based on relevant parts of the input data.
So, to answer your question about explainable AI being able to reason, then the answer is no – no reasoning, or at least not for now. Even though ChatGPT might in many cases provide a certain level of reasoning, it’s still far from being reliable. Many other AI models provide no reasoning at all. Hopefully, one day we will have mechanisms able to provide reliable and human-understandable explanations for at least the majority of models used in practice – that would be the holy grail of our work.
Wisconsin vs. Loomis
In 2013, Eric Loomis was found driving a car that had been involved in a shooting. To determine his sentence, the judge used both his previous criminal record as well as an output from the COMPAS algorithm. He was classified as being at high risk of re-offending, and sentenced to six years.
COMPAS itself has faced controversy multiple times. Based on a study by Julia Dressel and Hany Farid from Dartmouth College, it was found to be working with a level of precision similar to the judgement of random strangers on the internet.
Here we can see the problems with black box AI directly, most prominently in the case of Wisconsin vs. Loomis, where the defendant appealed the ruling because he felt that the judge made the decision based on an algorithm whose workings were secretive and could not be checked or validated. He argued that this was unfair and went against his right to have a fair trial.
The appeal went up to the Wisconsin Supreme Court, who ultimately ruled against Loomis. Their ruling, however, urged caution in the use of algorithms. We mustn’t forget that even felons are still people with rights – and the right to an explanation of a ruling is one of them. You must be able to show what you have based your ruling on, and here, explainable AI can help considerably.
Let’s go back to the courtroom. You mentioned that we need a balance between not being overwhelmed, but also getting a detailed enough explanation of the decision. Is there any way that judges could ask for either a more or less detailed explanation?
One way how this can be tackled is something similar to a “sensitivity slider”. A judge working on a case could ask the AI to give them a list of the ten most decisive factors for the suggested ruling. They can consider them, and if they need more granularity, just increase the sensitivity and see even more factors, which individually hold less weight but as a whole give a broader picture.
In other words, the judge can first look at the most significant argument for or against sending someone to jail, and then they can look at an even broader picture if needed.
This also applies to different levels of understanding. An explanation for a skilled judge might consist of 50 different factors, uncovering a high level of detail and nuance. While the same system can explain the case to a first-year law student by using just the three most important factors, summing up the gist of the case with the most hard-hitting evidence.
While on the subject of court rulings, let’s get to AI biases. There is mounting evidence that court rulings are biased toward certain groups of people. Can explainable AI help find those biases?
Of course, but it’s not that the AI will directly tell you there is something wrong. But by examining which attributes matter more than others in decision making, you can see the bias of the training data that was projected into the model… or indeed, the world.
There was a famous case where explainable AI was used to study how a machine-learning model classified different types of objects in pictures, including horses. They thought the algorithm worked brilliantly until they found out that instead of looking at important parts of the picture that should be relevant for the prediction, the algorithm just focused on the watermark. This led the model to implicitly create a rule like this – “if there is a watermark, say ‘horse’”.
“It’s not that the AI will directly tell you there is something wrong. But by examining which attributes matter more than others in decision making, you can see the bias of the training data… or indeed, the world.”
If you use an XAI algorithm on thousands of court-case final verdicts and the predictions of the AI model that served as recommendations for the judge, you can get a nice image of how the underlying AI model actually creates its decision. In this way, you can potentially discover biases present in the model.
To fix such biases you can clean your training data and retrain the model, adjust training strategy to explicitly punish the model when it makes a prediction based on some bias, or use another “de-biasing” procedure, such as deliberately including more of some specific types of rulings, or giving them more weight in the input.
Let’s finish up on the most important subject – responsibility. Who is responsible when the AI makes the decision? How do you think the future of responsibility for decisions will evolve?
I don’t believe we should let any kind of AI do important decision making in the foreseeable future. I see AI and humans as more like co-workers.
There are many things I would let AI do solely on its own. I will let it recommend me the best phone. The same goes for maybe 80–90% of the decision making in many professions. Let the AI do the tedious and repetitive work, so you have time and mental capacity to focus on the 10% of difficult cases.
Especially in critical decision making, a human should always have the last word. AI can be an advisor to a doctor, judge or soldier, but they must make the final call and weigh out all the pros and cons.
Let’s not forget that AI does make mistakes. It’s far from infallible. And it will never be!
There will always be errors, and people have to learn to question AI. I believe that questioning it and not believing its outputs outright is what will, in the end, add to the acceptance of the technology.
In these times, when we’re observing the growth of very powerful generative models such as ChatGPT or Midjourney, we will have to learn how to check the trustworthiness of many things on the web, in the news or on TV, as creating things like Deepfake will be easy.
So yes, we will always need humans to be making the final call and thinking critically about the outputs they get from the AI. Just as you don’t believe everything you find on Google (or I would like to believe so), you will learn to use your common sense when working with AI. In this way, it can help us in many areas while keeping the risks minimal.
Martin Tamajka has been working with different forms of AI in high-responsibility domains since his student years. In the past, he focused on the analysis of multidimensional medical images and image analysis in general. He now works on deep learning and computer vision and aims to increase the transparency and reliability of neural networks through methods of explainability and interpretability.
AUTHOR: Andrej Kras
PHOTOS: Dominika Behúlová