Cynthia Rudin is the Gilbert, Louis, and Edward Lehrman Distinguished Professor of Computer Science at Duke University. She directs the Interpretable Machine Learning Lab, and her goal is to design predictive models that people can understand. Her lab applies machine learning in many areas, such as healthcare, criminal justice, and energy reliability. She holds degrees from the University at Buffalo and Princeton. She is the recipient of the 2022 Squirrel AI Award for Artificial Intelligence for the Benefit of Humanity from the Association for the Advancement of Artificial Intelligence (the “Nobel Prize of AI”), as well as the INFORMS Society of Data Mining Prize in 2024. She received a 2022 Guggenheim fellowship, and is a fellow of the American Statistical Association, the Institute of Mathematical Statistics, and the Association for the Advancement of Artificial Intelligence.

Opening the Black Box in the Age of Generative and Agentic AI

Gulan: When we interviewed you last year, you described the interpretability of systems such as ChatGPT as an unresolved frontier, comparing their internal processes to information blended into a smoothie. Since then, generative AI has evolved from answering questions to performing increasingly complex tasks through autonomous agents. Has meaningful progress been made in understanding how these systems reach decisions, or is their growing autonomy advancing faster than our ability to interpret and control them?

Dr. Cynthia Rudin: Interestingly, not much. There are a lot of researchers trying very hard to understand the internal calculations of large language models (LLMs) or build more interpretable LLMs but, as far as I can tell, none have succeeded. However, there have been advances in learning specialized models and smaller models, as well as developing agents that can use interpretable tools. I’m still not sure whether we truly need interpretability for a lot of LLM operations since they often provide information that I can check myself. So even if the LLM is a complete black box, if it sends me to a website where I can check its results, or gives me the name of a medical condition I can look up, or gives me a mathematical proof I can check, then I’m happy. The problem is that LLMs are extremely difficult to troubleshoot because they are black boxes.

People are worried about what autonomous agents will do, so they are building protections against unexpected behavior – perhaps that won’t be needed if the models were more interpretable (and thus controllable).

Gulan: The AI industry increasingly presents explanation tools as evidence that complex models are becoming more transparent. However, your work has consistently distinguished between explaining the output of a black box and building a model whose reasoning is inherently interpretable. As explanation techniques become more sophisticated, how can policymakers and the public determine whether an AI system is genuinely understandable or merely producing a convincing explanation after the decision has already been made?

Dr. Cynthia Rudin: Good question! And it gets at the heart of why I work in interpretability – because explanations are always either flawed or incomplete for black box models. (If not, you wouldn’t need a black box model, since you’d already have an interpretable model.)

The answer to your question depends on the type of data you’re working with. If it’s tabular data (the kind of data that is stored on a spreadsheet), then you rarely get more accuracy with a black box model, and we need policy makers to understand that. For instance, predicting whether someone is going to commit another crime, or predicting whether someone is going to default on a loan – those are questions that don’t benefit from complex or black box models. If, instead, you are working with images, there are excellent interpretable neural networks for computer vision. So again, there’s no need for completely black box models. For large language models, that’s more complicated. As I said, even if the LLMs are black boxes so that we don’t understand how or why they output what they do, sometimes they can provide you with enough information that you can check the answer yourself, for instance, it might tell you a reference to look up to find the information from a reliable source. Or, it can tell you what reasoning process it used, so you can check that reasoning directly. That way, you might at least be able to determine whether it’s wrong, but if it is, there’s not really an easy way to fix it. You could try prompting it differently, but that’s not troubleshooting the LLM, it’s just asking it to try again!

Gulan: In your recent work, you have examined the possibility of creating models that are interpretable to those affected by their decisions without making their complete decision boundaries available to competitors or potential attackers. Does this offer a practical compromise between public accountability and legitimate concerns about intellectual property and security, or could companies use secrecy as another justification for withholding essential information?

Dr. Cynthia Rudin: Yes, that was the goal of that project. It was to prove that it is possible for a company to use a model where each decision had a complete and faithful explanation attached to it, so that anyone affected by the decision could completely understand what values were being added up to get their result (and yes, it is just adding up values). On the other hand, while each decision is understandable, the full model can be rather large so that it is more difficult for another entity to figure it out. So it’s a good compromise between accountability and legitimate concerns about intellectual property and security. If a company claims they must keep a secret, someone could point to our technique and say “hold on, I’m not so sure you need to keep that secret.”

Gulan: You have also warned that proprietary AI developers can restrict independent research capable of challenging claims about their systems. As a small number of powerful companies increasingly control access to advanced models, training data, and computing infrastructure, are we facing a scientific accountability problem in which the public is expected to trust claims that independent researchers cannot properly test or disprove?

Dr. Cynthia Rudin: Yes, absolutely. I think you’re referring to an article I wrote about wearable devices for heart monitoring. It makes me really upset that there is no decent large-scale dataset available to test heart monitoring algorithms – we just have to believe what the companies say, and I don’t believe them based on the data I’ve seen. But the problem – in general – is quite bad – I’ve seen articles about companies that make LLMs, where they create a version of the LLM that can pass a safety test (perhaps they make it less powerful so it has fewer capabilities), but then they change the version of the LLM, and now it is really powerful and really dangerous, and no one knows that they did it. Companies heavily restrict access to researchers, so there is really no way for researchers to do the kinds of safety evaluations they were hoping for.

Gulan: Your research demonstrates that interpretable models can assist with medical decisions, including breast cancer prediction and critical-care monitoring. Yet healthcare institutions are under growing pressure to adopt generative AI and commercially developed systems whose internal reasoning may be unclear. What standard of evidence should hospitals require before allowing such systems to influence diagnosis or treatment, and who should be held responsible when an opaque model contributes to medical harm?

Dr. Cynthia Rudin: This question needs to be handled for each domain separately, but one thing in common to many domains is that interpretable models should be used for high stakes decisions unless it is not possible to practically construct an interpretable model. We could make a few exceptions to this, for instance, if the black box is 100% accurate, or if the result is easily checkable by a doctor. Using interpretable models could really change everything about many important healthcare decisions.

In the U.S., I’m sure there will be plenty of lawsuits over the use of black box models. You can already read in the news about lawsuits against big insurance companies for using algorithms to make high stakes decisions that were decided incorrectly. (So far, those haven’t been generative AI models as far as I know, and possibly not even AI.) One challenge for patients will be knowing whether an AI algorithm was even used at all since the processes for making decisions are not always transparent.

Gulan: In your work on the “Rashomon Set,” you argue that there may be many models with similarly strong predictive performance, allowing decision-makers to select one that also satisfies requirements such as fairness or simplicity. Could this approach change how we think about bias in AI by demonstrating that discriminatory outcomes are not an unavoidable consequence of accuracy, but often reflect choices made by model developers and institutions?

Dr. Cynthia Rudin: Yes, it could, but it’s important to remember there are many sources of uncertainty besides just what model you select based on a data set. Data can be really messy. I’ve never seen a clean real-world dataset! Patient records can be messed up in all kinds of weird pathological ways. Same with loan decision data. Criminal justice data is incredibly noisy and messy. Discriminatory outcomes might simply be due to the processing of the data. One of the benefits of interpretable models is that it helps you troubleshoot not only the model but also the data. It can help you find the biases so you can remove them.

Gulan: Countries and regions with emerging digital economies, including the Kurdistan Region of Iraq, are beginning to introduce AI into healthcare, education, banking, and public administration, often by importing systems developed elsewhere. What safeguards should they establish before deploying these technologies, and could starting with interpretable systems allow them to avoid some of the mistakes made by countries that adopted black-box AI before developing effective oversight?

Dr. Cynthia Rudin: Yes, definitely! Interpretability for high stakes situations is really important to establish right away before black boxes end up being the default. Sadly, I don’t have much hope given how much people are pushing black box models, and how little responsibility the black box model companies take if something goes wrong. Regulations would be helpful here, as well as a financial incentive for companies to ensure that their models don’t cause harm.

Another thing I keep hoping for is that governments or other organizations will develop big publicly available datasets that allow the public, including academic researchers, to solve some of these really big important problems like heart monitoring. That way, there’s no profit margin for companies wanting to sell black box models when they aren’t more accurate than publicly available interpretable models.

By Kobin Ferhad