As part of our AI research series, we explore the relative effectiveness of different methods for explaining the outputs of AI to consumers in the context of the use of determining consumers’ creditworthiness.
Read the research note (PDF)[2]
Read the research note annex (PDF)[3]
We tested whether participants were able to identify errors caused either by incorrect data used by a credit scoring algorithm or by flaws in the algorithm’s decision logic itself.
The method of explaining algorithm-assisted decisions significantly impacted participants’ ability to judge these decisions. However, we found that the impact of the explanations we tested varied depending on the type of error, in ways that were not anticipated. For example, while simply providing an overview of the data that was available to the algorithm impaired participants ability to identify errors in data input, it helped participants challenge errors in the algorithm’s decision logic, such as the algorithm failing to use a relevant piece of information about the consumer. Surprisingly, it helped to do so more than explanations which focussed more directly on the decision logic itself.
We propose two hypotheses to explain the inconsistent effects of our explanation genres. First, additional information may make it more difficult to spot errors because there is simply more information to review. Second, additional information about the algorithm’s decision logic may encourage participants to focus on whether this decision logic was followed rather than if the decision logic was sound.
We found that providing additional information about the inner workings of the algorithm was well received by consumers and gave them greater confidence in their ability to disagree with the algorithm’s decisions. However, the research finds that more information may not always be helpful for decision-making and could lead to worse outcomes for consumers by impairing their ability to challenge errors. When and where this is true clearly depends on the specific context.
These findings reiterate the value of testing accompanying materials that may be provided to consumers when explaining AI, ML and/or algorithmic decision-making. Our findings also underscore the importance of testing consumers’ decision-making within the relevant context, rather than relying solely on self-reported attitudes.
Future research could look to explore how we can best explain AI assisted decisions in other contexts within financial services, the specific mechanisms for how explainability methods may impact consumers, alternative ways of presenting explanation genres, and the broader consumer journey beyond recognising errors.
Our AI research series
We want to enable the safe and responsible use of AI in UK financial markets, driving growth, competitiveness and innovation in the sector. As part of our effort to deepen our understanding of AI and its potential impact on financial services, our team is undertaking research on transparency and the potential role of AI explainability models.
This is part of a series of research notes on how AI intersects with financial services to spark discussion on these issues, drawing on a variety of academic and regulatory perspectives. We hope that these notes are of interest to those who build models, financial firms, and consumer groups in understanding complex debates on building and implementing AI systems.
Authors
Cameron Belton, Daniel Bogiatzis-Gibbons, Isaac Keeley, Jackie Spang, Cagatay Turkay, Yulu Pi.
Disclaimer
Research notes contribute to the work of the FCA by providing rigorous research results and stimulating debate. While they may not necessarily represent the position of the FCA, they are one source of evidence that the FCA may use while discharging its functions and to inform its views. The FCA endeavours to ensure that research outputs are correct, through checks including independent referee reports, but the nature of such research and choice of research methods is a matter for the authors using their expert judgement. To the extent that research notes contain any errors or omissions, they should be attributed to the individual authors, rather than to the FCA.