This paper is part of a set of Research Notes which set out to inform public discussion and provide rigorous research on topics that relate to AI.
Read the Research Note (PDF)[2]
This note examines the potential usefulness and limitations of large language models (LLMs) such as OpenAI’s GPT series in consumer-facing financial services.
To explore practical applications, we conducted 2 pilot projects:
- Simplifying financial concepts: We asked GPT-3.5 and GPT-4 to generate simplified definitions of complex financial terms, tailored to lower reading ages and supported by relevant examples.
- Providing consumer guidance: We compared the effectiveness of LLM-generated responses in a fixed chatbot for cash savings queries with a traditional website-based Q&A format.
These pilots aimed to assess both the methodologies firms might use to evaluate consumer outcomes from LLMs and to deepen our understanding of this evolving technology.
The 3 main lessons we learned from running 2 pilots were:
- LLMs demonstrate strong potential in simplifying complex information, enhancing readability and accessibility. However, validating their outputs requires a robust evaluation framework that combines human judgement with automated tools.
- The effectiveness of LLMs is context-dependent. Outcomes such as user comprehension and engagement are influenced by how the model is embedded within the customer journey, including content design and delivery.
- There is a clear appetite for AI-driven assistance. Many users responded positively to automated support, indicating a readiness to engage with intelligent systems in decision-making processes.
Alongside this research, we have published an engagement paper, Proposal for AI Live Testing[4] – which outlines proposals for live AI model testing pilots. These initiatives are designed to support the safe and responsible deployment of AI in financial services and to promote positive outcomes for UK consumers and markets.
Authors
Shuaib Ahmed, Rhosyn Almond, Cameron Belton, Daniel Bogiatzis-Gibbons, Krishane Patel, Manasi Phadnis, Patrick Sholl, Jackie Spang.
Disclaimer
Research notes contribute to the work of the FCA by providing rigorous research results and stimulating debate. While they may not necessarily represent the position of the FCA, they are one source of evidence that the FCA may use while discharging its functions and to inform its views. The FCA endeavours to ensure that research outputs are correct, through checks including independent referee reports, but the nature of such research and choice of research methods is a matter for the authors using their expert judgement. To the extent that research notes contain any errors or omissions, they should be attributed to the individual authors, rather than to the FCA.