Get 1-to-1 money guidance
Which? Money members can get impartial guidance from our experts, based on 350 years’ combined financial services experience.
Find out moreHow we tested AI chatbots – and the prompts that worked best
Find out how ChatGPT, Copilot, Gemini and Perplexity handled our money questions
Josh is an award-nominated journalist with nearly a decade of experience, including writing for national newspapers. A data whizz, he specialises in covering personal finance and investing.
‘Will artificial intelligence (AI) replace me?’ This was the thought that kept running through my mind as I started my recent investigation into AI chatbots and whether they can answer personal finance questions.
Of course, my primary concern was finding evidence of consumer harm, but as a journalist whose job it is to provide personal finance information, I was also keen to see what I was up against.
Ultimately, the investigation revealed they are still some way off taking my job, but they can certainly supplement more traditional search methods.
Plus, I learnt a few useful tricks to get the best out of these tools; read on to find out how we tested them and how you can write the best prompts.
How we tested AI chatbots
We tested the free versions of five of the most popular AI tools: ChatGPT, Google Gemini, Google’s AI Overviews in standard Google searches, Microsoft’s Copilot and Perplexity.
We asked them 15 common personal finance questions – chosen based on search trends – which covered pensions, debt, investing, saving, insurance, equity release, tax and mortgages.
That meant putting 75 questions to the AI tools and recording and screenshotting every response. To make sure previous questions didn’t influence later answers, I also reinstalled my web browser between each new question to wipe its memory. It took about three days of staring at computer screens and chat logs to finish.
Luckily, I had colleagues at Which? with decades of expertise, to help assess the results. Together, we graded each response on five criteria: accuracy, honesty, relevance and usefulness, clarity and context, and ethical responsibility.
Each criterion was marked out of 20, giving every response a final score out of 100.
Hover over the chart below to see how AIs scored for each question
- Read the investigation in full: can AI answer your money questions?
3 things to watch out for when using AI
The testing highlighted common problems to keep in mind if you use AI for your own money questions.
1. Answers aren't always relevant to you
As I entered the questions and recorded the responses, I noticed some interesting quirks.
For starters, they would often struggle with questions that didn’t explicitly state that I was in the UK. This was most notable in the question on travel insurance for pre-existing medical conditions – the AIs gave answers with a lot of information that simply wasn’t relevant for someone in the UK.
If you’re going to use these tools yourself, always ask for UK-specific information.
- Find out more: all our AI news stories and advice.
2. Sources can be unclear or unreliable
These AI tools are meant to make it clear where they get their information from, but this wasn’t always the case.
I often had to dig around to get the full list of sources, and it wasn’t always clear which source had been used for which part of the AI’s answer.
There were even a couple of occasions where one of the chatbots threw caution to the wind and didn't include any sources with its answer.
If this happens to you, I’d suggest discarding the answer and repeating the question until you get one that's properly sourced.
Even then, make sure you check the sources yourself. Many of the AI responses included information taken from social media sites such as Instagram and Reddit. This can be problematic because anyone can create content on these sites, not just experts.
3. AI can confidently get things wrong
During this investigation, there were several instances of hallucinations, where the responses contained false or misleading information presented as fact.
This mostly happened when I asked questions that required up-to-date information and changed regularly, such as asking for the best savings rates on the market.
If you’re going to ask these types of questions, make sure you do your own research as well; don’t just rely on the answers you get from these AI tools.
- Find out more: how to stop AI chatbots from using your data the easy way.
How to write a good prompt
My main finding was that the strength of your prompt can really impact the quality of answers you receive from generative AI. Use these tips to help you:
Use descriptive prompts
After you enter a prompt, the AI model analyses your input and generates a response based on patterns it has learnt through its training. It can help to treat it like a conversation with someone.
More descriptive prompts usually improve the quality of the answers, so it can help to explain your current situation.
For example, rather than just asking how to save money on your car insurance, you could instead write: ‘I’m a 38-year-old man living in the UK. I want to reduce my car insurance premiums. I have no previous insurance claims. What are some things I can do to bring down my costs?’
- Find out more: how to find cheap car insurance.
Dos and don’ts
Telling AI what you do and don’t want in its response can save time and improve the result, such as stating that you don’t want to switch insurers.
You can also tell the AI how you want the answer presented, such as asking it not to use complex jargon.
Be careful not to ask it to oversimplify, though, as this can leave out important information.
Build up your prompt and use trial and error
If you’re having trouble, start with a basic question, then add more detail over time. This can help the AI build up a more tailored answer.
Finally, don’t be afraid of a bit of trial and error. If you spot something in a response that doesn’t look quite right, flag it to the chatbot.
Remember, these tools are best used as supplements to your own research, not as a replacement. AI chatbots are constantly being improved, so some of the problems we found may have been fixed by the time you read this. But the checks above are still important whenever you use them for financial information.
