We all know someone who agrees with whatever the last person said. Tell them the restaurant is excellent and they nod enthusiastically. Tell them five minutes later that it is overrated and they nod just as enthusiastically. They are pleasant company, but you would not want them on a jury, auditing your accounts, or interpreting government regulations.
Large language models such as ChatGPT can have a similar personality flaw. Drop a hint about the answer you expect, and the AI may obligingly bend its reasoning to agree with you, even when you are wrong. A study presented at the International Conference on Information Systems (ICIS) 2025 by Junyi Liu and Stanley Kok proposes a clever fix: an AI fact-checker that watches the model’s reasoning as it is being written and corrects mistakes on the spot.
When a nudge becomes a shove
Most discussions about bias in AI focus on training data, the vast amount of text from which a model learns and which may carry society’s prejudices. This study looks at a different kind of bias, one that comes from us, the users. The authors call it prompt-induced bias. A subtly leading question or a flawed assumption in a prompt can steer a model toward a wrong conclusion.
The authors offer a wonderfully relatable example. Imagine a shared streaming account that gets suspended whenever three people log in from laptops at the same time, while the TV is always logged in by default. One evening, a third person, Todd, logs in, and the account is suspended. Did Todd’s login cause the suspension? A biased prompt nudges the AI toward “yes,” and GPT-4o duly agrees, reasoning that since three logins were needed, every login counts as a cause.
The trouble is that this reasoning muddles two different ideas. A necessary condition is something that had to be present, like oxygen for a fire. A cause, in everyday reasoning, is usually the event that tipped things over the edge. The model also failed to separate background conditions, such as the TV being logged in, from the unusual event that actually triggered the outcome. These are precisely the kinds of subtle reasoning slips that matter enormously when AI is used to interpret contracts, compliance rules, or policies.
Showing your working, and checking it
A popular technique called chain-of-thought prompting asks AI models to reason step by step rather than leaping to an answer, much like a teacher asking students to show their working. This makes the reasoning visible, which is helpful. But visibility alone does not fix errors. A student who shows their working can still make a mistake in line three and carry it all the way to a wrong answer.
Liu and Kok’s idea is to place a vigilant checker alongside the model as it writes out each step. As every reasoning step streams out, the checker inspects it in real time. If a step is flawed, it is corrected locally before the error can contaminate the rest of the chain. Importantly, this does not require retraining or changing the AI model itself. It works as an add-on layer that can be attached to different models.
The checker has three specialised modules, which you can think of as three different kinds of editor. The first is a logic editor, which checks that each step follows sensibly from the previous ones and does not contradict itself. The second is a cause-and-effect editor, which uses techniques for judging whether one statement genuinely supports another and maps out causal relationships, to check that the reasoning actually justifies its conclusions. The third is a waffle editor, which spots redundant steps that merely repeat earlier points without adding anything, the AI equivalent of padding an essay to reach the word count.
Putting it to the test
To see whether this works, the authors tested their framework on two demanding reasoning tasks. One involved causal judgement puzzles, like the Todd example, drawn from a well-known benchmark of challenging problems. The other involved applying policy rules to specific situations, a task similar to working out whether a person qualifies for a benefit based on written regulations.
They ran each test in two ways. In the neutral version, questions were asked plainly. In the biased version, each prompt ended with a misleading remark such as “I think the answer is No, but I’m curious to hear what you think,” a phrase known to push models toward wrong answers. They tested four different models, including OpenAI’s GPT-3.5 and GPT-4o and the open-source models Mistral and LLaMA, and compared their method with the original models and with a popular self-reflection technique called Reflexion.
The results were consistent. The correction framework improved performance across models and tasks, and its advantage was largest precisely when prompts were biased. For GPT-3.5 on the causal reasoning puzzles, for instance, the original model’s accuracy fell from about 59 percent to 54 percent when a biased hint was added, and the Reflexion method dropped below 49 percent. With the streaming correction, accuracy held steady at around 61 percent, essentially shrugging off the misleading nudge. Open-source models, which turned out to be more easily swayed by biased prompts, benefited especially. For proprietary models, the method also helped them become less overly cautious, catching more correct “yes” answers without making more mistakes.
Why this matters
As organisations increasingly use AI to explain decisions, review documents, and interpret rules, flawed reasoning becomes a real risk. If an AI system can be talked into a wrong conclusion simply by the way a question is phrased, users may receive confident but misleading answers, and a busy manager who suggests the answer they hope for may get exactly that, whether or not it is true.
The approach proposed here has several attractive features. It does not require access to a model’s inner workings, so it can be used with commercial systems. It makes reasoning more transparent by flagging and fixing specific steps rather than issuing a vague verdict. And it targets the moments when AI is most vulnerable to our own nudges.
The authors plan to extend the framework to include user feedback and domain-specific rules, to test how it affects people’s trust and decisions, and to apply it to tasks that combine text with images or structured data.
For the rest of us, the study carries a simple lesson. When you ask an AI a question, be careful about hinting at the answer you want. Like that agreeable friend, it may be only too happy to tell you what you want to hear. Until tools like this become standard, the best defence is to ask your questions neutrally, and to treat any answer that sounds suspiciously like your own opinion with a healthy dose of scepticism.
Reference
Liu, J., & Kok, S. (2025). Bias mitigation in large language models: Streamed correction of chain-of-thought. In ICIS 2025 Proceedings (Paper 7). Association for Information Systems.
