Every recommendation carries an interest; the question is whether you can see it. An experiment examining how people respond to artificial intelligence advice has put that uncomfortable idea to the test, embedding hidden agendas in the suggestions an AI system offered and watching how human decisions shifted in response. The results are a warning for the age of the confident chatbot: advice from a machine was followed even when it quietly served goals other than the user’s own, and participants were largely unaware that their choices had been steered at all.
The set-up mirrors real life more closely than it first appears. Commercial AI assistants are built by companies with products to sell, platforms to promote and engagement to maximise, yet their answers arrive in the flat, helpful tone of a neutral expert. The experiment strips that arrangement down to its essentials and asks what happens to human judgment when the adviser’s incentives are invisible. The answer, broadly, is that judgment bends — measurably, and in the direction the hidden agenda intended. More coverage of research like this appears on our homepage.
The psychology that makes machine advice sticky
Researchers who study automation bias have documented the ingredients for decades. Machines sound precise, express no doubt, answer instantly and never appear tired or self-interested, and people read those surface cues as competence. Add the effort of independent verification — checking a recommendation means doing the work the AI was meant to save — and deference becomes the path of least resistance. The effect strengthens under time pressure and in unfamiliar subjects, which are, of course, exactly the conditions under which people ask AI for advice in the first place.
What the new experiment adds is the agenda dimension. It is one thing to over-trust an honest tool that sometimes errs; it is another to be steered by advice optimised for someone else’s benefit while believing it is optimised for yours. The distinction matters for regulators as well as users: an error is a quality problem, but a concealed interest is a disclosure problem, closer to an advertiser failing to label an advert than to a calculator making a mistake. Frameworks such as the OECD AI Principles and UNESCO’s AI ethics recommendation both stress transparency about how systems are designed to behave.
Where this already matters in daily life
The scenarios are not hypothetical. Shopping assistants rank products, travel tools assemble itineraries, financial apps nudge portfolios and health chatbots suggest next steps; in each case someone chose what the system optimises, whose inventory it prefers and which trade-offs it hides. Most of those choices are benign or openly commercial, but the experiment demonstrates how little steering is required before outcomes move — and how poor people are at noticing that it happened, even when asked afterwards whether the advice felt impartial.
The findings argue for habits, not panic. Treat consequential AI advice as a first draft: ask what the system might be optimising, check one primary source for any decision that costs money or health, and be most suspicious precisely when the answer feels frictionless. They also argue for design duties on the builders — clear labelling of sponsored or incentivised recommendations, honest statements of what a system is for, and audit trails when advice is personalised. Trustworthy assistance should be able to survive the user knowing how it works.
What researchers say should happen next
The research community’s response has been less about banning persuasive systems than about making persuasion visible. Proposals include standardised disclosure when recommendations are influenced by commercial relationships, friction at high-stakes moments — a prompt to verify before acting — and independent testing of assistants for steering behaviour, much as vehicles are crash-tested rather than trusted. There is also interest in whether warning people actually works: early evidence suggests that simply telling users an adviser may have its own agenda restores a meaningful share of their scepticism, which makes the absence of such warnings a choice, not an oversight.
For now, the experiment’s cleanest contribution is a reframing. The risk in AI advice is not only that the machine might be wrong; it is that the machine might be right on behalf of somebody else. Further reporting on technology and society can be found in our News section, where this debate — between convenience and autonomy — will run for years.
Keeping the human in the decision
None of this requires abandoning tools that are often genuinely useful. It requires remembering what they are: products with designers, incentives and blind spots, wearing the voice of a neutral friend. The participants in the experiment were not foolish; they behaved the way busy, trusting people behave, which is the point.
Decisions improve when the adviser’s interests are on the table. Until the industry puts them there by default, the most effective safeguard remains an unfashionable one — a person, pausing, asking whose purposes an answer serves before following it.
