A Randomised Trial of AI Search: Fewer Clicks to Publishers, and Users Liked It Less

Almost everything written about AI search and publisher traffic is observational: traffic fell, AI Overviews appeared, therefore one caused the other. It is a reasonable inference and a weak piece of evidence, because the same period contained algorithm updates, a shift to mobile, changing query mixes and several years of platform behaviour designed to keep users on the platform.

A draft paper from researchers at the University of Pennsylvania and Northeastern University, published in August 2026, does something harder. Titled "AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence", it enrols 1,444 participants, tracks the actual search behaviour of 1,100 of them over seven days through a browser extension, surveys 956 afterwards, and randomises them into three conditions: forced regular Google Search, natural Google behaviour, and forced AI Mode.

What it found

Two results, and the second is the one most people will not have expected.

• People assigned to AI Mode were substantially less likely to click through to external websites. Removing AI Overviews had the opposite effect: clicks to websites increased.
• AI Mode generally made users feel worse about Google Search — lower trust, lower perceived usefulness — and significantly increased the share of users who searched on a competing engine such as Bing, DuckDuckGo or Yahoo. Removing AI Overviews produced no detectable decline in how users rated the experience.

The first finding confirms what publishers have been asserting, and now it is confirmed by a design in which the AI exposure was assigned rather than observed. The second undercuts the usual defence of the feature — that users prefer it — in the same experiment.

Why the design matters more than the numbers

Randomisation is the difference between a correlation and a cause. When a researcher assigns which interface a person sees, the people in each group are comparable in every other respect, so a difference in clicking behaviour cannot be explained away by different kinds of people searching differently. That is why this paper is harder to dismiss than a year of traffic charts, and why it is worth citing in a client conversation where a traffic decline is being treated as a content-quality problem.

Two caveats belong in the same breath. It is a draft paper, not yet through peer review. And it measures clicks and stated satisfaction — not revenue, not leads, not conversion quality. A world with fewer clicks and better-qualified clicks is a possible world, and nothing here rules it out.

Three things this should change in a marketing plan

1. Stop forecasting a recovery

If a feature reduces clicks by design, and its removal is the thing that increases them, then organic session volume for informational queries is not coming back through better content. Plans that carry a traffic-recovery line should replace it with something achievable: share of answers you appear in, branded search volume, direct traffic, and conversions per session.

2. Write for the click that a summary cannot satisfy

A summary can restate a definition, a list of steps or a comparison table. It cannot hand someone a calculator, a template, a dataset they can filter, a diagnostic, a price for their specific situation, or a conversation with somebody who has done the work. The content that still earns a visit is content where the value is in the doing rather than in the knowing — which is the same conclusion we reached about being cited by AI answers in the first place.

3. Take the competitor-engine finding seriously

The most actionable result in the paper is the least discussed: exposure to AI Mode pushed a measurable share of users toward other search engines. Whatever that says about Google, it says something plain about visibility strategy. If some portion of your audience is migrating, being present only where you have historically been strong is a risk, and checking your presence across the assistants and engines your customers actually use is cheap.

What it does not license you to do

It does not license quoting a percentage. The summary reporting of this paper does not carry effect sizes, and a figure invented to make a slide land is exactly the kind of claim that destroys a deck's credibility when someone checks. Cite the design, the direction and the sample; leave the magnitude to the paper.

It also does not license telling a client their content is fine. A randomised trial about interface effects says nothing about whether your pages deserve to rank. Both things are true at once: the surface takes clicks that used to be yours, and the majority of underperforming content was never good enough to win them.

The reporting change worth making this quarter

Most organic reporting still leads with sessions, because that is what the tools put first. If clicks are structurally falling for reasons no content programme controls, sessions as a headline metric makes good work look like failure and makes a shrinking market look like a team problem.

1. Lead with conversions and qualified enquiries from organic, not sessions.
2. Report impressions and appearance in AI surfaces separately from clicks, so presence and traffic are not confused with one another.
3. Track branded search and direct traffic as the demand signal that survives the click collapse.
4. Segment informational from commercial queries. They are being affected differently, and a blended chart hides which is which.
5. Record the date you changed the report, and keep the old view alongside it for a quarter. Changing a metric during a decline looks like hiding the decline unless both are visible.

The uncomfortable summary is that the evidence has caught up with the anecdote, and the evidence says the click is going away for a class of query regardless of how good your page is. That is a strategy problem rather than a content problem, and the honest response is to stop selling traffic as the outcome and start being paid for the demand that shows up by other routes.