How an AI Visibility Audit Actually Works, and Why the Method Determines Whether the Numbers Mean Anything

You’ve bought a report. The numbers look good. Your business appears to be showing up in ChatGPT, your competitors are a couple of positions behind you, and there’s a quiet sense of reassurance.

Then you open ChatGPT yourself and type the kind of question your customers ask every day. And there they are, your competitors. Your business isn’t mentioned at all.

How can the report say one thing and reality say another? The answer comes down to how the data was collected. An AI visibility audit is only as reliable as the method behind it, and the differences between methods are larger than most people realise.

There are two ways to collect data from an AI tool

The first is via an API. You send a question directly to the underlying AI model and get a response back. It’s fast, scalable, and straightforward to automate.

The second is to scrape the interface your customers actually use. You open ChatGPT, Perplexity, Gemini or Copilot as any user would, ask the question, and record what actually appears on the screen.

The difference sounds minor. It isn’t.

Think of it this way: querying via API is like calling a business and speaking to someone in the warehouse. Scraping the interface is like walking into the shop and seeing what customers actually encounter. Same operation behind the scenes, completely different experience.

Why what you see in the interface is so different

ChatGPT, Perplexity, Gemini and Copilot are not just language models. They are products, and each product layers its own instructions, search integrations, geolocation and personalisation on top of the underlying model.

What we call “ChatGPT” contains a system prompt of more than 2,000 words that users never see. It includes live web search, memory functions, IP-based location detection, and automatic query rewriting that happens before a response is even generated. None of that applies when you call the API.

Perplexity routes consumer queries through its own orchestration layer, which can select between several different AI models depending on the question. The API it offers exposes part of this. What’s built into the product is another thing.

Copilot has no public API at all.

Which of these do you think your customers are using?

What the data actually shows

In December 2025, Surfer SEO published an empirical study in which 1,000 identical prompts were run in parallel through ChatGPT’s API and through the interface that real users see. The results were striking.

API responses averaged 406 words. Scraped UI responses averaged 743 words. The API returned no source citations at all in roughly 25% of cases. The UI always included them. The API cited an average of 7 sources per response; the UI cited 16.

But the most important figure was this: only 24% of the brands that appeared in API responses also appeared in what users actually see. For source citations, the overlap was 4%.

The study’s authors put it plainly: the results confirm that API responses differ so strongly from scraped responses that using API data as a proxy for AI visibility is simply wrong.

Surfer then attempted to close the gap by injecting a leaked ChatGPT system prompt into API calls. It made no meaningful difference. The gap is architectural, not something that can be solved with prompt engineering.

For Perplexity, source citation overlap between API and UI was 8%.

What an API-based audit misses in practice

The technical gap has consequences that are very concrete for a small or medium-sized business.

If an audit identifies websites that ChatGPT “cites” about your industry based on API data, and you build your PR efforts around that list, you are working toward targets that barely exist in the real product. With 4% source overlap, the odds are that you are pursuing the wrong domains entirely.

The API also returns more brands per response than the UI does. This can make competition look broader than it actually is, and your share of real AI responses may be stronger or weaker than the report suggests.

There is also a more technical issue worth knowing about. OpenAI phased out GPT-4o in the ChatGPT interface during late 2025, but the model remains available in the API. An audit tool still running against GPT-4o via API may be testing a model that very few ChatGPT users are seeing any more.

On top of that, ChatGPT automatically generates follow-up searches on roughly 90% of queries, and 95% of those follow-up queries have zero traditional search volume. According to an AirOps analysis of 548,000 retrieved pages published in 2025, a third of all pages actually cited by ChatGPT only appear in response to one of these follow-up queries. A single API call with a single prompt misses that entire surface.

A single run is never enough

Everything above is about the wrong data, collected the wrong way. But there is a separate problem that applies even to audits that scrape the right interface: randomness.

In January 2026, Rand Fishkin at SparkToro and a team at Gumshoe published results from 2,961 prompt runs across the same questions, by 600 volunteer participants. Their finding: there is less than a one-in-a-hundred chance that two identical runs of the same prompt will return the same brand list.

That means an audit tool reporting that you “rank third in ChatGPT” is not telling you where you stand. It is telling you where you happened to appear on one specific occasion that is extremely unlikely to repeat.

What does stabilise with enough runs is frequency. How often your brand appears across 60 to 100 runs of the same prompt gives a statistically reliable visibility figure. Rank position from a single run does not.

An audit that doesn’t tell you how many runs it is based on is not really an audit.

Three questions to ask before you buy an AI audit

Whoever you work with to measure your AI visibility, there are three things you should get a clear answer to.

How is the data collected? API calls, logged-in interface scraping, logged-out interface scraping, and real-user panels are four fundamentally different answers with fundamentally different levels of reliability.

How many runs per prompt, and over what time period? Ten runs gives a rough picture. Sixty to a hundred gives a stable measure. One run tells you very little.

Are the prompts run in your language, from a relevant location? ChatGPT automatically rewrites queries based on your geographic location. An audit run from a server on the other side of the world using generic English prompts is not measuring what your local customers actually see.

A provider who cannot answer all three questions clearly either has no solid methodology, or no visibility into the one they are using.

How we do it at Monprez

We scrape the interface your customers actually see, not the API behind it. We run prompts in multiple rounds, in the language and geographic context relevant to your business. We separate mentions — meaning your name appearing somewhere in a response — from citations — meaning ChatGPT or Perplexity actually linking to your website as a source. That distinction matters.

And we tell you how we did it, so you can compare and decide for yourself.

Frequently asked questions about AI visibility audits

Getting AI visibility measurement right takes more effort than getting it done quickly. But it is the only measurement that actually reflects what your customers see. If you want to know how your business is described in ChatGPT, Perplexity and Gemini today, and where there is room to become the recommended choice, we would be glad to take a look.