AI Visibility Checker: What It Should Actually Measure

"Am I visible in AI search?" is an easy question to ask and a very easy one to answer badly.

Most AI visibility checkers work like this: you type a prompt, they send it to an engine, they search the answer for your brand name, and they give you a percentage. Every step in that chain has a defensible-looking bug in it. The result is a number that moves when nothing about your visibility has changed, and stays still when something has.

Here is what a check has to get right, and what we learned building one.

Being mentioned and being cited are not the same thing

This is the distinction that most checkers collapse, and it matters more than any other.

An AI answer can name you in its prose: "tools like Semrush and Ahrefs can do this". It can also cite you, listing your page among the sources underneath. These are different outcomes with different causes and different fixes. A mention means the model has absorbed your brand into its understanding of the category. A citation means a specific page of yours was retrieved and judged worth linking.

You can have either without the other, and you want both.

Now the bug. The AI Overview text an API returns is markdown, and it embeds its sources inline as links — [some anchor text](https://semrush.com/blog/...). If your checker searches that raw text for "semrush", it finds a match inside the URL and reports a brand mention. Semrush was not named in that sentence. It was linked.

So the naive check counts citations as mentions, inflates every brand that gets linked, and cannot tell the two apart at all. The fix is to strip link targets before counting prose mentions, and to count citations separately from the answer's own sources list, where they can be counted exactly rather than inferred:

[anchor](https://example.com/page)  ->  [anchor]

Anchor text stays, because if the answer wrote your brand name as the visible link text, that is a real mention. The target goes.

If you are evaluating a tool, this is the single question worth asking it: does your mention count include brand names that only appear inside source URLs? You will learn a lot from how quickly it can answer.

"No answer" is not the same as "an answer without you"

For any given prompt there are three outcomes, not two:

1. An AI answer appeared and named or cited you.
2. An AI answer appeared and did not.
3. No AI answer appeared at all.

The third is common and it is not a failure state — plenty of queries still return ten blue links. A checker that reports "0% visibility" without separating these tells you that you lost a game that was never played. Worse, it makes your score depend on how often the engine chose to generate an answer that day, which is not something you control or should be graded on.

There is a fourth state too, and it is the one that quietly corrupts data: the lookup itself failed. A timeout, a rate limit, a malformed response. That must never be recorded as a zero. A failed lookup and a confirmed absence look identical in a database if you let them, and once stored, no later analysis can separate them.

Two surfaces, not one

"AI search" is not a single place. Google alone runs AI Overviews and AI Mode, and they answer the same prompt differently, from different retrieval, with different citation behaviour. Checking one and reporting "your AI visibility" is measuring a third of the room.

The chat assistants — ChatGPT, Perplexity, Claude — are a separate problem again. They cost several times more per prompt to measure, and in the reference data we worked from they accounted for a small minority of raw brand mentions compared to Google's surfaces. That is a reason to sequence them second, not to pretend they do not exist. Any checker that covers one surface should say which one.

The prompts decide the answer

This is where most reports become fiction.

If a tool lets you type in the prompts, you will type in prompts you hope to win. Everyone does. You will pick phrasings that sound like your positioning, and the report will reflect your positioning back at you.

The honest source is the set of queries your site already appears for — real queries, from Search Console, not invented ones. And they should be ranked by impressions, not by clicks.

That ordering is deliberate and it is worth understanding. A query with impressions and no clicks is exactly the shape of a query where an AI answer may be absorbing the traffic: people are searching, Google is showing you, and nobody is arriving. Rank by clicks instead and you surface the queries that are already working — the ones least likely to have a problem.

If you have no Search Console data, a checker has to fall back to something, and it should tell you it did. A run built from keyword-tool suggestions is a legitimate answer to a different question, and silently substituting one for the other makes two incomparable reports look like a trend.

The useful output is not your score

A single visibility percentage is the least actionable number in the report. It tells you there is a problem, which you probably suspected, and nothing about what to do.

What you can act on is who got cited instead, at page level. Not "competitor.com" — the exact URL. A citation gap that names the competing page tells you what to go and read: how long it is, how it is structured, what question it answers in its first paragraph, whether it has a table where you have prose. A domain name tells you the name of a company you already knew about.

Collect the URLs, not just the domains. If you build this yourself, capture them from the first response, because you cannot recover them later without paying for the query again.

What a check still cannot tell you

Honesty about the limits matters as much as the measurement:

• It is a snapshot. Generative answers vary. One run is a data point, not a verdict, and the trend is the thing worth watching.
• It cannot tell you why. No API returns the retrieval reasoning. You get the outcome and the sources; the causation is inference.
• It does not measure traffic. Being cited in an AI answer is not a visit. Treat it as a leading indicator, not a revenue number.

How to check yours

You can do a rough version by hand today, free: take your top twenty Search Console queries by impressions, run each one, and record three things per query — was there an AI answer, were you named in the prose, were you cited in the sources. Twenty queries is about an hour and it will tell you more than most dashboards.

If you want the automated version, our AI Visibility tool does exactly the above across both Google surfaces, pulling its prompts from your connected Search Console account and storing each run so the trend is readable. It is on the Agency plan, and a run costs three analysis credits.

And if what you actually want to know is whether AI crawlers can read your pages at all — a different and more fundamental question — the AI Search Readiness Audit is free, needs no signup, and runs on any URL.