Methodology

How nthrun measures AI search visibility, in enough detail to check our work.

Why AI answers vary

Large language models sample their output. Ask the same engine the same question twice and you can get two different answers, with different brands in them. That is not a flaw in the measurement, it is the thing being measured: your real visibility is the rate at which you appear across many answers, not whether you appeared in one.

We never pin temperature or seed

Every query goes to the engine's standard API with default sampling. We deliberately do not set temperature, seed, top_p, or top_k, and we do not enable any caching. Pinning those parameters would make our numbers more repeatable and less true: your customers ask with defaults, so we measure with defaults.

Repeated runs

Each prompt is run between 1 and 20 times per engine (you choose; more runs give a tighter range). The engines under test are ChatGPT (OpenAI), Perplexity, Gemini (Google), and Claude (Anthropic), each queried through its own official API.

How a mention is counted

Every response is checked in two independent ways, and counts as a mention if either one hits:

• Direct text match. A case-insensitive, word-boundary-aware search for your brand name and any aliases you configured, tolerant of simple possessive and plural forms ("Acme's", "Acmes").
• Entity extraction. A separate model call lists every company, brand, product, or platform named in the response, in order of first appearance. This extraction always runs on the same model (gpt-4o-mini) no matter which engine produced the response: the extractor is the measurement instrument, and keeping it constant is what makes results comparable across engines.

The extracted entity list also gives us your brand's position in the answer (was it named first or fifth?) and the competitor table (which other brands the engine reaches for, and how often).

Mention rate and the range

The mention rate is the plain count: mentioned in 7 of 10 runs is 70%. Around it we report a 95% Wilson score confidence interval, computed from the run count. With few runs the interval is wide, and we show it that way on purpose: a 4-run check genuinely tells you less than a 20-run check, and the range is what says so. We never collapse the range into a single confident-looking score.

Full transcripts, always

Every run stores the engine's complete response, verbatim and untruncated, and every number on a report links back to the runs it was computed from. If you doubt a mention, read the answer yourself.

When a run fails

If a run fails for a technical reason (a provider error, a timeout), it never enters the statistics: only runs that produced an answer are counted, so a failure is not mistaken for a non-mention. Credits for runs that never completed are refunded automatically. A completed run that found no mention is a real, paid-for result.