Track AI Brand Mentions Over Time: Schedules and Drift

To track AI brand mentions over time, run the same buyer questions in the same assistants with the same settings on a schedule (monthly by hand, weekly with a script). Log every brand and cited domain per answer and compare runs prompt by prompt. Record the model and whether web search ran, to separate real change from updates and noise.

The first check tells you where you stand. This guide is about the second and later checks: what to hold constant, what causes drift, and how to tell a real change from noise. The comparison script at the end runs on the same CSV log as the free ChatGPT brand check.

Hold these fixed

If any of these change between runs, a change in your numbers may come from your setup, not the assistants.

Save the raw answers, not just your tallies, so you can re-score old runs if you refine the rules.

What causes drift

Some changes are real, some come from the assistant, and some are noise.

How often to run

Running more often doesn't make a single run more accurate. More answers per run does.

Compare two runs

This script compares two logs prompt by prompt (won, lost, up, down, same) and tests whether the overall mention rate changed by more than chance would explain:

import csv
import math
import sys
from collections import defaultdict

BRAND = "Northwind"


def load(path):
    """{(assistant, prompt_id): [True/False per run]} - did the answer name BRAND?"""
    out = defaultdict(list)
    for r in csv.DictReader(open(path, encoding="utf-8")):
        brands = [b.strip() for b in r["brands_in_order"].split(";")]
        out[(r["assistant"], r["prompt_id"])].append(BRAND in brands)
    return out


def two_proportion_p(k1, n1, k2, n2):
    """Two-sided p-value for 'the mention rate changed' (normal approximation)."""
    p = (k1 + k2) / (n1 + n2)
    se = math.sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
    if se == 0:
        return 1.0
    z = (k2 / n2 - k1 / n1) / se
    return math.erfc(abs(z) / math.sqrt(2))


before, after = load(sys.argv[1]), load(sys.argv[2])
print(f"{'prompt':<16}{'before':>8}{'after':>8}  change")
for key in sorted(set(before) | set(after)):
    b, a = before.get(key, []), after.get(key, [])
    if not b or not a:
        print(f"{'/'.join(key):<16}  only in one run: not comparable")
        continue
    rb, ra = sum(b) / len(b), sum(a) / len(a)
    change = "won" if rb == 0 and ra > 0 else "lost" if rb > 0 and ra == 0 else (
        "up" if ra > rb else "down" if ra < rb else "same")
    print(f"{'/'.join(key):<16}{sum(b)}/{len(b):<6}{sum(a)}/{len(a):<6}  {change}")

k1 = sum(sum(v) for v in before.values()); n1 = sum(len(v) for v in before.values())
k2 = sum(sum(v) for v in after.values()); n2 = sum(len(v) for v in after.values())
print(f"\nmention rate {k1}/{n1} = {k1 / n1:.0%} -> {k2}/{n2} = {k2 / n2:.0%}, "
      f"p = {two_proportion_p(k1, n1, k2, n2):.2f}")

python compare_runs.py log_2026-09.csv log_2026-10.csv on two small example logs (the brands are placeholders, not real measurements):

prompt            before   after  change
chatgpt/P1      1/3     3/3       up
chatgpt/P2      0/3     1/3       won
chatgpt/P3      3/3     2/3       down

mention rate 4/9 = 44% -> 6/9 = 67%, p = 0.34

The headline looks like a 23-point gain. With 9 answers per month, p = 0.34 says a change this size would turn up often by chance alone. Same rates with more answers, using the same test:

A rule of thumb from this: on 100 answers a run, a 20-point move is solid evidence; a 10-point move needs about 200 answers a run before it stands out. The prompt-by-prompt view is still useful at small sizes. It tells you where to look, such as which cited sources changed on a prompt you lost, even when the total isn't conclusive.

Pitfalls

• Comparing different prompt sets. If you added prompts, compare only the prompts present in both runs. The script marks the others as not comparable.
• Mixing searched and unsearched answers. If one month used search more often than the other, the rates move for that reason alone. Compare within each mode.
• Reacting to one prompt. One prompt going from 3/3 to 2/3 is one answer's difference.
• Forgetting branded prompts. Mention rate on branded prompts is always high. Keep them out of the totals and use them to track what the assistant says about you.

Related guides

• How to check if ChatGPT mentions your brand (free), which defines the log format.
• AI share of voice: measure it in ChatGPT and Perplexity.
• How AI assistants choose which brands and sources to mention.
• Measure AI share of voice in Python, for collecting the answers through the APIs.

Do it without code

The free LLM visibility query builder gives you a ready-to-run audit input from your brand, category and competitors, with a fixed example of the output. It doesn't call any AI model from your browser.

At scale

LLM Visibility Monitor on Apify is built for scheduled runs. It saves the generated question set so every run asks the same questions, compares each run with your previous run for the same brand, country and language (questions won, lost and unchanged), and records the model it requested and the model that actually answered on every row. You can schedule it weekly through Apify. It's free to start, then pay-as-you-go.

Disclosure: I build the datatooly query builder and the Apify actor. The comparison script works on any hand-collected log.

Sources: OpenAI: overview of OpenAI crawlers, OpenAI Help Center: Searching the web with ChatGPT, and the two-proportion z-test (computed in the script above).