Samples and confidence
Why a measurement fires a batch of probes, what samples per run buys you, and how to read the confidence band on a Visibility Score.
Updated October 20, 2026
Ask a model the same question twice and you get two different answers. That is not a fault to engineer around; it is the thing being measured. Presence is a probability, and a probability needs probes.
Why a measurement is a batch
A single probe on a buying question is close to a coin flip. So each measurement fires a batch: several probes of the same prompt, whose results are combined into one point with one band.
That is also why the trend chart plots batches rather than individual answers. A chart of single probes is a chart of coin flips.
What samples per run buys
More probes per batch narrow the band. The returns diminish in a predictable way: one to three is a large improvement, three to ten is a modest one, and beyond that you are paying for decimal places.
Multiple samples per run are available on Agency and Enterprise. Below that a run is one probe, and a weekly cadence accumulates the same statistical power over time instead, which for most prompts is fine.
Reading the band
The band is the range the true rate is likely to sit in, given how many probes are behind it.
| What you see | What to do |
|---|---|
| Narrow band | Treat the number as real. |
| Wide band | Collect more before concluding anything. The chart is telling you it does not know yet. |
| A move inside the band | Nothing. It will come back. |
| A move outside the band | Something changed. Go and find out what. |
| Bands that overlap between two periods | Not a difference. Do not report it as one. |
That last row is the one that costs people credibility. Two percentages that differ by six points with overlapping bands are not a six-point gain.
Cadence against samples
Both increase your probe count and they are not equivalent.
- More samples per run narrows the band on today's measurement. Use it when you need a confident number now.
- A faster cadence gives you more points over time. Use it when you need to see movement sooner.
A prompt you are actively working wants both. A prompt you are monitoring wants neither.
The honest report
A client asking "where are we" wants one number. The honest answer is a number, a range and a probe count, and giving all three is what stops you reporting a gain in week three that evaporates in week four.
When a stakeholder cannot read a band, report the lower bound instead of the midpoint. "At least 31 percent" is defensible, understandable, and will not have to be walked back.
Questions about samples and confidence
- How is the band calculated?
- It is a Wilson interval on the probes behind the rate, which behaves sensibly at small sample sizes where a naive interval does not.
- Why is my band still wide after a month?
- Because width depends on probe count, not elapsed time. A monthly prompt with one sample has four probes after four months.
- What is a batch?
- One measurement of a prompt. The probes inside it are the individual answers, and the rate and band are computed across them.