Every AI Benchmark Has an Observer

1. The score is a reading, not the model

A thermometer reading is not the weather. A speedometer reading is not the car. In the same way, a benchmark score is not the model. It is a reading taken under specified conditions.

That distinction disappears easily because scores are compact. A table turns thousands of interactions into one decimal place. A leaderboard arranges uncertain, multidimensional systems into a clean vertical order. The compression is useful. It is also dangerous. Once the measurement history has been removed, the number starts to look intrinsic: the model has an accuracy, has a reasoning level, is number four.

The more defensible statement is relational:

The measurement relation can be written as

$$\begin{aligned} \mathcal O_t &\coloneqq (D,I,P,\theta,J,\mu,A), \\ \hat y_i &\coloneqq \operatorname{Run}(M_t,x_i\mid I,P,\theta), \\ r_i &\coloneqq \operatorname{Score}_{J,\mu}(\hat y_i,c_i), \\ s_{\mathrm{obs}}(M_t;\mathcal O_t) &\coloneqq A(r_1,\ldots,r_n), \qquad (x_i,c_i)\in D. \end{aligned}$$

Here $M_t$ is the tested model or application snapshot; $D$ is the task sample; $I$ is the system boundary and interface; $P$ is the prompt and interaction protocol; $\theta$ contains runtime and inference settings; $J$ is the judge; $\mu$ is the scoring metric; $A$ is the aggregation rule; and $c_i$ is the reference, rubric, or decision context for task $i$.

This is not an argument that measurement is impossible. It is an argument for saying what was measured. A model evaluated through a fixed multiple-choice template at temperature zero is not the same tested system as that model operating through a conversational interface with tools, memory, retries, and a human reviewer. They may share weights, but the measurement boundary differs.

Dow and colleagues make this design space explicit through dimensions such as evaluation setting, task type, input source, interaction style, duration, metric type, and scoring method. The point is not that every evaluation must become enormous. It is that every evaluation has already made these choices, whether it records them or not.

We will call the combined set of choices the observer. The observer is not merely the person reading the result. It is the whole apparatus that converts behavior into evidence: the construct, protocol, data, prompts, runtime, scorers, rules, and reporting decisions. Every benchmark has one. A benchmark without a declared observer does not become neutral; its assumptions simply become harder to inspect.

2. What the observer contains

The observer can be decomposed into seven connected layers.

2.1 The construct

First comes the thing the evaluation claims to measure: factual recall, mathematical reasoning, instruction following, usefulness, safety, robustness, or something else. These labels are not raw observations. They are theoretical constructs. The benchmark operationalizes them through observable tasks and scoring rules.

Jacobs and Wallach describe how a construct can become mismatched with its operationalization. Their analysis focuses on measurement and fairness, but the general lesson travels: reliability and validity do not arise merely because a number is computed consistently. We must ask whether the observable procedure represents the concept named in the conclusion.

If a benchmark calls itself a test of reasoning but awards credit only for final-answer agreement, it may detect some combination of reasoning, memorized patterns, guessing, parsing the question, and exploiting artifacts. The score can still be useful. The construct claim must remain narrower than the label suggests unless additional evidence separates those mechanisms.

2.2 The system boundary

Second comes the unit under test. Is it a frozen base model? A chat model with a system prompt? A retrieval-augmented application? An agent with tools? A production workflow with human escalation? Each boundary supports a different statement.

A base-model benchmark can help compare model snapshots under controlled conditions. It cannot, by itself, establish the reliability of the final application. Conversely, an end-to-end workflow evaluation can establish performance for that configured system, but it may not isolate which component caused the result.

The boundary should be named before the score is interpreted. Otherwise the capability of an application is casually attributed to a model, or a model-level result is treated as proof that an application will work.

2.3 The input world

Third comes the test population: which tasks exist, which examples represent them, how those examples were collected, and what was excluded. A benchmark is a miniature world. Its task mixture declares what matters, and its sampling process determines what can be seen.

Dehghani and colleagues use the phrase benchmark lottery for the fragility created when factors other than fundamental algorithmic superiority affect which method appears superior. Their experiments show that relative performance can change when the benchmark tasks change. The lesson is not that rankings never contain information. It is that a ranking is conditional on the selected world.

Dataset construction can also affect apparent generalization. In their analysis of ImageNet-v2, Engstrom and colleagues identify statistical bias introduced by standard dataset-replication procedures. After remeasuring selection frequencies and correcting for the bias they studied, they estimate a bias-corrected accuracy gap of $(3.6 \pm 1.5)$ percentage points, versus the original $(11.7 \pm 1.0)$-point drop. This is a bounded result about that replication analysis, not a claim that distribution shift is unimportant. It shows why even a careful new test set is part of the measurement apparatus rather than a transparent window onto a model.

2.4 The interface and prompt

Fourth comes the representation of the task to the model. Instruction wording, answer ordering, examples, delimiters, tool descriptions, output schemas, and conversation history can all alter behavior. A prompt is not decorative metadata around the test. It is an experimental condition.

Polo and colleagues study evaluation across many prompt variants and introduce PromptEval to estimate a distribution of performance rather than relying on one template. This reframes prompt sensitivity. Instead of asking which single prompt is the real one, an evaluator can ask how the system performs across a defensible family of prompts, then report a median, quantile, or spread appropriate to the decision.

For deployed software, the relevant family may be the range of instructions real users produce. For a controlled research comparison, the family may be a set of semantically equivalent templates. Either way, the prompt belongs in the evidence record.

2.5 The runtime conditions

Fifth come the inference settings and surrounding machinery: model version, provider endpoint, temperature, top-p, seed when available, maximum tokens, retry policy, context truncation, retrieval index, tool versions, and safety filters. These conditions determine which behavior is sampled and which outputs are allowed to reach the judge.

A score from a mutable hosted endpoint is also time-indexed. If the provider changes the model behind an alias, a later reproduction may be measuring another snapshot. Recording the date is therefore not clerical decoration; it is part of identifying the tested system.

2.6 The judge and scoring rule

Sixth comes the conversion from outputs into credit. Exact match, unit tests, human preference, rubric scoring, and model judging each observe different properties and introduce different error modes.

Zheng and colleagues examine strong language models used as judges for open-ended responses. Alongside useful agreement results, they identify limitations including position, verbosity, self-enhancement, and reasoning biases. An automated judge can make evaluation scalable. It does not disappear from the experiment. Its identity, prompt, order randomization, rubric, and calibration belong beside the resulting score.

Human evaluation is not an observer-free escape hatch either. Rater recruitment, instructions, expertise, compensation, blinding, disagreement handling, and aggregation all shape the result. Human judgment can be the right instrument, but it remains an instrument.

2.7 The aggregation and report

Seventh comes the final compression. Mean or median? Macro or micro average? Best of several attempts or first attempt? Per-task weighting or pooled examples? Confidence interval or bare point estimate? Failed requests excluded or counted? One number can be produced from many defensible rules.

The report then decides what survives. If only the aggregate appears, a model that is excellent on frequent low-cost cases and unsafe on a rare high-cost case can look strong. Liang and colleagues demonstrate a multi-scenario, multi-metric approach in HELM, including metrics beyond accuracy. The broader principle is that a scorecard should preserve trade-offs the decision maker actually needs.

3. Why rankings move without anyone cheating

People often explain a changed ranking through one of two stories: the model improved, or the benchmark was manipulated. There is a large ordinary space between them. Rankings can move because the observer changed.

Select a different mixture of tasks and the models' comparative strengths receive different weights. Reword prompts and one model may be more stable than another. Replace exact match with a semantic judge and partially correct answers may receive credit. Change the judge and stylistic preferences can enter the score. Update the test set and the sampled population shifts. Aggregate by task instead of by example and small categories gain weight.

None of these changes is automatically improper. Some are improvements. The error occurs when the revised score is narrated as though nothing but the model changed.

This is also why benchmark saturation creates pressure. When many systems approach the ceiling, small artifacts can decide the order. Bowman and Dahl argue that strong benchmark design requires attention to properties including validity, reliability, size, annotation, and social bias. If a benchmark no longer separates the systems in a way aligned with the intended construct, another decimal place does not restore meaning.

Contamination adds a distinct problem. Sainz and colleagues describe levels of benchmark contamination and the risk that exposure to test data can overestimate performance. For many large models, the full training corpus is not available to the evaluator, so contamination is difficult to rule out completely. A responsible report should distinguish what was checked, what was inferred, and what remains unknown. A high public-benchmark score may still be informative, but it should not be treated as equivalent to performance on a fresh private sample drawn from the intended deployment.

4. What a benchmark score can legitimately support

The existence of an observer does not make scores arbitrary. It defines their jurisdiction.

A well-documented score can support claims such as:

• under protocol P, model snapshot A outperformed model snapshot B on task sample D by metric M;
• the tested application met a specified threshold on a representative acceptance set;
• performance was stable across a declared family of prompt variants;
• a model exhibited a particular failure under a behavioral test;
• one configuration offered a better measured trade-off among quality, latency, and cost for a stated workflow.

The same score usually cannot establish that a model is generally more intelligent, will be more useful for every user, is safe in untested contexts, or will preserve the result after its surrounding system changes.

This is the difference between skepticism and nihilism. Nihilism says no measurement means anything. Disciplined skepticism says meaning comes from the relationship between the claim and the instrument.

Wallach and colleagues frame generative-system evaluation as a measurement challenge and propose using lenses from social-science measurement to interrogate validity. That framing helps because a benchmark is not merely a bag of questions. It is an instrument connecting a concept to observations. The evaluator's task is to make that connection inspectable.

5. Replace the naked score with an observer card

The smallest practical improvement is to publish an observer card beside every important result. It need not be long. It must be sufficient to reconstruct what the number refers to.

Claim

State the narrow conclusion before presenting the score. Replace “Model A is better” with “Model A produced a higher first-attempt rubric score on this support-resolution set under the following configuration.”

System identity

Record the model and application versions, provider, endpoint or snapshot identifier, evaluation date, tool and retrieval versions, and any mutable aliases.

Task population

Record the dataset name and version, split, sample size, selection procedure, exclusions, provenance boundaries, and a hash or immutable identifier where possible. Describe how the sample relates to the target use.

Interaction protocol

Publish the system and user prompt templates, examples, answer format, number of turns, tool access, memory state, retry policy, and any human intervention. If there are several prompt variants, state how they were selected or sampled.

Runtime

Record temperature, top-p, maximum output length, seed support, repetitions, timeout handling, truncation, and failure policy.

Scoring

Name the metric, judge, judge version, rubric, rater instructions, blinding and randomization procedures, disagreement handling, and known judge limitations.

Aggregation and uncertainty

State weighting, aggregation, missing-output treatment, variance across runs or prompts, and confidence intervals where justified. Preserve per-task or per-risk slices needed for the decision.

Decision boundary

Finally, say what decision the result is allowed to inform. A research leaderboard, a procurement shortlist, a production launch, and a high-risk release require different evidence.

The observer card prevents a benchmark result from becoming an orphaned number. It also makes disagreements more productive. Instead of arguing whether the score is true, participants can ask which layer of the observer should change and why.

6. Evaluate behavior, not only averages

Aggregate accuracy answers “how often did the tested outputs satisfy this rule across this sample?” It does not reveal every behavior that matters.

Ribeiro and colleagues introduce CheckList as a methodology for behavioral testing of NLP models. The software-testing analogy is useful: a product team does not accept one test-suite pass rate as a complete account of system behavior. It writes targeted tests for invariances, directional expectations, boundaries, and known failure modes.

For an AI workflow, behavioral tests might ask:

• Does irrelevant demographic information change the decision?
• Does the system preserve an answer when wording changes but meaning does not?
• Does it refuse requests that cross a declared boundary?
• Does it recover after a tool fails?
• Does it cite the retrieved source rather than an unsupported memory?
• Does it escalate a high-cost ambiguity to a human?

These tests may not produce a glamorous single ranking. They often produce better engineering information. A failure names a condition that can be reproduced, assigned, corrected, and tested again.

7. The seven-question benchmark reading protocol

Before accepting, repeating, or purchasing on the basis of a model score, ask seven questions.

1. What exact construct is being claimed?

Translate the headline noun into an observable claim. If the report says “reasoning,” what behavior counts as evidence of reasoning, and what alternatives could also produce credit? If it says “helpfulness,” helpful to whom, for which task, under which constraints?

2. What system boundary was tested?

Identify whether the score belongs to a model snapshot, a prompted chat system, an application with retrieval and tools, or a human-machine workflow. Do not transfer the result across boundaries without additional evidence.

3. Which inputs and sampling process define the test?

Inspect task composition, example provenance, exclusions, freshness, contamination risk, subgroup coverage, and relation to the deployment population. Ask which important cases cannot appear in the sample.

4. Which prompts, interactions, and runtime settings were used?

Treat prompt templates, demonstrations, answer orders, tools, retries, temperature, context management, and versions as experimental conditions. Where plausible alternatives exist, look for stability rather than one optimized prompt.

5. Who or what scored the outputs, and by what rule?

Inspect exact-match logic, tests, rubric, model judge, human raters, ordering, blinding, calibration, and disagreement. Ask which qualities the scoring rule cannot observe.

6. How stable is the result under defensible alternatives?

Look for repeated runs, prompt distributions, alternate judges, resampled tasks, uncertainty intervals, and per-slice results. Stability does not prove universal validity, but fragility limits the claim.

7. What decision is this score supposed to support?

A number is useful only relative to a decision. State the threshold, cost of error, reversibility, monitoring plan, and evidence still required. The best model on an academic average may not be the best system for a constrained workflow.

If a benchmark report cannot answer all seven questions, that does not make it worthless. It tells you how much interpretive load remains on the reader.

8. Worked example: choosing a support assistant

Imagine a company choosing between two language-model systems for customer support. A public leaderboard gives System A a higher general score. The tempting decision is to select A and move on.

The seven-question protocol changes the evaluation.

The construct is not general intelligence. It is successful support resolution under company policy. The system boundary includes retrieval over current documentation, a ticketing integration, a refusal layer, latency limits, and human escalation. The input world should be a privacy-reviewed sample of actual issue types, including rare account-security and billing cases. The prompt family should include short messages, disorganized narratives, non-native phrasing, follow-up turns, and contradictory information. Runtime evaluation should include retrieval misses and tool timeouts.

The judge should not be one undifferentiated preference score. Deterministic checks can verify policy citations, required fields, and tool outcomes. Trained reviewers can assess whether the answer solves the problem without inventing facts. High-cost failures should be reported separately from ordinary dissatisfaction. Cost, latency, escalation rate, and recovery after failure belong beside response quality.

Suppose A remains slightly better on average, while B is more reliable on account-recovery cases and far cheaper. The right decision now depends on traffic composition, failure cost, and routing design. The company might choose B, choose A with a stricter security escalation, or route cases between them. The benchmark has not failed. It has become specific enough to support engineering.

9. Limitations and boundary of this note

This note does not claim that all benchmarks are equally fragile, that rankings are meaningless, or that a declared observer removes every bias. It is not an empirical meta-analysis of benchmark instability. It does not provide a universal method for proving intelligence, validity, safety, or deployment fitness.

The observer vocabulary is a measurement metaphor, not a claim about quantum observation or consciousness. Models need not be aware of being tested for the apparatus to affect the recorded result.

The cited studies cover different domains, periods, and evaluation regimes. Results about ImageNet replication do not automatically quantify language-model distribution shift. Results about prompt variation do not determine the stability of every prompt or model. Findings about particular model judges do not establish that every automated judge fails in the same way. Each source should be read within its own experimental boundary.

Finally, documentation does not replace testing. An observer card can make a weak evaluation legible; it cannot make the underlying sample representative or the construct valid. The objective is not more paperwork. It is to expose the choices that control the meaning of the score, then invest evaluation effort where the decision carries risk.

10. Frequently asked questions

Are benchmarks useless?

No. Benchmarks enable repeatable comparison, reveal progress under shared protocols, and reduce the cost of testing. They become misleading when a conditional result is presented as an intrinsic property or when the protocol no longer matches the decision.

Is the observer just another word for bias?

No. Some observer choices introduce bias; others define the measurement correctly. A metric must choose what counts. A dataset must sample something. A judge must apply a rule. The goal is not to eliminate all choices, which is impossible, but to justify and expose them.

Can one number ever be useful?

Yes, when the construct is narrow, the protocol is stable, the aggregation matches the decision, and important failure modes are reported elsewhere. A single number is a useful index, not a complete model of the system.

If everyone uses the same standardized benchmark, is the problem solved?

Standardization improves comparability. It does not prove that the benchmark represents the desired construct or deployment population. It can also intensify optimization toward a familiar test. Standardized scores are strongest when combined with fresh, decision-specific evidence.

Does human evaluation remove the observer?

No. Human evaluation changes the observer. It may capture qualities a deterministic metric misses, but it introduces rater populations, instructions, expertise, context, incentives, disagreement, and aggregation choices that should be recorded.

How can a small team compare models without building a research lab?

Start with the decision. Build a small representative acceptance set, add targeted tests for expensive failures, keep prompts and runtime fixed for the first comparison, repeat ambiguous cases, and record the observer card. Use public benchmarks to form a shortlist, not to outsource the final decision.

Related research

The measurement problem continues at the point where models become agents. AI Agents Need a Runtime, Not Another Prompt examines how generated possibilities acquire—or fail to acquire—operational authority across memory, tools, workflows, and action.

References

1. P. Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alexandra Chouldechova, and Hanna Wallach. 2024. Dimensions of Generative AI Evaluation Design.
2. Hanna Wallach et al. 2025. Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge. Proceedings of the 42nd International Conference on Machine Learning.
3. Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021. The Benchmark Lottery.
4. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, and Aleksander Madry. 2020. Identifying Statistical Bias in Dataset Replication. Proceedings of the 37th International Conference on Machine Learning.
5. Lianmin Zheng et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track.
6. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP 2023.
7. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL 2020.
8. Percy Liang et al. 2022. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023.
9. Samuel R. Bowman and George Dahl. 2021. What Will it Take to Fix Benchmarking in Natural Language Understanding?. NAACL 2021.
10. Abigail Z. Jacobs and Hanna Wallach. 2021. Measurement and Fairness. ACM FAccT 2021.
11. Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. 2024. Efficient multi-prompt evaluation of LLMs. NeurIPS 2024.

Version and status ledger

• Publication: Nillow Research Note 001
• Version: 1.1
• Status: Public research note
• Scope: Practical interpretation and documentation of AI benchmark measurements
• Revision policy: Material corrections or conceptual changes increment the version and remain visible in the article record