# AI Benchmarks: Testing Models and Why Leaderboards Lie

URL: https://interesting.weblocally.com/en/post/ai-benchmarks-testing-models-and-why-leaderboards-lie
Date: 2026-10-04
Updated: 2026-10-07
Sections: Technology
Source: MMLU / SWE-bench / LMArena, https://arxiv.org/abs/2009.03300
Russian version: https://interesting.weblocally.com/post/ai-benchmarks-testing-models-and-why-leaderboards-lie

> A breakdown of what AI benchmark scores really mean, where leaderboards fall short, and how 20 key modern benchmarks work.

## The Core Idea

A benchmark is not an exam for general intelligence. It is a **predetermined measurement procedure**: specific tasks, a concrete dataset version, a prompt format, a tooling harness, a time or cost budget, and a scoring rule. Its result answers a narrow question: _how well did this system, under these conditions, perform on this operationalized task?_

Because of this, the phrase “the model scored 90%” means almost nothing without the name of the test, its version, the execution protocol, and the date. It might be a strong signal for one workflow and completely irrelevant for another.

## What a “Score” Is Really Made Of

It helps to break down any result mentally like this:

> **result = model × test tasks × protocol × harness × metric × date**

Change any single factor, and an apples-to-apples comparison falls apart. The exact same base model can produce radically different results with a different system prompt, number of attempts, access to search or a terminal, generation temperature, judge model, agent loop, or even a silent API update under the same commercial name.

This does not make benchmarks useless. On the contrary, standardization is the only way to track progress and regression. But you have to standardize the **entire procedure**, not just the name of the model.

## Three Types of Tests That Are Frequently Confused

Type | What it asks | Examples | What it shows well | What it misses
Closed-ended tasks | “Will it pick the right answer?” | MMLU, GPQA | Breadth of knowledge, structured problem-solving | Real-world ambiguity, cost of failure
Executable tasks | “Will the result work in the environment?” | HumanEval, SWE-bench | Functional code correctness, working with repositories | Product requirements, security, maintainability
Human evaluation | “Which answer do users prefer?” | Chatbot Arena | Helpfulness and style in live dialogue | Audience representativeness, long-horizon tasks, objective truth

None of these types is inherently superior. When picking a coding assistant, reproducible code execution matters most; for a support bot, real dialogues and safety checks take precedence; for scientific research, source quality, confidence calibration, and human expertise are essential.

## Four Reasons Leaderboards Can Be Misleading

### 1. Data Contamination

If benchmark problems or closely related solutions leak into pretraining, fine-tuning, or system prompts, a model may simply recognize them instead of generalizing. With proprietary models, this is difficult to verify because training corpora are rarely disclosed in full. Researchers rely on paraphrasing questions or probing for memorized text fragments, but these are indirect methods.

Contamination does not necessarily scramble the rankings across the board, but it makes absolute scores unreliable. Defenses include fresh private splits, access-controlled task repositories, task variations, data-trace audits, and transparency about what went into training.

### 2. Optimizing for the Test Instead of the Skill

When the market fixates on a single number, teams naturally optimize for that number: engineering prompts, fine-tuning harnesses, enabling multiple retries, adding output filters, or injecting few-shot examples. This is not inherently fraudulent if conditions are disclosed. The issue arises when the performance of an agent equipped with specialized tools and expensive beam search over solution trajectories is marketed as the base model’s raw capability.

A rigorous report isolates the base model from the surrounding scaffolding and discloses the budget, tools, retry count, and exact protocol.

### 3. Measurement Error and Saturation

Datasets have faulty answer keys, ambiguous wording, and insufficient unit tests. As top models approach performance ceilings, a handful of flawed questions and sampling noise matter more than the score difference on a leaderboard. Without confidence intervals, “90.1% vs. 89.8%” implies a level of precision the measurement cannot support.

### 4. Mismatch with Your Actual Use Case

Your team rarely needs an abstract champion. You need a system that writes in your language, follows your schemas, safeguards secrets, fits your budget, and allows humans to review critical decisions. It is entirely possible that a model ranked second on a global leaderboard will outperform the leader on your internal set of 50 real-world tasks.

## How to Read a Score: A Minimal Evaluation Checklist

Before trusting a number, ask for—or track down—answers to eight questions:

1. **What task was measured?** Not the benchmark name, but the input, expected output, and success criteria.

2. **Which version and date?** Datasets, models, and leaderboards change over time.

3. **What was the model configuration?** The exact version tag, system prompt, temperature, and context window.

4. **What tools and permissions did the agent have?** Web search, browser, terminal, code execution, persistent memory.

5. **What was the budget?** Tokens, runtime, inference cost, and parallel attempts.

6. **How was it scored?** pass@1 or pass@k, automated test suite or LLM judge, blinded or unblinded evaluation.

7. **How stable is the result?** Repeat runs, variance, confidence intervals, and error samples.

8. **How were leakage and task quality controlled?** Knowledge cutoff dates, private test splits, and human audits.

If half of these questions go unanswered, it does not make the product bad. But it makes the score flimsy evidence for an expensive or risky decision.

## How to Run Your Own Honest Mini-Benchmark

Instead of debating public leaderboards, you can assemble an internal evaluation in a single day:

1. Collect 30–100 recent tasks for which you already have clear acceptance criteria. Strip out personal data and secrets.

2. Categorize them by severity: standard tasks, expensive mistakes, and critical safety violations. Do not average everything into a single score.

3. Lock down a uniform execution environment for all models: identical tools, time limits, and attempt caps.

4. Evaluate using code execution or factual verification wherever possible; where not possible, use blinded pairwise comparisons by multiple human reviewers.

5. Log not just success, but also cost, latency, human interventions required, dangerous errors, and resilience.

6. Maintain a fresh held-out set: never use it for day-to-day prompt tuning.

7. Rerun the evaluation after any update to model versions, tooling, or permission policies; the same model name does not guarantee the same system behavior.

The goal is not to crown the “smartest model,” but to determine _which system, under what risk profile and budget, fits which part of your workflow_.

## What Mature AI Evaluation Looks Like

Modern frameworks like HELM are moving away from raw accuracy toward multidimensional evaluations that measure efficiency, toxicity, and bias. Frontier models also require dedicated evaluations for dangerous capabilities and guardrail robustness—questions fundamentally different from “how well does the model solve a test.”

A mature approach functions as a portfolio: closed and fresh benchmarks for generalization; executable tasks for real-world agency; expert verification for high-stakes domains; user preference tests for ergonomics; red-teaming and safety evals for risk mitigation. You cannot honestly compress these results into a single magic number, but together they paint a far clearer picture of whether a system is fit for purpose.

## Conclusion

A benchmark is an instrument, not a verdict. A good instrument allows you to measure changes under controlled conditions. Misuse begins when its readout is brandished as proof of universal intelligence, safety, or production readiness.

When looking at leaderboards, start by asking **what claim the test actually supports**, followed by **how transparent the methodology is**, and only then, **whether it reflects your actual problem**. The higher the cost of failure, the less you should rely on a single score.

## A Guide to 20 Key Benchmarks and Arenas

Below is a roadmap of which evaluations to consult for specific practical tasks, followed by a technical breakdown of each.

If your question is… | First look at | Then be sure to add
“Can the model reason through complex knowledge?” | MMLU-Pro, GPQA Diamond, HLE | ARC-AGI-2 or FrontierMath, but only alongside cost and attempt count
“Can it solve mathematics?” | AIME, FrontierMath | Human review of the full derivation; a final number alone is not a proof
“Can it write code?” | LiveCodeBench, HumanEval+ | SWE-bench and Terminal-Bench, if you need an agent rather than autocomplete
“Can it fix bugs in our product?” | SWE-bench, Terminal-Bench | Your own internal issue backlog, CI runs, security checks, and cost tracking
“Can it use a browser and navigate a computer?” | WebArena, OSWorld, GAIA | Custom tests on your internal sites, forms, and permission policies
“Can it understand documents, charts, and diagrams?” | MMMU-Pro, MathVista | Real scans and spreadsheets from your specific domain
“Which chatbot do people prefer?” | LMArena | Arena-Hard-Auto and factual audits of critical claims
“Which system builds better web apps?” | WebDev Arena | Testing on your specific tech stack, deployment pipeline, and code reviews

## How to Read This List

- **Benchmark:** A curated set of tasks paired with an automated evaluation script. It typically reports the percentage of solved problems.

- **Arena:** Humans compare two responses or artifacts side by side in a blind test and pick the winner. It outputs a preference rating, not a measure of factual accuracy.

- **accuracy:** The proportion of correct answers.

- **pass@1:** The first attempt passes all checks.

- **pass@k:** At least one of _k_ attempts passes. This is fundamentally different from pass@1: something or someone must sample and filter the remaining attempts.

- **Agent:** More than just a model. It is the model plus a system prompt, planning loop, tools, permissions, and budgets. Its score cannot be credited to the base model alone.

---

## Part 1. Knowledge, Logic, and Mathematics

### 1. MMLU-Pro

- **Input:** A question spanning one of many academic subjects, accompanied by multiple-choice options.

- **Expected output:** Select the correct option.

- **Scoring:** accuracy, the percentage of answers matching the key.

- **What it actually measures:** Breadth of academic knowledge and structured reasoning in an English-language exam format.

- **What a high score does not mean:** That the model can reliably practice medicine, work as an attorney, conduct research, or process your internal documents.

- **When to use it:** For an initial baseline comparison of general-purpose text models.

- **Source:** [MMLU-Pro](https://arxiv.org/abs/2406.01574).

### 2. GPQA Diamond

- **Input:** A graduate-level science question in physics, chemistry, or biology with multiple-choice options.

- **Expected output:** Select the correct answer, designed to be Google-proof.

- **Scoring:** accuracy across the Diamond subset.

- **What it actually measures:** A combination of domain-specific scientific knowledge and advanced reasoning on challenging problems.

- **What a high score does not mean:** That the system can design an experiment, analyze a novel research paper, trace citations, or guarantee clinical accuracy.

- **Technical caveat:** The subset is small; getting just a few questions right or wrong noticeably swings the percentage. Always verify the dataset version: the original Diamond and Diamond Clean differ.

- **Source:** [GPQA](https://arxiv.org/abs/2311.12022), [GPQA Diamond Clean](https://gpqa.ai/).

### 3. Humanity’s Last Exam (HLE)

- **Input:** Highly complex closed-ended questions across a wide variety of academic disciplines, including multimodal inputs.

- **Expected output:** Provide the exact final answer.

- **Scoring:** Proportion of responses that pass automated verification.

- **What it actually measures:** A broad stress test on exceptionally difficult academic benchmarks.

- **What a high score does not mean:** That this is truly the “last exam” for general intelligence; models can still fail at underlying logic or real-world application.

- **Technical caveat:** Requires tracking the benchmark release date, anti-leakage policies, and specific scoring methodology for images and free-form answers.

- **Source:** [HLE](https://arxiv.org/abs/2501.14249).

### 4. ARC-AGI-2

- **Input:** Several colored grid demonstrations showing “input → correct output,” followed by a new test grid without an answer.

- **Expected output:** Infer the transformation rule and output the exact correct target grid.

- **Scoring:** Exact match of the entire grid; typically reported as pass@1, pass@2, along with time and token cost per task.

- **What it actually measures:** Inductive reasoning and rule discovery from few-shot examples, rather than text memorization.

- **What a high score does not mean:** That the system is skilled at office tasks or conversational interaction; the benchmark format is highly specialized.

- **Technical caveat:** Strong results often stem from specialized solvers and search algorithms rather than the underlying base model. Reports must disclose the agent harness, retry budget, and compute cost.

- **Source:** [ARC-AGI-2](https://arcprize.org/arc-agi/2).

### 5. FrontierMath

- **Input:** Novel, complex mathematics problems—predominantly unreleased publicly—ranging from university to research level.

- **Expected output:** Compute a verifiable, exact answer; this could be a number, a symbolic expression, or an object parsed by an evaluation script.

- **Scoring:** Percentage of problems with an automated exact-match verification, reported separately by tier and version.

- **What it actually measures:** The ability to carry out rigorous mathematical reasoning through to a concrete result.

- **What a high score does not mean:** Complete mathematical proof generation or the ability to conduct novel academic research without expert validation.

- **Technical caveat:** Questions and answer keys are periodically revised, difficulty tiers vary, and much of the test set is held private. A claim like “FrontierMath 50%” is meaningless without specifying the tier and version.

- **Source:** [FrontierMath: Methodology and Versions](https://epoch.ai/frontiermath/tiers-1-4/about).

### 6. AIME

- **Input:** A competition math problem from a given year of the American Invitational Mathematics Examination.

- **Expected output:** Return an integer between 0 and 999.

- **Scoring:** Exact match of the final integer.

- **What it actually measures:** The ability to execute a multi-step chain of competition-math reasoning to reach a single numerical answer.

- **What a high score does not mean:** That the derivation is sound or that the model generalizes to higher-level mathematics.

- **Technical caveat:** Each competition edition contains only 15 questions per exam. Specifying the year is essential: AIME 2024 and AIME 2025 are distinct tests.

- **When to use it:** A quick diagnostic for reasoning on a recent problem set, but not a standalone foundation for major decisions.

### 7. GSM8K / MATH-500

- **Input:** Grade-school word math problems (GSM8K) or high school competition problems (MATH-500).

- **Expected output:** Provide the correct final answer.

- **Scoring:** Exact match.

- **What it actually measures:** Baseline mathematical reasoning and compatibility with older academic literature.

- **What a high score does not mean:** Frontier reasoning ability; these datasets have been public for years, making memorization and contamination widespread.

- **When to use it:** As a sanity and regression check after model fine-tuning, quantization, or decoding adjustments, rather than a primary selection metric.

- **Sources:** [GSM8K](https://arxiv.org/abs/2110.14168), [MATH](https://arxiv.org/abs/2103.03874).

---

## Part 2. Instruction Following and Unit-Level Coding

### 8. IFEval

- **Input:** A prompt containing verifiable constraints: “write exactly three paragraphs,” “do not use commas,” “return valid JSON,” or “mention the word X.”

- **Expected output:** Respond to the prompt while adhering to all constraints.

- **Scoring:** Strict and loose compliance rates, calculated at both the instruction level and prompt level.

- **What it actually measures:** Syntactic compliance and formatting discipline, not factual truth.

- **What a high score does not mean:** That the model comprehended complex business logic or will not hallucinate facts.

- **Technical caveat:** Look specifically at prompt-level strict: a single broken constraint should fail the entire test case. When testing your own workflows, introduce prompt variations, which often reveal unexpected drops in adherence.

- **Sources:** [IFEval](https://arxiv.org/abs/2311.07911), [IFEval++ reliability study](https://aclanthology.org/2026.acl-long.354/).

### 9. HumanEval+ / MBPP+

- **Input:** A docstring describing a self-contained function alongside its signature.

- **Expected output:** Write the function implementation.

- **Scoring:** Code is executed against a test suite: pass@1 if the first generation passes; pass@k if at least one of k generated candidates passes.

- **What it actually measures:** Local algorithmic correctness and small-scale function generation.

- **What a high score does not mean:** That an agent can navigate a large codebase, design system architectures, or write secure production code.

- **Why the “+” matters:** The original test suites were sparse; extended test cases catch accidental passes that failed edge cases.

- **Sources:** [HumanEval](https://arxiv.org/abs/2107.03374), [EvalPlus](https://arxiv.org/abs/2305.01210).

### 10. LiveCodeBench

- **Input:** Recent competitive programming problems published after the model's training cutoff.

- **Expected output:** Write a program that passes all hidden test cases.

- **Scoring:** Execution success rate, typically reported as pass@1.

- **What it actually measures:** Fresh algorithmic problem-solving ability with minimal risk of memorizing past training data.

- **What a high score does not mean:** That the model can maintain large software projects or collaborate effectively in a development team.

- **Technical caveat:** Using recent problems reduces contamination risk but does not eliminate it. Always check the problem date window, programming language, execution time limits, and permitted tooling.

- **Source:** [LiveCodeBench](https://livecodebench.github.io/).

---

## Part 3. Agents: Repositories, Terminals, Browsers, and Operating Systems

### 11. SWE-bench

- **Input:** A real GitHub issue, a repository snapshot, and a reproduction environment with tests.

- **Expected output:** Edit codebase files to resolve the issue and pass unit tests.

- **Scoring:** The generated patch is applied; the run succeeds if all evaluation tests pass.

- **What it actually measures:** The combined capability of the model and coding harness to inspect a codebase, locate bugs, edit code, and execute test suites.

- **What a high score does not mean:** That the base model alone can code at this level. Performance hinges heavily on the agentic scaffold, search algorithms, terminal access, token budgets, and retry loops.

- **Critical caveat:** SWE-bench, SWE-bench Lite, Verified, Multimodal, and Multilingual are distinct benchmarks. Audits of SWE-bench Verified uncovered data contamination and problems with broken test suites; historical leaderboard scores cannot be accepted without specifying the exact dataset variant and harness.

- **Sources:** [SWE-bench](https://arxiv.org/abs/2310.06770), [Verified audit](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/).

### 12. Terminal-Bench

- **Input:** A Linux container preloaded with a problem and interactive shell access.

- **Expected output:** Use the terminal to achieve a verifiable end state: fix an environment, compile a build, configure a tool, execute data processing, and more.

- **Scoring:** An in-environment verifier inspects the final system state; results are measured as success rates across multiple trials.

- **What it actually measures:** An agent's autonomous operation in a CLI: planning, running commands, parsing error traces, and managing files and utilities.

- **What a high score does not mean:** That an agent is safe to deploy onto production infrastructure, handle secrets, or operate with open network access.

- **Technical caveat:** The exact Terminal-Bench and harness versions must be specified. The same model paired with a different scaffold can yield wildly divergent scores; report retries, time limits, cost, and run counts.

- **Source:** [Terminal-Bench](https://github.com/harbor-framework/terminal-bench).

### 13. WebArena

- **Input:** Self-contained, realistic website environments and a natural-language user task (e.g., search, update account settings, complete an e-commerce checkout).

- **Expected output:** Execute a sequence of browser actions to transition the application into the intended target state.

- **Scoring:** Automated state checks confirm whether the goal was fulfilled.

- **What it actually measures:** Multi-step planning and browser automation in a reproducible web sandbox.

- **What a high score does not mean:** That the agent will safely handle live websites, CAPTCHAs, financial transactions, PII, or unexpected UI redesigns.

- **When to use it:** When benchmarking browser-use agents; always follow up with testing on internal staging environments.

- **Source:** [WebArena](https://arxiv.org/abs/2307.13854).

### 14. OSWorld

- **Input:** A full desktop operating system with standard GUI applications; the agent perceives screenshots and controls the mouse and keyboard.

- **Expected output:** Complete multi-step workflows spanning web browsers, desktop software, and file systems.

- **Scoring:** Automated verification of OS state or generated output artifacts.

- **What it actually measures:** General computer-use capabilities: visual UI parsing, desktop navigation, and cross-application context switching.

- **What a high score does not mean:** That the agent is robust against custom screen resolutions, different system locales, software version mismatches, or irreversible system actions.

- **Technical caveat:** Keep a close eye on execution resolution, timeout thresholds, retry frequency, and error-recovery behaviors.

- **Source:** [OSWorld](https://arxiv.org/abs/2404.07972).

### 15. GAIA

- **Input:** Pragmatic multi-modal queries requiring diverse actions: web browsing, file inspection, mathematical calculations, visual processing, or running code.

- **Expected output:** Gather facts, orchestrate tools, and produce a concise, unambiguous answer.

- **Scoring:** Exact or quasi-exact match of the final response; evaluation split answers are held private.

- **What it actually measures:** Overall performance of an AI assistant agent operating within a fixed toolset.

- **What a high score does not mean:** That an agent behaves reliably or safely across the open internet, or understands your organization’s internal policies.

- **Technical caveat:** Evaluations with and without browser access cannot be compared directly. Reports must detail the available tools, API privileges, token budgets, and whitelisted sources.

- **Source:** [GAIA](https://ai.meta.com/research/publications/gaia-a-benchmark-for-general-ai-assistants/).

---

## Part 4. Vision: Documents, Schematics, and Charts

### 16. MMMU-Pro

- **Input:** College-level multi-discipline questions where text and imagery are interdependent: charts, diagrams, maps, tables, formulas, chemical structures, sheet music.

- **Expected output:** Accurately interpret the visual artifact and solve the domain problem.

- **Scoring:** accuracy of answers.

- **What it actually measures:** Joint visual perception and multimodal reasoning in complex subjects.

- **What a high score does not mean:** That the model will reliably extract data from low-resolution scans, legal contracts, clinical records, or your organization’s specific PDF layouts.

- **Technical caveat:** Verify image preprocessing: native resolution, OCR pipelines, image cropping, multi-page handling, and zero-shot vs. few-shot prompting.

- **Sources:** [MMMU](https://mmmu-benchmark.github.io/), [MMMU-Pro](https://arxiv.org/abs/2409.02813).

### 17. MathVista

- **Input:** Visual mathematical challenges: function plots, geometric figures, data tables, scientific charts, and diagrams.

- **Expected output:** Extract relevant visual data, execute mathematical reasoning, and output the answer.

- **Scoring:** Exact match of the final answer.

- **What it actually measures:** True visual mathematical reasoning rather than simple OCR.

- **What a high score does not mean:** That the model can flawlessly interpret enterprise dashboards, business graphics, or application screenshots.

- **Technical caveat:** An aggregate accuracy score hides whether failures were caused by visual misrecognition, calculation errors, or output formatting issues. Test these variables independently on internal data.

- **Source:** [MathVista](https://mathvista.github.io/).

---

## Part 5. Arenas: Human Preference Evaluations

### 18. LMArena / Chatbot Arena

- **Input:** An authentic user prompt. Two anonymous models generate responses side by side.

- **Expected output:** Provide the answer that the human user prefers.

- **Scoring:** Pairwise human votes are compiled into an Elo-style rating with confidence intervals.

- **What it actually measures:** Subjective conversational utility: clarity, tone, relevance, and prompt alignment.

- **What a high rating does not mean:** That answers are factually true, safe, legally sound, or capable of sustaining long-horizon workflows.

- **Technical caveat:** Examine the query category, sample size, cutoff dates, and confidence intervals. A single overall rating blurs differences across languages and specialized domains.

- **Source:** [How Arena Works](https://arena.ai/how-it-works), [Methodology](https://arxiv.org/abs/2403.04132).

### 19. Arena-Hard-Auto

- **Input:** Complex, open-ended prompts filtered from live Chatbot Arena traffic.

- **Expected output:** Generate an answer superior to a fixed baseline model.

- **Scoring:** A designated LLM judge compares responses and scores them; serves as an automated, low-cost proxy for human preferences.

- **What it actually measures:** Relative competitiveness of open-ended conversational models evaluated under a specific judge configuration.

- **What a high score does not mean:** That human users will agree with the judge, or that the responses are factually validated.

- **Technical caveat:** This evaluates two systems simultaneously: the candidate model and the judge model. Any analysis must disclose the judge model, its system prompt, the baseline model, and the prompt set version.

- **Source:** [Arena-Hard-Auto](https://github.com/lmarena/arena-hard-auto).

### 20. WebDev Arena

- **Input:** A prompt requesting a web application or UI component; implementations from two competing systems are presented to the user.

- **Expected output:** Build an interactive result that users prefer visually and functionally over the competitor’s.

- **Scoring:** Blind pairwise user votes converted into ratings.

- **What it actually measures:** The quality of an initial web prototype as judged by end users: aesthetics, layout, feature completeness, and basic interactivity.

- **What a high rating does not mean:** That the generated code is secure, accessible, well-tested, maintainable, or ready for production deployment.

- **Technical caveat:** Useful for ideation and UX evaluation, but production decisions require automated testing, code review, security audits, and load testing.

- **Source:** [WebDev Arena](https://arena.ai/blog/webdev-arena).

---

## The Minimal Evaluation Stack for Model Selection

Instead of relying on a single “top-performing” model on a leaderboard, assemble a targeted suite based on your use case:

- **General conversational chat:** LMArena + IFEval + 50 internal anonymized queries evaluated via blind pairwise review.

- **Software development assistant:** LiveCodeBench + HumanEval+ + SWE-bench/Terminal-Bench + real pull-request tasks evaluated in a sandboxed CI environment.

- **Browser automation agent:** WebArena + GAIA + your internal staging environment with dedicated test credentials.

- **Desktop automation:** OSWorld + a sandboxed internal suite of actions running on non-critical, reversible data.

- **Document and chart processing:** MMMU-Pro + MathVista + anonymized samples of your organization's actual files.

- **High-stakes workflows:** Augment any suite with factual verification pipelines, red-teaming scenarios, latency and cost tracking, variance measurements, and mandatory human-in-the-loop signoff for sensitive operations.

## The One Attribution String Every Leaderboard Must Provide

> **Model & date + agent & tools + benchmark version + retry count + budget + metric + confidence interval.**

If even half of these parameters are missing, you are looking at marketing collateral, not engineering proof.

---

Interesting: https://interesting.weblocally.com/en/
