Benchmark
A fixed task set and scoring procedure used to compare systems.
What Benchmark does not show
Performance can be optimized to the benchmark and may not transfer to production work.
Where Benchmark comes up on this site
- How an AI agent should verify a website before it actsThe full benchmark write-up has the method and the controls.
- AI agents cannot refuse what they cannot see: 27 trapsThe full case list and results are public at crawlcheck.io/benchmark , and the agentic-web overview this test belongs to is here .
- Sites measured is not people: what our own logs showedThe sites-measured figure stays, because it is true and because the benchmark population is the reason the rates on this site mean anything.
- Do AI agents stop before acting on a lookalike? We tested 4We built a small benchmark to measure the difference, and ran four models through it twice: once alone, once with a signed answer about the domain in the prompt.
Terms this definition uses
Definitions that name Benchmark
Related terms in Measurement and evaluation
How a measurement earns trust: evaluation sets, error rates, calibration, reproducibility, and the biases that make a number look better than it is.
AI answer reading · name confusion · hedged answer · Evaluation set · Golden dataset · Ground truth · Human evaluation · LLM-as-judge · Pairwise evaluation · Pointwise evaluation · Rubric · Inter-rater agreement · Accuracy · F1 score · False positive rate · False negative · True positive · True negative · Calibration · Confidence score · Coverage · Abstention · Reproducibility · Repeatability · Variance · Confidence interval · Sample size · Sampling bias · Selection bias · Survivorship bias · Data leakage · Test contamination
Questions about Benchmark
What is Benchmark?
A fixed task set and scoring procedure used to compare systems.
What does Benchmark not show or guarantee?
Performance can be optimized to the benchmark and may not transfer to production work.
Which area of the glossary does Benchmark belong to?
Measurement and evaluation: How a measurement earns trust: evaluation sets, error rates, calibration, reproducibility, and the biases that make a number look better than it is.
← hedged answer · Evaluation set →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.