Inter-rater agreement
The extent to which independent reviewers assign compatible judgments.
What Inter-rater agreement means
High agreement can reflect clear criteria or shared bias; low agreement signals ambiguity that averages conceal.
Terms this definition uses
Related terms in Measurement and evaluation
How a measurement earns trust: evaluation sets, error rates, calibration, reproducibility, and the biases that make a number look better than it is.
AI answer reading · name confusion · hedged answer · Benchmark · Evaluation set · Golden dataset · Ground truth · Human evaluation · LLM-as-judge · Pairwise evaluation · Pointwise evaluation · Rubric · Accuracy · F1 score · False positive rate · False negative · True positive · True negative · Calibration · Confidence score · Coverage · Abstention · Reproducibility · Repeatability · Variance · Confidence interval · Sample size · Sampling bias · Selection bias · Survivorship bias · Data leakage · Test contamination
Questions about Inter-rater agreement
What is Inter-rater agreement?
The extent to which independent reviewers assign compatible judgments.
Which area of the glossary does Inter-rater agreement belong to?
Measurement and evaluation: How a measurement earns trust: evaluation sets, error rates, calibration, reproducibility, and the biases that make a number look better than it is.
See it in the full glossary · 731 terms across 19 areas.