Corpus
The collection of documents available to a search or retrieval system.
What Corpus does not show
It defines what can be found; no ranking method can retrieve a document absent from the corpus.
Where Corpus comes up on this site
- How to check if AI crawlers can read your siteThe heaviest page measured in this corpus delivered 476,540 bytes and 2.3% visible text .
- Do AI crawlers render JavaScript? Check what they seeIn this corpus it is three: The first is fatal on any client that does not execute scripts.
- Three of our own checks passed because we measured nothingCorpus figures are on the dataset page .
- AI visibility vendors audited: 11 of 23 have llms.txtDisclosure: CrawlCheck publishes this piece and sells a tool in the same category. crawlcheck.io is not in its own corpus, because a scanner cannot grade its own host from the outside view it uses for everyone else; check our files from any client you…
- When an LLM summarises your audit report and gets it wrongThe site scored 90 out of 100, sat at the 86th percentile of our corpus, and still graded C.
Terms this definition uses
Definitions that name Corpus
score version · percentile rank · BM25 · CCBot
Related terms in Retrieval engineering
How pages become chunks, vectors and candidates inside an answer engine, and why being retrieved is not being cited.
Document ingestion · Data connector · Parsing · OCR · Layout-aware parsing · Semantic chunking · Fixed-size chunking · Chunk overlap · Parent-child retrieval · Metadata filtering · Vector database · Vector index · Sparse vector · Dense vector · Cosine similarity · Dot-product similarity · Approximate nearest neighbor · HNSW · Top-k · Retrieval recall · Retrieval precision · Mean reciprocal rank · Normalized discounted cumulative gain · Relevance score · Candidate generation · Query rewriting · Query expansion · HyDE · Multi-query retrieval · Reciprocal rank fusion · Late interaction · Lexical retrieval · Semantic search · Exact-match retrieval · Freshness boosting · Authority weighting · Source diversity · Deduplication · Near-duplicate detection · Semantic cache · Retrieval latency · Retrieval cutoff · Citation mapping · Answer synthesis
Questions about Corpus
What is Corpus?
The collection of documents available to a search or retrieval system.
What does Corpus not show or guarantee?
It defines what can be found; no ranking method can retrieve a document absent from the corpus.
Which area of the glossary does Corpus belong to?
Retrieval engineering: How pages become chunks, vectors and candidates inside an answer engine, and why being retrieved is not being cited.
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.