Deduplication
Removing records judged to represent the same content or entity.
What Deduplication means
Aggressive rules can collapse distinct versions; weak rules leave repeated evidence that falsely appears independent.
Definitions that name Deduplication
Related terms in Retrieval engineering
How pages become chunks, vectors and candidates inside an answer engine, and why being retrieved is not being cited.
Corpus · Document ingestion · Data connector · Parsing · OCR · Layout-aware parsing · Semantic chunking · Fixed-size chunking · Chunk overlap · Parent-child retrieval · Metadata filtering · Vector database · Vector index · Sparse vector · Dense vector · Cosine similarity · Dot-product similarity · Approximate nearest neighbor · HNSW · Top-k · Retrieval recall · Retrieval precision · Mean reciprocal rank · Normalized discounted cumulative gain · Relevance score · Candidate generation · Query rewriting · Query expansion · HyDE · Multi-query retrieval · Reciprocal rank fusion · Late interaction · Lexical retrieval · Semantic search · Exact-match retrieval · Freshness boosting · Authority weighting · Source diversity · Near-duplicate detection · Semantic cache · Retrieval latency · Retrieval cutoff · Citation mapping · Answer synthesis
Questions about Deduplication
What is Deduplication?
Removing records judged to represent the same content or entity.
Which area of the glossary does Deduplication belong to?
Retrieval engineering: How pages become chunks, vectors and candidates inside an answer engine, and why being retrieved is not being cited.
← Source diversity · Near-duplicate detection →
See it in the full glossary · 731 terms across 19 areas.