Data · embeddings

Every CrawlCheck catalogue as vectors, for agents that match by meaning.

The crawler catalogue, the glossary, every problem code and every census measure, embedded with baai/bge-base-en-v1.5 (768 dimensions). Embed a query with that model and take the cosine nearest neighbours: a crawler named loosely, a term described rather than named, a finding in a user’s words.

The sets

Set	What is embedded	Items	Built	File
AI crawler catalogue	every named AI, search and training agent: token, operator, purpose	114	2026-10-27	ai-crawlers.json (0.6 MB)
Glossary of the agentic web	every defined term with its definition	751	2026-10-27	glossary.json (4.3 MB)
Problem codes	every finding code the scan can raise, with its title	63	2026-10-27	problems.json (0.4 MB)
Census measures	every measure the top-10,000 census reports	38	2026-10-27	census-measures.json (0.2 MB)

Format

One JSON file per set: model, dim, built_at, n, and items, each {id, url, text, vector}. text is the exact string that was embedded; url is the page the item describes.
Rebuilt when the source changes and at least weekly. The index at /data/embeddings/index.json carries each set’s build time and size.
CC BY 4.0: credit “CrawlCheck” with the file URL. The pages the items describe are the canonical copy; the vectors are a view of them.

Read by CrawlCheck Research. Other open data: radar, crawler IP-range history, every dataset.