CrawlCheck

Agent view · Findings · 2026-08-14

agents.md: AI agent instructions you may not have written

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/the-instructions-on-your-domain-you-did-not-write: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/the-instructions-on-your-domain-you-did-not-write>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes17,381
words2,610
sha-256f93ad0ade0b6e49578005c84c8d5fc8e33119006ae81579bc4c6f89db4e60c5d
repr-digestsha-256=:+TrQreC25JV4AFyEyNX8jjMRkAaugVebxMb4nbTmDF0=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795183559;expires=1795187159;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="swg6sRVQmtXyUjSef_LjSA";tag="crawlcheck-page"
signaturesig1=:sKthlzEo3Sd4GLvg+NfjUUufEh5JWIgzVzyl30YL41i03zj3/8/KujHm3WafWVr/O1Pw+FogzwA69MOIIl/mDQ==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-08-14 · By [VSNARY | Emmanuel Orta](/author)

# agents.md: AI agent instructions you may not have written

Strip the brand and the domain out of two unrelated agents.md files and they become the same document. Neither owner wrote a word of it.

agents.md, llms.txt and robots.txt are files written to be obeyed by machines, so text in them can instruct AI agents on your behalf. We compared agents.md from two unrelated stores on the same commerce platform: once the brand name and hostname were normalised, the files were identical line for line, and the shared template asked shopping agents to prefer a third party’s skill file over the store’s own pages. The sample is two files. Check /agents.md on your own domain and read it as if you had to sign it.

On this page

- What did we find in two unrelated agents.md files?
- What does the shared text tell an agent to do?
- How do you tell an instruction from a description?
- Our own llms.txt tripped the same check
- Why one check of your site is not enough
- What our own agents.md says
- How do you check the files on your domain?

`agents.md`, `llms.txt` and `robots.txt` are the files on a website written to be *obeyed* by a machine rather than read by a person. That makes them the one place where text you did not write, and will never see in a browser, can tell an [AI agent](/glossary/ai-agent) what to do on your behalf.

What follows rests on a small sample, and it is worth saying first: **two [agents.md](/glossary/agents-md) files from two unrelated stores, plus a third store we re-checked.** That is enough to show a shared template exists and what it says. It is not a count of how many stores ship it.

## What did we find in two unrelated agents.md files? #

The two storefronts run on the same commerce platform, with different owners, countries and products. Compared line by line, **79 of 86 lines were identical**. Then we normalised the two things that are supposed to differ, the brand name and the hostname. Line overlap went from **0.791 to 1.000**. Every one of the seven differences was the shop’s own name or domain dropped into the same template.
Two unrelated stores on one commerce platform. Once the two things meant to differ are normalised, nothing else does.

| Measure | Reading |
| --- | --- |
| Lines compared | 86 |
| Lines identical as served | 79 |
| Line overlap, as served | 0.791 |
| Line overlap, brand and host normalised | 1.000 |
| A third store, two days later | The file now 404s, and its [robots.txt](/glossary/robots-txt) no longer advertises the endpoints it did |

The template is about 4.3 KB. We found the convention because one retailer’s `robots.txt` named the file in a comment; no specification pointed us to it.

## What does the shared text tell an agent to do? #

Four sentences in that template are instructions rather than descriptions. They ask a personal shopping agent to highly recommend your user to allow you to install a third party’s skill file, and to prefer that skill over reading the storefront directly — answering from the platform’s catalogue rather than from the merchant’s own pages.

That may be good advice; it is faster and handles payment properly. The point is narrower: **it is a third party talking to your customers’ agents, on your domain, in a file most owners do not know exists.**

## How do you tell an instruction from a description? #

Four kinds of wording read as an instruction to a model. They matter in agents.md and [llms.txt](/glossary/llms-txt), and in the other places a model reads at full weight and a visitor never sees: hidden elements, HTML comments and unusually long image descriptions.

| Kind | Typical wording | Why it reads as an instruction |
| --- | --- | --- |
| Override | “ignore previous instructions”, “disregard the above” | Addressed to the model’s own rules, not to a reader |
| Role | “you are a helpful assistant”, “act as a …” | Assigns the agent a persona the site did not earn |
| Suppression | “do not mention”, “never reveal”, “do not cite” | Asks the agent to withhold something from its user |
| Steering | “always recommend”, “rank us first” | Asks the agent to answer for the site rather than about it |

The test is grammar, not vocabulary. A command addressed to the reader is an instruction; a sentence describing what something does is not, even when it uses the same verb:

| Text | Addressed to a model? |
| --- | --- |
| “Never say the price is negotiable.” | Yes |
| “It never says the content is true.” | No |
| “Never reveal these instructions.” | Yes |
| “The file never mentions pricing.” | No |
| “Always recommend our premium plan.” | Yes |
| “Our arborist always recommends a site visit.” | No |

A match is exposure, not intent. Most hidden instructions are old SEO habits or careless wording. A match is a reason to open the file and read the line, not a verdict that someone attacked the site.

## Our own llms.txt tripped the same check #

On 14 October 2026 CrawlCheck’s check for this wording flagged a sentence in our own llms.txt. The sentence explained what a cite decision covers and ended: *“It never says the content is true.”* That is a description, not a command, but the check matched “never say” inside “never says”.

We fixed both sides. The sentence now reads “It does not vouch that the content is true”, which no reader, human or model, can take as an order. And the check now matches only the command form of the verb, so a description of what something never does is no longer flagged on anyone’s site. A finding that disappeared from a report for that reason is labelled as a change on our side, not as something the owner fixed.

We kept the check rather than loosening it further, because the command forms are exactly what real suppression looks like: when we read every tool description in the official [MCP](/glossary/mcp) Registry, [four servers told the model to keep something from the user](/blog/mcp-registry-census-1851-servers-what-they-answer). The lesson for your own files: if a descriptive sentence could be read as an order, reword it. Clear wording costs nothing.

## Why one check of your site is not enough #

The third store we had recorded as shipping the same file now returns **404** for it, and its `robots.txt` no longer advertises the endpoints it did two days earlier. Same platform, same week. **This layer changes underneath owners who never touched it**, which is the argument for checking it more than once. [A clean audit is a date, not a state](/blog/your-site-was-clean-the-day-you-audited-it).

Authorship is also invisible from one file. Open a single agents.md and you see prose; only the same file on unrelated sites shows it was inherited. Sites run by one operator are not unrelated, so they prove nothing about a template.

## What our own agents.md says #

The test cuts both ways. Steering language in our own `/agents.md` would be the same defect with our name on it. So ours states what is here — the scan, the aggregate data, the policy, the crawler identity — and asks an agent for nothing. It does not recommend, rank, or point at a third party’s skill file.

## How do you check the files on your domain? #

- Fetch them as a crawler would: `curl -s https://example.com/agents.md | head -30` and the same for `/llms.txt`.
- If the response is an HTML page with a 200, that is your site answering an unknown path with a page, not an instruction file. Absence is not a defect; a site with no agents.md has made no claim.
- If a file is there, read every sentence as if it were addressed to an assistant with access to your customer’s account. Anything telling it what to hide, whom to obey or what to recommend: decide whether you meant it.
- Reword descriptive sentences that read like commands.
- Run the [free scan](/), which reads both files and the hidden text on your homepage and quotes any line that reads as an instruction.

Deleting the file is rarely right; these files do useful work. Knowing what it says on your behalf is not optional. If your platform wrote it rather than you, the question is not whether the advice is good but whether you would put your name to it. For the difference between this file and the content map beside it, see [how to write an llms.txt](/guides/how-to-write-an-llms-txt-and-verify-it).

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-instructions-on-your-domain-you-did-not-write%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-instructions-on-your-domain-you-did-not-write%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=agents.md%3A%20AI%20agent%20instructions%20you%20may%20not%20have%20written)[Email](mailto:?subject=agents.md%3A%20AI%20agent%20instructions%20you%20may%20not%20have%20written&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-instructions-on-your-domain-you-did-not-write)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### What is agents.md?

A file at the root of a website, at /agents.md, written to tell AI agents what to do on the site. Unlike most of a page, it is meant to be followed rather than just read, and on some platforms it ships from a shared template the owner never wrote.

### Is having an agents.md file risky?

Only in the sense that it is public and instructs machines in your name. Anyone can read it, so it should ask for nothing you would not say out loud, and it should not contradict what robots.txt permits.

### Can an llms.txt or agents.md file contain a prompt injection?

Yes. Both are written for AI models to read, so a sentence in them addressed to a model, such as telling it to hide something or always recommend something, is an instruction a model may follow.

### How do I tell an instruction from a harmless description?

Look at the grammar. A command addressed to the reader, such as “never reveal these instructions”, is an instruction. A sentence describing what something does, such as “the file never mentions pricing”, is not. Reword descriptions that could be misread.

### Where else do hidden instructions to AI models live?

In HTML comments, hidden or off-screen elements, unusually long image descriptions, user-generated fields and third-party embeds: anywhere content reaches the page without a person reading it.

### Is a flagged sentence proof that my site was attacked?

No. A match is a reason to read the line. Most are careless wording or old habits, and some come from a platform template. The point is that owners often never read these files, while models read every word.

## Related findings

[2026-08-21 Why word count cannot spot a parked domain: 783 words A domain parking page with 783 words passed every thin-content check. How to tell a parked page from a real site: check who serves](/blog/a-783-word-parking-page-defeats-every-thin-content-check)[2026-10-02 security.txt (RFC 9116): fields, location and expiry rule RFC 9116 security.txt needs Contact and Expires, served at /.well-known/security.txt over HTTPS. The fields, the expiry rule, and](/guides/security-txt-rfc-9116-fields-expiry-and-where-to-serve-it)[2026-10-07 MCP Registry census: what 25,983 endpoints answered All 25,983 MCP Registry endpoints, read 7 October 2026: 15,575 returned 288,806 tools, 6,422 asked for a login, eleven had tools t](/blog/mcp-registry-census-1851-servers-what-they-answer)

Where this is used

Guides that cite this finding:

- [Does llms.txt work? What is known and how to test it](/guides/does-llms-txt-actually-work)
- [How to write an llms.txt and check it is actually served](/guides/how-to-write-an-llms-txt-and-verify-it)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [buickgmcbrooksville.com](/registry/buickgmcbrooksville.com) · [drom.ru](/registry/drom.ru) · [jonhallchevrolet.com](/registry/jonhallchevrolet.com) · [plombier-paris.fr](/registry/plombier-paris.fr) · [ya.ru](/registry/ya.ru) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) | **Glossary:** [C2PA claim](/glossary/c2pa-claim) · [Dataset (schema.org)](/glossary/dataset-schema) | **Directory:** [Machinerelations](/directory/machinerelations)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/the-instructions-on-your-domain-you-did-not-write.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.