CrawlCheck

Directory · ML platforms and experiment tracking

BentoML — what a crawler receives from bentoml.com

Website measurement, not a product review. This page records what machine clients received from bentoml.com. It does not evaluate BentoML software, features, pricing or support.

68
Dgrade

AI assistants are being shown something other than your site.

AI visibility 68/100, measured 2026-11-12 04:33 UTC from outside its network, as each named AI and search crawler and as a browser. An engine can technically reach this site and would struggle to get anything quotable out of it.

Held at D A high-severity finding sets the letter.

Open the full report bentoml.com ↗

861. ReachCan a named answer-engine crawler get your pages at all?622. ReadOnce it has the bytes, can it find the words?493. QuoteIs there a specific fact it can state and attribute?

9

findings

✗

llms.txt on its host

✗

entity map on its host

10

scans on record since 2026-09-18

The three stages, in the order an engine hits a site

Each stage gates the next: an engine that cannot reach a page never reads it, and one that cannot read it never quotes it.

86

Stage 1 · 40% of the score

Reach

Can a named answer-engine crawler get your pages at all?

Every named crawler was served the same page a browser gets.

62

Stage 2 · 30% of the score

Read

Once it has the bytes, can it find the words?

The content is buried in markup, or the files that guide a crawler are missing.

49

Stage 3 · 30% of the score

Quote

Is there a specific fact it can state and attribute?

An assistant would have to paraphrase your page instead of quoting a fact.

The stage to fix first is quote, the lowest of the three.

What to fix

9 findings on this site, 3 of them scored serious. Every one is named below; the top one is shown in full, free. What we saw, why each matters, how to fix it and the bytes behind it are what a licence buys — or we implement the whole list for you.

1 High 2 Medium 4 Low 2 Note

1robots.txt is unreadableHTTP 502.HighSettingConfirmed

What we saw

HTTP 502. /robots.txt

Why it matters

A 5xx on robots.txt tells a standards-following crawler to treat the whole site as disallowed (RFC 9309). A 401 or 403 is a 4xx, which the standard treats as no file and so allows everything, though some crawlers read it as a refusal. Either way the rules you wrote are not the rules a crawler read. This is the single most expensive file on the site to get wrong.

How to fix it

robots.txt could not be fetched. Allow it through the security layer; it is the one file every crawler reads first.

Setting: a setting on the server, CDN, DNS or a plugin — no code and no writing

On Next.js (detected)

  1. App Router: add app/robots.ts that returns a MetadataRoute.Robots object (rules per userAgent, sitemap URL). Pages Router or static export: put robots.txt in public/.
  2. Ship one of the two, not both: the route handler and a public/robots.txt collide.

All Next.js fixes

Site deployed from GitHub

  1. Add this workflow file as .github/workflows/crawlcheck.yml (change robots-path if robots.txt lives elsewhere).
  2. Run it from the Actions tab: it opens one pull request with the corrected robots.txt, every change listed. Review, merge, deploy.

How the action works

Paste this

A minimal valid file. Serve it at https://bentoml.com/robots.txt as text/plain with a 200, and let it through any bot or firewall rule (use your sitemap’s real address if it is not /sitemap.xml).

User-agent: *
Disallow:

Sitemap: https://bentoml.com/sitemap.xml

The whole corrected robots.txt for bentoml.com · built from the live file, every change listed at the top.

Fix it with a coding agent

Paste into Claude Code, Cursor or Codex, opened on the code that serves this site.

Show prompt
You are working in the codebase (or server/CDN configuration) that serves bentoml.com. A crawler audit found the problem below. Fix it at the source, keep the change as small as possible, and do not change anything a visitor sees unless the fix requires it.

Finding: robots.txt is unreadable (ROBOTS_BLOCKED, high severity)
Page: https://bentoml.com/robots.txt
What was measured: HTTP 502.
Why it matters: A 5xx on robots.txt tells a standards-following crawler to treat the whole site as disallowed (RFC 9309). A 401 or 403 is a 4xx, which the standard treats as no file and so allows everything, though some crawlers read it as a refusal. Either way the rules you wrote are not the rules a crawler read. This is the single most expensive file on the site to get wrong.
How to fix it: robots.txt could not be fetched. Allow it through the security layer; it is the one file every crawler reads first.
Detected platform: Next.js. Make the change where this platform controls it.
Rule and examples: https://crawlcheck.io/rules/ROBOTS_BLOCKED
Full report (markdown): https://crawlcheck.io/r/x8rofgd4hd.md

When you are done:
1. Tell me which files or settings you changed and why.
2. If the fix is a server, CDN or cache setting you cannot change from the code, say exactly where it is set.
3. After deploying, re-scan at https://crawlcheck.io/?url=bentoml.com . The fix counts when ROBOTS_BLOCKED is gone from the new report.
Why this was decided
RuleROBOTS_BLOCKED · rule pack sv34 · scope core
Evaluated2026-11-12 04:33:02 UTC
Decisiond1:4b3381a72badb3c2e8e1 · finding f1:e3e5575d92ac70263c39 resolve either at /api/graph/<id> with a key
Observercrawlcheck-worker build be1e3120 · audit v4

What it rests on

IdentityStatusBytessha256Kept
read: /robots.txt50216bb2fb7b51f18already_held

27 more in the manifest.

Artifacts: manifest:aef4b75cfa45b9c9a031029bf814887531199a3121992807e61a19bed252a0c8

Grade effect Caps the letter at D. Without it the letter would be C (score 68). Moves no points; the score is built from the measured rows.

Rule ROBOTS_BLOCKED under scoring rules v34 · fingerprint ROBOTS_BLOCKED|/robots.txt · persisting (seen in 2 of 11 readings)

Verify the fix Re-scan after the change. This finding is cleared when ROBOTS_BLOCKED is no longer reported at /robots.txt; if it comes back after clearing, it is marked a regression, not a new finding.

ROBOTS_BLOCKED · full transcript

2www and the apex both serve the site instead of one redirecting🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Medium
3Almost nothing a machine receives from this page is readable text🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Medium
4The origin returned a server error on the sitemap path/sitemap.xml🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Low
5The origin returned a server error on the sitemap path/sitemap_index.xml🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Low
6The origin returned a server error on the sitemap path/wp-sitemap.xml🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Low
7Whether a sitemap exists could not be determined — every path was refused🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Low
8No llms.txt/llms.txt🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Note
9A cached copy of this page older than an hour was served🔒 What we saw, why it matters, the fix and the evidence bytes are in the licensed report.Note

Open the other 8

What we saw, why it matters, how to fix it, and the evidence bytes for every finding — on this report and every re-scan. From $29 a month.

Unlock the findings

Or have them fixed

We implement the list on your site and re-measure, with a sealed before and after. One site, $100 once.

Get this fixed — $100

Sites like yours that fixed the same problem

Matched on Amazon CloudFront. A fix counts once two readings confirm it; sites are not named. Read from the technology record of 2026-11-05.

serves no /llms.txt

10 sites on a stack like yours have made this fix (Published llms.txt); 10 still hold it. 103 on any stack.

  • similar site A (WordPress + Amazon CloudFront + WP Engine; shares CDN): fixed 2026-11-10, still fixed today
  • similar site B (Amazon CloudFront; shares CDN): fixed 2026-11-06, still fixed today
  • similar site C (Amazon CloudFront; shares CDN): fixed 2026-11-06, still fixed today

serves no /agents.md (optional)

2 sites on a stack like yours have made this fix (Published agents.md); 1 still hold it. 22 on any stack.

  • similar site A (Amazon CloudFront; shares CDN): fixed 2026-11-11, still fixed today
  • similar site B (Amazon CloudFront; shares CDN): fixed 2026-11-08, came back

The signed answer lists every match.

The chain a crawler follows

The chain breaks at robots.txt. Everything after that point is only reachable by a crawler guessing the conventional path.

✗robots.txtEvery crawler reads this first
✗sitemapNot named in robots.txt and none resolves
✗llms.txtPoints onward, links stay on this host
✗entity graphParses as JSON and points home

A machine file that does not parse is worth less than one that is absent, because it looks present. Rows behind this.

Where the bytes go

Of the 418,116 decompressed bytes the homepage delivers, 1.9% is text a reader or a model can actually use. Most of what a crawler downloads here is not words.

  • Readable text 1.9% · 7,813 B
  • Markup & attributes 27.2% · 113,624 B
  • Structured data (JSON-LD) 0.9% · 3,882 B
  • Inline CSS 0.3% · 1,073 B
  • Inline JavaScript 57.3% · 239,723 B
  • Inline SVG 12.4% · 52,001 B

The individual blocks that weigh the most, so the fix is a search rather than a hunt:

  • Inline JavaScript · 237,530 B (56.8%) — a framework state payloadopens {"props":{"pageProps":{"global":{"name":"Bento: Run Inference at Scale
  • Inline SVG · 9,956 B (2.4%) — unnamedopens <path d="M128.467 2.735C128.393 2.50167 128.292 2.27833 128.164 2.0716
  • Inline SVG · 9,956 B (2.4%) — unnamedopens <path d="M128.467 2.735C128.393 2.50167 128.292 2.27833 128.164 2.0716
  • Inline SVG · 9,956 B (2.4%) — unnamedopens <path d="M128.467 2.735C128.393 2.50167 128.292 2.27833 128.164 2.0716
  • Inline SVG · 4,225 B (1%) — unnamedopens <g clip-path="url(#clip0_36_336)"><path d="M42.89 0C43.4259 0 43.9288
Section average over 2 scans on this score version · held at 68 since 2026-11-05
0501002026-10-27 16:05 · 70/100 · 6 findings2026-11-12 04:33 · 68/100 · 9 findings2026-10-272026-11-12

What changed

  1. 2026-11-12score moved 72 → 69score-down
  2. 2026-11-12robots.txt named a sitemapresolved
  3. 2026-11-12resolved: the machine files cost more than the pageresolved
  4. 2026-11-12a new finding appeared: whether a sitemap exists could not be determined — every path was refusedappeared
  5. 2026-11-12a new finding appeared: the origin returned a server error on the sitemap pathappeared
  6. 2026-11-12a new finding appeared: robots.txt is unreadableappeared
  7. 2026-10-22score moved 72 → 74score-up
  8. 2026-10-06a sitemap became readableresolved
  9. 2026-10-06robots.txt no longer named a sitemap againreturned
  10. 2026-10-06a new finding appeared: the machine files cost more than the page againreturned
  11. 2026-10-02score moved 70 → 66score-down
  12. 2026-10-02robots.txt named a sitemapresolved

10 scans on record, first 2026-09-18. Score movements are shown only between readings taken under the same score version; finding codes are comparable across all of them.

Is this your site?

This listing reads the public record. Claim it and Watch keeps that record on your terms: every list this page holds back, finding age, and twice-daily re-measurement. Free for 30 days, no card.

Claim the record for bentoml.com

Is this your company?

This listing came from a fetch, not a submission, so nobody was asked. If bentoml.com is yours you can prove it in a minute and for nothing — a DNS TXT record or one meta tag on the home page, both checkable by anyone.

Claiming marks you as the verified owner and lets you correct the category and the company name. It moves no score, no rank and no position.

Claim this listing — free or ask about a verified profile

Dispute or re-scan

If a reading here is wrong, it is our instrument that is wrong, and we want to know. Tell us what is wrong and the site is re-measured; the result is published as measured. Nothing about a listing, a payment or a request changes a number.

By email instead: hello@crawlcheck.io. Category “ML platforms and experiment tracking” is our label for what the product is primarily sold as; it is not scored. Back to the directory.