Directory · ML platforms and experiment tracking
MosaicML — what a crawler receives from mosaicml.com
Website measurement, not a product review. This page records what machine clients received from mosaicml.com. It does not evaluate MosaicML software, features, pricing or support.
AI assistants like ChatGPT can read your site, but find little they can quote from your pages.
AI visibility 85/100, measured 2026-10-31 03:35 UTC from outside its network, as each named AI and search crawler and as a browser. An engine can reach and read this site. What is left is giving it something specific to say.
Held at C A medium-severity finding sets the letter.
4
findings
✗
llms.txt on its host
✗
entity map on its host
6
scans on record since 2026-09-18
The three stages, in the order an engine hits a site
Each stage gates the next: an engine that cannot reach a page never reads it, and one that cannot read it never quotes it.
Stage 1 · 40% of the score
Reach
Can a named answer-engine crawler get your pages at all?
Every named crawler was served the same page a browser gets.
- What each crawler getsn/a
- Crawler permissionsn/a
- Machine-file chainn/a
- Speed100
- Server & hosting75
- Internal linksn/a
- Mobilen/a
- Exposed files and secretsn/a
Stage 2 · 30% of the score
Read
Once it has the bytes, can it find the words?
The content is buried in markup, or the files that guide a crawler are missing.
- Page weight & textn/a
- Machine files60
- Sitemapn/a
- Crawl wasten/a
- Accessibility structuren/a
- Image signalsn/a
- Instructions for agentsn/a
- URL naming100
- Error pagesn/a
Stage 3 · 30% of the score
Quote
Is there a specific fact it can state and attribute?
Not measured on this scan; left out rather than counted as zero.
- Structured datan/a
- Local presence & NAPn/a
- Claim consistencyn/a
- Authors & trust signalsn/a
- Linked datan/a
- Quotable contentn/a
- Entity corroborationn/a
- Page subjectn/a
The stage to fix first is read, the lowest of the three.
What to fix
4 findings on this site, 1 of them scored serious. Every one is named below; the top one is shown in full, free. What we saw, why each matters, how to fix it and the bytes behind it are what a licence buys — or we implement the whole list for you.
1 Medium 1 Low 2 Note
1robots.txt does not return 200HTTP 404.MediumSetting
What we saw
HTTP 404. /robots.txt
Why it matters
The response code on robots.txt determines how crawlers treat the whole site. Anything other than 200 or a clean 404 is risky.
How to fix it
Serve robots.txt with status 200 as text/plain. A 404 or 410 means no file: under RFC 9309 a crawler may then fetch everything, so any rule you meant to publish is not in force. A 429 is read by Google as a server error, which pauses crawling: exempt robots.txt from rate limits.
Setting: a setting on the server, CDN, DNS or a plugin — no code and no writing
Site deployed from GitHub
- Add this workflow file as
.github/workflows/crawlcheck.yml(changerobots-pathif robots.txt lives elsewhere). - Run it from the Actions tab: it opens one pull request with the corrected robots.txt, every change listed. Review, merge, deploy.
Paste this
A minimal valid file. Serve it at https://www.mosaicml.com/robots.txt as text/plain with a 200, and let it through any bot or firewall rule.
User-agent: *
Disallow:
Sitemap: https://www.mosaicml.com/sitemap.xmlThe whole corrected robots.txt for www.mosaicml.com · built from the live file, every change listed at the top.
Fix it with a coding agent
Paste into Claude Code, Cursor or Codex, opened on the code that serves this site.
Show prompt
You are working in the codebase (or server/CDN configuration) that serves mosaicml.com. A crawler audit found the problem below. Fix it at the source, keep the change as small as possible, and do not change anything a visitor sees unless the fix requires it. Finding: robots.txt does not return 200 (ROBOTS_NOT_200, medium severity) Page: https://mosaicml.com/robots.txt What was measured: HTTP 404. Why it matters: The response code on robots.txt determines how crawlers treat the whole site. Anything other than 200 or a clean 404 is risky. How to fix it: Serve robots.txt with status 200 as text/plain. A 404 or 410 means no file: under RFC 9309 a crawler may then fetch everything, so any rule you meant to publish is not in force. A 429 is read by Google as a server error, which pauses crawling: exempt robots.txt from rate limits. Rule and examples: https://crawlcheck.io/rules/ROBOTS_NOT_200 Full report (markdown): https://crawlcheck.io/r/pk1qcsy09t.md When you are done: 1. Tell me which files or settings you changed and why. 2. If the fix is a server, CDN or cache setting you cannot change from the code, say exactly where it is set. 3. After deploying, re-scan at https://crawlcheck.io/?url=mosaicml.com . The fix counts when ROBOTS_NOT_200 is gone from the new report.
Why this was decided
| Rule | ROBOTS_NOT_200 · rule pack sv34 · scope core |
|---|---|
| Evaluated | 2026-10-31 03:35:31 UTC |
| Decision | d1:f1a41407d2996d5ef59a · finding f1:93457fe80937294d9bd2 resolve either at /api/graph/<id> with a key |
| Observer | crawlcheck-worker build 77ae0f18 from LAX · audit v4 |
What it rests on
| Identity | Status | Bytes | sha256 | Kept |
|---|---|---|---|---|
| read: /robots.txt | 404 | 906 | 90c032f35272 | already_held |
5 more in the manifest.
Artifacts: manifest:bf6d163d071e70cdf9f3bf0e55d87b5131756088de01aba139e4a165d803f6e2
Grade effect Caps the letter at C. Without it the letter would be A (score 85). Moves no points; the score is built from the measured rows.
Rule ROBOTS_NOT_200 under scoring rules v34 · fingerprint ROBOTS_NOT_200|/robots.txt · persisting (seen in 7 of 7 readings)
Verify the fix Re-scan after the change. This finding is cleared when ROBOTS_NOT_200 is no longer reported at /robots.txt; if it comes back after clearing, it is marked a regression, not a new finding.
ROBOTS_NOT_200 · full transcript
Open the other 3
What we saw, why it matters, how to fix it, and the evidence bytes for every finding — on this report and every re-scan. From $29 a month.
Unlock the findingsOr have them fixed
We implement the list on your site and re-measure, with a sealed before and after. One site, $100 once.
Get this fixed — $100The main product
We fix the 4 findings on mosaicml.com. $100, once.
1 of them are scored serious. One site. You keep the grade for free — this is the part where the list stops being a list.
- Every eligible finding implemented on the site itself: machine files, schema, NAP, settings. Not included: new content, redesigns, plugin or theme replacement, custom integrations (the terms list them)
- An entity map authored for your business, not a template with the name swapped
- Measured again afterwards — the before and the after, sealed, so it is evidenced rather than claimed
- Thirty days of monitoring included
- Work starts within two business days
What it covers · one payment, nothing recurring
What changed
- 2026-10-31the stale edge copy clearedresolved
- 2026-10-31the homepage no longer read as mostly markupresolved
- 2026-10-31resolved: a named crawler was sent somewhere the ordinary client was notresolved
- 2026-10-31no answer engine was refusedresolved
- 2026-10-17the homepage began taking more than one redirect to reachappeared
- 2026-10-17a new finding appeared: a named crawler was sent somewhere the ordinary client was not againreturned
- 2026-10-17an answer engine was refused againreturned
- 2026-10-04resolved: a named crawler was sent somewhere the ordinary client was notresolved
- 2026-10-04no answer engine was refusedresolved
6 scans on record, first 2026-09-18. Score movements are shown only between readings taken under the same score version; finding codes are comparable across all of them.
Is this your site?
This listing reads the public record. Claim it and Watch keeps that record on your terms: every list this page holds back, finding age, and twice-daily re-measurement. Free for 30 days, no card.
Claim the record for mosaicml.comIs this your company?
This listing came from a fetch, not a submission, so nobody was asked. If mosaicml.com is yours you can prove it in a minute and for nothing — a DNS TXT record or one meta tag on the home page, both checkable by anyone.
Claiming marks you as the verified owner and lets you correct the category and the company name. It moves no score, no rank and no position.
Claim this listing — free or ask about a verified profile
Dispute or re-scan
If a reading here is wrong, it is our instrument that is wrong, and we want to know. Tell us what is wrong and the site is re-measured; the result is published as measured. Nothing about a listing, a payment or a request changes a number.
By email instead: hello@crawlcheck.io. Category “ML platforms and experiment tracking” is our label for what the product is primarily sold as; it is not scored. Back to the directory.