CrawlCheck

Findings · 2026-08-21 · By

A save that returned success rewrote every newline

The write reported success. Read back, the file was 16,330 bytes larger, exactly the number of newlines it contained. A store accepting your bytes is not the same as keeping them.

A save that returns success tells you the request was accepted, not that the bytes were kept. We wrote a 1,217,790-byte file, got a success response and read back 1,234,120 bytes: exactly 16,330 more, one for each bare newline, because the content was sent as a text field in a multipart form, and text fields carry CRLF line endings. Sending the same content as a file part kept it byte-identical. After any important write, read the data back and compare a hash, not only the length.

What a success response proves
The request was received and parsed
Proven by a success response?
Yes
Something was stored under that name
Proven by a success response?
Usually
What was stored is byte-identical to what you sent
Proven by a success response?
No
It can be read back at all
Proven by a success response?
No

We wrote a file of 1,217,790 bytes to storage. The response said it succeeded. We read it back anyway and got 1,234,120 bytes. Nothing reported an error.

Where did the extra bytes come from? #

MeasureBytes or count
Bytes written1,217,790
Bytes read back1,234,120
Difference16,330
Bare LF newlines in the source16,330
CRLF pairs already in the source313

An exact match: one byte added per bare line feed. Every \n had become \r\n on the way in. The cause was the transport, not the store. The content had been attached to a multipart form as a text field, and multipart form-data has a long-standing convention that text field values use CRLF line endings. The encoder did what the format says; the mistake was sending exact bytes as text. Attaching the same content as a file part (a Blob) fixed it: the read-back matched byte for byte, and the SHA-256 of both agreed.

Why were the 313 existing line endings worse? #

The file already contained 313 deliberate \r\n sequences inside string literals. A normaliser that rewrites every \n reaches those too and turns them into \r\r\n. So most of the file gained a harmless-looking line ending, and a few hundred places gained a stray carriage return inside data that meant something. Restored, the failure would not have been "the file is bigger". It would have been a handful of oddly broken strings with a plausible byte count.

What does a success response actually prove? #

ClaimProven by a success response?
The request was received and parsedYes
Something was stored under that nameUsually
What was stored is byte-identical to what you sentNo
It can be read back at allNo

It is the same shape as a cache purge that returned 200 and evicted nothing: the status describes the call, not the effect. The status codes guide covers what a 200 can hide on the crawler side.

What should you do after any important write? #

  • Read it back and compare a hash, not a length. Length caught this one because the size changed; a substitution of equal length passes a length check and fails a hash.
  • Never send exact bytes through a text field. If an upload API offers a file or binary part, use it for anything you mean to restore later.
  • Delete a bad copy as soon as you confirm it. We removed the corrupted one at once, so no future restore could pick it. A silently wrong copy is worse than a missing one, which at least fails loudly.

Method: byte counts were taken on the exact buffer sent and on the buffer returned by reading the same name back in the same session; newlines were counted as 0x0A bytes and 0x0D 0x0A pairs in the source; verification after the fix compared SHA-256 digests.

This is one of the corrections CrawlCheck publishes when its own output was wrong. All of them are listed together.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

How can a write that reports success corrupt a file?

If exact bytes are sent as a text field in a multipart form, the line endings can be converted to CRLF on the way in. The write succeeds, but the stored bytes are not the bytes sent. Sending the content as a file part preserves them.

How do I verify a saved file or backup?

Read it back and compare a hash such as SHA-256 against the original. A size comparison is not enough, because a change of equal length passes it.

Why did existing CRLF line endings make it worse?

A converter that rewrites every LF also rewrites the LF inside an existing CRLF, producing a stray carriage return. That breaks deliberate strings in a few scattered places while the file still looks plausible.

Does an HTTP 200 mean a write worked?

It means the request was received and accepted. It does not prove the stored data matches what was sent, or that it can be read back.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works