Findings · 2026-08-21 · By VSNARY | Emmanuel Orta

A backup that returned success and quietly rewrote every newline in the file

The write reported success. Read back, the file was 16,330 bytes larger — exactly the number of newlines it contained. A store confirming it accepted your bytes is not the same as confirming it kept them.

We wrote a 1,217,790-byte file to a key-value store. The API returned success: true. We read it back anyway, and got 1,234,120 bytes.

Sixteen thousand three hundred and thirty bytes appeared out of nowhere, and the response that produced them reported no error of any kind.

Where the extra bytes came from

The number is not arbitrary. We counted the newlines in the file we sent:

An exact match. One byte added per bare line feed. Every \n in the file had been rewritten as \r\n somewhere between our process and storage.

The cause was the transport, not the store. We attached the payload as a text field in a multipart form:

form.append("value", sourceString);      // text field — newlines normalised
form.append("value", new Blob([bytes]));  // file part — bytes preserved

Multipart form-data has a long-standing convention that text field values use CRLF line endings. The encoder was not malfunctioning; it was doing exactly what the format says. The mistake was ours — handing binary-exact content to an API designed to carry text.

Switching the same payload to a Blob file part fixed it. Read-back then matched the source byte for byte, and the SHA-256 of both agreed.

Why 313 makes this worse than it looks

The file already contained 313 genuine CRLF sequences — inside string literals, where they were deliberate. A normaliser that rewrites every \n as \r\n reaches those too, and turns \r\n into \r\r\n.

So the corruption was not uniform. Most of the file gained a harmless-looking line ending. A few hundred places gained a stray carriage return inside data that meant something. If this copy had ever been restored, the failure would not have been “the file is bigger”. It would have been a handful of oddly broken strings scattered through a million lines, with a byte count that looked plausible.

The general shape

A write endpoint returning success tells you the request was accepted. It does not tell you that what came back out is what went in. Those are different claims, and only one of them was tested.

This is the same failure we wrote up when a cache purge returned HTTP 200 and evicted nothing: the status code described the API call, not the effect. Here the status described the acceptance, not the fidelity. In both cases the check that catches it costs one extra request.

It is worth being precise about what a successful write does and does not establish:

What we do now

• Read back and compare a digest. Not a length — a hash. Length would have caught this one because the size changed, but a substitution of equal length would pass a length check and fail a hash.
• Never send exact bytes through a text field. If an API offers a binary or file part, that is the one to use for anything you intend to restore later.
• Delete the bad copy immediately. We removed the corrupted snapshot the moment we confirmed it, so no future restore can select it. A silently-wrong backup is worse than a missing one, because a missing one fails loudly at the moment you need it.

The snapshot existed because the same source had gone six days without one while fifteen builds shipped. It would have been a bad week to discover that the one backup in the store was quietly the wrong bytes.

Method

Byte counts were taken with a length check on the exact buffer sent and on the buffer returned by a subsequent read of the same key, in the same session. Newline counts were taken by counting 0x0A occurrences and 0x0D 0x0A pairs in the source. Verification after the fix compared SHA-256 digests of both buffers.

Related findings

All findings · The dataset · How the dataset works