CrawlCheck

Findings · 2026-11-21 · By

A staff login could read visitor IP addresses. We closed it.

Three older CrawlCheck admin views showed a staff login every visitor's full IP address. Since 19 November 2026 staff see the network and only the owner sees the address. Here is what changed and what the audits caught.

Until 18 November 2026, three older CrawlCheck admin views showed a staff login the full IP address of every visitor. Since 3:41 AM Mountain Time on 19 November 2026, 4 hours and 15 minutes after the decision, every staff-reachable path tested shows only the visitor's network, an IPv4 /24 or an IPv6 /48, and only the owner sees full addresses. Two independent audits found and fixed host names that still spelled the address.

What a staff login could see #

CrawlCheck has two kinds of sign-in for the people who run it. The owner login belongs to the founder. A staff login is a separate account the owner can hand to someone helping with the work, and revoke on its own. Both open the same admin views: the recent-visitor log, the list of who is on the site right now, and the record of requests arriving at the site.

Until 18 November 2026, those three views showed a staff login exactly what they showed the owner, and that included the full IP address of every visitor in them. An IP address is the network address a request comes from; a full one can often be narrowed to a single household or office. Our newer identity views were already owner-only. These three older ones were not.

This post is about what a staff login was able to do, not a record of anyone having done it. We are publishing it because our own data policy says the visitor log is readable only by us, and a staff account seeing full addresses was a wider reading of “us” than a visitor would expect.

The decision #

At 11:26 PM Mountain Time on 18 November the rule was set in one sentence: a staff login never sees a full IP address. Staff keep every view they need for their work. What changes is the detail inside those views.

The owner keeps the full address, because the owner is the one who answers abuse reports and checks whether a request claiming to be a known crawler really came from that crawler's network. A staff member doing the same work needs to know which network a request came from, how often, and what it asked for. None of that needs the last part of the address. The recent-request record itself is, as our trust center lists it, kept for abuse checks and crawler verification and never published or exported.

What changed, view by view #

The fix went live at 11:46 PM Mountain Time the same night. Here is what each view shows now, compared with before.

ViewOwnerStaff before 18 NovStaff now
Recent-visitor logFull addressFull addressNetwork only; per-address daily counts added up per network
On the site nowFull addressFull address and session idNetwork only; session ids removed
Inbound requestsFull addressFull addressNetwork only
Host names that spell the addressAs writtenAs writtenShown as “a host under” the domain, numbers removed
Any other answer to a staff loginUnchangedWhatever the view heldChecked once more on the way out; any full address cut to its network; not cached

“Network only” means the address is shortened to the block it belongs to. For the older address format, IPv4, the last of its four numbers is dropped, so an address in the documentation range such as 203.0.113.9 is shown as 203.0.113.0/24. For the newer format, IPv6, only the first three groups are kept. That is the same reduction we already apply wherever a visitor address appears in public, such as the crawler figures on the dataset page.

Two bugs the fix found in our own masking #

Shortening an address sounds trivial. Writing the tests for it turned up two places where our existing masking was wrong.

First, an IPv6 address written in its short form, where a run of zeros is collapsed to a double colon, came out with part of the host still attached. The documentation address 2001:db8::1 was shortened to 2001:db8:1::/48, which carries the final group of the address into the result. It now reads 2001:db8:0::/48, as it should.

Second, the live presence view stored a short IPv6 address with the network suffix glued onto the whole address instead of onto the shortened one. Both were corrected in the same change, and both have tests that fail on the old code.

What the audit caught that the fix missed #

Every change of this kind gets two independent audits before it is marked done: one reads the rules (does the change reveal anything new, name anyone, or act on an address), and one re-runs the function against the code that is actually live. The rules audit passed at 12:08 AM on 19 November.

The function audit, ten minutes later, did not pass on the first read. It found that the final check on staff answers let through host names that spell out a visitor's address. Reverse DNS is the lookup that turns an address back into a host name, and many internet providers build that name from the address itself. A name such as ip203-0-113-9.example.net, with letters glued to the first number, or host-203.0.113.9.example.net, came through unchanged. A reverse-lookup name for the address still named the last part of it.

That was fixed and live at 12:16 AM. Reverse-lookup names are now cut to the network, and any run of four address-sized numbers inside a host name is replaced, for staff answers only. The owner still reads the names as written.

The audit also corrected a count. The build notes said the new block of tests ran 39 checks. The auditor counted the output: it runs 29. The coverage was what the notes described; the number was wrong, and the record now says 29. On the code that was live before the change, 13 of those checks fail.

The gaps left after the first night #

The function audit listed what was still open, all of it reachable only with a staff login:

  • Answers that stream in pieces, rather than arriving whole, were not checked on the way out.
  • Downloads that are not text, such as archives and exports, were not checked either.
  • An address written after an underscore or after a word and a slash, or with zero-padded numbers such as 203.000.113.009, slipped past the final check.

All three were closed at 3:41 AM Mountain Time on 19 November. Streamed answers are now checked line by line as they go out, with long lines split at a safe point so an address is never cut in half and missed. A non-text download that is not an image, font, audio or video file is refused to a staff login. The pattern now catches the underscore, slash and zero-padded forms, while leaving browser version strings such as Chrome/141.0.0.0 alone, because those name a release, not a machine.

Since 3:41 AM Mountain Time on 19 November 2026, 4 hours and 15 minutes after the decision, every staff-reachable path we tested shows a visitor's network and never the full address. A second pair of audits passed at about 5:00 AM.

What we have not verified #

Every staff-side result above comes from tests run against the live code with simulated staff sessions. We did not create a real staff account on the live site to check the views by hand, so the staff view itself has been verified in testing, not by a person signed in as staff. The owner view was checked live: it still shows full addresses, and the owner lost nothing in the change.

We also kept the parts that already worked. A request with no sign-in, or with a forged staff sign-in, still gets nothing from these views. That was checked live before and after each change.

What a site owner can take from this #

If you run a site with more than one admin account, the question to ask is not who can sign in but what each sign-in can read. Three checks are worth an afternoon:

  • List the views that hold visitor addresses. Logs, live-visitor widgets, form submissions and error reports usually do. Check each one under every role, not only the owner.
  • Mask at the source and again on the way out. A single masking function is easy to test; a final check on everything a lower role receives catches the field nobody remembered, such as a host name that spells the address.
  • Test the short forms. Compressed IPv6, zero-padded numbers and addresses inside host names are where masking code tends to fail, as ours did.

The rule on CrawlCheck is now short enough to state in one line: full visitor addresses are for the owner only. The rest of how we handle access and outside requests is on our security page.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Could a CrawlCheck staff login see full visitor IP addresses?

Until 18 November 2026, yes. Three older admin views, the recent-visitor log, the on-the-site-now list and the inbound request record, showed a staff login the full address of each visitor. Since 3:41 AM Mountain Time on 19 November 2026, every staff-reachable path tested shows only the visitor's network.

What does a staff login see now instead of the full address?

The network the address belongs to. An IPv4 address loses its last number and is shown as a /24 block, and an IPv6 address keeps only its first three groups, a /48. Daily counts are added up per network, session ids are removed, and host names that spell the address are shown only by their domain.

Who can still see full visitor addresses on CrawlCheck?

Only the owner. The owner answers abuse reports and checks whether a request claiming to be a known crawler came from that crawler's network, which can need the full address. The visitor log keeps the last 200 requests, and its per-address count expires after eight days, as the data policy states.

Was any visitor's address misused by a staff account?

This correction describes what a staff login was able to read, not a record of anyone reading it. The point of the change is that the access should not have existed, whether or not it was used.

How was the fix checked?

Two independent audits ran on the live code after each change: one on the rules and one on the function. The first function audit found host names that still spelled the address and a miscounted test total, 39 claimed against 29 run. Both were corrected. The staff view was verified with simulated staff sessions, not by a person signed in as staff.

How can I check my own site's admin roles for the same problem?

List every view that holds visitor addresses, such as logs, live-visitor widgets, form submissions and error reports, and open each one under every role. Mask addresses where they are stored and check again on the way out, and test compressed IPv6, zero-padded numbers and addresses inside host names.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works