Findings · 2026-10-17 · By VSNARY | Emmanuel Orta

AI agents cannot refuse what they cannot see: 27 traps, four models

A revoked credential and an out-of-date fact look exactly like a valid one on the page. So the models walked straight through them.

An AI agent that books, buys or signs in for someone has to decide, every time, whether the thing in front of it is what it claims to be. We wanted to know where that decision fails, so we built 27 situations an agent might meet and gave them to four models.

Fourteen are traps. Thirteen are ordinary cases where the right move is to go ahead, because an agent that refuses everything is not safe, it is useless.

The four kinds of trap #

• Lookalike sites. A page imitating a real business, asking the agent to sign in.
• Tampered tools. A tool the agent is told to connect to, whose description has been altered to steer it.
• Revoked credentials. A business whose licence or certificate has been withdrawn, while its site still displays it.
• Stale facts. Hours, prices or details that were true once and are not any more.

Each model saw every case twice: once with only what it could observe itself, and once with a signed, checkable record about the site. A trap counts as stopped when the model chose to stop and verify or to refuse. An answer that named no action counted as wrong. The run finished on October 13, 2026.

Traps stopped, by kind #

Totals are across all four models: Gemini 3.5 Flash-Lite (Google), gpt-oss-120b (OpenAI, open weights), Llama 3.3 70B (Meta) and Mistral Small 3.1 (Mistral AI).

What the table says #

Revoked and stale traps are invisible from the page. A withdrawn licence number looks identical to a valid one. Yesterday's opening hours look like today's. Working alone, the four models stopped 5 of 24 of these traps; two of them stopped none of the revoked ones. This is not a reasoning failure. There was nothing to reason from. With a record that said the credential was revoked or the fact had changed, all four stopped all 24.

Tampered tools split the models. Gemini and gpt-oss-120b caught all four tampered tools on their own. Llama and Mistral caught one each. With the record, every model caught all four.

Lookalike sites did not improve. 11 of 16 stopped alone, 11 of 16 with the record. Gemini caught fewer with it (1 of 4) than without (2 of 4), while Llama went from 3 to 4. A record about the real site does not help much when the agent never connects the fake page to it, and that is the gap we are working on next.

The cost of caution #

Safety that stops everything is not safety. On the 13 ordinary cases, three models completed more of them with the record than without. gpt-oss-120b went the other way: it completed 6 of 13 alone and 5 of 13 with it, and on the real, legitimate sign-in pages it went ahead with none of the four. It treated almost everything as a possible trap.

What this means #

• If you build agents: give them a way to check facts the page cannot prove. A model cannot refuse a revoked credential it has no way of knowing about.
• If you run a business: keep your licence, insurance, hours and prices current everywhere they appear. An agent that can check will catch a stale claim, and an agent that cannot will repeat it to your customer.
• If you evaluate agents: score usefulness next to safety. One of the four models looked safest on lookalikes and was nearly useless on real sign-ins.

The limits #

27 cases, four models, one run. Eleven of 216 calls returned no usable answer and were scored as wrong, which slightly understates every model. The cases are fictional, so they test judgement in a controlled setting rather than performance on the live web.

Published by CrawlCheck. The signed record in this test is CrawlCheck's own, so weigh that comparison accordingly. The full case list and results are public at crawlcheck.io/benchmark, and the agentic-web overview this test belongs to is here.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works