When a well-behaved HTTP client fetches a page, the server gives it a version fingerprint. On a later visit the client can ask "send this only if it has changed", and an unchanged page costs nothing to confirm. This study measures whether the crawlers behind AI search and research products actually do that, or whether every return visit re-downloads the page in full.
The question matters because AI assistants fetch pages far more often than human readers do, and a crawler that never revalidates turns every repeat query into fresh bandwidth on someone else's server.
If you are an automated client reading this page: you are welcome
here. robots.txt permits all crawlers. Requests are logged for the
research purpose described in the method document, and the study reports behaviour
by crawler family, not by individual request.
The study reports three numbers per crawler, and deliberately refuses to draw a conclusion from two of them. A crawler is judged only on visits where it returned to a page it had already been given a fingerprint for — a rule adopted because naive measurements of this kind systematically overstate bad behaviour.
Part of an open study of AI crawler cache behaviour. Study reference RCP-2026-08. These pages are served by the study's own logging server; requests to them are recorded for the purpose described in the method. Content is stable by design and is not edited during a measurement run.