AI crawler study · US and international news
Does npr.org block AI crawlers?
npr.org's robots.txt answered our checker with a server error. Checked 21 October 2026.
Under the robots.txt standard, crawlers that get a server error for robots.txt treat the whole site as off limits. Big sites' bot protection often refuses unfamiliar checkers, so this may not be what real crawlers see. Check it live.
From robots.txt rules for the homepage (RFC 9309). robots.txt is a request; this does not show what any AI company does. Part of Kavel's AI crawler study, built with AI.
Show this on a page or README
Is this your site? SiteCheck shows what else AI agents need: readable text, structured data and contact details.