One in 10 county government websites block good bots

Last week we introduced the good bot scan, a tool that tells you whether your website lets a well-behaved bot in. This is the follow-up we promised: what happens when that bot knocks on the door of every county government website in the United States.

We keep a list of 2,858 county, parish, and borough homepages. ScanGovBot scans every one of them, following robots.txt, waiting between visits, and identifying itself on every single request, including the Lighthouse run. Nothing we send looks like a person's browser. In September 2026 we went back through the whole list to answer one question: is the digital front door open?

Why this matters

Government websites operate as a public service. Their whole purpose is to be found and read. The bots these sites turn away are not scrapers hammering the server; they are the tools that make public information usable:

• search engines and the AI assistants people now use to ask "when is the county clerk open" or "how do I appeal my property assessment"
• accessibility and quality checkers, like ours
• link checkers, uptime monitors, archive crawlers and the state and federal tools that inventory government sites

A well-behaved bot announces itself, follows robots.txt, and visits slowly. Blocking it by user agent does nothing to a bot that pretends to be Chrome, which is exactly what the abusive ones do. The only bots a user-agent block reliably stops are the honest ones.

What we scanned

• 100% (2,858 counties) — every U.S. county, parish, and borough homepage, one per county, compiled and checked by hand over the summer of 2026.
• Up to 3 requests per site — a basic HTTP request, a headless browser, and a Google Lighthouse audit, all identified as ScanGovBot.
• September 2026 — checked on ScanGov's regular scan schedule (August or September 2026), plus one fresh request to every homepage on September 13, 2026.

What we learned

• 85.2% (2,435 counties) let the bot in — most of the country is doing fine.
• 9.9% (284 counties) turn an identified, polite bot away — with a firewall, a content delivery network (CDN) rule, or a captcha.
• 3.7% (107 counties) ask bots to stay out in robots.txt — that is the site's call and we honor it, but it has the same effect on the public.
• 13.7% (391 counties) cannot be read by a well-behaved bot, put together.
• Most of the blocking is not a decision a county made. A handful of hosting platforms and CDN defaults account for most of it.

What counts as a block

An audit makes up to three kinds of request, all with the same ScanGovBot identity:

• A basic request. The kind of HTTP request any script, feed reader, link checker or accessibility tool makes. We use it to read the page's title, description, social tags and reading level.
• A headless browser. If the site refuses the basic request, we try again with a headless Chromium browser, still identified as ScanGovBot.
• Lighthouse. Google's page-quality tool, running in headless Chrome, which produces the accessibility, performance, and best-practice scores.

All three requests carry the same name. The basic request and the headless browser send Mozilla/5.0 (compatible; ScanGovBot/1.0; +https://scangov.com/bot), and Lighthouse sends the same string with Lighthouse appended. Anyone reading a server log can see exactly who visited and follow the link to our bot page, which explains what the bot does, how often it visits, and how to keep it out if you want to. That is the whole point of a good bot: you never have to guess.

That gives us four outcomes for the 284 sites that block something:

The remaining 32 counties are not blocking anyone: they have an expired or misissued certificate, a DNS problem, or a server that times out or errors for every visitor.

How the block happens

Three mechanisms cover nearly all of it:

• A plain "403 Forbidden." 169 sites. The server looks at the user agent string, sees it is not a mainstream browser, and refuses. In our earlier probe of these sites, the same request with a browser user agent got a 200. This is user-agent sniffing, not bot detection: it stops the bots that identify themselves and lets through the ones that lie.
• A Cloudflare challenge page. 78 sites. Cloudflare serves a "checking your browser" interstitial to every visitor it does not recognize. A real browser passes it in a second. A basic request cannot. Headless browsers usually cannot either.
• A captcha page. 33 sites. The server answers with HTTP 202 and a page that immediately redirects to a captcha. This one is sneaky: 202 means "accepted," so many tools treat it as success and read the captcha page as if it were the homepage. Ours did too, until we caught it while preparing this post (see "About the data").

Who does the blocking

"Other sites" on Cloudflare, CloudFront, or Akamai means the CDN is the only thing we can see; the customer behind it varies. See "About the data" for how we attribute platforms.

This is the part that matters most. We resolved every blocking site's DNS to find its host, and the blocks cluster tightly:

• 2.5% (71 counties) on Granicus govAccess, a government content management system (CMS) platform fronted by Akamai, return a plain 403 to the bot — one platform, one setting.
• 0.7% (19 counties) on CivicPlus sit behind a Cloudflare challenge. CivicPlus hosts hundreds of county sites that do let us in, so this looks like a per-account or newer-plan default rather than a platform-wide choice.
• 0.7% (19 counties) in Colorado sit on the state's shared platform on Amazon Web Services (AWS) CloudFront, which returns 403 to the bot for every one of them.
• 0.9% (27 counties) on SiteGround hosting get the captcha treatment.
• 3.4% (98 counties) sit behind Cloudflare on other platforms, with either the challenge page or a firewall rule. We can see Cloudflare; we cannot see who set the rule.
• 0.9% (26 counties) share one nginx setup on Google Cloud and return a plain 403. We could not identify the vendor.

In other words, if four or five platforms changed one default, roughly two thirds of the county websites that block good bots would open up, and almost none of those counties would notice anything had changed.

Who's responsible

• Executives
• Digital service teams
• Third-party vendors
• Government leaders
• Website managers
• IT staff
• Hosting providers

About the data

• Blocking tiers come from each site's most recent ScanGov audit, which records whether the basic request, the headless browser, and Lighthouse got through, plus one fresh plain request to every homepage, following redirects and retrying the www. host when the bare host had a certificate error, the same way our scanner does.
• Block mechanism comes from the status code and body of that request, and from an earlier probe (September 7) that compared the bot user agent with a browser user agent, basic requests only. That one-off comparison is the only time we sent a browser user agent; a real scan identifies as ScanGovBot on every request, as documented on our bot page.
• Platform attribution comes from DNS: canonical name (CNAME) chains (for example *.granicusgovaccess.net, *.civicplus.io) and shared address blocks, checked by hand. Where we can only see a CDN, we say so.
• What we got wrong, and fixed. The 33 captcha sites answered with HTTP 202, which our scanner had been treating as a successful fetch, so for a few weeks we computed their scores on the captcha page. We fixed the scanner on September 13: it now recognizes a captcha or browser-check page wherever it appears and records it as a block with its own status code (996, "Challenge page"). If a site challenges both the basic request and the headless browser, we stop and record the block rather than let Lighthouse's Chrome try its luck, because a site that walls off both identified requests has blocked a good bot, whatever a full browser can squeeze through. We re-audited those sites, and they now show on scangov.org as blocked rather than graded. We count them as blocked here. The good bot scan reports the same thing when it runs into one.
• A site's behavior changes: a few that blocked us in August let us in on September 13, and vice versa. Every number is as of the date given.

Raw data

One row per county:

• domain — the county's homepage domain
• agency — the county, parish, or borough name and state
• result — the overall outcome: open, which block tier, robots.txt, or broken
• robots_txt — what the site's robots.txt says: allowed, disallowed, or crawl-delay
• plain_fetch_status — the HTTP status code from the basic request
• plain_fetch_host — whether the bare or www host responded
• block_style — how the site blocks: Cloudflare challenge, a plain 403, a captcha
• headless_browser — whether the headless browser request got through
• lighthouse — the Lighthouse audit's result
• platform — the hosting platform or CDN we attributed
• server_header — the site's Server HTTP header
• cloudflare — whether Cloudflare sits in front of the site
• last_audit — date of the most recent audit

Download CSV

What to do next

• Run the good bot scan on your domain.
• If it fails, look at your CDN and firewall bot settings. "Block automated traffic" and "challenge unknown bots" are common defaults that were never chosen on purpose.
• Allow bots that identify themselves and honor robots.txt, and rate-limit instead of blocking. Cloudflare, Akamai, and AWS all support verified-bot and allow-list rules. Our bot page has everything a firewall administrator needs to allow ScanGovBot by name.
• Use robots.txt for the bots you actually want to keep out, and say so by name.

About ScanGov

ScanGov monitors websites for AI-readiness, accessibility, security, and usability, and helps teams build better digital experiences. Learn more about ScanGov.