Good bot scan

What is a good bot?

• It visits slowly and follows the rules in your robots.txt file.
• It gives its name on every visit, so you can see who is there.
• It has a page that explains who runs it and why.

About

This tool checks whether ScanGovBot can reach your website. It looks at three things:

• Your robots.txt file, to see whether it allows ScanGovBot in and how long it asks the bot to wait between visits
• A basic request, like the one your browser makes
• A full browser, in case the site turns away the basic request

If we cannot get in, we cannot scan your site. The site probably turns away other good bots too.

This tool only tells you about ScanGovBot. Another good bot may get a different result on the same site. If your site blocked us, check your robots.txt file and your firewall, web application firewall (WAF), and content delivery network (CDN) bot settings. The block may be a setting no one chose.

Common issues

• Your robots.txt file tells ScanGovBot to stay out, or asks it to wait too long between visits
• A firewall blocks visits that do not come from a person
• A captcha or "checking your browser" page is shown instead of the page (status 996)
• The server sends back an error, such as 403 or 503
• The site has a security certificate problem
• The site takes too long to answer

Access isn't all-or-nothing

A failed check does not always mean your site blocks every good bot. Most sites let some bots in and keep others out in these ways:

• robots.txt: your file can name one bot and tell it to stay out, while the rest are welcome.
• Allow lists: the site lets in a few well-known bots and blocks the rest.
• Block lists: the site names and blocks one bot, while the others still get in.
• Firewall defaults: many firewalls and CDNs block bots they do not recognize, even if no one asked them to.
• Traffic limits: a bot you allow can still be shut out for visiting too often, or for the network it comes from.

ScanGovBot

Learn more about ScanGovBot.