TechScoutBot
The crawler behind TechScout. It visits a site's homepage to detect the technologies it's built with — CMS, frameworks, analytics, CDN, payments — for our public technology lookup.
How it behaves
- Identifies itself with the user agent
TechScoutBot. - Requests a single page per site — it does not spider your whole site.
- Caches results, so it won't re-request the same site for weeks.
- Honors
robots.txtand backs off when a site returns a rate-limit response.
Excluding TechScoutBot
To ask it not to crawl your site, add this to your robots.txt:
User-agent: TechScoutBot Disallow: /
It picks up the change on its next visit. You can also request removal of a specific listing from that site's profile page.