Bot support

TKLinkCheckBot

TKLinkCheckBot is the link checker run by Twelve Kites for Link Health, a service that finds broken links on our clients’ websites.

If you’ve seen it in your logs, it’s because a website we look after links to a page on yours, and we’re checking that the link still works.

What it does

  • It requests only the specific URLs that our clients’ pages link to. It doesn’t crawl your site, follow links on your pages, or try to discover other content.
  • It usually sends a single HEAD request. If that doesn’t give a clear answer, it sends a GET and reads at most the first 64 KB of the response before closing the connection.
  • It records only what it needs to tell whether the link works: the HTTP status, any redirects, and the final URL. It doesn’t store, index, or reuse your content, and it isn’t used to train AI models.
  • It rechecks a working link about once a month, and a failing link a few times over a day or two before reporting it as broken.

How it behaves

  • It’s slow by design. It never has more than one request open to your site at a time, and waits at least two seconds between requests to the same site.
  • It obeys robots.txt. It reads your /robots.txt and follows the rules for the user agent TKLinkCheckBot, or for * if there are none for it specifically. URLs you disallow are never requested.
  • It honours Crawl-delay for TKLinkCheckBot or *, up to 60 seconds.
  • It backs off when your server answers 429 Too Many Requests or 503 Service Unavailable, or sends a Retry-After header, waiting longer each time (up to a day).
  • It doesn’t try to get round bot protection. If it meets a challenge page, it stops, and a person checks the link in an ordinary browser instead.

How to recognise it

Requests carry this user agent:

TKLinkCheckBot/1.0 (+https://12kites.com/tklinkcheckbot)

TKLinkCheckBot runs on cloud infrastructure, so its IP addresses change. We use cryptographically signed requests (Web Bot Auth) so that it can be verified without relying on IP addresses.

How to stop it, or slow it down

To stop it checking any URL on your site, add this to your /robots.txt:

User-agent: TKLinkCheckBot
Disallow: /

To slow it down instead:

User-agent: TKLinkCheckBot
Crawl-delay: 10

It rereads robots.txt at least once a day, so changes take effect within 24 hours.

Contact

If TKLinkCheckBot is causing you a problem, or you have a question about it, email [email protected]. Please include the URL it requested and the time, if you have them.