About FrankBot

FrankBot is the crawler behind FR^NK, a content intelligence platform operated by Digital News Agency Ltd. If you foundFrankBot/1.0 in your logs, this page explains what it was doing there.

What it reads

FrankBot fetches pages that are already accessible to the public. It presents no credentials, defeats no paywalls and submits no forms. It reads a site when one of our clients analyses their own site, their market or their competitors, and it stores findings about pages rather than mirroring sites.

The AI-visibility check

As part of an audit, FrankBot sends a small number of extra requests that identify themselves with the user agents of well-known AI crawlers, such as GPTBot and ClaudeBot. Those requests test whether a site blocks AI crawlers; they come from our servers, not from those companies, and the result is reported to the site being audited.

Load

Requests are bounded per audit and retried with backoff. A typical audit fetches a few dozen pages once, not continuously.

Questions or blocking

If you want your site excluded from a specific client's analysis, or have any question about a request you saw, write to us through the contact form on this site and include your domain. Blocking the literal user agent stringFrankBot at your edge also works.