Reading frank.online

About FrankBot

FrankBot is the crawler behind FR^NK, an SEO and AI visibility tool operated by Frank Online Ltd. If you found FrankBot/1.0 in your logs, this page explains what it was doing there. Its full user agent is:

FrankBot/1.0 (+https://frank.online/crawler)

What it reads

FrankBot fetches pages that are already accessible to the public. It presents no credentials, defeats no paywalls and submits no forms. It reads a site when one of our clients analyses their own site, their market or their competitors, and it stores findings about pages rather than mirroring sites.

The AI-visibility check

FrankBot does not send requests under other crawlers' names. To tell whether AI engines can read a site, FR^NK reads the site's robots.txt and asks the engines themselves, which reach the site from their own networks.

Opting out

The robots.txt token is FrankBot. The FR^NK free audit does not run on a site whose robots.txt carries this group, and does not open it as a competitor:

User-agent: FrankBot
Disallow: /

Load

Requests are bounded per audit and retried with backoff. A typical audit fetches a few dozen pages once, not continuously.

Questions or blocking

If you want your site excluded from anything else, or have any question about a request you saw, write to us at [email protected] or through frank.online/talk, and include your domain. We add it to our exclusion list. Blocking the literal user agent string FrankBot at your edge also works.