AI visibility

Should lenders allow GPTBot and OAI-SearchBot in robots.txt?

Allow the search crawlers that decide whether you can be cited: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot. Training crawlers such as GPTBot and ClaudeBot are a licensing choice that does not affect search citation. Check that your CDN or firewall is not blocking the search bots.

  • Updated 19 November 2026
  • By Ayan Chakraborty

No cost, no obligation - a written audit within 5 working days.

By Ayan Chakraborty, Founder, SBA Innovation Pvt Ltd, Kolkata. Last updated . Reviewed by Ayan Chakraborty.

Should lenders allow GPTBot and OAI-SearchBot in robots.txt?

Allow OAI-SearchBot, because it is the crawler that supports search results in ChatGPT and decides whether your pages can be cited there. GPTBot is a different crawler, used to collect content that may help train models; allowing or blocking it is a licensing decision for your business and does not decide whether ChatGPT search can cite you.

The same logic applies to other vendors: the search crawlers are what matter for citations, and the training crawlers are a separate choice. The most common failure is not robots.txt at all but a CDN or web application firewall that quietly blocks bots, as we found in one lender engagement. After you edit robots.txt, check server logs for real hits from the search crawlers and review firewall rules monthly.

Crawlers and what they affect (confirm in each vendor's documentation)
CrawlerOperatorPurposeAffects search citation?
OAI-SearchBotOpenAISearch results in ChatGPTYes
GPTBotOpenAITraining dataNo
ChatGPT-UserOpenAIFetches pages when a user asksIndirectly
PerplexityBotPerplexitySearch indexYes
Claude-SearchBotAnthropicSearch quality for ClaudeYes
ClaudeBotAnthropicTraining dataNo
GooglebotGoogleSearch, AI Overviews and AI ModeYes
BingbotMicrosoftBing and CopilotYes
A robots.txt allowing the search crawlers (add your own Disallow rules for private paths)
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

What is llms.txt, and does a lender need one?

llms.txt is a proposed plain-text file at the root of a site that lists important pages for language models. It is not a standard that search engines follow: Google has said it does not use llms.txt for Search, and we do not report it as a lever for citations. A lender does not need one.

If you already have one, there is no harm in keeping it current, but spend no more time on it than that, and do not report it to your board as a lever. Time is better spent on what decides citation: letting search crawlers in, putting key facts in plain HTML, writing direct answers, keeping your entity data consistent and measuring over repeated runs. Treat llms.txt as optional housekeeping.

From our client work: the firewall, not robots.txt

In the consumer lending engagement, a CDN bot-management rule was blocking GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended. The rules were rewritten to a deliberate allow-list, and an llms.txt and an agent-readable sitemap were published alongside.

Method: Engagement 031: before-and-after CDN and robots rules compared in a crawler access change log (identifiers removed in the published evidence).

What did not work: Relying on robots.txt alone; the block sat in the CDN rules, so the robots file looked fine while the crawlers were refused.

Read case study 031 Prompt set, baseline, assumptions and footnotes.

Primary sources

FAQ

Frequently asked questions

Does blocking GPTBot stop ChatGPT citing me?

No. GPTBot is a training crawler, while OAI-SearchBot supports ChatGPT search results. You can block GPTBot as a licensing choice and still be cited through OAI-SearchBot, provided it is allowed.

How do I check whether bots can reach my site?

Test with curl using each crawler's user agent, check your CDN and firewall rules and read server logs for real hits from OAI-SearchBot, Claude-SearchBot and PerplexityBot. Do this monthly.

Is Google-Extended needed for AI Overviews?

No. AI Overviews and AI Mode use the Googlebot index. Google-Extended controls certain Gemini training and grounding uses and does not affect Search ranking or AI Overviews eligibility.

Should a lender block training crawlers?

It is a business choice about licensing your content. It does not affect whether search assistants cite you. Decide with your legal and content teams, and record the decision.

Next step

Want this checked against your own campaigns?

The free Growth Audit reviews your funnel, tracking and compliance gaps and returns a written plan within 5 working days.

  • Free of charge
  • Delivered in 5 working days
  • No lock-in