By Ayan Chakraborty, Founder, SBA Innovation Pvt Ltd, Kolkata. Last updated . Reviewed by Ayan Chakraborty.
Should lenders allow GPTBot and OAI-SearchBot in robots.txt?
Allow OAI-SearchBot, because it is the crawler that supports search results in ChatGPT and decides whether your pages can be cited there. GPTBot is a different crawler, used to collect content that may help train models; allowing or blocking it is a licensing decision for your business and does not decide whether ChatGPT search can cite you.
The same logic applies to other vendors: the search crawlers are what matter for citations, and the training crawlers are a separate choice. The most common failure is not robots.txt at all but a CDN or web application firewall that quietly blocks bots, as we found in one lender engagement. After you edit robots.txt, check server logs for real hits from the search crawlers and review firewall rules monthly.
| Crawler | Operator | Purpose | Affects search citation? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search results in ChatGPT | Yes |
| GPTBot | OpenAI | Training data | No |
| ChatGPT-User | OpenAI | Fetches pages when a user asks | Indirectly |
| PerplexityBot | Perplexity | Search index | Yes |
| Claude-SearchBot | Anthropic | Search quality for Claude | Yes |
| ClaudeBot | Anthropic | Training data | No |
| Googlebot | Search, AI Overviews and AI Mode | Yes | |
| Bingbot | Microsoft | Bing and Copilot | Yes |
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /What is llms.txt, and does a lender need one?
llms.txt is a proposed plain-text file at the root of a site that lists important pages for language models. It is not a standard that search engines follow: Google has said it does not use llms.txt for Search, and we do not report it as a lever for citations. A lender does not need one.
If you already have one, there is no harm in keeping it current, but spend no more time on it than that, and do not report it to your board as a lever. Time is better spent on what decides citation: letting search crawlers in, putting key facts in plain HTML, writing direct answers, keeping your entity data consistent and measuring over repeated runs. Treat llms.txt as optional housekeeping.
From our client work: the firewall, not robots.txt
In the consumer lending engagement, a CDN bot-management rule was blocking GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended. The rules were rewritten to a deliberate allow-list, and an llms.txt and an agent-readable sitemap were published alongside.
Method: Engagement 031: before-and-after CDN and robots rules compared in a crawler access change log (identifiers removed in the published evidence).
What did not work: Relying on robots.txt alone; the block sat in the CDN rules, so the robots file looked fine while the crawlers were refused.
Read case study 031 Prompt set, baseline, assumptions and footnotes.