Should a Startup Block AI Crawlers?

The short answer

No, in almost every case. A startup wants to be found, and AI assistants can only cite pages their crawlers can read. Blocking makes sense for publishers protecting paid content, not for a company marketing a product.

AI crawlers at a glance

• Default: Allow them
• Names to know: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended
• Where it is set: robots.txt, and sometimes the CDN or firewall
• Block only: Private, paid or sensitive content

How to check your site

• Read robots.txt. It sits at yoursite.com/robots.txt.
• Check the CDN. Some services, such as Cloudflare, have settings that block AI bots.
• Check the logs. See whether the crawlers visit.
• Test. Ask an assistant to read one of your pages.

What blocking costs

A blocked company can still be mentioned from what other sites say about it. But its own pages cannot be read or cited, so it loses control of the description.

We allow AI crawlers on studiohuncho.com, and publish our facts at /facts and /llms.txt.

Related reading: What Is llms.txt and Does a Startup Need One?, How to Get Recommended by ChatGPT

FAQ

Does blocking stop AI from mentioning us?

No. It only stops the crawler reading your pages.

Is robots.txt binding?

No. It is a request that well-behaved crawlers follow.

Can we allow search but block training?

Partly. Some companies run separate crawlers for search and for training, and each can be set separately.

Building a site that search engines and AI tools can read? Talk to us.