What is an AI crawler?
- Short for
- AI crawler (AI bot)
- Topic
- AI search
- Read
- 2 minutes
- Updated
- 19 November 2026
An AI crawler is a bot that AI companies send to read web pages1, either to answer a question right now or to train future models2. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are the names most sites see. If a crawler cannot read your pages, the assistant behind it cannot cite them3.
- 1
Crawlers fetch the page HTML. Most do not run scripts.
- 2
Some companies run separate crawlers for search and for training.
- 3
No read, no quote, no link back to your page.
Why do AI crawlers matter for a startup?
AI assistants only quote pages their crawlers can reach. A startup that blocks them by accident, often through a CDN or firewall setting, can still be mentioned from what other sites say, but loses control of how it is described.
How do you check what AI crawlers see on your site?
4 parts, in the order most teams tackle them.
- a.Read robots.txt
It sits at yoursite.com/robots.txt and says which bots may visit.
- b.Check the CDN
Some services, such as Cloudflare, have settings that block AI bots.
- c.Check the logs
See whether the crawlers actually visit, and which pages they read.
- d.Test a page
Ask an assistant to read one of your pages and summarise it.
What does this look like in practice?
What are the common mistakes?
- ✕
Blocking every bot by default because a CDN template did it.
- ✕
Assuming a block stops AI tools from mentioning you. It only stops them reading your pages.
- ✕
Putting key copy where crawlers cannot see it, inside images or scripts.
See also whether AI assistants can read JavaScript websites.
robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may visit.
AI search13Server-side renderingServer-side rendering means the server sends a page as finished HTML, so the text is there before any script runs.
AI search03llms.txtllms.txt is a short Markdown file at the root of a website that tells AI tools what the company does and which pages matter most.
AI searchNeed this done for your startup?
Studio Huncho is a creative studio in Bengaluru for funded AI and SaaS startups: brand, website, product and launch videos. Share a brief and we reply with a scoping doc within 48 hours.
.webp)
