What is an AI crawler?

What is an AI crawler?
Short for
AI crawler (AI bot)
Topic
AI search
Read
2 minutes
Updated
19 November 2026
Definition

An AI crawler is a bot that AI companies send to read web pages1, either to answer a question right now or to train future models2. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are the names most sites see. If a crawler cannot read your pages, the assistant behind it cannot cite them3.

  1. 1

    Crawlers fetch the page HTML. Most do not run scripts.

  2. 2

    Some companies run separate crawlers for search and for training.

  3. 3

    No read, no quote, no link back to your page.

01

Why do AI crawlers matter for a startup?

AI assistants only quote pages their crawlers can reach. A startup that blocks them by accident, often through a CDN or firewall setting, can still be mentioned from what other sites say, but loses control of how it is described.

02

How do you check what AI crawlers see on your site?

4 parts, in the order most teams tackle them.

  1. a.Read robots.txt

    It sits at yoursite.com/robots.txt and says which bots may visit.

  2. b.Check the CDN

    Some services, such as Cloudflare, have settings that block AI bots.

  3. c.Check the logs

    See whether the crawlers actually visit, and which pages they read.

  4. d.Test a page

    Ask an assistant to read one of your pages and summarise it.

03

What does this look like in practice?

We allow AI crawlers on studiohuncho.com and keep our company facts in two places they read easily: /facts and /llms.txt.

From our work
04

What are the common mistakes?

  • Blocking every bot by default because a CDN template did it.

  • Assuming a block stops AI tools from mentioning you. It only stops them reading your pages.

  • Putting key copy where crawlers cannot see it, inside images or scripts.

Full guideRead our guide: should a startup block AI crawlers?

See also whether AI assistants can read JavaScript websites.

Related termsAll 30 terms
12robots.txt

robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may visit.

AI search
13Server-side rendering

Server-side rendering means the server sends a page as finished HTML, so the text is there before any script runs.

AI search
03llms.txt

llms.txt is a short Markdown file at the root of a website that tells AI tools what the company does and which pages matter most.

AI search
Clutch★★★★★4.9 from 8 verified reviews

Need this done for your startup?

Studio Huncho is a creative studio in Bengaluru for funded AI and SaaS startups: brand, website, product and launch videos. Share a brief and we reply with a scoping doc within 48 hours.

VIEW CASE STUDY