What is robots.txt?

What is robots.txt?
Short for
robots.txt file
Topic
AI search
Read
2 minutes
Updated
19 November 2026
Definition

robots.txt is a plain text file at the root of a website1 that tells crawlers which parts of the site they may visit. It follows the Robots Exclusion Protocol, published as an internet standard (RFC 93092) in 2022. It is a request that well-behaved crawlers follow, not a lock3.

  1. 1

    It always lives at yoursite.com/robots.txt.

  2. 2

    The Robots Exclusion Protocol, written up as an internet standard in September 2022.

  3. 3

    Anyone can read the file, and bad bots can ignore it.

01

Why does robots.txt matter for a startup?

The same file now decides whether search engines and AI assistants can read your pages. One wrong line can hide a whole site from the tools buyers use to shortlist companies, so it is worth checking after every launch or migration.

02

What goes in a robots.txt file?

4 parts, in the order most teams tackle them.

  1. a.User-agent

    The crawler a rule applies to, such as GPTBot, or * for every crawler.

  2. b.Disallow

    Paths that crawler should not visit, such as /admin.

  3. c.Allow

    Exceptions inside a disallowed path.

  4. d.Sitemap

    The address of your sitemap.xml, so crawlers find every page.

03

What does this look like in practice?

Our own robots.txt at studiohuncho.com/robots.txt blocks nothing. It has one line, pointing crawlers to the sitemap.

From our work
04

What are the common mistakes?

  • Leaving a staging rule such as "Disallow: /" in place after launch.

  • Using robots.txt to hide private pages. The file is public and is only a request.

  • Blocking AI search crawlers when you only meant to opt out of training.

Full guideRead our guide: should a startup block AI crawlers?
Related termsAll 30 terms
11AI crawler

An AI crawler is a bot that AI companies send to read web pages, either to answer a question right now or to train future models.

AI search
03llms.txt

llms.txt is a short Markdown file at the root of a website that tells AI tools what the company does and which pages matter most.

AI search
04Schema markup

Schema markup is structured data in a page's code that labels what the page is about: the company, the article, the questions it answers, the people behind it.

AI search
Clutch★★★★★4.9 from 8 verified reviews

Need this done for your startup?

Studio Huncho is a creative studio in Bengaluru for funded AI and SaaS startups: brand, website, product and launch videos. Share a brief and we reply with a scoping doc within 48 hours.

VIEW CASE STUDY