What is robots.txt?
- Short for
- robots.txt file
- Topic
- AI search
- Read
- 2 minutes
- Updated
- 19 November 2026
robots.txt is a plain text file at the root of a website1 that tells crawlers which parts of the site they may visit. It follows the Robots Exclusion Protocol, published as an internet standard (RFC 93092) in 2022. It is a request that well-behaved crawlers follow, not a lock3.
- 1
It always lives at yoursite.com/robots.txt.
- 2
The Robots Exclusion Protocol, written up as an internet standard in September 2022.
- 3
Anyone can read the file, and bad bots can ignore it.
Why does robots.txt matter for a startup?
The same file now decides whether search engines and AI assistants can read your pages. One wrong line can hide a whole site from the tools buyers use to shortlist companies, so it is worth checking after every launch or migration.
What goes in a robots.txt file?
4 parts, in the order most teams tackle them.
- a.User-agent
The crawler a rule applies to, such as GPTBot, or * for every crawler.
- b.Disallow
Paths that crawler should not visit, such as /admin.
- c.Allow
Exceptions inside a disallowed path.
- d.Sitemap
The address of your sitemap.xml, so crawlers find every page.
What does this look like in practice?
Our own robots.txt at studiohuncho.com/robots.txt blocks nothing. It has one line, pointing crawlers to the sitemap.
From our work
What are the common mistakes?
- ✕
Leaving a staging rule such as "Disallow: /" in place after launch.
- ✕
Using robots.txt to hide private pages. The file is public and is only a request.
- ✕
Blocking AI search crawlers when you only meant to opt out of training.
An AI crawler is a bot that AI companies send to read web pages, either to answer a question right now or to train future models.
AI search03llms.txtllms.txt is a short Markdown file at the root of a website that tells AI tools what the company does and which pages matter most.
AI search04Schema markupSchema markup is structured data in a page's code that labels what the page is about: the company, the article, the questions it answers, the people behind it.
AI searchNeed this done for your startup?
Studio Huncho is a creative studio in Bengaluru for funded AI and SaaS startups: brand, website, product and launch videos. Share a brief and we reply with a scoping doc within 48 hours.
.webp)
