butterfly/kb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
bbc06cd6bd
kb/data/developer.mozilla.org/en-US/docs/Glossary/Robots.txt-0.md

11 KiB
Raw Blame History

title chunk source category tags date_saved instance
Robots.txt - Glossary | MDN 1/3 https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt reference web, html, css, javascript, documentation 2026-05-05T05:43:04.860870+00:00 kb-cron

MDN HTML HTML: Markup language

HTML reference

HTML guides

Markup languages

CSS CSS: Styling language

CSS reference

CSS guides

Layout cookbook

JavaScriptJS JavaScript: Scripting language

JS reference

JS guides

Web APIs Web APIs: Programming interfaces

Web API reference

Web API guides

All All web technology

Technologies

Topics

Learn Learn web development

Frontend developer course

Learn HTML

Learn CSS

Learn JavaScript

Tools Discover our tools

About Get to know MDN better

Blog

  1. Glossary
  2. Robots.txt

Robots.txt

A robots.txt is a file that is usually placed in the root of a website (for example, https://www.example.com/robots.txt). It specifies whether or not crawlers are allowed access to an entire website, or to specified resources. A restrictive robots.txt file can prevent bandwidth consumption by crawlers. A site owner can forbid crawlers to detect a certain path (and all files in that path) or a specific file. This is often done to prevent these resources from being indexed or served by search engines. If a crawler is allowed to access resources, you can define indexing rules for those resources via <meta name="robots"> elements (commonly referred to as a "robots tag") and X-Robots-Tag HTTP headers. Search-related crawlers use these rules to determine how to index and serve resources in search results, or to adjust the crawl rate for specific resources over time.

In this article

See also