kb/data/developer.mozilla.org/en-US/docs/Glossary/Crawler-0.md

title	chunk	source	category	tags	date_saved	instance
Crawler - Glossary | MDN	1/3	https://developer.mozilla.org/en-US/docs/Glossary/Crawler	reference	web, html, css, javascript, documentation	2026-05-05T05:28:39.870629+00:00	kb-cron

• Skip to main content
• Skip to search

MDN HTML HTML: Markup language

HTML reference

• Elements
• Global attributes
• Attributes
• See all…

HTML guides

• Responsive images
• HTML cheatsheet
• Date & time formats
• See all…

Markup languages

• SVG
• MathML
• XML

CSS CSS: Styling language

CSS reference

• Properties
• Selectors
• At-rules
• Values
• See all…

CSS guides

• Box model
• Animations
• Flexbox
• Colors
• See all…

Layout cookbook

• Column layouts
• Centering an element
• Card component
• See all…

JavaScriptJS JavaScript: Scripting language

JS reference

• Standard built-in objects
• Expressions & operators
• Statements & declarations
• Functions
• See all…

JS guides

• Control flow & error handing
• Loops and iteration
• Working with objects
• Using classes
• See all…

Web APIs Web APIs: Programming interfaces

Web API reference

• File system API
• Fetch API
• Geolocation API
• HTML DOM API
• Push API
• Service worker API
• See all…

Web API guides

• Using the Web animation API
• Using the Fetch API
• Working with the History API
• Using the Web speech API
• Using web workers

All All web technology

Technologies

• Accessibility
• HTTP
• URI
• Web extensions
• WebAssembly
• WebDriver
• See all…

Topics

• Media
• Performance
• Privacy
• Security
• Progressive web apps

Learn Learn web development

Frontend developer course

• Getting started modules
• Core modules
• MDN Curriculum
• Check out the video course from Scrimba, our partner

Learn HTML

• Structuring content with HTML module

Learn CSS

• CSS styling basics module
• CSS layout module

Learn JavaScript

• Dynamic scripting with JavaScript module

Tools Discover our tools

• Playground

• HTTP Observatory

• Border-image generator

• Border-radius generator

• Box-shadow generator

• Color format converter

• Color mixer

• Shape generator

About Get to know MDN better

• About MDN

• Advertise with us

• Community

• MDN on GitHub

Blog

1. Glossary
2. Crawler

Crawler

A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages. Typically search engines (e.g., Google, Bing, etc.) use crawlers to build indexes.

In this article

• See also

See also

• Related glossary terms:
  • Search engine
• Web crawler on Wikipedia