Glossary

robots.txt

Definition

The robots.txt file is a file on your website. In it, you specify which areas search engines and AI crawlers are allowed to visit. For example, you can allow Google access while excluding certain AI crawlers. Important: The robots.txt file does not protect confidential content and does not automatically remove pages from search results.

Written by

Ortwin Oberhauser

Co-Founder, CTO, and Digital Visibility Strategist