How to Enable and Disable Crawlers in Squarespace

Every public website can be accessed by both people and automated bots known as crawlers. These bots scan and index your content for different purposes depending on who operates them. Search engines like Google use crawlers to include your site in search results, while newer AI companies may scan your content to train their models. Squarespace gives you a built‑in way to request that certain AI crawlers avoid your site by adding exclusions to your robots.txt file.

A few things to understand before adjusting these settings:

• Requesting that AI crawlers avoid your site doesn’t guarantee compliance, but it’s currently the strongest available option.

• Blocking AI crawlers may reduce your visibility and traffic.

• Squarespace does not earn revenue from any third‑party crawler activity.

• If you want to hide your site from search engines entirely, use Squarespace’s search‑visibility settings instead.

Excluding Your Site From Known AI Crawlers

To request AI crawlers not scan your site:

Open the Settings panel. Click Crawlers. Check the box next to “Block known artificial intelligence crawlers."

When you enable this option, Squarespace updates your robots.txt file to request that the following bots do not crawl your site:

AI2Bot Ai2Bot-Dolma aiHitBot Amazonbot anthropic-ai Applebot-Extended Bytespider CCBot ClaudeBot cohere-ai cohere-training-data-crawler DuckAssistBot FacebookBot Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video GPTBot img2dataset Meta-ExternalAgent MyCentralAIScraperBot omgili omgilibot Quora-Bot TikTokSpider YouBot

As AI evolves, Squarespace may expand this list. You can also request additional bots through customer support.

Important: Blocking AI crawlers does not remove any content they may have collected before you enabled this setting.

Understanding Robots.txt

Your robots.txt file is a publicly accessible document that well‑behaved crawlers read to understand what they should or shouldn’t scan. Historically, this file was used to manage search‑engine indexing, but today it also serves as a way to request exclusion from AI crawlers. However, robots.txt is only a request—malicious or non‑compliant crawlers may ignore it.

The only guaranteed way to prevent crawlers from accessing your content is to make your site private.

Why the Setting Isn’t Enabled by Default

Crawlers have always scanned websites unless specifically told not to. Many of them, including search engines and AI tools, help drive traffic and visibility. Because blocking AI crawlers could reduce your reach, Squarespace leaves the setting off by default so site owners can decide what’s best for their goals.

There is currently no way to block AI training while still allowing AI‑powered search or chatbot visibility from the same company.

Reasons You Might Leave the Setting Unchecked

Keeping your site open to crawlers generally increases your chances of being discovered. Search engines rely on crawling to index your content, and many AI tools now provide backlinks or citations that can bring new visitors to your site.

For example, if someone asks an AI assistant for the best restaurant in a city, you’d likely want your business included in the answer.

Note: It’s not currently possible to block AI crawlers on specific pages only—your request applies to the entire site.

Why Squarespace Calls Them “Known Artificial Intelligence Crawlers”

There is no universal standard for blocking AI crawlers. Each AI company must individually support and honor robots.txt instructions. The list in your settings includes only the crawlers that have publicly agreed to follow these rules.

Need help with adjusting crawler settings or need an expert to make a decision for you, get in touch with a free consultation!