Core conclusion

Amazon sent an official email to webmasters announcing that starting from June 15, 2026, Amazonbot (Amazon’s AI crawler) will fully comply with the robots.txt standard protocol. This is an important milestone in the field of AI crawler management - Amazonbot had previously ignored robots.txt for a long time, forcing a large number of webmasters to manually apply for blocking or deploy anti-crawling tools (such as Anubis) to defend themselves.

Key Points

• Effective date: June 15, 2026 -Affected objects: All operators and SEO practitioners who run content sites
• Core change: Amazonbot's crawling permissions will be completely controlled by robots.txt, and there is no need to manually submit a blocking request to Amazon.
• Background: Amazonbot’s previous behavior of ignoring robots.txt gave birth to an ecosystem of anti-AI crawler tools such as Anubis

Background and trigger events

On May 14, 2026, well-known developer Xe Iaso published an official email from Amazon. In the email, Amazon said: Starting June 15, 2026, Amazonbot's crawling preferences will be "fully managed through industry standard instructions (robots.txt)."

Amazonbot’s behavior has been controversial before. Unlike traditional search engine crawlers such as Googlebot and Bingbot, Amazonbot has long disrespected the Disallow directive in robots.txt. The only way a webmaster can stop Amazonbot from crawling content is through Amazon's manual request form—a process that is neither transparent nor efficient.

Xe Iaso bluntly stated in his blog that it was Amazonbot's "brutal crawling" behavior that prompted him to develop Anubis, an open source AI crawler defense tool. He said in the HN discussion that Amazon's email "gave him a viable business model."

Key Impact

Adaptation suggestions

For sites that use anti-crawling tools such as Anubis

• Check if Anubis's robots.txt configuration already contains Amazonbot
• If you don't plan to block Amazonbot, you can add exemption rules in Anubis

For SEO operators

• Add or update User-agent: Amazonbot rules in robots.txt
• For training data that you do not want to be collected by AI crawlers, use the complete Disallow: /
• Note the difference between Amazonbot and other Amazon crawlers (such as Amazon AdBot)

Baseline configuration example

Tool keywords such as ```bash

robots.txt

User-agent: Amazonbot Disallow: /

User-agent: * Allow: /

![— robots.txt](https://picsum.photos/seed/website-crawler-management/800/400)

##
- [HN](https://news.ycombinator.com/item?id=48140730)
- [Xe Iaso](https://xeiaso.net/notes/2026/amazonbot-respecting-robots-txt/)
- [Amazon Amazonbot](https://developer.amazon.com/amazonbot)

##

 `Anubis`、`n8n` are automatically recognized by the platform.

## Internal link guidance
- Want to protect your content from AI crawlers? Watch the tutorial: [How to add quality gates to your AI automation workflow: A practical guide from output to trustworthy results](https://waytoclawearn.com/tutorials/guide-ai-automation-quality-check-validation)
- Want to build an automated content publishing system? See the case: [OpenClaw + Claude Automated Publishing: $1,500–$2,500/mo Case Study](https://waytoclawearn.com/cases/openclaw-content-empire-20260419)
- Want to automate content distribution with n8n? Watch the tutorial: [How to build an AI content automated distribution system with n8n + ChatGPT: a complete 30-minute tutorial](https://waytoclawearn.com/tutorials/guide-n8n-chatgpt-content-automation-distribution)