What RFC standard governs robots.txt parsing?

RFC 9309 (Robots Exclusion Protocol) is the formal internet standard establishing exact matching priority rules, wildcard handling, and longest-prefix precedence.

When both Allow and Disallow match with the same length, which wins?

Under RFC 9309 section 2.2.2, when an Allow and Disallow directive match the same number of path characters, the Allow rule takes precedence.

QuickCoderHub

Online robots.txt Tester & Validator (RFC 9309)

Test your website's robots.txt rules against web search crawlers and AI bots. Verify path patterns, wildcards (*), end anchors ($), and rule precedence without leaving your browser.

1. robots.txt Content

2. Test Specific URL Path

Batch URL Test Matrix

Test URL Path	Status	Matched Directive

RFC 9309 Robots Exclusion Protocol Standards

The robots.txt file instructs web robots and search engine crawlers which parts of a site should not be accessed. Standardized by RFC 9309 in 2022, modern crawlers follow strict path-matching algorithms:

Key Matching Rules

Longest Match Priority: If multiple directives match a given URL, the rule with the longest path length takes precedence (e.g., Allow: /p/a/ beats Disallow: /p/).
Tie-Breaking: When an Allow and a Disallow rule match the exact same number of characters, the Allow directive wins.
Pattern Operators: The asterisk (*) matches any sequence of 0 or more characters, while the dollar sign ($) anchors the match to the exact end of the URL.