sophie/nuke
1
0
Fork 0
Code Packages 1 Activity Actions

fix(robots2policy): handle multiple user agents under one block (#925)

Browse source
This commit is contained in:
Jason Cameron 2025-09-06 22:35:19 -04:00 • committed by GitHub
parent 87c2f1e0e6
commit 82099d9e05
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 208 additions and 58 deletions
Download patch file Download diff file Expand all files Collapse all files

25
cmd/robots2policy/testdata/consecutive.robots.txt vendored Normal file
View file

@ -0,0 +1,25 @@
# Test consecutive user agents that should be grouped into any: blocks
User-agent: *
Disallow: /admin
Crawl-delay: 10
# Multiple consecutive user agents - should be grouped
User-agent: BadBot
User-agent: SpamBot
User-agent: EvilBot
Disallow: /
# Single user agent - should be separate
User-agent: GoodBot
Disallow: /private
# Multiple consecutive user agents with crawl delay
User-agent: SlowBot1
User-agent: SlowBot2
Crawl-delay: 5
# Multiple consecutive user agents with specific path
User-agent: SearchBot1
User-agent: SearchBot2
User-agent: SearchBot3
Disallow: /search