sophie/nuke
1
0
Fork 0
Code Packages 1 Activity Actions

Bump AI-robots.txt rules to version 1.31 (#538)

Browse source 
* Bump AI-robots.txt rules to version 1.31

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
Dryusdan 2025-05-23 18:15:12 +02:00 • committed by GitHub
parent c78d830ecb
commit 11081aac08
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 11 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

5
web/static/robots.txt
View file

@ -9,6 +9,8 @@ User-agent: Brightbot 1.0
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: cohere-ai
@ -21,6 +23,7 @@ User-agent: FacebookBot
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: FriendlyCrawler
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
@ -37,6 +40,7 @@ User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User/1.0
User-agent: NovaAct
User-agent: OAI-SearchBot
User-agent: omgili
@ -55,6 +59,7 @@ User-agent: TikTokSpider
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: wpbot
User-agent: YouBot
Disallow: /