sophie/nuke
1
0
Fork 0
Code Packages 1 Activity Actions

Bump ai.robots.txt to v1.34 (#632)

Browse source
This commit is contained in:
Dryusdan 2025-06-08 20:54:47 +02:00 • committed by GitHub
parent 9539668049
commit 281b6c5c00
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 14 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

11
web/static/robots.txt
View file

@ -2,9 +2,11 @@ User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: aiHitBot
User-agent: Amazonbot
User-agent: Andibot
User-agent: anthropic-ai
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: bedrockbot
User-agent: Brightbot 1.0
User-agent: Bytespider
User-agent: CCBot
@ -33,7 +35,6 @@ User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: imgproxy
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-agent: meta-externalagent
@ -47,10 +48,16 @@ User-agent: omgili
User-agent: omgilibot
User-agent: Operator
User-agent: PanguBot
User-agent: Panscient
User-agent: panscient.com
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: QualifiedBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
@ -60,6 +67,8 @@ User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: wpbot
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: YouBot
Disallow: /