sophie/nuke
1
0
Fork 0
Code Packages 1 Activity Actions

Bump ai.robots.txt to v1.37 (#689)

Browse source 
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
This commit is contained in:
Dryusdan 2025-06-18 19:30:53 +02:00 • committed by GitHub
parent 97c1d4f353
commit 1d5fa49eb0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 11 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

8
web/static/robots.txt
View file

@ -21,7 +21,9 @@ User-agent: Cotoyogi
User-agent: Crawlspace
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: EchoboxBot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: FriendlyCrawler
@ -42,6 +44,7 @@ User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User/1.0
User-agent: MyCentralAIScraperBot
User-agent: NovaAct
User-agent: OAI-SearchBot
User-agent: omgili
@ -54,12 +57,17 @@ User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: Poseidon Research Crawler
User-agent: QualifiedBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-CT
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: Sidetrade indexer bot
User-agent: TikTokSpider