57c0b2b22c
Perplexity has some proper documentation available for their crawlers, with published IP addresses: https://docs.perplexity.ai/guides/bots. Signed-off-by: Timon de Groot <timon.degroot@team.blue>
8 lines
339 B
YAML
8 lines
339 B
YAML
# User agents that index exclusively for search in for AI systems.
|
|
# Each entry should have a positive/ALLOW entry created as well, with further documentation.
|
|
# Exceptions:
|
|
# - Claude-SearchBot: No published IP allowlist
|
|
- name: "ai-crawlers-search"
|
|
user_agent_regex: >-
|
|
OAI-SearchBot|Claude-SearchBot|PerplexityBot
|
|
action: DENY
|