nuke/data/crawlers
Timon de Groot 57c0b2b22c
Add IP mapped Perplexity user agents (#1393)
Perplexity has some proper documentation available for their crawlers,
with published IP addresses: https://docs.perplexity.ai/guides/bots.

Signed-off-by: Timon de Groot <timon.degroot@team.blue>
2026-01-15 19:57:31 -05:00
..
_allow-good.yaml Implement FCrDNS and other DNS features (#1308) 2025-11-26 22:24:45 -05:00
ai-search.yaml Add IP mapped Perplexity user agents (#1393) 2026-01-15 19:57:31 -05:00
ai-training.yaml Split up AI filtering files (#592) 2025-06-01 20:21:18 +00:00
alibaba-cloud.yaml fix(default-config): also block alibaba cloud (#1005) 2025-08-20 23:01:49 +00:00
applebot.yaml Add Applebot definition (#589) 2025-05-31 10:18:32 -04:00
bingbot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
commoncrawl.yaml chore(default-config): allowlist common crawl (#753) 2025-07-04 00:10:45 +00:00
duckduckbot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
googlebot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
huawei-cloud.yaml fix(default-config): block Huawei Cloud (#1004) 2025-08-20 22:40:07 +00:00
internet-archive.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
kagibot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
marginalia.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
mojeekbot.yaml Fix: mojeekbot regex (#351) 2025-04-24 14:24:41 +00:00
openai-gptbot.yaml Opt-in policies for OpenAI and MistralAI bots (#590) 2025-05-31 16:48:57 -04:00
openai-searchbot.yaml Opt-in policies for OpenAI and MistralAI bots (#590) 2025-05-31 16:48:57 -04:00
perplexitybot.yaml Add IP mapped Perplexity user agents (#1393) 2026-01-15 19:57:31 -05:00
qwantbot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
tencent-cloud.yaml feat(default-config): block tencent cloud by default (#1216) 2025-10-24 19:43:42 +00:00
yandexbot.yaml Implement FCrDNS and other DNS features (#1308) 2025-11-26 22:24:45 -05:00