nuke/data/crawlers
Corry Haines de7dbfe6d6
Split up AI filtering files (#592)
* Split up AI filtering files

Create aggressive/moderate/permissive policies to allow administrators to choose their AI/LLM stance.

Aggressive policy matches existing default in Anubis.

Removes `Google-Extended` flag from `ai-robots-txt.yaml` as it doesn't exist in requests.

Rename `ai-robots-txt.yaml` to `ai-catchall.yaml` as the file is no longer a copy of the source repo/file.

* chore: spelling

* chore: fix embeds

* chore: fix data includes

* chore: fix file name typo

* chore: Ignore READMEs in configs

* chore(lib/policy/config): go tool goimports -w

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-01 20:21:18 +00:00
..
_allow-good.yaml Add Applebot definition (#589) 2025-05-31 10:18:32 -04:00
ai-search.yaml Split up AI filtering files (#592) 2025-06-01 20:21:18 +00:00
ai-training.yaml Split up AI filtering files (#592) 2025-06-01 20:21:18 +00:00
applebot.yaml Add Applebot definition (#589) 2025-05-31 10:18:32 -04:00
bingbot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
duckduckbot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
googlebot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
internet-archive.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
kagibot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
marginalia.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00
mojeekbot.yaml Fix: mojeekbot regex (#351) 2025-04-24 14:24:41 +00:00
openai-gptbot.yaml Opt-in policies for OpenAI and MistralAI bots (#590) 2025-05-31 16:48:57 -04:00
openai-searchbot.yaml Opt-in policies for OpenAI and MistralAI bots (#590) 2025-05-31 16:48:57 -04:00
qwantbot.yaml feat: enable loading config fragments (#321) 2025-04-23 07:01:28 -04:00