sophie/nuke
1
0
Fork 0
Code Packages 1 Activity Actions

chore(docs/manifest): allow common crawl to test with the team

Browse source 
Signed-off-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
Xe Iaso 2025-07-04 00:22:43 +00:00
parent 7c0996448a
commit 2f1e78cc6c
No known key found for this signature in database
1 changed files with 1 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/manifest/cfg/anubis/botPolicies.yaml
View file

@ -11,6 +11,7 @@
## /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
bots:
- import: (data)/crawlers/commoncrawl.yaml
# Pathological bots to deny
- # This correlates to data/bots/deny-pathological.yaml in the source tree
# https://github.com/TecharoHQ/anubis/blob/main/data/bots/deny-pathological.yaml