feat: implement a client for Thoth, the IP reputation database for Anubis (#637)

* feat(internal): add Thoth client and simple ASN checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): cached ip to asn checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: go mod tidy

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(thoth): minor testing fixups, ensure ASNChecker is Checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): make ASNChecker instances

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): add GeoIP checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): store a thoth client in a context

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: refactor Checker type to its own package

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(thoth): add thoth mocking package, ignore context deadline exceeded errors

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): pre-cache private ranges

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/policy/config): enable thoth ASNs and GeoIP checker parsing

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(thoth): refactor to move checker creation to the checker files

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(policy): enable thoth checks

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thothmock): test helper function for loading a mock thoth instance

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat: wire up Thoth, make thoth checks part of the default config

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(thoth): mend staticcheck errors

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(admin): add Thoth docs

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(policy): update Thoth links in error messages

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(docs/manifest): enable Thoth

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: add THOTH_INSECURE for contacting Thoth over plain TCP in extreme circumstances

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(thoth): use mock thoth when credentials aren't detected in the environment

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(cmd/anubis): better warnings for half-configured Thoth setups

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(botpolicies): link to Thoth geoip docs

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
Xe Iaso 2025-06-16 11:57:32 -04:00 committed by GitHub
parent 823d1be5d1
commit e3826df3ab
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
39 changed files with 1101 additions and 82 deletions

View file

@ -9,10 +9,13 @@ anubistest
apk
Applebot
archlinux
asnc
asnchecker
asns
aspirational
badregexes
bdba
berr
betteralign
bingbot
bitcoin
blogging
@ -25,6 +28,7 @@ Brightbot
broked
Bytespider
cachebuster
cachediptoasn
Caddyfile
caninetools
Cardyb
@ -89,9 +93,14 @@ Fordola
forgejo
fsys
fullchain
gaissmai
Galvus
geoip
geoipchecker
gha
gipc
gitea
godotenv
goland
gomod
goodbot
@ -101,6 +110,7 @@ goyaml
GPG
GPT
gptbot
grpcprom
grw
Hashcash
hashrate
@ -113,6 +123,7 @@ hostable
htmlc
htmx
httpdebug
Huawei
hypertext
iaskspider
iat
@ -120,11 +131,14 @@ ifm
Imagesift
imgproxy
inp
IPTo
iptoasn
iss
isset
ivh
Jenomis
JGit
joho
journalctl
jshelter
JWTs
@ -164,7 +178,6 @@ mojeekbot
mozilla
nbf
netsurf
NFlag
nginx
nobots
NONINFRINGEMENT
@ -241,11 +254,14 @@ subrequest
SVCNAME
tagline
tarballs
tarrif
techaro
techarohq
templ
templruntime
testarea
thoth
thothmock
Tik
Timpibot
torproject
@ -270,6 +286,7 @@ websecure
websites
Webzio
wildbase
withthothmock
wordpress
Workaround
workdir