lib: move config to yaml (#307)

* lib: move config to yaml

Signed-off-by: Xe Iaso <me@xeiaso.net>

* web: run go generate

Signed-off-by: Xe Iaso <me@xeiaso.net>

* Add Haiku to known instances (#304)

Signed-off-by: Asmodeus <46908100+AsmodeumX@users.noreply.github.com>

* Add headers bot rule (#300)

* Closes #291: add headers support to bot policy rules

* Fix config validator

* update docs for JSON -> YAML

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: document http header based actions

Signed-off-by: Xe Iaso <me@xeiaso.net>

* lib: add missing test

Signed-off-by: Xe Iaso <me@xeiaso.net>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Asmodeus <46908100+AsmodeumX@users.noreply.github.com>
Co-authored-by: Asmodeus <46908100+AsmodeumX@users.noreply.github.com>
Co-authored-by: Neur0toxine <pashok9825@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

lib/policy/config/testdata/bad/badregexes.yaml

@@ -0,0 +1,7 @@
+bots:
+- name: path-bad
+  path_regex: "a(b"
+  action: DENY
+- name: user-agent-bad
+  user_agent_regex: "a(b"
+  action: DENY

lib/policy/config/testdata/bad/invalid.yaml

@@ -0,0 +1 @@
+bots: []

lib/policy/config/testdata/bad/nobots.yaml

@@ -0,0 +1 @@
+{}

lib/policy/config/testdata/good/allow_everyone.yaml

@@ -0,0 +1,6 @@
+bots:
+- name: everyones-invited
+  remote_addresses:
+    - "0.0.0.0/0"
+    - "::/0"
+  action: ALLOW

lib/policy/config/testdata/good/block_cf_workers.yaml

@@ -0,0 +1,5 @@
+bots:
+  - name: cloudflare-workers
+    headers_regex:
+      CF-Worker: .*
+    action: DENY

lib/policy/config/testdata/good/challengemozilla.yaml

@@ -0,0 +1,4 @@
+bots:
+- name: generic-browser
+  user_agent_regex: Mozilla
+  action: CHALLENGE

lib/policy/config/testdata/good/everything_blocked.yaml

@@ -0,0 +1,4 @@
+bots:
+- name: everything
+  user_agent_regex: .*
+  action: DENY