ef52550e70
Closes #372 Fun YAML fact of the day: What is the difference between how these two expressions are parsed? ```yaml foo: > bar ``` ```yaml foo: >- bar ``` They are invisible in yaml, but when you evaluate them to JSON the difference is obvious: ```json { "foo": "bar\n" } ``` ```json { "foo": "bar" } ``` User-Agent strings, URL path values, and HTTP headers _do_ end in newlines in HTTP/1.1 wire form, but that newline is usually stripped before the server actually handles it. Also HTTP/2 is a thing and does not terminate header values with newlines. This change makes Anubis more aggressively detect mistaken uses of the yaml `>` operator and nudges the user into using the yaml `>-` operator which does not append the trailing newline. I had honestly forgotten about this YAML behavior because it wasn't relevant for so long. Oops! Glad I released a beta. Whenever you get into this state, Anubis will throw a config parsing error and then give you a message hinting at the folly of your ways. ``` config.Bot: regular expression ends with newline (try >- instead of > in yaml) ``` Big thanks to https://yaml-multiline.info, this helped me realize my folly instantly. @aiverson, this is official permission to say "told you so". Signed-off-by: Xe Iaso <me@xeiaso.net>
49 lines
No EOL
983 B
JSON
49 lines
No EOL
983 B
JSON
{
|
|
"bots": [
|
|
{
|
|
"import": "(data)/bots/ai-robots-txt.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/bots/cloudflare-workers.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/bots/headless-browsers.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/bots/us-ai-scraper.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/googlebot.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/bingbot.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/duckduckbot.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/qwantbot.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/internet-archive.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/kagibot.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/marginalia.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/crawlers/mojeekbot.yaml"
|
|
},
|
|
{
|
|
"import": "(data)/common/keep-internet-working.yaml"
|
|
},
|
|
{
|
|
"name": "generic-browser",
|
|
"user_agent_regex": "Mozilla|Opera",
|
|
"action": "CHALLENGE"
|
|
}
|
|
],
|
|
"dnsbl": false
|
|
} |