fix(config): remove trailing newlines in regexes (#373)
Closes #372 Fun YAML fact of the day: What is the difference between how these two expressions are parsed? ```yaml foo: > bar ``` ```yaml foo: >- bar ``` They are invisible in yaml, but when you evaluate them to JSON the difference is obvious: ```json { "foo": "bar\n" } ``` ```json { "foo": "bar" } ``` User-Agent strings, URL path values, and HTTP headers _do_ end in newlines in HTTP/1.1 wire form, but that newline is usually stripped before the server actually handles it. Also HTTP/2 is a thing and does not terminate header values with newlines. This change makes Anubis more aggressively detect mistaken uses of the yaml `>` operator and nudges the user into using the yaml `>-` operator which does not append the trailing newline. I had honestly forgotten about this YAML behavior because it wasn't relevant for so long. Oops! Glad I released a beta. Whenever you get into this state, Anubis will throw a config parsing error and then give you a message hinting at the folly of your ways. ``` config.Bot: regular expression ends with newline (try >- instead of > in yaml) ``` Big thanks to https://yaml-multiline.info, this helped me realize my folly instantly. @aiverson, this is official permission to say "told you so". Signed-off-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
parent
c669b47b57
commit
ef52550e70
8 changed files with 61 additions and 9 deletions
21
lib/policy/config/testdata/bad/regex_ends_newline.json
vendored
Normal file
21
lib/policy/config/testdata/bad/regex_ends_newline.json
vendored
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
{
|
||||
"bots": [
|
||||
{
|
||||
"name": "user-agent-ends-newline",
|
||||
"user_agent_regex": "Mozilla\n",
|
||||
"action": "CHALLENGE"
|
||||
},
|
||||
{
|
||||
"name": "path-ends-newline",
|
||||
"path_regex": "^/evil/.*$\n",
|
||||
"action": "CHALLENGE"
|
||||
},
|
||||
{
|
||||
"name": "headers-ends-newline",
|
||||
"headers_regex": {
|
||||
"CF-Worker": ".*\n"
|
||||
},
|
||||
"action": "CHALLENGE"
|
||||
}
|
||||
]
|
||||
}
|
||||
17
lib/policy/config/testdata/bad/regex_ends_newline.yaml
vendored
Normal file
17
lib/policy/config/testdata/bad/regex_ends_newline.yaml
vendored
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
bots:
|
||||
- name: user-agent-ends-newline
|
||||
# Subtle bug: this ends with a newline
|
||||
user_agent_regex: >
|
||||
Mozilla
|
||||
action: CHALLENGE
|
||||
- name: path-ends-newline
|
||||
# Subtle bug: this ends with a newline
|
||||
path_regex: >
|
||||
^/evil/.*$
|
||||
action: CHALLENGE
|
||||
- name: headers-ends-newline
|
||||
# Subtle bug: this ends with a newline
|
||||
headers_regex:
|
||||
CF-Worker: >
|
||||
.*
|
||||
action: CHALLENGE
|
||||
Loading…
Add table
Add a link
Reference in a new issue