e0781e4560
* feat: add robots2policy CLI utility to convert robots.txt to Anubis challenge policies * feat: add documentation for robots2policy CLI tool * feat: implement crawl delay handling as weight adjustment in Anubis rules * feat: add various robots.txt and YAML configurations for user agent handling and crawl delays * test: add comprehensive tests for robots2policy conversion and parsing * fix: update example URL in usage instructions for robots2policy CLI * Update metadata check-spelling run (pull_request) for json/robots2policycli Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com> on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev> * docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI * Update cmd/robots2policy/main.go Co-authored-by: Xe Iaso <me@xeiaso.net> Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com> * Update cmd/robots2policy/main.go Co-authored-by: Xe Iaso <me@xeiaso.net> Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com> * fix(robots2policy): use sigs.k8s.io/yaml Signed-off-by: Xe Iaso <me@xeiaso.net> * feat(config): properly marshal bot policy rules Signed-off-by: Xe Iaso <me@xeiaso.net> * chore(yeetfile): expose robots2policy in libexec Signed-off-by: Xe Iaso <me@xeiaso.net> * fix(yeetfile): put robots2policy in $PATH Signed-off-by: Xe Iaso <me@xeiaso.net> * Update metadata check-spelling run (pull_request) for json/robots2policycli Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com> on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev> * style: reorder imports * refactor: use preexisting structs in config * fix: correct flag check in main function * fix: reorder fields in AnubisRule struct for better alignment * style: improve alignment of struct fields in AnubisRule and OGTagCache * Update metadata check-spelling run (pull_request) for json/robots2policycli Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com> on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev> * fix: add validation for generated Anubis rules from robots.txt * feat: add batch processing for robots.txt files to generate Anubis CEL policies * fix: improve usage message and error handling for input file requirement * refactor: update AnubisRule structure to use ExpressionOrList for improved expression handling * refactor: reorganize policy definitions in YAML files for consistency and clarity * fix: correct indentation in blacklist and complex YAML files for consistency * test: enhance output comparison in robots2policy tests for YAML and JSON formats * Revert "fix: improve usage message and error handling for input file requirement" This reverts commit ddcde1f2a326545d3ef2ec32e5e03f55f4f931a8. * fix: improve usage message and error handling in robots2policy Signed-off-by: Jason Cameron <git@jasoncameron.dev> --------- Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com> Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com> Signed-off-by: Xe Iaso <me@xeiaso.net> Signed-off-by: Jason Cameron <git@jasoncameron.dev> Co-authored-by: Xe Iaso <me@xeiaso.net> |
||
|---|---|---|
| .. | ||
| advice.md | ||
| allow.txt | ||
| candidate.patterns | ||
| excludes.txt | ||
| expect.txt | ||
| line_forbidden.patterns | ||
| patterns.txt | ||
| README.md | ||
| reject.txt | ||
check-spelling/check-spelling configuration
| File | Purpose | Format | Info |
|---|---|---|---|
| dictionary.txt | Replacement dictionary (creating this file will override the default dictionary) | one word per line | dictionary |
| allow.txt | Add words to the dictionary | one word per line (only letters and 's allowed) |
allow |
| reject.txt | Remove words from the dictionary (after allow) | grep pattern matching whole dictionary words | reject |
| excludes.txt | Files to ignore entirely | perl regular expression | excludes |
| only.txt | Only check matching files (applied after excludes) | perl regular expression | only |
| patterns.txt | Patterns to ignore from checked lines | perl regular expression (order matters, first match wins) | patterns |
| candidate.patterns | Patterns that might be worth adding to patterns.txt | perl regular expression with optional comment block introductions (all matches will be suggested) | candidates |
| line_forbidden.patterns | Patterns to flag in checked lines | perl regular expression (order matters, first match wins) | patterns |
| expect.txt | Expected words that aren't in the dictionary | one word per line (sorted, alphabetically) | expect |
| advice.md | Supplement for GitHub comment when unrecognized words are found | GitHub Markdown | advice |
Note: you can replace any of these files with a directory by the same name (minus the suffix) and then include multiple files inside that directory (with that suffix) to merge multiple files together.