* feat: first implementation of honeypot logic
This is a bit of an experiment, stick with me.
The core idea here is that badly written crawlers are that: badly
written. They look for anything that contains `<a href="whatever" />`
tags and will blindly use those values to recurse. This takes advantage
of that by hiding a link in a `<script>` tag like this:
```html
<script type="ignore"><a href="/bots-only">Don't click</a></script>
```
Browsers will ignore it because they have no handler for the "ignore"
script type.
This current draft is very unoptimized (it takes like 7 seconds to
generate a page on my tower), however switching spintax libraries will
make this much faster.
The hope is to make this pluggable with WebAssembly such that we force
administrators to choose a storage method. First we crawl before we
walk.
The AI involvement in this commit is limited to the spintax in
affirmations.txt, spintext.txt, and titles.txt. This generates a bunch
of "pseudoprofound bullshit" like the following:
> This Restoration to Balance & Alignment
>
> There's a moment when creators are being called to realize that the work
> can't be reduced to results, but about energy. We don't innovate products
> by pushing harder, we do it by holding the vision. Because momentum can't
> be forced, it unfolds over time when culture are moving in the same
> direction. We're being invited into a paradigm shift in how we think
> about innovation. [...]
This is intended to "look" like normal article text. As this is a first
draft, this sucks and will be improved upon.
Assisted-by: GLM 4.6, ChatGPT, GPT-OSS 120b
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(honeypot/naive): optimize hilariously
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(honeypot/naive): attempt to automatically filter out based on crawling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use mazeGen instead of bsGen
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add honeypot docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: fix spelling metadata
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Implement FCrDNS and other DNS features
* Redesign DNS cache and methods
* Fix DNS cache
* Rename regexSafe arg
* Alter verifyFCrDNS(addr) behaviour
* Remove unused dnsCache field from Server struct
* Upd expressions docs
* Update docs/docs/CHANGELOG.md
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor(dns): simplify FCrDNS logging
* docs: clarify verifyFCrDNS behavior
Add a note to the documentation for `verifyFCrDNS` to clarify that it returns true when no PTR records are found for the given IP address.
* fix(dns): Improve FCrDNS error handling and tests
The `VerifyFCrDNS` function previously ignored errors returned from reverse DNS lookups. This could lead to incorrect passes when a DNS failure (other than a simple 'not found') occurred. This change ensures that any error from a reverse lookup will cause the FCrDNS check to fail.
The test suite for FCrDNS has been updated to reflect this change. The mock DNS lookups now simulate both 'not found' errors and other generic DNS errors. The test cases have been updated to ensure that the function behaves correctly in both scenarios, resolving a situation where two test cases were effectively duplicates.
* docs: Update FCrDNS documentation and spelling
Corrected a typo in the `verifyFCrDNS` function documentation.
Additionally, updated the spelling exception list to include new terms and remove redundant entries.
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* refactor: move lib/policy/config to lib/config
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: don't set global loggers anymore
Ref #864
You were right @kotx, it is a bad idea to set the global logger
instance.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(config): add log sink support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/policies): add logging block documentation
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd/anubis): revert this change, it's meant to be its own PR
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add file logging smoke test
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: don't expose the old log file time format string
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(internal): add ListOr[T any] type
This is a utility type that lets you decode a JSON T or list of T as a
single value. This will be used with Redis Sentinel config so that you
can specify multiple sentinel addresses.
Ref TecharoHQ/botstopper#24
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(store/valkey): add Redis(R) Sentinel support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
check-spelling run (pull_request) for Xe/redis-sentinel
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore(store/valkey): remove pointless comments
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document the Redis™ Sentinel configuration options
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(store/valkey): Redis™ Sentinel doesn't require a password
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* internal/log: Implement logging of HOST when using subrequest auth
The host header wouldn't be set on subrequest auth, so we need to look for X-Forwarded-Host header when logging requests.
* chore: add changelog entry
---------
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Co-authored-by: Xe Iaso <xe.iaso@techaro.lol>
* feat(anubis): add /healthz route to metrics server
Also add health check test for Docker Compose and update documentation
for health checking Anubis with Docker Compose.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
This is not used yet, but it will be part of a larger strategy around
adding/removing weight based on JA4H (and other) fingerprint matches
with Thoth.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(decaymap): add Delete method
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(lib/challenge): refactor Validate to take ValidateInput
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): implement store interface
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/store): all metapackage to import all store implementations
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(policy): import all store backends
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): use new challenge creation flow
Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.
This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(decaymap): fix documentation typo
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(lib): fix SA4004
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/store): make generic storage interface test adaptor
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(decaymap): invert locking process for Delete
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/store): add bbolt store implementation
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(devcontainer): adapt to docker compose, add valkey service
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): make challenges live for 30 minutes by default
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/store): implement valkey backend
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/store/valkey): disable tests if not using docker
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/policy/config): ensure valkey stores can be loaded
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/store-interface
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore(devcontainer): remove port forwards because vs code handles that for you
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(default-config): add a nudge to the storage backends section of the docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(docs): listen on 0.0.0.0 for dev container support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(policy): document storage backends
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG and internal links
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/policies): don't start a sentence with as
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: fixes found in review
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* feat(config): opengraph passthrough configuration
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(ogtags): use config.OpenGraph for configuration
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: wire up ogtags config in most of the app
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(ogtags): return default tags if they are supplied
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: make OpenGraph legal so we have some sanity in reviewing
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use OpenGraph.Enabled
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): load default config file if one is not specified in spawnAnubis
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(config): fix ST1005
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document open graph defaults and its new home in the policy file
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(installation): point to weight threshold new home
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: rename default to override
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(default-config): add off-by-default opengraph settings to bot policy file
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(anubis): make build
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix build
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor(ogtags): optimize URL construction and memory allocations
* test(ogtags): add benchmarks and memory usage tests for OGTagCache
* refactor(ogtags): optimize OGTags subsystem to reduce allocations and improve request runtime by up to 66%
* Update docs/docs/CHANGELOG.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* refactor(ogtags): optimize URL string construction to reduce allocations
* Update internal/ogtags/ogtags.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* test(ogtags): add fuzz tests for getTarget and extractOGTags functions
* fix(ogtags): update memory calculation logic
Prev it would say that we had allocated 18pb
=== RUN TestMemoryUsage
mem_test.go:107: Memory allocated for 10k getTarget calls: 18014398509481904.00 KB
mem_test.go:135: Memory allocated for 1k extractOGTags calls: 18014398509481978.00
Now it's fixed with
=== RUN TestMemoryUsage
mem_test.go:109: Memory allocated for 10k getTarget calls:
mem_test.go:110: Total: 630.56 KB (0.62 MB)
mem_test.go:111: Per operation: 64.57 bytes
mem_test.go:140: Memory allocated for 1k extractOGTags calls:
mem_test.go:141: Total: 328.17 KB (0.32 MB)
mem_test.go:142: Per operation: 336.05 bytes
* refactor(ogtags): optimize meta tag extraction for improved performance
* Update metadata
check-spelling run (pull_request) for json/ogmem
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore: update CHANGELOG for recent optimizations and version bump
* refactor: improve URL construction and meta tag extraction logic
* style: cleanup fuzz tests
---------
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat(lib): ensure that clients store cookies
If a client is misconfigured and does not store cookies, then they can
get into a proof of work death spiral with Anubis. This fixes the
problem by setting a test cookie whenever the user gets hit with a
challenge page. If the test cookie is not there at challenge pass time,
then they are blocked. Administrators will also get a log message
explaining that the user intentionally broke cookie support and that this
behavior is not an Anubis bug.
Additionally, this ensures that clients being shown a challenge support
gzip-compressed responses by showing the challenge page at gzip level 1.
This level is intentionally chosen in order to minimize system impacts.
The ClearCookie function is made more generic to account for cookie
names as an argument. A correlating SetCookie function was also added to
make it easier to set cookies.
* chore(lib): clean up test code
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(playwright): Add support to run tests in Docker/Podman
* fix command name
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Henri Vasserman <henv@hot.ee>
* up the pw version as it is in package.json
* add convenience npm scripts
* chore: changelog update
Also removed a period from my other item.
* chore: fix spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Henri Vasserman <henv@hot.ee>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* refactor: reorder import statements in fetch.go and fetch_test.go
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: optimize struct field alignment to reduce memory usage
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(ogtags): enhance target URL handling for OGTagCache, support Unix sockets
Closes: #323#319
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs: update CHANGELOG.md to include Opengraph passthrough support for Unix sockets
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs: update CHANGELOG.md to include Opengraph passthrough support for Unix sockets
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(ogtags): add option to consider host in Open Graph tag cache key
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(ogtags): add option to consider host in OG tag cache key
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* test(ogtags): enhance tests for OGTagCache with host consideration scenarios
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): extract constants for HTTP timeout and max content length
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): restore fetchHTMLDocument method for cache key generation
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): replace maxContentLength field with constant and ensure HTTP scheme is set correctly
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(fetch): add proxy headers
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(logging): centralize logger creation in GetLogger function
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(logging): rename GetLogger to GetRequestLogger for clarity
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: streamline error handling and response methods
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(lib): Split anubis.go up into some smaller specialized methods
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(http): simplify error response handling by using respondWithStatus
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore(lib): run goimports
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat(cmd/anubis): compute full XFF header
this one is pretty important to not pass
through blindly, as many applications and
frameworks will trust them
* feat(cmd/anubis): skip XFF compute if remote address is loopback
* docs: update CHANGELOG
* fix: improve error handling for resource closing and JSON encoding in MakeChallenge
* chore: update CHANGELOG with recent changes and improvements
* refactor: simplify RenderIndex function and improve error handling
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(fetch): improve error handling for Content-Type parsing
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(fetch): rename OgHandledError to ErrOgHandled for statichcheck to like me
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Add Open Graph tag support (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Prevent nil pointer dereference in test (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat!: Implement Open Graph tag caching and passthrough functionality (WIP)
I'm going to sleep. currently tags are passed to renderIndex.
see https://github.com/TecharoHQ/anubis/issues/131
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Add configuration for air tool with build and logger settings
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Move OG tags to base template (og-tags)
Moves the Open Graph (OG) tags from the index template to
the base template. This allows OG tags to be set on any
page, not just the index. Also adds a
BaseWithOGTags function to the web package to allow
passing OG tags to the base template. Removes the
ogTags parameter from the Index function and template.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Delete CHANGELOG.md
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Add language attribute to HTML tag in template
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): Fix nil pointer ref
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Add timeout to http client (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: fix line endings & indentation
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: add inspection comment for GoBoolExpressions in UnchangingCache
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Implement Open Graph tag fetching and caching
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(og-tags): Simplify Open Graph tag extraction logic
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(og-tags): Add nil check in isOGMetaTag and enhance test cases
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Add approved tags and prefixes for Open Graph extraction
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* test(og-tags): Update tests with approved tags and improve clarity
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: Add changelog notes
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: Improve stability of the target fetcher?
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: Update template error handling and improve Open Graph tag integration
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: format files and remove deubg logs
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Credit CELPHASE for mascot design (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Credit CELPHASE for mascot design (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Allow twitter prefixed OG tags by default
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: replace /tmp with /var
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update docs/docs/CHANGELOG.md
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* Update docs/docs/admin/configuration/open-graph.mdx
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* chore: add fediverse to default prefixes (#og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Remove og-query-distinct flag
This commit removes the `og-query-distinct` flag and
associated logic. URLs with different query parameters
will now always be treated as the same cache key for Open
Graph tags. This simplifies the caching logic and
improves performance.
Additionally, the http client used for fetching OG tags
is now a member of the OGTagCache struct, rather than a
global variable. This improves testability and allows
for more flexible configuration in the future.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update docs/docs/admin/configuration/open-graph.mdx
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* docs: remove og tags references
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: rename url > u to not overlap package name
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update internal/ogtags/cache.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* Update internal/ogtags/cache.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* fix(tests): Don't use network when network access is disabled
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Handle nil URL in GetOGTags (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: sort installation docs alphabetically
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): validate that no duplicate requests are made
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style(tests): remove unused ok var
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs: convert to table fmt
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Enhance OG tag fetching and caching
Adds additional approved OG tags (`keywords`, `author`), improves
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update generated templ's after format
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): update integration_test.go to reflect the new behavior of fetchHTMLDocument
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Revert "data/botPolicies: allow iMessage scraper by default (#178)"
This reverts commit 21a9d777
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Simplify ogTags access in cache test.
Didn't know this was possible! wow!
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Handle request timeouts when fetching OG tags (#og-tags)
Cache a nil result for half the TTL to avoid repeatedly
requesting a timed-out URL.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: make OG tags passthrough option function.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Handle timeouts and non-200 responses when fetching OG tags (og-tags)
- Cache empty results for timeouts and non-200 status codes
to avoid spamming the server.
- Use a non-nil empty map to represent empty results in the
cache, as nil would be a cache miss.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): switch to http.MaxBytesReader
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore(og-tags): add noindex, nofollow meta tag and update error line numbers
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
