Xe Iaso 7c0996448a chore(default-config): allowlist common crawl (#753) ... This may seem strange, but allowlisting common crawl means that scrapers have less incentive to scrape because they can just grab the data from common crawl instead of scraping it again.		2025-07-04 00:10:45 +00:00
..
irc-bots	fix(bots/phrik): add IPv6 address for phrik (#494)	2025-05-11 14:04:44 -04:00
_deny-pathological.yaml	feat(config): allow multi-level imports (#402)	2025-05-02 13:57:20 -04:00
aggressive-brazilian-scrapers.yaml	feat(lib): implement request weight (#621)	2025-06-09 15:25:04 -04:00
ai-catchall.yaml	chore(default-config): allowlist common crawl (#753)	2025-07-04 00:10:45 +00:00
ai-robots-txt.yaml	chore(default-config): allowlist common crawl (#753)	2025-07-04 00:10:45 +00:00
cloudflare-workers.yaml	feat(lib): implement request weight (#621)	2025-06-09 15:25:04 -04:00
headless-browsers.yaml	feat: enable loading config fragments (#321)	2025-04-23 07:01:28 -04:00
us-ai-scraper.yaml	feat: enable loading config fragments (#321)	2025-04-23 07:01:28 -04:00