68a71c6a99
* Add Applebot definition Adds Apple's search indexing bot, and allowlists it by default. Allowlisted by default because it is equivalent to Googlebot/Bingbot. Remove Applebot from `ai-robots-txt.yaml` for the same reasons. Remove `Applebot-Extended` from `ai-robots-txt.yaml` as it has no effect. * chore: spelling Signed-off-by: Xe Iaso <me@xeiaso.net> --------- Signed-off-by: Xe Iaso <me@xeiaso.net> Co-authored-by: Xe Iaso <me@xeiaso.net> |
||
|---|---|---|
| .. | ||
| irc-bots | ||
| _deny-pathological.yaml | ||
| aggressive-brazilian-scrapers.yaml | ||
| ai-robots-txt.yaml | ||
| cloudflare-workers.yaml | ||
| headless-browsers.yaml | ||
| us-ai-scraper.yaml | ||