68a71c6a99
* Add Applebot definition Adds Apple's search indexing bot, and allowlists it by default. Allowlisted by default because it is equivalent to Googlebot/Bingbot. Remove Applebot from `ai-robots-txt.yaml` for the same reasons. Remove `Applebot-Extended` from `ai-robots-txt.yaml` as it has no effect. * chore: spelling Signed-off-by: Xe Iaso <me@xeiaso.net> --------- Signed-off-by: Xe Iaso <me@xeiaso.net> Co-authored-by: Xe Iaso <me@xeiaso.net> |
||
|---|---|---|
| .. | ||
| apps | ||
| bots | ||
| clients | ||
| common | ||
| crawlers | ||
| botPolicies.json | ||
| botPolicies.yaml | ||
| embed.go | ||