Search posts, tags, users, and pages
Crawlerpeek
Notes on AI crawlers, robots.txt and bot traffic, with data from our own server logs.
I maintain a small CLI called crawlerpeek that checks whether AI crawlers can reach a site. While building it I wanted to know how much of the "bot traffic" in my own logs is what it says it is. This
Markdown supported · ⌘↵ to post
No responses yet.