Bait, Keep, or Block: A Smarter Way to Handle AI Crawlers

Most advice about AI crawlers comes down to two buttons: block everything, or allow everything. Both are wrong, and for the same reason — they treat every bot as identical when they're nothing alike.