Scraping Ethics: How to Configure robots.txt and User-Agent to Avoid the Ban Hammer

We often talk about ethics in tech as a moral obligation, but in web scraping, ethics is a functional requirement. If you ignore a site's boundaries, you aren't just being "rude"—you are actively trig