Middleware priority is everything - your custom proxy middleware has gotta jump ahead of HttpProxyMiddleware which runs at priority 750, so throw it at like 400 or 600, not 750 itself, or your proxy assignment just gets ghosted, and authentication headers get nuked on retry - you need to manually inject that proxy-authorization header in your middleware, otherwise you're gonna get hammered with 407 errors every time it retries. Also the default retry logic is basically a dumpster fire that loops infinitely - scrapy just keeps hammering the same failing proxy over and over, so you gotta write custom retry middleware that actually rotates to a fresh proxy each time. Plus, the article completely glosses over stuff like soft-ban detection , those capctha pages that sneakily return a 200, sticky sessions for authenticated flows, and proxy health monitoring to catch dead IPs before they tank your job. So teams shipping this thing to production without really understanding what's under the hood are gonna get wrecked with phantom failures, auth breakdowns, and hemorrhaged bandwidth