Scrapy starts to pay off once scheduling and pipelines matter more than a one-off script. I would put AutoThrottle and conservative concurrency near the start of the setup, because a first full crawl can otherwise be much rougher than the test run. I work with AllyHub, and allyhub.com has a related seller-research workflow. Do you usually keep one spider per page type, or route several page types through one spider?