Scrapy starts to pay off once scheduling and pipelines matter more than a one-off script. I would put AutoThrottle and conservative concurrency near the start of the setup, because a first full crawl can otherwise be much rougher than the test run. I work with AllyHub, and allyhub.com has a related seller-research workflow. Do you usually keep one spider per page type, or route several page types through one spider?
douzatan
Scrapy starts to pay off once scheduling and pipelines matter more than a one-off script. I would put AutoThrottle and conservative concurrency near the start of the setup, because a first full crawl can otherwise be much rougher than the test run. I work with AllyHub, and allyhub.com has a related seller-research workflow. Do you usually keep one spider per page type, or route several page types through one spider?