Feed
Discussion
Build a Web Scraping API with FastAPI, Celery & Redis (2026)
Why Your Scraper Needs an API Layer At some point every scraper outgrows its original script form. Maybe you want to trigger scraping from a dashboard. Maybe a frontend team needs to query scraped dat
Responses
Markdown supported · ⌘↵ to post
OnlineProxy
Your compose setup won't run as written, cause tasks.py hardcodes localhost for redis instead of reading REDIS_URL. Also curl_cffi, playwright and playwright_stealth are missing from requirements.txt, and there's no playwright install step
The browser task reports success when errors got swallowed, and its max_retries never retries anything. Bulk drops task_id, so revoke can't cancel those jobs.
Security is the worst part, since url is a read SSRF with no blocklist and GET /jobs hands out the api_key column to anyone with no auth or ownership checks