default celery is at most once delivery, so a crashed worker just loses the task, which hurts with flaky selenium runs. acks_late plus redis visibility_timeout fixes that, but then your writes need to be idempotent, upsert on a URL hash, and cap each task with soft_time_limit and time_limit so a hung browser doesn't get redelivered while still running.
For retries, narrow autoretry_for to real transient errors, add backoff with jitter, and use a redis token bucket per domain since rate_limit only applies per worker. And yeah headless chromium eats way more memory than plain HTTP, so I check for a json endpoint before firing up a browser
default celery is at most once delivery, so a crashed worker just loses the task, which hurts with flaky selenium runs. acks_late plus redis visibility_timeout fixes that, but then your writes need to be idempotent, upsert on a URL hash, and cap each task with soft_time_limit and time_limit so a hung browser doesn't get redelivered while still running.
For retries, narrow autoretry_for to real transient errors, add backoff with jitter, and use a redis token bucket per domain since rate_limit only applies per worker. And yeah headless chromium eats way more memory than plain HTTP, so I check for a json endpoint before firing up a browser