System Design Drill Day 5: Load Balancing and Horizontal Scaling

One check before the third server goes live: add up the database connections each box will open. Three servers with their usual worker pools can hit MySQL's limit together, even when traffic looks fine.

The rolling-deploy exercise would be even stronger with a checkout running while old and new application versions overlap. Database compatibility is one boundary; session payloads and queued-job formats are two more. Testing that mixed-version window makes “any server can handle any request” a property of the release process as well as the steady state.

One gotcha I'd add from a rollout like this: when you sync APP_KEY across the new boxes, every existing encrypted cookie dies at the same moment. We watched a chunk of users get bounced mid session during the cutover. If you're on Laravel 11, APP_PREVIOUS_KEYS lets you rotate instead of flipping the switch, so old cookies keep decrypting until they expire naturally. Worth calling out before the big sale, not after.

One crash fixture I would add: a worker sends the report email, then dies before recording completion. A scheduler lock stops concurrent starts, but a retry can still repeat that external side effect. Keep the same report/run ID on retry and test the provider's duplicate handling, where supported. Where it cannot deduplicate, record that remaining risk. Would you include that crash point in the Day 3/Day 5 exercise?

The readiness-vs-liveness split is the detail most first-time setups get wrong: folding Redis into readiness means one cache blip pulls every server out of rotation at once. The duplicate report email is the other classic: a scheduled job needs a single owner the moment more than one box exists.