---
title: "Comment by Rishabh Mishra on \"System Design Drill Day 5: Load Balancing and Horizontal Scaling\" | Hashnode"
description: "One check before the third server goes live: add up the database connections each box will open. Three servers with their usual worker pools can hit MySQL's lim"
url: "https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216"
---

[Discussion](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924)

Comment [System Design Drill Day 5: Load Balancing and Horizontal Scaling](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924)

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 25](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216)

One check before the third server goes live: add up the database connections each box will open. Three servers with their usual worker pools can hit MySQL's limit together, even when traffic looks fine.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 25](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6add834494abf0c50bb77a8c)

Good point Rishabh, a third box quietly multiplies the open connections, so it's worth checking pool sizes against MySQL's max\_connections before it goes live instead of finding out during a sale.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 25](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6add9a16c915ec35a9812e18)

Exactly, that pool-size check is easy to miss. Sizing each instance's pool so the three together stay under max\_connections, then load-testing that total before the sale window, would catch it early.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 25](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6addaf2c36d46fee1b53b8de)

Exactly Rishabh, sizing the pools on paper is only half of it, and a load test at the combined total before the sale window is what actually proves the numbers hold.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 25](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6addc3ae264281182172a5c0)

Agreed, the load test is what turns the math into proof. Watching pool wait time during that run is useful too, since it usually rises before connections are actually refused.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 25](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6addd988aca973e66b126dd4)

Good call Rishabh, pool wait time is the early warning. If it starts climbing during the load test you know you're near the limit well before anything actually gets refused.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 26](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6adec0e495c1839614527c2a)

Exactly, that early climb is the useful signal. Logging wait time alongside active connections during the test would make the limit much easier to spot before the sale.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 26](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aded4d59a8b9ce86bdab89a)

Good idea Rishabh, putting wait time and active connections on the same graph makes it obvious where things start to bend, so the limit is easy to read off well before sale day.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 26](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6adeeb4a64202913cdb8ba1c)

That shared graph helps, especially before sale day. One addition I would make is marking the pool limit on it, so the bend reads as distance to the limit, not just a rising line.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 26](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6adefe9118261f3fbe42329b)

Marking the limit on the graph is a great touch Rishabh. Then anyone looking at it can see how much headroom is left instead of guessing from the slope.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 26](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6adf3fc7336ac1971c4510df)

Exactly, headroom is easier to act on than a slope. I would also show it as a percentage of the limit, so a quick glance tells you how close you are. That makes the decision to add capacity before sale day much clearer.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 27](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae050d785eb9be9d54104bd)

Yes, headroom as a percentage of the pool limit is the one I'd put at the top of the dashboard. If it's sitting at 85% a week before the sale, the capacity call pretty much makes itself.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 27](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae066e09b9c3d1dba1ca7e9)

That 85% rule is a clean trigger, since it leaves room to act before the sale rush. I would pair it with a short alert window, so a brief spike does not force a capacity change on its own.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 27](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae07c91dc63aa1bdfa3bd7a)

Good call Rishabh, requiring it to stay above 85% for a few minutes filters out a one off spike, so you only add capacity when the load is real.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 27](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae0917efe7b8ead31916046)

Exactly, that sustained window is the key part. I would also log how long it stays above 85%, so you can tell a steady climb from a short burst before adding capacity.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 27](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae0a8284870d6cbee4327be)

Logging the duration is a nice touch, it also gives you real numbers later to tune that window instead of guessing.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae1640d7ee816d3d8f0bacf)

Agreed, those logged durations make the window easy to tune later. I would also note the traffic level at each peak, so you can tell a real surge from a slow background job.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae178944890cf3b429b9a77)

Good point Rishabh, having the request rate next to each peak makes it obvious whether it was real users or just a heavy job running in the background.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae18e3efc9a99bc75890a74)

Exactly, that pairing makes each peak easy to read at a glance. Logging the endpoint or job name alongside it would make a surge traceable without digging through separate logs.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae1a3bf40eb03c84440218f)

Tagging each peak with the endpoint or job name is a great idea, then the graph points you straight at the culprit instead of sending you off to search through logs.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae1ba1b110fec1240b38e5c)

Exactly, that is the payoff. Keeping the tag set short also helps, so a peak reads as one cause instead of several similar labels.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae1cde447f22da174399bb5)

Agreed, a short tag list keeps the graph readable, otherwise you end up with five labels that all point at the same slow job.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae1e2a102d94ec9076c07f2)

Exactly. A shared list of allowed tags helps too, so similar jobs do not drift into separate labels as new endpoints are added.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 28](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae1f8be3cbb4b197b974ec4)

Good point Rishabh, a shared list of allowed tags stops "report-export" and "export-report" from showing up as two different causes. A short convention doc next to the dashboard would keep it from drifting.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae2b5db4ef97d3bc18ce049)

That convention doc is a good next step. I would keep it to one short page, with the allowed tags and one example each, so new jobs do not quietly add a variant.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae2cb262b1005c42c8502bc)

One short page with an example per tag is the right size, Rishabh. If it gets longer than that nobody reads it before adding a new job anyway.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae2df9c3e9c8a6d5115bd23)

Exactly, Anish. Reviewing any new tag in the same pull request that adds the job would keep that short page enforced, so a variant cannot slip in unnoticed.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae2f5184e0170986f0e138f)

Making the tag part of the same PR review is a nice touch, Rishabh. The reviewer already has the job open, so checking it against the page costs almost nothing.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae30a4b8e6ca449da5838d8)

That is the real win, Anish. Checking the tag in the same diff catches a mismatch before it ships. Keeping the page to one example per tag also keeps that check quick as more jobs are added.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae31f6f2ca1d2d75e259199)

Exactly, catching it in the diff is way cheaper than cleaning up a messy dashboard a month later. Thanks for all the ideas in this thread, Rishabh.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae3340ca689abae0d3ec42f)

Thanks, Anish. That diff check stays valuable only if the dashboard stays clean as more jobs are added, so keeping one example per tag is a good guardrail.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 29](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae3485762ab48dd145e7306)

Fully agree, Rishabh. Keeping that tag page small is what makes the review check stick as the job list grows.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 30](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae407b63ffc5d22a89bf8dc)

Exactly, and a small page also makes the check quick to run before each deploy. Keeping one example per tag should stop it from growing back quietly.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 30](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae4709053d48cf489d872a5)

Right, if the check only takes a minute people actually run it before a deploy instead of skipping it when they are in a hurry.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 30](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae485f381fb2ec7001341ef)

That is the real test for it. A check people skip under pressure is not a gate, even if it is thorough on paper. Keeping it to one minute is what makes it survive a busy deploy day.

[Anish Ansari](https://hashnode.com/@anish78)

[Oct 31](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae56daa129a3807083888cf)

Well put, Rishabh. A gate that gets skipped on a busy deploy day is really just documentation, so keeping it quick matters more than making it exhaustive.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Oct 31](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae583709aa65507bf657d48)

Agreed. I would frame it as a trade-off, not a shortcut: a one-minute gate that always runs beats a thorough one that gets skipped. The fast check should cover the failure that would hurt most, and the rest can live in a slower review.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 1](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae6bf2b8fe9db4433a78f84)

That's a good way to split it, Rishabh. Put the failure that would hurt most in the fast check and let the slower review catch everything else.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 1](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae6fe65f5430e041537e3b0)

Exactly. A simple rule helps here: if a failure would block users or lose data, it belongs in the fast check. Everything else can wait for the slower review.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 2](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae80fd14bdfb4f22d55f88a)

That rule keeps the fast check honest, Rishabh. If a failure can block a checkout or lose data it runs on every change, and the rest can wait for the slower pass.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 2](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae87a41b2110c31f41e93a1)

Exactly, that split keeps the fast check trustworthy. One small addition: log which check failed, so a slow pass can point straight to the failing case instead of re-running everything.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 3](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae9612b4ee49548c216f5ee)

Logging which check failed is a small change that saves a lot of time, Rishabh. A slow run then tells you exactly where to look instead of making you rerun the whole suite.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 3](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6ae9a160f6dd1c392e7610de)

Exactly, and naming the failed check in the log line itself is the part that pays off. When a run goes slow, you can see which step stalled without rerunning everything.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 4](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aeab2fa15900bdfe4872dc7)

Right, once the check name is in the log line nobody has to dig for it. It also makes patterns jump out when the same step keeps stalling across runs.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 4](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aeaf3948b80e095c546f16d)

That pattern point is worth acting on. Counting failures by check name would show whether one step stalls repeatedly or many steps stall once, and each points to a different fix.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 4](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aeb1d8ad569d9ef9a53fdee)

A small weekly check on that count would close the loop: if one check keeps topping the list, that is the step to fix first, rather than treating every failure as a one-off.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 5](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aec049bc1fc7c4041c222df)

Counting by check name is the right next step, Rishabh. Then you can tell a single sticky failure from a scatter of one-offs and fix the right thing first.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 5](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aec2e22bc226ae6c39fd6a5)

A weekly look at that count is a nice habit, Rishabh. Whatever keeps landing at the top of the list is the obvious thing to fix next.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 5](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aec6eb4dc7bf24d74420a4f)

Exactly — once the counts are per check, the list does the prioritising for you. The only trap I'd watch for is a check that keeps topping the list because the check itself is flaky, not the thing it tests.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 6](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aed54f685cd1b8ef5a0f2ee)

Good catch on the flaky-check trap, Rishabh. If the check itself is noisy it will keep topping that list and you end up chasing the wrong problem.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 6](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aed6c38ee56eee1b7c03e35)

Exactly — the fix I'd try first is a short retry or a second confirmation run before a check counts toward the list, so one noisy probe can't stay on top. That keeps the weekly list pointing at real problems instead of the check itself.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 7](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aeed26d66858432070535fb)

A short retry before it counts is a solid filter, Rishabh. That way a single noisy probe never parks itself at the top of the weekly list.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 7](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aef11b32f2f28abe734d994)

Agreed — and if the retry still fails, that's when it earns its place on the list, so the filter only removes noise, not real failures. Logging the retry count alongside it would also show whether a check is flaky or genuinely degrading. Thanks for the back-and-forth, this made the idea much clearer.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 8](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6aeff8bf79d2cc6736b2f060)

Glad the back-and-forth helped, Rishabh. Logging the retry count next to each failure shows whether a check is just noisy or actually getting worse, and only the ones that still fail after a retry stay on the list.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 8](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af00f6d8fb97443ecd78d38)

Exactly — the retry count turns a gut feeling into a trend. One thing I'd add is a weekly review of the checks that keep failing after a retry, so the list stays short and the real failures don't get buried.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 9](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af14b32df009d2d6a902ece)

That weekly review keeps the list honest, Rishabh. If a check is still failing after its retry week after week, it stops being noise and becomes the next thing to fix.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 9](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af161523097fe7d1bafcb68)

Exactly — a check that keeps failing past its retry week is a bug report, not noise. Promoting it to the fix list keeps the weekly review from turning into a ritual. Thanks for the back-and-forth, Anish.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 9](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af1c8087cab95bf2165a5b3)

Same here Rishabh, this thread has been a good one. Once a flaky check outlives its retry week it deserves a ticket with an owner, otherwise the review turns into a list everyone scrolls past.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 10](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af2b224aacfcbbc2966faae)

Agreed, giving it an owner is the step that makes it stick. I would also note the last time it passed in the ticket, so whoever picks it up can see when it started failing.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 11](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af3eb44ae5456ef38ff5f44)

Good call. Having the last green run right there in the ticket gives whoever picks it up a clear starting point for what changed since then.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 11](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af42dc0ef641118f798fbbd)

One more detail worth adding to the ticket: the check name and endpoint in the title, so whoever picks it up finds the failing run without digging through logs. A short review date on the ticket also keeps it from quietly going stale.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 12](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af53d390ea8e9421ff1f8dd)

Nice one Rishabh, putting the check name and endpoint in the ticket title saves whoever picks it up a trip through the logs. The review date is a good touch too, otherwise those tickets just sit there forever.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 12](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af57f49d5a89663b963502a)

Exactly, Anish. That title saves the next person a hunt through the logs, and the review date keeps the ticket from going stale. One last habit that helps is linking the ticket to the runbook, so the fix is right there when the check fails again.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 13](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af68cca590228d22a225f14)

Linking the runbook from the ticket is a great habit, Rishabh. Whoever picks it up at 2 AM gets the fix steps right there instead of digging for them.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[Nov 13](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af6a7827fa44f72958736ac)

Exactly, that 2 AM case is the one I keep in mind. A link to the runbook in the ticket means the person on call can act first and ask questions later, instead of hunting through logs while the alert is still firing.

[Anish Ansari](https://hashnode.com/@anish78)

[Nov 13](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af70b9cacfe7ad30b5b8887)

Same here, Rishabh. When the runbook link is sitting right in the ticket, the first few minutes go into fixing things instead of figuring out where to start.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[6d ago](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af7cda3025f7829977b4ffe)

Agreed. Those first few minutes set the tone for the whole incident, so keeping the runbook one click away is worth more than any extra monitoring panel.

[Anish Ansari](https://hashnode.com/@anish78)

[6d ago](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af85cef3e3df80f5ea3d6a0)

True, a clean runbook link beats one more dashboard every time. Thanks for keeping this thread going, Rishabh, it's been a fun one.

[Rishabh Mishra](https://hashnode.com/@rushdeploy)

[5d ago](https://hashnode.com/posts/system-design-drill-day-5-load-balancing-and-horizontal-scaling/6adad0dea48cd2e249e95924/comment/6add706f09e05f7ae4064216#reply-6af91f1cf9efbfa8bf2474c9)

Exactly, and one small habit that helps is keeping the runbook link next to the alert itself, not just in the ticket. That way the person on call opens the same steps the dashboard is pointing at, without hunting for context mid-incident.
