--- title: "Discussion on \"Stop Pinging Kafka - A Zero-Cost Health Check Pattern (and What Can Go Wrong)\" | Hashnode" description: "Discussion on \"Stop Pinging Kafka - A Zero-Cost Health Check Pattern (and What Can Go Wrong)\". Why Kafka health checks can trigger consumer rebalances, restart storms, and cascading failures - and how to monitor your actual clients without making the cluster less reliable." url: "https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7" --- [Feed](https://hashnode.com/feed) / Discussion # [Stop Pinging Kafka - A Zero-Cost Health Check Pattern (and What Can Go Wrong)](https://abhishekpareek.dev/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong?utm_source=hashnode&utm_medium=feed) Why Kafka health checks can trigger consumer rebalances, restart storms, and cascading failures - and how to monitor your actual clients without making the cluster less reliable. Oct 19 · 11 min read · abhishekpareek.dev Read the article - [#kafka](https://hashnode.com/tag/kafka) - [#rebalancing](https://hashnode.com/tag/rebalancing) - [#consumergroups](https://hashnode.com/tag/consumergroups) - [#kafka-health-checks](https://hashnode.com/tag/kafka-health-checks) - [#distributed-systems](https://hashnode.com/tag/distributed-systems) - [#kubernetes-livenessprobe](https://hashnode.com/tag/kubernetes-livenessprobe) - [#sre-devops](https://hashnode.com/tag/sre-devops) ## Written by [abhishek pareek](https://hashnode.com/@abhishek-pareek) @abhishek-pareek · Vancouver, Canada [View profile](https://hashnode.com/@abhishek-pareek) Architecting resilient distributed systems by balancing throughput, reliability, and the trade-offs that matter at scale. Senior engineer with 10+ years of experience designing, operating, and evolving production Kubernetes platforms and distributed systems. I make architectural decisions with a clear understanding of the operational tradeoffs and take ownership of the outcomes. My current work focuses on large-scale Kubernetes platforms and AI infrastructure, while I continue to explore agentic AI to accelerate thinking for more intelligent, reliable, and impactful platform operations. Based in Vancouver. ## Responses 7 Markdown supported · ⌘↵ to post [Abhinav Sharma](https://hashnode.com/@abhinavs1920) [Oct 20](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad7e5af9a5ea76ea2a5f709) The restart-storm angle is the counterintuitive part, a health check that kills and restarts consumers can trigger the very rebalances that then look like downtime. Piggybacking on the statistics callback instead of publishing test messages also keeps the topics clean. [Damiano Bianda](https://hashnode.com/@damianobianda) [Oct 20](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad71841044eb377c226fae7) The "the health check itself made the cluster less reliable" pattern shows up far beyond Kafka, and your GC-pauses-to-restart-storm chain is the sharpest example of the monitor and the monitored sharing a fate. The failure mode I'd want a follow-up on: how do you notice when the piggybacked statistics themselves stop arriving? If the callback stalls silently, you end up with the calmest dashboard of all - the one that says nothing is wrong because nothing is being measured. Did you end up putting a watchdog on the absence of callbacks, or is staleness just accepted as a blind spot? [Abhinav Sharma](https://hashnode.com/@abhinavs1920) [Oct 20](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad6e7f65b4fbbb2012e1bc2) The restart storm section is the part every team rediscovers on its own. A health check that restarts the client turns one slow broker into a full fleet outage. [Rishabh Mishra](https://hashnode.com/@rushdeploy) [Oct 20](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad6d99c8b41f070ec2d91f0) One more guard worth adding: wire this check to the readiness probe rather than liveness. A false unhealthy then pulls the pod out of service instead of restarting it, which avoids both the restart storm and the consumer rebalance the article describes. [abhishek pareek](https://hashnode.com/@abhishek-pareek) [Oct 20](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad6d99c8b41f070ec2d91f0#reply-6ad6ee30d28d916a1e4b7502) good point [Rishabh Mishra](https://hashnode.com/@rushdeploy "Rishabh Mishra") ! [Rishabh Mishra](https://hashnode.com/@rushdeploy) [Oct 20](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad6d99c8b41f070ec2d91f0#reply-6ad702f7c399e7d1392aa301) Thanks! One detail worth pinning down: keep the liveness probe cheap, just a process-alive check. Put the Kafka state behind a readiness endpoint with a small consecutive-failure threshold, so one blip doesn't drain the pod. That pairing covers both failure modes without mixing them. [netproxy](https://hashnode.com/@argumentmoney8117) [Oct 19](https://hashnode.com/posts/stop-pinging-kafka-a-zero-cost-health-check-pattern-and-what-can-go-wrong/6ad65e65cb89e7ce1f6ae2c7/comment/6ad69f70d0d0b34960b500c0) The section on the stats callback firing while the broker is unreachable stood out to me — a health check that only looks at "did the callback fire recently" would happily report healthy during a disconnect. Parsing the broker connection state off-thread feels like the crucial half of this pattern that most write-ups skip.