---
title: "Discussion on \"Chaos drills for a streaming system: what restarting and killing every component taught me\" | Hashnode"
description: "Discussion on \"Chaos drills for a streaming system: what restarting and killing every component taught me\". How to design a failure drill that measures what operators see, not uptime: 0 of 42,950 readings lost, and 4 bugs found and fixed in a Kafka pipeline."
url: "https://hashnode.com/posts/kafka-chaos-testing-restart-vs-kill/6afc48e1485f929a3f847aeb"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Chaos drills for a streaming system: what restarting and killing every component taught me](https://techblog01.hashnode.dev/kafka-chaos-testing-restart-vs-kill?utm_source=hashnode&utm_medium=feed)

How to design a failure drill that measures what operators see, not uptime: 0 of 42,950 readings lost, and 4 bugs found and fixed in a Kafka pipeline.

1d ago · 13 min read · techblog01.hashnode.dev  
Read the article

-   [#kafka](https://hashnode.com/tag/kafka)
-   [#chaos-engineering](https://hashnode.com/tag/chaos-engineering)
-   [#python](https://hashnode.com/tag/python)
-   [#devops](https://hashnode.com/tag/devops)
-   [#reliability](https://hashnode.com/tag/reliability)

## Written by

[Sudhanshu Wani](https://hashnode.com/@TechBlog01)

@TechBlog01

  
[View profile](https://hashnode.com/@TechBlog01)

Full-stack engineer building real-time data pipelines, moving into cloud and AI.

I'm a full-stack engineer working mostly on backends and real-time systems: Python and FastAPI, Kafka and MQTT, Postgres and TimescaleDB, with React on the front end and CI and infrastructure around it. I write about what I build and break: the design trade-offs, the numbers I measured, and what I'd do differently. Now I'm moving into cloud and AI, and writing up what I learn along the way.

## Responses 1

Markdown supported · ⌘↵ to post

[Mike Dabydeen](https://hashnode.com/@firelinks)

[5h ago](https://hashnode.com/posts/kafka-chaos-testing-restart-vs-kill/6afc48e1485f929a3f847aeb/comment/6afd97034300b22e5f05ab85)

For "recovered" I'd keep your on-time-window rule and change what you report: a spread over repeated kills, because where the kill lands in the commit cycle decides how much the processor replays. Quix Streams commits offsets and state every 5 seconds by default, and a killed processor resumes from the last commit. A kill just after a commit replays almost nothing; one just before it replays close to five seconds of input, which has to drain before the screen is on time again.

So I'd run each kill about 20 times, with a random delay so the signal lands at different points in that cycle, and report the median and the worst run side by side. Your single-sample caveat already names the gap, and I'd expect this to be the variable behind most of it.
