---
title: "Comment by Eugene Chernysh on \"Designing for Failure: Strategies to Build Resilient, Always-On Services\" | Hashnode"
description: "Hi! Thanks for the insights. Just a quick question: in your experience, what has been the most challenging part of implementing chaos engineering in a productio"
url: "https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca/comment/68765ff3f3b59d43f032aeff"
---

[Discussion](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca)

Comment [Designing for Failure: Strategies to Build Resilient, Always-On Services](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca)

[Eugene Chernysh](https://hashnode.com/@EChernysh)

[Jul 15, 2025](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca/comment/68765ff3f3b59d43f032aeff)

Hi! Thanks for the insights. Just a quick question: in your experience, what has been the most challenging part of implementing chaos engineering in a production environment? how do you ensure it doesn’t cause disruptions for end users?

[Abhishek Vajarekar](https://hashnode.com/@abhishekanantvajarek)

[Jul 15, 2025](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca/comment/68765ff3f3b59d43f032aeff#reply-6876608ed65a01c96a38e1ad)

There wasn’t one "most challenging" part, but rather a series of hurdles we had to address. One challenge was ensuring chaos tests didn’t accidentally impact production or end-user experience, like simulating network latency without affecting live traffic. Another was managing non-critical dependencies that could still cause cascading issues. We solved these by setting up tight monitoring, isolating failure points, and ensuring recovery mechanisms like circuit breakers and rollbacks were ready to act, which allowed us to minimize disruptions.

[Eugene Chernysh](https://hashnode.com/@EChernysh)

[Jul 15, 2025](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca/comment/68765ff3f3b59d43f032aeff#reply-687660f0f2d246670cf65684)

[Abhishek Vajarekar](https://hashnode.com/@abhishekanantvajarek) Sounds like a solid strategy. How do you prioritize which failure scenarios to test, and do you ever have to balance testing depth with the risk of introducing too many disruptions?

[Abhishek Vajarekar](https://hashnode.com/@abhishekanantvajarek)

[Jul 15, 2025](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca/comment/68765ff3f3b59d43f032aeff#reply-68766174ae9c123ce2288574)

[Eugene Chernysh](https://hashnode.com/@EChernysh) We prioritize failure scenarios based on what’s most likely to go wrong and how it could affect the system. For exmaple, we first test things like network lag or databse issues, since those are pretty common. When it comes to balacing how deep we go with testing, we make sure to test in isolated enviroments first, with good monitoring to avoid messing up production. We focus on depth, but always make sure there are recovery steps ready if anything goes wrong.

[Eugene Chernysh](https://hashnode.com/@EChernysh)

[Jul 15, 2025](https://hashnode.com/posts/designing-for-failure-strategies-to-build-resilient-always-on-services/676945c78ff68c96283020ca/comment/68765ff3f3b59d43f032aeff#reply-687661ab8e03967f9ec5fa80)

Nice! Thanks
