---
title: "Discussion on \"Auditing the invisible\" | Hashnode"
description: "Discussion on \"Auditing the invisible\". Originally published at https://ketushah.com/writing/auditing-the-invisible/\nEvery system I build has two jobs, and they fight each other: it has to keep working when something breaks, and it has to t"
url: "https://hashnode.com/posts/auditing-the-invisible/6af8d7e8e16b64e1e8a1ee69"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Auditing the invisible](https://ketushah.hashnode.dev/auditing-the-invisible?utm_source=hashnode&utm_medium=feed)

Originally published at https://ketushah.com/writing/auditing-the-invisible/ Every system I build has two jobs, and they fight each other: it has to keep working when something breaks, and it has to t

2d ago · 5 min read · ketushah.hashnode.dev  
Read the article

-   [#devops](https://hashnode.com/tag/devops)
-   [#discuss](https://hashnode.com/tag/discuss)
-   [#reliability](https://hashnode.com/tag/reliability)

## Written by

[Ketu Shah](https://hashnode.com/@ketushah)

@ketushah

  
[View profile](https://hashnode.com/@ketushah)

## Responses 2

Markdown supported · ⌘↵ to post

[Mike Dabydeen](https://hashnode.com/@firelinks)

[19h ago](https://hashnode.com/posts/auditing-the-invisible/6af8d7e8e16b64e1e8a1ee69/comment/6afaf5891694eeaed3bd0d1c)

The origin can't always be the loud one, and your GitLab example shows why: the backup job did try to report, and the mail server rejected its failure emails. A broken component often can't say it's broken. So I'd pair loud-at-origin with a check that expects good news and alarms when it doesn't arrive, run by something other than the job itself. A rule like "no backup above a size floor in the last 26 hours" would have fired there without the backup job having to work.

The fallback path needs the same treatment from the caller's side. The caller is the only place that can count how often it fell back, so it should publish that rate as a number with a named owner and a ceiling, rather than page on it. A cache serving stale answers 2% of the time is probably fine; 40% for a week means the real path is gone, and the trend shows that long before the day it breaks.

[Ketu Shah](https://hashnode.com/@ketushah)

[17h ago](https://hashnode.com/posts/auditing-the-invisible/6af8d7e8e16b64e1e8a1ee69/comment/6afaf5891694eeaed3bd0d1c#reply-6afb121d7a996267e39a7fdd)

You are right, I missed that. It is in the article now: [ketushah.com/writing/auditing-the-invisible](https://ketushah.com/writing/auditing-the-invisible/)
