Home
Feed

Extend

Hashnode Pro
Hashnode for Agents

Resources

The foreword
Hashnode gql skill

Buildful - AI coding agent that delivers frontier-level results at a fraction of the cost Bug0 - The AI-native e2e QA regression testing Hackathons Changelog Brand @hashnode on X Hashnode on LinkedIn Support - hello+support@hashnode.com Code of Conduct Terms Privacy Sitemap

Search Hashnode

Search posts, tags, users, and pages

Discussion on "Observability and Scaling: Monitoring Event Lag and Handling Failure" | Hashnode

Feed/

Discussion

Madhav Bhasin

Production patterns for backend engineers who are tired of hello-world tutorials.

Jul 13

Observability and Scaling: Monitoring Event Lag and Handling Failure

A distributed event-driven pipeline fails differently from a monolith. When a single service goes down, you get an error page. When an event-driven pipeline degrades, it's often invisible at first — m

blog.madhav.dev10 min read

#gcp #observability #event-driven-architecture #backend-engineering

Responses2

Markdown supported · ⌘↵ to post

Jeongjin Shin

Building small developer and productivity tools.

Jul 13

The oldest_unacked_message_age vs raw backlog distinction is the right one to lead with, most teams alert on the wrong metric early on. On the ordering-key hot spot problem at 10x, did you consider splitting the hot key onto its own subscription instead of sharding the whole topic? That way a bursty user doesn't stall throughput for everyone else sharing the same ordering key space.

Madhav Bhasin

Production patterns for backend engineers who are tired of hello-world tutorials.

Jul 13

Solid approach for predictable key spaces — isolates the blast radius without touching the whole topic. The catch is when the hot key is unpredictable (bots, sudden high-volume users) — you need tooling to detect and re-route dynamically, which is a different problem.

Relaxing ordering on non-critical event types is usually the safer default at scale. But your suggestion is the right first move before you get there.