The logs that really save you at 3 a.m. are the ones that let you reconstruct the chain of events, not just identify the service that failed.
Imagine a payment gets authorised, the ledger records the debit, but the downstream settlement worker times out before acknowledging the message. On retry, you need to know whether the original operation committed, whether the retry reused the same idempotency key, and which message attempt produced each log. A line saying settlement failed tells you almost nothing.
I'd carry the trace ID, payment or operation ID, idempotency key, and message attempt through the relevant services, while keeping sensitive payment data out of the logs. OpenTelemetry's trace and span correlation helps connect the events, but the business identifiers are what let you understand what actually happened to that payment.
There's a useful distinction here: logs should explain the decision and state transition, while traces show the path the request took. Having both makes a huge difference when the failure only appears under production timing and retries.
The logs that really save you at 3 a.m. are the ones that let you reconstruct the chain of events, not just identify the service that failed.
Imagine a payment gets authorised, the ledger records the debit, but the downstream settlement worker times out before acknowledging the message. On retry, you need to know whether the original operation committed, whether the retry reused the same idempotency key, and which message attempt produced each log. A line saying
settlement failedtells you almost nothing.I'd carry the trace ID, payment or operation ID, idempotency key, and message attempt through the relevant services, while keeping sensitive payment data out of the logs. OpenTelemetry's trace and span correlation helps connect the events, but the business identifiers are what let you understand what actually happened to that payment.
There's a useful distinction here: logs should explain the decision and state transition, while traces show the path the request took. Having both makes a huge difference when the failure only appears under production timing and retries.