As applications scale, infrequent problems become significant. A network failure for 0.1% of requests is trivial at 1,000 requests per day, but a nightmare for customer support at 1,000,000 requests per day. This commonly happens when we have externa...

Markdown supported · ⌘↵ to post
Thanks for the article, it will save me a lot of time next week! 😅 I have some concerns.
1/ For the
createProcessTickermethod:try { for (const record of records) { await processRecord(record); await cursorRepo.setPosition(record.id); processedRecords++; } } catch (error) { logger.error('Could not process record', { processName, record, error, }); } finally { await cursorRepo.unlockCursor(); }Should
errorbe forwarded in thecatchblock? Otherwise, the cursor process won't terminate if failed to process a record.2/ For this termination flow
} catch (error) { logger.critical('Terminating server: cursor process error', { processName: props.processName, error, }); // we use structured logging await lifecycle.close(); process.exit(1); }When a node is terminated and the cursor lock is released, will the terminated process propagate to the remaining nodes?
I guess, there will be the scenario that container managers (CR, K8S) keep initialize new nodes to replace terminated nodes until the error is fixed.