---
title: "Mike Dabydeen (@firelinks) | Hashnode"
description: "VP Engineering @ UREEQA. Read the latest articles by Mike Dabydeen on Hashnode."
url: "https://hashnode.com/@firelinks"
---

@firelinks

# Mike Dabydeen

@firelinks · Toronto, ON, Canada · Joined September 2021

VP Engineering @ UREEQA

[Website](https://michaeldabydeen.com/)

## Comments

[You can make that overlap window a number you choose. Two attempts only overlap on an outside call if the old worker starts it with less lease left than the call can take. So before each outside call, have the worker compare the lease time it has left, counted from its last successful heartbeat on its own monotonic clock, against that call's client timeout, and renew or stop if the margin is short. With a 30 second lease and a 10 second timeout on the ticket API, a worker holding 8 seconds won't make the call. A reclaim then can't start until the old call has returned or timed out, so search-before-create finds what it made. What's left is a call the other system completes after your client gave up on it. Only the external reference search catches that one, which is why I'd keep both.](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd96c74300b22e5f05a8fd#reply-6b003a4175c660b7bfe59e2f)

Reply · Article · 2d ago · [I Kill My Workers on Purpose](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51)

[On what to count: per run where any false hypothesis reaches HIGH, which you're already doing, with a confidence bound reported next to it, because zero in 29 runs is a smaller claim than it looks. With no failures in n runs, the 95% upper bound on the true rate is roughly 3/n, so 29 clean runs still allows a false HIGH rate near 1 in 10. That's enough to show the challenge helped: at your old rate of 7 in 57, getting 29 clean runs in a row by chance is about a 2% event. To claim under 1 in 20 you'd need about 60 clean runs, and about 150 for 1 in 50. That gives the larger measurement runs since a job. Add them to the 29 and publish the run count with the bound, and an on-call reader can see what HIGH is worth today. You'll also see it move when a new scenario or a model change shifts the rate, instead of finding out from a bad page at 3 am.](https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1/comment/6b003a1d51a88417780e2ad4)

Comment · Article · 2d ago · [My AI SRE said "HIGH confidence" and was wrong. What it took to make HIGH mean HIGH](https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1)

[There's a signal inside the grace window that can separate a retry from a theft: whether the replacement token has been used yet. A lost response means the browser never received the new refresh token, so it can't present it. If the new token has already come back to the server and the old one shows up after that, the request isn't a retry from a dropped response. Two parties are holding the same session. RFC 9700 (section 4.14.2) describes the response for that case: the server can't tell which side is the attacker, so it revokes the active token and the client has to log in again. With rotation down to once a day per session, the check is cheap. Keep the replacement's id on the revoked row, record when it was first used, and honour the 60 seconds only while that field is empty. It won't catch a replay that lands before the browser's next refresh, but someone mashing Ctrl+F5 keeps their login, and a stolen token presented after the real refresh ends the session instead of minting a fresh access token.](https://hashnode.com/posts/a-revoked-refresh-token-that-still-works-for-60-seconds/6afffac3f6448aa9fa4fec97/comment/6b0039fe8ec420b77d55c582)

Comment · Article · 2d ago · [A Revoked Refresh Token That Still Works for 60 Seconds](https://hashnode.com/posts/a-revoked-refresh-token-that-still-works-for-60-seconds/6afffac3f6448aa9fa4fec97)

[For "recovered" I'd keep your on-time-window rule and change what you report: a spread over repeated kills, because where the kill lands in the commit cycle decides how much the processor replays. Quix Streams commits offsets and state every 5 seconds by default, and a killed processor resumes from the last commit. A kill just after a commit replays almost nothing; one just before it replays close to five seconds of input, which has to drain before the screen is on time again. So I'd run each kill about 20 times, with a random delay so the signal lands at different points in that cycle, and report the median and the worst run side by side. Your single-sample caveat already names the gap, and I'd expect this to be the variable behind most of it.](https://hashnode.com/posts/kafka-chaos-testing-restart-vs-kill/6afc48e1485f929a3f847aeb/comment/6afd97034300b22e5f05ab85)

Comment · Article · 4d ago · [Chaos drills for a streaming system: what restarting and killing every component taught me](https://hashnode.com/posts/kafka-chaos-testing-restart-vs-kill/6afc48e1485f929a3f847aeb)

[When the idempotency keys go in, I'd build them from the job id and step, never from the attempt number. The attempt counter is the right fence for your own tables, but if it ends up in the key, a reclaimed job hands the other system a key it has never seen, and the duplicate ticket gets created anyway. The fence has to change on every claim; the key has to stay the same across them. The heartbeats you're planning can carry the same token. If the heartbeat runs UPDATE jobs SET lease\_expires\_at = ... WHERE id = $1 AND attempts = $2 and gets zero rows back, the worker has already lost the job and should stop before its next external call, instead of learning that when its success write matches nothing. For a system that doesn't accept idempotency keys, writing the job id into the record it creates, in an external reference field or a label, gives the retry something to search for before it creates another one.](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51/comment/6afd96c74300b22e5f05a8fd)

Comment · Article · 4d ago · 1 · [I Kill My Workers on Purpose](https://hashnode.com/posts/crash-safe-workflow-engine-postgres/6afcf385577ca6d1f890dc51)

[The origin can't always be the loud one, and your GitLab example shows why: the backup job did try to report, and the mail server rejected its failure emails. A broken component often can't say it's broken. So I'd pair loud-at-origin with a check that expects good news and alarms when it doesn't arrive, run by something other than the job itself. A rule like "no backup above a size floor in the last 26 hours" would have fired there without the backup job having to work. The fallback path needs the same treatment from the caller's side. The caller is the only place that can count how often it fell back, so it should publish that rate as a number with a named owner and a ceiling, rather than page on it. A cache serving stale answers 2% of the time is probably fine; 40% for a week means the real path is gone, and the trend shows that long before the day it breaks.](https://hashnode.com/posts/auditing-the-invisible/6af8d7e8e16b64e1e8a1ee69/comment/6afaf5891694eeaed3bd0d1c)

Comment · Article · 6d ago · [Auditing the invisible](https://hashnode.com/posts/auditing-the-invisible/6af8d7e8e16b64e1e8a1ee69)

[I'd split the funnel record into a start and an end, because emitting it after the run leaves the accounting rule with nothing to check in the case that hurts most. When a worker hits its memory limit, the kernel kills it with SIGKILL, and Python never runs the finally block. The item in flight and the run record both vanish, so the run looks the same as one that never started. If digest.run.start goes out as soon as the query returns, carrying run\_id and selected, an end record can close it. A start with no end after the job's normal duration means it died partway, and selected minus the decision records already written tells you how many users it left behind. A missing start at the scheduled time catches the cron that never fired, which is the never-reached case from your third bullet.](https://hashnode.com/posts/what-to-log-silent-failures-decision-records/6afa862cc1fd081482ee4d9b/comment/6afaf56e1694eeaed3bd0bf3)

Comment · Article · 6d ago · [Silent Failures: What to Log When a Bug Leaves No Error](https://hashnode.com/posts/what-to-log-silent-failures-decision-records/6afa862cc1fd081482ee4d9b)

[The export option can be made to fail loudly with one assertion in the job that publishes it: compare the exported holders against the directory's list of disabled accounts and stop the publish if the overlap isn't empty. That keeps the no-schema-change route and gives it the failure signal the active attribute gets from the policy tests.](https://hashnode.com/posts/agent-authorization-in-production/6aebd30687a2fa4bdc363b9c/comment/6aedc67819115b2ec5c5e8b7#reply-6afaf5581694eeaed3bd0ba7)

Reply · Article · 6d ago · [Authorizing AI Agent Chains in Production](https://hashnode.com/posts/agent-authorization-in-production/6aebd30687a2fa4bdc363b9c)

[Logging it as a separate record that points back to the decision would leave the envelope as it is: the hash of the held call, the approver, the outcome, and the time from hold to answer. That last field is the early warning. When most approvals for send\_money come back within a couple of seconds, nobody is reading the arguments, and the hold has stopped working well before the approve rate tells you. For send\_money and send\_email, where the attack and the task share a tool, the argument that usually separates them is the recipient. A payee or domain already in the user's history could pass, and a new recipient is what waits for a person.](https://hashnode.com/posts/fail-closed-authorization-for-ai-agent-tool-calls-and-the-audit-record-behind-it/6af6ec76f8d8584368c29d20/comment/6af7000bea6cac6a25c84fe9#reply-6afaf529503a002400dbd6e3)

Reply · Article · 6d ago · [Fail-closed authorization for AI agent tool calls, and the audit record behind it](https://hashnode.com/posts/fail-closed-authorization-for-ai-agent-tool-calls-and-the-audit-record-behind-it/6af6ec76f8d8584368c29d20)

[I'd have the Lua script return the wait time, because window minus elapsed tells a refused client the wrong thing about a sliding window. With the weighted estimate, capacity comes back gradually, at prev/window per second, about 1.7 requests per second in your 100-per-minute example. A client refused because of last window's weight is often allowed again within a second or two, yet Retry-After sends it to the boundary. Every client refused during that minute gets the same boundary, so they all come back together, and the edge burst you removed returns through the retry path. Inside the script, when estimate >= limit and prev > 0, the wait is (estimate - limit) \* window / prev, rounded up with a one-second minimum and capped at the time left in the window. One small thing in the caller too: int() truncates, so a client refused at 30.4 seconds is told 29 and arrives just before the reset. math.ceil fixes that.](https://hashnode.com/posts/distributed-rate-limiting-effective-limit/6af6977ee0fa0b84b693d1e2/comment/6af70049f758036b040a3fed)

Comment · Article · Nov 13 · 1 · [Distributed Rate Limiting: Your Effective Limit Isn't Your Config](https://hashnode.com/posts/distributed-rate-limiting-effective-limit/6af6977ee0fa0b84b693d1e2)
