The async-does-not-mean-parallel confusion is the single most expensive misunderstanding in this space. Marking a handler async makes it yield on IO, and a forward pass is not IO - it is CPU or GPU work that holds the loop, so one slow inference stalls every other request on that worker and the symptom looks like the whole service degrading at once. Running inference in a threadpool or a separate worker process is the fix, and it also gives you somewhere to put a queue. The memory half deserves the same emphasis: without a bound on in-flight requests, load does not produce a slow service, it produces an OOM kill, because every waiting request is holding tensors. Rejecting early once the queue is full is friendlier than accepting work you cannot finish, and it is what turns an outage into elevated error rates.
The async-does-not-mean-parallel confusion is the single most expensive misunderstanding in this space. Marking a handler async makes it yield on IO, and a forward pass is not IO - it is CPU or GPU work that holds the loop, so one slow inference stalls every other request on that worker and the symptom looks like the whole service degrading at once. Running inference in a threadpool or a separate worker process is the fix, and it also gives you somewhere to put a queue. The memory half deserves the same emphasis: without a bound on in-flight requests, load does not produce a slow service, it produces an OOM kill, because every waiting request is holding tensors. Rejecting early once the queue is full is friendlier than accepting work you cannot finish, and it is what turns an outage into elevated error rates.