For a small service, I would begin with a bounded local queue when one process owns the model and the workload is predictable. The important limit is across the whole deployment: several HTTP workers each owning a queue can multiply the intended concurrency and load several copies of the model.
A separate serving runtime becomes useful when batching, GPU sharing, or independent scaling is the actual requirement. I would also test client cancellation in either design. Releasing an HTTP slot does not necessarily stop an inference thread, so admission accounting must follow the work until it truly finishes.
The async-does-not-mean-parallel confusion is the single most expensive misunderstanding in this space. Marking a handler async makes it yield on IO, and a forward pass is not IO - it is CPU or GPU work that holds the loop, so one slow inference stalls every other request on that worker and the symptom looks like the whole service degrading at once. Running inference in a threadpool or a separate worker process is the fix, and it also gives you somewhere to put a queue. The memory half deserves the same emphasis: without a bound on in-flight requests, load does not produce a slow service, it produces an OOM kill, because every waiting request is holding tensors. Rejecting early once the queue is full is friendlier than accepting work you cannot finish, and it is what turns an outage into elevated error rates.