Been there. Cold starts are real, but the actual problem is usually what you discovered: wasteful initialization. Module-level init is the right move.
That said, 3.2s median suggests something else was happening too. If you're still on Lambda, consider whether gRPC over HTTP/2 is the right choice here. We ditched gRPC for internal services and moved to Postgres subscriptions for async work. Eliminated a whole category of connection overhead.
For stateful services like yours, Lambda was probably the wrong tool. Did you evaluate moving to ECS or even just a box.