If using pure JavaScript to build a node app, yes, what we can gain from the modern bundler is very little.
But if you're using TypeScript, a transpiling/ building step is unavoidable. That's where the modern bundlers shine.
What nodemon + esbuild (or swc) does is transpiling your entire codebase superfast and start your server whenever a code change.
HMR could help accelerate the feedback loop if we have a considerable backend codebase, even if we've replaced babel/ tsc with esbuild/ swc.