There are a lot of ways to write a controller and a lot of types of objects that are supposedly called a controller... but you are on to something.
A well-written controller can adhere to SRP, but way too often I've seen the rule violated.
An example of a good controller: light, short, orchestrates input from/output to a user, everything else is outside it.
An example of a bad controller: fat, do-everything compilation of logic. Also, whatever passes for a controller in Angular.