Although this isn't a very popular opinion, I've found Lumen to be really great to build an API with.
It's very light and I haven't come up against any problems when trying to create a fairly complex API. There's nothing in there that I've thought "Oh damn! We should be using a full-blown Laravel installation."
Admittedly, it is missing a couple of features that you might be used to if you're familiar with Laravel, but if you don't mind simplifying/separating some stuff out (i.e. OAuth in a separate service OR just using API keys, for example) and writing a little bit of piping, you can achieve great things.
The performance of Lumen is great for our needs and we've managed to create a really flexible and efficient API with it that conforms to the JSON-API standard, uses OAuth for access delegation and much more, without tying us to a particular package.
I'd be happy to go into more detail separately about some of the things we've done to make this work well.