You could just make a standalone Python module that classifies images and outputs their class. Then call this from Go as an external command. It doesn't seem like a lot of integration is needed, so I'd go with this simple approach.
(You might also consider just using Python for the api server. Nothing against Go but it sounds like a simple layer so maybe that the language doesn't really matter.)