My first thought is to avoid low level access control management. Consider to apply authorization at service model level (API endpoint). Different models represents different composition of physical entities so you can set which composition available to who.
You can create a restricted/sensitive model which contains:
And you can also create less restricted model which contains:
Each data model (composition) have its own API endpoint. Then you can apply any type of authorization logic on them. You can use an external authorization server or you can store access rights assignments in each MS while you can still use an external authentication server.