We have built something in-house for managing tenant configurations very similar to what you have described. It does layering and overrides, and there's a client that attaches to applications and picks up config changes dynamically. So no restarts are required when you change configurations.
I've been looking for out of the box solutions for the same and found Spring Cloud Config. Check it out. cloud.spring.io/spring-cloud-config