Yes, it's absolutely safe to use CSS Flexbox, provided you use appropriate fallback. With the Hashnode web app, we are using CSS Flexbox in most of the components. If your audience uses IE11 and above, you should definitely take advantage of CSS Flexbox.
As you have rightly pointed out, most of the browsers have implemented most of the Flexbox functionalities.
Talking about the fallbacks, here are a few which we are using:
display property:display: -webkit-box;
display: -ms-flexbox;
display: flex;
order property:-webkit-box-ordinal-group: 3;
-ms-flex-order: 2;
order: 2;
flex-direction property:-webkit-box-orient: horizontal;
-webkit-box-direction: normal;
-ms-flex-direction: row;
flex-direction: row;
Similarly, there's a fallback for all the Flexbox keywords. Use this tool (Pleeease Play) for generating, and confirming the fallback rules.
TIP: A Complete Guide to Flexbox should be your CSS Flexbox bible.