If you want the layout (padding, margins) to scale with font-size, then yes. Otherwise, no. Sometimes padding and margins do look best when they're scaled relative to font-size.

My option is, "rem" and "em" are both unsafe for layout because of rounding to "px". I reject it.

%

I think, (r)em are good for fonts, but should not be used for layout. There are other units which are a lot better fitting.