Really if you have a proper viewport meta, are designing in %/EM font sizes with EM based elastic layout (in addition to the semi-fluid and responsive you should also have), and following things like the WCAG guidelines, this should take care of itself in all but ONE corner case, the new HDX displays that don't lie to us about their scaling... like the Kindle Fire HDX. (Apple retina even with viewport meta and -webkit-text-resize:none STILL bold faced lie to us, and interestingly that's a good thing!)

A good rule of thumb is that you should avoid using webfonts, particularly those with thin-glyphs or non-standard aspect ratios for characters on any "flow text" -- your actual content paragraphs. Universally they make you either use larger sizes than you should need, or compromise legibility. GOOD use of webfonts is to do so with a eye-dropper for a handful of visual cutesies -- NEVER use them on flow text unless you want someone somewhere at sometime to bitch about legibility.

Another thing is that serif fonts often compromise legibility on screen media. This flies in the face of print thinking, where it's universally accepted that serifs aid in the flow of text legibility... and they do when you have the dpi to render them properly or analog typeset print; but on screen we just don't have the pixels on most desktops and laptops (though this is changing slowly) to render serif fonts clearly... so times, garamond, all the classics? DON'T DO IT! As the mantra being pushed 15 years ago went: "the web is not print!"

For MOST system level sans-serif fonts ("Segoe ui",arial,helvetica,sans-serif), the SMALLEST size you should allow is 0.85REM, which works out to 14px for VGA/96dpi/16px/100%/Windows small users, and 17px for 8514/120dpi/20px/125%/Windows Large/Win7+ Medium users, and for things like 200% scale media center systems or retina displays anywhere from 24 to 28px. Again why you're supposed to use %/em is so the entire layout (even your media queries) auto-scale to the default font size the user or OS has chosen for the browser!

ANYTHING less than 14px is generally speaking a giant middle finger to usability. ANYONE telling you otherwise doesn't know enough about websites or accessibility to be flapping their gums on the topic... Even so NEVER declare your font sizes in pixels without a damned good reason -- the only legitimate reasons to do so being for things like behind gilder-levin image replacement or other advanced layout techniques. You declare a font size in pixels on your content, you've told users with accessibility needs to go plow themselves.

SINCE your layout widths, paddings, line-heights and media queries should all (for the most part) be EM based -- or REM, though if you care about legacy users you can't use that yet :( -- the entire layout will auto-scale to the target device or browser settings, and you can leverage that for things like that HDX corner case as a correction. IF you declare the most commonly used font size and face on BODY, you then only need to change the size for things like headings (which from a logical document structure point of view is the ONLY legitiamate reason to have text of different sizes) or things like tag clouds. Delcare it once, work relative enlarging off that only when needed!

Then for things like the HDX you can trap them thusly:

@media
    (-webkit-min-device-pixel-ratio:2) and (min-width:1600px), 
    (min-resolution:172dpi) and (min-width:1600px)
{
    html { font-size:200%; }
}

Boom, instantly enlarging the entire layout (since HTML is body's parent) for higher DPI high resolution units.

Again, the WCAG says use EM, so FREAKING USE EM!!!

One other thing to always keep in mind is that every font face usually has different metrics at the same declared size. I made a simple image some time ago to illustrate just how bad that can get:

http://www.cutcodedown.com/images/faceCompare.png

If you declared font-size:1em; for all those faces, the top half is what you'd get on "normal" systems, the bottom half is what you'd get on classic windows if you turn on "large fonts" or whatever they were calling it that YEAR... notice how none of those faces are actually the same size despite the same declaration of size. You also have to keep in mind that with system fonts, you can't rely on the renderer treating the same fonts the same, or that any font you choose will even exist on the target. Whilst webfonts are supposed to help with that, the differences in rendering quality and the majority of webfonts in circulation do more harm than good. You might get a more consistent appearance across platforms, but typically at the same time that consistency costs you usability and accessibility. This is why having a semi-fluid layout that adjusts to the content instead of fixed layout that forces you to design content to the layout is a VERY important thing... and something the fools dicking around in Photoshop under the DELUSION that they are "designers" are typically ignorant of -- just like how they are utterly unaware of every other aspect of accessibility.

A final note, I HIGHLY recommend taller line heights to improve long line legibility. People say 120% to 130%, I go for 150% as default for screen media, sometimes pushing as much as 175%. It just makes the beginning and end of lines clearer. I know on mobile it can be tempting to try and lower the line-height to try and stuff more text on the screen, but if you are sacrificing legibility or "visual flow" to do so, you're not helping the user.

As @gskema it is a good idea to copy or follow some frameworks styles for fonts, but it won't always help you if you use a special font for your site. The things that I've been reading about usability and readability is that the minimum font size that you can use for users to read it without any difficulty is 16px, that should be your base and minimum size, but as always it depends on every font, some are naturally bigger than others but that's kind of a standard. You could put your body to be at default 16px and then use ems to change sizes for desktop and stuff.

Also never use line-height at 100% or 1 use it at 120% or 130% for it to be easier to read. Hope this work for you.