The Shadow DOM that Polymer is using is an HTML5 specification you can read about here. A Virtual DOM, used by React, is an in memory representation of the actual DOM.
Mutating the actual DOM isn't very performant, the Virtual DOM being pure javascript is thus way faster. This is a key benefit to using a Virtual DOM and that's one of the reasons why everybody is so hyped about it.
Personally I haven't worked with the Shadow DOM, but reading through the spec, the Shadow DOM has been created to enable better composition of the DOM:
A method of combining multiple DOM trees into one hierarchy and how these trees interact with each other within a document, thus enabling better composition of the DOM.
TL;DR; The Virtual DOM is an in-memory representation of the actual DOM whereas the Shadow DOM is an HTML5 specification created to enable better composition of the DOM.