In a nutshell - they are different solutions for solving different problems developed by different companies.

Shadow DOM

Shadow DOM solves a single problem that has plagued javascript developers since the inception of HTML - lack of encapsulation. You include a third party stylesheet that includes some generic styling for table and now your calendar widgets are all broken - sounds familiar ? Congratulations - the future of the web platform has a solution for you : it is the Shadow DOM.

Shadow DOM was initially implemented by browser developers to implement encapsulated widgets like HTML 5 slider input element.

As Dimitri Glazkov has described eloquently in this post:

You see, browser developers realized that coding the appearance and behavior of HTML elements completely by hand is a) hard and b) silly. So they sort of cheated.

They created a boundary between what you, the Web developer can reach and what’s considered implementation details, thus inaccessible to you. The browser however, can traipse across this boundary at will. With this boundary in place, they were able to build all HTML elements using the same good-old Web technologies, out of the divs and spans just like you would.

The Shadow DOM spec makes exactly same encapsulation facility available to web developers:

This specification describes a method of combining multiple DOM trees into one hierarchy and how these trees interact with each other within a document, thus enabling better composition of the DOM.

Alternatives

It is not that browser's haven't had other approaches for encapsulation before:

1. Frames: Frames are heavy and are too isolated - they have their own execution context. Events don't bubble through frame boundaries. While they have their use cases (widgets containing third party content in a site), they are often not suitable for fine grained encapsulation.

2. Applets/Flash: Require plugins to be installed, require separate toolchains to be developed, plagued with security issues.

Polymer

Polymer is a reference implementation from Google that works by polyfilling a huge number of DOM apis for legacy browsers. While polymer is quite performant in chrome, do run real world benchmarks if a significant fraction of your users come from other/older browsers.

Virtual DOM

Virtual DOM is a solution for expensive layout recomputation and browser reflow that happens when the DOM gets mutated. While mutations themselves are not too expensive, successive reflows on multiple mutations can add up to a poor user experience - especially for highly interactive/dynamic web applications.

React (the original pioneer of virtual dom)

React is a component oriented UI library that utilizes virtual DOM. It is proven in large scale codebases and allows you drop down to direct DOM manipulation without too much hassle for performance critical components.

React creates a tree data structure that maps to the DOM hierarchy and allows developers to write their code as if the entire component hierarchy was being re-rendered on every action. However these re-renderings are processed as mutations in the virtual dom (the virtual term indicates that any native api provided by the host environment - browser is not in use here) which are asynchronously applied in batch wise fashion to the DOM, thereby avoiding reflows.

Interoperability

React aims on interoperability with Shadow DOM but does not intend to use shadow DOM as a foundational layer for its component model:

In word of Sebastian Markbage:

We’re not going to use it at all at Facebook. We’re not going to build React on it because there’s a strong model difference – imperative in Web Components to declarative in React. Web Components doesn’t have an idiomatic way to define things like where events go. How do you pass data when everything is a string? We see it more as an interop layer that lets various frameworks talk to each other.

Encapsulation

What about the original problem we set out to solve ? Encapsulation.

The component model may provide a great solution for encapsulating DOM sub-trees, it does not provide the isolation that Shadow DOM provides.

The solution for style encapsulation that has become popular in React community is through inline styling. The community has developed tools that make management of inline styling easier - popular solutions include Khan/aphrodite and FormidableLabs/radium

I have personally faced significant friction from other UI developers when proposing inline styles and many teams might find a better tradeoff by adopting strong conventions.

Tighter integration:

The community has come up with multiple solutions that integrate these technologies more tightly:

• Wildhoney/ReactShadow:

With ReactShadow you can apply a Shadow DOM root inside of your component. Under normal React.js conditions, your styles are written inline for style encapsulation – with ReactShadow your styles can now be moved into their rightful place – within CSS documents!

• Wildhoney/Maple.js:

Maple.js is a React webcomponents based framework mixing ES6 with Custom Elements, HTML Imports and Shadow DOM. It has in-built support for SASS and JSX, including a Gulp task for vulcanizing your project.

The Shadow DOM that Polymer is using is an HTML5 specification you can read about here. A Virtual DOM, used by React, is an in memory representation of the actual DOM.

Mutating the actual DOM isn't very performant, the Virtual DOM being pure javascript is thus way faster. This is a key benefit to using a Virtual DOM and that's one of the reasons why everybody is so hyped about it.

Personally I haven't worked with the Shadow DOM, but reading through the spec, the Shadow DOM has been created to enable better composition of the DOM:

A method of combining multiple DOM trees into one hierarchy and how these trees interact with each other within a document, thus enabling better composition of the DOM.

TL;DR; The Virtual DOM is an in-memory representation of the actual DOM whereas the Shadow DOM is an HTML5 specification created to enable better composition of the DOM.