Disclaimer: I led development of gaming.youtube.com. We chose Polymer at the time for non-technical reasons.
Some downsides:
Performance is not great.
Lack of coherent data architecture (e.g. Redux).
Difficult to manage state change (as opposed to React) due to a more limited programming model - e.g. template ifs and repeats don't know to reevaluate in many cases. This makes it harder to integrate with third party js libraries.
Inferior tooling vs React.
Philosophically Polymer is more about the DOM and React is more JS-focused. Some of this is personal preference.
Some downsides: Performance is not great. Lack of coherent data architecture (e.g. Redux). Difficult to manage state change (as opposed to React) due to a more limited programming model - e.g. template ifs and repeats don't know to reevaluate in many cases. This makes it harder to integrate with third party js libraries. Inferior tooling vs React.
Philosophically Polymer is more about the DOM and React is more JS-focused. Some of this is personal preference.