It's probably going to boil down to the incentives issue - I'd bet it will be harder to become rich overnight by launching a sidechain that is pegged to bitcoin than to launch a new altcoin from scratch (especially if it's funded by an ICO.) On the flip side, hopefully that means that developers who are more inspired by building ideas than by the prospect of overnight wealth will end up experimenting with sidechains tech because there's ALSO less downside if you fail.
With regard to lightning we're already seeing some "layer 1.5" solutions such as eltoo that are trying to make channel opens and closes more scalable. I've also heard of some folks working on "layer 3" infrastructure though it's too early to tell what the time frame is to see working software.
Prediction markets, or more generally, information markets, have long been theorized and in my opinion are sorely needed if we want to build truly free markets. Theoretically we should be able to make better collective decisions if we just enable people to put their money where their mouth is... of course, there are also a number of negative consequences that can result. It seems like the "wisdom of the crowd" can work in many cases, but not all... and such markets can create perverse incentives that lead to things such as assassination markets. But you have to take the good with the bad, when you're building new tools - it's difficult if not impossible to build a useful tool that can't be abused.