That's an interesting point about prediction markets. We individuals tend to invest in the stock market even when we know the market as a whole is wiser than us as individuals, because on the whole the market goes up, and anyways there are ways to track overall market performance. For prediction markets, I suppose there would need to be similar incentives somehow, otherwise every individual who doesn't have special information would be better off predicting what the overall market predicts, which doesn't help.
I'm guessing I just don't understand how prediction markets work. Hoping someone will correct me.
This is an excellent post. I've been struggling myself to understand to what extend deontological values and the inherent irrationality of humans need to be factored into consequentialist decision making. I've become more and more convinced that values and social norms matter much more than I had previously thought.
The last chapter of Global Catastrophic Risks (Bostrom and Circovic) covers global totalitarianism. Among other things they mention how improved lie-detection technology, anti-aging research (to mitigate risks of regime change), and drugs to increase docility in the population could plausibly make a totalitarian system permanent and stable. Obviously an unfriendly AGI could easily do this as well.