tl;dr: Dan Luu has a detailed post where he tracks in detail past predictions and argues that contra Karnofsky, Arb, etc, the track record of futurists is overall quite bad. Relevantly to this audience, he further argues that this is evidence against the validity of current longtermist efforts in long-range predictions.

(I have not finished reading the post). 

______

I've been reading a lot of predictions from people who are looking to understand what problems humanity will face 10-50 years out (and sometimes longer) in order to work in areas that will be instrumental for the future and wondering how accurate these predictions of the future are. The timeframe of predictions that are so far out means that only a tiny fraction of people making those kinds of predictions today have a track record so, if we want to evaluate which predictions are plausible, we need to look at something other than track record.

The idea behind the approach of this post was to look at predictions from an independently chosen set of predictors (Wikipedia's list of well-known futurists¹) whose predictions are old enough to evaluate in order to understand which prediction techniques worked and which ones didn't work, allowing us to then (mostly in a future post) evaluate the plausibility of predictions that use similar methodologies.

Unfortunately, every single predictor from the independently chosen set had a poor record and, on spot checking some predictions from other futurists, it appears that futurists often have a fairly poor track record of predictions so, in order to contrast techniques that worked with techniques that I didn't, I sourced predictors that have a decent track record from my memory, an non-independent source which introduces quite a few potential biases.

Something that gives me more confidence than I'd otherwise have is that I avoided reading independent evaluations of prediction methodologies until after I did the evaluations for this post and wrote 98% of the post and, on reading other people's evaluations, I found that I generally agreed with Tetlock's "Superforecasting" on what worked and what didn't work despite using a wildly different data set.

In particular, people who were into "big ideas" who use a few big hammers on every prediction combined with a cocktail party idea level of understanding of the particular subject to explain why a prediction about the subject would fall to the big hammer generally fared poorly, whether or not their favored big ideas were correct. Some examples of "big ideas" would be "environmental doomsday is coming and hyperconservation will pervade everything", "economic growth will create near-infinite wealth (soon)", "Moore's law is supremely important", "quantum mechanics is supremely important", etc. Another common trait of poor predictors is lack of anything resembling serious evaluation of past predictive errors, making improving their intuition or methods impossible (unless they do so in secret). Instead, poor predictors often pick a few predictions that were accurate or at least vaguely sounded similar to an accurate prediction and use those to sell their next generation of predictions to others.

By contrast, people who had (relatively) accurate predictions had a deep understanding of the problem and also tended to have a record of learning lessons from past predictive errors. Due to the differences in the data sets between this post and Tetlock's work, the details are quite different here. The predictors that I found to be relatively accurate had deep domain knowledge and, implicitly, had access to a huge amount of information that they filtered effectively in order to make good predictions. Tetlock was studying people who made predictions about a wide variety of areas that were, in general, outside of their areas of expertise, so what Tetlock found was that people really dug into the data and deeply understood the limitations of the data, which allowed them to make relatively accurate predictions. But, although the details of how people operated are different, at a high-level, the approach of really digging into specific knowledge was the same.

Because this post is so long, this post will contain a very short summary about each predictor followed by a moderately long summary on each predictor. Then we'll have a summary of what techniques and styles worked and what didn't work, with the full details of the prediction grading and comparisons to other evaluations of predictors in the appendix.

• Ray Kurzweil: 7% accuracy
  • Relies on: exponential or super exponential progress that is happening must continue; predicting the future based on past trends continuing; optimistic "rounding up" of facts and interpretations of data; panacea thinking about technologies and computers; cocktail party ideas on topics being predicted
• Jacque Fresco: predictions mostly too far into the future to judge, but seems very low for judgeable predictions
  • Relies on: panacea thinking about human nature, the scientific method, and computers; certainty that human values match Fresco's values
• Buckminster Fuller: too few predictions to rate, but seems very low for judgeable predictions
  • Relies on: cocktail party ideas on topics being predicted to an extent that's extreme even for a futurist
• Michio Kaku: 3% accuracy
  • Relies on: panacea thinking about "quantum", computers, and biotech; exponential progress of those
• John Naisbitt: predictions too vague to score; mixed results in terms of big-picture accuracy, probably better than any futurist here other than Dixon, but this is not comparable to the percentages given for other predictors
  • Relies on: trend prediction based on analysis of newspapers
• Gerard K. O'Neill: predictions mostly too far into the future to judge, but seems very low for judgeable predictions
  • Relies on: doing the opposite of what other futurists had done incorrectly, could be described as "trying to buy low and sell high" based on looking at prices that had gone up a lot recently; optimistic "rounding up" of facts and interpretations of data in areas O'Neill views as underrated; cocktail party ideas on topics being predicted
• Patrick Dixon: 10% accuracy; also much better at "big picture" predictions than any other futurist here (but not in the same league as non-futurist predictors such as Yegge, Gates, etc.)
  • Relies on: extrapolating existing trends (but with much less optimistic "rounding up" than almost any other futurist here); exponential progress; stark divide between "second millennial thinking" and "third millennial thinking"
• Alvin Toffler: predictions mostly too vague to score; of non-vague predictions, Toffler had an incredible knack for naming a trend as very important and likely to continue right when it was about to stop
  • Relies on: exponential progress that is happening must continue; a medley of cocktail party ideas inspired by speculation about what exponential progress will bring
• Steve Yegge: 50% accuracy; general vision of the future generally quite accurate
  • Relies on: deep domain knowledge, font of information flowing into Amazon and Google; looking at what's trending
• Bryan Caplan: 100% accuracy
  • Relies on: taking the "other side" of bad bets/predictions people make and mostly relying on making very conservative predictions
• Bill Gates / Nathan Myhrvold / old MS leadership: timeframe of predictions too vague to score, but uncanny accuracy on a vision of the future as well as the relative importance of various technologies
  • Relies on: deep domain knowledge, discussions between many people with deep domain knowledge, font of information flowing into Microsoft

[...]