In this post, I compared forecasts from Ajeya Cotra and from forecasters in the Existential Risk Persuasion Tournament (XPT) relating to some of the inputs to Cotra’s biological anchors model.

Here, I give my personal take on which of those forecasts seem more plausible.

Note that:

• I’m only considering the inputs to the bio anchors model which we have XPT forecasts for. This notably excludes the 2020 training requirements distribution, which is a very important driver of model outputs.
• My take is based on considering the explicit arguments that Cotra and the XPT forecasters gave, rather than on independent research.
• My take is subjective.
• I’ve been working with the Forecasting Research Institute (who ran the XPT) since November 2022, and this is a potential source of bias.
• I’m publishing this post in a personal capacity and it hasn’t gone through FRI’s review process.
• I originally wrote this early in 2023. I've tried to update it as new information came out, but I likely haven't done a comprehensive job of this.

To recap, here are the relevant forecasts:

See workings here and here^[1]. 
*The 'most aggressive' and 'most conservative' forecasts can be considered equivalent to 90% confidence intervals for the median estimate.^[2]

Hardware

• For FLOP/$ in 2025, I think both Cotra and the XPGT forecasters are wrong, but Cotra will prove more right.
  • Epoch’s current estimate of highest GPU price-performance is 4.2e18 FLOP per $.
  • They also find a trend in GPU price-performance of 0.1 OOM/year for state of the art GPUs. So I’ll extrapolate 4.2e18 to 5.97E+18.
• For compute price halving time to 2100, I think it depends how likely you think it is that novel technologies like optical computing will reduce compute prices in future.
  • This is the main argument Cotra puts forward for expecting such low prices.
  • It’s an argument made in XPT too, but less weight is put on it.
    • Counterarguments given in XPT: fundamental physical limits, progress getting harder, rare materials capping how much prices can drop, catastrophe/extinction, optimisation shifting to memory architectures.
      • Cotra mentions some but not all of these (she doesn’t mention rare materials or memory architectures).
  • Cotra flags that she thinks after 2040 her forecasts on this are pretty unreliable.^[3]
  • But, because of how wrong their 2024 and 2030 forecasts seem to be, I’m not inclined to put much weight on XPT forecasts here either.
  • I’ll go with the most aggressive XPT figure, which is close to Cotra’s.
    • I don’t have an inside view on the likelihood of novel technologies causing further price drops.
• Note that the disagreement about compute price halving times drives a lot of the difference in model output.

Willingness to spend

• On the most expensive training run by 2025, I think Cotra is a bit too aggressive and XPT forecasters are much too conservative.
  • In 2022, Cotra updated downwards a bit on the likelihood of a $1bn training run by 2025.
    • There isn’t much time left for Cotra to be right.
  • Cotra was predicting $20m by the end of 2020, and $80m by the end of 2021.
    • GPT-3 was $4.6m in 2020. If you buy that unreleased proprietary models are likely to be 2-8x more expensive than public ones (which Cotra argues),^[4] that XPT forecasters missed this consideration, and that GPT-3 isn’t proprietary and/or unreleased (flagging because I’m unsure what Cotra actually means by proprietary/unreleased), then this could be consistent with Cotra’s forecasts. 
  • Epoch estimates that GPT-4 cost $50m to train at some point in 2022. 
    • Again, this could be in line with Cotra’s predictions.
    • More importantly, GPT-4 costs make XPT forecasters look quite wrong already - their 2024 prediction was surpassed in 2022. This is especially striking in the context of:
      • XPT forecasters making their predictions three years after Cotra made hers.
      • The possibility that at a given time unreleased models might be 2-8x more expensive than the most expensive released model.
  • It seems to me that both Cotra and XPT forecasters will probably end up wrong, but that XPT forecasters will be much more wrong. 
    • I’ll somewhat arbitrarily pick halfway between Cotra’s conservative estimate ($300m) and her $1bn forecast for my best guess - so $650m.
• On the doubling time of spending on compute, I lean towards Cotra over XPT.
  • I feel a bit confused about how Cotra gets her 2.5 year doubling time, but I think it rests on:
    • Assuming $1bn in 2025 and the incentive of building a transformative model.
    • Getting a doubling time from 2025-2040 of 2 years, by estimating how much companies would be willing to spend on a project overall, and then ratios of overall spending to spending on compute for final training runs.
    • Then assuming that growth slows down, hitting a cap at 1% of GDP and eventually syncing up with the GDP growth rates of the largest national economy, which Cotra estimates at 3%.
  • My main reaction to this is that there are lots of assumptions here and therefore many ways that it could turn out wrong.
  • I feel disagreement on some specific points, though I’m not sure how much of Cotra’s 2.5 year doubling time is based on these things:
    • $1bn in 2025 seems too high (see above).
    • There being a transparent incentive of building a transformative model seems like an upper bound, not a median estimate to me.
    • Using Manhattan/Apollo projects as anchors for 2100 also seems like an upper bound to me:
      • It’s possible that we’d get a huge countrywide effort like this, but also possible that governments don’t get involved in this way. 
      • For a company to spend 1% of GDP you’d need loads of concentration of wealth - which might happen, or might not.
      • I agree that it seems unlikely to be higher than that - but my median guess would be lower.
  • But, because of how wrong their 2024 and 2030 forecasts seem to be, I’m inclined to basically ignore XPT forecasts here.
  • I’ll extend Cotra’s doubling time a bit, because I have a few arguments for slower doubling times, to 3 years.
  • Note that the disagreement about spending on compute doubling times drives a lot of the difference in model output (it’s the single biggest driver of difference among the inputs we have XPT forecasts for).

Algorithmic progress

• I think comparability here is low ish.
  • XPT forecasters were asked by what factor training efficiency on ImageNet classification would improve by 2024 and 2030. Extrapolating this to a halving time for the compute required for TAI which holds till 2100 seems like a bit of a stretch.
  • That said, Cotra is substantially basing her forecasts on past trends on specific benchmarks, so I still think the XPT forecasts should be viewed as somewhat indicative.
• One of the reviewers of our post on these results (Daniel Kokotajlo) pointed out that this 2022 Erdil and Besiroglu paper shows that efficiency has been improving faster than the 2020 Hernandez and Brown paper Cotra based her forecasts on suggest - halving every 9 months, as opposed to every 13-16 months. That’s a decrease in halving time by around 30%.
• Cotra further bumps up the 13-16 month estimate from Hernandez and Brown, arguing that progress on narrow tasks which can be directly optimized for is likely to be faster than TAI progress.
  • Kokotajlo offered an argument in the other direction: estimating algorithmic improvements using quantitative improvements on existing benchmarks only fails to take into account qualitative improvements where whole new capabilities emerge. 
  • Without having looked into it, I expect the magnitude of the effect from Cotra’s argument to be bigger than the effect from Kokotajlo’s argument - but it still makes me want to bump down Cotra’s forecasts a bit.
• My best guess is to bump Cotra’s forecasts down by 35%, which gives me a halving time of 1.3-2.3 years, depending on the anchor.

My best guess

* See workings here.

Where this leaves me

• I expect that sometimes the XPT forecasts will be closer to the truth, and sometimes Cotra’s will be.
  • Specifically, I expect Cotra to be more right on: 
    • FLOP per $ in 2025
    • Compute price halving time from 2025 to 2100
    • Compute cost for most expensive training run to 2025
    • Doubling time of spending on compute for the most expensive training run to 2025 (years)
  • And XPT forecasters to be more right on:
    • Halving time of compute requirements from 2025 to 2100 (years)
• The net effect of my best guess inputs is an output that’s slightly more aggressive than Cotra’s original model output. 
• I don’t think I can conclude from this ‘53% by 2050 is roughly my TAI timeline’, for a few reasons:
  • I haven’t considered all of the inputs to Cotra’s model, most notably the 2020 training computation requirements distribution. Without forming a view on that, I can’t really say that ~53% represents my overall view.
  • I feel pretty uncertain about this sort of modeling in general. It feels very sensitive to assumptions and inputs. If it were really hard to get the model to put any significant probability on TAI this century, I’d take that as an update (similarly with the model making TAI soon look very very likely). But for most middling values I’m not personally inclined to base too much on them.

1. ^{^}

   This spreadsheet uses as a template Cotra's publicly available spreadsheet, linked to from her report.

2. ^{^}

    For the relevant questions in the XPT, forecasters were asked to provide their 5th, 25th, 50th, 75th, and 95th percentile forecasts. In this analysis we use the term, ‘median’ to refer to analyses using the group’s median forecast for the 50th percentile of each question. We use the term ‘most aggressive’ to refer to analyses using the group medians for the 5th percentile estimate of the question relating to hardware costs, and the 95th percentile estimate for the questions relating to willingness to spend and algorithmic progress. (I.e., this uses the lowest plausible hardware costs and the highest plausible willingness to spend and algorithmic efficiency to give the highest plausible likelihood of TAI.) We use the term ‘most conservative’ to refer to  analyses using the group medians for the 95th percentile estimate of the question relating to hardware costs, and the 5th percentile estimate for the questions relating to willingness to spend and algorithmic progress. (I.e., this uses the highest plausible hardware costs and the lowest plausible willingness to spend and algorithmic efficiency to give the lowest plausible likelihood of TAI.) The most aggressive and most conservative estimates can be considered equivalent to 90% confidence interval for the median estimate. See here for context on which XPT questions map to which biological anchors inputs.

3. ^{^}

   “This forecast feels most solid and plausible out to ~2040 or so, beyond which it feels substantially more murky and likely incorrect.” p. 4 

   “Of all the quantitative estimates in this document, I consider these forecasts the most likely to be knowably mistaken. While most of the other quantitative estimates in this document have a lot more absolute uncertainty associated with them, there is a lot more low-hanging fruit left in improving short- and medium-term hardware price forecasts. For example, my understanding is that semiconductor industry professionals regularly write highly detailed technical reports forecasting a number of hardware cost-efficiency metrics, and I have neither read any of this literature nor interviewed any hardware experts on this question.” p. 30

4. ^{^}

   “I would guess that the most compute-intensive training run for an unreleased and/or proprietary model (e.g., a language model powering Google Assistant or Google Translate) is already ~2-8x larger than AlphaStar’s ~1.3e23, costing ~$2-8M.” p. 36 “[N]ote that there will probably be a non-trivial delay between the first time a training run of size X is completed and the first time such a training run is published, and my forecasts are about the former”. p. 37