Re your first point that it doesn't seem like we're making very fast progress:

People have very different views of how fast we're progressing. There's ultimately no objective measure because for any measure you could log or exponentiate it which also gives a measure of where we are but a very different picture. The evolutionary perspective suggests that if we're moving from insect to lizard abilities in a matter of years, we're going fast. From the 80k podcast with Paul Christiano:

"look at what evolution was able to do with varying amounts of compute. If you look at what each order of magnitude buys you in nature, you’re going from insects to small fish to lizards to rats to crows to primates to humans. Each of those is one order of magnitude, roughly, so you should be thinking of there are these jumps. It is the case that the different between insect and lizard feels a lot smaller to us and is less intuitive significance than the difference between primate and human or crow and primate"

Although this is usually used to argue for moderate to fast takeoff, it also favors of ignorance, i.e. at best, just don't know how fast we're progressing because this contradicts our intuitive sense.

FYI I can't confirm your observation that AI researchers don't believe in short timelines.

Thanks for this post!

I agree with the argument and only want to comment on an aside you made.

"I’m aware that some folks around CEA assign very high rates of return, in excess of 30% / year, to investment in movement-building and outreach. I think this is an epistemic error, but that would probably be a longer discussion"

This seems like a very important discussion. Could you outline your argument? If anyone else has an opinion on this please do comment! I'm mainly interested in how to decide between prioritisation research and capacity-building. These causes seem some of the most promising right now so I think we're facing this decision and it's important to get it right.