Answering on the LW thread

This seems to complement @nostalgebraist's complaint that much of work on AI timelines (Bio Anchors, AI 2027) rely on a few load-bearing assumptions (e.g. the permanence of Moore's law, the possibility of software intelligence explosion) and then doing a lot of work crunching statistics and Fermi estimations to "predict" an AGI date, when really the end result is overdetermined by those beginning assumptions and not affected very much by changing the secondary estimations. It is thus largely a waste of time to focus on improving those estimations when there is a lot more research to be done on the actual load-bearing assumptions:

• Is Moore's law going to continue indefinitely?
• Is software intelligence explosion plausible? (If yes, does it require concentration of compute?)
• Is technical alignment easy?
• ...

Which are the actual cruxes for the most controversial AI governance questions like:

• How much should we worry about regulatory capture?
• Is it more important to reduce the rate of capabilities growth or for the US to beat China?
• Should base models be open-sourced?
• How much can friction when interacting with the real world (e.g. time needed to build factories and perform experiments (poke @titotal), regulatory red tape, labor unions, etc.) prevent AGI?
• How continuous are "short-term" AI ethics efforts (FAccT, technological unemployment, military uses) with "long-term" AI safety?
• How important is it to enhance collaboration between US, European and Chinese safety organizations?
• Should EAs work with, for, or against frontier AI labs?
• ...

I'd like to thank Sam Altman, Dario Amodei, Demis Hassabis, Yann LeCun, Elon Musk, and several others who declined to be named for giving me notes on each of the sixteen drafts of this post I shared with them over the past three months. Your feedback helped me polish a rough stone of thought into a diamond of incisive criticism. 

??? Was this meant for April's Fools Day? I'm confused.

It doesn't matter what you think they should have done, the fact is, Murati and Sutskever defected to Altman's side after initially backing his firing, almost certainly because the consensus discourse quickly became focused on EA and AI safety and not the object-level accusations of inappropriate behavior.

The "highly inappropriate behavior" is question was nearly entirely about violating safety protocols, and by the time Murati and Sutskever defected to Altman's side the conflict was clearly considered by both sides to be a referendum on EA and AI safety, to the point of the board seeking to nominate rationalist Emmett Shear as Altman's replacement.

Show me a 1966 study showing 70% of a representative sample of the general population mistake ELIZA for an human after 5 minutes of conversation.

I know this is an April's Fools joke, but EAs and AI safety people should do more thinking about how to value-align human organizations while still making them instrumentally effective (see e.g. @Scott Alexander's A Paradox of Ecclesiology, the social and intellectual movements tag).

Plenty AI safety people have tried to do work in AI, with a, let's say, mixed track record:

• Be too relaxed in organization and orthodoxy and bottom-up in control, you wind up starting the AI race in the first place because the CEO you picked turned out to be a pathological liar and plenty of your new hires more committed to him and acceleration than safety.
• Be too strict in organization and orthodoxy and top-down in control, the sole AI safety work you manage to publish is a seven-page Word document with mistaken mathematical signs and the only thing you're known for is getting linked with eight violent deaths.

... probably there should be a golden mean between the two. (EleutherAI seems to be a rare success story in this area.)

I think you're interpreting as ascendancy what is mostly just Silicon Valley realigning to the Republican Party (which is more of a return to the norm both historically and for US industrial lobbies in general). None of the Democrats you cite are exactly rising stars right now.

Wait, I heard that one before...