Here are some questions of mine. I haven't done a ton to follow discussions of AI safety, which means my questions will either be embarrassingly naive or will offer a critical outside perspective. Please don't use any that fit the former case :)

• It seems like there's a decent chance that whole brain emulations will come before de novo AI. Is there any "friendly WBE" work it makes sense to do to prepare for this case, analogous to "friendly AI" work?

• Around the time AGI comes in to existence, it's important how cheap and fast the hardware available to run it on is. If hardware is relatively expensive and slow, we can anticipate a slower (and presumably more graceful) transition. Is there anything we can do to nudge the hardware industry away from developing ever-faster chips, so hardware will be relatively expensive and slow at the time of the transition? For example, Musk could try to hire away researchers at semiconductor firms to work on batteries or rocket ships, but this could only be a temporary solution: the wages for such researchers would rise in response to the shortage, likely leading to more students going in to semiconductor research. (Hiring away professors that teach semiconductor research might be a better idea, assuming American companies are bad at training employees.)

• In this essay, I wrote: "At some point our AGI will be just as smart as the world's AI researchers, but we can hardly expect to start seeing super-fast AI progress at that point, because the world's AI researchers haven't produced super-fast AI progress." I still haven't seen a persuasive refutation of this position (though I haven't looked very hard). So: Given that human AI researchers haven't produced a FOOM, is there any reason to expect that an AI equal to the level of the human AI research community would produce a FOOM? (EDIT: A better framing might be whether or not we can expect chunky, important AI insights to be discovered in the future, or whether AGI will come out of many small cumulative developments. I suppose the idea that the brain implements a single cortical algorithm should push us in the direction of believing there is at least one chunky undiscovered insight?)

1) What careers that directly contribute to AI alignment should someone consider who is not likely suited as a researcher in fields like math or decision theory? 2) Which first steps are recommendable to end up in such a career?

Would it be valuable to develop a university level course on AI safety engineering to be implemented in hundreds of universities that use Russell's book worldwide, to attract more talented minds to the field? Which are the steps that would cause this to happen?

GiveWell's Holden Karnofsky assessed the Singularity Institute in 2012, and provided a thoughtful, extensive critique of the mission and approach which still remains tied for the top post on Lesswrong. It seems the EA Meta-charity evaluators are still hesitant to name AI Safety (and more broadly, Existential Risk Reduction) as a potentially effective target for donations. What are you doing to change that?

Assuming human-level AGI is expensive and of potential military value, it seems likely the governments of USA and probably other powers like China will be strongly involved in its development.

Is it now time to create an official process of international government-level coordination about AI safety? Is it realistic and desirable?

Do you think that the human race is more likely to be wiped out in a world with AGI or in a world without AGI? Why?

It seems to me that even the most optimistic versions of friendly super-AI are discordant with current values across society. Why isn't there more discussion about how AI development itself can be regulated, delayed and stopped? What's going on in this space? What might work?