Thanks for the great list of resources!

Coincidentally I just discovered the Jim Rutt Show podcast recently and I've been enjoying it.

Thanks for writing this, in my opinion the field of complex systems provides a useful and under-explored perspective and set of tools for AI safety. I particularly like the insights you provide in the "Complex Systems for AI Safety" section, for example that ideas in complex systems foreshadowed inner alignment / mesa-optimisation.  

I'd be interested in your thoughts on modelling AGI governance as a complex system, for example race dynamics.

I previously wrote a forum post on how complex systems and simulation could be a useful tool in EA for improving institutional decision making, among other things: https://forum.effectivealtruism.org/posts/kWsRthSf6DCaqTaLS/what-complexity-science-and-simulation-have-to-offer  

I can think of a few other areas of direct impact which could particularly benefit from talented software engineers:

Improving climate models is a potential route for high impact on climate change, there are computational modelling initiatives such as the Climate Modeling Alliance and startups such as Cervest. It would also be valuable to contribute to open source computational tools such as the Julia programming language and certain Python libraries etc.

There is also the area of computer simulations for organisational / government decision making, such as Improbable Defence (disclosure: I am a former employee and current shareholder), Simudyne  and Hash.ai. I've heard anecdotally that a few employees of Hash.ai are sympathetic to EA, but I don't have first hand evidence of this. 

More broadly there are many areas of academic research, not just AI safety, which could benefit from more research software engineers. The Society of Research Software Engineering aims to provide a community for research engineers and to make this a more established career path. This type of work in academia tends to pay significantly lower than private sector software salaries, so this is worse for ETG, but on the flip side this is an argument for it being a relatively neglected opportunity.

Is there a list of the ideas that the fellows were working on? I'd be curious. 

It's not surprising to me that there aren't many "product focused" traditional startup style ideas in the longtermist space, but what does that leave? Are most of the potential organisations research focused? Or are there some other classes of organisation that could be founded? (Maybe this is a lack of imagination on my part!)

Very useful to know, thanks for the context!

Congratulations, this is really great to hear, and seems like a fantastic opportunity!

Out of interest, what was the sequence of events, did you already have a PhD program lined up when you applied for funding? Or are you going to apply for one now that you have the funding? Also had you already discussed this with your current employer before applying for funding?

I only ask because I have been considering attempting to do something similar!

This is a good point, although I suppose you could still think of this in the framing of "just in time learning", i.e. you can attempt a deep RL project, realise you are hopelessly out of your depth, then you know you'd better go through Spinning Up in Deep RL before you can continue. 

Although the risk is that it may be demoralising to start something which is too far outside of your comfort zone.

I massively agree with the idea of "just do a project", particularly since it's a better way of getting practice of the type of  research skills (like prioritisation and project management) that you will need to be a successful researcher.

I suppose the challenge may be choosing a topic for your project, but reaching out to others in the community may be one good avenue for harvesting project ideas. 

What are your thoughts on re-implementing existing papers? It can be a good way to develop technical skills, and maybe a middle ground between learning pre-requisites and doing your own research project? Or would you say it's better to just go for your own project?

These links are excellent! I hadn't come across these before, but I am really excited about the idea of using roleplay and table top games as a way of generating insight and getting people to think through problems. It's great to see this being applied to AI scenarios.