The EA community has spent a lot of time thinking about transformative AI. In particular, there is a lot of research on x-risks from transformative AI, and on how transformative AI development will unfold. However, advances in AI have many other consequences which appear crucial for guiding strategic decisionmaking in areas besides AI risk, and I haven't seen/found much material about these implications.
Here is one example of why this matters. In the upcoming decades, AI advancements will likely cause substantial changes to what the world looks like. The more the world changes, the less likely it is that research done earlier still applies to that context. The degree to which research is affected by this will depend on the type of research, but I expect the average effect to be relatively large. Therefore, we should discount the value of research in proportion to the expected loss in generalizability over time.
Another way in which AI could influence the value of research is by being able to entirely automate it. If such AI is quick enough, and able to decide what types of research should be done, then there's no role for humans to play in doing research anymore. Thus, from that point onwards, human capital ceases to be useful for research. Furthermore, such AI could redo research that was done until that point, so (to a first approximation) the impact of research done beforehand would cease when AI has these capabilities. Similarly to the previous consideration, it implies that we should discount the value of research (and career capital) over time by the probability of such development occurring.
I suspect that there are many other ways in which AI might affect our prioritization. For example, it could lower the value of poverty reduction interventions (due to accelerated growth), or increase the value of interventions that allow us to influence decisionmaking/societal values. It should also change the relative value of influencing certain key actors, based on how powerful we expect them to become as AI advances.
I'd really appreciate any thoughts on these considerations or links to relevant material!
Will -- many of these AGI side-effects seem plausible -- and almost all are alarming, with extremely high risks of catastrophe and disruption to almost every aspect of human life and civilization.
My main take-away from such thinking is that human individuals and institutions have very poor capacity to respond to AGI disruptions quickly, decisively, and intelligently enough to avoid harmful side-effects. Even if the AGI is technically 'aligned' enough not to directly cause human extinction, its downstream technological, economic, and cultural side-effects seem so dangerously unpredictable that we are very unlikely to manage them well.
Thus, AGI would be a massive X-risk amplifier in almost every other domain of human life. As I've argued many times, whatever upsides we can reap from AGI will still be there in a century, or a millennium, but whatever downsides are imposed by AGI could start hurting us within a few years. There's a huge temporal asymmetry to consider. (Maybe we can solve alignment in the next few centuries, and we'd feel reasonably safe proceeding with AGI research. But maybe not. There's every reason to take our time when we're facing a Great Filter.)
Therefore it seem... (read more)
It's very interesting to have your views on this.
Another question: Would you be worried that the impact of humanity on the world (more precisely, industrial civilization) could be net-negative if we aligned AI with human values ?
One of my fears is that if we include factory farms in the equation, humanity causes more suffering than wellbeing, simply because animals are more numerous than humans and often have horrible lives. (if we include wild animals, this gets more complicated).
So if we were to align AI with human values only, this would boost factory farming and keep it running for a long time, making the overall situation much worse.
I'm aware that cultivated meat could help solve the issue, but this seems far from automatic - many people in animal welfare don't seem so optimistic about that. It could not work out for quite a number of reasons:
https://www.forbes.com/sites/briankateman/2022/09/06/optimistic-longtermism-is-terrible-for-animals/?sh=328a115d2059
https://www.forbes.com/sites/briankateman/2022/12/07/if-we-dont-end-factory-farming-soon-it-might-be-here-forever/?sh=63fa11527e3e
So nice to see you back on the forum!
I agree with most of your comment, but I am very surprised by some points:
Does this mean that you consider plausible an improvement in productivity of ~100,000 x in a 5 year period in the next 20 years? As in, one hour of work would become more productive than 40 years of full time work 5 years earlier? That seems... (read more)
'Relatedly, laws around capital ownership. If almost all economic value is created by AI, then whoever owns the aligned AI (and hardware, data, etc) would have almost total economic power. Similarly, if all military power is held by AI, then whoever owns the AI would have almost total military power. In principle this could be a single company or a small group of people. We could try to work on legislation in advance to more widely share the increased power from aligned AI. '
I'm a bit worried that even if on paper ownership of AI is somehow spread over a large proportion of the population, people who literally control the AI could just ignore this.
On point 2, re: defense-dominant vs. offense-dominant future technologies - even if technologies are offense-dominant, the original colonists of a solar system are likely to maintain substantial control over settled solar systems, because even if they tend to lose battles over those systems, antimatter or other highly destructive weapons can render the system useless to would-be conquerors.
In general I expect interstellar conflict to look vaguely Cold War-esque in the worse cases, because the weapons are likely to be catastrophically powerful, hard to defend against (e.g. large bodies accelerated to significant fractions of lightspeed), and visible after launch, with time for retaliation (if slower than light).
Might be a good time to update Are We Living At The Most Influential Time in History?.