given Tyler Cowen's recent assessment of o3 being AGI.

Good grief. Maybe we can retrospectively say that AGI was created in the 1990s. Maybe Babbage's Analytical Engine was AGI. 

This is an absurd and discrediting statement. Sorry, Tyler Cowen, you've said some things I thought were interesting or insightful on blogs or podcasts before, and Ezra Klein respects you a lot, which goes a long way for me because I respect Ezra a lot. But this is sheer madness. 

As Andrew Connor points out in the quoted text in this post, there are simple tests of fairly rudimentary intelligence like ARC-AGI-2 that, as far as we know (the full test results are still forthcoming), o3 isn't human-level on — and it's not like ARC-AGI-2 is a test of AGI, nor will ARC-AGI-3 be. Getting human-level on ARC-AGI-2 or ARC-AGI-3 is necessary, but not sufficient, for AGI. It's a very low bar to clear. 

And that's just for starters. I recently posted a great talk by François Chollet here that points out many weaknesses in LLMs, and I can't imagine o3 overcomes all or even most of them. 

There are all kinds of tests we could come up with for AGI, such as a deliberately challenging, adversarial Turing test with no time limits and with judges who are savvy about LLMs. Or the ability to fully and autonomously replace a highly skilled knowledge worker such as a journalist, an academic research assistant, or a paralegal with only the same level of input and supervision that a human in that role would require. Can current frontier AI systems like o3 do long-term, autonomous hierarchical planning at that level of complexity? No. Absolutely not. Not even close. 

I think people should interpret Tyler Cowen making such a ridiculous statement as a signal that the AGI discourse is rotten and off the rails. This sucks. 

I used to feel so strongly about effective altruism. But my heart isn't in it anymore.

I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven't been able to sustain a vegan diet for more than a short time. And so on.

But there isn't a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.

These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:

-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI's takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk has dominated the EA Forum for a while. It feels like AGI is what the movement is mostly about now, so now I just disagree with most of what effective altruism is about.

-The extent to which LessWrong culture has taken over or "colonized" effective altruism culture is such a bummer. I know there's been at least a bit of overlap for a long time, but ten years ago it felt like effective altruism had its own, unique culture and nowadays it feels like the LessWrong culture has almost completely taken over. I have never felt good about LessWrong or "rationalism" and the more knowledge and experience of it I've gained, the more I've accumulated a sense of repugnance, horror, and anger toward that culture and ideology. I hate to see that become what effective altruism is like.

-The stories about sexual harassment are so disgusting. They're really, really bad and crazy. And it's so annoying how many comments you see on EA Forum posts about sexual harassment that make exhausting, unempathetic, arrogant, and frankly ridiculous statements, if not borderline incomprehensible in some cases. You see these stories of sexual harassment in the posts and you see evidence of the culture that enables sexual harassment in the comments. Very, very, very bad. Not my idea of a community I can wholeheartedly feel I belong to.

-Kind of a similar story with sexism, racism, and transphobia. The level of underreaction I've seen to instances of racism has been crazymaking. It's similar to the comments under the posts about sexual harassment. You see people justifying or downplaying clearly immoral behaviour. It's sickening.

-A lot of the response to the Nonlinear controversy was disheartening. It was disheartening to see how many people were eager to enable, justify, excuse, downplay, etc. bad behaviour. Sometimes aggressively, arrogantly, and rudely. It was also disillusioning to see how many people were so... easily fooled.

-Nobody talks normal in this community. At least not on this forum, in blogs, and on podcasts. I hate the LessWrong lingo. To the extent the EA Forum has its own distinct lingo, I probably hate that too. The lingo is great if you want to look smart. It's not so great if you want other people to understand what the hell you are talking about. In a few cases, it seems like it might even be deliberate obscurantism. But mostly it's just people making poor choices around communication and writing style and word choice, maybe for some good reasons, maybe for some bad reasons, but bad choices either way. I think it's rare that writing with a more normal diction wouldn't enhance people's understanding of what you're trying to say, even if you're only trying to communicate with people who are steeped in the effective altruist niche. I don't think the effective altruist sublanguage is serving good thinking or good communication.

-I see a lot of interesting conjecture elevated to the level of conventional wisdom. Someone in the EA or LessWrong or rationalist subculture writes a creative, original, evocative blog post or forum post and then it becomes a meme, and those memes end up taking on a lot of influence over the discourse. Some of these ideas are probably promising. Many of them probably contain at least a grain of truth or insight. But they become conventional wisdom without enough scrutiny. Just because an idea is "homegrown" it takes on the force of a scientific idea that's been debated and tested in peer-reviewed journals for 20 years, or a widely held precept of academic philosophy. That seems just intellectually the wrong thing to do and also weirdly self-aggrandizing.

-An attitude I could call "EA exceptionalism", where people assert that people involved in effective altruism are exceptionally smart, exceptionally wise, exceptionally good, exceptionally selfless, etc. Not just above the average or median (however you would measure that), but part of a rare elite and maybe even superior to everyone else in the world. I see no evidence this is true. (In these sorts of discussions, you also sometimes see the lame argument that effective altruism is definitionally the correct approach to life because effective altruism means doing the most good and if something isn't doing the most good, then it isn't EA. The obvious implication of this argument is that what's called "EA" might not be true EA, and maybe true EA looks nothing like "EA". So, this argument is not a defense of the self-identified "EA" movement or community or self-identified "EA" thought.)

-There is a dark undercurrent to some EA thought, along the lines of negative utilitarianism, anti-natalism, misanthropy, and pessimism. I think there is a risk of this promoting suicidal ideation because it basically is suicidal ideation.

-Too much of the discourse seems to revolve around how to control people's behaviours or beliefs. It's a bit too House of Cards. I recently read about the psychologist Kurt Lewin's study on the most effective ways to convince women to use animal organs (e.g. kidneys, livers, hearts) in their cooking during meat shortages during World War II. He found that a less paternalistic approach that showed more respect for the women's was more effective in getting them to incorporate animal organs into their cooking. The way I think about this is: you didn't have to be manipulated to get to the point where you are in believing what you believe or caring this much about this issue. So, instead of thinking of how to best manipulate people, think about how you got to the point where you are and try to let people in on that in an honest, straightforward way. Not only is this probably more effective, it's also more moral and shows more epistemic humility (you might be wrong about what you believe and that's one reason not to try to manipulate people into believing it).

-A few more things but this list is already long enough.

Put all this together and the old stuff I cared about (charity effectiveness, giving what I can, expanding my moral circle) is lost in a mess of other stuff that is antithetical to what I value and what I believe. I'm not even sure the effective altruism movement should exist anymore. The world might be better off if it closed down shop. I don't know. It could free up a lot of creativity and focus and time and resources to work on other things that might end up being better things to work on.

I still think there is value in the version of effective altruism I knew around 2015, when the primary focus was on global poverty and the secondary focus was on animal welfare, and AGI was on the margins. That version of effective altruism is so different from what exists today — which is mostly about AGI and has mostly been taken over by the rationalist subculture — that I have to consider those two different things. Maybe the old thing will find new life in some new form. I hope so.

I think you make an important point that I'm inclined to agree with. 

Most of the discourse, theories, intuitions, and thought experiments about AI alignment was formed either before the popularization of deep learning (which started circa 2012) or before the people talking and writing about AI alignment started really caring about deep learning. 

In or around 2017, I had an exchange with Eliezer Yudkowsky in an EA-related or AI-related Facebook group where he said he didn't think deep learning would lead to AGI and thought symbolic AI would instead. Clearly, at some point since then, he changed his mind. 

For example, in his 2023 TED Talk, he said he thinks deep learning is on the cusp of producing AGI. (That wasn't the first time, but it was a notable instance and an instance where he was especially clear on what he thought.)

I haven't been able to find anywhere where Eliezer talks about changing his mind or explains why he did. It would probably be helpful if he did.

All the pre-deep learning (or pre-caring about deep learning) ideas about alignment have been carried into the ChatGPT era and I've seen a little bit of discourse about this, but only a little. It seems strange that ideas about AI itself would change so much over the last 13 years and ideas about alignment would apparently change so little. 

If there are good reasons why those older ideas about alignment should still apply to deep learning-based systems, I haven't seen much discussion about that, either. You would think there would be more discussion.

My hunch is that AI alignment theory could probably benefit from starting with a fresh sheet of paper. I suspect there is promise in the approach of starting from scratch in 2025 without trying to build on or continue from older ideas and without trying to be deferential toward older work.

I suspect there would also be benefit in getting out of the EA/Alignment Forum/LessWrong/rationalist bubble. 

Yesterday, I watched this talk by François Chollet, which provides support for a few of the assertions I made in this post.

I think it’s still very relevant! I don’t think this talk’s relevance has diminished. It’s just important to also have that more recent information about o3 in addition to what’s in this talk. (That’s why I linked the other talk at the bottom of this post.)

By the way, I think it’s just o3 and not o1 that achieves the breakthrough results on ARC-AGI-1. It looks like o1 only gets 32% on ARC-AGI-1, whereas the lower-compute version of o3 gets around 76% and the higher-compute version gets around 87%.

The lower-compute version of o3 only gets 4% on ARC-AGI-2 in partial testing (full testing has not yet been done) and the higher-compute version has not yet been tested.

Chollet speculates in this blog post about how o3 works (I don’t think OpenAI has said much about this) and how that fits in to his overall thinking about LLMs and AGI:

Why does o3 score so much higher than o1? And why did o1 score so much higher than GPT-4o in the first place? I think this series of results provides invaluable data points for the ongoing pursuit of AGI.

My mental model for LLMs is that they work as a repository of vector programs. When prompted, they will fetch the program that your prompt maps to and "execute" it on the input at hand. LLMs are a way to store and operationalize millions of useful mini-programs via passive exposure to human-generated content.

This "memorize, fetch, apply" paradigm can achieve arbitrary levels of skills at arbitrary tasks given appropriate training data, but it cannot adapt to novelty or pick up new skills on the fly (which is to say that there is no fluid intelligence at play here.) This has been exemplified by the low performance of LLMs on ARC-AGI, the only benchmark specifically designed to measure adaptability to novelty – GPT-3 scored 0, GPT-4 scored near 0, GPT-4o got to 5%. Scaling up these models to the limits of what's possible wasn't getting ARC-AGI numbers anywhere near what basic brute enumeration could achieve years ago (up to 50%).

To adapt to novelty, you need two things. First, you need knowledge – a set of reusable functions or programs to draw upon. LLMs have more than enough of that. Second, you need the ability to recombine these functions into a brand new program when facing a new task – a program that models the task at hand. Program synthesis. LLMs have long lacked this feature. The o series of models fixes that.

For now, we can only speculate about the exact specifics of how o3 works. But o3's core mechanism appears to be natural language program search and execution within token space – at test time, the model searches over the space of possible Chains of Thought (CoTs) describing the steps required to solve the task, in a fashion perhaps not too dissimilar to AlphaZero-style Monte-Carlo tree search. In the case of o3, the search is presumably guided by some kind of evaluator model. To note, Demis Hassabis hinted back in a June 2023 interview that DeepMind had been researching this very idea – this line of work has been a long time coming.

So while single-generation LLMs struggle with novelty, o3 overcomes this by generating and executing its own programs, where the program itself (the CoT) becomes the artifact of knowledge recombination. Although this is not the only viable approach to test-time knowledge recombination (you could also do test-time training, or search in latent space), it represents the current state-of-the-art as per these new ARC-AGI numbers.

Effectively, o3 represents a form of deep learning-guided program search. The model does test-time search over a space of "programs" (in this case, natural language programs – the space of CoTs that describe the steps to solve the task at hand), guided by a deep learning prior (the base LLM). The reason why solving a single ARC-AGI task can end up taking up tens of millions of tokens and cost thousands of dollars is because this search process has to explore an enormous number of paths through program space – including backtracking.

There are however two significant differences between what's happening here and what I meant when I previously described "deep learning-guided program search" as the best path to get to AGI. Crucially, the programs generated by o3 are natural language instructions (to be "executed" by a LLM) rather than executable symbolic programs. This means two things. First, that they cannot make contact with reality via execution and direct evaluation on the task – instead, they must be evaluated for fitness via another model, and the evaluation, lacking such grounding, might go wrong when operating out of distribution. Second, the system cannot autonomously acquire the ability to generate and evaluate these programs (the way a system like AlphaZero can learn to play a board game on its own.) Instead, it is reliant on expert-labeled, human-generated CoT data.

It's not yet clear what the exact limitations of the new system are and how far it might scale. We'll need further testing to find out. Regardless, the current performance represents a remarkable achievement, and a clear confirmation that intuition-guided test-time search over program space is a powerful paradigm to build AI systems that can adapt to arbitrary tasks.

I’m curious what you both think of my impression that the focus on near-term AGI has completely taken over EA and sucked most of the oxygen out of the room.

I was probably one of the first 1,000 people to express an interest in organized effective altruism, back before it was called “effective altruism”. I remember being in the Giving What We Can group on Facebook when it was just a few hundred members, when they were still working on making a website. The focus then was exclusively on global poverty.

Later, when I was involved in a student EA group from around 2015 to 2017, global poverty was still front and centre, animal welfare and vegetarianism/veganism/reducetarianism was secondary, and the conversation about AI was nipping at the margins.

Fast forward to 2025 and it seems like EA is now primarily a millennialist intellectual movement focused on AGI either causing the apocalypse or creating utopia within the next 3-10 years (with many people believing it will happen within 5 years), or possibly as long as 35 years if you’re way far out on the conservative end of the spectrum.

This change has nothing to do with FTX and probably wouldn’t be a reason for anyone at Anthropic to distance themselves from EA, since Anthropic is quite boldly promoting a millennialist discourse around very near-term AGI.

But it is a reason for me not to feel an affinity with the EA movement anymore. It has fundamentally changed. It’s gone from tuberculosis to transhumanism. And that’s just not what I signed up for.

The gentle irony is that I’ve been interested in AGI, transhumanism, the Singularity, etc. for as long as I’ve been interested in effective altruism, if not a little longer. In principle, I endorse some version of many of these ideas.

But when I see the kinds of things that, for example, Dario Amodei and others at Anthropic are saying about AGI within 2 years, I feel unnerved. It feels like I’m at the boundary of the kind of ideas that it makes sense to try to argue against or rationally engage with. Because it doesn’t really feel like a real intellectual debate. It feels closer to someone experiencing some psychologically altered state, like mania or psychosis, where attempting to rationally persuade someone feels inappropriate and maybe even unkind. What do you even do in that situation?

I recently wrote here about why these super short AGI timelines make no sense to me. I read an article today that puts this into perspective. Apple is planning to eventually release a version of Siri that merges the functionality of the old, well-known version of Siri and the new soon-to-be-released version that is based on an LLM. The article says Apple originally wanted to release the merged version of Siri sooner, but now this has been delayed to 2027. Are we going to have AGI before Apple finishes upgrading Siri? These ideas don’t live in the same reality.

To put a fine point on it, I would estimate the probability of AGI being created by January 1, 2030 to be significantly less than the odds of Jill Stein winning the U.S. presidential election in 2028 as the Green Party candidate (not as the leader of either the Democratic or Republican primary), which, to be clear, I think will be roughly as likely as her winning in 2024, 2020, or 2016 was. I couldn’t find any estimates of Stein’s odds of winning either the 2028 election or past elections from prediction markets or election forecast models. At one point, electionbettingodds.com gave her 0.1%, but I don’t know if they massively rounded up or if those odds were distorted by a few long-shot bets on Stein. Regardless, I think it’s safe to say the odds of AGI being developed by January 1, 2030 are significantly less than 0.1%.

If I am correct (and I regret to inform you that I am correct), then I have to imagine the credibility of EA will diminish significantly over the next 5 years. Because, unlike FTX scamming people, belief in very near-term AGI is something that many people in EA have consciously, knowingly, deliberately signed up for. Whereas many of the warning signs about FTX were initially only known to insiders, the evidence against very near-term AGI is out in the open, meaning that deciding to base the whole movement on it now is a mistake that is foreseeable and… I’m sorry to say… obvious.

I feel conflicted saying things like this because I can see how it might come across as mean and arrogant. But I don’t think it’s necessarily unkind to try to give someone a reality check under unusual, exceptional circumstances like these.

I think EA has become dangerously insular and — despite the propaganda to the contrary — does not listen to criticism. The idea that EA has abnormal or above-average openness to criticism (compared to what? the evangelical church?) seems only to serve the function of self-licensing. That is, people make token efforts at encouraging or engaging with criticism, and then, given this demonstration of their open-mindedness, become more confident in what they already believed, and feel licensed to ignore or shut down criticism in other instances.

It also bears considering what kind of criticism or differing perspectives actually get serious attention. Listening to someone who suggests that you slightly tweak your views is, from one perspective, listening to criticism, but, from another perspective, it’s two people who already agree talking to each other in an echo chamber and patting themselves on the back for being open-minded. (Is that too mean? I’m really trying not to be mean.)

On the topic of near-term AGI, I see hand-wavey dismissal of contrary views, whether they come from sources like Turing Prize winner and FAIR Chief AI Scientist Yann LeCun, surveys of AI experts, or superforecasters. Some people predict AGI will be created very soon and seemingly a much larger number think it will take much longer. Why believe the former and not the latter? I see people being selective in this way, but I don’t see them giving principled reasons for being selective.

Crucially, AGI forecasts are a topic where intuition plays a huge role, and where intuitions are contagious. A big part of the “evidence” for near-term AGI that people explicitly base their opinion on is what person X, Y, and Z said about when they think AGI will happen. Someone somewhere came up with the image of some people sitting in a circle just saying ever-smaller numbers to each other, back and forth. What exactly would prevent that from being the dynamic?

When it comes to listening to differing perspectives on AGI, what I have seen more often than engaging with open-mindedness and curiosity is a very unfortunate, machismo/hegemonic masculinity-style impulse to degrade or humiliate a person for disagreeing. This is the far opposite of "EA loves criticism”. This is trying to inflict pain on someone you see as an opponent. This is the least intellectually healthy way of engaging in discourse, besides, I guess, I don’t know, shooting someone with a gun if they disagree with you. You might as well just explicitly forbid and censor dissent.

I would like to believe that, in 5 years, the people in EA who have disagreed with me about near-term AGI will snap out of it and send me a fruit basket. But they could also do like Elon Musk, who, after predicting fully autonomous Teslas would be available in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, and 2024, and getting it wrong 9 years in a row, now predicts fully autonomous Teslas will be available in 2025.

In principle, you could predict AGI within 5 years and just have called it a few years too soon. If you can believe in very near-term AGI today, you will probably be able to believe in very near-term AGI when 2030 rolls around, since AI capabilities will only improve.

Or they could go the Ray Kurzweil route. In 2005, Kurzweil predicted that we would have “high-resolution, full-immersion, visual-auditory virtual reality” by 2010. In 2010, when he graded his own predictions, he called this prediction “essentially correct”. This was his explanation:

The computer game industry is rapidly moving in this direction. Technologies such as Microsoft’s Kinect allows players to control a videogame without requiring controllers by detecting the player's body motions. Three-dimensional high-definition television is now available and will be used by a new generation of games that put the user in a full-immersion, high-definition, visual-auditory virtual reality environment.

Kurzweil’s gradings of his own predictions are largely like that. He finds a way to give himself a rating of “correct” or “essentially correct”. Even though he was fully incorrect. I wonder if Dario Amodei will do the same thing in 2030.

In 2030, there will be the option of doubling down on near-term AGI. Either the Elon Musk way — kick the can down the road — or the Ray Kurzweil way — revisionist history. And the best option will be some combination of both.

When people turn out to be wrong, it is not guaranteed to increase their humility or lead to soul searching. People can easily increase their defensiveness and their aggression toward people who disagree with them.

And, so, I don’t think merely being wrong will be enough on its own for EA to pull out of being a millennialist near-term AGI community. That can continue indefinitely even if AGI is over 100 years away. There is no guarantee that EA will self-correct in 5 years.

For these reasons, I don’t feel an affinity toward EA any more — it’s nothing like what it was 10 or 15 years ago — and I don’t feel much hope for it changing back, since I can imagine a scenario where it only gets worse 5 years from now.

When I was most involved in in-person EA organizing/outreach/activism around 2015 to 2017, it seemed to me at the time that the focus of the movement was something roughly like this:

Global poverty: 80%

Animal welfare: 15%

AI and existential risk: 5%

Now, organizations like 80,000 Hours and individuals like Will MacAskill are saying they’re pivoting to focusing exclusively on AGI. Judging from what I see online, it seems like the focus of the movement is currently something like:

AGI: 95%

Global poverty and animal welfare: 5%

It seems like the movement has really changed. It started as a movement around charity effectiveness in the cause area of global poverty, and now it’s a movement or community devoted to talking about AGI. This is a big change and it has probably alienated a lot of people who formerly felt an affinity for the EA movement, as well as put off people who would have been interested in what the EA movement used to be but who aren’t on board with the current movement’s beliefs about AGI.

I recently wrote about why I don’t believe in the predictions of AGI within 5 years here. For me, at this point, the EA movement has almost completely killed off its credibility. I don’t think there is any way to undo what has been done. The “effective altruism” label is now owned by people who think AGI is coming soon and as the years tick on and it becomes increasingly clear AGI is not coming soon, the term “effective altruism” will be seen by more mainstream parts of the world as even more fringe, weird, and dubious than it is today.

I’m not sure what people who don’t believe in near-term AGI and who want to focus on global poverty and/or animal welfare should do. Maybe there would be value in creating some kind of spin-off term? A term that creates a clear distinction between “effective altruism the movement about AGI” and “a movement that focuses on charity effectiveness in the cause area(s) of global poverty and/or animal welfare”. The benefit of coining and popularizing such a term would be to draw a clear distinction between what EA used to be ten years ago and what EA is now.

Using such a term wouldn’t necessarily require disavowing and distancing yourself or your organization from the EA movement or from self-identified EA organizations. Maybe some people would want to do that (I don’t know), but the primary purpose would just be to clearly differentiate your beliefs and your focus from the people who believe in a relatively imminent Singularity and who treat that as the most important thing in the world to focus on right now.

Society has long operated on a simple bargain: we contribute our labor and, in return, gain income, security, and a stake in the economy. Governments tax our wages to fund public services; corporations rely on human workers to create value; and in return, workers expect their efforts to be rewarded with opportunity and security. This centuries-old implicit bargain will soon come under strain.

Maybe this is a small detail to focus on, but I often see a problem when people try to tell a story along the lines of "society was stable and harmonious since time immemorial and then this new, disruptive, dangerous technology came along". I see a problem with this story.

Slavery wasn't abolished in the United States until 1866. Two centuries ago would be 1825. So, is anything resembling a fair bargain exchanging labour for "income, security, and a stake in the economy" really "centuries-old"?

North America and other parts of the world also have a history of indentured servitude. 

Even well into the 1900s, workers were treated in a way that was exploitative and violent. For example, Ford's management and security tried to violently suppress union activities, in at least one case killing some of the workers. 

The brief overview of the history of labour under the heading "Revisiting Our Social Contract" also doesn't mention slavery, indentured servitude, or other ugly, violent parts of this history. 

AGI or transformative AI would, in theory, cause fundamental changes to the way society organizes itself around productive labour, capital investment, government revenue, the welfare state, and so on. Yes. This possibility does raise social, political, and ethical questions. Yes. I get that when you're writing an article like this, you often just need to quickly put together a framing device to get the conversation off the ground. 

But this framing device just seemed a little too whitewashed for my taste.

Thanks.

Sorry for replying to this ancient post now. (I was looking at my old EA Forum posts after not being active on the forum for about a year.)

Here's why this answer feels unsatisfying to me. An incredibly mainstream view is to care about everyone alive today and everyone who will be born in the next 100 years. I have to imagine over 90% of people in the world would agree to that view or a view very close to that if you asked them.

That's already a reason to care about existential risks and a reason people do care about what they perceive as existential risks or global catastrophic risks. It's the reason most people who care about climate change care about climate change.

I don't really know what the best way to express the most mainstream view(s) would be. I don't think most people have tried to form a rigorous view on the ethics of far future people. (I have a hard enough time translating my own intuitions into a rigorous view, even with exposure to academic philosophy and to these sorts of ideas.) But maybe we could conjecture that most people mentally apply a "discount rate" to future lives, so that they care less and less about future lives as the centuries stretch into the future, and at some point it reaches zero.

Future lives in the distant future (i.e. people born significantly later than 100 years from now) only make an actionable different to existential risk when the estimated risk is so low that it changes the expected value math to account for 10^16 or 10^52 or whatever it is hypothetical future lives. That feels like an important insight to me, but its applicability feels limited.

So: people who don't take a longtermist view of existential risk already have a good reason to care about existential risk.

Also: people who take a longtermist view of ethics don't seem to have a good reason to think differently about any other subject than existential risk. At least, that's the impression I get from trying to engage open-mindedly and charitably with this new idea of "longtermism".

Ultimately, I'm still kind of annoyed (or at least perplexed) by "longtermism" being promoted as if it's a new idea with broad applicability, when:

1. A longtermist view of existential risk has been promoted in discussions of existential risk for a very long time. Like, decades.

2. If longtermism is actionable for anything, it's for existential risk and very little (if anything) else.

3. Most people are already bought in to caring about existential risk for relatively "neartermist" reasons.

When I heard the hype about longtermism, I was expecting there to be more meat on the bone.