Wait what, we're (or GV is) defunding animal stuff to focus more on AI stuff? That seems really bad to me, I feel like 'PR' damage to EA is much more coming from the 'AI eschaton' side than the 'help the animals' side (and also that interventions on animal welfare are plausibly much more valuable than AI)^[1]

1. ^{^}

   e.g. here and here

I think if you subscribe to a Housing-Theory-Of-Everything or a Lars Doucet Georgist Persepctive^[1] then YIMBY stuff might be seen as an unblocker to good political-economic outcomes in everything else.

1. ^{^}

   Funny story version here

Which particular resolution criteria do you think it's unreasonable to believe will be met by 2027/2032 (depending on whether it's the weak AGI question or the strong one)?

Two of the four in particular stand out. First, the Turing Test one exactly for the reason you mention - asking the model to violate the terms of service is surely an easy way to win. That's the resolution criteria, so unless the Metaculus users think that'll be solved in 3 years^[1] then the estimates should be higher. Second, the SAT-passing requires "having less than ten SAT exams as part of the training data", which is very unlikely in current Frontier models, and labs probably aren't keen to share what exactly they have trained on.

it is just unclear whether people are forecasting on the actual resolution criteria or on their own idea of what "AGI" is. 

No reason to assume an individual Metaculus commentator agrees with the Metaculus timeline, so I don't think that's very fair.

I don't know if it is unfair. This is Metaculus! Premier forecasting website! These people should be reading the resolution criteria and judging their predictions according to them. Just going off personal vibes on how much they 'feel the AGI' feels like a sign of epistemic rot to me. I know not every Metaculus user agrees with this, but it is shaped by the aggregate - 2027/2032 are very short timelines, and those are median community predictions. This is my main issue with the Metaculus timelines atm.

I actually think the two Metaculus questions are just bad questions. 

I mean, I do agree with you in the sense that they don't fully match AGI, but that's partly because 'AGI' covers a bunch of different ideas and concepts. It might well be possible for a system to satisfy these conditions but not replace knowledge workers, perhaps a new market focusing on automation and employment might be better but that also has its issues with operationalisation.
 

1. ^{^}

   On top of everything else needed to successfully pass the imitation game

The Metaculus timeline is already highly unreasonable given the resolution criteria,^[1] and even these people think Aschenbrenner is unmoored from reality.

1. ^{^}

   Remind me to write this up soon

> Says he's stuck in bed and only going to take a stab

> Posts a thorough, thoughtful, point-by-point response to the OP in good faith

> Just titotal things

-      -     -     -      -     -     -      -     -     -      -     -     -      -     -     

On a serious note, as Richard says it seems like you agree with most of his points, at least on the 'EA values/EA-as-ideas' set of things. It sounds like atm you think that you can't recommend EA without recommending the speculative AI part of it, which I don't think has to be true.

I continue to appreciate your thoughts and contributions to the Forum and have learned a lot from them, and given the reception you get^[1] I think I'm clearly not alone there :)

1. ^{^}

   You're probably by far the highest-upvoted person who considers them EA critical here? (though maybe Habryka would also count)

Apologies for not being clear! I'll try and be a bit more clear here, but there's probably a lot of inferential distance here and we're covering some quite deep topics:

Supposing we do in fact invent AGI someday, do you think this AGI won’t be able to do science? Or that it will be able to do science, but that wouldn’t count as “automating science”?

Or maybe when you said “whether 'PASTA' is possible at all”, you meant “whether 'PASTA' is possible at all via future LLMs”?

So on the first section, I'm going for the latter and taking issue with the term 'automation', which I think speaks to mindless, automatic process of achieving some output. But if digital functionalism were true, and we successful made a digital emulation of a human who contributed to scientific research, I wouldn't call that 'automating science', instead we would have created a being that can do science. That being would be creative, agentic, with the ability to formulate it's own novel ideas and hypotheses about the world. It'd be limited by its ability to sample from the world, design experiments, practice good epistemology, wait for physical results etc. etc. It might be the case that some scientific research happens quickly, and then subsequent breakthroughs happen more slowly, etc.

My opinions on this are also highly influenced by the works of Deutsch and Popper too, who essentially argue that the growth of knowledge cannot be predicted, and since science is (in some sense) the stock of human knowledge, and since what cannot be predicted cannot be automated, scientific 'automation' is in some sense impossible.

Maybe you’re assuming that everyone here has a shared assumption that we’re just talking about LLMs...but I bet that if you were to ask a typical senior person in AI x-risk (e.g. Karnofsky) whether it’s possible that there will be some big AI paradigm shift (away from LLMs) between now and TAI, they would say “Well yeah duh of course that’s possible,” and then they would say that they would still absolutely want to talk about and prepare for TAI, in whatever algorithmic form it might take.

Agreed, AI systems are larger than LLMs, and maybe I was being a bit loose with language. On the whole though, I think much of the case by proponents for the importance of working on AI Safety does assume that current paradigm + scale is all you need, or rest on works that assume it. For instance, Davidson's Compute-Centric Framework model for OpenPhil states right in that opening page:

In this framework, AGI is developed by improving and scaling up approaches within the current ML paradigm, not by discovering new algorithmic paradigms. 

And I get off the bus with this approach immediately because I don't think that's plausible.

As I said in my original comment, I'm working on a full post on the discussion between Chollet and Dwarkesh, which will hopefully make the AGI-sceptical position I'm coming from a bit more clear. If you end up reading it, I'd be really interested in your thoughts! :)

Yeah I definitely don't mean 'brains are magic', humans are generally intelligent by any meaningful definition of the words, so we have an existence proof there that it is possible to be instantiated in some form.

I'm more sceptical of thinking science can be 'automated' though - I think progressing scientific understanding of the world is in many ways quite a creative and open-ended endeavour. It requires forming beliefs about the world, updating them due to evidence, and sometimes making radical new shifts. It's essentially the epistemological frame problem, and I think we're way off a solution there.

I think I have a big similar crux with Aschenbrenner when he says things like "automating AI research is all it takes" - like I think I disagree with that anyway but automating AI research is really, really hard! It might be 'all it takes' because that problem is already AGI complete!

Thanks JP! No worries about the documentation, I think the main features and what they correspond to on the Forum are fairly easy to interpret.

As for the redirection, I actually queried the bot site directly after seeing it you had one for that purpose, so I haven't actually tested the redirection!

Interesting on the holiday seasonality, it'd be interesting to see and I might look into it. My expectation given the top-level data is that extraneous community events are what bump engagement more, but I could be wrong.

Thanks for sharing toby, I had just finished listening to the podcast and was about to share it here but it turns out you beat me to it! I think I'll do a post going into the interview (Zvi-style)^[1] and bringing up the most interesting points and cruxes, and why the ARC Challenge matters. To quickly give my thoughts on some of the things you bring up:

• The ARC Challenge is the best benchmark out their imo, and it's telling that labs don't release their scores on it. Chollet says in the interview that they test it but because they score badly, the don't release them.
• On timelines, Chollet says that OpenAI's success led the field to 1) stop sharing Frontier research and 2) make the field focus on LLMs alone, thereby setting back timelines to AGI. I'd also suggest that the 'AGI in 2/3 years' claims don't make much sense to me unless you take an LLMs+scaling maximalist perspective.

And to respond to some other comments here:

• To huw, I think the AI Safety field is mixed. The original perspective was that ASI would be like an AIXI model, but the success of transformers have changed that. Existing models and dependents could be economically damaging, but taking away the existential risk undermines the astronomical value of AI Safety from an EA perspective.
• To OCB, I think we just disagree about how far LLMs are away from this. i think less that ARC is 'neat' and more that it shows a critical failure model in the LLM paradigm. In the interview Chollet argues that the 'scaffolding' is actually the hard part of reasoning, and I agree with him.
• To Mo, I guess Chollet's perspective would be that you need 'open-endedness' to be able to automate many/most work? A big crux I think here is whether 'PASTA' is possible at all, or at least whether it can be used as a way to bootstrap everything else. I'm more of the perspective that science is probably the last thing that can possibly be automated, but that might depend on your definition of science. 
  • I'm quite sceptical of Davidson's work, and probably Karnofsky's, but I'll need to revisit them in detail to treat them fairly. 
  • The Metaculus AGI markets are, to me, crazy low. In both cases the resolution criteria are some LLM unfriendly, it seems that people are more going off 'vibes' and not reading the fine print. Right now, for instance, any OpenAI model will be easily discovered in a proper imitation game by asking it to do something that violates the terms of service.

I'll go into more depth on my follow-up post, and I'll edit this bit of my comment wiht a link once I'm done.

1. ^{^}

   In style only, I make no claims as to quality

Precommitting to not posting more in this whole thread, but I thought Habryka's thoughts deserved a response

IMO, it seems like a bad pattern that when someone starts thinking that we are causing harm that the first thing we do is to downvote their comment

I think this is a fair cop.^[1] I appreciate the added context you've added to your comment and have removed the downvote. Reforming EA is certainly high on my list of things to write about/work on, so would appreciate your thoughts and takes here even if I suspect I'll ending up disagreeing with diagnosis/solutions.^[2]

My guess is it would be bad for evaporative cooling reasons for people like me to just leave the positions from which they could potentially fix and improve things

I guess that depends on the theory of change for improving things. If it's using your influence and standing to suggest reforms and hold people accountable, sure. If it's asking for the community to "disband and disappear", I don't know. Like, I don't know in many other movements would that be tolerated with significant influence and funding power?^[3] If one of the Lightcone Infrastructure team said "I think lightcone infrastructure in its entirety should shut down and disband, and return all funds" and then made decisions about funding and work that aligned with that goal and not yours, how long should they expect to remain part of the core team?

Maybe we're disagreeing about what we mean by the 'EA community' implicitly here, and I feel that sometimes the 'EA Community' is used as a bit of a scapegoat, but when I see takes like this I think "Why should GWWC shut down and disband because of the actions of SBF/OpenAI?" - Like I think GWWC and its members definitely count as part of the  EA Community, and your opinion seems to be pretty maximal without much room for exceptions.

(Also I think it's important to note that your own Forum use seems to have contributed to instances of evaporative cooling, so that felt a little off to me.)

I am importantly on the Long Term Future Fund, not the EA Infrastructure Fund

This is true, but LTFF is part of EA Funds, and to me is clearly EA-run/affiliated/associated. It feels like its odd that you're a grantmaker who decides where money to the community, from one of its most well-known and accessible funds, and you think that said community should disperse/disband/not grow/is net-negative for the world. That just seems rife for weird incentives/decisions unless, again, you're explicitly red-teaming grant proposals and funding decisions. If you're using it to "run interference" from the inside, to move funding away from the EA community and its causes, that feels a lot more sketchy to me.

1. ^{^}

   Never downvote while upset I guess

2. ^{^}

   I think I've noted before that there's a very large inferential difference between us, as we're two very different people

3. ^{^}

   Unless it was specifically for red-teaming