When AI Safety people are also vegetarians, vegans or reducetarian, I am pleasantly surprised, as this is one (of many possible) signals to me they're "in it" to prevent harm, rather than because it is interesting.
I am skeptical that exhibiting one of the least cost-effective behaviors for the purpose of reducing suffering would be correlated much with being "in it to prevent harm". Donating to animal charities, or being open to trade with other people about reducing animal suffering, or other things that are actually effective seem like much better indicator that people are actually in it to prevent harm.
My best guess is that most people who are vegetarians, vegans or reducetarians, and are actually interested in scope-sensitivity, are explicitly doing so for signaling and social coordination reasons (which to be clear, I think has a bunch going for it, but seems like the kind of contingent fact that makes it not that useful as a signal of "being in it to prevent harm").
Have you actually just talked to people about their motives for reducing their animal product consumption, or read about them? Do you expect them to tell you it's mostly for signaling or social coordination, or (knowingly or unknowingly) lie that it isn't?
I'd guess only a minority of EAs go veg mostly for signaling or social coordinaton.
Other reasons I find more plausibly common, even among those who make bigger decisons in scope-sensitive ways:
Finding it cost-effective in an absolute sense, e.g. >20 chickens spared per year, at small personal cost (usually ignoring opportunity costs to help others more). (And then maybe falsely thinking you should go veg to do this, rather than just cut out certain animal products, or finding it easier to just go veg than navigate exceptions.)
Wanting to minimize the harm they personally cause to others, or to avoid participating in overall harmful practices, or deontological or virtue ethics reasons, separately from actively helping others or utilitarian-ish reasons.
Not finding not eating animal products very costly, or not being very sensitive to the costs. It may be easy or take from motivational "budgets" psychologically separate from wo
I have had >30 conversations with EA vegetarians and vegans about their reasoning here. The people who thought about it the most seem to usually settle on it for signaling reasons. Maybe this changed over the last few years in EA, but it seemed to be where most people I talked to where at when I had lots of conversations with them in 2018.
I agree that many people say (1), but when you dig into it it seems clear that people incur costs that would be better spent on donations, and so I don't think it's good reasoning. As far as I can tell most people who think about it carefully seem to stop thinking its a good reason to be vegan/vegetarian.
I do think the self-signaling and signaling effects are potentially substantial.
I also think (4) is probably the most common reason, and I do think probably captures something important, but it seems like a bad inference that "someone is in it to prevent harm" if (4) is their reason for being vegetarian or vegan.
when you dig into it it seems clear that people incur costs that would be better spent on donations, and so I don't think it's good reasoning.
I’ve thought a lot about this, because I’m serious about budgeting and try to spend as little money as possible to make more room for investments and donations. I also have a stressful job and don’t like to spend time cooking. I did not find it hard to switch to being vegan while keeping my food budget the same and maintaining a high-protein diet.
Pea protein is comparable in cost per gram protein to the cheapest animal products where I live, and it requires no cooking. Canned beans are a bit more expensive but still very cheap and also require no cooking. Grains, ramen and cereal are very cheap sources of calories. Plant milks are more expensive than cow's milk, but still fit in my very low food budget. Necessary supplements like B12 are very cheap.
On a day-to-day basis, the only real cooking I do is things like pasta, which don’t take much time at all. I often go several weeks without doing any real cooking. I’d bet that I both spend substantially less money on food and substantially less time cooking than the vast majority of omnivore... (read more)
Do you mean financial costs, or all net costs together, including potentially through time, motivation, energy, cognition? I think it's reasonably likely that for many people, there are ~no real net (opportunity) costs, or that it's actually net good (but if net good in those ways, then that would probably be a better reason than 1). Putting my thoughts in a footnote, because they're long and might miss what you have in mind.[1]
Ya, that seems fair. If they had the option to just stop thinking and feeling bad about it and chose that over going veg, which is what my framing suggests they would do, then it seems the motivation is to feel better and get more time, not avoid harming animals through their diets. This would be like seeing someone in trouble, like a homeless person, and avoiding them to avoid thinking and feeling bad about them. This can be either selfish or instrumentally other-regarding, given opportunity costs.
If they thought (or felt!) the right response to the feelings is to just go veg and not to just stop thinking and feeling bad about it, then I would say they are in it to prevent harm, just guided by their feelings. And their feelings might not be very scope-sensitive, even if they make donation and career decisions in scope-sensitive ways. I think this is kind of what virtue ethics is about. Also potentially related: "Do the math, then burn the math and go with your gut".
1. ^
Financial/donations: It's not clear to me that my diet is more expensive than if I were omnivorous. Some things I've substituted for animal products are cheaper and others are more expensive. I haven't carefully worked through this, though. It's also not clear that if it were more expensive, that I would donate more, because of how I decide how much to donate, which is based on my income and a vague sense of my costs of living, which probably won't pick up differences due to diet (but maybe it does in expectation, and maybe it means donating less later, becaus
4
Habryka
I meant net costs all together, tough I agree that if you take into account motivation "net costs" becomes a kind of tricky concept, and many people can find it motivating, and that is important to think about, but also really doesn't fit nicely into a harm-reducing framework.
I mean, being an onmivore would allow you to choose between more options. Generally having more options very rarely hurts you.
Overall I like your comment.
2
MichaelStJules
Ya, I guess the value towards harm reduction would be more indirect/instrumental in this case.
I think this is true of idealized rational agents with fixed preferences, but I'm much less sure about actual people, who are motivated in ways they wouldn't endorse upon reflection and who aren't acting optimally impartially even if they think it would be better on reflection if they did.
By going veg, you eliminate or more easily resist the motivation to eat more expensive animal products that could have net impartial opportunity costs. Maybe skipping (expensive) meats hurts you in the moment (because you want meat), but it saves you money to donate to things you think matter more. You'd be less likely to correctly — by reflection on your impartial preferences — skip the meat and save the money if you weren't veg.
And some people are not even really open to (cheaper) plant-based options like beans and tofu, and that goes away going veg. That was the case for me. My attitude before going veg would have been irrational from an impartial perspective, just considering the $ costs that could be donated instead.
Of course, some people will endorse being inherently partial to themselves upon reflection, so eating animal products might seem fine to them even at greater cost. But the people inclined to cut out animal products by comparing their personal costs to the harms to animals probably wouldn't end up endorsing their selfish motivation to eat animal products over the harms to animals.
The other side is that a veg*n is more motivated to eat the more expensive plant-based substitutes and go to vegan restaurants, which (in my experience) tend to be more expensive.
I'm not inclined to judge how things will shake out based on idealized models of agents. I really don't know either way, and it will depend on the person. Cheap veg diets seem cheaper than cheap omni diets, but if people are eating enough plant-based meats, their food costs would probably increase.
Here are pr
4
MichaelStJules
I guess this is a bit pedantic, but you originally wrote "My best guess is that most people who are vegetarians, vegans or reducetarians, and are actually interested in scope-sensitivity, are explicitly doing so for signaling and social coordination reasons". I think veg EAs are generally "actually interested in scope-sensitivity", whether or not they're thinking about their diets correctly and in scope-sensitive ~utilitarian terms. "The people who thought about it the most" might not be representative, and more representative motivations might be better described as "in it to prevent harm", even if the motivations turn out to be not utilitarian, not appropriately scope-sensitive or misguided.
1
defun
Happy to hear this from someone working on AI Safety.
Do you think EA/EA orgs should do something about it?
(Crypto has a similar problem. There are people genuinely interested in it but many people (most?) are just into it for the money)
2
yanni kyriacos
I actually think it is a huge risk because when the shit hits the fan I want to know people are optimising for the right thing. It is like the old saying; "it is only a value if it costs you something"
Anyway, I'd like to think that orgs suss this out in the interview stage.
I met Australia's Assistant Minister for Defence last Friday. I asked him to write an email to the Minister in charge of AI, asking him to establish an AI Safety Institute. He said he would. He also seemed on board with not having fully autonomous AI weaponry.
All because I sent one email asking for a meeting + had said meeting.
Advocacy might be the lowest hanging fruit in AI Safety.
Akash's Speaking to Congressional staffers about AI risk seems similar:
Like you, Akash just cold-emailed people:
There's a lot of concrete learnings in that writeup; definitely worth reading I think.
[PHOTO] I sent 19 emails to politicians, had 4 meetings, and now I get emails like this. There is SO MUCH low hanging fruit in just doing this for 30 minutes a day (I would do it but my LTFF funding does not cover this). Someone should do this!
(Speaking as someone on LTFF, but not on behalf of LTFF)
How large of a constraint is this for you? I don't have strong opinions on whether this work is better than what you're funded to do, but usually I think it's bad if LTFF funding causes people to do things that they think is less (positively) impactful!
We probably can't fund people to do things that are lobbying or lobbying-adjacent, but I'm keen to figure out or otherwise brainstorm an arrangement that works for you.
1
yanni kyriacos
Hey Linch, thanks for reaching out! Maybe send me your email or HMU here yannikyriacos@gmail.com
4
huw
(I think you blanked out their name from their email footer but not from the bottom of the email, but not sure how anonymous you wanted to keep them.)
Another thing you can do is send comments proposed legislation on regulations.gov. I did so last week about a recent californian bill on open-sourcing model weights (now closed). In the checklist (screenshot below) they say: "the comment process is not a vote – one well supported comment is often more influential than a thousand form letters". There are people much more qualified on AI risk than I over here, so in case you didn't know, you might want to keep an eye on new regulation coming up. It doesn't take much time and seems to have a fairly big impact.
Yesterday Greg Sadler and I met with the President of the Australian Association of Voice Actors. Like us, they've been lobbying for more and better AI regulation from government. I was surprised how much overlap we had in concerns and potential solutions: 1. Transparency and explainability of AI model data use (concern)
2. Importance of interpretability (solution)
3. Mis/dis information from deepfakes (concern)
4. Lack of liability for the creators of AI if any harms eventuate (concern + solution)
5. Unemployment without safety nets for Australians (concern)
6. Rate of capabilities development (concern)
They may even support the creation of an AI Safety Institute in Australia. Don't underestimate who could be allies moving forward!
My previous take on writing to Politicians got numbers, so I figured I'd post the email I send below.
I am going to make some updates, but this is the latest version:
---
Hi [Politician]
My name is Yanni Kyriacos, I live in Coogee, just down the road from your electorate.
If you're up for it, I'd like to meet to discuss the risks posed by AI. In addition to my day job building startups, I do community / movement building in the AI Safety / AI existential risk space. You can learn more about AI Safety ANZ by joining our Facebook group here or the PauseAI movement here. I am also a signatory of Australians for AI Safety - a group that has called for the Australian government to set up an AI Commission (or similar body).
Recently I worked with Australian AI experts (such as Good Ancestors Policy) in making a submission to the recent safe and response AI consultation process. In the letter, we called on the government to acknowledge the potential catastrophic and existential risks from artificial intelligence. More on that can be found here.
There are many immediate risks from already existing AI systems like ChatGPT or Midjourney, such as disinformation or improper implementation in various ... (read more)
Thanks for sharing, Yanni, and it is really cool that you managed to get Australia's Assistant Minister for Defence interested in creating an AI Safety Institute!
Did you mean to include a link?
The Metaculus' question you link to involves meeting many conditions besides passing university exams:
RIP to any posts on anything earnest over the last 48 hours. Maybe in future we don't tag anything April Fools and it is otherwise a complete blackout on serious posts 😅
I think the hilarity is in the confusion / click bait. Your idea would rob us of this! I think the best course of action is for anyone with a serious post to wait until April 3 :|
2
Toby Tremlett
Not a solution to everything mentioned here- but a reminder that you can click "customize feed" at the top of the page and remove all posts tagged april fools.
1
yanni kyriacos
nah let's lean all the way in, for one day a year, the wild west out here.
1
yanni kyriacos
Damn just had the idea of a "Who wants to be Fired?" post.
Remember: EA institutions actively push talented people into the companies making the world changing tech the public have said THEY DONT WANT. This is where the next big EA PR crisis will come from (50%). Except this time it won’t just be the tech bubble.
Is this about the safety teams at capabilities labs?
If so, I consider it a non-obvious issue, whether pushing a talented people into an AI safety role at, e.g., DeepMind is a bad thing. If you think that is a bad thing, consider providing a more detailed argument, and writing a top-level post explaining your view.
If, instead, this is about EA institutions pushing people into capabilities roles, consider naming these concrete examples. As an example, 80k has a job advertising a role as a prompt engineer at Scale AI. That does not seem to be a very safety-focused role, and it is not clear how 80k wants to help prevent human extinction with that job ad.
A judgement I'm attached to is that a person is either extremely confused or callous if they work in capabilities at a big lab. Is there some nuance I'm missing here?
Some of them have low p(doom from AI) and aren't longtermists, which justifies the high-level decision of working on an existentially dangerous technology with sufficiently large benefits.
I do think their actual specific actions are not commensurate with the level of importance or moral seriousness that they claim to attach to their work, given those stated motivations.
1
yanni kyriacos
Thanks for the comment Linch! Just to spell this out:
"Some of them have low p(doom from AI) and aren't longtermists, which justifies the high-level decision of working on an existentially dangerous technology with sufficiently large benefits. "
* I would consider this acceptable if their p(doom) was ~ < 0.01%
* I find this pretty incredulous TBH
"I do think their actual specific actions are not commensurate with the level of importance or moral seriousness that they claim to attach to their work, given those stated motivations."
* I'm a bit confused by this part. Are you saying you believe the importance / seriousness the person claims their work has is not reflected in the actions they actually take? In what way are you saying they do this?
4
Linch
I dunno man, at least some people think that AGI is the best path to curing cancer. That seems like a big deal! If you aren't a longtermist at all, speeding up the cure for cancer by a year is probably worth quite a bit of x-risk.
Yes.
Lab people shitpost, lab people take competition extremely seriously (if your actual objective is "curing cancer," it seems a bit discordant to be that worried that Silicon Valley Startup #2 will cure cancer before you), people take infosec not at all seriously, the bizarre cultish behavior after the Sam Altman firing, and so forth.
1
Dicentra
Yes. I think most people working on capabilities at leading labs are confused or callous (or something similar, like greedy or delusional), but definitely not all. And personally, I very much hope there are many safety-concerned people working on capabilities at big labs, and am concerned about the most safety-concerned people feeling the most pressure to leave, leading to evaporative cooling.
Reasons to work on capabilities at a large lab:
* To build career capital of the kind that will allow you to have a positive impact later. E.g. to be offered relevant positions in government
* To positively influence the culture of capabilities teams or leadership at labs.
* To be willing and able to whistleblow bad situations (e.g. seeing emerging dangerous capabilities in new models, the non-disparagement stuff).
* [maybe] to earn to give (especially if you don't think you're contributing to core capabilities)
To be clear, I expect achieving the above to be infeasible for most people, and it's important for people to not delude themselves into thinking they're having a positive impact to keep enjoying a lucrative, exciting job. But I definitely think there are people for whom the above is feasible and extremely important.
Another way to phrase the question is "is it good for all safety-concerned people to shun capabilities teams, given (as seems to be the case) that those teams will continue to exist and make progress by default?" And for me the strong answer is "yes". Which is totally consistent with wanting labs to pause and thinking that just contributing to capabilities (on frontier models) in expectation is extremely destructive.
1
yanni kyriacos
Thanks so much for your thoughtful comment! I appreciate someone engaging with me on this rather than just disagree ticking. Some thoughts:
1. Build Career Capital for Later Impact:
* I think this depends somewhat on what your AGI timelines are. If they're short, you've wasted your time and possibly had a net-negative effect.
* I think there is a massive risk of people entering an AGI lab for career capital building reasons, and then post rationalising their decision to stay. Action changes attitudes faster than attitude changes actions after all.
2. Influence the Culture:
* Multiple board members attempted this at OpenAI got fired, what chance does a single lower level employee?
* I have tried changing corporate culture at multiple organisations and it is somewhere between extremely hard and impossible (more on that below).
3. Be Prepared to Whistle-Blow:
* I was a Whistle-Blower during my time at News Corp. It was extremely difficult. I simply do not expect a meaningful number of people to be able to do this. You have to be willing to turn your life upside down.
* This can be terrible for a person's mental health. We shouldn't be vary careful openly recommending this as a reason to stay at a Lab.
* As mentioned above, I expect a greater number of people to turn into converts than would-be Whistle-Blowers
* I think if you're the kind of person who goes into a Lab for Career Capital that makes you less likely to be a Whistle-Blower TBH.
4. Be Prepared to Whistle-Blow:
* Sorry, you can make good money elsewhere.
1
Dicentra
(1) I agree if your timelines are super short, like <2yrs, it's probably not worth it. I have a bunch of probability mass on longer timelines, though some on really short ones
Re (2), my sense is some employees already have had some of this effect (and many don't. But some do). I think board members are terrible candidates for changing org culture; they have unrelated full-time jobs, they don't work from the office, they have different backgrounds, most people don't have cause to interact with them much. People who are full-time, work together with people all day every day, know the context, etc., seem more likely to be effective at this (and indeed, I think they have been, to some extent in some cases)
Re (3), seems like a bunch of OAI people have blown the whistle on bad behavior already, so the track record is pretty great, and I think them doing that has been super valuable. And 1 whistleblower seems much better than several converts is bad. I agree it can be terrible for mental health for some people, and people should take care of themselves.
Re (4), um, this is the EA Forum, we care about how good the money is. Besides crypto, I don't think there are many for many of the relevant people to make similar amounts of money on similar timeframes. Actually I think working at a lab early was an effective way to make money. A bunch of safety-concerned people for example have equity worth several millions to tens of millions, more than I think they could have easily earned elsewhere, and some are now billionaires on paper. And if AI has the transformative impact on the economy we expect, that could be worth way more (and it being worth more is correlated with it being needed more, so extra valuable); we are talking about the most valuable/powerful industry the world has ever known here, hard to beat that for making money. I don't think that makes it okay to lead large AI labs, but for joining early, especially doing some capabilities work that doesn't push the mos
1
Closed Limelike Curves
In my experience, they're mostly just impulsive and have already written their bottom line ("I want to work on AI projects that look cool to me"), and after that they come up with excuses to justify this to themselves.
1
yanni kyriacos
Hi! Thanks for your comment :)
I basically file this under "confused".
2
Linch
Interesting, in your ontology I'd much more straightforwardly characterize it as "callous"
1
yanni kyriacos
Sorry, I will explain further for clarity. I am Buddhist and in that philosophy the phrase "confused" has an esoteric usage. E.g. "that person suffers because they're confused about the nature of reality". I think this lab person is confused about how much suffering they're going to create for themselves and others (i.e. it hasn't been seen or internalised).
[GIF] A feature I'd love on the forum: while posts are read back to you, the part of the text that is being read is highlighted. This exists on Naturalreaders.com and would love to see it here (great for people who have wandering minds like me)
I agree with you, and so does our issue tracker. Sadly, it does seem a bit hard. Tagging @peterhartree as a person who might be able to tell me that it's less hard than I think.
4
yanni kyriacos
As someone who works with software engineers, I have respect for how simple-appearing things can actually be technically challenging.
2
Lorenzo Buonanno
For what it's worth, I would find the first part of the issue (i.e. making the player "floating" or "sticky") already quite useful, and it seems much easier to implement.
I think acting on the margins is still very underrated. For e.g. I think 5x the amount of advocacy for a Pause on capabilities development of frontier AI models would be great. I also think in 12 months time it would be fine for me to reevaluate this take and say something like 'ok that's enough Pause advocacy'.
Basically, you shouldn't feel 'locked in' to any view. And if you're starting to feel like you're part of a tribe, then that could be a bad sign you've been psychographically locked in.
I think if you work in AI Safety (or want to) it is very important to be extremely skeptical of your motivations for working in the space. This applies to being skepticism of interventions within AI Safety as well.
For example, EAs (like most people!) are motivated to do things they're (1) good at (2) see as high status (i.e. people very quietly ask themselves 'would someone who I perceive as high status approve of my belief or action?'). Based on this, I am worried that (1) many EAs find protesting AI labs (and advocating for a Pause in general) cringy and/or awkward (2) Ignore the potential impact of organisations such as PauseAI.
We might all literally die soon because of misaligned AI, so what I'm recommending is that anyone seriously considering AI Safety as a career path spends a lot of time on the question of 'what is really motivating me here?'
fwiw i think this works in both directions - people who are "action" focussed probably have a bias towards advocacy / protesting and underweight the usefulness of research.
I expect (~ 75%) that the decision to "funnel" EAs into jobs at AI labs will become a contentious community issue in the next year. I think that over time more people will think it is a bad idea. This may have PR and funding consequences too.
My understanding is that this has been a contentious issue for many years already.
80,000 hours wrote a response to this last year, Scott Alexander had written about it in 2022, and Anthropic split from OpenAI in 2021. Do you mean that you expect this to become significantly more salient in the next year?
1
yanni kyriacos
Thanks for the reply Lorenzo! IMO it is going to look VERY weird seeing people continue to leave labs while EA fills the leaky bucket.
1
yanni kyriacos
Yeah my prediction lacked specificity. I expect it to become quite heated. I'm imagining 10+ posts on the topic next year with a total of 100+ comments. That's just on the forum.
4
OllieBase
I'd probably bet against this happening FWIW. Maybe a Manifold market?
Also, 100+ comments on the forum might not mean it's necessarily "heated"—a back and forth between two commenters can quickly increase the tally on any subject so that part might also need specifying further.
1
yanni kyriacos
I spent about 30 seconds thinking about how to quantify my prediction. I'm trying to point at something vague in a concrete way but failing. This also means that I don't think it is worth my time making it more concrete. The initial post was more of the "I am pretty confident this will be a community issue, just a heads up".
I have written 7 emails to 7 Politicians aiming to meet them to discuss AI Safety, and already have 2 meetings.
Normally, I'd put this kind of post on twitter, but I'm not on twitter, so it is here instead.
I just want people to know that if they're worried about AI Safety, believe more government engagement is a good thing and can hold a decent conversation (i.e. you understand the issue and are a good verbal/written communicator), then this could be an underrated path to high impact.
Another thing that is great about it is you can choose how many emails to send and how many meetings to have. So it can be done on the side of a "day job".
The Greg / Vasco bet reminded me of something: I went to buy a ceiling fan with a light in it recently. There was one on sale that happened to also tick all my boxes, joy! But the salesperson warned me "the light in this fan can't be replaced and only has 10,000 hours in it. After that you'll need a new fan. So you might not want to buy this one." I chuckled internally and bought two of them, one for each room.
So I did a quick check today - I've sent 19 emails to politicians about AI Safety / x-risk and received 4 meetings. They've all had a really good vibe, and I've managed to get each of them to commit to something small (e.g. email XYZ person about setting up an AI Safety Institute). I'm pretty happy with the hit rate (4/19). I might do another forum quick take once I've sent 50.
Two jobs in AI Safety Advocacy that AFAICT don't exist, but should and probably will very soon. Will EAs be the first to create them though? There is a strong first mover advantage waiting for someone -
1. Volunteer Coordinator - there will soon be a groundswell from the general population wanting to have a positive impact in AI. Most won't know how to. A volunteer manager will help capture and direct their efforts positively, for example, by having them write emails to politicians
2. Partnerships Manager - the President of the Voice Actors guild reached out... (read more)
The general public wants frontier AI models regulated and there doesn't seem to be grassroots focussed orgs attempting to capture and funnel this energy into influencing politicians. E.g. via this kind of activity. This seems like massively low hanging fruit. An example of an organisation that does this (but for GH&W) is Results Australia. Someone should set up such an org.
My impression is that PauseAI focusses more on media engagement + protests, which I consider a good but separate thing. Results Australia, as an example, focuses (almost exclusively) on having concerned citizens interacting directly with politicians. Maybe it would be a good thing for orgs to focus on separate things (e.g. for reasons of perception + specialisation). I lean in this direction but remain pretty unsure.
5
joepio
Founder of PauseAI here. I know our protests are the most visible, but they are actually a small portion of what we do. People always talk about the protests, but I think we actually had most results through invisible volunteer lobbying. Personally, I've spent way more time sending emails to politicians and journalists, meeting them and informing them of the issues. I wrote an Email Builder to get volunteers to write to their representatives, gave multiple workshops on doing so, and have seen many people (including you of course!) take action and ask for feedback in the discord.
I think combining both protesting and volunteer lobbying in one org is very powerful. It's not a new idea of course - orgs like GreenPeace have been using this strategy for decades. The protests create visibility and attention, and the lobbying gives the important people the right background information. The protests encourage more volunteers to join and help out, so we get more volunteer lobbyists! In my experience the protests also help with getting an in with politicians - it creates a recognizable brand that shows you represent a larger group.
It breaks my heart when I see eulogy posts on the forum. And while I greatly appreciate people going to the effort of writing them (while presumably experiencing grief), it still doesn't feel like enough. We're talking about people that dedicated their lives to doing good, and all they get is a post. I don't have a suggestion to address this 'problem', and some may even feel that a post is enough, but I don't. Maybe there is no good answer and death just sucks. I dunno.
This is an extremely "EA" request from me but I feel like we need a word for people (i.e. me) who are Vegans but will eat animal products if they're about to be thrown out. OpportuVegan? UtilaVegan?
I think the term I've heard (from non-EAs) is 'freegan' (they'll eat it if it didn't cause more animal products to be purchased!)
1
yanni kyriacos
This seems close enough that I might co-opt it :)
https://en.wikipedia.org/wiki/Freeganism
3
Toby Tremlett
If you predictably do this, you raise the odds that people around you will cook some/ buy some extra food so that it will be "thrown out", or offer you food they haven't quite finished (and that they'll replace with a snack later.
So I'd recommend going with "Vegan" as your label, for practical as well as signalling reasons.
3
yanni kyriacos
Yeah this is a good point, which I've considered, which is why I basically only do it at home.
Has anyone seen an analysis that takes seriously the idea that people should eat some fruits, vegetables and legumes over others based on how much animal suffering they each cause?
I.e. don't eat X fruit, eat Y one instead, because X fruit is [e.g.] harvested in Z way, which kills more [insert plausibly sentient creature].
"Pens Down" to mean 'Artificial Super Intelligence in my opinion is close enough that it no longer makes sense to work on whatever else I'm currently working on, because we're about to undergo radical amounts of change very soon/quickly'.
For me, it is probably when we have something as powerful as GPT-4 except it is agentic and costs less than $100 / month. So, that looks like a digital personal assistant that can execute an instruction like "have a TV delivered for me by X date, under Y price and organise installation and wall mounting."
This is obviously a question mainly for people who don't work full time on AI Safety.
I don't know if this can be answered in full-generality.
I suppose it comes down to things like:
• Financial runway/back-up plans in case your prediction is wrong
• Importance of what you're doing now
• Potential for impact in AI safety
1
yanni kyriacos
I agree. I think it could be a useful exercise though to make the whole thing (ASI) less abstract.
I find it hard to reconcile that (1) I think we're going to have AGI soon and (2) I haven't made more significant life changes.
I don't buy the argument that much shouldn't change (at least, in my life).
2
Chris Leong
Happy to talk that through if you'd like, though I'm kind of biased, so probably better to speak to someone who doesn't have a horse in the race.
1
yanni kyriacos
I suppose it is plausible for a person to never have a "Pens Down" moment if;
1. There is a FOOM
2. They feel they won't be able to positively contribute to making ASI safe / slowing it down
1
yanni kyriacos
I'm somewhat worried we're only 2-3 years from this, FWIW. I'd give it a ~ 25% chance.
I think the average EA worries too much about negative PR related to EA. I think this is a shame because EA didn't get to where it is now because it concerned itself with PR. It got here through good-old-fashioned hard work (and good thinking ofc).
Two examples:
1. FTX.
2. OpenAI board drama.
On the whole, I think there was way too much time spent thinking and talking about these massive outliers and it would have been better if 95%+ of EAs put their head down and did what they do best - get back to work.
I think it is good to discuss and take action... (read more)
Thanks Yanni, I think a lot of people have been concerned about this kind of thing.
I would be surprised if 80,000 hours isn't already tracking this or something like it - perhaps try reaching out to them directly, you might get a better response that way
My long thoughts:
1. 80k don't claim to only advertise impactful jobs
They also advertise jobs that help build career impact, and they're not against posting jobs that cause harm (and it's often/always not clear which is which). See more in this post.
They sometimes add features like marking "recommended orgs" (which I endorse!), and sometimes remove those features ( 😿 ).
2. 80k's career guide about working at AI labs doesn't dive into "which lab"
See here. Relevant text:
I think [link to comment] the "which lab" question is really important, and I'd encourage 80k to either be opinionated about it, or at least help people make up their mind somehow, not just leave people hanging on "which lab" while also often recommending people go work at AI labs, and also mentioning that often that work is net-negative and recommending reducing the risk by not working "in certain positions unless you feel awesome about the lab".
[I have longer thoughts on how they could do this, but my main point is that it's (imo) an important hole in their recommendation that might be hidden from many readers]
3. Counterfactual / With great power comes great responsibility
If 80k wouldn't do all this, should we assume there would be no job board and no guides?
I claim that something like a job board has critical mass: Candidates know the best orgs are there, and orgs know the best candidates are there.
Once there's a job board with critical mass, it's not trivial to "compete" with it.
But EAs love opening job boards. A few new EA job boards pop up every year. I do think there would be an alternative. And so the question seems to me to be - how well are 80k using their critical mass?
4. What results did 80k's work actually cause?
First of all: I don't actually know, and if someone from 80k would respond, that would be way better than my guess.
Still, here's my guess, which I think would be better than just responding to the poll:
* Lots of engineers who care about AI Safet
3
Guy Raveh
And there's always the other option that I (unpopularly) believe in - that better publicly available AI capabilities are necessary for meaningful safety research, thus AI labs have contributed positively to the field.
Even though I've been in the AI Safety space for ~ 2 years, I can't shake the feeling that every living thing dying painlessly in its sleep overnight (due to AI killing us) isn't as bad as (i.e. is 'better' than) hundreds of millions of people living in poverty and/or hundreds of billions of animals being tortured.
This makes me suspicious of my motivations. I think I do the work partly because I kinda feel the loss of future generations, but mainly because AI Safety still feels so neglected (and my counter factual impact here is larger).
Thanks for sharing. I suspect most of the hundreds of millions of people living in poverty would disagree with you, though, and would prefer not to painlessly die in their sleep tonight.
I think its possible we're talking passed each other?
5
NickLaing
I don't think he's talking past you. His point seems that the vast majority of the hundreds of millions of people living in poverty both have net positive lives, and don't want to die.
Even with a purely hedonistic outlook, it wouldn't be better for their lives to end.
Unless you are not talking about the present, but a future far worse than today's situation?
1
yanni kyriacos
I'm saying that on some level it feels worse to me that 700 million people suffer in poverty than every single person dying painlessly in their sleep. Or that billions of animals are in torture factories. It sounds like I'm misunderstanding Jason's point?
2
NickLaing
I would contend they are not "suffering" in poverty overall, because most of their lives are net positive. There may be many struggles and their lives are a lot harder than ours, but still better than not being alive at all.
I agree with you on the animals in torture factories, because their lives are probably net negative unlike the 700 million in poverty.
3
titotal
If AI actually does manage to kill us (which I doubt), It will not involve everybody dying painlessly in their sleep. That is an assumption of the "FOOM to god AI with no warning" model, which bears no resemblance to reality.
The technology to kill everyone on earth in their sleep instantaneously does not exist now, and will not exist in the near-future, even if AGI is invented. Killing everyone in their sleep is orders of magnitude more difficult than killing everyone awake, so why on earth would that be the default scenario?
2
Stephen Clare
I think you have a point with animals, but I don't think the balance of human experience means that non-existence would be better than the status quo.
Will talks about this quite a lot in ch. 9 of WWOTF ("Will the future be good or bad?"). He writes:
And, of course, for people at least, things are getting better over time. I think animal suffering complicates this a lot.
I think https://www.wakingup.com/ should be considered for effective organisation status. It donates 10% of revenue to the most effective causes and I think reaching nondual states of awakening could be one of the most effective ways for people in rich countries to improve their wellbeing.
Also related (though more tangentially): https://podcast.clearerthinking.org/episode/167/michael-taft-and-jeremy-stevenson-glimpses-of-enlightenment-through-nondual-meditation/
The catchphrase I walk around with in my head regarding the optimal strategy for AI Safety is something like: Creating Superintelligent Artificial Agents* (SAA) without a worldwide referendum is ethically unjustifiable. Until a consensus is reached on whether to bring into existence such technology, a global moratorium is required (*we already have AGI).
I have heard rumours that an AI Safety documentary is being made. Separate to this, a good friend of mine is also seriously considering making one, but he isn't "in" AI Safety. If you know who this first group is and can put me in touch with them, it might be worth getting across each others plans.
Something I'm confused about: what is the threshold that needs meeting for the majority of people in the EA community to say something like "it would be better if EAs didn't work at OpenAI"?
Imagining the following hypothetical scenarios over 2024/25, I can't predict confidently whether they'd individually cause that response within EA?
Ten-fifteen more OpenAI staff quit for varied and unclear reasons. No public info is gained outside of rumours
There is another board shakeup because senior leaders seem worried about Altman. Altman stays on
What are some historical examples of a group (like AI Safety folk) getting something incredibly wrong about an incoming Technology? Bonus question: what led to that group getting it so wrong? Maybe there is something to learn here.
In the 90's and 2000's, many people such as Eric Drexler were extremely worried about nanotechnology and viewed it as an existential threat through the "gray goo" scenario. Yudkowsky predicted drexler style nanotech would occur by 2010, using very similar language to what he is currently saying about AGI.
It turned out they were all being absurdly overoptimistic about how soon the technology would arrive, and the whole drexlerite nanotech project flamed out by the end of the 2000's and has pretty much not progressed since. I think a similar dynamic playing out with AGI is less likely, but still very plausible.
Do you have links to people being very worried about gray goo stuff?
(Also, the post you link to makes this clear, but this was a prediction from when Eliezer was a teenager, or just turned 20, which does not make for a particularly good comparison, IMO)
1
yanni kyriacos
I hope you're right. Thanks for the example, it seems like a good one.
9
Ives Parr
This is probably a good exercise. I do want to point out a common bias about getting existential risks wrong. If someone was right about doomsday, we would not be here to discuss it. That is a huge survivorship bias. Even catestrophic events which lessen the number of people are going to be systemically underestimated. This phenomenon is the anthropic shadow which is relevant to an analysis like this.
1
yanni kyriacos
Yeah, Case Studies as Research need to be treated very carefully (i.e. they can still be valuable exercises but the analyser needs to be aware of their weaknesses)
5
saulius
There were many predictions about AI and AGI in the past (maybe mostly last century) that were very wrong. I think I read about it in Superintelligence. A quick Google search shows this article which probably talks about that.
0
yanni kyriacos
Thanks!
5
saulius
Cultured meat predictions were overly optimistic, although many of those predictions might have been companies hyping up their products to attract investors. There's also probably a selection bias where the biggest cultured meat optimisits are the ones who become cultured meat experts and make predictions
I recently discovered the idea of driving all blames into oneself, which immediately resonated with me. It is relatively hardcore; the kind of thing that would turn David Goggins into a Buddhist.
Gemini did a good job of summarising it:
This quote by Pema Chödron, a renowned Buddhist teacher, represents a core principle in some Buddhist traditions, particularly within Tibetan Buddhism. It's called "taking full responsibility" or "taking self-blame" and can be a bit challenging to understand at first. Here's a breakdown:
What would be the pros and cons of adding a semi-hidden-but-permanent Hot Takes section to the Forum? All of my takes are Hot and due to time constraints I would otherwise not post at all. Some would argue that someone like me should not post Hot Takes at all. Anyway, in true lazy fashion here is ChatGPT on the pros and cons:
Pros:
Encourages diverse perspectives and stimulates debate.
Can attract more engagement and interest from users.
Provides a platform for expressing unconventional or controversial ideas.
Fosters a culture of intellectual curiosity and ope
This feels like it could just be a genre of Quick Takes that people may choose to post?
2
NickLaing
That's an interesting one - I'm a fan of hot takes myself :D. I think "Quick takes" does the job on these though, even if the posts are a bit longer. I'm not sure we need another section. Maybe a "Hot takes" tab could be added to signify that the thought behind a take isn't so deep?
One of the seminal texts in marketing science is The Long and The Short of it by Field and Binet. They argue that for maximum effectiveness, marketing should aim to create two distinct work streams and results; immediate sales and longer term sales.
They argue the tactics that go into either are distinct (e.g. short term = create a sense of urgency, long term = match brand associations with category associations).
This feels like a good analogy for AI Safety Advocacy / Governance - keep talking about short term things people can buy now (in an Au... (read more)
Feels kinda obnoxious to write a quick take along the lines of "I'm thinking about writing a post on X, does anyone actually give a sh*t? Otherwise I won't write it."
I just wanted to check, since I can't place my finger on why it feels obnoxious but it certainly does.
How impactful could non-dual meditation be for improving wellbeing?
Are there any EAs out there who have practiced non-dual meditation? Specifically, anything from the Dzogchen or Mahamudra traditions of Tibetan Buddhism?
More secular teachers would be Loch Kelly, Sam Harris, Michael Taft.
This has been a life changing experience for me and I'm wondering whether it could be a blind spot for EA.
I'd also just love to chat with someone else who has experienced non-duality / awakening through this form of meditation :)
Prediction: In 6-12 months people are going to start leaving Deepmind and Anthropic for similar sounding reasons to those currently leaving OpenAI (50% likelihood).
> Surface level read of what is happening at OpenAI; employees are uncomfortable with specific safety policies. > Deeper, more transferable, harder to solve problem; no person that is sufficiently well-meaning and close enough to the coal face at Big Labs can ever be reassured they're doing the right thing continuing to work for a company whose mission is to build AGI.
Basically, this is less about "OpenAI is bad" and more "Making AGI is bad".
Very quick thoughts on setting time aside for strategy, planning and implementation, since I'm into my 4th week of strategy development and experiencing intrusive thoughts about needing to hurry up on implementation;
I have a 52 week LTFF grant to do movement building in Australia (AI Safety)
I have set aside 4.5 weeks for research (interviews + landscape review + maybe survey) and strategy development (segmentation, targeting, positioning),
Then 1.5 weeks for planning (content, events, educational programs), during which I will get feedback from others on th
What would stop you from paying for an LLM? Take an extreme case; Sam Altman turns around tomorrow and says "We're racing to AGI, I'm not going to worry about Safety at all."
Would that stop you from throwing him $20 a month?
Help clear something up for me: I am extremely confused (theoretically) how we can simultaneously have:
1. An Artificial Superintelligence
2. It be controlled by humans (therefore creating misuse of concentration of power issues)
My intuition is that once it reaches a particular level of power it will be uncontrollable. Unless people are saying that we can have models 100x more powerful than GPT4 without it having any agency??
I don't think we need to explicitly alert the reader when we've received help from an LLM to write something (even if it wrote the bulk of the post). That's it, my quickest ever Quick Take.
There have been multiple occasions where I've copy and pasted email threads into an LLM and asked it things like:
What is X person saying
What are the cruxes in this conversation?
Summarise this conversation
What are the key takeaways
What views are being missed from this conversation
I really want an email plugin that basically brute forces rationality INTO email conversations.
Tangentially - I wonder if LLMs can reliably convert peoples claims into a % through sentiment analysis? This would be useful for Forecasters I believe (and rationality in general)
How bad does factory farming need to get before we WANT TO accelerate AI capabilities?
This article suggests that the DAILY "total painfulness of fish suffering is equivalent to the painfulness of one hundred fifty million human deaths" and that most of these deaths are via "slow suffocation, crushing, or live disemboweling".
Let's assume the creation of AGI had a >80% probability of leading to two outcomes for non-human animals: extinction or liberation.
At some point over the next 5 years virtually every person reading this will become a Pause advocate, the only question is whether that occurs now, in 2025, 2026, 2027, 2028 or 2029.
When AI Safety people are also vegetarians, vegans or reducetarian, I am pleasantly surprised, as this is one (of many possible) signals to me they're "in it" to prevent harm, rather than because it is interesting.
I am skeptical that exhibiting one of the least cost-effective behaviors for the purpose of reducing suffering would be correlated much with being "in it to prevent harm". Donating to animal charities, or being open to trade with other people about reducing animal suffering, or other things that are actually effective seem like much better indicator that people are actually in it to prevent harm.
My best guess is that most people who are vegetarians, vegans or reducetarians, and are actually interested in scope-sensitivity, are explicitly doing so for signaling and social coordination reasons (which to be clear, I think has a bunch going for it, but seems like the kind of contingent fact that makes it not that useful as a signal of "being in it to prevent harm").
Have you actually just talked to people about their motives for reducing their animal product consumption, or read about them? Do you expect them to tell you it's mostly for signaling or social coordination, or (knowingly or unknowingly) lie that it isn't?
I'd guess only a minority of EAs go veg mostly for signaling or social coordinaton.
Other reasons I find more plausibly common, even among those who make bigger decisons in scope-sensitive ways:
- Finding it cost-effective in an absolute sense, e.g. >20 chickens spared per year, at small personal cost (usually ignoring opportunity costs to help others more). (And then maybe falsely thinking you should go veg to do this, rather than just cut out certain animal products, or finding it easier to just go veg than navigate exceptions.)
- Wanting to minimize the harm they personally cause to others, or to avoid participating in overall harmful practices, or deontological or virtue ethics reasons, separately from actively helping others or utilitarian-ish reasons.
- Not finding not eating animal products very costly, or not being very sensitive to the costs. It may be easy or take from motivational "budgets" psychologically separate from wo
... (read more)I’ve thought a lot about this, because I’m serious about budgeting and try to spend as little money as possible to make more room for investments and donations. I also have a stressful job and don’t like to spend time cooking. I did not find it hard to switch to being vegan while keeping my food budget the same and maintaining a high-protein diet.
Pea protein is comparable in cost per gram protein to the cheapest animal products where I live, and it requires no cooking. Canned beans are a bit more expensive but still very cheap and also require no cooking. Grains, ramen and cereal are very cheap sources of calories. Plant milks are more expensive than cow's milk, but still fit in my very low food budget. Necessary supplements like B12 are very cheap.
On a day-to-day basis, the only real cooking I do is things like pasta, which don’t take much time at all. I often go several weeks without doing any real cooking. I’d bet that I both spend substantially less money on food and substantially less time cooking than the vast majority of omnivore... (read more)
I met Australia's Assistant Minister for Defence last Friday. I asked him to write an email to the Minister in charge of AI, asking him to establish an AI Safety Institute. He said he would. He also seemed on board with not having fully autonomous AI weaponry.
All because I sent one email asking for a meeting + had said meeting.
Advocacy might be the lowest hanging fruit in AI Safety.
[PHOTO] I sent 19 emails to politicians, had 4 meetings, and now I get emails like this. There is SO MUCH low hanging fruit in just doing this for 30 minutes a day (I would do it but my LTFF funding does not cover this). Someone should do this!
You should probably also blank their job title (which would make it easy to work out who they are) and their phone number (!)
A periodic reminder that you can just email politicians and then meet them (see screenshot below).
Yesterday Greg Sadler and I met with the President of the Australian Association of Voice Actors. Like us, they've been lobbying for more and better AI regulation from government. I was surprised how much overlap we had in concerns and potential solutions:
1. Transparency and explainability of AI model data use (concern)
2. Importance of interpretability (solution)
3. Mis/dis information from deepfakes (concern)
4. Lack of liability for the creators of AI if any harms eventuate (concern + solution)
5. Unemployment without safety nets for Australians (concern)
6. Rate of capabilities development (concern)
They may even support the creation of an AI Safety Institute in Australia. Don't underestimate who could be allies moving forward!
My previous take on writing to Politicians got numbers, so I figured I'd post the email I send below.
I am going to make some updates, but this is the latest version:
---
Hi [Politician]
My name is Yanni Kyriacos, I live in Coogee, just down the road from your electorate.
If you're up for it, I'd like to meet to discuss the risks posed by AI. In addition to my day job building startups, I do community / movement building in the AI Safety / AI existential risk space. You can learn more about AI Safety ANZ by joining our Facebook group here or the PauseAI movement here. I am also a signatory of Australians for AI Safety - a group that has called for the Australian government to set up an AI Commission (or similar body).
Recently I worked with Australian AI experts (such as Good Ancestors Policy) in making a submission to the recent safe and response AI consultation process. In the letter, we called on the government to acknowledge the potential catastrophic and existential risks from artificial intelligence. More on that can be found here.
There are many immediate risks from already existing AI systems like ChatGPT or Midjourney, such as disinformation or improper implementation in various ... (read more)
RIP to any posts on anything earnest over the last 48 hours. Maybe in future we don't tag anything April Fools and it is otherwise a complete blackout on serious posts 😅
Something(!) needs to be done. Otherwise, it's just a mess for clarity and the communication of ideas.
Remember: EA institutions actively push talented people into the companies making the world changing tech the public have said THEY DONT WANT. This is where the next big EA PR crisis will come from (50%). Except this time it won’t just be the tech bubble.
A judgement I'm attached to is that a person is either extremely confused or callous if they work in capabilities at a big lab. Is there some nuance I'm missing here?
[GIF] A feature I'd love on the forum: while posts are read back to you, the part of the text that is being read is highlighted. This exists on Naturalreaders.com and would love to see it here (great for people who have wandering minds like me)
![]()
I think acting on the margins is still very underrated. For e.g. I think 5x the amount of advocacy for a Pause on capabilities development of frontier AI models would be great. I also think in 12 months time it would be fine for me to reevaluate this take and say something like 'ok that's enough Pause advocacy'.
Basically, you shouldn't feel 'locked in' to any view. And if you're starting to feel like you're part of a tribe, then that could be a bad sign you've been psychographically locked in.
I think if you work in AI Safety (or want to) it is very important to be extremely skeptical of your motivations for working in the space. This applies to being skepticism of interventions within AI Safety as well.
For example, EAs (like most people!) are motivated to do things they're (1) good at (2) see as high status (i.e. people very quietly ask themselves 'would someone who I perceive as high status approve of my belief or action?'). Based on this, I am worried that (1) many EAs find protesting AI labs (and advocating for a Pause in general) cringy and/or awkward (2) Ignore the potential impact of organisations such as PauseAI.
We might all literally die soon because of misaligned AI, so what I'm recommending is that anyone seriously considering AI Safety as a career path spends a lot of time on the question of 'what is really motivating me here?'
I expect (~ 75%) that the decision to "funnel" EAs into jobs at AI labs will become a contentious community issue in the next year. I think that over time more people will think it is a bad idea. This may have PR and funding consequences too.
I have written 7 emails to 7 Politicians aiming to meet them to discuss AI Safety, and already have 2 meetings.
Normally, I'd put this kind of post on twitter, but I'm not on twitter, so it is here instead.
I just want people to know that if they're worried about AI Safety, believe more government engagement is a good thing and can hold a decent conversation (i.e. you understand the issue and are a good verbal/written communicator), then this could be an underrated path to high impact.
Another thing that is great about it is you can choose how many emails to send and how many meetings to have. So it can be done on the side of a "day job".
The Greg / Vasco bet reminded me of something: I went to buy a ceiling fan with a light in it recently. There was one on sale that happened to also tick all my boxes, joy! But the salesperson warned me "the light in this fan can't be replaced and only has 10,000 hours in it. After that you'll need a new fan. So you might not want to buy this one." I chuckled internally and bought two of them, one for each room.
Be the meme you want to see in the world (screenshot).
So I did a quick check today - I've sent 19 emails to politicians about AI Safety / x-risk and received 4 meetings. They've all had a really good vibe, and I've managed to get each of them to commit to something small (e.g. email XYZ person about setting up an AI Safety Institute). I'm pretty happy with the hit rate (4/19). I might do another forum quick take once I've sent 50.
Two jobs in AI Safety Advocacy that AFAICT don't exist, but should and probably will very soon. Will EAs be the first to create them though? There is a strong first mover advantage waiting for someone -
1. Volunteer Coordinator - there will soon be a groundswell from the general population wanting to have a positive impact in AI. Most won't know how to. A volunteer manager will help capture and direct their efforts positively, for example, by having them write emails to politicians
2. Partnerships Manager - the President of the Voice Actors guild reached out... (read more)
The general public wants frontier AI models regulated and there doesn't seem to be grassroots focussed orgs attempting to capture and funnel this energy into influencing politicians. E.g. via this kind of activity. This seems like massively low hanging fruit. An example of an organisation that does this (but for GH&W) is Results Australia. Someone should set up such an org.
It breaks my heart when I see eulogy posts on the forum. And while I greatly appreciate people going to the effort of writing them (while presumably experiencing grief), it still doesn't feel like enough. We're talking about people that dedicated their lives to doing good, and all they get is a post. I don't have a suggestion to address this 'problem', and some may even feel that a post is enough, but I don't. Maybe there is no good answer and death just sucks. I dunno.
This is an extremely "EA" request from me but I feel like we need a word for people (i.e. me) who are Vegans but will eat animal products if they're about to be thrown out. OpportuVegan? UtilaVegan?
Has anyone seen an analysis that takes seriously the idea that people should eat some fruits, vegetables and legumes over others based on how much animal suffering they each cause?
I.e. don't eat X fruit, eat Y one instead, because X fruit is [e.g.] harvested in Z way, which kills more [insert plausibly sentient creature].
What is your "Pens Down" moment?
"Pens Down" to mean 'Artificial Super Intelligence in my opinion is close enough that it no longer makes sense to work on whatever else I'm currently working on, because we're about to undergo radical amounts of change very soon/quickly'.
For me, it is probably when we have something as powerful as GPT-4 except it is agentic and costs less than $100 / month. So, that looks like a digital personal assistant that can execute an instruction like "have a TV delivered for me by X date, under Y price and organise installation and wall mounting."
This is obviously a question mainly for people who don't work full time on AI Safety.
I think the average EA worries too much about negative PR related to EA. I think this is a shame because EA didn't get to where it is now because it concerned itself with PR. It got here through good-old-fashioned hard work (and good thinking ofc).
Two examples:
1. FTX.
2. OpenAI board drama.
On the whole, I think there was way too much time spent thinking and talking about these massive outliers and it would have been better if 95%+ of EAs put their head down and did what they do best - get back to work.
I think it is good to discuss and take action... (read more)
A potential failure mode of 80k recommending EAs work at AI labs:
I think this is sufficiently important / likely that 80k should consider tracking these people over time to see if this is a real issue.
What is your best guess of the overall impact of 80k interviewing AI labs on their podcast + listing AI lab vacant roles?
Poll: https://www.guidedtrack.com/programs/qj0ykwn/run
Even though I've been in the AI Safety space for ~ 2 years, I can't shake the feeling that every living thing dying painlessly in its sleep overnight (due to AI killing us) isn't as bad as (i.e. is 'better' than) hundreds of millions of people living in poverty and/or hundreds of billions of animals being tortured.
This makes me suspicious of my motivations. I think I do the work partly because I kinda feel the loss of future generations, but mainly because AI Safety still feels so neglected (and my counter factual impact here is larger).
I don't think... (read more)
Thanks for sharing. I suspect most of the hundreds of millions of people living in poverty would disagree with you, though, and would prefer not to painlessly die in their sleep tonight.
I think https://www.wakingup.com/ should be considered for effective organisation status. It donates 10% of revenue to the most effective causes and I think reaching nondual states of awakening could be one of the most effective ways for people in rich countries to improve their wellbeing.
The catchphrase I walk around with in my head regarding the optimal strategy for AI Safety is something like: Creating Superintelligent Artificial Agents* (SAA) without a worldwide referendum is ethically unjustifiable. Until a consensus is reached on whether to bring into existence such technology, a global moratorium is required (*we already have AGI).
I thought it might be useful to spell that out.
I think it would be good if lots of EAs answered this twitter poll, so we could get a better sense for the communities views on the topic of Enlightenment / Awakening: https://twitter.com/SpencrGreenberg/status/1782525718586413085?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet
I have heard rumours that an AI Safety documentary is being made. Separate to this, a good friend of mine is also seriously considering making one, but he isn't "in" AI Safety. If you know who this first group is and can put me in touch with them, it might be worth getting across each others plans.
Something I'm confused about: what is the threshold that needs meeting for the majority of people in the EA community to say something like "it would be better if EAs didn't work at OpenAI"?
Imagining the following hypothetical scenarios over 2024/25, I can't predict confidently whether they'd individually cause that response within EA?
- Ten-fifteen more OpenAI staff quit for varied and unclear reasons. No public info is gained outside of rumours
- There is another board shakeup because senior leaders seem worried about Altman. Altman stays on
- Superalignment team
... (read more)What are some historical examples of a group (like AI Safety folk) getting something incredibly wrong about an incoming Technology? Bonus question: what led to that group getting it so wrong? Maybe there is something to learn here.
In the 90's and 2000's, many people such as Eric Drexler were extremely worried about nanotechnology and viewed it as an existential threat through the "gray goo" scenario. Yudkowsky predicted drexler style nanotech would occur by 2010, using very similar language to what he is currently saying about AGI.
It turned out they were all being absurdly overoptimistic about how soon the technology would arrive, and the whole drexlerite nanotech project flamed out by the end of the 2000's and has pretty much not progressed since. I think a similar dynamic playing out with AGI is less likely, but still very plausible.
I recently discovered the idea of driving all blames into oneself, which immediately resonated with me. It is relatively hardcore; the kind of thing that would turn David Goggins into a Buddhist.
Gemini did a good job of summarising it:
This quote by Pema Chödron, a renowned Buddhist teacher, represents a core principle in some Buddhist traditions, particularly within Tibetan Buddhism. It's called "taking full responsibility" or "taking self-blame" and can be a bit challenging to understand at first. Here's a breakdown:
What it Doesn't Mean:
- Self-Flagellation:
... (read more)What would be the pros and cons of adding a semi-hidden-but-permanent Hot Takes section to the Forum? All of my takes are Hot and due to time constraints I would otherwise not post at all. Some would argue that someone like me should not post Hot Takes at all. Anyway, in true lazy fashion here is ChatGPT on the pros and cons:
Pros:
- Encourages diverse perspectives and stimulates debate.
- Can attract more engagement and interest from users.
- Provides a platform for expressing unconventional or controversial ideas.
- Fosters a culture of intellectual curiosity and ope
... (read more)One of the seminal texts in marketing science is The Long and The Short of it by Field and Binet. They argue that for maximum effectiveness, marketing should aim to create two distinct work streams and results; immediate sales and longer term sales.
They argue the tactics that go into either are distinct (e.g. short term = create a sense of urgency, long term = match brand associations with category associations).
This feels like a good analogy for AI Safety Advocacy / Governance - keep talking about short term things people can buy now (in an Au... (read more)
Feels kinda obnoxious to write a quick take along the lines of "I'm thinking about writing a post on X, does anyone actually give a sh*t? Otherwise I won't write it."
I just wanted to check, since I can't place my finger on why it feels obnoxious but it certainly does.
Is there a Slack group or something similar for Founders of early-stage (EA) startups?
How impactful could non-dual meditation be for improving wellbeing?
Are there any EAs out there who have practiced non-dual meditation? Specifically, anything from the Dzogchen or Mahamudra traditions of Tibetan Buddhism?
More secular teachers would be Loch Kelly, Sam Harris, Michael Taft.
This has been a life changing experience for me and I'm wondering whether it could be a blind spot for EA.
I'd also just love to chat with someone else who has experienced non-duality / awakening through this form of meditation :)
Prediction: In 6-12 months people are going to start leaving Deepmind and Anthropic for similar sounding reasons to those currently leaving OpenAI (50% likelihood).
> Surface level read of what is happening at OpenAI; employees are uncomfortable with specific safety policies.
> Deeper, more transferable, harder to solve problem; no person that is sufficiently well-meaning and close enough to the coal face at Big Labs can ever be reassured they're doing the right thing continuing to work for a company whose mission is to build AGI.
Basically, this is less about "OpenAI is bad" and more "Making AGI is bad".
Very quick thoughts on setting time aside for strategy, planning and implementation, since I'm into my 4th week of strategy development and experiencing intrusive thoughts about needing to hurry up on implementation;
- I have a 52 week LTFF grant to do movement building in Australia (AI Safety)
- I have set aside 4.5 weeks for research (interviews + landscape review + maybe survey) and strategy development (segmentation, targeting, positioning),
- Then 1.5 weeks for planning (content, events, educational programs), during which I will get feedback from others on th
... (read more)What would stop you from paying for an LLM? Take an extreme case; Sam Altman turns around tomorrow and says "We're racing to AGI, I'm not going to worry about Safety at all."
Would that stop you from throwing him $20 a month?
(I currently pay for Gemini)
Help clear something up for me: I am extremely confused (theoretically) how we can simultaneously have:
1. An Artificial Superintelligence
2. It be controlled by humans (therefore creating misuse of concentration of power issues)
My intuition is that once it reaches a particular level of power it will be uncontrollable. Unless people are saying that we can have models 100x more powerful than GPT4 without it having any agency??
I don't think we need to explicitly alert the reader when we've received help from an LLM to write something (even if it wrote the bulk of the post). That's it, my quickest ever Quick Take.
There have been multiple occasions where I've copy and pasted email threads into an LLM and asked it things like:
I really want an email plugin that basically brute forces rationality INTO email conversations.
Tangentially - I wonder if LLMs can reliably convert peoples claims into a % through sentiment analysis? This would be useful for Forecasters I believe (and rationality in general)
How bad does factory farming need to get before we WANT TO accelerate AI capabilities?
This article suggests that the DAILY "total painfulness of fish suffering is equivalent to the painfulness of one hundred fifty million human deaths" and that most of these deaths are via "slow suffocation, crushing, or live disemboweling".
Let's assume the creation of AGI had a >80% probability of leading to two outcomes for non-human animals: extinction or liberation.
How might we do the math on this?
I figured out why dudes like forecasting so much
At some point over the next 5 years virtually every person reading this will become a Pause advocate, the only question is whether that occurs now, in 2025, 2026, 2027, 2028 or 2029.