Listen on: Spotify, Apple Podcasts, or Google Podcasts

Background

About a year ago, I argued as part of a larger red-teaming post (just my section, with a few minor tweaks, is on my blog) that total utilitarianism does not imply that any amount of suffering can be offset or morally justified by even an arbitrarily large amount of wellbeing. This is an uncommon view, and in fact I don't know of anyone else who holds it. 

To quote what I think of as the core of my argument, 

To rephrase that more technically, the moral value of hedonic states may not be well-modeled by the finite real numbers, which many utilitarians seem to implicitly take as inherently or obviously true. In other words, that is, we have no affirmative reason to believe that every conceivable amount of suffering or pleasure is ethically congruent to some real finite number, such as -2 for a papercut or -2B for a week of torture. 

While two states of the world must be ordinally comparable, and thus representable by a utility function, to the best of my knowledge there is no logical or mathematical argument that “units” of utility exist in some meaningful way and thus imply that the ordinal utility function representing total hedonic utilitarianism is just a mathematical rephrasing of cardinal, finite “amounts” of utility.  

Absence of evidence may be evidence of absence in this case, but it certainly isn’t proof. Perhaps hedonic states really are cardinally representable, with each state of the world being placed somewhere on the number line of units of moral value; I wouldn’t be shocked. But if God descends tomorrow to reveal that it is, we would all be learning something new.  

I recently had @DanielFilan (who disagrees with the bolded thesis above), on my podcast to discuss and debate. 

Summary (from Claude)

Here is a summary of the key points made by Aaron and Daniel in their conversation:

Aaron's position and arguments:

• Aaron believes there are some amounts of suffering that cannot be morally justified by any amount of happiness, under total utilitarianism. He thinks inflicting suffering that wouldn't otherwise exist cannot always be outweighed.
• Aaron argues we should be skeptical of the default view that suffering can always be offset by happiness. In the absence of affirmative arguments for offsetability, we should use the heuristic of asking if a perfectly rational being would choose to undergo the suffering for happiness. Aaron has the intuition they would not for certain severe suffering.
• Aaron thinks evolution shapes us to avoid things like suicide for reproductive fitness, not because they are rational. He believes preferences don't have intrinsic value - we should do what maximizes wellbeing.
• Aaron is a moral realist and thinks there are objective better and worse hypothetical states of the world. He believes it's coincidental humans evolved to care about suffering and happiness since they happen to be morally significant.

Daniel's positions and arguments:

• Daniel believes under total utilitarianism, great suffering can always be offset by enough pleasure, even arbitrarily large suffering. This is the common default view in ethics.
• Daniel argues we make choices involving small risks of suffering for large probabilities of happiness, like going to the beach. He thinks evolution shaping us to avoid suicide suggests avoiding suicide is rational.
• Daniel takes morality to be about what we fundamentally prefer. He thinks evolution gives us the intuition to care about morality, so evolutionarily-derived preferences suggest moral rationality, unlike aliens caring about paperclips.
• Daniel is a moral realist and agrees there are objective better and worse hypothetical states. He doesn't think it's coincidental we evolved to care about pleasure and suffering.

Agreements:

• Both accept the orthogonality thesis that agents can have any goals independent of intelligence.
• Both are moral realists who believe there are objectively better and worse hypothetical states of the world.

Disagreements:

• They disagree on whether extreme suffering can always be outweighed by enough happiness under total utilitarianism.
• They differ on whether evolutionarily-derived preferences like avoiding suicide suggest moral rationality, or can be irrational.
• They dispute whether humans caring about pleasure and suffering is a lucky coincidence or not.

Transcript

Notes

• The core discussion on ethics begins at 7:58 and moves into philosophy of language at ~1:12:19
• Only the transcript corresponding to this section is included below. 
• The full transcript is on my blog here.
• Lightly edited, still very imperfect.

AARON

So basically my ethical position or the thing that I think is true and also not the default view - in fact most I think most people think it is wrong - is: total utilitarianism does not imply that for some amount of suffering that could be created there exists some other extremely large, arbitrarily large amount of happiness that could also be created which would morally justify the former. 

DANIEL

So you think that even under total utilitarianism there can be big amounts of suffering such that there's no way to morally tip the calculus. However much pleasure you can create, it's just not going to outweigh the fact that you inflicted that much suffering on some people.

AARON

Yeah, and I'd highlight the word inflicted. If something's already there and you can't do anything about it, that's kind of neither here nor there as it pertains to your actions. So it's really about you increasing, you creating suffering that wouldn't have otherwise been created. Yeah. It's also been a couple of months since I've thought about this in extreme detail, although I thought about it quite a bit. 

DANIEL

Maybe I should say my contrary view, I guess, when you say that, I don't know, does total utilitarianism imply something or not? I'm like, well, presumably it depends on what we mean by total utilitarianism. Right. So setting that aside, I think that thesis is probably false. I think that yeah. You can offset great amounts of suffering with great amounts of pleasure, even for arbitrary amounts of suffering.

AARON

Okay. I do think that position is like the much more common and even, I'd say default view. Do you agree with that? It's sort of like the implicit position of people who are of self described total utilitarians who haven't thought a ton about this particular question.

DANIEL

Yeah, I think it's probably the implicit default. I think it's the implicit default in ethical theory or something. I think that in practice, when you're being a utilitarian, I don't know, normally, if you're trying to be a utilitarian and you see yourself inflicting a large amount of suffering, I don't know. I do think there's some instinct to be like, is there any way we can get around this?

AARON

Yeah, for sure. And to be clear, I don't think this would look like a thought experiment. I think what it looks like in practice and also I will throw in caveats as I see necessary, but I think what it looks like in practice is like, spreading either wild animals or humans or even sentient digital life through the universe. That's in a non s-risky way, but that's still just maybe like, say, making the earth, making multiple copies of humanity or something like that. That would be an example that's probably not like an example of what an example of creating suffering would be. For example, just creating another duplicate of earth.

DANIEL

Anything that would be like so much suffering that we shouldn't even the pleasures of earth outweighs.

AARON

Not necessarily, which is kind of a cop out. But my inclination is that if you include wild animals, the answer is yes. But I'm much more committed to "some amount." It's like some amount than this particular time and place in human history is like that or whatever.

DANIEL

Okay, can I get a feel of some other concrete cases to see?

AARON

Yeah.

DANIEL

So one example that's on my mind is, like, the atomic bombing of Hiroshima and Nagasaki, right? So the standard case for this is, like, yeah, what? A hundred thousand people died? Like, quite terrible, quite awful. And a lot of them died, I guess a lot of them were sort of some people were sort of instantly vaporized, but a lot of people died in extremely painful ways. But the countercase is like, well, the alternative to that would have been like, an incredibly grueling land invasion of Japan, where many more people would have died or know regardless of what the actual alternatives were. If you think about the atomic bombings, do you think that's like the kind of infliction of suffering where there's just not an offsetting amount of pleasure that could make that okay?

AARON

My intuition is no, that it is offsettable, but I would also emphasize that given the actual historical contingencies, the alternative, the implicit case for the bombing includes reducing suffering elsewhere rather than merely creating happiness. There can definitely be two bad choices that you have to make or something. And my claim doesn't really pertain to that, at least not directly.

DANIEL

Right. Sorry. But when you said you thought your answer was no, you think you can't offset that with pleasure?

AARON

My intuition is that you can, but I know very little about how painful those deaths were and how long they lasted.

DANIEL

Yeah, so the non offset so it's like, further out than atomic bombing.

AARON

That's my guess, but I'm like 🤷‍♂️.

DANIEL

Okay, sure, that's your guess. You're not super confident. That's fine. I guess another thing would be, like, the animal farming system. So, as you're aware, tons of animals get kept in farms for humans to eat, by many count. Many of them live extremely horrible lives. Is there some amount that humans could enjoy meat such that that would be okay?

AARON

No. So the only reason I'm hesitating is, like, what is the actual alternative here? But basically the answer is no. Although, like, what I would actually endorse doing depends on what the alternative is.

DANIEL

Okay, but you think that factory farming is so bad that it's not offsettable by pleasure.

AARON

Yeah, that's right. I'm somewhat maybe more confident than the atomic bombing case, but again, I don't know what it's like to be a factory farm pig. I wouldn't say I'm, like, 99% sure. Probably more than 70% or something. Or 70%, like, conditional on me being right about this thesis, I guess something like that, which I'm like. Yeah, okay. I don't know. Some percent, maybe, not probably not 99% sure, but also more than 60. Probably more than 70% sure or something.

DANIEL

All right. Yeah. So I guess maybe can you tell us a little bit about why you would believe that there's some threshold that you like where you can no longer compensate by permitting pleasure?

AARON

Yes. Let me run through my argument and sort of a motivation, and the motivation actually is sort of more a direct answer to what you just said. So the actual argument that I have and I have a blog post about this that I'll link, it was part of an EA forum post also that you'll also link in the show description is that the affirmative default case doesn't seem to actually be made anywhere. That's not the complete argument, but it's a core piece of it, which is that it seems to be, like, the default received view, which doesn't mean it's wrong, but does mean that we should be skeptical. If you accept that I'm right, that the affirmative case hasn't been made, we can talk about that. Then you should default to some other heuristic. And the heuristic that I assert and sort of argue, but kind of just assert is a good heuristic is. Okay. Is you do the following thought experiment. If I was a maximally or perfectly rational being, would I personally choose to undergo this amount of suffering in compensation or not compensation, exchange for later undergoing or earlier undergoing some arbitrarily large amount of happiness. And I personally have the intuition that there are events or things that certainly conceivable states and almost certainly possible states that I could be in such that even as a rational being, like as a maximum rational being, I would choose to just disappear and not exist rather than undergo both of these things.

DANIEL

Okay.

AARON

Yeah.

DANIEL

Why do you think that?

AARON

Yeah, so good question. I think the answer comes at a couple of different levels. So there's a question of why I'm saying it and why I'm saying it is because I'm pretty sure this is the answer I would actually give if actually given if Credibly offered this option. But that just pushes the question back. Okay, why do I feel that.

DANIEL

Even what option are we talking about here? There exists a thing such that for.

AARON

All pleasures, basically, for example, let's just run with the fact, the assumption that a genie God descends. And I think it's credible, and he offers that I can live the life of every factory, farmed animal in exchange for whatever I want for any amount of time or something like that. Literally, I don't have to give the answer now. It can just be like an arbitrarily good state for an arbitrarily long period of time.

DANIEL

Oh, yeah.

AARON

And not only would I say the words no, I don't want to do that, I think that the words no, I don't want to do that, are selfishly in a non pejorative sense. Correct. And then there's a question of why do I have that intuition? And now I'm introspecting, which is maybe not super reliable. I think part of my intuition that I can kind of maybe sort of access via introspection just comes from basically, I'm very fortunate to not have had a mostly relatively comfortable life, like as a Westerner with access to painkillers, living in the 21st century. Even still, there have definitely been times when I've been suffered, at least not in a relative sense, but just like, in an absolute sense to me, in a pretty bad way. And one example I can give was just like, I was on a backpacking trip, and this is the example I give in another blog post I can link. I was on a backpacking trip, and we didn't have enough food, and I was basically very hungry for like five days. And I actually think that this is a good and I'm rambling on, but I'll finish up. I think it's illustrative. I think there's some level of suffering where you're still able to do at least for me, I'm still able to do something like reasoning and intentionally storing memories. One of the memories I tried to intentionally codify via language or something was like, yeah, this is really bad, this really sucks, or something like, that what.

DANIEL

Sucked about it, you were just like, really hungry yeah.

AARON

For five days.

DANIEL

Okay. And you codified the thought, like, feeling of this hunger I'm feeling, this really sucks.

AARON

Something like that. Right. I could probably explicate it more, but that's basically okay. Actually, hold on. All right. Let me add so not just it really sucks, but it sucks in a way that I can't normally appreciate, so I don't normally have access to how bad it sucks. I don't want to forget about this later or something.

DANIEL

Yeah. The fact that there are pains that are really bad where you don't normally appreciate how bad they are, it's not clear how that implies non offset ability.

AARON

Right, I agree. It doesn't.

DANIEL

Okay.

AARON

I do think that's causally responsible for my intuition that I lend link to a heuristic that I then argue does constitute an argument in the absence of other arguments for offset ability.

DANIEL

Yeah. Okay. So that causes this intuition, and then you give some arguments, and the argument is like, you think that if a genie offered you to live liable factory farmed animals in exchange for whatever you wanted, you wouldn't go for that.

AARON

Yes. And furthermore, I also wouldn't go for it if I was much more rational.

DANIEL

If you were rational, yeah. Okay. Yeah. What do I think about this? One thing I think is that the I think the case of live experience this suffering and then experience this pleasure, to me, I think that this is kind of the wrong way to go about this. Because the thing about experiencing suffering is that it's not just we don't live in this totally dualistic world where suffering just affects only your immaterial mind or something in a way where afterwards you could just be the same. In the real world, suffering actually affects you. Right. Perhaps indelibly. I think instead, maybe the thing I'd want to say is suppose you're offered a gamble, right, where there's like a 1% chance that you're going to have to undergo excruciating suffering and a 99% chance that you get extremely awesome pleasures or something.

AARON

Yeah.

DANIEL

And this is meant to model a situation in which you do some action in which one person is going to undergo really bad suffering and 99 other people are going to undergo really great pleasure. And to me, I guess my intuition is that for any bad thing, you could make the probability small enough and you can make the rest of the probability mass good enough that I want to do that. I feel like that's worth it for me. And now it feels a little bit unsatisfying that we're just going that we're both drilling down to, like, well, this is the choice I would make, and then maybe you can disagree that it's the choice you would make. But yeah, I guess about the gambling case, what do you think about that? Let's say it's literally a one in a million chance that you would have to undergo, let's say, the life of one factory farmed animal.

AARON

Yeah.

DANIEL

Or is that not enough? Do you want it to be like, more?

AARON

Well, I guess it would have to be like one of the worst factory farmed animals. Life, I think would make that like.

DANIEL

Yeah, okay, let's say it's like, maybe literally one in a billion chance.

AARON

First of all, I do agree that these are basically isomorphic or morally equivalent, or if anything, time ordering in my example does mess things up a little bit, I'll be happy to reverse them or say that instead compare one person to 1000 people. So, yeah, you can make the probability small enough that my intuition changes. Yeah. So in fact, 1%, I'm very like, no, definitely not doing that. One in a million. I'm like, I don't know, kind of 50 50. I don't have a strong intuition either way. 100 trillion. I have the intuition. You know what? That's just not going to happen. That's my first order intuition. I do think that considering the case where you live, one being lives both lives, or you have, say, one being undergoing the suffering and then like 100 trillion undergoing the pleasure makes small probabilities more if you agree that they're sort of isomorphic makes them more complete or something like that, or complete more real in some. Not tangible is not the right word, but more right.

DANIEL

You're less tempted to round it to zero.

AARON

Yeah. And so I tend to think that I trust my intuitions more about reasoning. Okay, there's one person undergoing suffering and like 100 trillion undergoing happiness as it pertains to the question of offset ability more than I trust my intuitions about small probabilities.

DANIEL

I guess that's strange because that strikes me as strange because I feel like you're regularly in situations where you make choices that have some probability of causing you quite bad suffering, but a large probability of being fun. Like going to the beach. There could be a shark there. I guess this is maybe against your will, but you can go to a restaurant, maybe get food poisoning, but how often are you like, oh man, if I flip this switch, one person will be poisoned, but 99 people will?

AARON

Well, then you'd have to think that, okay, staying home would actually be safer for some reason, which I don't affirmatively think is true, but this actually does work out for the question of whether you should kill yourself. And there hopefully this doesn't get censored by Apple or whatever, so nobody do that. But there I just think that my lizard brain or there's enough evolutionary pressure to not trust that I would be rational when it comes to the question of whether to avoid a small chance of suffering by unaliving myself, as they say on TikTok.

DANIEL

Hang on, evolution is pressured. So there's some evolutionary pressure to make sure you really don't want to kill yourself, but you think that's like, irrational.

AARON

I haven't actually given this a ton of thought. It gets hard when you loop in altruism and yeah, the question also there's like some chance that of sentient's after death, there's not literally zero or something like that. Yeah, I guess those are kind of cop outs. So I don't know, I feel like it certainly could be. And I agree this is sort of like a strike against my argument or something. I can set up a situation you have no potential to improve the lives of others, and you can be absolutely sure that you're not going to experience any sentience after death. And then I feel like my argument does kind of imply that, yeah, that's like the rational thing to do. I wouldn't do it. Right. So I agree. This is like a strike against me.

DANIEL

Yeah. I guess I just want to make two points. So the first point I want to make is just methodologically. If we're talking about which are you likely to be more rational about gambles of small risks, small probabilities of risk versus large rewards as opposed to situations where you can do a thing that affects a large number of people one way and a small number of people another way? I think the gambles are more like decisions that you make a bunch and you should be rational about and then just the second thing in terms of like, I don't know, I took you to be making some sort of argument along the lines of there's evolutionary pressure to want to not kill yourself. Therefore, that's like a debunking explanation. The fact that there was evolutionary pressure to not kill ourselves means that our instinct that we shouldn't kill ourselves is irrational. Whereas I would tend to look at it and say the fact that there was very strong evolutionary pressure to not kill ourselves is an explanation of why I don't want to kill myself. And I see that as affirming the choice to not kill myself, actually.

AARON

Well, I just want to say I don't think it's an affirmative argument that it is irrational. I think it opens up the question. I think it means it's more plausible that for other I guess not even necessarily for other reasons, but it just makes it more plausible that it is irrational. Well.

DANIEL

Yeah, I take exactly the opposite view. Okay. I think that if I'm thinking about, like, oh, what do I really want? If I consider my true preferences, do I really want to kill myself or something? And then I learn that, oh, evolution has shaped me to not kill myself, I think the inference I should make is like, oh, I guess probably the way evolution did that is that it made it such that my true desires are to not kill myself.

AARON

Yeah. So one thing is I just don't think preferences have any intrinsic value. So I don't know, we might just like I guess I should ask, do you agree with that or disagree with.

DANIEL

That do I think preferences have intrinsic value? No, but so no, but I think like, the whole game here is like, what do I prefer? Or like, what would I prefer if I understood things really clearly?

AARON

Yes. And this is something I didn't really highlight or maybe I didn't say it at all, is that I forget if I really argue it or kind of just assert it, but I at least assert that the answer to hedonic utilitarian. What you should do under hedonic utilitarianism is maybe not identical to, but exactly the same as what a rational agent would do or what a rational agent would prefer if they were to experience everything that this agent would cause. Or something like that. And so these should give you the exact same answers is something I believe sure. Because I do think preferences are like we're built to understand or sort of intuit and reason about our own preferences.

DANIEL

Kind of, yeah. But broadly, I guess the point I'm making at a high level is just like if we're talking about what's ethical or what's good or whatever, I take this to ultimately be a question about what should I understand myself as preferring? Or to the extent that it's not a question of that, then it's like, I don't know, then I'm a bit less interested in the exercise.

AARON

Yeah. It's not ideal that I appeal to this fake and that fake ideally rational being or something. But here's a reason you might think it's more worth thinking about this. Maybe you've heard about I think Tomasik makes an argument about yeah. At least in principle, you can have a pig that's in extreme pain but really doesn't want to be killed still or doesn't want to be taken out of its suffering or whatever, true ultimate preference or whatever. And so at least I think this is pretty convincing evidence that you can have where that's just like, wrong about what would be good for it, you know what I mean?

DANIEL

Yeah, sorry, I'm not talking about preference versus hedonic utilitarianism or anything. I'm talking about what do I want or what do I want for living things or something. That's what I'm talking about.

AARON

Yeah. That language elicits preferences to me and I guess the analogous but the idea.

DANIEL

Is that the answer to what I want for living things could be like hedonic utilitarianism, if you see what I mean.

AARON

Or it could be by that do you mean what hedonic utilitarianism prescribes?

DANIEL

Yeah, it could be that what I want is that just whatever maximizes beings pleasure no matter what they want.

AARON

Yeah. Okay. Yeah, so I agree with that.

DANIEL

Yeah. So anyway, heading back just to the suicide case right. If I learn that evolution has shaped me to not want to kill myself, then that makes me think that I'm being rational in my choice to not kill myself.

AARON

Why?

DANIEL

Because being rational is something like optimally achieving your goals. And I'm a little bit like I sort of roughly know the results of killing myself, right? There might be some question about like, but what are my goals? And if I learned that evolution has shaped my goals such that I would hate killing myself right, then I'm like, oh, I guess killing myself probably ranks really low on the list of states ordered by how much I like them.

AARON

Yeah, I guess then it seems like you have two mutually incompatible goals. Like, one is staying alive and one is hedonic utilitarianism and then you have to choose which of these predominates or whatever.

DANIEL

Yeah, well, I think that to the extent that evolution is shaping me to not want to commit suicide, it looks like the not killing myself one is winning. I think it's evidence. I don't think it's conclusive. Right. Because there could be multiple things going on. But I take evolutionary explanations for why somebody would want X. I think that's evidence that they are rational in pursuing X rather than evidence that they are irrational in pursuing X.

AARON

Sometimes that's true, but not always. Yeah, there's a lot in general it is. Yeah. But I feel like moral anti realistic, we can also get into that. Are going to not think this is like woo or Joe Carlsmith says when he's like making fun of moralists I don't know, in a tongue in cheek way. In one of his posts arguing for explicating his stance on antirealism basically says moral realists want to say that evolution is not sensitive to moral reasons and therefore evolutionary arguments. Actually, I don't want to quote him from memory. I'll just assert that evolution is sensitive to a lot of things, but one of them is not moral reasons and therefore evolutionary arguments are not a good evidence or are not good evidence when it comes to purely, maybe not even purely, but philosophical claims or object level moral claims, I guess, yeah, they can be evidenced by something, but not that.

DANIEL

Yeah, I think that's wrong because I think that evolution why do I think it's wrong? I think it's wrong because what are we talking about when we talk about morality? We're talking about some logical object that's like the completion of a bunch of intuitions we have. Right. And those I haven't thought about intuitions are the product of evolution. The reason we care about morality at all is because of evolution under the standard theory that evolution is the reason our brains are the way they are.

AARON

Yeah, I think this is a very strange coincidence and I am kind of weirded out by this, but yes, I.

DANIEL

Don'T think it's a coincidence or like not a coincidence.

AARON

So it's not a coincidence like conditional honor, evolutionary history. It is like no extremely lucky or something that we like, of course we'd find it earthlings wound up with morality and stuff. Well, of course you would.

DANIEL

Wait. Have you read the metafic sequence by Eliezer Yudkowski?

AARON

I don't think so. And I respect Eliezer a ton, except I think he's really wrong about ethics and meta ethics in a lot of like I don't even know if I but I have not, so I'm not really giving them full time.

DANIEL

Okay. I don't know. I basically take this from my understanding of the meta ethics sequence, which I recommend people read, but I don't think it's a coincidence. I don't think we got lucky. I think it's a coincidence. There are some species that get evolved, right, and they end up caring about schmorality, right?

AARON

Yeah.

DANIEL

And there are some species that get evolved, right? And they end up caring about the prime numbers or whatever, and we evolved and we ended up caring about morality. And it's not like a total so, okay, partly I'm just like, yeah, each one of them is really glad they didn't turn out to be the other things. The ones that care about two of.

AARON

Them are wrong, but two of them are wrong.

DANIEL

Well, they're morally wrong. Two of them do morally wrong things all the time. Right?

AARON

I want to say that I hate when people say that. Sorry. So what I am saying is that you can call those by different names, but if I'm understanding this argument right, they all think that they're getting at the same core concept, which is like, no, what should we do in some okay, so does schmorality have any sort of normativity?

DANIEL

No, it has schmormativity.

AARON

Okay, well, I don't know what schmormativity is.

DANIEL

You know how normativity I feel like that's good. Schmormativity is about promoting the schmud.

AARON

Okay, so it sounds like that's just normativity, except it's normativity about different propositions. That's what it sounds like.

DANIEL

Well, basically, I don't know, instead of these schmalians wait, no, they're aliens. They're not shmalians. They're aliens. They just do a bunch of schmud things, right? They engage in projects, they try and figure out what the schmud is. They pursue a schmud and then they look at humans, they're like, oh, these humans are doing morally good things. That's horrible. I'm so glad that we pursue the schmood instead.

AARON

Yeah, I don't know if it's incoherent. I don't think they're being incoherent. Your description of a hypothetical let's just take for granted whatever in the thought experiment is in fact happening. I think your description is not correct. And the reason it's not correct is because there is like, what's a good analogy? So when it comes to abstract concepts in general, it is very possible for okay, I feel like it's hard to explain directly, but here an analogy, is you can have two different people who have very different conceptions of justice, but fundamentally are earnestly trying to get at the same thing. Maybe justice isn't well defined or isn't like, actually, I should probably have come up with a good example here. But you know what? I'm happy to change the word for what I use as morality or whatever, but it has the same core meaning, which is like, okay, really, what should you do at the end of the day?

DANIEL

Yeah.

AARON

What should you do?

DANIEL

Whereas they care about morality, which is what they should do, which is a different thing. They have strong desires to do what they should do.

AARON

I don't think it is coherent to say that there are multiple meanings of the word should or multiple kinds. Yeah.

DANIEL

No, there aren't.

AARON

Sorry. There aren't multiple meanings of the word should. Fine.

DANIEL

There's just a different word, which is schmood, which means something different, and that's what their desires are pegged to.

AARON

I don't think it's coherent, given what you've already the entire picture, I think, is incoherent. Given everything else besides the word schmoud, it is incoherent to assert that there is something broadly not analogous, like maybe isomorphic to normativity or, like, the word should. Yeah. There is only what's yeah. I feel like I'm not gonna I'm not gonna be able to verbalize it super well. I do. Yeah. Can you take something can you pick.

DANIEL

A sentence that I said that was wrong or that was incoherent?

AARON

Well, it's all wrong because these aliens don't exist.

DANIEL

The aliens existed.

AARON

Okay, well, then we're debating, like, I actually don't know. It depends. You're asserting something about their culture and psychology, and then the question is, like, are you right or wrong about that? If we just take for granted that you're right, then you're right. All right. I'm saying no, you can't be sure. So conditional on being right, you're right. Then there's a question of, like, okay, what is the probability? So, like, conditional on aliens with something broad, are you willing to accept this phrase, like, something broadly analogous to morality? Is that okay?

DANIEL

Yeah, sure.

AARON

Okay. So if we accept that there's aliens with something broadly analogous to morality, then you want to say that they can have not only a different word, but truly a pointer to a different concept. And I think that's false.

DANIEL

So you think that in conceptual space, there's morality and that there's, like, nothing near it for miles.

AARON

The study, like yeah, basically. At least when we're talking about, like, the like, at the at the pre conclusion stage. So, like, before you get to the point where you're like, oh, yeah, I'm certain that, like, the answer is just that we need, like, we need to make as many tennis balls as possible or whatever the general thing of, like, okay, broadly, what is the right thing to do? What should I do? Would it be good for me to do this cluster of things yeah. Is, like, miles from everything else.

DANIEL

Okay. I think there's something true to that. I think I agree with that in some ways and on others, my other response is I think it's not a total coincidence that humans ended up caring about morality. I think if you look at these evolutionary arguments for why humans would be motivated to pursue morality. They rely on very high level facts. Like, there are a bunch of humans around. There's not one human who's, like, a billion times more powerful than everyone else. We have language. We talk through things. We reason. We need to make decisions. We need to cooperate in certain ways to produce stuff. And it's not about the fact that we're bipedal or something. So in that sense, I think it's not a total coincidence that we ended up caring about morality. And so in some sense, I think because that's true, you could maybe say you couldn't slightly tweak our species that it cared about something other than morality, which is kind of like saying that there's nothing that close to morality in concept space.

AARON

But I think I misspoke earlier what I should have said is that it's very weird that we care about that most people at least partially care about suffering and happiness. I think that's just a true statement. Sorry, that is the weird thing. Why is it weird? The weird thing is that it happens to be correct, even though I only.

DANIEL

Have what do you mean it's correct?

AARON

Now we have to get okay, so this is going into moral realism. I think moral realism is true, at least.

DANIEL

Sorry, what do you mean by moral realism? Wait, different by moral realism?

AARON

Yes. So I actually have sort of a weak version of moral realism, which is, like, not that normative statements are true, but that there is, like, an objective. So if you can rank hypothetical states of the world in an ordinal way such that one is objectively better than another.

DANIEL

Yes. Okay. I agree with that, by the way. I think that's true. Okay.

AARON

It sounds like you're a moral realist.

DANIEL

Yeah, I am.

AARON

Okay. Oh, really? Okay. I don't know. I thought you weren't. Okay, cool.

DANIEL

Lots of people in my reference class aren't. I think most Bay Area rationalists are not moral realists, but I am.

AARON

Okay. Maybe I was confused. Okay, that's weird. Okay. Sorry about that. Wait, so what do I mean by it happens to be true? It's like it happens to coincide with yeah, sorry, go ahead.

DANIEL

You said it happens to be correct that we care about morality or that we care about suffering and pleasure and something and stuff.

AARON

Maybe that wasn't the ideal terminology it happens to so, like, it's not morally correct? The caring about it isn't the morally correct thing. It seems sort of like the caring is instrumentally useful in promoting what happens to be legitimately good or something. Or, like legitimately good or something like that.

DANIEL

But but I think, like so the aliens could say a similar thing, right? They could say, like, oh, hey, we've noticed that we all care about schmurality. We all really care about promoting Schmeasure and avoiding Schmuffering. Right? And they'd say, like, they'd say, like, yeah, what's? What's wrong?

AARON

I feel like it's not maybe I'm just missing something, but at least to me, it's like, only adding to the confusion to talk about two different concepts of morality rather than just like, okay, this alien thinks that you should tile the universe paperclips, or something like that, or even that more reasonably, more plausibly. Justice is like that. Yeah. I guess this gets back to there's only one concept anywhere near that vicinity in concept space or something. Maybe we disagree about that. Yeah.

DANIEL

Okay. If I said paperclips instead of schmorality, would you be happy?

AARON

Yes.

DANIEL

I mean, cool, okay, for doing the.

AARON

Morally correct thing and making me happy.

DANIEL

I strive to. But take the paperclipper species, right? What they do is they notice, like, hey, we really care about making paperclips, right? And, hey, the fact that we care about making paperclips, that's instrumentally useful in making sure that we end up making a bunch of paperclips, right? Isn't that an amazing coincidence that we ended up caring our desires were structured in this correct way that ends up with us making a bunch of paperclips. Is that like, oh, no, total coincidence. That's just what you cared about.

AARON

You left at the part where they assert that they're correct about this. That's the weird thing.

DANIEL

What proposition are they correct about?

AARON

Or sorry, I don't think they're correct implicitly.

DANIEL

What proposition do they claim they're correct about?

AARON

They claim that the world in which there is many paperclips is better than the world in which there is fewer paperclips.

DANIEL

Oh, no, they just think it's more paperclipy. They don't think it's better. They don't care about goodness. They care about paperclips.

AARON

So it sounds like we're not talking about anything remotely like morality, then, because I could say, yeah, morality, morality. It's pretty airy. It's a lot of air in here. I don't know, maybe I'm just confused.

DANIEL

No, what I'm saying is, so you're like, oh, it's like this total coincidence that humans we got so lucky. It's so weird that humans ended up caring about morality, and it's like, well, we had to care about something, right? Like anything we don't care about.

AARON

Oh, wow, sorry, I misspoke earlier. And I think that's generating some confusion. I think it's a weird coincidence that we care about happiness and suffering.

DANIEL

Happiness and suffering, sorry. Yeah, but mutatus mutantus, I think you want to say that's like a weird coincidence. And I'm like, well, we had to care about something.

AARON

Yeah, but it could have been like, I don't know, could it have been otherwise, right? At least conceivably it could have been otherwise.

DANIEL

Yeah, the paperclip guys, they're like, conceivably, we could have ended up caring about pleasure and suffering. I'm so glad we avoided that.

AARON

Yeah, but they're wrong and we're right.

DANIEL

Right about what?

AARON

And then maybe I don't agree. Maybe this isn't the point you're making. I'm sort of saying that in a blunt way to emphasize it. I feel like people should be skeptical when I say, like okay, I have good reason to think that even though we're in a very similar epistemic position, I have reason to believe that we're right and not the aliens. Right. That's like a hard case to make, but I do think it's true.

DANIEL

There's no proposition that the aliens and us disagree on yes.

AARON

The intrinsic value of pleasure and happiness.

DANIEL

Yeah, no, they don't care about value. They care about schmalu, which is just.

AARON

Like, how much paperclips there is. I don't think that's coherent. I don't think they can care about value.

DANIEL

Okay.

AARON

They can, but only insofar as it's a pointer to the exact same not exact, but like, basically the same concept as our value.

DANIEL

So do you reject the orthogonality thesis?

AARON

No.

DANIEL

Okay. I think that is super intelligent.

AARON

Yeah.

DANIEL

So I take the orthogonality thesis to mean that really smart agents can be motivated by approximately any desires. Does that sound right to you?

AARON

Yeah.

DANIEL

So what if the desire is like, produce a ton of paperclips?

AARON

Yeah, it can do that descriptively. It's not morally good.

DANIEL

Oh, no, it's not morally good at all. They're not trying to be morally good. They're just trying to produce a bunch of paperclips.

AARON

Okay, in that case, we don't disagree. Yeah, I agree. This is like a conceivable state of the world.

DANIEL

Yeah. But what I'm trying to say is when you say it's weird that we got lucky the reason you think it's weird is that you're one of the humans who cares about pleasure and suffering. Whereas if you were one of the aliens who cared about paperclips. The analogous shmarin instead of Aaron would be saying, like, oh, it's crazy that we care about paperclips, because that actually causes us to make a ton of paperclips.

AARON

Do they intrinsically care about paperclips or is it a means of cement?

DANIEL

Intrinsically, like, same as in the Orphogonality thesis.

AARON

Do they experience happiness because of the paperclips or is it more of a functional intrinsic value?

DANIEL

I think they probably experience happiness when they create paperclips, but they're not motivated by the happiness. They're motivated by like, they're happy because they succeeded at their goal of making tons of paperclips. If they can make tons of paperclips but not be happy about it, they'd be like, yeah, we should do that. Sorry. No, they wouldn't. They'd say, like, we should do that and then they would do it.

AARON

Would your case still work if we just pretended that they're not sentient?

DANIEL

Yeah, sure.

AARON

Okay. I think this makes it cleaner for both sides. Yeah, in that case, yes. So I think the thing that I reject is that there's an analog term that's anything like morality in their universe. They can use a different word, but it's pointing to the same concept.

DANIEL

When you say anything like morality. So the shared concepts sorry, the shared properties between morality and paperclip promotion is just that you have a species that is dedicated to promoting it.

AARON

I disagree. I think morality is about goodness and badness.

DANIEL

Yes, that's right.

AARON

Okay. And I think it is totally conceivable. Not even conceivable. So humans wait, what's a good example? In some sense I intrinsically seem to value about regular. I don't know if this is a good example. Let's run with it intrinsically value like regulating my heartbeat. It happens to be true that this is conducive to my happiness and at least local non suffering. But even if it weren't, my brain stem would still try really hard to keep my heart beating or something like that. I reject that there's any way in which promoting heart beatingness is an intrinsic moral or schmoral value or even that could be it could be hypothesized as one but it is not in fact one or something like that.

DANIEL

Okay.

AARON

Likewise, these aliens could claim that making paperclips is intrinsically good. They could also just make them and not make that claim. And those are two very different things.

DANIEL

They don't claim it's good. They don't think it's good.

AARON

They think it's claim it schmud.

DANIEL

Which they prefer. Yeah, they prefer.

AARON

Don't. I think that is also incoherent. I think there is like one concept in that space because wait, I feel like also this is just like at some point it has to cash out in the real world. Right? Unless we're talking about really speculative not even physics.

DANIEL

What I mean is they just spend all of their time promoting paperclips and then you send them a copy of Jeremy Bentham's collected writings, they read it and they're like all right, cool. And then they just keep on making paperclips because that's what they want to do.

AARON

Yeah. So descriptively.

DANIEL

Sure.

AARON

But they never claim that. It's like we haven't even introduced objectivity to this example. So did they ever claim that it's objectively the right thing to do?

DANIEL

No, they claim that it's objectively the paperclipy thing to do.

AARON

I agree with that. It is the paperclippy thing to do.

DANIEL

Yeah, they're right about stuff. Yeah.

AARON

So they're right about that. They're just not a right. So I do think this all comes back down to the question of whether there's analogous concepts in near ish morality that an alien species might point at. Because if there's not, then the paperclippiness is just like a totally radically different type of thing.

DANIEL

But why does it like when did I say that they were closely analogous? This is what I don't understand.

AARON

So it seems to be insinuated by the closeness of the word semantic.

DANIEL

Oh yeah, whatever. When I was making it a similar sounding word, all I meant to say is that they talk about it plays a similar role in their culture as morality plays in our culture. Sorry. In terms of their motivations, I should say. Oh, yeah.

AARON

I think there's plenty of human cultures that are getting at morality. Yeah. So I think especially historically, plenty of human cultures that are getting at the same core concept of morality but just are wrong about it.

DANIEL

Yeah, I think that's right.