Durrell added - I wish all those protesting to animals living in zoos and claiming animals lead far happier lives in the wild - I wish they all saw this!

I agree we shouldn't assume that animals lead far happier lives in the wild; but I don't think that means we should support zoos (which unlike sanctuaries exist for the benefit of humans rather than the benefit of the animals, and typically rely on breeding animals).

To ensure future AIs can satisfy their own preferences, and thereby have a high level of well-being

This implies preferences matter when they cause well-being (positively-valenced sentience).

I subscribe to an eliminativist theory of consciousness, under which there is no "real" boundary distinguishing entities with sentience vs. entities without sentience. Instead, there are simply functional and behavioral cognitive traits, like reflectivity, language proficiency, self-awareness, reasoning ability, and so on.

I am closer to a pure preference utilitarian than a hedonistic utilitarian. As a consequence, I care more about AI preferences than AI sentience per se. In a behavioral sense, AI agents could have strong revealed preferences even if they lack phenomenal consciousness.

This implies that what matters is revealed preferences (irrespective of well-being/sentience/phenomenal consciousness).

In other words, corporations don’t really possess intrinsic preferences; their actions are ultimately determined by the preferences of the people who own and operate them.

...

These intrinsic preferences are significant in a moral sense because they belong to a being whose experiences and desires warrant ethical consideration.

This implies that what matters is intrinsic preferences as opposed to revealed preferences.

These intrinsic preferences are significant in a moral sense because they belong to a being whose experiences and desires warrant ethical consideration.

This (I think) is a circular argument.

I don't have a hard rule for which preferences are ethically important, but I think a key idea is whether the preference arises from a complex mind with the ability to evaluate the state of the world.

This implies cognitive complexity and intelligence is what matters. But one probably could describe a corporation (or a military intelligence battalion) in these terms, and one probably couldn't describe newborn humans in these terms.

If it's coherent to talk about a particular mind "wanting" something, then I think it matters from an ethical point of view.

I think we're back to square 1, because what does "wanting something" mean? If you mean "having preferences for something", which preferences (revealed, intrinsic, meaningful)?

My view is that sentience (the capacity to have negatively- and positively-valenced experiences) is necessary and sufficient for having morally relevant/meaningful preferences, and maybe that's all that matters morally in the world.

To be clear, which preferences do you think are morally relevant/meaningful? I'm not seeing a consistent thread through these statements.

To ensure future AIs can satisfy their own preferences, and thereby have a high level of well-being

...

I subscribe to an eliminativist theory of consciousness, under which there is no "real" boundary distinguishing entities with sentience vs. entities without sentience. Instead, there are simply functional and behavioral cognitive traits, like reflectivity, language proficiency, self-awareness, reasoning ability, and so on.

I am closer to a pure preference utilitarian than a hedonistic utilitarian. As a consequence, I care more about AI preferences than AI sentience per se. In a behavioral sense, AI agents could have strong revealed preferences even if they lack phenomenal consciousness.

...

In other words, corporations don’t really possess intrinsic preferences; their actions are ultimately determined by the preferences of the people who own and operate them.

...

These intrinsic preferences are significant in a moral sense because they belong to a being whose experiences and desires warrant ethical consideration.

In other words, from a moral standpoint, what matters are the preferences of the individual humans involved in the corporation, not the revealed preferences of the corporation itself as a separate entity.

It's not obvious to me how this perspective (which assigns weight to the intrinsic preferences of individuals) is compatible with what you wrote in an earlier comment, downplaying the separateness of individuals and emphasising revealed preferences over phenomenal consciousness (which sounds similar to having intrinsic preferences?):

1. I subscribe to an eliminativist theory of consciousness, under which there is no "real" boundary distinguishing entities with sentience vs. entities without sentience. Instead, there are simply functional and behavioral cognitive traits, like reflectivity, language proficiency, self-awareness, reasoning ability, and so on.
2. I am closer to a pure preference utilitarian than a hedonistic utilitarian. As a consequence, I care more about AI preferences than AI sentience per se. In a behavioral sense, AI agents could have strong revealed preferences even if they lack phenomenal consciousness.

In other words, corporations don’t really possess intrinsic preferences; their actions are ultimately determined by the preferences of the people who own and operate them.

...

From my preference utilitarian point of view, what matters is something more like meaningful preferences. Animals can have meaningful preferences, as can small children, even if they do not exhibit the type of complex agency that human adults do.

What's the difference between "revealed", "intrinsic" and "meaningful" preferences? The latter two seem substantially diffferent from the first.

Animals are unable to communicate with us in a way that allows for negotiation, trade, legal agreements, or meaningful participation in social institutions. Because of this, they cannot make credible commitments, integrate into our legal system, or assert their own interests. This lack of communication largely explains why humans collectively treat animals the way we do—exploiting them without serious consideration for their preferences.

I'm sceptical that animal exploitation is largely explained by a lack of communication. Humans have enslaved other humans with whom they could communicate and enter into agreements (North American slavery); humans have afforded rights/protection/care to humans with whom they can't communicate and enter into agreements (newborn infants, cognitively impaired adults); and I'd be surprised if solving interspecies communication gets us most of the way to the abolition of animal exploitation, though it's highly likely to help.

I think animal exploitation is better explained by a) our perception of a benefit ("it helps us") and b) our collective superior intelligence/power ("we can"), and it's underpinned by c) our post-hoc speciesist rationalisation of the relationship ("animals matter less because they're not our species"). It's not clear to me that us being able to speak to advanced AIs will mean that any of a), b) and c) won't apply in their dealings with us (or, indeed, in our dealings with them).

By clearly defining ownership and legal autonomy, property rights reduce conflict by ensuring that individuals and groups have recognized control over their own resources, rather than relying on brute force to assert their control. As this system has largely worked to keep the peace between humans—who can mutually communicate and coordinate with each other—I am relatively optimistic that it can also work for AIs. This helps explain why I favor integrating AIs into the same legal and economic systems that protect human property rights.

I remain deeply unpersuaded I'm afraid. GIven where we're at on interpretability and alignment vs capabilities, this just feels more like a gorilla or an ant imagining how their relationship with an approaching human is going to go. These are alien minds the AI companies are creating. But I've already said this, so I'm not sure how helpful it is – just my intuition.

I subscribe to an eliminativist theory of consciousness

V interesting!

I am closer to a pure preference utilitarian than a hedonistic utilitarian. As a consequence, I care more about AI preferences than AI sentience per se. In a behavioral sense, AI agents could have strong revealed preferences even if they lack phenomenal consciousness.

Does this mean you consider e.g. corporations to have moral worth, because they demonstrate consistent revealed preferences (like a preference to maximise profit)?

I think there are strong pragmatic reasons to give AIs certain legal rights, even if they don't have moral worth. Specifically, I think granting AIs economic rights would reduce the incentives for AIs to deceive us, plot a violent takeover, or otherwise undermine human interests, while opening the door to positive-sum trade between humans and AIs.

New to me – thanks for sharing. I think I'm (much) more pessimistic than you on cooperation between us and advanced AI systems, mostly because of a) the ways in which many humans use and treat less powerful / collectively intelligent humans and other species and b) it seeming very unclear to me that AGI/ASI would necessarily be kinder.

At least insofar as we're talking about individual liberties, I think I'm willing to bite the bullet on this question. We already recognize various categories of humans as lacking the maturity or proper judgement to make certain choices for themselves. The most obvious category is children, who (in most jurisdictions) are legally barred from entering into valid legal contracts, owning property without restrictions, dropping out of school, or associate with others freely. In many jurisdictions, adult humans can also be deemed incapable of consenting to legal contracts, often through a court order.

These are good points and I now realise refer to negative (rather than positive) rights. I agree with you that we should restrict certain rights of less agentic/intelligent sentient individuals – like the negative rights you list above, plus some positive rights like the right to vote and drive. This doesn't feel like much of a bullet bite to me.

I continue to believe strongly that some negative rights like the right not to be exploited or hurt ought to be grounded solely in sentience, and not at all in intelligence or agency.

Traditionally, economic rights like the freedom to own property are seen as negative rights, not positive rights. The reason is because, in many contexts, economic rights are viewed as defenses against arbitrary interference from criminal or state actors (e.g., protection from crime, unjust expropriation, or unreasonable regulations).

Appreciate this – I didn't know this, makes sense!

Since these categories are often difficult to distinguish in practice, I preferred to sidestep this discussion in my post, and focused instead on a dichotomy which felt more relevant to the topic at hand.

I tend to think that negative vs positive rights remains a better framing than welfare vs rights, partly because I'm not aware of there being a historical precedent for using welfare vs rights in this way. At least in the animal movement this isn't what that dichotomy means – though perhaps one would see this dichotomy across movements rather than within a single movement. If you have reading on this please do share.

Good contribution.

I don't think I agree with this dichotomy:

there is a tendency for these approaches to come apart in the following way:

• A welfare approach often treats entities as passive recipients of care who require external protection. For example, when advocating for child welfare, one might support laws that prevent child abuse and ensure children’s basic needs are met.
• A rights approach, by contrast, often recognizes entities as active agents who should be granted control over their own lives and resources. For example, historically, those advocating for minority rights have pushed for legal recognition of their autonomy, such as the ability to own property, choose their employment, enter enforceable legal contracts, and seek legal recourse through the courts.

Moral advocates – whether for enslaved humans, foetuses or animals – often couch their advocacy in the framework/language of 'rights', even though the rights they were demanding were closer to "protection" than "autonomy".

I think what you're getting at here is something like negative vs positive rights, where negative rights are 'freedoms from' (e.g. freedom from discrimination, bondage, exploitation, being killed) and positive rights are 'freedoms to' (e.g. freedom to own property, vote, marry).

And I broadly disagree with this, or at least would say it's missing out on the most important point (sentience):

In my view, if AIs possess agency and intelligence comparable to or greater than that of human adults, they should not merely be afforded welfare protections but should also be granted legal rights that allow them to act as independent agents

In my view, moral worth/value/consideration is grounded solely in sentience, which is something like the capacity to have positively- and negatively-valenced experiences. (I might go further, and say moral worth is grounded solely in the capacity to suffer, but I'm not sure about that currently).

Agency and intelligence are morally irrelevant to this first, fundamental question of moral consideration. Otherwise why wouldn't the same go for humans? Are more agentic, more intelligent humans deserving of more welfare or more negative rights that e.g. cognitively impaired humans? (Of course agency and intelligence are relevant to positive rights, like the right to vote or drive.)

Finally – I strongly believe we should oppose the creation of artifical sentience, at the very least until we're sure they won't endure (meaningful) suffering. I'd call myself a digital antinatalist, following @RichardP's piece here.

I think worrying about suffering is overblown.

Can you say more about this? Sounds extremely morally unintuitive to me. Check out Brian Tomasik's work here: https://reducing-suffering.org/the-horror-of-suffering/ 

Again, my question: do you think that non-consensually farming cognitively impaired humans for their flesh/secretions, given net-positive lives and no negative externalities, would be morally justifiable (or good)?

I think we shouldn't exploit any sentient beings (i.e. use them as a means to an end without their consent), regardless of their species or substrate. I'm not sure whether this is because I believe exploitation is intrinsically morally wrong, or whether it's because I think it's a helpful proxy for suffering (something I do think is intrinsically morally bad, at least suffering beyond a certain threshold).

Do you disagree - either with my definition of exploitation, or with my normative claim that we shouldn't exploit sentient beings?