Introduction

In my earlier post Estimating value from pairwise comparisons I wrote about a reasonable statistical model for the pairwise comparison experiments that Nuño Sempere at QURI have been doing (see also his sequence on estimating value). While writing that post I started thinking about fields where utility extraction is important and decided to take a look at health economics and environmental economics. This post is a write-up of my attempt at a light survey of the literature on this topic, with particular attention paid on pairwise experiments.

What do I mean by pairwise comparisons? Suppose I ask you "Do you prefer to lose your arm or your leg?" That's a binary pairwise comparison between the two outcomes $A$, $B$, where $A=\text{lose a leg}$ and $B=\text{lose an arm}$. Such comparison studies are truly widespread, going back at least to McFadden (1973), which has $23000$ Google Scholar citations! Models such as these are called discrete choice models, and I will refer to them as dichotomous (binary) comparisons as well, which is terminology I've seen in the economics literature. These models cannot measure the scale of the preferences properly though. There are many reasons why we care about the scale of preferences/utilities. For instance, we need scaling to compare preferences between different studies, and we need scales when we face uncertainty, as part of expected utility theory.

To take scale into account we can ask questions such as "How many times worse would it be to lose an arm than losing a leg?". Then you might answer, say, $1/10$, so you think losing a leg is ten times worse than losing an arm. Or $10$, so you think losing an arm is ten times worse than losing a leg. These questions are harder than the corresponding binary questions though, and I can image respondents being flabbergasted by them. Questions of this kind are called graded (or ratio) comparisons in the literature. The idea is old -- it goes way back to Thurstone (1927)!

I'm excited about the prospect of using pairwise comparisons on a large scale. Here are some applications:

1. Estimate the value of research, both in the context of academia and effective altruism. This post presents a small-scale experiment in the EA context. It would be interesting to do a similar experiment inside of academia. Probably more rigorous and lengthy though. In my experience many academics do not feel that their or other people's work is important. They research whatever is publishable since it's their job. Attempting to quantify researchers understanding of the value of their and other people's research could at least potentially push some researchers into a more effective direction.
2. Estimating the value of EA projects. This should be pretty obvious. One of the potentials of the pairwise value estimation method is crowd-sourcing -- since it's so easy to say "I prefer $A$ to $B$", or perhaps "$A$ is $10$ times better than $B$" -- the bar for participation is likely to be lower than, say, participating in Metaculus, which is a real hassle. Possible applications would be crowd-sourcing of valuation of small projects, e.g. something like Relative Impact of the First 10 EA Forum Prize Winners.
3. Descriptive ethics. You could estimate moral weights for various species. You could get an understanding about how people vary in the their moral valuations. You could run experiments akin to the experiments underlying moral foundations theory, but with a much more quantitative flavor. I haven't thought deeply about it, but I imagine studies of this sort would be important in the context of moral uncertainty.

Summary of thoughts from the short literature review

See the linked post for more information.

1. I had a peek at value estimation in economics and marketing. There is a sizable literature here, and more work is needed to figure out what exactly is relevant for effective altruists. Discrete choice models are applied a lot in economics, but these models are not able to estimate the scaling of the values. Marketing researchers prefer graded pairwise comparisons, which is equivalent to the pairwise method used here, but with limits on how much you can prefer one choice to another.

2. I'm enthusiastic about the prospects of doing larger-scale paired comparison studies on EA topics. The first step would be to finish the statistical framework I started on here, then do a small-scale study suitable for a methodological journal in e.g., psychology or economics. Then we could run a study on a larger scale.

3. Most examples I've seen in health economics, environmental economics, and marketing are only tangentially related to effective altruism. (I don't claim they don't exist -- there's probably many studies in health economics relevant to EA). But the topics of cognitive burden and experimental design is relevant for anyone who's involved with value estimation. It would be good to have at least a medium effort report on these topics -- I would certainly appreciate it! The literature probably contains a good deal of valuable insights for those sufficiently able and motivated to trudge through it.

4. There is a reasonable number of statistical papers on the graded comparisons. But mostly from the $50$s - $70$s. These will be very difficult to read unless you're at the level of a capable master student of statistics. But summarizing and extending their research could potentially be an effective thesis!