This is great!

Just wanted to flag that the image of your results doesn't include the last few spreadsheet rows, so it misses out on GDP results for India, which then gets mentioned in the discussion

Yep I've applied! Thanks for thinking of it

Welcome to the EA forum Richard! 

Three comments: 

Should we use risk-averse decision theories?

Note that risk-weighted utility theories (REUT) are not the only way of rationalising Sheila’s decision-making. For example, it might be the case that when Sheila goes to Leighton Moss and sees no birds, she feels an immense regret, because post-hoc she knows she would have been better off if she'd gone to Shapwick Heath. So the decision matrix looks like this, where  δ is the loss of utility due to Sheila's regret at the decision she made. 

That is, I have re-described the situation such that even using expected utility theory, for large enough δ, Sheila is still being rational to go to Shapwick Heath. Of course,  where Sheila would not feel such regret (δ= 0), or the regret would not be large enough to change the result, then on expected utility theory (EUT) Sheila is not acting rationally. 

It’s not clear to me how to choose between EUT and REUT for describing situations like this. My intuition is that, if upon making the decision I could immediately forget that I had done so, then I would not succumb to the Allais paradox (though I probably would succumb to it right now). So I’m weakly inclined to stick with EU. But I haven't thought about this long and hard. 

The relevance to AI strategy

Considerations about risk-aversion are relevant for evaluating different AGI outcomes and therefore are relevant for determining AI strategy. See Holden Karnofsky's Questions about AI strategy: ‘insights about how we should value “paperclipping” vs. other outcomes could be as useful as insights about how likely we should consider paperclipping to be’.  In this case "other outcomes" would include the long miserable and the long happy future. 

Pushing back on the second conclusion

While I agree with the first of the conclusions (“if you are sufficiently averse to risk, and you wish to donate some money to do good, then you should donate it to organisations working to hasten human extinction rather than ones working to prevent it”), I don’t agree with the second (“whether or not you are averse to risk, this is what you should do”).  

In essence, I think the total hedonist utilitarian has no intrinsic reason to take into account the risk-preferences of other people, even if those people would be affected by the decision.

To see this, imagine that Sheila is facing a classic scenario in moral philosophy. A trolley is on track to run over five people, but Sheila has the chance to divert the trolley such that it only runs over one person. Sheila is a total hedonist utilitarian and  believes the world would be better off if she diverted the trolley. We can easily imagine Sheila’s friend Zero having different moral views and coming to the opposite conclusion. Sheila is on the phone to Zero while she is making this decision. Zero explains the reasons he has for his moral values and why he thinks she should not divert the trolley. Sheila listens, but by the end is sure that she does not agree with Zero’s axiology. Her decision theory suggests she should act so as to maximise the amount of utility in the world and divert the trolley. Given her moral values, to do otherwise  would be to act irrationally by her own lights.

If Sheila is risk-neutral and is the leader in the rock-climbing example above, then she has no intrinsic reason to respect the risk preferences of those she affects. That is, the total hedonist utilitarian must bite the bullet and generally reject Pettigrew’s conclusion from the rock-climbing example that "she should descend.” Of course, in this specific case, imposing risks on others could annoy them and this annoyance could plausibly outweigh the value to the total hedonist utilitarian of carrying on up the mountain.

 I think there are good deontological reasons to intrinsically respect the risk-preferences of those around you. But to the extent that the second argument is aimed at the total hedonist utilitarian, I do not think it goes through. For a longer explanation of this argument, see this essay where I responded to an earlier draft of this paper. And I'd love to hear where people think my response goes wrong.