This post is designed to teach people with basic medical and operational knowledge how to do a simple cost-benefit analysis with life-years as the unit of analysis. I am distilling cost-benefit analysis down to the simplest procedure that will allow a non-expert to produce something useful. This framework has some 'magic numbers' and heuristics I do not explain, but I want a clean presentation. To do a proper analysis, consult this guideline. This post may be updated in the future due to feedback.
This framework is appropriate for the analysis of a relatively simple change in operations. This could include interventions, procedures, or controls. For this post, I'm going to call the thing you are doing the policy. Even if the policy is meant to be applied in a large area, the analysis should always focus on a single model site. This makes it concrete, and makes all of the costs and benefits easier to visualize and list.
Estimating costs
A proper cost-benefit analysis includes an analysis of both the time to implement the program and the monetary costs of things that must be purchased. But far too often, people only look at the money cost, and ignore the time. So to avoid that mistake, this framework focuses almost entirely on the time cost. This is a better approach, because for most policies, the time cost is bigger than the capital and supply costs. If you have an estimate of the annualized money cost, divide it by the GDP per capita of the country the money is coming from, and add that number to the annualized FTE cost you calculate. But if not, just ignore the money and focus on the time.
The first step to estimating costs is to consult the full operational plan for implementing the new policy. Look at everyone who needs to know about the change, and everyone who will have to do anything different. Tally things on a person-by-person basis. Estimate how much time an individual in a particular role will spend on the policy, ideally by consulting several of them, and then multiply by the number of people in that role. Never round any time cost down to zero. Five additional seconds per procedure can be significant, when multiplied by the annual number of procedures.
Be sure to include
1) line staff. How much time will they spend learning about the policy, reviewing it, asking questions about it, and then actually doing it? Think concretely, and if possible, look back at the training and verification time involved in similar policies.
2) customers or patients. How much time, if any, are they required to spend as a result of the policy?
3) managers and executives. How much time will they spend learning about the policy, running training sessions for the staff, monitoring the staff to ensure that the policy is being implemented properly, and doing disciplinary procedures for staff who fail to comply?
4) any outside people who are affected, like suppliers or partners or governing bodies.
Estimate and add up the initial time estimates and the yearly time estimates separately. Then divide the initial number by five. However, if employee turnover time is less than 5 years, their training costs should instead be divided by the turnover time. Then add this to the yearly time, for the total annualized time cost. Divide the hours by 2000 for the annualized FTE cost of the policy.
Estimating benefits
To estimate benefits, you need a study or evidence showing the health benefits of what you're doing. If you cannot find a published study, do an expert elicitation: ask neutral experts for their best-case estimate, their worst-case estimate, and their expected estimate. Then make a triangular distribution from that and aggregate the estimates.
Often the benefit estimate will be in terms of an odds ratio or a relative risk ratio. Turn this into a number of individual cases by multiplying by the number of people in the site each year, and if applicable, the existing prevalence.
List the expected annual number of lives saved (which could be a small percentage chance of saving a life) and all illnesses prevented per year of operation of the new policy. You are going to convert them all into life-years gained, and add them up. To do so, look up the DALY value of all of the illnesses or conditions the policy is expected to prevent here.
1) Every life saved is worth 20 life-years (future years are discounted).
2) Preventing a permanent illness or injury is worth 20 life-years times its DALY adjustment.
3) Preventing a temporary illness or injury is worth its DALY adjustment, times the expected length of the condition.
Social benefits can be converted into DALY values by estimating how the policy will affect the mental health of the affected population, and using the DALY scores for anxiety or depression. This will be speculative, so be sure to add appropriate caveats to the conclusion if social benefits are a large factor.
If the policy produces any monetary benefits, divide the benefit number by the GDP per capita of the country receiving the the benefits.
Decision
Compare the annualized FTE cost to the annualized life-years gained. Ideally you have a 90% confidence interval for one or both, and this uncertainty should be presented and considered (Here's a Google Sheets template for doing Monte Carlo analysis). But if you only have estimates for the averages, use this heuristic:
If the FTE cost is less than the life-year benefit, the policy passes the cost-benefit test. It is good, and should be a priority, assuming it does not have a high money cost you were unable to estimate.
If the FTE cost is greater than the life-year benefit, it's not recommended as a priority. There are probably better things you can be doing.
If the FTE cost is more than five times the life-year benefit, the policy fails the cost-benefit test. It is probably a waste of time, and you should only do it if you are legally forced to.
Executive summary: This post provides a simplified framework for non-experts to conduct cost-benefit analyses of policies and interventions using life-years as the benefit metric.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
This is great! You should consider doing a whole series of posts like this, especially focusing on common misconceptions--the time cost thing was something I fuzzily knew about, but hadn't ever explicitly considered as the single major cost of a policy, such that money cost can be basically ignored.
I'm surprised that a policy only fails the cost-benefit test if the FTE cost five times more expensive than the benefit, and anything less ineffective than that is simply not a priority. What's the reasoning behind that?
The official framing is that a DALY is valued at 2 to 4 times GDP per capita, so given uncertainty, it's probably good if you're buying a DALY for less than GDP per capita and probably bad if you're paying 5x.
My framing is that the disutility of working a job, holding income constant, is probably between 0.2 and 1 DALY.
Interesting! Can you link to any (reasonably simple) example CEAs where this process is applied?
Sadly no, although many EA-style shallow dives use a similar approach. The target audience for this is people like industrial hygienists or nurse managers who want to do an analysis of an operational change. I posted this because a couple people at a recent conference asked me for a guide like this and there was nothing like it available.
Also, do you have any recommendations for estimating the disvalue of a policy that curtails people's freedom (eg. by increasing the price of a good they value)?
There are standard approaches for valuing the loss of consumer surplus from price changes. Traditionally, moving money from one entity to another is just a transfer, not a cost, but there is a deadweight loss associated with price changes, and we measure that as a cost. But you have to have an estimate for how many trades will not happen as a result of the price change.
There are no existing metrics for valuing loss of freedom in DALY terms. You'd basically have to do a proper survey, using similar methodology to the one that generates the DALY losses of various health states.
Thank you!