We all make decisions every day. Some of these decisions are pretty inconsequential, such as what to have for an afternoon snack. Some of these decisions are quite consequential, such as where to live or what to dedicate the next year of your life to. Finding a way to make these decisions better is important.
The folks at Charity Science Health and I have been using the same method to make many of our major decisions for the past for years -- everything from where to live to even deciding to create Charity Science Health. The method isn’t particularly novel, but we definitely think the method is quite underused.
Here it is, as a ten step process:
-
Come up with a well-defined goal.
-
Brainstorm many plausible solutions to achieve that goal.
-
Create criteria through which you will evaluate those solutions.
-
Create custom weights for the criteria.
-
Quickly use intuition to prioritize the solutions on the criteria so far (e.g., high, medium, and low)
-
Come up with research questions that would help you determine how well each solution fits the criteria
-
Use the research questions to do shallow research into the top ideas (you can review more ideas depending on how long the research takes per idea, how important the decision is, and/or how confident you are in your intuitions)
-
Use research to rerate and rerank the solutions
-
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable
-
Repeat steps 8 and 9 until sufficiently confident in a decision.
Which charity should I start?
The definitive example for this process was the Charity Entrepreneurship project, where our team decided which charity would be the best possible charity to create.
Come up with a well-defined goal: I want to start an effective global poverty charity, where effective is taken to mean a low cost per life saved comparable to current GiveWell top charities.
Brainstorm many plausible solutions to achieve that goal: For this, we decided to start by looking at the intervention level. Since there are thousands of potential interventions, we placed a lot of emphasis on plausibly highly effectve, and chose to look at GiveWell’s priority programs plus a few that we thought were worthy additions.
Come up with research questions that would help you determine how well each solution fits the criteria: We came up with the following list of questions and research process.
Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: Since this choice was important and we were pretty uninformed about the different interventions, we did shallow research into all of the choices. We then produced the following spreadsheet:
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable / Repeat steps 8 and 9 until sufficiently confident in a decision: We then researched the top eight more deeply, with a keen idea to turn them into concrete charity ideas rather than amorphous interventions. When re-ranking, we came up with a top five, and wrote up more detailed reports --SMS immunization reminders,tobacco taxation,iron and folic acid fortification,conditional cash transfers, and a poverty research organization. A key aspect to this narrowing was also talking to relevant experts, which we wish we did earlier on in the process as it could quickly eliminate unpromising options.
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: As we researched further, it became more clear that SMS immunization reminders performed best on the criteria being highly cost-effective, with a high strength of evidence and easy testability. However, the other four finalists are also excellent opportunities and we strongly invite other teams to invest in creating charities in those four areas.
Which condo should I buy?
Brainstorm many plausible solutions to achieve that goal: For this, I searched around on Zillow and found several candidate properties.
Create custom weights for the criteria: For this decision, I wanted to turn things roughly into a personal dollar value, where I could calculate the benefits minus the costs. The costs were the purchasing cost of the condo turned into a monthly mortgage payment, plus the annual HOA fee, plus the property tax. The benefits were the expected annual rent plus half of Zillow’s expectation for how much the property would increase in value over the next year, to be a touch conservative. I also added some more arbitrary bonuses: +$500 bonus if there was a dishwasher, a +$500 bonus if there was a balcony, and up to +$1000 depending on how much I liked the size of the kitchen. I also added +$3600 if there was a parking space, since the space could be rented out to others as I did not have a car. Solutions would be graded on benefits minus costs model.
Quickly use intuition to prioritize the solutions on the criteria so far: Ranking the properties was pretty easy since it was very straightforward, I could skip to plugging in numbers directly from the property data and the photos.
Property |
Mortgage |
Annual fees |
Annual increase |
Annual rent |
Bonuses |
Total |
A |
$7452 |
$5244 |
$2864 |
$17400 |
+$2000 |
+$9568 |
B |
$8760 |
$4680 |
$1216 |
$19200 |
+$1000 |
+$7976 |
C |
$9420 |
$4488 |
$1981 |
$19200 |
+$1200 |
+$8473 |
D |
$8100 |
$8400 |
$2500 |
$19200 |
+$4100 |
+$9300 |
E |
$6900 |
$4600 |
$1510 |
$15000 |
+$3600 |
+$8610 |
Come up with research questions that would help you determine how well each solution fits the criteria: For this, the research was just to go visit the property and confirm the assessments.
Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: Pretty easy, not much changed as I went to actually investigate.
How should we fundraise?
Come up with a well-defined goal: I want to find the fundraising method with the best return on investment.
Brainstorm many plausible solutions to achieve that goal: For this, our Charity Science Outreach team conducted a literature review of fundraising methods and asked experts, creating a list of the 25 different fundraising ideas.
Create criteria through which you will evaluate those solutions / Create custom weights for the criteria: The criteria we used here was pretty similar to the criteria we later used for picking a charity -- we valued ease of testing, the estimated return on investment, the strength of the evidence, and the scalability potential roughly equally.
Come up with research questions that would help you determine how well each solution fits the criteria: We created this rubric with questions:
-
What research says on it (e.g. expected fundraising ratios, success rates, necessary pre-requisites)
-
What are some relevant comparisons to similar fundraising approaches? How well do they work?
-
What types/sizes of organizations is this type of fundraising best for?
-
How common is this type of fundraising, in nonprofits generally and in similar nonprofits (global health)?
-
How one would run a minimum cost experiment in this area?
-
What is the expected time, cost, and outcome for the experiment?
-
What is the expected value?
-
What is the expected time cost to get best time per $ ratio (e.g., would we have to have 100 staff or huge budget to make this effective)?
-
What further research should be done if we were going to run this approach?
Who should we hire?
Come up with a well-defined goal: I want to hire the employee who will contribute the most to our organization.
Brainstorm many plausible solutions to achieve that goal: For this, we had the applicants who applied to our job ad.
Person |
Autonomy |
Communication |
Creativity |
Break down |
Learn new skills |
Values fit |
Prior experience |
A |
High |
Medium |
Low |
Low |
High |
Medium |
Low |
B |
Medium |
Medium |
Medium |
Medium |
Medium |
Medium |
Low |
C |
High |
Medium |
Medium |
Low |
High |
Low |
Medium |
D |
Medium |
Medium |
Medium |
High |
Medium |
Low |
High |
E |
Low |
Medium |
High |
Medium |
Medium |
Low |
Medium |
Come up with research questions that would help you determine how well each solution fits the criteria: The initial written application was already tailored toward this, but we designed a Skype interview to further rank our applicants.
Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: After our Skype interviews, we re-ranked all the applicants.
Person |
Autonomy |
Communication |
Creativity |
Break down |
Learn new skills |
Values fit |
Prior experience |
A |
High |
High |
Low |
Low |
High |
High |
Low |
B |
Medium |
Medium |
Medium |
Medium |
Low |
Low |
Low |
C |
High |
Medium |
Low |
High |
High |
Medium |
Medium |
D |
Medium |
Low |
Medium |
High |
Medium |
Low |
High |
E |
Low |
Medium |
High |
Medium |
Medium |
Low |
Medium |
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: While “MVP testing” may not be polite to extend to people, we do a form of MVP testing by only offering our applicants one month trials before converting to a permanent hire.
Which television show should we watch?
Come up with a well-defined goal: Our friend group wants to watch a new TV show together that we’d enjoy the most.
Brainstorm many plausible solutions to achieve that goal: We all each submitted one TV show, which created our solution pool.
Create criteria through which you will evaluate those solutions / Create custom weights for the criteria: For this decision, the criteria was the enjoyment value of each participant, weighted equally.
Come up with research questions that would help you determine how well each solution fits the criteria: For this, we watched the first episode of each television show and then all ranked each one.
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: We then watched the winning television show, which was Black Mirror. Fun!
Which statistics course should I take?
Come up with a well-defined goal: I want to learn as much statistics as fast as possible, without having the time to invest in taking every course.
Brainstorm many plausible solutions to achieve that goal: For this, we searched around on the internet and found ten online classes and three books.
Name |
Cost |
Estimated hours |
Depth score |
Breadth score |
How interesting |
Credential level |
$465 |
150 |
10 |
9 |
3 |
5 |
|
Probability and Statistics, Statistical Learning, Statistical Reasoning |
$0 |
150 |
8 |
10 |
4 |
2 |
Critically Evaluate Social Science Research and Analyze Results Using R |
$320 |
144 |
6 |
6 |
5 |
4 |
http://online.stanford.edu/Statistics_Medicine_CME_Summer_15 |
$0 |
90 |
5 |
2 |
7 |
0 |
$0 |
60 |
6 |
5 |
6 |
0 |
|
Statistical Reasoning for Public Health |
$0 |
40 |
5 |
2 |
4 |
2 |
$0 |
20 |
1 |
4 |
6 |
0 |
|
$0 |
8 |
3 |
1 |
5 |
1 |
|
$0 |
5 |
1 |
2 |
10 |
0 |
|
$0 |
1 |
1 |
1 |
11 |
0 |
|
$0 |
60 |
6 |
5 |
6 |
0 |
|
$0 |
25 |
5 |
5 |
2 |
0 |
|
$25 |
50 |
7 |
3 |
3 |
0 |
|
$17 |
20 |
2 |
4 |
8 |
0 |
Come up with research questions that would help you determine how well each solution fits the criteria: For this, the best we could do would be to do a little bit from each of our top class choices, while avoiding purchasing the expensive ones unless free ones did not meet our criteria.
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: Only the first three felt deep enough. Only one of them was free, but we were luckily able to find a way to audit the two expensive classes. After a review of all three, we ended up going with “Master Statistics with R”.
-
This post is also on LessWrong.
Just wanted to say that I found this article really helpful and already sent it to many people who asked me for how they should make a decision. Please never take it down :D
You can also do a Monte Carlo analysis to see how outcomes might differ when you are not confident how strongly you want to prioritise and/or weight. Maybe some of these research investigations are not actually necessary!
80k also recommends a similar process for making career decisions. There's a sketch here: https://80000hours.org/career-guide/personal-fit/ And this tool leads you through a similar process: https://80000hours.org/career-decision/
One thing I noticed is that some decision-making experts, such as the author of Values Focused Decision Making, recommends generating options after you've defined your criteria, because it helps you spot more options. In our process we generate options before defining the criteria, because we found that people find the 'define criteria' step too abstract by itself.
Thank you very much for this post, I really enjoyed your talk on the EAGxVirtual this year (which is what brought me here). I tried to look into the statistics courses you listed since I have similar goals to those you stated. I was wondering about the credential scores on your top course: I looked a bit into it on Coursera and have struggled to find anything about its credentials. Can you enlighten me about it? Thank you very much in advance :)
Is it worth cross-posting this to LessWrong? Anna Salamon is leading an effort to get LessWrong used again as a locus of rationality conversation, and this would fit well there.
Thanks, I did not know that. Done.