After evaluating the Metaculus community’s comments in the Forecast With GiveWell series, GiveWell selected five top contributors to share the $3,500 prize pool:
- NickLaing – $1,500
- katifish – $1,500
- dimaklenchin – $200
- sleepyaardvark – $200
- Zaldath – $100
Special recognition also goes to katifish, who made numerous high-quality comments and reached the maximum individual prize payout. We’re also grateful to the Manifund donors who contributed to the prize pool: Austin Chen, Saul Munn, Anton Makiievskyi, and Ryan Kidd.
The Forecast With GiveWell series aims to strengthen GiveWell’s decision-making, surfacing perspectives and analysis. Forecasts and comments can be compared with GiveWell’s public comments and reasoning, helping GiveWell sense-check the rationales behind its funding decisions and incorporate new information. GiveWell shared the following reflections on the winning comments:
Thank you to everyone who commented on our forecasts! We evaluated comments for clarity (10 points), reasoning (10 points), and impact (10 points). These were judged blind, i.e., without seeing the commenters’ names. Three GiveWell researchers evaluated each comment, including the researcher most closely involved in the grant related to the prediction.Overall, we found the comments to be clear and well-reasoned, and they engaged deeply with complex topics related to a variety of GiveWell’s grantmaking areas. The most impactful comments were ones that brought in additional evidence or knowledge, critiqued our approach, critiqued specific parameters we used in our models, or called out areas where our arguments or models were unclear.Our top-rated comment was by NickLaing on the question: “Will the study of supplying eyeglasses to near-vision impaired adults in Kenya and India find a boost to household consumption of the following levels?” This comment was well-informed, clear, and well-argued. It stood out for its argument of specific ways the study could be biased, including conflict of interest risk, implementation methods, data collection and sampling techniques, and lack of blinding. The commenter dug into our cost-effectiveness analysis and offered alternative suggestions for both implementation and evaluation of this program or one like it.The second-highest rated comment was also by NickLaing. It was the most critical comment of them all, in response to, “Will the study of oral rehydration solution and zinc distribution in Sierra Leone find a statistically significant increase in ORS/zinc usage by the recipients?” The arguments here are well-articulated and highlight some of the potential weaknesses of this grant. Although most claims aren’t related to the prediction itself, and we would have liked to see more evidence for some key claims, we appreciated the level of thought that went into this critique and the engagement with GiveWell’s assumptions and decision-making.Two comments by katifish had the third and fourth highest ratings. The first two relate to our ILC India grant predictions: First “Will in-line chlorination devices serve at least 5.8 million people in Andhra Pradesh at the end of the 4th year of Evidence Action's program?” Katifish’s response to this question was very detailed, sense-checking our estimates with evidence from other types of infrastructure projects in India. It was overall a strongly reasoned comment that included ways the commenter could be wrong or what would make them change their mind. One potential misunderstanding in this comment about what Evidence Action’s program would do (it will install chlorinators on existing water systems, not install new water points) was an important flag for us to make sure we are clearly explaining grant activities in our communications.The fourth-highest rated comment, on the prediction, “Will the household water chlorination rate be 65% or higher for households served in the first two years of Evidence Action's program in Madhya Pradesh and Andhra Pradesh?” introduced a few critiques and new ideas that we hadn’t previously incorporated into our work. For example, katifish pointed out that we didn’t fully explain our reasoning for adjusting results from a study we rely on to estimate chlorine levels in household water. This comment also brought up a consideration we haven’t yet incorporated or thought much about, which is how boiling and seasonality might affect residual chlorine measurements.We’re excited to continue to add forecasts to Metaculus to get even more meaningful engagement with our work. External feedback is an important part of what we do, as we strive for transparency and truth-seeking in order to help people as much as possible.Thank you to everyone who participated!
The Forecast With GiveWell questions remain open, and GiveWell and Metaculus would welcome additional forecasts and reasoning shared on these questions.
Congratulations again to our commenting prize winners, NickLaing, katifish, dimaklenchin, sleepyaardvark, and Zaldath!
Here are all the winning comments:
Note that we awarded a 10.5th place because katifish hit the prize cap of $1,500 with the first $100 of their ninth place comment. Unfortunately the rules we wrote were a bit ambiguous about what should happen in this scenario, so we decided to award the leftover $100 from ninth place as an additional prize.