Hi Nick, thanks for the thoughtful response. I think you make a lot of good points and I agree that there are numerous incentives can can lead an M+E provider to bias results positively. That's why there is a ton of bad M+E out there.
One main reaction: for an employee who works in an M+E org, there is arguably no worse situation than being pressured to skew your results positively, or even worse, taking on projects where you know a certain results is expected by your clients. It makes you feel you work is meaningless, and really sucks. And when you are put in this situations, you sure as hell don't want to work for the same client again.
Yes, i hear you that for bean-counters in an organization (or those who get dividends in a for-profit org), there are strong incentives to make clients happy and get more contracts. But I think that the job-satisfaction incentive for rank-and-file employees skews the other way. And in the course of my experience, I think it is this latter incentive toward truth-telling that has dominated in most cases.
[Disclaimer: I'm the Chief Economist of IDinsight, an M+E provider who has worked with GiveWell and many others. I have a LOT of experience with evaluators being pressured to sugarcoat results, or lack thereof. ]
Strong disagree on this conclusion that M+E providers are inherently biased.
Yes, there are situations where M+E have incentives that can lead to bias. For instance, if an NGO hires an M+E provider to do an external evaluation of themselves, the NGO is therefore the 'client' of the researchers. This can be problematic, since the NGO will need to approve deliverables before payments are made. I've been involved in these situations and it is tricky.
But in general, arrangements can be made to align incentives with the truth. For instance if a funder (like GiveWell) hires an M+E provider to do an evaluation of one of its grantees, the incentives of the M+E provider are aligned with the funder, who hopefully would like to know the unvarnished truth. We've done numerous evaluations for GiveWell (most notably the New Incentives RCT) and have never felt any incentive to skew results one way or another.
From an organizational perspective, a well-run evaluation organization has much stronger long-term incentives to have a reputation for being honest, transparent, and truth-seeking, rather than getting repeat business from any particular client.
By "very careful", I mean they shouldn't make the case that their org is higher-impact than the current org unless they are damn sure. And this is an extremely difficult judgement call to make, when comparing two organizations whose mission is social impact. Given that impact is integral to an EA's worldview, it would be a pretty incendiary accusation for a headhunter to make the case that org X is higher-impact than org Y, so someone should switch jobs. It's one thing to make this case if hiring someone away from Exxon, but another to make the case within a community of arguably impactful organizations. I think these kinds of tactics have potential to cause major rifts within the community so should be avoided.
OP here. Thanks for all of the engagement with this post and for the varying opinions. People have brought up some important points on the benefits of headhunting (increased information, better outcomes for employees, overall better job matches, etc), and I agree with a lot of what is said. After taking these into account and mulling what has been said, here's where I stand (subject to change):
Good points- I take back my earlier "Clearly..." statement, and agree it needs to also include utility gains for the worker in the calculation.
Just to clarify, I wouldn't be advocating that orgs don't hire from peer orgs. Of course, post jobs, make them widely known, take and consider applications from all place. But I think it's different to spend money on dedicated staff to directly target and aggressively recruit staff from friendly orgs within your ecosystem.
Thanks so much for the engagement. We at Giving Green share your concern around some of CATF's activities around carbon capture, though I wouldn't go as far as to say that CATF's work on 45Q is "net harmful". Instead we acknowledge there are tradeoffs from CATF's strategy in this sector that have uncertain overall impacts. We have noted this element as a "Key Uncertainty" in our report. The relevant text is copied at the bottom of this post.
Our recommendation of CATF was primarily based on our assessment of their work in Shipping/Aviation and Enhanced Geothermal, which were two of our focus areas this year. However, we are not specifically recommending restricted donations for two reasons:
From the 'Key Uncertainties' Section of our Deep Dive on CATF
"Advocacy for incentives for power sector CCUS and captured CO2 storage via EOR: CATF’s advocacy for enhancements to the US Section 45Q tax credit included continued eligibility for power sector applications of carbon capture utilization and storage (CCUS). There is concern that the tax incentives may extend the life of US coal and natural gas-fired power plants. One analysis suggests that 45Q could increase the operating years of an otherwise end-of-life coal plant into the 2040s, resulting in at least 6 million metric tons of additional CO2e emissions. CATF claims that it foresees little deployment of CCS in the US power sector but that the plants that use it will help bring down its cost through learning by doing, resulting in accelerated uptake in emerging economies. While Giving Green thinks CCS could be a valuable technology in contexts such as heavy industry or power plants in emerging economies, we share concerns about incentivizing its use for US fossil fuel power plants. In addition, CATF has continued to advocate for the inclusion of enhanced oil recovery (EOR) for storage of captured emissions or atmospheric removals in subsidies such as 45Q. CATF argues that EOR is climate beneficial, that it serves as the primary niche market for scaling capture technologies, and that it can help transition to large-scale saline storage. Others question the need to subsidize it, especially given concerns over environmental justice, the potential to prolong oil extraction, and the involvement of the fossil fuel industry in the trajectory of emerging climate technologies like DAC.