Alternative title: should SoGive red-team malaria vaccines?
 

We’ve been seeing a lot of excitement about malaria vaccines – e.g. the first thing mentioned by the EA wins 2023 post was the R21 vaccine.

We at SoGive looked into malaria vaccines about a year ago, and came away with a slightly more cautious impression. Bear in mind though, (a) we were trying to answer a different question^[1]; (b) a lot has changed in a year.

The purpose of this post is to outline these (currently tentative) doubts, and explore whether there's appetite for us to research this more carefully.

The main things we’re still unsure of

• At first glance, malaria vaccines appear less cost-effective than existing malaria interventions (nets/SMC^[2]). Are they, in fact, less cost-effective?
• In light of this, does it make sense to advocate for their rollout?

We thank 1Day Sooner for their helpful comments and constructive collaboration – we sent them a draft of this shortly before publishing. We also thank our contacts at Malaria Consortium and AMF; when we spoke to them in 2022 for our earlier review of malaria vaccines, their comments were very helpful. Some earlier work done by current/former members of the SoGive team has also provided useful groundwork for the thinking here, so thank you to Isobel Phillips, Ishaan Guptasarma, Scott Smith.

Be aware that any indications of cost-effectiveness in this post are extremely rough, and may change materially if we were to conduct this research.

Malaria vaccines may be materially (10x??) less effective than nets/SMC

Based on the research we did a year ago, it seems that malaria vaccines significantly underperform bednets and SMC. Several items in this table are caveated; it’s worth reviewing the version in the appendix which sets out the details.

* SMC is only excluded from this table for brevity, not because of any preference for bednets over SMC.

** Malaria reduction figures are estimates under study conditions

Vaccine costs look high…

When we created this table c.1 year ago, the key message from this table is that costs for vaccines are materially higher than for bednets or SMC, which is significantly driven by logistical difficulties, such as the need for multiple doses and a cold supply chain (i.e. the vaccines have to be kept at a low temperature while they are transported). At the time, we focused on RTS,S because there was more information available.

At that stage, we guessed that R21 would likely have similar costs to RTS,S. Somewhat to our surprise, it does seem that R21 costs may be lower than RTS,S costs. We weren’t clear on the costs of R21, however when we shared a draft of this with 1Day Sooner, they helpfully pointed us to their Dec 2023 Vaccination Status Report. It seems they believe that each dose costs $3.90 on its own, and the all-in cost of delivering the first dose to a person is $25 per full course. 

... and there doesn't * seem * to be an offsetting efficacy benefit.

Although the efficacy numbers look similar, there are several complicating factors not captured in this table. For example, a consideration about the ages of beneficiaries may somewhat make bednets look better for anyone doing a back-of-the-envelope calculation from the above table. This is expanded on in the research plan below.

Numbers are caveated – we haven’t done this research yet.

And just to stress, the cost-effectiveness comparisons (10x less effective) are very rough, back-of-the-envelope figures. 10x is calculated solely by observing that the cost per person appears to be about an order of magnitude higher for R21; there are several other considerations which could move the needle (possibly significantly).

To indicate how materially the 10x estimate could move after doing this work properly, a gut-feel-based 75% confidence interval is 2x-30x, and a gut-feel-based 90% confidence interval is 0.3x-100x. 

Is worse cost-effectiveness reason not to advocate for something?

Should we use advocacy efforts on getting pre-qualification of R21 to happen sooner, or should we use our advocacy efforts on something else, e.g. to get more aid money spent more effectively, which presumably includes more malaria nets and SMC? 

It’s not obvious what the answer to that question is, but it seems reasonable to believe that the cost-effectiveness of vaccines may at least be a relevant part of the story.

The research plan (see below) sets out a number of ways advocating for malaria vaccines might make sense, even if they are not the most cost-effective.

What happens next?

At this stage, the purpose of writing this is to see whether someone else has already resolved these questions.

If not, we would consider looking into this more carefully. Here's our plan for what we would do:

Research plan

• Review and update our understanding of the R21 vaccine as at today. 
  • Is our understanding of the costs correct, or could we be misunderstanding? (e.g. maybe the $2-$4 mentioned by the WHO only captures part of the costs) Ideally, it would also be useful to understand why the R21 costs are materially lower than the RTS,S costs.
    • Update: after we shared a draft of this with 1Day Sooner, they pointed us to their Dec 2023 Vaccination Status Report. It seems they believe that each dose costs $3.90 on its own, and the all-in cost of delivering the first dose to a person is $25 per full course. The research should review/verify this.
  • What’s an up-to-date picture of the efficacy? The numbers in the table are from older papers, and more up to date figures may change the figures.
    • How strong is the evidence base? Normally the evidence from, say, just one RCT would not be considered strong evidence, and might cause us to apply a replicability adjustment.
    • Does the efficacy percentage appropriately reflect the complications of multiple doses (e.g. not all those who got the first dose will get the next dose, etc; is that appropriately captured)
  • Plus any other considerations which would help us determine whether the cost-effectiveness of R21 really is less good than the cost-effectiveness of bednets or SMC, and if so, by how much.
    • This would include any of the other factors in the GiveWell CEA that have been glossed over in this short write-up, such as 
      • Adjustments for differences between trial conditions and actual usage (e.g. net usage may be worse in real life)
      • Adjustment for net coverage years lost due to residual nets from previous distributions
      • Insecticide resistance (note that we have previously critiqued GiveWell’s approach to insecticide resistance adjustments, arguing that they have not incorporated relevant RCT evidence; we might not incorporate these concerns in this analysis)
      • This list is not exhaustive
      • This would likely be implemented by simply taking a copy of the GiveWell model and adjusting it. 
      • Critiquing/red-teaming GiveWell’s analysis is out of scope for this piece of research.
    • The cost-effectiveness analysis should consider ages – vaccines are geared towards under-5s, who have a higher baseline malaria mortality than the general population. However the c.$2 cost per person for bednets is not focused on under-5s. This means that a back-of-the-envelope cost-effectiveness estimate from the figures in the table above would flatter the relative cost-effectiveness figures for bednets. Thank you to 1Day Sooner for highlighting this point.
• Understand the Imperial College cost-effectiveness study on R21 (Schmit et al 2023)
  • When we did our earlier review, the Imperial College cost-effectiveness study on RTS,S (Topazian, Schmit et al 2022) seemed to confirm our impression that RTS,S is not the most cost-effective intervention (at least most of the time).
  • The pre-print for the equivalent paper for R21 states “Under an assumed dose price of US$3 we estimated a median incremental cost-effectiveness ratio compared with current interventions of <...> $36 [range $126, $34] and $33 [range $158, $27] per DALY averted in perennial and seasonal settings, respectively”. 
  • We understand this report was co-authored by the R21 study team.
  • If the findings are correct, this appears to be in the same ballpark as bednets and SMC, which seems surprising given the high costs. 
    • Note that the point around ages above may be an offsetting factor which could conceivably bridge this gap (at least partially)
  • Thank you to the team at 1Day Sooner for flagging this.
• Does the information about malaria vaccines change our views on the cost-effectiveness of bednets and SMC?
  • In principle there could be interaction effects (i.e. the extent to which bednets or SMC reduce malaria might be smaller if the beneficiaries are also receiving vaccines). In practice, this might not be such a big deal, since the overlap probably won’t be large.
  • The new information about the fact that malaria vaccines seem to have a smaller effect on more severe malaria than on malaria as a whole might cause us to think twice about the assumption in GiveWell’s models which says that this effect size is the same regardless of the severity of the malaria. (This is essentially questioning the “Ratio of the reduction in malaria mortality to the reduction in malaria incidence” item in GiveWell’s CEA). Note that we largely want to avoid red-teaming GiveWell’s analysis for this research, so this point may be de-scoped.
• Arguably the policy decision should be based not on today’s cost-effectiveness, but on future cost-effectiveness.
  • A deeper understanding of the cost structure – is there much scope for it to come down further? 
  • How might vaccine efficacy evolve over time?
  • With any of these considerations about future cost effectiveness, it’s useful to distinguish between future changes which will happen anyway, and those which will be accelerated/supported by rolling out the vaccines sooner.
  • Might some kind of forecasting help with this?
  • It might be appropriate to consider this comparatively – i.e. will the cost-effectiveness of bednets/SMC decline over time (e.g. because of diminishing marginal returns of some sort)? If so, might vaccine cost-effectiveness overtake it? If so, when would we expect vaccine cost-effectiveness to overtake bednet/SMC cost-effectiveness? How much sooner than that overtaking point should we start rolling out vaccines? 
• Are there reasons to want to support the roll-out of the vaccine, even if R21’s cost-effectiveness is somewhat behind that of the most cost-effective malaria interventions?
  • For example, those bodies whom you are lobbying to fund the vaccine roll-out might otherwise have been expected to use funds on activities which are less cost-effective than the R21 vaccine. E.g. because they have different cost-effectiveness thresholds.
  • Is there a reason why including a mixture of cost-effective and somewhat less cost-effective interventions together could be a high-impact outcome? E.g. maybe there are reasons why full malaria elimination is particularly valuable, or, related to the previous point, perhaps funders who are being lobbied are keen to ensure that full malaria elimination is achieved. 

If anyone thinks this research plan is incomplete, we would be interested to hear further thoughts.

Similarly, if anyone has immediate answers to the questions, we would be happy to hear their thoughts.

We would be very happy if this analysis weren’t needed, and the purpose of this post is to explore whether it’s useful to conduct this work. If it seems like it’s useful, and there’s enough people who would value it, we may conduct it, especially since our past work on this topic leaves us well-positioned. Also, this is not intended in a mercenary way, but if donors were willing to provide SoGive with financial contributions towards this work, that would make us significantly more likely to prioritise this work.

Appendix: table with more details

This is a repeat of the table which appeared earlier in this document, but with more details/sources/references.

Note 1: 

For brevity, this table references malaria nets but not SMC. This is not intended to indicate some kind of preference for nets over SMC. Indeed, if the table did include a column for SMC, the overall picture would be very similar to what we see for bednets.

Note 2:

When we first produced this table, the cost per person for R21 was populated as

Unknown, but likely similar to the costs for RTS,S.

The rationale we gave was:

We haven’t done a careful review of the costs of the R21 vaccine, and we haven’t seen an explicit estimate of the costs, but we know that there are structural issues with both vaccines which drive costs upwards for both. Furthermore, the R21 vaccine uses an “adjuvant” (or additive) called Matrix M, and we know that this adjuvant has at least one feature which will lead to it being less cost-effective than the AS01 adjuvant used for the RTS,S vaccine. Source: “No such TLR4 ligand is present in the MM adjuvant which could lead to an improved safety profile and lower costs of manufacture” Venkatraman et al 2019 preprint https://www.medrxiv.org/content/10.1101/19009282v1.full.pdf or https://www.medrxiv.org/content/10.1101/19009282v1 Note that we haven’t done a careful review of all the components, and there may be other elements which point in the other direction.

When we updated it for publication recently, we populated this cell with

$8-$16, see caveats

We expanded on this as follows:

We didn’t carefully review this number, so this is caveated, but the WHO said $2-$4 per dose, so we just multiplied that by 4 (4 doses).

Later in the document, we clarified that we had not checked if this was the “all-in” cost, i.e. whether it included logistical and other costs.

After we shared a draft of this with 1Day Sooner, they pointed us to their Dec 2023 Vaccination Status Report. It seems they believe that each dose costs $3.90 on its own, and the all-in cost of delivering the first dose to a person is $25 per full course. 

Thank you to 1Day Sooner for this helpful contribution.

Note 3: 

The cost per person protected by a bednet is calculated as a straight (i.e. unweighted) average of the cost per malaria net (i.e. this row in GiveWell’s CEA), which comes to $4.48. This is multiplied by the number of bednets needed per person (0.49 nets, see elsewhere in the table) to get to $2.18.

Note 4:

It’s common for more than one person to sleep under one net because of bed-sharing. Source of the estimate of 2.06 people sleeping under one net is GiveWell: Number of people covered per net [Oct 2022] (public). 0.49 bednets per person (i.e. about half a bednet per person) is calculated as (1/2.06).

Note 5:

RTS,S must be kept between 2-8°C. R21 also requires 2-8°C, but exhibits "thermostability for 2 weeks at 25°C and 40°C", which would make distribution in Sub-Saharan Africa easier. (See the table on page 3 of this UNICEF document: https://www.unicef.org/supply/media/19456/file/Malaria%20-%20Vaccine%20-%20QA%20-%20October%202023%20-%20English%20.pdf)

Thank you to 1Day Sooner for flagging this.

Note 6:

RTS,S Clinical Trials Partnership, 2015, The Lancet: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5626001/ “The coprimary endpoints of efficacy to clinical malaria over the first 12 months after dose 3 were 55·8% (97·5% CI 50·6–60·4) in children aged 5–17 months and 31·3% (23·6–38·3) in infants aged 6–12 weeks.” Note that this is at 12 months (another row of this table looks at more severe cases of malaria, which is assessed at 18 months)

Note 7:

Datoo et al, 2021, The Lancet: Efficacy of a low-dose candidate malaria vaccine, R21 in adjuvant Matrix-M, with seasonal administration to children in Burkina Faso: a randomised controlled trial - The Lancet. “At 1 year, vaccine efficacy remained high, at 77% (67–84)”

Note 8:

This 45% figure is the reduction in the incidence of uncomplicated episodes of Plasmodium falciparum malaria according to Pryce et al 2018, a Cochrane review of bednets. Study conditions are largely similar to real life, but this figure doesn’t include any adjustments for insecticide resistance, or for the possibility that the 36 months between distributions might be longer than the time period at which malaria incidence was assessed in the studies (which varied between the 5 studies in the Pryce metastudy).

Note 9:

RTS,S Clinical Trials Partnership, 2015, The Lancet: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5626001/ “compared with 171 children who experienced at least one episode of severe malaria in the C3C [control] group, 116 children experienced at least one episode of severe malaria in the R3R [treatment] group (32·2%, 13·7 to 46·9)” This is at 18 months.

Note 10:

The estimate of 44.4% was calculated by simple rescaling, i.e. 32.2% * 77%/55.8%. We have not checked whether the linearity assumptions implicit in this calculation are reasonable.

Note 11:

GiveWell assumes that the same 45% factor applies to both general malaria reductions and to the more severe reductions associated with fatal malaria. This isn’t obviously correct, and they provide some discussion of it in their models (see the comment in this cell), or in their intervention report on SMC.

1. ^{^}

   The question we were answering was not “should we advocate for more malaria vaccines?”, but rather “should the existence of malaria vaccines change our minds about funding existing malaria interventions (bednets and SMC)?”

2. ^{^}

   SMC = seasonal malaria chemoprevention. GiveWell recommends Malaria Consortium’s SMC work.