This post aims to present why I consider mass media interventions very promising for the GHD cause area, and is in line with the excellent research from Rosie Bettle (while at Founders Pledge). I’ll also talk about implementation features and give a practitioner’s perspective on how to approach them.

I will discuss 3 main points:

• Mass media interventions can be effective in many areas, in many contexts, with many formats
• Mass media interventions can be cheap to implement and thus may easily become cost-effective
• Mass media interventions can be relatively easy to implement but are somewhat hard to design and evaluate

Note: I’m the co-founder of NOVAH, an AIM-incubated non-profit using mass media to prevent violence against women in Rwanda. In the third section in particular, I will share my view based on my experience at NOVAH.

Mass media interventions are effective in many areas, in many contexts, with many formats

On top of Rosie’s aforementioned research, Janet Pauketat (from the Sentience Institute) also finds that mass media interventions are effective, and Jamie Harris finds that public opinion can be shaped. 
Discussing why is the subject of many theories and I will leave it out of the scope of this post. It is also clear that most of the time effect sizes are small, but I will argue later that this is structurally balanced by the wide reach that mass media can offer.

I want to stress the diversity of areas in which mass media has been proven to be effective:

• Changing individual behaviors on taboo/ sensitive topics. This is typically the case for family planning (see the RCT by DMI, or the work FEM is currently undertaking), or for HIV testing (see this Cochrane review). This may partly be explained because misconceptions on taboo topics are frequent (think of beliefs regarding contraceptives side effects). Mass media also works on other sensitive topics that may be strongly ingrained  in social norms, such as female genital mutilation, early-marriage, intimate partner violence (see this valuable meta review from A. Peterman); and gender attitudes at large (e.g. the belief about how long girls' education should be).
• Creating (or destroying) a shared common good, by changing behaviours that affect society at large. Attitudes and behaviors towards others may change because of mass media. A famous study by Elisabeth Levy-Paluck shows how a popular Rwandan radio drama positively influenced behaviors and norms perception regarding intergroup prejudice post-genocide. On the other side, again in Rwanda (a country where because of the hilly landscape radio is very popular), a recent study shows that the hate radio RTLM increased violence during the genocide where it was aired. Similar effects have been shown in pre-war Germany regarding antisemitism. Other recent studies also show effects on positive cooperation; e.g. Indian voters have been persuaded to punish vote-buying candidates or that Afghan citizens cooperated more with military forces to remove roadside bomb threats after exposure to messaging on the radio (this was before the Taliban's return to power).
• Educating people towards positive individual or family health behaviors. A systematic review demonstrates positive behavior change on a wide range of topics: “one-off behaviors such as tuberculosis testing or vasectomy, episodic behaviors such as vaccinations, use of oral rehydration therapy, and early initiation of breastfeeding, and habitual behaviors such as nightly bed net use, handwashing, consumption of iron and foods rich in vitamin A”. This RCT from DMI in Burkina Faso is a typical example of a successful program. In a completely different context, a repeated public awareness campaign on Norwegian TV improved stroke recognition.
• Increasing positive individual non-health behaviours. A recent RCT in Uganda demonstrated how a movie could increase exam grades through role modelling. Recent research in Tanzania (qualitative for now, quantitative study in progress) also shows how adolescent girls avoided age-disparate transactional sex after listening to a drama. Finally, there is promising research on the use of mass-media for mental health, to teach “auto-CBT” to the audience (RCT in progress in Bangladesh).

The examples above are telling, and they also show the variety of the contexts where mass media has been proven to work: high-income countries, low and middle-income countries, post-war settings, etc. covering a large range of topics (health, education, social norms, politics, etc.). 

I also want to emphasize the diversity of the formats used: short films, TV shows, radio ads, radio talk shows, medium to long radio soap operas, and even video games or chatbots.

At NOVAH we conducted our own analysis to decide on which format to use. We believe there is no consensus in the literature on what should be the key requirements in terms of length or format. So we decided to focus on adapting the messaging to the format currently preferred in the context we work in (which is radio dramas in the case of Rwanda).

Mass media interventions are cheap to implement and thus may easily become cost-effective

The cost per listener is low. Production and distribution are relatively cheap - e.g. in Rwanda, you can produce a minute of content on the radio for 50-100$ and air it for 10-20$/minute, excluding promotional ads. On the other hand, the audience reached can be very large (e.g. there are 6M daily listeners on Radio Rwanda, out of a 14M  population). So, in the case of Rwanda, for example, if 5% of Radio Rwanda listeners have listened to a 15-minute episode, the cost per listener is 1350$/300 000=0.0045$/ listener.

Several reports previously found mass media interventions to be cost-effective, including those for FEM (Rethink Priorities), DMI (The Lancet), and the Founders Pledge report analyzes some interventions in detail. Several AIM research reports have also come to similar conclusions (one on IPV, which led to NOVAH’s founding almost a year ago; another one on caregiver awareness).
I believe that if a campaign even has some effect, there are good chance for it to be cost-effective, even if the effect size is small.

One often overlooked positive impact is that change can last over decades, as the knowledge gained and/or the new behaviors can often be transferred from one generation to the next (think about vaccines, FGMs, contraceptives, or drunk driving).

Because effect sizes are often small and the effect may be more diffuse, describing the effectiveness of mass media interventions can be harder than for a concrete,  more visible intervention. Furthermore, their evaluation may be complex, which could explain why these interventions are not more widespread. This brings me to the operational side of mass media interventions, and their specific pros and cons.

Mass media interventions are relatively easy to implement and somewhat hard to design and evaluate

It is relatively easy to learn about content appreciation and reach. Gathering information on viewership/ listenership and understanding whether the content is liked and successful can be done using focus groups or data collected by the media (for instance texts from listeners a  radio station may receive) - though running your own surveys can be somewhat costly..

But reach and appreciation do not say much on the campaign’s success at changing behaviors. 

I believe message design is not a straightforward science: tips and lessons from previous studies exist, but there is not always a “magic recipe” on how to design a successful campaign - at least this was not the case on entertainment-education to prevent IPV.

Measuring impact itself is hard. What constitutes a study’s control group for a media campaign? How do you handle attribution? How long does the effect last? These are complex questions that can only be answered on a case-by-case basis.

Thus, I believe that most of the time evaluation and message design should go hand in hand. The main challenge, which was also identified in IPA’s best bets reports, is to shorten feedback loops as much as possible. Production (especially TV production) is expensive, and re-recording a full series is too expensive: you need to get it right the first time, but how do you do it?

At NOVAH, we took a few approaches to address this:

• Investing upfront: in high quality formative research, talented script writers and early on feedback from national and international experts (e.g. asking them to rank potential messages) - it all brought a lot of value.
• Iterating on the message rather than on the product: we found out the main challenge arises when a message that is too far from your audience (e.g. too progressive, not relevant, not understood, not actionable) rather than from content that is unappealing (after all, producing good content is the job of many professionals, and they are good at it). So we produced short stories, each representing one potential message, and we tested them for feedback in the rural areas where we work.
• Qualitative data: we discovered that qualitative evaluation can bring a lot of value, including validating the different steps of the theory of change, exploring a broad range of potential behaviour changes, and pinpointing what works and what doesn’t. It’s a tool public health academics frequently rely on. Finally, listeners’ groups can also be used to continuously test new episodes.

Once the messaging is ready, it’s easy to scale in a given context: just air on different channels, or air your product more frequently.

Let’s finally stress that a rigorous impact evaluation can be done (but may be costly). Most of the academic papers mentioned earlier use extremely rigorous methodologies to measure impact. One way to do this is to “treat” participants yourself: respondents are (randomly) invited to collective hearings or viewings, and thus exposed to the campaign. Others use the geographical limit of radio signals (or use places where the radio doesn’t reach because of the mountainous terrain as control groups). FEM is developing an extremely valuable innovation: devices that emit other radio waves in someone’s home, effectively replacing your own show, ensuring these families form a perfect control group. For TV studies, researchers have used the possibility to randomize the digital signal, and of course, for web/ social media randomization is easy.

It is important to keep in mind that any impact evaluation results will only be valid for that specific content, in that context. The external validity of a media campaign will remain unproven - but a detailed analysis of the changes in content or context may help to get a feeling about it. 

Conclusion

Research has shown that mass media can be effective in behaviour change in different contexts and with different formats. Moreover, because of the high potential reach of the media campaign and the relatively low production and distribution cost per person touched, the cost-effectiveness of mass media interventions can be highly promising. However, there is no clear pathway for designing the message or the evaluation. 

Producing and evaluating a media campaign seems like navigating troubled waters: one needs to be at ease with uncertainty. Zoom out, and the big picture appears highly positive. I believe this is partly why mass media interventions may be overlooked and also why they’re so promising.

If something in this post sparked your interest and you’d like to discuss, please don’t hesitate to reach out to alexis@novah.ngo.