Ron Berman

Angela Duckworth, Ahra Ko, Katherine L. Milkman, Joseph S. Kay, Eugen Dimant, Dena Gromet, Aden Halpern, Youngwoo Jung, Madeline K. Paxson, Ramon A. Silvera Zumaran, Ron Berman, Ilana Brody, Colin Camerer, Elizabeth Canning, Hengchen Dai, Marcos Gallo, Hal E. Hershfield, Matthew Hilchey, Ariel Kalil, Kathryn Kroeper, Amy Lyon, Benjamin Manning, Nina Mazar, Michelle Michelini, Susan Mayer, Mary Murphy, Philip Oreopoulos, Sharon Parker, Renante Rondina, Dilip Soman, Christophe Van den Bulte (2025), National Megastudy Shows that Email Nudges to Elementary School Teachers Boost Student Math Achievement, Particularly When Personalized, Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2418616122

Abstract: In response to the alarming recent decline in US math achievement, we conducted a national megastudy in which 140,461 elementary school teachers who collectively taught 2,992,027 students were randomly assigned to receive a variety of behaviorally informed email nudges aimed at improving students’ progress in math. Specifically, we partnered with the nonprofit educational platform Zearn Math to compare the impact of 15 different interventions with a reminder-only megastudy control condition. All 16 conditions entailed weekly emails delivered to teachers over 4-wk in the fall of 2021. The best-performing intervention, which encouraged teachers to log into Zearn Math for an updated report on how their students were doing that week, produced a 5.06% increase in students’ math progress (3.30% after accounting for the winner’s curse). In exploratory analyses, teachers who received any behaviorally informed email nudge (vs. a reminder-only megastudy control) saw their students’ math progress boosted by an average of 1.89% during the 4-wk intervention period; emails referencing personalized data (i.e., classroom-specific statistics) outperformed emails that did not by 2.26%. While small in size, these intervention effects were consistent across school socioeconomic status and school type (public, private, etc.) and, further, persisted in the 8-wk post-intervention period. Collectively, these findings underscore both how difficult it is to change behavior and the need for large-scale, rigorous, empirical research of the sort undertaken in this megastudy.

Katherine L. Milkman, Sean F. Ellis, Dena Gromet, Alex S. Luscher, Rayyan S. Mobarak, Madeline K. Paxson, Ramon A. Silvera Zumaran, Robert Kuan, Ron Berman, Neil A. Lewis Jr. John A. List, Mitesh S. Patel, Christophe Van den Bulte, Kevin G. Volpp, Maryann V. Beauvais, Jonathan K. Bellows, Cheryl A. Marandola, Angela Duckworth (2024), Megastudy Shows That Reminders Boost Vaccination But Adding Free Rides Does Not, Nature. https://doi.org/10.1038/s41586-024-07591-x

Abstract: Encouraging routine COVID-19 vaccinations is likely to be a crucial policy challenge for decades to come. To avert hundreds of thousands of unnecessary hospitalizations and deaths, adoption will need to be higher than it was in the autumn of 2022 or 2023, when less than one-fifth of Americans received booster vaccines. One approach to encouraging vaccination is to eliminate the friction of transportation hurdles. Previous research has shown that friction can hinder follow-through and that individuals who live farther from COVID-19 vaccination sites are less likely to get vaccinated. However, the value of providing free round-trip transportation to vaccination sites is unknown. Here we show that offering people free round-trip Lyft rides to pharmacies has no benefit over and above sending them behaviourally informed text messages reminding them to get vaccinated. We determined this by running a megastudy with millions of CVS Pharmacy patients in the United States testing the effects of (1) free round-trip Lyft rides to CVS Pharmacies for vaccination appointments and (2) seven different sets of behaviourally informed vaccine reminder messages. Our results suggest that offering previously vaccinated individuals free rides to vaccination sites is not a good investment in the United States, contrary to the high expectations of both expert and lay forecasters. Instead, people in the United States should be sent behaviourally informed COVID-19 vaccination reminders, which increased the 30-day COVID-19 booster uptake by 21% (1.05 percentage points) and spilled over to increase 30-day influenza vaccinations by 8% (0.34 percentage points) in our megastudy. More rigorous testing of interventions to promote vaccination is needed to ensure that evidence-based solutions are deployed widely and that ineffective but intuitively appealing tools are discontinued.

Ron Berman and Christophe Van den Bulte (2022), False Discovery in A/B Testing, Management Science, 68 (9), pp. 6762-6782. 10.1287/mnsc.2021.4207

Abstract: We investigate what fraction of all significant results in website A/B testing is actually null effects (i.e., the false discovery rate (FDR)). Our data consist of 4,964 effects from 2,766 experiments conducted on a commercial A/B testing platform. Using three different methods, we find that the FDR ranges between 28% and 37% for tests conducted at 10% significance and between 18% and 25% for tests at 5% significance (two sided). These high FDRs stem mostly from the high fraction of true null effects, about 70%, rather than from low power. Using our estimates, we also assess the potential of various A/B test designs to reduce the FDR. The two main implications are that decision makers should expect one in five interventions achieving significance at 5% confidence to be ineffective when deployed in the field and that analysts should consider using two-stage designs with multiple variations rather than basic A/B tests.

Mitesh Patel, Katherine L. Milkman, Linnea Gandhi, Heather N. Graci, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Jake Rothschild, Modupe Akinola, John Beshears, Jonathan E. Bogard, Alison Buttenheim, Christopher F. Chabris, Gretchen B. Chapman, James J. Choi, Hengchen Dai, Craig R. Fox, Amir Goren, Matthew D. Hilchey, Jillian Hmurovic, Leslie K. John, Dean Karlan, Melanie Kim, David Laibson, Cait Lamberton, Brigitte C. Madrian, M. Meyer, Maria Modanu, Jimin Nam, Todd Rogers, Renante Rondina, Silvia Saccardo, Maheen Shermohammed, Dilip Soman, Jehan Sparks, Caleb Warren, Megan Weber, Ron Berman, Chalanda N. Evans, Seung Hyeong Lee, Christopher K. Snider, Eli Tsukayama, Christophe Van den Bulte, Kevin Volpp, Angela Duckworth (2022), A Randomized Trial of Behavioral Nudges Delivered through Text Messages to Increase Influenza Vaccination Among Patients with an Upcoming Primary Care Visit, American Journal of Health Promotion, 37 (3), pp. 324-332.

Abstract: Purpose: To evaluate if nudges delivered by text message prior to an upcoming primary care visit can increase influenza vaccination rates. Design: Randomized, controlled trial. Setting: Two health systems in the Northeastern US between September 2020 and March 2021. Subjects: 74,811 adults. Interventions: Patients in the 19 intervention arms received 1-2 text messages in the 3 days preceding their appointment that varied in their format, interactivity, and content. Measures: Influenza vaccination. Analysis: Intention-to-treat. Results: Participants had a mean (SD) age of 50.7 (16.2) years; 55.8% (41,771) were female, 70.6% (52,826) were White, and 19.0% (14,222) were Black. Among the interventions, 5 of 19 (26.3%) had a significantly greater vaccination rate than control. On average, the 19 interventions increased vaccination relative to control by 1.8 percentage points or 6.1% (P = .005). The top performing text message described the vaccine to the patient as “reserved for you” and led to a 3.1 percentage point increase (95% CI, 1.3 to 4.9; P < .001) in vaccination relative to control. Three of the top five performing messages described the vaccine as “reserved for you.” None of the interventions performed worse than control. Conclusions: Text messages encouraging vaccination and delivered prior to an upcoming appointment significantly increased influenza vaccination rates and could be a scalable approach to increase vaccination more broadly.

Katherine L. Milkman, Linnea Gandhi, Mitesh Patel, Heather N. Graci, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Jake Rothschild, Jonathan E. Bogard, Ilana Brody, Christopher F. Chabris, Edward Chang, Gretchen B. Chapman, Jennifer E. Dannals, Noah J. Goldstein, Amir Goren, Hal E. Hershfield, Alexander Hirsch, Jillian Hmurovic, Samantha Horn, Dean Karlan, Ariella Kristal, Cait Lamberton, M. Meyer, Allison H. Oakes, Maurice Schweitzer, Maheen Shermohammed, Joachim H. Talloen, Caleb Warren, Ashley Whillans, Kuldeep N. Yadav, Julian J. Zlatev, Ron Berman, Chalanda N. Evans, Rahul Ladhania, Jens Ludwig, Nina Mazar, Sendhil Mullainathan, Christopher K. Snider, Jann Spiess, Eli Tsukayama, Lyle Ungar, Christophe Van den Bulte, Kevin Volpp, Angela Duckworth (2022), A 680,000-Person Megastudy of Nudges to Encourage Vaccination in Pharmacies, Proceedings of the National Academy of Sciences, 119 (6). 10.1073/pnas.211512611

Abstract: Encouraging vaccination is a pressing policy problem. To assess whether text-based reminders can encourage pharmacy vaccination and what kinds of messages work best, we conducted a megastudy. We randomly assigned 689,693 Walmart pharmacy patients to receive one of 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination or to a business-as-usual control condition that received no messages. We found that the reminder texts that we tested increased pharmacy vaccination rates by an average of 2.0 percentage points, or 6.8%, over a 3-mo follow-up period. The most effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top performing intervention included two texts delivered 3 d apart and communicated to patients that a vaccine was “waiting for you.” Neither experts nor lay people anticipated that this would be the best-performing treatment, underscoring the value of simultaneously testing many different nudges in a highly powered megastudy.

Katherine L. Milkman, Mitesh Patel, Linnea Gandhi, Heather N. Graci, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Modupe Akinola, John Beshears, Jonathan E. Bogard, Alison Buttenheim, Christopher F. Chabris, Gretchen B. Chapman, James J. Choi, Hengchen Dai, Craig R. Fox, Amir Goren, Matthew D. Hilchey, Jillian Hmurovic, Leslie K. John, Dean Karlan, Melanie Kim, David Laibson, Cait Lamberton, Brigitte C. Madrian, M. Meyer, Maria Modanu, Jimin Nam, Todd Rogers, Renante Rondina, Silvia Saccardo, Maheen Shermohammed, Dilip Soman, Jehan Sparks, Caleb Warren, Megan Weber, Ron Berman, Chalanda N. Evans, Christopher K. Snider, Eli Tsukayama, Christophe Van den Bulte, Kevin Volpp, Angela Duckworth (2021), A Megastudy of Text-Based Nudges Encouraging Patients to Get Vaccinated at an Upcoming Doctor’s Appointment, Proceedings of the National Academy of Sciences, 118 (20). 10.1073/pnas.2101165118

Abstract: Many Americans fail to get life-saving vaccines each year, and the availability of a vaccine for COVID-19 makes the challenge of encouraging vaccination more urgent than ever. We present a large field experiment (N = 47,306) testing 19 nudges delivered to patients via text message and designed to boost adoption of the influenza vaccine. Our findings suggest that text messages sent prior to a primary care visit can boost vaccination rates by an average of 5%. Overall, interventions performed better when they were 1) framed as reminders to get flu shots that were already reserved for the patient and 2) congruent with the sort of communications patients expected to receive from their healthcare provider (i.e., not surprising, casual, or interactive). The best-performing intervention in our study reminded patients twice to get their flu shot at their upcoming doctor’s appointment and indicated it was reserved for them. This successful script could be used as a template for campaigns to encourage the adoption of life-saving vaccines, including against COVID-19.

Daniela Schmitt and Ron Berman, Suspenseful and Surprising Content.

Abstract: Not much research has been done about the impact of content types on subscriber demand and long-term revenue. Answering this question empirically is hard because editors make an endogenous decision about the content they solicit and the allocation to the paid and free sections. One type of content, however, shows promise in answering this question - suspenseful and surprising content. Suspenseful events are events for which there is high variance in the uncertainty of their outcome. For example, a close match between two soccer teams. Surprising events are those where the realized outcome is very different than the expectation prior to the event. An example would be an upset victory by an underdog soccer team. Other important contexts where suspense and surprise are common include political debates, voting and elections. Because these events have inherent uncertainty, their realized outcome allows us to use randomness for empirical identification purposes when estimating demand and consumer preferences. Our research project focuses on empirically determining how content editors should treat surprising and suspenseful events - how much coverage should they allocate to them, and whether the content should be paid or free.

Qi Yu, Ron Berman, Eric Bradlow (Working), The Dark Side of Adding a Category: Will Existing Ones Pay the Price.

Abstract: ‘‘More is better’’ has been a belief held by many retailers when they manage product assortments. We challenge this conventional wisdom by demonstrating that a retailer may face more price sensitive demand for existing products when expanding its assortments. To measure the effects of assortment expansion on price sensitivity, we exploit the state of Washington’s privatization of liquor sales in 2012 that generated exogenous variation in retailers’ assortments over time. We find that customers are on average more price sensitive when purchasing from other drink categories after a store started to carry liquor but its impact is heterogeneous. To understand the differential changes in the price sensitivity across product categories depending on whether they are complements or substitutes to the new one, we build a demand model that simultaneously estimates the degree of complementarity between product categories and the changes in price sensitivity upon assortment expansion. We find that the increase in the price sensitivity happens in product categories that are complements to the new one, and that these changes cannot be rationalized by alternative explanations, e.g., correlated preferences across product categories and changes in error variance. Based on the demand estimates, we conduct counterfactual simulations and show that the observed prices are consistent with retailers’ (biased) belief that the price sensitivity does not vary with assortment, which results in significant profit loss.

Qi Yu, Ron Berman, Eric Bradlow, Pricing Strategy Post Assortment Expansion.

Ron Berman, Leonid Pekelis, Aisling Scott, Christophe Van den Bulte, p-Hacking and False Discovery in A/B Testing.