Barbara Mellers

Pavel Atanasov, Jens Witkowski, Barbara Mellers, Philip Tetlock (2024), Crowd prediction systems: Markets, polls, and elite forecasters, International Journal of Forecasting.

Abstract: What systems should we use to elicit and aggregate judgmental forecasts? Who should be asked to make such forecasts? We address these questions by assessing two widely-used crowd prediction systems: prediction markets and prediction polls. Our main test compares a prediction market against team-based prediction polls, using data from a large, multi-year forecasting competition. Each of these two systems uses inputs from either a large, sub-elite or a small, elite crowd. We find that small, elite crowds outperform larger ones, whereas the two systems are statistically tied. In addition to this main research question, we examine two complementary questions. First, we compare two market structures, continuous double auction (CDA) markets and logarithmic market scoring rule (LMSR) markets, and find that the LMSR market produces more accurate forecasts than the CDA market, especially on low-activity questions. Second, given the importance of elite forecasters, we compare the talent-spotting properties of the two systems, and find that markets and polls are equally effective at identifying elite forecasters. Overall, the performance benefits of "superforecasting" hold across systems. Managers should move towards identifying and deploying small, select crowds to maximize forecasting performance.

Barbara Mellers and Siyuan Yin (2023), Reference-Point Theory: An Account of Individual Differences in Risk Preferences, Perspectives on Psychological Science. https://doi.org/10.1177/17456916231190393

Abstract: We propose an account of individual differences in risk preferences called “reference-point theory” for choices between sure things and gambles. Like most descriptive theories of risky choice, preferences depend on two drivers—hedonic sensitivities to change and beliefs about risk. But unlike most theories, these drivers are estimated from judged feelings about choice options and gamble outcomes. Furthermore, the reference point is assumed to be the less risky option (i.e., sure thing). Loss aversion (greater impact of negative change than positive change) and pessimism (belief the worst outcome is likelier) predict risk aversion. Gain seeking (greater impact of positive change than negative change and optimism (belief the best outcome is likelier) predict risk seeking. But other combinations of hedonic sensitivities and beliefs are possible, and they also predict risk preferences. Finally, feelings about the reference point predict hedonic sensitivities. When decision makers feel good about the reference point, they are frequently loss averse. When they feel bad about it, they are often gain seeking. Three studies show that feelings about reference points, feelings about options and feelings about outcomes predict risky choice and help explain why individuals differ in their risk preferences.

Pavel Atanasov, J. Witkowski, Barbara Mellers, Philip Tetlock (Under Review), The person-situation debate revisited: Forecasting skill matters more than elicitation method.

Philip Tetlock, Lu Yunzi, Barbara Mellers (2022), False Dichotomy Alert: Improving Subjective-Probability Estimates vs. Raising Awareness of Systemic Risk, International Journal of Forecasting.

E. Karger, J.T Monrad, Barbara Mellers, Philip Tetlock (Under Review), Reciprocal scoring: A method for forecasting unanswerable questions.

Katherine L. Milkman, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Predrag Pandiloski, Yeji Park, Aneesh Rai, Max Bazerman, John Beshears, Lauri Bonacorsi, Colin Camerer, Edward Chang, Gretchen B. Chapman, Robert Cialdini, Hengchen Dai, Lauren Eskreis-Winkler, Ayelet Fishbach, James J. Gross, Samantha Horn, Alexa Hubbard, Steven J. Jones, Dean Karlan, Tim Kautz, Erika Kirgios, Joowon Klusowski, Ariella Kristal, Rahul Ladhania, George Loewenstein, Jens Ludwig, Barbara Mellers, Sendhil Mullainathan, Silvia Saccardo, Jann Spiess, Gaurav Suri, Joachim H. Talloen, Jamie Taxer, Yaacov Trope, Lyle Ungar, Kevin Volpp, Ashley Whillans, Jonathan Zinman, Angela Duckworth (2021), Megastudies Improve the Impact of Applied Behavioural Science, , 600 (), pp. 478-483.

Abstract: Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens’ decisions and outcomes. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy—a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.

Ike Silver, Barbara Mellers, Philip Tetlock (2021), Wise teamwork: Collective confidence calibration predicts the effectiveness of group discussion, Journal of Experimental Social Psychology .

Siyuan Yin, H. Arkes, John McCoy, Morris A. Cohen, Barbara Mellers (2021), Conflicting Goals Influence Physicians’ Expressed Beliefs to Patients and Colleagues, Medical Decision Making, 41 (5), pp. 505-514. https://doi.org/10.1177/0272989X211001841

Abstract: Background Physicians who communicate their prognostic beliefs to patients must balance candor against other competing goals, such as preserving hope, acknowledging the uncertainty of medicine, or motivating patients to follow their treatment regimes. Objective To explore possible differences between the beliefs physicians report as their own and those they express to patients and colleagues. Design An online panel of 398 specialists in internal medicine who completed their medical degrees and practiced in the United States provided their estimated diagnostic accuracy and prognostic assessments for a randomly assigned case. In addition, they reported the diagnostic and prognostic assessments they would report to patients and colleagues more generally. Physicians answered questions about how and why their own beliefs differed from their expressed beliefs to patients and colleagues in the specific case and more generally in their practice. Results When discussing beliefs about prognoses to patients and colleagues, most physicians expressed beliefs that differed from their own beliefs. Physicians were more likely to express greater optimism when talking to patients about poor prognoses than good prognoses. Physicians were also more likely to express greater uncertainty to patients when prognoses were poor than when they were good. The most common reasons for the differences between physicians’ own beliefs and their expressed beliefs were preserving hope and acknowledging the inherent uncertainty of medicine. Conclusion To balance candor against other communicative goals, physicians tended to express beliefs that were more optimistic and contained greater uncertainty than the beliefs they said were their own, especially in discussions with patients whose prognoses were poor.

Ville Satopää, Marat Salikhov, Philip Tetlock, Barbara Mellers (2021), Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model, International Journal of Forecasting.

Christopher Karvetski, Carolyn Meinel, Daniel Maxwell, Lu Yunzi, Barbara Mellers, Philip Tetlock (2021), Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales, International Journal of Forecasting.