Evidence of Heterogeneity in the Direction and Magnitude of Narrative Effects on Transportation and Counterarguing Across Three Independent Samples

Abstract

We report on 3 studies that reexamine two well-studied mechanisms of narrative persuasion, transportation and counterarguing, with two secondary data analyses and a preregistered follow-up experiment featuring identical messages and measures. This project employed internal replication, an undervalued method in communication research, to explore the stability of findings across three samples—a large national sample of U.S. adults (Study 1, secondary data analysis), a sample of U.S. state legislators (Study 2, secondary data analysis), and a convenience sample from Prolific, an online crowdsourcing platform (Study 3, preregistered follow-up study). Results show substantial evidence of treatment effect heterogeneity: a narrative message produced significantly less transportation than a nonnarrative message in Study 1, but the same narrative produced significantly more transportation in Study 3. The narrative also produced more counterarguing in Study 1 and Study 2, but not in Study 3. Indirect effects models revealed different patterns across the 3 studies. Such heterogeneity of the exact same treatment across different samples and populations highlights the need to qualify broad claims of generalizability of seemingly well-established narrative mechanisms.