Experiments normally measure the average effect, but the effect can be different for different people.
The second key idea for moving beyond simple experiments is heterogeneity of treatment effects. The experiment of Schultz et al. (2007) powerfully illustrates how the same treatment can have different effects on different kinds of people (Figure 4.4), but this analysis of heterogeneity is actually quite unusual for an analog age experiment. Most analog age experiments involve a small number of participants that are treated as interchangeable “widgets” because little about them is known pre-treatment. In digital experiments, however, these data constraints are less common because researchers tend to have more participants and know more about them. In this different data environment, we can estimate heterogeneity of treatment effects in order to provide clues about how the treatment works, how it can be improved, and how it can be targeted to those mostly likely to benefit.
Two examples of heterogeneity of treatment effects in the context of social norms and energy use come from additional research on the Home Energy Reports. First, Allcott (2011) used the large sample size (600,000 households) to further split the sample and estimate the effect of the Home Energy Report by decile of pre-treatment energy usage. While Schultz et al. (2007) found differences between heavy and light users, Allcott (2011) found that there were also differences within the heavy and light user group. For example, the heaviest users (those in the top decile) reduced their energy usage twice as much as someone in the middle of the heavy user group (Figure 4.7). Further, estimating the effect by pre-treatment behavior also revealed that there was not a boomerang effect even for the lightest users (Figure 4.7).
In a related study, Costa and Kahn (2013) speculated that the effectiveness of the Home Energy Report could vary based on a participant’s political ideology and that the treatment might actually cause people with certain ideologies to increase their electricity use. In other words, they speculated that the Home Energy Reports might be creating a boomerang effect for some types of people. To assess this possibility, Costa and Kahn merged the Opower data with data purchased from a third-party aggregator that included information such as political party registration, donations to environment organizations, and household participation in renewable energy programs. With this merged dataset, Costa and Kahn found that the Home Energy Reports produced broadly similar effects for participants with different ideologies; there was no evidence that any group exhibited boomerang effects (Figure 4.8).
As these two examples illustrate, in the digital age, we can move from estimating average treatment effects to estimating the heterogeneity of treatment effects because we can have many more participants and we know more about those participants. Learning about heterogeneity of treatment effects can enable targeting of a treatment where it is most effective, provide facts that stimulate new theory development, and provide hints about a possible mechanism, the topic to which I now turn.