Don’t put on the kettle just yet…

On my Twitter feed, I noticed some scientists complaining about the lack of citation in a recent Guardian column, “The psychology behind a nice cup of tea“, reporting that research finds “hot drinks warm our personalities as well as our bellies”. So I put on my detective cap, fired up Google, and found the original study in about 30 seconds with the search term “holding warm drinks study” (it was number 3 on the list). But because I’m a psychologist and am interested in the recent efforts to increase the replicability of psychological science, I also Googled “holding warm drinks replication”. There I found a pre-registered replication plan, which is a document that details exactly how a team of scientists plan to replicate an existing research finding and how they plan to analyse the results. That way, there can be little leeway to engage in what scientists call researcher degrees of freedom, or changing small things about the analysis until a proposed effect “works”. So how does the original study hold up?

What did the original studies find?

In the first study, 41 people held a cup of either hot or iced coffee for a few seconds during an elevator ride to the study room while the experimenter wrote their details on a clipboard. Once they got to the room, they were asked to read a description of a person who was “intelligent, skillful, industrious, determined, practical, and cautious” and then rate ten personality traits on a seven-point scale. Five of those traits were words related to warmth/coldness. The average ratings on those five traits were slightly higher among the people who had held the hot coffee (4.71 vs 4.25), while the ratings for the other five traits were not as different.

In a second study, 53 people held a hot or cold therapeutic pad “under the guise of a product evaluation”. After they finished evlauating the product, they were offered a reward for participation, either a Snapple drink or a $1 ice cream gift certificate. Some people were told the Snapple would be for themselves and the gift certificate could be a treat for a friend, while the others were told the Snapple could be for a friend and the gift certificate for themselves. The people who held a warm pad were more likely to choose the gift for a friend (54%) than were the people who held a cold pad (25%).

What did the pre-registered replication find?

The replication study only repeated the second study from the original paper. Although the first study was not included in the replication, there were statistical and methodological problems that made the second study a better choice for replication efforts. The researchers calculated that they would need about 300 participants to be relatively sure that they could detect effects a bit smaller than the one originally reported. They conducted the study three times, at three different universities (Kenyon College, Michigan State University, and the University of Manchester). One study retained the Snapple/ice cream choice, while the other two used different rewards (Snapple/cupcake or juice/smoothie).

None of the three replication studies showed that people who handled the warm pack were more likely to choose a reward for a friend. In fact, in each study the warm pack-holders were actually slightly more likely to choose the selfish reward. About 57% of people who tested the warm pack chose the selfish reward, compared to 51% of people who tested the cold pack, although this 6% difference was not quite large enough to be statistically significant.

What can we conclude?

First of all, let’s be clear about what we cannot conclude. We can’t conclude that the original study was in any way fraudulent. Given how null hypothesis testing works, we’d expect a lot of “statistically significant” findings to turn out to be false positives (David Colquhoun gives a really cogent explanation of this). This is why we need replications to be taken more seriously. Also, it’s hard to blame the original media story’s author*. Because this is in my field, I immediately thought to check for replications and knew exactly what words to search for and how to scan the list of thousands of results for the relevant evidence. This is a skill we should all practice as responsible consumers of media (and a skill we teach in our Rewrite the Headlines project).

What we can conclude is that holding something warm versus cold is unlikley to have a reliable effect on people’s generosity, but that doesn’t mean you shouldn’t offer someone a hot drink when they come in on a cold day.

References

Lawrence E. Williams, and John A. Bargh (2008). Experiencing Physical Warmth Promotes Interpersonal Warmth. Science, 322(5901): 606–607. doi:  10.1126/science.1162548

Dermot Lynott, Katherine S. Corker, Jessica Wortman, Louise Connell, M. Brent Donnellan, Richard E. Lucas, Kerry O’Brien (2014). Replication of “Experiencing Physical Warmth Promotes Interpersonal Warmth”. Social Psychology, 45, pp. 216-222. DOI: 10.1027/1864-9335/a000187