I’m So Sorry For Psychology’s Loss, Whatever It Is

Here are some recent extraordinary events:

The bloggers at Data Colada published a four-part series (1, 2, 3, 4) alleging fraud in papers co-authored by Harvard Business School professor Francesca Gino. She responded by suing both them and Harvard for $25 million.

Earlier, the Colada boys had found evidence of fraud in a paper co-authored by Duke professor Dan Ariely. The real juicy bit? There’s a paper written by both Ariely and Gino in which they might have independently faked the data for two separate studies in the same article. Oh, and the paper is about dishonesty.

Also, there's this gem:

(Both Ariely and Gino deny any wrongdoing. Since we're now in the business of suing blogs, let me state that I, of course, have no idea if Ariely, Gino, or anybody else ever engaged in research misconduct. There's no evidence that I have any ideas at all! I'm just a bunch of bees!)

Gino's coauthors are scrambling to either find out if their data is solid, or to assure others that it is. She has students who are trying to get jobs right now; God help them. Ariely still has his job, but he runs a big lab, is involved in multiple companies, and collaborates with a lot of people, so if he eventually does go down, he'll take a lot of people with him.

All of that is bad. But there's an extra uncomfortable fact that nobody seems to mention, perhaps because they don't see it, or perhaps because they don't want it to be true.

This whole debacle matters a lot socially: careers ruined, reputations in tatters, lawsuits flying. But strangely, it doesn't seem to matter much scientifically. That is, our understanding of psychology remains unchanged. If you think of psychology as a forest, we haven't felled a tree or even broken a branch. We've lost a few apples.

That might sound like a dunk on Gino and Ariely, or like a claim about how experimental psychology is wonderfully robust. It is, unfortunately, neither. It is actually a terrifying fact that you can reveal whole swaths of a scientific field to be fraudulent and it doesn't make a difference. It's also a chance to see exactly what's gone wrong in psychology, and maybe how we can put it right.

It's a Wonderful, Fraudulent Life

Gino's work has been cited over 33,000 times, and Ariely's work has been cited over 66,000 times. They both got tenured professorships at elite universities. They wrote books, some of which became bestsellers. They gave big TED talks and lots of people watched them. By every conventional metric of success, these folks were killing it.

Now let's imagine every allegation of fraud is true, and everything Ariely and Gino ever did gets removed from the scientific record, It's a Wonderful Life-style. (We are, I can't stress this enough, imagining this. Buzz buzz, I’m bees.) What would change?

Not much. Let's start with Ariely. He's famous for his work on irrationality, which you could charitably summarize as “humans deviate from the rules of rationality in predictable ways,” or you could uncharitably summarize as “humans r pretty dumb lol.” He's a great popularizer of this research because he has a knack for doing meme-able studies, like one where, uh, men reported their sexual preferences while jerking off. But psychologists have been producing studies where humans deviate from the rules of rationality for 50 years. We've piled up hundreds of heuristics, biases, illusions, effects, and paradoxes, and if you scooped out Ariely's portion of the pile, it would still be a giant pile. A world without him is scientifically a very similar world to the one we have now.

Same goes for Gino. Much of her work is also part of the big pile of cognitive biases, and, just like Ariely, that pile would be huge with or without her. For the rest, you can judge for yourself the four studies that were recently retracted:

• Participants said they wanted cleaning products more after they were forced to argue against something they believed (vs. arguing for the thing they believed).

• Participants either wrote about 1) a duty or obligation, 2) a hope or aspiration, or 3) their usual evening activities. Then they imagined networking at a corporate event. The people who wrote about the duty or obligation said they felt more “dirty, tainted, inauthentic, ashamed, wrong, unnatural, impure” while imagining the networking event than people who wrote about their hopes/aspirations or their evening activities.

• Participants who were given the opportunity to lie about the outcome of a coin toss (they could get more money if they lied), and who did indeed lie, later came up with more uses for a newspaper in 1 minute.

• Participants completed as many math problems as they could in 1 minute, and they could lie about how many they got right (they could get more money if they lied). Then they filled out a form where they reported how much time and money they spent coming to the lab, for which they were compensated up to a certain amount (here they could also get more money if they lied). Some participants signed at the top of the form, and some signed at the bottom. The participants who signed at the bottom lied more than the participants who signed at the top.

(I'm describing these studies in experimental history terms—as in, people doing things. The authors described these results as “inauthenticity causes feelings of impurity” and “dishonesty leads to creativity” and “signing makes ethics salient.” See what a difference it makes to talk about people and the things they did!)

Looking over the rest of Gino's papers, these studies seem like pretty standard examples of her research. I'll only speak for myself here: if I found out that every single one of these studies had been nothing more than Gino running create_fake_data.exe on her computer over and over again, I wouldn't believe anything different about the human mind than I already believe now. (...)

As a young psychologist, this chills me to my bones. Apparently is possible to reach the stratosphere of scientific achievement, to publish over and over again in “high impact” journals, to rack up tens of thousands of citations, and for none of it to matter. Every marker of success, the things that are supposed to tell you that you're on the right track, that you're making a real contribution to science—they might mean nothing at all. So, uh, what exactly am I doing?

I'm So Sorry For Your Loss, Whatever It Is

But hey, these are just three people, albeit three pretty famous people. Maybe the impact of any single scientist is simply too small to be seen from a distance. If you deleted a whole bunch of papers from across the literature, though, that would really make a difference, and we’d have to rebuild big parts of the field from the ground up. Right?

No, not really. We did delete those papers, and nothing much happened. In 2015, a big team of researchers tried to redo 100 psychology studies, and about 60% failed to replicate. This finding made big waves and headlines, and it's already been cited nearly 8,000 times.

But the next time someone brings it up, ask them to name as many of the 100 studies as they can. My bet is they top out at zero. I'm basically at zero myself, and I've written about that study at length. (I asked a few of my colleagues in case I'm just uniquely stupid, and their answers were: 0, 0, 0, 0, 1, and 3.)

This is really weird. Imagine if someone told you that 60% of your loved ones had died in a plane crash. Your first reaction might be disbelief and horror—“Why were 60% of my loved ones on the same plane? Were they all hanging out without me?”—but then you would want to know who died. Because that really matters! The people you love are not interchangeable! Was it your mom, your best friend, or what? It would be insane to only remember the 60% statistic and then, whenever someone asked you who died in that horrible plane crash, respond, “Hmm, you know, I never really looked into it. Maybe, um, Uncle Fred? Or my friend Clarissa? It was definitely 60% of my loved ones, though, whoever it was.”

So if you hear that 60% of papers in your field don’t replicate, shouldn't you care a lot about which ones? Why didn't my colleagues and I immediately open up that paper's supplement, click on the 100 links, and check whether any of our most beloved findings died? The answer has to be, “We just didn't think it was an important thing to do.” We heard about the plane crash and we didn't even bother to check the list of casualties. What a damning indictment of our field!

by Adam Mastroianni, Experimental History |  Read more:
Image: author's father; Francesco Gino and Dan Ariely

[ed. As true as ever. Probably related in part to a 'publish or perish' mindset and the need to continually maintain relevance and expertise in your field of study. ]