Why We Can't Agree on What Makes a Good Movie

A recent study by NYU psychology professor Pascal Wallisch found that taste in films is highly idiosyncratic—and that critics and audiences don't seem to like the same things.
illustration: a hand plucking the fifth star from a line of stars

“Did we even see the same movie?”

If you’ve ever uttered that exasperated line, you know how deflating it can be to leave the theater gushing, only to have your enthusiasm punctured by the curmudgeon you were just sharing popcorn with.

And as this year’s summer blockbuster lineup shapes up, promising the usual mix of sequels, superhero flicks, and animated kid-pleasers, we can (most likely) look forward to yet another season of passionate arguments about what constitutes a good film. Is there anything more alienating than hearing the whole office buzzing about—or opening the newspaper to a rave review of—something you yawned through? 

That thing we call taste—it’s deeply personal, and even more idiosyncratic than we may have thought, at least according to new research by NYU psychology professor Pascal Wallisch and Rutgers University’s Jake Whritner, a graduate of Tisch’s Cinema Studies Program and Wallisch’s former student. The pair surveyed 3,204 ordinary viewers about 209 major motion pictures released between 1985 and 2004, asking them to rank movies they’d seen on a four-star system, and then compared those to existing rankings of those same films by professional critics from 42 sources.

They found that there was only a very modest correlation in movie preferences among the study participants, suggesting that their tastes were highly idiosyncratic. In contrast, agreement among critics was significantly higher than among the study participants. However, the correlation between critics and non-critics was no higher than among the study participants themselves.

In other words, if you were looking for someone to recommend a movie you'd love, instead of consulting a professional critic’s review, you would do just as well asking a random person on the street. Or, as the study’s authors put it in their paper, “Ironically, something about being a critic seems to make the recommendations of critics unsuitable—relatively speaking—for predicting the movie taste of regular people.”

Coming on the heels of last year’s what-is-the-role-of-the-critic kerfuffle over Batman vs. Superman—wherein much was written about the fact that critics almost universally panned a major box-office hit—that conclusion has the potential to ruffle some feathers. (In response to the study, some have already leapt to critics' defense.)

But the authors acknowledge that there could be any number of explanations for the gulf between professional assessments and popular taste: Perhaps critics approach movies with a different aim from that of many moviegoers, or maybe their opinions are synchronized because they share similar training and therefore beliefs about art.

And Wallisch insists that the point of the study wasn’t as much to take aim at the film theory crowd as much as it was to explore the mystery of how human brains create the experience of reality. In fact, he’s a movie buff and lifelong Roger Ebert fan who, out of respect for the late critic, deliberately delayed the publication of these results until well after Ebert’s death in 2013. Perhaps Ebert was beloved not because his thumbs up or down ratings perfectly predicted those of his readers—this study would seem to make that notion unlikely—but rather because audiences appreciated his writing and his insights, Wallisch suggests.

Wallisch studies visual perception—his psychological analysis of why people saw “the dress” differently was featured everywhere from Slate to the New York Post—but this foray into film represents a new approach for him. In vision science, he explains, “we usually present our participants with dots or bars and ask questions like, ‘Is this bar oriented vertically, or is it a little off?’” While these carefully controlled experiments can help researchers zero in on how the brain responds to simple visual cues, it’s difficult to determine what these findings say about people’s perceptions of the complex world around them.

Asking people about movies, Wallisch realized, might be a way to see how people respond to a stimulus that has more in common with real life than rectangles on a screen. “Movies are like a piece of virtual reality I can deploy to study your reaction,” Wallisch says. “Let’s say we both watch the same movie. We experience the same situation. But do we even see the same thing?”

The answer, according to previous experiments, is yes: The eye movements of different people watching the same commercially produced movie are synchronized, as is brain activity at the back of the brain in the areas involved in perceptual processing.

But activity in large regions of the prefrontal cortex doesn’t synchronize between movie viewers, which could be a clue as to why so many of us have had the experience of hating a film our date loved.

“If it’s the same movie and our brain is acting in roughly the same way, why do we disagree so strongly on whether or not we liked it?” Wallisch asks. “If the difference isn’t coming from the movie itself, it must tap into something in your personality—your world view, what makes you laugh, what makes you cry, what you find offensive, what upsets you.”

In Wallisch’s recent study, men and women didn’t show systematic differences in movie tastes, and neither did different age groups (though the sample skewed young in general, with an average age of 25.14 years)—a finding that might surprise marketers who deliberately target specific demographic groups for various films. But Wallisch says that this is in keeping with what he’s seen in other studies of perception.

“People focus on demographic groups because that kind of information is easy to ascertain,” he says. “But it doesn’t tell you the whole story. People are much more complicated than that.”

A savvier approach might be to slice potential audiences into “taste clusters,” or groups that have rated movies similarly in the past, regardless of what else they may or may not have in common. Netflix has experimented with a similar strategy in the algorithm for its personalized recommendations, Wallisch says, but the accuracy of the predictions depends on having people rank a large number of movies. In most settings, viewers are more likely to weigh in only when they hated or loved something, which is why a survey like Wallisch’s—which, by forcing participants to rank everything they’ve seen, captures the middle, too—is so valuable.

Though there was very little overall agreement among movie-watchers in the study—ratings for any given film varied by an average of 1.25 stars—the correlation between ratings was highest for people who had seen about half the movies in the sample, which the researchers suggest might “represent the average taste of the casual moviegoer in the United States.” And there were also a few “superpredictors,” or individuals whose ratings were more likely to sync up with others. Because these ordinary viewers were so accurate in their predictions of audiences' taste, Wallisch suggests that perhaps they—not professional critics—should be the ones making movie recommendations. (No superpredictors were found among the professional critics.)

“Imagine if these people who know what audiences actually like could be brought on as Hollywood consultants,” Wallisch says. “You could bring them in in the first step of the movie making process, and know what they like is what many people would like.”

Beyond the commercial potential, though, the study raises a lot of sticky questions that Wallisch and his colleagues plan to investigate with further research. If people can’t agree on the quality of a movie like The Shawshank Redemption, what else might they disagree on? And could entertainment preferences be a predictor of how well people get along—a possible index for romantic matchmaking or professional teambuilding? Do certain types of educational experiences push people to agree more?

“Taste is complex,” Wallisch marvels. There’s a lot to explore.

In one follow-up study, for example, he’s taking a closer look at horror movie fans to see if there are any personality traits—such as extroversion, introversion, neuroticism, conscientiousness, or openness to experiences—that unite them. Is it that people who like scary movies don’t find them frightening? Or that there’s something about the fear that they find enjoyable?

Wallisch believes that this research could get at something essential about people’s core beliefs, illuminating profound differences in how people—even those who appear to share many traits—see the world. He notes that when people argue about whether or not a movie was good, they behave as though they’re debating a matter of objective fact.

“In my opinion, that has profound implications for political discourse,” Wallisch says. “We somehow have to find a way to live with each other in this country and in this world. And to do that we need to respect and take seriously that people have real differences in values.”

Investigating what’s behind a a simple thumbs up or thumbs down rating could be a first step to mapping out those differences.

“What you like goes to the heart of who you are,” Wallisch says.