Abstract women looking up through a tube with waves

Researchers at the new Max Planck–NYU Center for Language, Music, and Emotion couple innovative methodologies with red-hot technology to understand how the brain responds to aesthetic experiences—and the mystery of what makes us human.

By Lindsy Van Gelder
Illustrations by Max Löffler

Many thinkers throughout history have feverishly held forth on the soul-stirring power of music and verse. Shakespeare famously called music “the food of love,” while Tolstoy said it was “the shorthand of emotion.” Charles Darwin declared, “If I had my life to live over again, I would have made a rule to read some poetry and listen to some music at least once a week.” According to Robert Frost, “Poetry is when an emotion has found its thought and the thought has found words.” Without music, Jane Austen wrote in Emma, “life would be a blank to me.” Even worse, said Friedrich Nietzsche, “life would be a mistake.” Plato agreed, opining that “music gives a soul to the universe, wings to the mind, flight to the imagination, and life to everything.” Keith Richards simply said, “It speaks in emotions, and if it’s in the bones, it’s in the bones.”

    And then there are the scientists who want to know how all that lovely lyrical sausage is made.

    The Max Planck–NYU Center for Language, Music, and Emotion (known as CLaME, pronounced like the airport baggage area) is housed in a rabbit warren of labs and offices in Arts and Science’s Psychology Building on the Washington Square campus. Its raison d’être, according to its website, is “connecting aesthetic and cognitive experiences.”

    The science of aesthetics—a group of fields and subfields variably known as empirical aesthetics, neuro-aesthetics, and the neuroscience of aesthetics—is highly cross-disciplinary, drawing on the work of neuroscientists, psychologists, musicians, musicologists, music historians, linguists, cultural anthropologists, computer scientists, artificial intelligence engineers, and more, says Catherine Hartley, an Arts and Science assistant professor of psychology and codirector of CLaME. But she and David Poeppel, the center’s other codirector and an Arts and Science professor of psychology and neural science, prefer to label themselves more broadly as cognitive scientists and cognitive neuroscientists who also work on questions of aesthetic experience.

    CLaME opened in March 2019 as a joint project between NYU and the Max Planck Society (MPS), the prestigious research organization based in Munich that, since its founding in 1948, has produced 18 Nobel Prize winners in physics, chemistry, and medicine. CLaME is one of 20 university scientific research partnerships around the world sponsored by MPS. Four of the university partners are American, and NYU is the first non–Ivy League institution to be so honored. (The other US partnerships are the Max Planck–Yale Center for Biodiversity Movement and Global Change, the Max Planck–Princeton Center for Plasma Physics, the Max Planck–Harvard Research Center for the Archaeoscience of the Ancient Mediterranean, and the Max Planck–Harvard Research Center for Quantum Optics.)

abstract illustration two faces kissing

    CLaME’s three research pillars—music, language, and emotion—“are things that people are good at and computers aren’t,” Poeppel notes. With the possible exception of outliers like Snowball, the boogying internet cockatoo, it is human minds that react emotionally to music and language, and therefore it is humans that CLaME studies.

    The beating heart of the center—or more precisely, its pulsing brain—is a room that would look a little like Dr. Frankenstein’s lightning-at-midnight laboratory if it weren’t for the cheery rocket-ship decor on the walls and the cot where participants recline as their brains are being scanned (some of them are children, hence the space motif).

    At the end of the cot is a large machine that test subjects stick their head into while they listen to stories, syllables, and other auditory input, from beeps and boops to Beethoven and the Beatles. This is the MEG, a magnetoencephalographer. Says Poeppel: “It’s a very big cousin to EEG,” or electroencephalogram. But of all such measurement devices in the researchers’ toolbox, MEG has perhaps the “highest sensitivity to human brain activity,” says Poeppel. It’s particularly helpful, he adds, at showing scientists “where in the brain the neuronal activity is coming from. For the kinds of phenomena we are pursuing, MEG is the ideal technology.”

    Although the utilization of computers, electrodes, and neuroimaging to study aesthetic experience is a relatively new sensation, interest in the topic of what we feel and why goes back to Aristotle, according to Poeppel—“we just have better machines now.” A pioneer closer to the present was Gustav Theodor Fechner, a 19th-century German philosopher, psychologist, and physicist. “He wrote a book in 1860—Elemente der Psychophysik—that is one of the foundations of modern psychology,” says Poeppel. “He said, let’s apply the rigorous principles of the quantitative sciences—math and science—to perception.”

    An expert on visual stimuli in particular, Fechner theorized formulas by which perception could be measured. At the time, this was a radical concept; prominent thinkers like Immanuel Kant insisted that the study of the mind was too subjective to qualify as a science. In 1876’s Vorschule der Ästhetik, Fechner further argued, Poeppel says, that “we should be using the rigorous methods of the natural sciences to study more interesting psychological phenomena [than perception], like looking at art, looking at landscapes, listening to music. He thought we should extend rigorous scientific methodology to aesthetic phenomena, and that, in some sense, is the birth of the field of empirical aesthetics.”

    Much recent scientific research on aesthetics has focused on the visual arts. (For example, several months before CLaME’s opening, the University of Pennsylvania inaugurated the Penn Center for Neuroaesthetics, whose mission is, according to its website, “investigating the nature and neural basis of beauty, art, design, and architecture.”) It’s not surprising that visual arts loom large, says Poeppel, given that “vision is by far the most studied domain in neuroscience—about 70 percent or so.”

    That crowding is one reason why two of CLaME’s chosen bailiwicks are music and language. Poeppel also stresses that while aesthetics are a dimension of CLaME’s turf, the center “also very much focuses on basic research on language, music, and emotion—that is to say, research that is not necessarily driven by aesthetic concerns.” The center sees itself as a clearinghouse for ideas on its three core subjects. It hosts conferences (including the 2019 Biennial Meeting of the Society for Music Perception and Cognition) and a lecture series (the titles of the three talks given since CLaME’s opening: “Musical Meaning within Super Semantics,” “Learning, Liking, and Learning to Like: Statistical Learning as a Guiding Principle of Musical Creativity and Emotion,” and “Towards Science of Interacting Minds”).

    So what are the great still-unanswered questions in the field? “Probably all of them,” Poeppel says with a laugh. One area that piques his own interest is the intersection of emotion, music, and memory. “You play a bar of music and it releases something, an unfolding in you. How the hell does that work? Like something you heard as a teenager, the first time you kissed someone. I can personally play entire Pink Floyd albums in my head from my first ginger hand-holding,” he says. “How is it that music is the kind of information that implants a memory in you that’s so compelling? That reaches into your heart and gut and yanks out a memory? It’s incredible.”

 

 

 

 

 

 

Although the utilization of computers, electrodes, and neuroimaging to study aesthetic experience is a relatively new sensation, interest in the topic of what we feel and why goes back to Aristotle.

Abstract illustration of chemical biology

    Postdoctoral research scientists are a vital part of CLaME. Keith Doelling is a Berkeley, California, native whose research focuses on the phenomenon of rhythms in the brain. “If a person is listening to a series of sentences or musical pieces, the brain starts to synchronize with the rhythms,” he explains. “That’s very cool, but the question is why. I ran a series of studies to show that it’s related directly to comprehension. The more you’re synchronizing, the better you understand what you’re listening to.” Doelling’s experiments also found that trained musicians are better synchronizers, implying, he says, that “there’s a natural base level and you can improve on it” with training.

    Claire Pelofi, originally from Montpellier, France, studies the different auditory responses of musicians and nonmusicians. In one experiment, she tested both groups as they listened to a sound known as a Shepard tone—formed by overlaying different tones separated by octaves that each rise or fall (depending on the nature of the Shepard tone in question) and repeat independently of one another. As a result, the sound seems to continuously get higher (or lower) but never really does. Shepard tones are the auditory equivalent of an optical illusion: as some of the tones rise in pitch, others drop (or vice versa), tricking the brain into hearing a waa-waa-waa sound that seems never to end. (Shepard tones are often used in movies to create tension, or to signify speed, as in the careening Batpod in The Dark Knight.) Test subjects were asked to compare the pitch shifts in two Shepard tones. But the testers deliberately used what they called ambiguous stimuli, because they knew that when the tones are separated by half an octave, judgment becomes much more difficult, and people pretty much do a mental coin toss on whether the pitch shift was up or down. The musicians in Pelofi’s sample had slower reaction times and less confidence in their assessments than nonmusicians, who blithely guessed away. Pelofi speculates that the musicians’ training to discern sounds in a complex setting, as in an orchestra playing, made them more aware of the ambiguity.

    Pablo Ripollés, who hails from Castellón, Spain, has been studying the role of reward in learning. “If something is rewarding, it’s going to release dopamine” in the brain, he says, and dopamine in turn enhances learning. “It’s a matter of arousal, whatever pumps you up,” Ripollés says. “There have been studies that show that if you give money to people, they learn better.” His work determined that learning new vocabulary words is so satisfying in and of itself that it alone promotes continued learning. In the test, “we would invent words,” he says, “like ‘The car had four blah-blahs.’ It could have been wheels, tires, or windows—you’re not sure. But then there would be a sentence, ‘The lock of the blah-blah was broken.’” The moment subjects grasped that “blah-blah” meant “door,” the dopamine kicked in. In this experiment, despite not learning a new word, simply solving the puzzle gave participants a rush.

    Ripollés also found that when dopamine is directly administered to subjects in what he calls a “pharmaceutical intervention,” it helps them learn. (Does that mean you should pop a dose before taking the SAT or GRE? He responds with an emphatic no. “If you take too much, it’s bad for you,” he warns, “and if you don’t have enough, you’re doing badly, too.” The sweet spot likely comes through natural interactions, he says.) Ripollés is currently studying the use of music as a dopamine catalyzer.

    CLaME’s website notes that it looks for researchers with “irreverent, innovative approaches.” Asked for an example, Hartley and Poeppel recounted when Ripollés and Pelofi had to set up an entire EEG lab at a conference last year in Telluride, Colorado—the geek version of a triathlon. “This event is held in a high school, and the best room we had was a kitchen,” recalls Ripollés. Their first move? “We had to unplug the refrigerator,” says Pelofi.

    It took four days for them to set up all the delicate equipment—cables, electrodes, amplifier—and assemble a lab capable of measuring a subject’s brain waves. They pull out a photo of Ripollés wearing a cap spiked with electrodes in the high school kitchen. Pelofi’s baby is crawling around in the foreground of the shot. “My baby took her very first step in this lab,” she notes.

    Hartley and Poeppel say that their role as administrators is to hire great researchers, provide them with maximum resources, and then get out of their way. The center’s seed money—half from MPS and half from NYU—will allow CLaME to flourish for five years; after that, they are expected to thrive with individual donations or institutional grants. “Ideally, if we do this right, after five years there will be so much cool research that we will be self-sustaining,” Poeppel says. But he and Hartley would like to start cranking up support sooner. “Science is hard and slow,” Poeppel says. “If you have a limited time horizon, you have to get things done that aren’t risky.” Adds Hartley: “Knowing there’s continuity is important, because it affects the kinds of problems you choose to work on. It’s tricky if there’s just a four-year range left. The methods we use—data collection, analysis of the data—take a long time. Add to that the process of trying to pin something down by doing a series of experiments.”

    To give their researchers the intellectual elbow room they need, the directors are hoping to find a benefactor to name CLaME after. “The John and Jane Smith Center would mean doing not two experiments, but 20,” Poeppel says. There will also be smaller naming opportunities for fellowships to sponsor the work of individual researchers, Hartley says. In New York City, she adds, “it takes about $100,000 to support someone for one year.”

 

 

 

 

Hartley and Poeppel say that their role as administrators is to hire great researchers, provide them with maximum resources, and then get out of their way. “Ideally, if we do this right, after five years there will be so much cool research that we will be self-sustaining.”

Abstract illustration of bioengineering

    There’s no shortage of ideas among the researchers about how they’d use any additional resources. “The center has a lot of opportunity to interface more directly with professional musicians, because it’s in Manhattan . . . near Juilliard, Carnegie Hall, Lincoln Center,” Doelling says. “There’s a lot to be learned from people who spend their days using music in all its varieties,” from comparing the responses of musicians with different styles to asking them whether they feel melancholic when playing a sad song, or if they simply worry about the technical details.

    Pelofi would use the funding to carry on a giant cross-cultural, multifaceted examination of music, including in remote rural communities in developing countries. “I would put the EEG lab in the middle of the forest,” she says. In particular, she’d explore the nature/nurture question as it applies to the emotional impact of music: “We know people can recognize emotion in a music that’s not [from] their own [culture], but that’s probably tied to acoustic features that also exist in prosody. Like, if you’re scared, you’re going to [here Pelofi switches to a rattled, high-pitched voice] speak like that. If you’re sad, you’ll speak in a low-pitched voice.”

    These features are also found in music, she adds, “but we don’t know exactly the tradeoff between cultural influences and bottom-line acoustic-driven phenomena—what’s learned and what’s innate.” The topic is especially relevant for Pelofi, a new mother, who recently learned that “one of the big universals in music is that, in all cultures, mothers or fathers sing lullabies at a certain rate frequency.”

    Ripollés is fascinated by the relationship between memory and music. He points out that people with dementia have the ability to hum and clap along with a song they remember when other mental functions are lost—although no one knows why. “One interesting idea that’s been tested mostly in rodents is that, when there’s a release in the hippocampus of dopamine—one of whose functions is to boost memory—it boosts the memory not only for what you are learning in that moment but also for, say, one hour afterwards,” Ripollés says. “And also before that—it’s retroactive, which is super-interesting. It’s well studied in rodents, less well in humans.” He would love to be able to conduct his experiments on a much larger scale, and on people—administering dopamine, testing their memories, and studying their brains using an MRI (magnetic resonance imaging) machine.

    As you might expect, music is an essential part of the non-work lives of the CLaME team members. Pelofi is a violinist, Poeppel plays acoustic guitar, and Doelling sings and is proficient on several instruments, including upright bass. Hartley and Ripollés are serious music fans. What Hartley listens to depends on whether she’s running (1990s hip-hop), cooking (Motown), or working (classical or ambient electronic), while Ripollés’s tastes run the gamut from Spanish punk to the strains of bagpipe.

    So what is it like to go to a concert or hear music on the radio when you’re an expert on the cortical signals and neural circuit dynamics that make the enjoyment possible? “There’s some postprocessing, and you may recognize some things about the experience that were particularly salient,” Hartley acknowledges. But if anything, awareness adds to the experience, Doelling says: “It’s a reflection of how complex and interesting our brains are that we have this thing—music—that isn’t strictly necessary for survival. It’s immensely complex, and yet we do it for fun. How cool is that?”

    Hartley points out that artificial intelligence is “trying to solve problems such as how can we recognize an emotion from a vocal recording, a text, or a video of people interacting? This is highly commercial stuff,” she says. “But these are things that humans are experts at from one year of age.”

    “Fresh out of the box,” Poeppel adds. “Even though we’re all computer-centric in our lifestyles, the experiences of how language works, how music works, and how emotion works are extremely compelling human experiences.” AI, as compelling and sexy as it is, “isn’t even close,” he declares. “These are fundamental human experiences. No matter how much data you collect and put in your machine, you’re not going to find that.”

 

 

 

 

“It’s a reflection of how complex and interesting our brains are that we have this thing—music—that isn’t strictly necessary for survival. It’s immensely complex, and yet we do it for fun. How cool is that?”
—Catherine Hartley

Abstract illustration for Quantum phenomena