The Ghost in the Machine
A small team at NYU Tandon’s Media and Games Network (MAGNET) is working on the first-ever production of "Hamlet" in the virtual world
There’s an urban legend among cinema cognoscenti that one of the first films ever made— all 50 seconds of a steam train moving towards the camera as it pulls into a station in the Lumière brothers’ 1895 classic, L’arrivée d’un train en gare de La Ciotat— caused such shock and disorientation that members of the audience screamed and ran toward the back of the room. Some even ran from the building. Incredibly, at least to our modern sensibilities, the audience was so unfamiliar with the medium, they instinctively ran away from what felt like an actual oncoming train.
A little over a century later, film audiences had similar visceral reactions to 3D cinema: gasping and ducking as webbed-winged creatures appeared to swoop dangerously close in Avatar. And with the advent of Virtual Reality (VR), the motion sickness and disorientation the medium sometimes causes is mitigated by the marvel of trekking through otherworldly jungles and desertscapes, often in the company of fellow virtual adventurers from across the world.
Theater of the Future, Now
Now, we are approaching yet another technological frontier. Javier Molina a young engineer who specializes in VR and motion capture—the process used in computer-generated images (CGI) filmmaking that enables such scenes as Harry Potter riding through the air astride a dragon—believes we will begin to see a form of theater that combines the immersive, interactive elements of gaming, the spectacle of film and the communal experience of live performance; a mash up that, initially, may startle, thrill and disorient us in new ways. In addition to being a faculty member at NYU’s Tandon School of Engineering, Molina is spearheading an initiative at the school’s Media and Games Network (MAGNET) to turn this idea into a reality, using none other than Hamlet, one of the most famous plays in the English language. While Shakespeare’s timeless tragedy has been reimagined in countless ways—from school halls to London’s Globe Theater to a Disney variation featuring lions in the Serengeti—To Be With Hamlet will be the first time the tale of madness and betrayal in Elsinore Castle will be performed live in virtual space.
Technology Serving Creativity
The team of engineers, designers and technicians uses gaming technology— specifically the commonly used animation tool, Ikinema, and the motion capture program, OptiTrack—to recreate the immediacy of theater in a virtual space. In their studio space at 2 Metrotech Center in downtown Brooklyn, 16 little cameras – about the size of CCTV cameras – are suspended from the walls, and capture the shape of the actor’s body by photographing him at 240 frames per second, creating a virtual “skeleton.” The actors must wear skintight suits, which ensure that nothing on the surface of their bodies interferes with the motion capture technology. A 3D body scan—created by photographing the actor from more than 250 angles under different lighting conditions—is then applied to create a photorealistic avatar. A virtual costume (complete with rustling material) is designed and added to the image. Finally, responsive motion capture technology allows the actor to perform live in the virtual dimension. The ambitious production will allow up to 15 audience members from remote locations to “attend” the performance through VR headsets. As the actors appear in front of this audience in an alternate dimension as Prince Hamlet of Denmark and the vengeful Ghost of his betrayed father, in reality they are in a modest, soundproofed studio at NYU Tandon, performing their lines in dark, skin-hugging bodysuits.
“I believe this is the future of theater,” says Molina. “We are embracing new technologies that are not yet perfect—that’s the reason why VR is not yet in everyone’s home. It’s still very expensive to run, and there isn’t a mainstream flow of hardware and software for it yet. But in 12 years it will be a lot more common..”
Accommodating the Space-Time Continuum
At the moment, the team is focusing on Act I: Scene V, in which Hamlet is visited by a 15-foot-tall apparition in full armor. Actor Roger Casey relishes the role.“As the Ghost I can disappear. We have a lot of fun. Instead of laying down during a break, we may experiment with different ways of stretching the tech—flying, lightning, fire for no reason, dance battles,” he says, noting that they've already tested versions of the scene using feedback from volunteer audience members. “We’re putting the audience on this parapet of a castle, high above the ocean in a barren place with sort of a foreboding atmosphere,” says the director David Gochfeld. “I believe that can give us a sense of identification with the character and an understanding of the story that would be difficult to access otherwise.”
But while the virtual world this team creates may be subject to their every whim and fancy, being on the technological cutting edge presents some very real obstacles in the physical world. As the director, Gochfeld must spend his time between the virtual space, which is what the audience perceives, and the physical space, which is where the actors are rehearsing. This existence in two realities has unique challenges. For example, the actors cannot wear VR headsets while they perform, as they would be unable to see their physical rehearsal space. “There’s a long cord involved that I might trip over as I’m performing,” Casey adds. At this point, the team has yet to make use of facial capture, and even though it will be a part of the final production, Casey explains that the actors would nevertheless have to use their bodies and faces in very specific ways to communicate emotion in this new medium. “It feels to me like an intersection between the bold body language of theater and the subtlety of film acting,” he says, “because your audience members might also be standing very close to you.” The actors are also unable to see the VR audience. The only way they can see what the spectators are experiencing and where each member of the audience is standing— each of whom is represented, in the virtual realm, by a floating pair of hands—is by watching themselves on a nearby monitor. To ensure that the actors don’t appear to be walking through walls or off castle parapets, simple props are used on set to represent objects that exist in the virtual world.
Casey added that working with a remote audience can be a challenge as well. “As an actor and musician I am used to feeding off the energy of a live audience,” he says. For this reason, says director Gochfeld, it’s important both for the actors to be able to see the audience, and for the audience members to see each other without interrupting the immersive quality of the experience. “We have to come up with a representation of other audience members that makes sense within the performance experience—some kind of ghostly apparition perhaps. One of the things people are concerned about with VR is that it can be very isolating if everybody is in their own headset and just seeing their own virtual world. Where a lot of people seem to get really excited about VR experiences is when they are shared. So how do we keep that aliveness of the actors for the audience, and how do we keep the actors aware that there is an audience there that they are performing for, as opposed to just performing in a vacuum?”
Audience Participation Drives the Element of Chance
Despite these challenges, Gochfeld believes that VR is more compatible with immersive theater—which relies on an element of chance—than it is with film. “Three-hundred-and-sixty [degree] filmmaking often fails because people are approaching it with the language of filmmaking, which means working within a frame and being able to cut and direct focus in very specific ways. You can’t do that in 360, where the audience chooses what they’re going to be looking at.” According to Gochfeld, the techniques of gaming, rather than film, are more useful to their project. “I think it’s pretty natural that when you feel like you’re in a place, you want to be able to explore it and interact with it in some way. So applying game techniques is important to story-telling in virtual reality,” he says.
To Be With Hamlet was inspired, in part, by Janet Murray’s book, Hamlet and the Holodeck, her 1998 study of the influence of interactive digital media on the art of story-telling. “It’s about how we think in 3D,” said Molina, “how we play, and how we move around.” Murray analyzes why the ability to participate in a story is so satisfying, and how the features of digital media may be harnessed to enhance narrative rather than overwhelm it. But the need to keep the audience engaged in the story is also part of the challenge of a live-action VR production. At the moment, the audience for To Be With Hamlet is able to walk around and explore the set at will. “They can walk away far enough so that they’re really not where the scene is happening,” said Gochfeld. “This presents some questions in terms of the story-telling: if you allow your audience to walk away, are you defeating yourself as a story-teller? We have to work on making the events of the story compelling enough so that they want to stay and watch them. Another thing we’re working on is how to build the world in such a way that it helps the story, in a way that contributes to the story-telling. If we could have clues to the story placed elsewhere around the castle, they could gain some of the narrative if they choose to walk away, and then maybe catch up with the action elsewhere.”
"Instead of running away from doing anything that advances our art and advances our brains, we should embrace it."
“I think it’s a marker of how our brains have developed – we require more to remain stimulated,” says Casey. “Theater purists might denounce this, but throughout history every generation has used the tools of their day to move theater forward.” Casey suggests that, in theater, VR might be compared with “projection mapping,” a technique that turns the surfaces of a set into a screen on which video or images of landscapes and props are projected. “Both are evolutions of the interactive visual medium in the theatrical space. Instead of running away from doing anything that advances our art and advances our brains, we should embrace it,” he said. For Casey, the power of VR also lies in its potential for empathy-building. “I once worked on a VR piece where I played a slave with my white scene partner as the slave master, and then we switched so that I was playing the slave master, and he was playing the slave. We then had great a discussion about what all of that felt like and what it meant, and how to apply that experience to our lives," he says. "We have to realize the enormous power we have here, in raising relatability in fiction.” Casey added that it is even more important given the current political landscape, where people of differing viewpoints are becoming more and more polarized. “We’re going to a place right now where we can use technology to affect that. What could be more powerful than experiencing something with your own eyes, ears and heart?”
Stepping In: Experiencing VR Firsthand
The team at MAGNET allowed me to experience a demo of a version of Act I: Scene V recorded in VR. Gochfeld pointed out it was “not quite the same,” since the acting wouldn’t be live, and I would be experiencing it alone as opposed to with other audience members. Nevertheless, I would get a good sense of the experience.
Gochfeld eased me into the headset and gave me two cordless handles, each of which had a button. These would be my controls. The feeling of calm that washed over me was first thing I noticed was as I looked up into the night sky—it was clear, covered in thousands of stars, with a slight rosiness on the horizon which suggested it was very early in the morning. The next thing I noticed was the ocean and the sounds of the waves. How good to get out from between the buildings of New York! I looked around to orientate myself, and saw that I was on a walled parapet of a stone Medieval-looking castle, surrounded by mountains and craggy cliffs. In front of a set of steps stood a blond avatar of Hamlet. The masked Ghost, who towered above me in a full set of armor, had already appeared, and their dialogue was underway as Hamlet and I both craned our necks to see the Ghost. I looked down at my body and saw—with a flip-flop of my stomach—a floating pair of hands and nothing else. When I pointed my hands in a particular direction, a blue arc would form in front of me. “How do I teleport?” I asked Molina as I pressed the button. “You’re doing it,” I heard him say, and noticed with a shock that I had jumped a few meters ahead and was standing right on the edge of the parapet, gazing over the low wall at the menacing ocean below. As I leaned forward to get a closer look, a red grid wall appeared in front of me. “What is this grid doing in front of me?” I asked Molina. “It’s to keep you safe,” he said. Was it preventing me from falling into the ocean? I wondered. Afterwards, I learned what he meant; the grid would appear when I was about to bump into an object in my physical environment. Of course, I had forgotten all about the real world.
I walked up the stairs and poked around the castle. I watched the acting from that vantage point, which was a bit like watching it in an amphitheater. Then I went back downstairs and watched the actors up close. At the closing of the scene, the Ghost climbed onto the low wall, having just told Hamlet to avenge his honor. I looked at Hamlet to see his reaction, and when I turned back to the Ghost, he had disappeared. Suddenly it was just me and Hamlet, alone in the early dawn. I felt what Hamlet must have been thinking: Was it all a dream? Am I losing my mind?