Reality 3.0 Finds Its Voice
Mark Skwarek on Charting the Territory Where Artificial Intelligence, Virtual Reality and Augmented Reality Converge
It’s been a year since we last checked in with Mark Skwarek lecturer of integrated digital media at the Tandon School of Engineering. A lot has happened in his world, which spans the realms of artificial intelligence, augmented reality (AR) and virtual reality (VR).
After returning from a conference in China and the Consumer Electronics show this spring, he characterizes the movement toward an integrated-reality culture as in a kind of toddler stage. “I think people are learning the vocabulary right now,” he said, likening where integrated digital media is currently to the early days of movies in which all the sets still looked like theater. “Cinema had so much more potential.”
What evolution have you seen in the past year for AR and VR?
Mark Skwarek: We’re starting to see a few experiences with a real polish, and a lot more detailed crafting of the environment through sound, animations, interactions and a sense of presence. It’s the difference between feeling like you’re a floating eye in space, observing the world around you, and actually feeling like you’re there and could reach out and grab things and really feel them. If we want to make an experience of someone riding a bike in the desert, we’ll have them sitting on a real bike to help complete the reality.
What are you working on at Tandon right now? What’s exciting to you?
MS: We’re doing a lot around NYU's public safety. One idea is to give security workers headsets and AR glasses so they make the campus safer. They would be able to move people through spaces more quickly if there was some sort of an emergency using real-time navigation. If there were a fire, for example, we could steer people away from that area. People in the building would have an app that would show a line on the ground to show where to go, or alternatively, tell people which areas to avoid.
It seems like the ultimate application of this kind of technology is to save lives. It’s fun to play games and do cool things, but if you can use it for safety or medicine, that’s really of value.
MS: Recently one of my graduate students, backed by the NYU School of Medicine, created a startup project that allows surgeons to basically reconstruct CAT scans in layers so they can see the patient’s information in 3D. Microsoft HoloLens has joined them.
Another app we made was an exploration to Mars with Google Cardboard hardware—basically a phone placed into a visor you wear on your face. Cardboard essentially makes technology accessible to anybody with a smartphone. The students created a rover you can drive around on the real Mars, so we wouldn’t have to move the resources back and forth from Earth. The rover would go off and find the metal or the different materials they’d need, mark where it was and then colonizers could start to use resources to build up some sort of sustainable environment.
Another VR experience we did was showing growth of stem cells and how to help bones heal more quickly with mechanical intervention on the cellular level.
One of the things I’m most proud of that we’re working on is an app with virtual reality to create peace. Since I was a young artist, I always thought I could use these technologies to bring people together. The idea is to get people who don’t like each other to wear the head-mount displays and then interact. They’d work together to solve a problem in a safe space and hopefully come out with a lower level of hate, or potentially come away with a level of respect for each other. It’s an iterative process. One experience isn’t going to solve all problems, so a lot of the experiences that we create might become slightly tweaked or specific to the problem that we’re trying to take on. The one that we have right now involves two halves of the body. One person controls the upper half and one has the lower half. Then you put each half together and they have to plan how to do something, like climb a wall. It’s still under development right now, so it would be like a work of art and it would be a learning experience from watching people. User testing would be part of the artwork.
Have there been any surprises for you?
The thing that surprised me was how popular VR and AR have become. After Pokémon Go people started taking augmented reality seriously. Before that you would have people weren’t really interested in it, and saying it won’t work on a large scale. But if it’s done correctly it can really have a large impact across the general public. Another thing that’s sort of surprising and disappointing at the same time, in the West is the lack of adoption beyond this game.
In China, for example, they’re much hungrier for the technology than we are. They’re excited about the new, whereas people here think of these things as a bit more of a gimmick. The QR code is very much alive everywhere else except America. That’s significant because it shows people want to have a digital experience at a specific place.
What’s the future for AR and VR?
Some of the things would be extensions of the current work that’s going on. We’re working on telepresence—the idea is that people can connect in the same virtual room via something like a hologram, like Princess Leia from Star Wars.
I’m also a big advocate of accessibility, so anybody with a smartphone would be able to participate. The idea of making something that’s going to live in a lab doesn’t really get me excited. It would be something that has a real impact on society. So first responders, for example, can see a real-time 3D model of the problem and it could be annotated with software that helps them.
Basically we’re always working the newest and coolest technology in the lab here, and can pivot very quickly. Mobile is the largest market right now; if you can do something that works on mobile you could have an epic impact. Glasses are starting to become pretty exciting.
How is that going to be different culturally? Google Glass had an element of ridiculousness to it. People hated it.
When Google Glass came out, you would get beat up if you went into a bar with them on. But if you look at somebody wearing Oculus Rift, it’s ridiculous at a completely different level. Like I’m wearing a box on the front of my face and there are wires going all over the room. We’re not hearing as many jokes about that quite as much. The revolution and the experience are very good; they’re able to overlook the shortcomings.
Everybody’s on their phones constantly but the phone goes back into your pocket. In the future, when everybody’s got their glasses on, it will be long-term exposure. You’re going to see levels of addiction. We’ve created this magical kind of thing but you do need to have somebody who’s grounded in reality thinking about what’s going on and the larger effect it’s having on the general population.
It’s going to come to a point where it’ll be better to be inside a virtual experience than it will be in the real world.
It almost sounds like you may end up with a bunch of haves and have-nots. What about people that can’t afford it or don’t have access or the right infrastructure to support it?
To really do this right you’ve got to make a technology that everyone can have. Everybody who has a smartphone right now will probably have the ability to have a full VR headset in the future. It’ll trickle down to a point where they can make something that’s within everybody’s reach.
People [regardless of where there live] would be able to go to a first-class school, all in VR, and never travel anywhere. But it’s like they’re in another world the whole time. So it’s giving access to people who might not [otherwise] have access, a real advantage.