Spring 2005
Vol. 1 No. 2

Virtual Patient Technology to Help Students Recreate
Rare Disorders and Perform Complex Procedures

The virtual patient project will allow interaction with and simulation of many complex elements of the human body that was not previously possible.
By Richard Pierce

Researchers in the Interactive Telecommunications Program at the Tisch School of the Arts’ Kanbar Institute and NYU’s School of Medicine are creating a computer behavioral engine that will allow interaction with and simulation of many complex elements of the human body that was not previously possible. The virtual patient project uses computer game technology and medical visualization technology to illustrate changes in a body as opposed to just representing the body frozen at a moment in time.

Spearheading the research is Jean-Marc Gauthier, teacher in the Interactive Telecommunications Program, and Martin Nachbar, director of advanced educational systems at the School of Medicine.

In the past, virtual reality technology for professional medical training could only explore statically an internal organ structure of the human body much like an automotive manual uses an exploded-view illustration of the car engine to make its point. More recent advances combined virtual reality applications with visual and tactile interactions that made it possible to recreate rare disorders and perform delicate or dangerous procedures without using live patients.

“We want to be able to simulate the complex elements of the human body,” said Nachbar. “For example, the interactions between the physiologic systems and the rules governing them would be one area of interest. There is a connection between pain perception and the control of blood pressure. Severe sudden pain may cause someone to faint as excessive neuronal stimulation or discharge may lead to a dramatic slowing of the heart rate and in insufficient blood supply to the brain.

“The research is endeavoring to find a way that will allow a computer user to influence a system of relationships between organs instead of just viewing them as static models,” added Nachbar. “We want to go beyond just the anatomic representation of the body by attempting to understand the human body as a whole.”

According to researchers, the main contribution of this exploration is the behavioral engine inspired by game-play techniques that deeply engage players in rich multi-layered interactivity.

“The behavioral engine currently being designed is one step beyond interactivity and offers responses which are much closer to real life behaviors,” said Gauthier. “We are working on creating an interaction between two people, the patient and the physician. The virtual patient can produce, for example, facial expressions and eye movements, and talks back in order to influence the course of the interaction.

“We are developing the interactive design around the concept of the physician, who is the viewer, understanding rather than controlling the virtual patient’s reactions,” added Gauthier. “This research helps me to teach students how to create virtual worlds where viewers influence a situation instead of controlling it. The addition of artificial intelligence with path planning behaviors inside the virtual patient helps to create physiological events that influence the viewer’s decision making.”

Gauthier noted that the current prototype is an animated 3-D character controlled by a set of interactive tools. The viewer is able to see both the exterior and interior elements of the body. The viewer can change the position of cameras around the patient, he said, and can select visualization tools from a 3-D toolbox in order to display specific views of the body. The toolbox includes a navigation tool, an X-ray tool, a pin camera tool, a slicing tool, a cutting tool, a pealing tool, a storyline tool, and a web tool.

The enterprise concept for the virtual patient project will be to provide an online service providing a real-time visual interface for digital data sent by laboratories and medical departments. The virtual patient can be used to create renderings of animated 3-D models at the end of a data processing pipeline. Physicians, radiologists, and laboratories will be able to email numeric data, pictures and even 3-D files. In return the virtual patient interface will provide visual responses that are unique and non-scripted.

Researchers explain that the next step will be to provide the appropriate plug-ins to parse and filter the client’s most common data formats inside the web browser of choice — for example, tif and Diacom files for radiology. The virtual patient setup is available inside a web browser on the desktop, displaying on-demand high resolution renderings of a dynamic virtual patient. A subscription fee will let the user input data in real time on the virtual patient’s web page.

Collaborators on the virtual patient project are Miro Kirov, Zach Rosen, Mike Olson, Marc Triola, Henry Feldman, Fabien Barati, Ganesh Ramanathan, Hillary Gauthier, and Maria Mayer. Funding for the research is provided, in part, by NYU’s CDCF 2004-2005 grant. An online demonstration of the virtual patient can be found at http://www.tinkering.net/vp.

Back to top