New Software Allows Scientists to Navigate the Brain in 3-D

Early cartographers would mark the unknown areas on their maps with the words “Terra Incognita.” Today’s neuroresearchers could use the same Latin phrase in their work with the human brain, as many areas of “uncharted water” remain to be explored. Two modern-day explorers at New York University are using three-dimensional mapping techniques to break new ground in the research and understanding of the brain.

Patrick Kelly, chairman of neurosurgery at the NYU Medical Center, and Jean-Marc Gauthier, assistant art professor and teacher of 3-D interactive worlds and virtual spaces in the Interactive Telecommunications Program at NYU’s Tisch School of the Arts, have collaborated to design a web-based software prototype that allows a viewer to upload a three-dimensional visualization of the brain that is made of thousands of MRI slices. This allows the viewer to enter the brain from any angle or point of view and see slices following his or her own path using the 3-D pixel information.

Kelly, an avid sailor, has years of hands-on experience navigating his sailboat in foggy weather among the islands off the coast of Maine. He maintains that his knowledge and insight into coastal navigation has been useful to him for understanding and improving his research and work as a neurosurgeon. “Navigating my sailboat has been a major inspiration to me in the work to develop a three-dimensional navigation system for operating on the human brain,” he said.

According to Kelly, the problems for surgeons operating on the brain are somewhat analogous to being on a sailboat in foggy weather with little to no visibility. “The navigational challenge for the sailor starts with gathering all the necessary available information in advance—navigational and current charts (maps), weather forecasts, tide tables, and mariner’s notices, for instance. Then I plan my course. And finally, I sail my planned course, but in real world conditions that require me to be constantly correcting and updating my information as I go.”

The top images (black and white) offer 2-D pictures of the brain. The four images below them illustrate how a Webcam image controls the 3-D navigation system inside the brain.

A neurosurgeon uses a similar three-step process. Before operating on a patient, he collects important imaging data that includes computed tomography (CT scanning), magnetic resonance imaging (MRI) and angiography (blood vessel study of the brain). This planning phase allows the surgeon to navigate the safest course to reach a tumor located deep inside the brain without damaging important areas of the brain and blood vessels. But even with all this imaging information, and with the more recent introduction of visualization tools available to neurosurgeons now, it’s still not always enough for the job at hand.

Gauthier has long been interested in finding new applications for the details that three-dimensional technology can provide. “It was through my real-time 3-D visualization process for the Dynamic Virtual Patient project a couple of years ago that Dr. Kelly and I discovered we were both passionate about resolving problems of 3-D navigation inside visual data,” said Gauthier. The challenge to both explorers now was to find a way to accommodate the nearly infinite amount of data needed in a 3-D replication of the inside of a human brain. There seemed to be no solution to the problem of excessive data, unless the two could find a new way of looking at things.

At the same time the two were discussing possible solutions, Gauthier was grappling with a similar type of problem while working on 3-D interactive maps of Manhattan. The amount of 3-D details requested when moving the virtual camera from a birds-eye view of 42nd Street to the details of a block on Times Square also seemed close to infinite. He started working on a small prototype where 3-D anatomic details were cloned and assembled together as they entered the line of sight of a virtual camera traveling inside the human body. “This camera concept is revolutionary because we usually create a virtual world with 3-D models, textures, and animations first, and then shoot a scene inside that virtual world. In this case you are entertaining the idea of introducing a virtual camera into a world that already exists and that you have yet to discover in its various dimensions,” he said.

Applying the same technique, Gauthier and Kelly were able to successfully develop a way to visualize not only the vast amounts of brain data but navigate through the virtual 3-D ‘cloud’ that resulted. The next step was to record the interactive experience of the surgeon navigating through the data. It was then that Gauthier suggested utilizing a webcam in order to track the head of the surgeon. This precise tracking of the surgeon’s head movements could allow control of the virtual camera from a distance without the need to touch a mouse and a keyboard. The hands-off approach, which is very useful in the operating room, uses computer vision and augmented reality in order to record and analyze how the surgeon moves though the brain in order to plan a path before surgery.

The whole path-planning process can be stored in a database and replayed later in the operating room. Tracking how surgeons navigate inside a 3-D brain can help for comparing planned path versus real path decisions that may occur prior to and during surgery. This type of information can be useful for the training of surgeons in addition to enabling a seasoned surgeon to revisit their path.
Kelly and Gauthier’s new Internet-based Web browser allows 3-D navigation inside a brain using a cloud of voxels, or pixels, located in space. Since the images of slices of the brain are displayed in space they can be visited from many angles, including new angles that were not included in the original pictures. This is key to enabling freer navigation. The viewer can then navigate the virtual brain in any direction regardless of the orientation of the original slices of the MRI.

To the current prototype Gauthier is now adding a layer of artificial intelligence in order to learn from the surgeon’s eye movements. The patient’s brains may be different, and each surgeon’s way of looking at a brain may also be different, but there may be some consistency in the way one particular surgeon approaches the brain. The software will be able to anticipate the surgeon’s needs the same way Amazon.com or Google learn about the habits of its customers and get better at displaying items that may fit their needs. This time higher resolution levels of details will be retrieved faster from an online database by anticipation of the surgeon’s interests.

For more information about the project and online demonstrations, visit www.tinkering. net. The brain project is a joint collaboration between ITP Tisch School of the Arts and the Department of Neurosurgery, School of Medicine, NYU. Collaborators for this project also include ITP student members of the Brain Group: Caroline Pino, Rocio Barcia, Jae In Lee, Sandra Villareal, and Chunxi Jiang. Some of the new ideas about visualization of the brain were also discussed in meetings of the Brain Group from Gauthier’s “Sciviz” class during spring 2006.

Back to top