Every year in the United States, millions of women are compressed, pulled, and mashed between the unfriendly plates of an X-ray machine, then sit on pins and needles as they await the results of their mammogram. Depending on the equipment, somewhere around 10 percent of these patients are called back for further screening—additional scans, ultrasounds, MRIs, and/or biopsies. (The newer 3D machines tend to have clearer images so require less frequent follow-up.) Most will ultimately walk away with a clean bill of health, but some 255,000 women will be diagnosed with a malignancy.
Krzysztof Geras, assistant professor of radiology at the Grossman School of Medicine and an affiliate faculty member at the Center for Data Science as well as the Courant Institute of Mathematical Sciences, has spent much of his career training artificial intelligence networks to help radiologists with the vital act of interpreting all those medical images—as well as demonstrating that AI may one day go head to head with humans at the same diagnostic task.
Geras was already immersed in the development of deep neural networks (DNNs) when he came to NYU’s Center for Data Science as a postdoc in 2016. Neural networks are trained to recognize patterns in the huge quantities of information they’re fed so that they can mimic the human brain. To compare the cancer-detecting skills of people and machines, Geras first built an enormous data set of more than 1 million images (at least four views of the breasts from 229,426 screening exams). The radiologists’ findings were typical of what they see in their patient populations: 96 percent of the cases were free of cancer, one half of one percent had a malignancy, and another 3.5 percent showed benign breast changes. Geras and his team then proceeded to teach the neural network how to make these distinctions.
What the researchers established is that humans and machines possess complementary abilities. “Radiologists are better at integrating information across different images and different imaging modalities,” says Geras. “They can look at a mammogram and then they can look at maybe an ultrasound and they can look at multiple images and synthesize this information into a decision.” They’re also good at deciphering shapes and patterns, and they bring context, says Nan Wu, a PhD student at the Center for Data Science who joined the project five years ago. “When they’re looking at a suspicious region, they might check the symmetry of the breast or the general density or maybe some other technique they’ve learned during their training.”
“Neural networks, on the other hand, have other advantages,” says Geras. “They don’t get tired. They don’t get bored. And they’re very fast—it only takes them a small fraction of a second to process these images, and they exhaustively search for everything.” In a face-off against 14 radiologists, the AI performed nearly as well as the full team and at least as well as a single radiologist.
Some 40 professors, students, and doctors have played a part in developing the system. After a few years at the Center for Data Science, Geras “realized that to make it happen I needed a bigger team.” He then applied for—and received—an affiliation at the School of Medicine. “I work a lot with radiologists at NYU Langone Health,” Geras says. “I wouldn’t be able to do this without them.” His dual appointments have allowed him to gather a cross-disciplinary group of computer scientists (including from Courant), AI and DNN specialists, and experts in medical imaging (the images’ creation) and radiology (their interpretation).
Geras then took the unusual step of posting a detailed report of the DNN’s design technology online and offering its tools for free to universities and hospitals around the world. Institutions from Australia to Finland have downloaded the network and plugged in their own scans. “[They told me] that what we put online generalized very well to their data,” Geras says. “That’s a very big win for us because we have shown that these results detect cancer across different populations and different manufacturers of X-ray equipment.” By open-sourcing his research, Geras hopes that “people will have the courage to try similar projects, with other types of imaging and other body parts.” Sharing the technology has also meant that “we are gaining a lot of publicity, which might help us in the future, either in terms of funding from charities, or if we ever decide to commercialize the work.”
The team continues to teach the system new tricks, such as the distinctly human skill of synthesizing and interpreting multiple views of the same breast. But for now, the best diagnostic results come from a combination model that uses both human and machine strengths. Future diagnostic protocols might involve using the DNN as a “second reader,” says Wu—having radiologists first estimate risk by traditional methods, then consult the network, and then possibly reassess their prediction.
Even if next-generation DNNs become more adept, she adds, the hands-on experience of doctors will help inform their design. And doctors’ communication skills will always be needed to meet patients’ emotional needs. “Radiologists will always be important,” Wu says. “We don’t really want to replace them.”
—Lindsy Van Gelder
Stone forests—agglomerations of pointed rock formations that resemble petrified trees—have long occupied the ranks of our planet’s more mysterious geological features. Scientists were stymied by the specifics of the processes that form the spear tip points of these jagged pinnacles, which can be found in Southern China, Malaysia, and Madagascar. But now a team of NYU mathematicians has solved that mystery using—of all things—a miniature mountain of candy.
Geologists agree that stone forests form when submerged rocks dissolve incrementally into the surrounding water. The researchers, who include
Leif Ristroph, an associate professor at the Courant Institute of Mathematical Sciences, and Jinzi Mac Huang, an assistant professor of mathematics at NYU Shanghai, conducted a series of experiments in the Applied Mathematics Lab to investigate how dissolution drives the specific creation of the forests’ trademark spikes.
Using computer simulations, they developed a math model that isolated the precise dissolution mechanisms that culminated in the structures’ sharp spires. They found that as the minerals in the rock dissolved, the surrounding water got heavier and flowed downward thanks to gravity, modifying the dissolution rate and leading over time to those dramatic tapers. To confirm the accuracy of the model, the team replicated the formation of stone forests by submerging blocks of soluble hard candy in water tanks and observing the dissolution process that reshaped them. As they’d hoped, the resulting “candy forests” (pictured above) sprouted points that could break skin.
Published in Proceedings of the National Academy of Sciences, the research has both practical and theoretical implications. “We are considering manufacturing sharp-tipped objects through dissolution,” says Huang. “Our work actually shows that we can control the final sharpness of this object through a carefully designed initial shape.” Such precise control could improve the manufacturing standards of needle-tipped elements used in crucial medical and research equipment.
Beyond that, however, lies the satisfaction inherent in understanding the world a little better; the shape of a natural object is, after all, an important clue to its history. “Our progress not only extended the frontier of the study of fluid-structure interactions,” Huang says. “It also brought us one step closer to understanding the past, and the future, of the rocks on our planet.”
Out for Justice
Alina Das’s interest in immigration law was borne of personal experience. Her parents immigrated to the US from India, and Das says: “I always had this sense of otherness, a feeling that because of the color of my skin, because of differences in culture and upbringing, I was never fully perceived as an American.” In her book No Justice in the Shadows: How America Criminalizes Immigrants, Das—School of Law professor and codirector of NYU’s Immigrant Rights Clinic—explores the racist roots of US immigration policy and how the criminal legal system is used to target immigrants.
Immigration law has always been influenced by racism and white supremacy
“Congress’s first act in 1790 was to limit the right to be naturalized to free, White persons. And from there, you see that concept turn up whenever it appears that people of color may develop a foothold. When Chinese immigrants were developing roots in California, legislators talked about Chinese women as prostitutes. One of the first antidrug ordinances in the country was a San Francisco anti-opium law clearly targeting Chinese people. Fast-forward a few years, and Congress passes the Chinese Exclusion Act.”
The Trump administration was “a magnifying glass for what was already happening”
“More people were deported in Obama’s first three years than in Trump’s. Obama defended it by talking about deporting felons, not families. So he used this language of criminality, saying that we are going after the ‘right’ people. [But] Trump targeted people who’d previously had some measure of security, with attacks on Deferred Action for Childhood Arrivals, and of course the family separation program. The openly racist statements Trump made to justify these policies made it much clearer what was happening.”
Deportation is a shortcut that doesn’t actually solve the problem
“The Trump administration used [the gang] MS-13 as its favorite talking point. But when people [previously] came to the US fleeing Central American conflicts, the Reagan administration didn’t recognize them as refugees because of its political ties to regimes in these countries. So those immigrants were not given support. Many were victimized and formed a gang to protect themselves. And instead of actually doing the real work to address gang issues, which has to do with gang intervention programs, education, economic opportunities, we decided to deport people. That’s how MS-13 became a transnational gang. If you’re deporting somebody who is committing violence without doing the work of addressing that violence, all you’re doing is spreading violence.”
Things are much the same under the Biden administration
“Much of the same harm is being done, but it comes with the veneer of legitimacy that the Trump administration didn’t even bother trying to adopt. We don’t have to talk about it because it’s not in our face and not being justified using the overtly racist terms that Trump did.”
The solution is to ensure that the criminal legal system isn’t used as a pipeline for deportation
“In my work, I see people who are American in every way, except on paper, being told they have to start over in a country where they may face persecution or at minimum be placed into forced, isolated poverty because they know no one. [Even those who commit crimes] don’t lose their humanity because of their worst choices. We are choosing to harm people in very drastic and cruel ways that really don’t provide any good. It doesn’t reverse the loss; it creates a ripple effect of harm.”
The Science Behind...
Ever wonder how delivery drones handle swinging payloads that dwarf their spindly frames without spinning out of control? Until now, not reliably! Luckily, Giuseppe Loianno and his team at NYU’s Agile Robotics and Perception Lab are on it. “Previous approaches leverage GPS or motion capture and do not consider perception and physical constraints,” Loianno says of the current crop of drones. “This approach can be reliable in remote areas but is subject to strong failures in urban environments where the signal is often shadowed. In addition, many systems consider the payload a disturbance.”
If it seems counterintuitive categorizing the cargo as an obstacle to the workings of a cargo delivery system, the robotics team agrees. The payload is, in fact, key to the researchers’ approach; it’s the heart of what Loianno calls “a perception and action system.” An onboard camera watches how the dangling payload is moving while an inertial measurement unit—a device like the one in your phone that measures acceleration and rotation—tracks how the drone itself is moving. These data sets work with a model predictive control system that anticipates what the payload will do next and how that squares (or doesn’t) with what the drone itself is about to do, ensuring that the physics of these elements remain synced in service of stability. “The user should imagine that that payload has its own motion or path in space,” explains Loianno, an assistant professor at the Tandon School of Engineering. “The robot moves to accommodate the load’s motion.”
The system has shown promising results in preliminary tests but is still a ways from wide deployment. Challenges include adjusting for extreme speed and weather as well as complications operating in GPS-denied environments. The team is currently working on a distributed approach involving multiple drones cooperatively managing payloads, with initial runs already showing exciting results. The project has ramifications beyond simply protecting your Amazon packages. Drones could streamline warehouse operations, transform object manipulation on construction sites, and even revolutionize supply deliveries in disaster zones. The sky is literally the limit.
Photos from top: SCIEPRO/Getty Images; courtesy of the Applied Math Lab; courtesy of NYU Law; Bestgreenscreen/iStock