High Energy Physics & High Performance Technology Coming Together at NYU

By Jimmy Kyriannis & Allen Mincer

The High Energy Physics Group in the NYU College of Arts and Science's Physics Department has recently joined the ATLAS experiment at CERN (the European Organization for Nuclear Research, located in Geneva, Switzerland). The ATLAS experiment is an international collaboration involving 1800 physicists from 150 institutions in 35 countries. The mission of the project is to explore the fundamental nature of matter and the basic forces that shape our universe by searching for new discoveries in the head-on collisions of protons of extraordinarily high energy.¹

Slated to begin collecting data sometime in 2007, ATLAS will study the particles created when two protons collide with a total energy of 14 teraelectronvolts (TeV)—about 15,000 times the energy equivalent of the proton mass. Among the many studies planned as part of this experiment is a search for the Higgs particle, the one remaining undetected particle predicted by what is now called "The Standard Model" of particle physics. This model has been very successful, agreeing with experiment in the hundreds of tests to which it has been put over the last three decades. However, as it is incomplete and raises some other thorny problems we won't discuss here, much theoretical work has gone into understanding how to extend this model. ATLAS will therefore also be looking for signs of new particles, predicted to be accessible to ATLAS by these "Beyond the Standard Model" theories. At CERN, the Large Hadron Collider (LHC) is being built in a tunnel of 27 kilometers in circumference, straddling the French-Swiss border near Geneva. Two counterrotating beams of protons will be accelerated and then smashed head-on at various points along the collider. As described on the ATLAS website, "the energy density in these high energy collisions is similar to the particle collision energy in the early universe less than a billionth of a second after the Big Bang.² Four main experiments (see figure, below), one of which is the ATLAS project, will measure the particle debris flowing out of these collisions, which may occasionally include the sought-after particles.

Figure 1. Overall View of the CERN LHC Experiments
(ATLAS Experiment Image; Copyright CERN.)

A detector at an LHC interaction region faces many difficulties. Bunches of counter-rotating protons will interact approximately 40 million times a second. A typical event may include 1000 outgoing particles, whose type, energy, momentum, and direction need to be measured. Some particle types interact readily, while others pass through a large amount of material with a very small probability of colliding. Particle momenta can be measured by following their paths in a detector-provided magnetic field, but every detector is made of matter, off of which the particles can scatter and therefore change direction.

No single detector type can simultaneously perform all the measurements that are required. The ATLAS detector, which is about the size of a five-story building (see figure 2), therefore consists of several nested detectors surrounding the interaction region, each of which is in turn made of several components. The detector closest to the beam of protons has an inner radius of less than two inches, an outer radius of more than three feet, and a length of more than 20 feet. The outermost detector, designed to measure the highly penetrating muon particles, has an outside length of about 150 feet and a radius of 36 feet. Both of these detectors will measure the position of particles passing through them to a precision of less than the thickness of a human hair.

The high interaction rate of the particles and fine grained multiple measurements by the detectors result in a tremendous amount of data: about 2 megabytes (MB) per event, at a rate of 40 million events per second. In fact, only a small fraction of the events can be kept. A sophisticated three-level trigger system—one part of which is being designed by the NYU group—culls the most interesting events and keeps the rate down to a just-manageable few hundred megabytes per second (MB/s).

Figure 2. The ATLAS Detector. At the size of a five-story building, weighing 7,000 tons, the detector records the product of protons colliding with each other at very high levels of energy. The resultant data from that collision streams from the detector over high-speed computer networks at approximately 320 megabytes per second.

ATLAS thus presents many data flow, networking, and computational challenges. Even after being abridged by the trigger system described above, the large amount of data produced is too great to reconstruct and distribute to all collaborating institutions. Rather, a multi-tier computing system has been developed to divide the load of collecting and processing the volume of information being gathered by the ATLAS detector.³

On average, data streams from the CERN detector at about 320 MB/s, requiring ten gigabits per second (Gbps) links from the detector for very high performance and reliability. The roughly 10 petabytes per year (1 PB is equal to approximately one billion megabytes) of raw data from the detector flows into a very high capacity CERN storage "Tier-0" facility. It is run through an initial processing step, distributed to ten global Tier-1 facilities around the world for further processing to yield datasets for physicists, and then made accessible to a much larger number of Tier-2 facilities for greater scientific analysis.

The volume of data generated by ATLAS is equaled by the computational power necessary to analyze it. Grid systems,⁴ for example, use net work s to interconnect supercomputers and computer clusters to yield a single system with performance characteristics that exceed those possible with a single computer. The Open Science Grid (www.opensciencegrid.org), in particular, is a distributed high performance computing facility that spans multiple campuses and provides processing resources to support the heavy computational loads of the LHC experiments, as well as other physics and biology research activities.

Transmitting large amounts of data between the ATLAS Tier facilities and research institutions necessitates very high performance networks. Rapid and low-latency remote network access and graphics display are also necessary for research activities and visualization of the analyzed data. In addition, communication among the many institutions and countries requires video transmission and conference capabilities, which also place demands on the network.

LHCNet is a dedicated network specifically designed to support the needs of CERN LHC research projects, such as ATLAS and CMS (Compact Muon Solenoid). The LHCNet network operates at 10 Gbps, spanning CERN, the StarLight communications facility in Chicago, and the Manhattan Landing (MANLAN) communications facility hosted by NYSERNet (www.nysernet.org) in downtown New York City.⁵ Participant research institutions then typically establish their connections to LHCNet via StarLight or MANLAN.

In Summer 2006, NYU obtained a 1 Gbps Ethernet connection to the LHCNet network presence at MANLAN, via Dense Wave Division Multiplexing (DWDM) technology on the NYU Metro Optical Net work.⁶ This connection, in fact, provides the campus with connectivity to two research networks: the United States LHCNet presence (USLHCNet) and the UltraLight network. By its nature, it also provides high-speed connectivity to the CERN facility. A single "DWDM lambda" (optical wavelength) represents the optical link from the NYU campus to MANLAN to establish the gigabit Ethernet connection, while Virtual LAN Network ( VLAN) technology over that Ethernet link makes the simultaneous transmission of both LHCNet and UltraLight networks possible.

While LHCNet supports the research needs of NYU's High Energy Physics Group, UltraLight is an experiment in hybrid optical networks. UltraLight is supported by the California Institute of Technology and CERN for research in hybrid networking, where network devices and Internet-connected computers use both IP (the Internet Protocol) and, simultaneously, direct fiber optic communications. The creation of fiber optic links between sites on a dynamic, on-demand basis is of particular interest to research groups that transmit very large amounts of data and require extremely high bandwidth performance (at least several Gbps) without congestion or interference from other network communications. Supporting this capability is a design goal for the next-generation Internet2 network, currently known as "NewNet.⁷ The basic building-blocks of this network are hybrid "nodes," which will ultimately enable on-demand optical connections between institutions on a national scale.

Research projects on the forefront of science continue to place novel demands on technology resources. The design of the computational and network models for ATLAS was a substantial undertaking and has furthered hybrid optical communications, high-speed data transfer techniques, and grid computing technologies through its application, even at this early stage in the project. The requirements for high performance computing technology across the academic disciplines are growing at a terrific rate. The national distributed computing facility, TeraGrid (www.teragrid.org), is similarly receiving quite a bit of attention from universities across the country. Perhaps not unexpectedly, information technology continues to develop and benefit from the research needs of the academic community as it rises to meet that demand.

Footnotes

1. http://atlasexperiment.org/
2. http://atlasexperiment.org/more_about_atlas.html
3. The ATLAS Tier computing structure is described in detail in the CERN ATLAS Technical Design Report, ATLAS TDR-017, CERN-LHCC-2005-022 (http://atlas-proj-computing-tdr.web.cern.ch/atlas-proj-computing-tdr/PDF/
   Computing-TDR-final-June20.pdf)
4. See Global Grid Computing at NYU by Richard Bonneau for more information about these systems.
5. NYU is a member organization of MANLAN and a close collaborator on research and academic-oriented network initiatives.
6. Please see High Performance & Optical Networks by Jimmy Kyriannis for more information about DWDM technology, hybrid optical networks, and the NYU Metro Optical Network.
7. A technical description of the NewNet network architecture is available at: www.internet2.edu/files/Internet2-New-Network-Tech-v0.9.pdf