PC Farms at Fermilab: Scaling up to Production Systems

Abstract

High Energy Physics (HEP) requires massive computational resources for data acquisition; design, control, and analysis of experiments; and, for theoretical analyses. To improve our understanding of fundamental physics, the CPU requirement will increase by more than an order of magnitude in the next two years. The increasing power of commodity computing hardware and network technology allows scientists to use clusters of relatively inexpensive computers for many HEP applications. Fermi National Accelerator Laboratory (FNAL) will demonstrate two prototype clusters under consideration for production systems, both based on Intel microprocessors, running the Linux operating system. One receives, analyzes, and filters data at 100 MBytes/sec from the CDF Collider Experiment using ATM and Fast Ethernet technologies. The second, a prototype PC FARM analyzes HEP data using loosely-coupled parallel processing techniques developed at FNAL. Both systems execute FNAL codes ported to Linux, and both demonstrate the scalability of such commodity-based computing for HEP tasks.

Contact person:

Elizabeth C. Schermerhorn
P. O. Box 480
Dryden, NY 13053
ecs@fnal.gov
607 840 3481 (phone)
607 844 3665 (fax)

Network requirements:

Booth size requested:

Description of Research Exhibit

The purpose of this exhibit is to demonstrate Fermilab's work in progress on scalable computing clusters built from commodity components which meet the high performance demands of HEP experiments. One cluster is a prototype system for real time data acquisition and analysis; the other is a prototype of an HEP event reconstruction farm. When the experiments begin taking data in 1999, full production systems must be ready to operate.

One of these experiments is CDF, with scientists from 50 universities and laboratories in 8 countries. During the next period of data acquisition, they anticipate an event rate with millions of separate collisions which will produce terabytes of data per second. Massive real time "Level 1" and "Level 2" hardware processors in the data acquisition system select the most interesting 1/10000 of the events for the software "Level 3 Filter", which further selects events to be recorded for detailed analyses offline.

The "Level 3 Filter" cluster we shall be demonstrating is a 1/16th scale prototype of a complete system and consists of five 200 MHz Pentium Pro nodes (4 with dual processors), interconnected with Fast Ethernet via two hubs. The single processor node connects via 155 Mbps ATM to a backbone 2.5 Gbps ATM switch, which in turn connects to ATM interfaces on four VME-bus single board computers (Motorola MVME-1600 family). These components are housed in two instrument racks, with external video monitors. HEP events from the CDF Collider Experiment will be read from memory buffers in the VME crate by the MVME-1600 computers, and feed via the ATM switch into the PC cluster.

In the cluster, the events will be analyzed ("reconstructed"), and those with most interesting physics passed to mass storage. This cluster will sustain data rates through the ATM switch of 100 MBps and represents 1/16th of the full online system. Modifications to Linux for this real time application were necessary and will be part of the presentation.

The "PC FARM" cluster is designed to be ten 233 MHz Pentium II nodes and is approximately 1/20th scale of one of the systems expected to be assembled in two years for the CDF experiment. These nodes will be interconnected with Fast Ethernet via a hub. They will be housed in two instrument racks, with external monitors.

The offline "PC FARM" takes the output of the "Level 3 Filter" to reconstruct each collision event in detail and store the data for easy access by 500 collaborating physicists. The Research Exhibit will demonstrate the capabilities of the cluster by running offline HEP analysis codes which have been ported to Linux and will display the reconstructed events.

Scientists from Fermilab, Academia Sinica and MIT will be at the exhibit to discuss the design and performance features and characteristics of the clusters. They will also be prepared to discuss the experience of porting complex physics codes to the Linux operating system, as well as the issues and science of the upcoming experiments.