Virginia Polytechnic Institute and State University

Cost-Conscious Supercomputing

Interconnects

Technology Partners Make Vision a Reality

With this many computers, how can they all communicate effectively? The sheer volume of traffic among 1100 computers is immense, and in a supercomputer, it has to travel quickly. Virginia Tech turned to cutting-edge technology from Mellanox Technologies for this critically important component.

From Israel to Cupertino

Israel-based Mellanox Technologies signed on to provide InfiniBand technology for the cluster as the primary communications fabric, as well as all necessary drivers, cards, and switches for the project. Since Mellanox had not yet ported their driver to the Mac OS X platform, that lent still more urgency to the project. But three engineers from Mellanox — Vu Pham from Mellanox U.S. and Dror Goldenberg and Edward Bortnikov from Mellanox’s Israel headquarters — set up camp in Apple’s Cupertino labs, and went to work.

“The assistance we received from Apple was invaluable in helping us meet our deadline.”

— Dror Goldenberg, Software Architect, Mellanox Corporation

“Both Apple and Mellanox had identified the project as a strategic one,” says Goldenberg, software architect at Mellanox. “We were pleasantly surprised by the abundance of online documentation, and the willingness of Apple’s Developer Technical Support team (DTS) to help us accomplish our aggressive goals. In less than two weeks, we had a solution outline.

“However,” Goldenberg continues, “it was clear that there was no chance to succeed unless we worked face-to-face with Apple developers. Therefore, we headed to Apple headquarters in Cupertino, where we spent two weeks studying the new platform, working on the Power Mac G5 prototypes, consulting with the DTS team, and writing sample drivers. Then, we moved to the real code. The effort was very fruitful, and we all were very excited when the 1100-node cluster was ranked as the third most powerful supercomputer in the world.”

Turbo-Powered Communications

InfiniBand is now available on Mac OS X for any user who wishes to use it in a high-performance environment.

“By adding an InfiniBand card to each Power Mac G5,” says Lockhart, “we get a 10-gigabit per second communications fabric. It’s full duplex, and capable of ensuring some very low latencies in transmitting data … on the order of 4.5 microseconds for certain packet sizes. The nice thing about the InfiniBand solution was that its price point matched other comparable products, it was five times as fast, and had lower latencies ... so that was a great coup for us.”

Five Gigabit Ethernet switches from Cisco also were added to the supercomputer’s infrastructure, providing a secondary communications fabric for management, data transfer, and other less latency-sensitive tasks. With built-in gigabit Ethernet standard on every system, putting each node on the network was a snap.

A Safe ‘Port’ in the Storm

Vu Pham, Dror Goldenberg, and Edward Bortnikov, software architects at Mellanox, were charged with porting the InfiniBand driver to Mac OS X. Since none of them had ever booted up a Mac before, let alone do any development on the Mac platform, all say their assignment was very ambitious. Factor in the need to complete the migration of a very sophisticated application to the new platform in little more than a month, and all players on the team were collectively holding their breath.

“The InfiniBand driver is a very nontrivial product — the latest specification is over 2000 pages long,” Goldenberg explains. “Due to the robust capabilities of InfiniBand, it’s much more complex than standard network drivers. Fortunately, our software is architected in a modular way that only requires reimplementing the library and kernel extensions that provide the OS-specific services for each new platform. Plus, the assistance we received from Apple was invaluable in helping us meet our deadline.”

Having visited Cupertino earlier, Goldenberg and Bortnikov returned to Israel with their plan in mind and completed their initial development on Power Mac G4 servers running Mac OS X. This required a development model that was strictly “follow the sun”: As one team finished a long day’s work, they would hand off their code to the next team to keep the project moving forward. When the coding was completed and the initial tests were passed, the code was ready to be installed and tested on the Power Mac G5 systems.

Just in Time Transition
A week before the software went into production, the Mellanox team received their first Power Mac G5 computers. From then on, says Goldenberg, the transition was “very smooth and painless.” The development effort also included the porting of Mellanox’s comprehensive testing and benchmarking suite to the Mac OS X platform, which ensured the stabilization of the InfiniBand driver.

While Bortnikov, Pham, and Goldenberg were busy with the driver porting project, Varadarajan was porting the Message Passing interface (MPI) stack and the benchmarks to Mac OS X. Combining the new driver with MPI and the benchmark worked with little effort, says Bortnikov. But he, Pham, and Goldenberg still had one last, nontrivial issue to deal with: scaling the application to 1100 nodes, and delivering 10-teraflop performance in two weeks.

“The Terascale cluster was the most significant scalability trial to date for both our hardware and Apple’s software — previously, our largest cluster was 256 nodes,” observes Bortnikov. “It was a tremendous logistical and human effort to assemble the 1100-computer cluster. But due to unlimited 24-hour pizza, soda, and student volunteers from Virginia Tech, we did it!

“Then,” he continues, “working around the clock at the Virginia Tech Terascale Computing Facility with our field application engineer team of Gene Crossley and Jeff Kirk leading the way, we managed to stabilize the network, the MPI port, the benchmarks, and make everything work together. The driver code was quite solid from the day it was ported to Mac OS X. All in all, we were very pleased to work on this software development project with Apple and Virginia Tech.”

Introduction 1 2 3 4 5 6 7 8 9

Download this story