IceCube: Data Warehouse
- Tape library and cluster
- Photo by D. Bogen
From the IceCube Laboratory, the data deemed the most critical is transferred via a NASA satellite called Transfer and Data Relay Satellite System (TDRSS) where it's stored in the University of Wisconsin - Madison's data warehouse. Data that isn't as critical is stored on tapes at the South Pole and then carried to Madison by the first people to return after the drilling and summer season has begun. When those tapes arrive, they are transferred into the data warehouse.
In Madison, disk arrays store the data and servers provide the data to the collaboration for further analysis. At over 400 Terabytes (approximately 400,000 Gigabytes) and growing, data storage and management demand a significant amount of equipment. Available storage hardware now includes hundreds of hard drives (ranging from 300GB to 1TB each, depending on age) in numerous Redundant Arrays of Inexpensive Disks (RAID) enclosures.
HP Proliant servers share the data with workstations and cluster nodes via Lustre, a high-performance filesystem. The RAID enclosures are connected to the Proliant servers via a fiber-optic Storage Area Network (SAN) to ensure rapid delivery of data off the disks.
- RAID arrays
- Photo by D. Bogen
RAID arrays protect the data from individual disk failures. However, to protect against other forms of data loss (ranging from large-scale hardware failure and building-wide calamity to accidental erasure) tape drives back up the data and allow off site storage. The primary backup hardware, a QualStar XLS tape library, currently holds hundreds of tapes and numerous LTO4 tape drives. It would be impossible to back up all the data every night, but only a small fraction of the data changes each day (new, incoming data, or recently analyzed results). Each night the backup software recognizes recent changes and duplicates it, along with a fraction of the full data set, and over the course of weeks creates a full copy on tape.
Two models of disk arrays form the primary storage for the project, the NexSan SATABeast and the Apple Xserve RAID.
Each SATABeast contains 42 500GB or 1TB SATA hard drives, divided into eight RAID 5 arrays with five disks each, and two drives available as spares to be automatically used in case any disk fails.
Each Xserve RAID contains 14 500GB or 750GB ATA hard drives, divided into two RAID 5 arrays with seven disks each.
Disk Servers
The disk storage is managed by HP Proliant DL 385 servers running RedHat Enterprise Linux.
Storage and Data Networks
The servers and disk storage connect via a fiber-optic storage network, which allows multiple servers to access the same storage in case of hardware failure or changing need. The storage network consists of Cisco and QLogic fiber-optic switches.
- 2 RAID array types and servers
- Photo by D. Bogen
Each server also has four gigabit ethernet jacks for three separate conventional networks. One public network for communication with client computers, and one private network for communication with a computation cluster (which filters the experimental data, generates simulated data, and produces of the load on the data storage).
Filesystem
To hide the complexities of the servers and disk storage from the end users and client computers, we use the Lustre filesystem to combine the multiple servers and storage into three filesystems. This allows us to present a number of pieces of equipment as coherent filesystems and hides much of the complexity of the system from end-users.
