PDD - Data Handling
By Section | Whole Paper | PDF (233 pages, ~6.62mb)
8 Data Handling-
8.1 System Elements
- 8.1.1 Software Management
- 8.1.2 System Engineering
- 8.1.3 Development Environment
- 8.1.4 Analysis Framework
- 8.1.5 Database
- 8.1.6 Visualization
- 8.1.7 Development Interfaces
- 8.1.8 Integration at Pole
- 8.1.9 Hardware
- 8.1.10 Data Distribution
- 8.2 Offline Data Flow
-
8.3 Data Model
- 8.3.1 High Multiplicity
- 8.3.2 Upgoing Tracks
- 8.3.3 Cascades/Taus
- 8.3.4 GRB Downgoing Muons
- 8.3.5 Icetop
- 8.3.6 Supernova
- 8.3.7 Prescaled Raw Data
- 8.3.8 Monitor
- 8.3.9 Calibration
- 8.3.10 Full-Sky Summary Histograms
- 8.3.11 Unfiltered Raw
- 8.4 Data Sample Organization
- 8.5 Latency
- 8.6 Schedule
- 8.7 Summary
8 Data Handling
Data from the IceCube sensors will be selected, reconstructed, filtered and analyzed to achieve the scientific goals of the project. The software used in this endeavor extends from firmware deep in the ice, through the DAQ, to the Data Handling, and finally into the analysis. In this section, we describe the Data Handling software and associated computing hardware.
The location of IceCube at the South Pole places special demands on what would otherwise be a straightforward software system. First, the Data Handling software must provide robust, fast and accurate filtering of the data in an essentially online environment. This is because IceCube's high data rate from downgoing muons results in a large data volume in spite of a small individual event size, and the satellite bandwidth for uploading data to the northern hemisphere is much too small to permit full raw data transfers. Second, the harsh environment of the South Pole and its inaccessibility for about nearly 3/4 of the year mean that "winter-overs" must be employed to maintain the detector. Since winter-overs will often not have expertise in all aspects of the detector, the interfaces used to control and monitor the Data Handling system at the Pole must be simple, user friendly, and battle-tested, and the associated computing hardware systems must be reliable and fault-tolerant. Third, in the absence of a centralized laboratory facility, and since the collaboration cannot meet as frequently as one might wish, it is crucial that interfaces between various subsystems are very well defined to allow for independent software development.
In sec. 8.1 we describe the general features of a Data Handling software system for the IceCube detector, and demonstrate that these features are available in existing software packages, or require no special techniques to custom build. Offline data flow is covered in sec. 8.2. The various streams into which we divide the data are discussed in sec. 8.3, and the way it is organized in sec. 8.4. Data latency requirements are described in sec. 8.5, and sec. 8.6 gives an outline of the schedule for implementation of the Data Handling system.
