The general concept for the surface system DAQ is shown in fig. 73. The proposed DAQ design revolves around three separate phases of event creation:
These tasks can be laid out in more detail by enumerating the messages that are exchanged on the various networks of the DAQ. The following items are also summarized in the figure. Since 100BaseT networking is assumed, bandwidth requirements are shown to be comfortably less than 8 MB/sec in each case.
One notable feature about this design is the prevalence of off-the-shelf computing and networking hardware from the earliest possible point in the data flow diagram. It should be further noted that the focus is on commercial hardware and software found in the world of e-commerce and Web services industry as opposed to process control. Furthermore, there is a clear emphasis on the use of commercial Ethernet networking protocols and hardware in place of more traditional bus-based architectures such as VME or CPCI. Both of these features are intentional and represent our belief that, given both the expected data rates and the difficult maintenance environment at the South Pole, these choices represent the best solutions for both system performance and stability. A corollary of the emphasis on off-the-shelf hardware products is that most of the effort in DAQ is software.
The system consists of an interconnected set of hardware components and software tasks. While this design does allow for a great deal of flexibility in how these components are grouped together and mapped onto various networks and processors, the overall system performance goals dictate that certain aspects of this grouping are fixed. In the following discussion, the distinction between system software processes and actual CPUs within the system should be noted. For example, a complete string of optical modules is said to terminate in a single processor, called the string processor.
For the purposes of system design, it has been important to decide that, since an optical module string represents a logical grouping of data for triggering purposes, all data from one string will always flow to a single processor and will not be distributed across multiple CPU's. While, for the purposes of this document, it is easiest to discuss the string processing program as executing on a single processor, namely the string processor, the system design does not require that each string be handled by a separate processor. It does, however, require that all data from all DOMs in a string are routed to a single system process that can deal with them in an efficient and complete manner.
At the surface, the twisted pairs from DOM pairs on a string are connected to a DOM Hub. Each twisted pair connected to the DOM Hub is defined as a channel. It is appropriate to design the Hub such that the number of channels per Hub is a factor of the number of DOM pairs on a string, i.e. 30. For example, if the DOM Hub accommodates six channels, five DOM Hubs are needed to service an entire string. If the DOM Hub accommodates ten channels, then only three DOM Hubs are needed (similarly, two DOM Hubs will suffice if a Hub has 15 channels).
The functions of the DOM Hub include power distribution and control for each DOM pair, message control, data flow management from the DOMs, downloading of software, firmware, and operating parameters, and generation of time calibration signals. The DOM Hub design is derived from the single-channel Test-Board used to read out string 18, and is the only significant piece of custom hardware awaiting realization. It is conceivable (and likely) that the design and layout of the DOM Hub will accommodate more than six channels. In what follows, however, we assume a conservative default of six channels per DOM Hub (12 DOMs).
The DOM Hub design is centered on an embedded processor that provides low level communications services between the surface and the DOM and provides high level communications services (i.e. TCP/IP) to the string processor. The DOM Hub will consist of 1U rack-mounted chassis with front-end electronics including FPGAs and FADCs for 12 DOMs, an embedded CPU daughter card for processing and communications functions, and a power supply capable of providing power for all attached DOMs. The back panel of the chassis contains connectors for direct attachment to DOM deployment cables, master clock distribution, and a standard 10/100BT network connector. The embedded processor is a credit-card-sized 200 MHz StrongARM CPU running Linux.
The DOM Hub will support TCP/IP protocol and will deliver data streams for each DOM to the string processor using the standard socket programming model. Furthermore, the DOM Hub will provide transparent web-based communications with individual DOMs so that maintenance and minimal testing functions may be performed remotely without the need of the full data acquisition system.
All real-time functions in the surface DAQ that depend on nanosecond-level timing are contained within the DOM Hub. Data transferred from the DOM Hub to downstream elements in the surface DAQ are time-stamped messages; subsequent processing needs only to ensure adequate data flow. The latencies of network operation do not introduce synchronization problems.
The DOM Hub receives a carefully distributed 40 MHz master clock. This master clock will be linked ultimately to GPS, providing a universal time reference to other GPS-linked data sets to ≥ 40 ns. The DOM Hub maintains a register capable of assigning a unique time-stamp for at least one day (44 bits). Although runs will likely not extend more than one day, it is straightforward to track time perhaps for as much as a year (51 bits). Test functions will be incorporated to ensure that every DOM Hub has exactly the same master time. These test functions will operate as separate, scheduled threads that will not interfere with normal data acquisition.
Assuming a PMT noise rate of 300 Hz, 60 DOMs per string, a fraction of 5% of hits needing full waveforms, 170 bytes per waveform and 8 bytes per non-waveform hit, one obtains a DOM data rate of about 5 kb/s and 0.3 MB/s per string. Because these data flow rates are modest, the surface DAQ can be network-based rather than a more traditional crate/backplane/bus implementation. Of course, with local coincidence, these rates would drop an order of magnitude further.
Data from each DOM is requested periodically, at a rate of ∼10 Hz. This provides adequately small latency for data streaming purposes, and maintains a reasonable transmission burst of 0.5–1 kb/request. Each request for data includes the time-mark signal as the first pulse. The DOM will ignore most of these, using them only as frequently as local clock drift characteristics require.
DOM data packets with local time-stamps are locally buffered in the DOM Hub. For each string, hits from DOMs connected to five DOM Hubs are sent over a String LAN (one 100BaseT network switch) to a String Processor (SP).
Data buffered in the DOM Hub is block-transferred periodically, ∼10 Hz, to the String Processor (SP). The SP is expected to be a standard PC CPU running a Linux OS. Here, data is converted to master time units. The desired space-time correlations characteristic of physics, traditionally defined with real-time electronic trigger systems and called "triggers," are found in our case through software algorithms in the SP. Simulations have shown that only about 5% of available CPU cycles of a 1 GHz PC are needed to perform typical coincidence algorithms for a 60-DOM string.
Each String Processor is also connected to a global Event LAN. String coincidences are reported by each String Processor to the Global Event Trigger on the Event LAN. The Global Event Trigger is envisioned as a single processor on the Event LAN, which is also connected to a set of processors that serve as Event Builders. Assuming a string coincidence rate of 60 Hz per string, an average of 4 hits per coincidence event, and 60 DOMs per string, one obtains a total data rate for all 80 strings of less than 0.2 MB/s for these messages.
When trigger conditions for a particular event type are fulfilled, the global event trigger informs the String Processor to tag all hits that occurred near enough in time to the global trigger time. If one assumes a global aggregate trigger rate for all trigger types of 1 kHz, 10 bytes per message, and 80 string processors, these "Look-back Requests" require an average bandwidth of 0.8 MB/s.
The String Processor sends the data for the tagged hits to one of a set of Event Builder CPUs. This also occurs over the Event LAN. Assuming 20 random noise hits per event, 12 muon hits per event, and the above values for the average number of bytes per hit and for the global event rate, one obtains an average bandwidth of 0.6 MB/s. This brings the total bandwidth usage on the Event LAN to about 1.6 MB/s, well within the capabilities of 100BaseT switches.
It may turn out that the waveform data retrieval task for the look-back requests may be just as easily handled by sending all waveform data associated with the string coincidence trigger messages, since the amount of data arriving at the SP is roughly the same as the look-back request traffic. This software architecture would eliminate the look-back process, yielding a data flow diagram without loops at this level. In this scenario, however, the destination of the waveform data needs to be considered carefully to ensure arrival at the appropriate Event Builder target CPU.
The Event Builder CPUs send fully constructed events to a disk server over the Online LAN. The disk server will write events to disk. Since the bulk of the event data is in the hits, the bandwidth on this network will be not significantly more than 0.6 MB/s. 1 MB/sec is a safe number to assume, well within the 100BaseT limits. In addition to the Event Builder processors and the disk server, the Online LAN also is connected to the processors that constitute the online filtering and data archiving system. Events are still represented in master clock time units, but also have the associated GPS times linked to each.
This design is attractive, in part, due to its high degree of modularity and flexibility. Thus to create a minimal test stand for DOM production and quality control, one can simply work with a single DOM Hub attached to one or more DOMs (or DOM pairs). Functionality for a whole string can be achieved by networking the DOM Hubs to an off-the-shelf String Processor CPU and installing the requisite software. In an international collaboration where hardware production, software development, testing and deployment will be carried out on multiple continents, this is a key advantage. Finally, with the proposed "network-heavy" scheme, IceCube will benefit from a tremendous amount of community experience and publicly-available tools for TCP and UDP-based networking, which far outstrips the resources available for bus-based architectures such as VME or Compact PCI.
The DOM and its associated DOM Hub design will accommodate the technical requirements of both IceTop and IceCube, with differences confined to firmware, software, and operating conditions. Within the counting house, the differences would be apparent only in the network configuration.
Each IceCube string requires 15 quads for the 60 DOMs. Unlike IceCube, the IceTop DOMs will not share a twisted quad with a neighboring DOM since the expected rates are higher than those of IceCube. The four IceTop DOMs thus require two quads, for a total number of 17 DOM twisted quads/string. In addition, there will be ancillary devices added to the string, such as pressure sensors used during deployment and freeze-back, so the expected total number of quads will be at least 18 per string, perhaps more. The fraction of counting house resources needed for IceTop is ∼15% relative to IceCube.
IceCube The 80 strings of IceCube will require 400 (240, 160) DOM Hubs in the counting house under the assumptions that a DOM Hub has 6 (10, 15) channels (two DOMs per channel in IceCube) and is housed in 1U mechanics. The DOM Hubs and 80 (perhaps less) associated String Processors will require something like 10 (6, 4) oor-to-ceiling racks. The space-saving that would be obtained with the 10- or 15-channel version is substantial, and provides ample motivation for high design density.
IceTop IceTop, described in detail elsewhere in this document, will consist of an ensemble of frozen water tanks situated just below the surface, and located near each of the 80 strings. Each string will have two tanks separated by a few meters to suppress single muon events, yielding a total of 160 tanks. To extend the dynamic range, each tank will be viewed by two DOMs operated at substantially different gain settings. This facilitates the capture of extremely high-energy air showers. The local coincidence requirement will be operational between the high-gain DOMs in the two tanks. The coincidence requirement will reduce to an acceptable level the rate induced by the relatively copious flux of uninteresting single muons penetrating the tank volume.
With 160 tanks and two DOMs/tank, IceTop's 320 DOMs will bring 320 twisted pairs (in 160 twisted quads) into the IceCube/IceTop counting house. These IceTop quads will plug into 54 (32, 22) DOM Hubs dedicated to IceTop, if the DOM Hub supports 6 (10, 15) channels. The IceTop DOM Hubs will connect to a dedicated IceTop Global Trigger CPU and Event Builder, with a LAN architecture analogous to IceCube. The output data stream will be linked to IceCube by both master clock times and GPS time units for online and offline analysis.
The system design described in this document is based on a loosely-coupled collection of subsystems whose operations are, for the most part, decoupled. One advantage of such a design is that it allows some independence in the design and implementation of individual sub-systems. However, in order to co-ordinate these elements into a single, coherent system, some degree of overall control is required. This function is provided by the experiment control system.
The IceCube detector can, at any given moment, be described as being in one of several operational states. These states serve as short, meaningful descriptions of just what operations are being performed by the detector. By enumerating which operations are allowed in which detector states, this state model also serve as a mechanism to control and co-ordinate operations of separate detector sub-systems.
Scope of Experiment Control The detector, as a whole, can be though of as being in one of the following states: "idle," "entering data-taking state," "running," "paused," "exiting data-taking state," "calibrating" or in one of several "testing" states. Each of these states has certain implications for the operations of each sub-system. For example, if the detector is taking data, the DOM string control sub-system should not be able to alter PMT high voltage settings on an active OM. Similarly, although it is permissible to alter individual settings for an inactive DOM (i.e. an DOM not included in the global trigger's list of active DOMs), it should not be possible to initiate LED beacons on this particular DOM since resulting light would affect data taking activities. Therefore, all sub-systems refer to experiment control as the single indicator of the system's current aggregate operational state. Each sub-system is responsible for designing its internal operations and states to be consistent with the overall detector behavior described by experiment control's current state.
Common Component Control Model In addition to providing all sub-systems with a consistent indication of the detector's overall state, experiment control must, in order to initiate and control data taking, be able to request sub-systems to perform certain actions. Depending on a particular sub-system's role in the detector's operation, these actions can be relatively simple or fairly complex. For example, when beginning data taking, DOM string control will be required to log current DOM parameter settings to a local database and change its state from "idle" to "data taking." The fact that the DOM string control sub-system is in the "data taking" mode implies that it will be unresponsive to user requests to reset critical DOM parameters. Other sub-systems may require more complex responses to experiment control commands.
To the extent that the complexity of these operations is only of concern to the sub-system itself, the details of these operations should remain concealed within the sub-system's operation. Each sub-system should be able to accept a high-level request from experiment control to move into a new state. Following this request, each sub-system will respond with its new state and status. Successful transitions of each sub-system into the appropriate state will lead experiment control to indicate that the detector has taken on the desired final state (e.g. "running"). The failure of any sub-system to appropriately move to the desired target state will indicate a system problem and will cause experiment control to bring all sub-systems back to a consistent and safe state.
It should be noted that with the loosely-coupled system described here, individual control operations are best handled directly by those sub-systems involved. That is, commands and requests for monitoring and status information will be processed by the sub-systems themselves, not by some single intervening agent (e.g. experiment control). The advantage of such a design is that additions and alterations to sub-system controls and interfaces need only be implemented in one place -the sub-system itself. They do not have to be further integrated into additional supervisory programs or levels.
However, it should be pointed out that, with this implementation model, it is possible for there to be a wide and confusing variety of user interfaces for each sub-system. Even with experiment control properly playing its detector-wide coordinating role, it will be possible to implement sub-system user interfaces in many different and potentially awkward styles. A certain degree of discipline in design and implementation of individual sub-system user interfaces will be required in order to produce a consistent overall look and feel. Given the breadth of functions requiring user interfaces and the necessity for effective remote (albeit secure) access, the adoption of a web-based user interface model is preferred.
DAQ Configuration Database The DOM string control sub-system will be responsible for keeping an accurate database of DOM-related parameters for each data taking session. While this list of DOM parameters and operational settings is not yet complete, it is safe to assume that, during any given data taking session, each DOM participating in the detector array will have 50 to 100 parameters that describe its intended operation. These will include static fiducial information such as serial number, string and position location, firmware version number, etc. It will also include per session information such as PMT high voltage, trigger threshold level, and, if some level of waveform capture is included in the DOM, parameters that describe operation of waveform capture and feature extraction algorithms.
This same database will also be used to control which DOMs are actually participating in data acquisition operations for detector array and, if required, insure that unused DOMs are powered down and/or not contributing unwanted triggers to string processor trigger processors. This same database will be used by the Global Trigger and Event Builder sub-system to control operations of the String Trigger processor algorithm.
Component Discovery and Self Configuration The IceCube detector will be deployed over a period of several years. Furthermore, during the summer deployment season, optical module strings will be deployed, tested and integrated at rates far in excess of that experienced by the current collaboration. While the physical logistics for drilling, deploying and cabling detector strings are critical to the success of this timetable, the speed with which the data acquisition system can adapt and re-configure to the addition or deletion of new optical module strings will be of equal importance.
Each sub-system located at the South Pole must be aware of its environment and capable of determining whether needed hardware and software components are available. This operation can be performed automatically when individual sub-systems are started or can be the result of an explicit re-configuration command. The most basic example of such an operation is enabling the experiment control sub-system to locate, via a set well-defined network operations, and establish communications with all other operating sub-systems. A more complex, but conceptually similar, configuration operation enables the DOM string control sub-system to determine which strings are present and which optical modules are in an operable state. This resource discovery scheme can be extended to on-line sub-systems as well by enabling the filter farm manager to discover which individual processors are available to participate in its operation.
If this technique is consistently applied across all sub-systems and is combined with accurate and complete documentation of the experiment cable plant, it becomes relatively simple to incorporate both additional DOM strings and computing resources as they are installed. The presence or absence of portions of the hardware and software system become apparent to operators as each sub-system reports its success or failure during a "configure" operation. In addition to speeding integration of new portions of the detector, this capability allows fast and effective resolution of complex system-wide equipment problems. This capability can also provide critical equipment status information when recovering from an unscheduled power loss.
Just as it is critical to be able to automatically incorporate new DOM strings and computing components as they are installed, it is also important to be able, under computer control, to mark portions of the system as off-line or "non-participating." The most obvious need for such a facility is during system debugging. It is often useful to remove elements from the system in order to investigate their role in generating specific error conditions. Unfortunately, it is not always convenient or possible to electrically disconnect or power down suspected hardware or software components. Some data acquisition problems may require this sort of system de-population for correct diagnosis and repair. If these problems occur during the winter season, the ability to remove some components from the system, via remote access, will be of great advantage.
"Security" can cover a wide range of capabilities and protections. It is worth reviewing the current use of the term in computing and communications systems. In current computing usage, the term security is based on four principles: authentication, authorization, privacy, and non-repudiation. Briefly, authentication involves the verification that a user has the identity they claim to have, authorization verifies that a given user has the authorization or privilege to perform a requested operation, privacy assures users that messages and data exchanged with another user will be private to only those two parties, and non-repudiation stipulates that requests received from a user cannot be repudiated at a later date (e.g. electronic transfer of funds). Each of these principal functions can be implemented separately and each with varying degrees of completeness. Furthermore, several different technologies can be used to achieve some or all of this functionality. While simple open password schemes are not considered to provide any real level of security, suitable software mechanisms, such as DES, Kerberos, and PKI (Public Key Infrastructure), exist and are readily available.
An exact statement of IceCube security requirements does not, as yet, exist. However, based on past AMANDA experience and reasonable projections of IceCube operations, several needs are obvious. Regardless of what network access level protections may be implemented upstream, such as screening of requestor IP addresses, it is important that all control commands be verified and screened by a security service at the South Pole site. This service should provide, at a minimum, strong authentication and authorization functions. These would basically regulate who would have access to data acquisition and on-line control and monitoring information and which specific capabilities would be allowed on an individual basis. While privacy and non-repudiation functions do not appear to have any role in experiment control and monitoring at present, any suitable security service will allow their implementation at a later date. In discussing security in the context of experiment control, it is assumed that a certain level of robust security is provided by the underlying operating system and network environment. It is assumed, for example that unauthorized access to system accounts has been eliminated and that network routers and their control ports have been made suitably safe. The following discussion centers on the application design decisions that will enhance this low-level security and protect the resulting system from unauthorized or inadvertent use.
One mechanism used in designing secure control systems is to create a finite number of "roles" that users can take on when logged onto a system. These roles are typically descriptive in nature (e.g. "operator," "implementers," "maintenance technician," "super-user"). Users requesting access to the data acquisition or on-line sub-systems at the South Pole will present their name, security credentials, and intended operational role to the security service. After authenticating user identity, the security service determines whether a user is authorized to take on the requested operational role. If these security checks are successful, the user then has all the privileges associated with that operational role.
One of the advantages of this scheme is that it creates a clean partition between security services and system operations. Security mechanisms can be centralized into a single point of access. Individual sub-systems are then able to treat all commands as having been issued by authenticated users acting in an authorized capacity (i.e. role). In addition to these implementation efficiencies, this scheme also clarifies overall system design and operation. Although commands to sub-systems will carry the identity, as well as the acting role, of the requestor, the design becomes one primarily driven by the requestor's role. The act of describing the set of all possible user operational roles implicitly describes all possible modes of system operation. For example, since each sub-system differentiates between normal operations allowed by an "operator" and those privileged commands reserved for "implementers," lists of permissible "operator" and "implementer" functions take on detector-wide meaning and significance. Furthermore, since, at the sub-system implementation level, every command must be checked against the role of the requestor, sub-system behavior becomes both well-organized, easily described and, hopefully, documented.
Slow Control The DOM string control sub-system will consist of a central control program and co-operating tasks executing on each string's dedicated control processor. Since the detector will grow by the addition of DOMs organized as strings and since the only electrical and logical path for communicating with a particular OM is through its string processor and cable, this design is a natural fit for the detector's topology.
The central slow control program will be responsible for all user and experiment control interface functions to the optical modules and will be responsible for all DOM-related interactions with the detector configuration database. These database interactions will include queries of DOMs currently active in the detector, setting and recording of internal DOM parameters for data taking, and recording of individual DOM calibration information acquired during low level calibration operations. Most DOM string control operations will be synchronous command or monitoring requests.
At present, there are only a small number of possible states for the DOM string control sub-system. These states should be sufficient to indicate the following DOM string control activities: off-line or idle, performing self-configuration and string discovery, performing one or more local DOM calibration functions, on-line and able to accept operator commands, and on-line but only accepting monitoring requests (as during normal data taking). Additional states can be added as needs arise. It should be noted that these states, as well as those of other sub-systems, will not correspond exactly to those of the experiment control program. They do, however, represent a list of possible sub-system behaviors that are consistent with the overall detector operation.
The central control program will also be responsible for discovering via their associated string processors, all DOMs presently in the detector. Once connected to the local area network, newly added strings will be incorporated into the central control program through a single "reconfigure" command. This same operation will allow strings temporarily removed from the detector for purposes of test or electronics repair to be marked as "non participating strings" and their respective DOMs recorded as "off line."
String Data Handling Each DOM in a string produces a stream of hits and waveforms that, via the String LAN, ultimately reaches the string processor. All operations surrounding data transfer, sorting and interactions with other system components (e.g. the global trigger) are the responsibility of the string data handling process. The first phase of its operations center on the delivery of string level hit information to the global trigger process. Upon reception of data from one or more DOMs, its initial task to sort this data into structures that allow quick, efficient determination of local coincidence during a sliding time window (typically 8 μs). When all coincidences for a given time period have been found, they are properly formatted and passed on to the global trigger process.
The second phase begins when the global trigger process responds with a list of time intervals of interest. It is then the string data handler's task to collect raw data from all DOMs for each of the time intervals of interest and forward the resulting data set to the appropriate Event Builder. In practice, these two phases will overlap and provision must be made to have several such operations in progress at any one time.
There are only a small number of possible states for the string handler process. These states should be sufficient to indicate the following activities: off-line or idle, performing self-configuration and DOM discovery, processing normal data triggers, paused due to operator request or error condition, and performing string specific tests. Additional states can be added as needs arise.
Global Trigger As described above, the global trigger is responsible for generating lists indicating the occurrence of detector-wide events of interest. It consumes the list of hits as determined by the string data handling process in each string processor and emits a list of UTC-based times that bracket hits considered to be possible events. The global trigger behaves synchronously with the aggregate collection of on-line string processors. That is, the global trigger waits until all string processors that are configured and active to report their contributions for a given time interval. Only when all string contributions are present, can the global trigger begin the process of determining what events might have occurred. Once this determination is complete, the Global Trigger selects an Event Builder for the current data sequence and passes a list of UTC time intervals to both the selected Event Builder and all string handling processes.
During normal operation, all 80 string processors will remain synchronized within the reporting period specified when data taking was initiated. This period can range from as low as ∼100 ms to perhaps a few seconds, without degradation of system performance or compromise of data integrity. Several global triggers may be found within one reporting period. As described above, for each reporting period, ALL string processors must forward string-level local coincidences, or their absence, to the global trigger in order to initiate the global triggering process. In the absence of such a report, the global trigger will re-synchronize all string processors and forward, to the Event Builder an indication of what reporting periods were discarded. Persistent failure of a string processor program or CPU will cause the global trigger manager to enter a non-operating error state. Experiment control will detect, during periodic system scans, this state change and force an abnormal end of run sequence for all data acquisition sub-systems. Co-ordination activities and generation of global triggers are performed by a single global trigger process executing on a dedicated processor. Since this activity requires the exchange of multiple messages between this process and each string's data handling process during each reporting period, high speed, low latency network connections between these processors is of great importance.
There are only a small number of possible states for the global trigger sub-system. These states should be sufficient to indicate the following activities: off-line or idle, performing selfconfiguration and string processor discovery, processing normal data triggers, paused due to operator request or error condition, and performing global trigger specific tests. Additional states can be added as needs arise. As with other sub-systems, it should be noted that these states need not correspond exactly to those of the experiment control program. They do, however, represent a list of possible sub-system behaviors that are consistent with the overall detector operation.
Event Builder The I/O requirements for IceCube are not very demanding, opening up many possible approaches to event building. We opt for a simple yet flexible approach based on commercial hardware. Its main elements are a farm of the order of five "event worker" PCs connected via a switched ethernet network and a control PC that manages the farm and distributes the work. This asynchronous approach offers a redundant, fault-tolerant, reliable and scalable solution that should work with minimal maintenance effort. The amount of CPU time needed for event building has not yet been studied in detail but is estimated not to be the defining factor for the size of the farm. The number of PCs is rather determined by the requirement of redundancy. To exploit the CPU capacity, several tasks will be incorporated in addition. We currently envision the following set of tasks:
A special control PC is used to control event building by selecting specific event worker PCs that are ready to accept data, usually in a round-robin way. In addition it will be used for configuration, system and data supervision and will host a graphical user interface. If a PC has crashed, the load is taken over by the other PCs in a dynamic way and the PC is question is automatically rebooted. All event working PCs's have identical functionality. Their number is completely scalable and can be adjusted according to needs. The buffered string-wise data will be "pushed" to an idle event worker PCs roughly every 10–100 s. The Control PC also provides a list of links to data belonging to the same trigger time stamp. In this way only pointers are modified and the PCs are running in an "I/O-free" mode. The data are actually only moved in the last step before writing the data to disk. Such a farm can be built based on today's standard networking equipment.
Calibrations serve a crucial and central role in the operation of the IceCube detector. They must, to the greatest extent possible, be automated. The present digital design allows calibrations to be carried out without any physical intervention to the data acquisition system. That is, all calibrations can be performed without disconnecting cables or physically re-configuring DOM electrical data paths. However, while this is a necessary condition for fully automated, unattended calibration operations, it is not a sufficient one. Further design and detailed descriptions of operational sequences are still required for automated operations to become a reality. While the following framework should allow such operations to take place, further discussions are needed in order to insure their successful operations within the current design.
DAQ Calibration Operations The primary calibration operation carried out within the DAQ system is that of synchronizing time bases for all DOMs with the GPS clock. This operation will take place automatically once every 5 to 10 seconds. The actual nanosecond-level signaling operations are performed by the DOM Hub and DOMs themselves. The DAQ's only role is in enabling such operations and supervising the dissemination of the calculated clock offsets for each DOM to both the DAQ configuration data base and, through an as yet unspecified mechanism, the Online system. The primary DAQ component involved at the supervisory level is the string processor. Since all data leaving the string processor is corrected to UTC, this is the rational point at which to control and implement needed time base corrections. All other calibration IceCube calibration operations are based on analysis of data collected by a carefully configured, but normally operating DAQ. Therefore, these calibration operations fall outside the scope and capability of the DAQ and are, more properly, Online functions.
Online Calibration Operations In addition to the time base synchronization calibration performed by the DAQ system, additional calibration operations are required for successful detector operation. These include, but are not limited to, determination of individual PMT high voltage settings, determination of proper PMT trigger thresholds and measurement of cable delays for all DOMs. As noted above, all of these calibration tasks are based on specific data analysis applied to the detector data stream. Therefore, these calibration operations are primarily operating in the Online domain. In fact, they can all be considered to be part of a special subset of normal data taking operations. It should be noted that some of these Online calibration operations will produce data required for correct operations within the DAQ system. In these cases, mechanisms must be in place for inserting these calibration values into the DAQ configuration data base.
In practice, these operations can be automated and orchestrated by the main supervisory program, experiment control. Experiment control can select a particular DAQ configuration and instruct DAQ to begin operations. It can then select a particular Online configuration and instruct Online to begin operation. With a suitable level of flexibility at the experiment control level, fairly complex sequences of calibrations can be carried out in an automated manner.
Operations and Error Logging The design described in this document partitions system behavior into a number of low-level tasks and assigns them to individual sub-systems. Furthermore, these sub-systems are organized together through a single experiment control sub-system that is able to orchestrate their operations into a single co-ordinated activity. While such a design has been proven to be both flexible and reliable, when problems do occur, the relatively "loose" connection between subsystems can make proper diagnosis difficult. A well-designed operations log is a key tool in allowing rapid discovery and correction of individual sub-system problems.
All sub-systems will be able to log messages to a single, system wide operations log. Each message will be tagged with the name of the sub-system as well as the time of entry. Each message will also include numerical and text fields to describe error conditions as well as global state information such as experiment control run number and state. While there will be provision for one or more free form text areas, the format will include mandatory fixed format fields to facilitate automated searching in one of several modes. At a minimum, the log will be searchable by sub-system, time and global run number. In practice, this operations log may be built on existing open-source or commercial database technology.
DAQ and Operations Monitoring It is anticipated that, at least for purposes of detector and data acquisition system monitoring, there will be a number of groups interested in having independent and uncoordinated access to one or more software sub-systems at the South Pole. As described earlier, design distinctions between monitoring requests and control commands will allow the use of robust, commercial-grade web server programs for a large portion of these user interactions. Routing requests for monitoring data through a web server concentrates most of the interactive data flow through a well-debugged program that acts as a buffer between off-site data requests and data acquisition and on-line sub-system programs. Unfortunately, this mechanism is still subject to periodic and occasionally unscheduled communications outages between the northern hemisphere and the South Pole. Given the wide distribution of local time zones within the IceCube collaboration, this can create limited or, at a minimum, inconvenient access to monitoring data.
With the interposition of a northern hemisphere server/cache gateway system, much of the data acquisition and on-line sub-system monitoring data can be made available to collaboration members at their home institutions during reasonable day-time hours. This gateway system would be configured to refresh its cache from sources at the south pole during times of expected communications availability and serve these monitoring data files in response to browser and client requests. During times of communications blackout between the south pole and northern hemisphere, collaboration members would be able to direct their browsers and client programs to the gateway system and obtain current, within 24 hr, detector and sub-system status. Furthermore, if it were deemed necessary to restrict general collaboration access to South Pole systems, this architecture would provide a straight forward mechanism for distributing detector monitoring information. Thus, up-to-date detector and data acquisition system performance data is made available collaboration-wide on a best-effort basis.
Portions of the monitoring and operational information will be needed for correct interpretation of data during later analysis. This can include current and cumulative trigger rates for all trigger types as well as Event Builder statistics and gross DOM data rates on a string by string basis. There must be a mechanism for identifying and passing monitoring data from the DAQ system to the Online system. This may consist of a separate data stream, or even the periodic inclusion of monitoring data in normal data streams.
Online Components The Online system is essentially a carefully organized portion of the Offline environment that has been targeted for execution at the data acquisition site under the control of the experiment control sub system. It has three primary responsibilities. First, it is the sole component responsible for copying the DAQ event stream from the shared disk system, located on the Online LAN, to both archive storage (i.e. tape) and, via satellite, to the northern hemisphere. As such, it is capable of reformatting and repackaging data from the Event Builder into those forms best consumed by offline analysis tasks. Secondly, it provides the framework and control mechanisms for executing any online filter operations used to analyze, reduce or reject data from any of the data streams emitted by event builders. And thirdly, as described above, the Online system contains filters and or programs that perform all calibration functions beyond those of DOM time base synchronization performed by the DAQ system
COTS Hardware and Software Components There has been a real effort, during the design of both the DAQ and Online systems, to maximize the potential for using commercial, off the shelf (COTS) hardware and software components. At a minimum, this includes use of open source operating systems (namely, Linux), standards-based network and communications software and PC-based computing platforms. Of course, we feel it also should allow for the purchase of standards-based components as required. This can, for example, include purchase of PC clusters that have well engineered cooling and maintenance features or disk systems that have extensive space management and diagnostic software packages.
String LAN Components As noted earlier, each string terminates in its own private network. In practice, this network consists of a set of DOM Hubs and a single processor connected to an off-the-shelf switched Ethernet hub. This design nicely encapsulates the entire string behind each string processor, which serves as the complete interface to the string. Based on system performance, we may choose to have two or more strings share the same string processor. If so, we need only increase the number of Ethernet attachments required for the commercial switched hub.
String, Global Trigger, and Event Builder Processors Depending upon the degree to which multiple OM strings can be accommodated on a single string LAN and processor, the IceCube DAQ, when fully implemented, may have on the order of 50 to 100 processors. By any measure, this is a substantial number of systems to house and maintain. When designing a system with such a large number of processors, several factors besides basic unit processor cost must be considered, such as operation in a low humidity and low pressure environment with an occasionally unreliable power source. Many vendors have addressed the ancillary issues involved in configuring, operating, and maintaining large clusters of PC-based Linux systems-most notably for use in the e-commerce industry. For these markets, vendors have addressed such issues as increased packing density (e.g. one processor per 1U rack space), improved ventilation and temperature monitoring, improved, tool-free, modular repair capabilities, and network-based remote diagnosis and control of individual processors. These systems typically have a full, but limited set of disk, communications, and PCI resources. In determining the functionality and interconnections needed to implement both the DAQ and Online systems, care has been taken to stay within the physical and logical constraints presented by these PC cluster implementations.
Event LAN The data rates presently anticipated indicate that the communications network used to connect all OM string processors with the single global trigger processor and multiple Event Builder processors can be adequately serviced by conventional data network (100BT or Gigabit Ethernet). As with the String LANs, this network is well bounded and only requires a local switch-based infrastructure.
If these data rate projections grow beyond the capabilities of either 100BT or Gigabit Ethernet technologies, alternate, off-the-shelf solutions can be inserted into this architecture without major perturbations to the system design. In particular, we have investigated the Myricom switched fabric inter-processor network as a potential solution.
Storage Systems and Online LAN As with network infrastructure, storage systems represent an area where system data rates dictate the technical solution. The primary storage facility shown in this design is that which provides the flexible data buffer between the DAQ and Online system partitions. This system design effectively increases performance by providing parallel computing resources for those paths with the highest data throughput rates. At the juncture of the DAQ and Online systems, this implies a shared network-based storage system with multiple access paths to both Event Builder and online processors. The current aggregate data rates indicate that a 100BT based commercial SAN (Storage Area Network) system will meet the necessary storage and throughput requirements. As with the selection of processors, storage system selection should not be based solely on hardware costs. Most modern commercial systems include system level features such as automatic mirroring, local automated backup and remote fault diagnosis and configuration tools. These facilities should receive high consideration in vendor and system selection.
As with the Event Builder LAN, the technical solutions are sensitive to the ultimate data rates expected within the system. If the current projected data rates prove inadequate, consideration will be given to their effect on the I/O bandwidth requirements of the shared disk system. Alternate solutions with substantially higher throughput rates do exist - most notably fiber channel based SAN arrays. Since these storage systems use individual PCI interfaces and a separate switched fabric interconnect for all disk related data, the costs - as well as the data throughput - increase dramatically.
