PDD - Digital Optical Module

By Section | Whole Paper | PDF (233 pages, ~6.62mb)

• 7.1 Overview
• 7.2 Digital Optical Module
  • 7.2.1 Pressure Housing
  • 7.2.2 Optical Sensor
  • 7.2.3 PMT HV Generator
  • 7.2.4 Optical Beacon
  • 7.2.5 Signal Processing Circuitry
  • 7.2.6 Local/Global Time Transformation
  • 7.2.7 Cable Electrical Length Measurement
  • 7.2.8 Data Flow and Feature Extraction
  • 7.2.9 Local Coincidence
  • 7.2.10 System Design Aspects
• 7.3 Network
  • 7.3.1 Copper Links
  • 7.3.2 Time-Base Distribution
• 7.4 Surface DAQ
  • 7.4.1 Overview
  • 7.4.2 DOM Hub
  • 7.4.3 String Processor
  • 7.4.4 IceCube and IceTop System Integration
  • 7.4.5 Experiment and Configuration Control
  • 7.4.6 Security Environment
  • 7.4.7 DAQ Components
  • 7.4.8 Calibration Operations
  • 7.4.9 DAQ and Online Monitoring
  • 7.4.10 DAQ Computing Environment
• 7.5 AMANDA Data Transmission Techniques

7.2 Digital Optical Module

The Digital Optical Module (DOM) is a self-contained data acquisition platform that is capable of capturing and digitizing real-time PMT pulses, storing data internally, and, when requested, transmitting them to a surface data acquisition (DAQ) system. It contains its own processor, memory, flash file system, and real-time operating system (RTOS). The DOM is capable of scheduling background communications tasks while acquiring data from a local waveform digitizer. All calibration functions needed to interpret data for physical content may be invoked under software control; some operate as scheduled tasks without interfering with data acquisition, and others require dedicated control of state.

A Field-Programmable Gate Array (FPGA) is used to implement most low-level logical functions of the DOM, facilitating fully programmable and flexible implementation of performance features. The FPGA is programmed automatically from a configuration file that the CPU retrieves from flash memory in the DOM after power-up and self-boot has occurred. The FPGA design and CPU programs can be replaced at will subsequent to deployment to enhance functionality. This capability has been exploited already in the AMANDA string 18 (DOMs only), using the satellite link to the south pole from the northern hemisphere.

The basic elements of the DOM are the optical sensor for Cerenkov light, an electronic circuit board for processing signals, a HV generator for the optical sensor, an optical beacon for calibration purposes, and the glass pressure housing. The general physical layout of a generic DOM is shown in fig. 62.

Figure 61: Block diagram of the "Test-Board" used in the AMANDA DOM string DAQ system.

Figure 62: Schematic profile view of a generic Digital Optical Module, showing pressure sphere, optical coupling gel, PMT, signal processing electronics board, LED flashed board, PMT base, and electrical penetrator.

7.2.1 Pressure Housing

The spherical glass pressure housings are standard, well-proven items of commerce, used in numerous oceanographic and maritime applications. For IceCube, the most important qualities are mechanical reliability, cost, optical transmission, and potassium content. Commonly known in the US as Benthospheres, a trade-name of Benthos, Inc., the dominant US supplier, these simple devices have an excellent record for reliability. Over 700 of these, from both Billings and Benthos, in various implementations and sizes, have been deployed in AMANDA without evidence of a single implosive failure. To ensure reliability against implosion, care must be taken not to chip either the surfaces of the hemispherical rims that form the equatorial ground-glass seal, or the holes that permit penetrators to carry signals and power across the boundary. Costs for standard pressure-tested Benthospheres are modest, in the range of $350.

For IceCube, a 13 in diameter sphere is the optimum size. While larger spheres are available that could a accommodate larger optical sensor, the costs for drilling larger diameter holes (with an appropriate margin of safety during deployment) rise very quickly, making larger sizes a poor overall trade-off.

Because the Cerenkov radiation mechanism produces the greatest intensity at ever-smaller wavelengths, limited ultimately by self-absorption, it is of interest to push the optical transmission limit of the glass to a value below the quantum efficiency limit of the PMT. For processing and mechanical strength, the manufacturers use a borosilicate glass that limits transmission to about 350 nm, well above the expected ∼220 nm limit for deep polar ice. It is conceivable that an exterior surface layer of an appropriate wavelength shifter for the component shorter than 350 nm could increase the effective light yield, and this is being explored.

The potassium content of the typical borosilicate glass includes a naturally occurring fraction of ⁴⁰K. The β particle from the decay of this isotope produces Cerenkov radiation at a level that dominates the observed PMT noise rate, unless steps are taken to reduce the overall potassium content of the glass. At the request of IceCube collaborators, manufacturers have been able to produce glass with substantially lowered potassium content without compromise to important processing, mechanical, or optical qualities. It is anticipated that IceCube optical modules will display noise rates very close to those of isolated dark-adapted PMTs.

7.2.2 Optical Sensor

The optical sensor is a medium-size (∼10 inch diameter) hemispherical 10-stage photomultiplier tube (PMT), made by Hamamatsu. PMTs very similar to the intended device have been deployed within AMANDA with generally excellent experience. These large PMTs offer surprisingly good time-resolution, as indicated by a transit-time-spread (TTS) for single photo-electron (SPE) pulses of about 2.5 ns rms. Despite their large photocathode area, these PMTs, in total equilibrated darkness, generate only ∼300 Hz or less of spontaneous noise pulses at temperatures less than 0°. Photocathode sensitivity extends well into the UV, limited by the optical transmission of the glass pressure sphere at 350 nm. For each detected photon, the PMT produces SPE pulses that have characteristic rise (fall) times of 7 (11) ns.

Due to the stochastic nature of the electron multiplication process, SPE pulses display significant variations in pulse shape and amplitude. The measured pulse area distribution of DOM #1 (the deepest), acquired over a period of 10 days, is shown in fig. 63. The histogram is formed by adding together consecutive ATWD samples for the pulses that rise above a digital threshold, and subtracting a measured pedestal. This simple method works well enough if the discriminator threshold that launches the ATWD is sufficiently low that nearly all SPE pulses result in a trigger. It should be noted that a fair fraction of the PMTs in string 18 do not have such good peak to valley ratios, and the spectrum shown in fig. 63 is better than most.

Figure 63: Measured pulse area distribution of DOM #1 measured over a period of 10 days in CY 2001. Most of hte histogram is SPE noise pulses. The stochastic nature of the electron amplification process leads to a very broad distribution such that some SPE signals fall below threshold.

To define a measure of gain, the peak region of spectra such as that shown in fig. 63 are fit to a simple a Gaussian form. From the fit parameters, an average gain can be extracted. Repeated measurements taken over 180 days in CY 2000 have demonstrated that the combined PMT + DOM signal processing electronics provide excellent gain stability. These results are shown in fig. 64. Because PMT gains were set to optimize the optical signal paths, which had wide variations, and because some PMTs displayed poor peak-to-valley ratios, high quality fits could be obtained for less than half of the deployed ensemble of DOMs. The mean of the distribution, ∼ -2,8% per year corresponds to drifts of 0.05% per week, far below the experimental requirement of 2% per week; none of this subset comes close to 2% per week.

Figure 64: Measured distribution of a subset of DOM gains, recorded over a period of 250 days in CY 2000. The mean in this ensemble is less than 3% per year, with none approaching the experimental requirement of ≤2%/week.

At a gain of 10⁷, a typical SPE pulse will have an amplitude of ∼8 mV into a 50 Ω load. Under these conditions, the signal displays some nonlinearity at around 2 V, reaching complete saturation at ∼4 V. The effective dynamic range under these conditions is approximately 500 PE. The PMT maximum output current at the anode is limited by space charge effects in the dynode chain. As signal size increases, the space charge effects appear first at the last dynode, then at the next to last, and so forth. Not only does the pulse amplitude display a non-linear relationship to the input photon signal, but the pulse shape begins to stretch out considerably. The effective dynamic range may, in principle, be increased by lowering the PMT gain below 10⁷. At some point in the range of a few millivolts, intrinsic electronic noise will degrade performance at the SPE level.

With careful circuit design and layout, however, ultimate performance may exceed 500 PE significantly. It seems feasible to utilize a dynode tap to obtain a signal that could extend the dynamic range at least another order of magnitude beyond what can be practically achieved from the anode signal. This general idea has been used in many past experiments, and should be practical for IceCube. This possibility is being explored actively.

7.2.3 PMT HV Generator

The PMT HV generator, known familiarly as the PMT base, is a multi-stage voltage multiplier, or Cockroft-Walton circuit, with an oscillator running at a few tens of kHz. The base must meet a number of important requirements, in particular, high reliability, high stability and low induced noise. A voltage multiplier has certain potential advantages since the voltage steps are smaller than that of a single stage multiplier. If arranged correctly, the base will naturally supply the most current to the dynode stages near the anode, where the pulse loading occurs. This produces an attractive low power design. On the other hand, there are more parts, which could affect MTBF.

The IceCube PMT base will have two separate sections. One section, supplying the potential difference between the photocathode and the first dynode, will be fixed at a voltage near the manufacturer's indicated maximum. This produces a uniform transit-time among the DOMs, and produces the best peak-to-valley ratio for the SPE pulse-height spectrum. The second section will be variable under software control, to obtain the desired 1 × 10⁷ gain. Experience with AMANDA string 18 shows that reliability is an issue, as two bases appear to have failed. All HV designs involving semiconductors must be based on conservative design practices, since even extremely rare discharge mechanisms can be fatal. Induced noise also appears to be present, at least in the timing measurements, which give better results with the PMT HV off. The re-engineering of the PMT base to improve reliability is recognized as an important task.

7.2.4 Optical Beacon

The string 18 DOMs are each equipped with six GaN LEDs, which emit predominantly in the near-UV at 380 nm. The luminous intensity and the pulsing rate may be varied over a wide range under software control. At their brightest, these beacons can be seen by OMs 200 m distant. Due to their high intrinsic capacitance, LEDs are not easy to pulse at ns speeds; the pulse width is ∼5 ns. The LEDs are broad angular emitters, spaced at 60° around a vertical axis, and are canted over to produce a roughly hemispherical source. Some AMANDA analog OMs have been equipped with similar optical beacon boards using GaN LEDs emitting at 450 nm; these have also proved to be useful for test purposes. The PMT HV must be turned down at the emitting DOM to avoid potentially harmful pulses.

The optical beacons can be used to knit together a highly over-constrained measure of relative DOM positions within the array. They can also be used to study optical properties of the ice at these wavelengths.

7.2.5 Signal Processing Circuitry

The block diagram of the DOM signal processing circuitry is shown in fig. 65. The principal logical and functional elements are:

• State control;
• Waveform capture and digitization;
• Local time generation;
• Local/master clock time transformation;
• Cable length measurement for timing offsets;
• Data flow;
• Local coincidence capability;
• Control and glue logic, provided by an FPGA;
• A low-power 32-bit ARM CPU for higher level operations;
• Power conversion and filtering;
• Monitoring and calibration;
• Message handling - send/receive .

State Control The DOM is operated as a slave in a master/slave relationship with the surface DAQ. Upon power-up, it enters a wait state for an interval of a few seconds, to permit downloading of new firmware or software. If no messages are received before time-out, the DOM will boot from Flash memory and await commands. Normal operation of the DOM is dominated by a data-taking state.

In addition, a local-time calibration process is periodically invoked, so that the local DOM time may be ultimately transformed to master clock time with nanosecond accuracy. The time calibration process appears as separately scheduled thread, with no deleterious impact upon the data-taking state. Other active threads are message receive/send and data transmission, also invisible to the data-taking process. There are several calibration modes, such as running optical beacons, cable length measurement, etc., during which normal data-taking is presumably inactive.

Figure 65: Block diagram of the digital optical module signal processing circuitry.

Information Capture The DOM is equipped with innovative circuitry that is well-matched to the PMT pulse characteristics and dynamic range. The capture of waveforms with ∼300 MHz 14 bit resolution is a daunting technical challenge if only conventional flash ADCs are considered. Power dissipation for the ensemble of FADCs would be several watts, imposing an undesirable operating condition; noise due to the large amount of digital activity and data flow presents an inhospitable engineering environment. Instead, the DOM concept takes advantage of the fact that, most of the time, nothing is happening: Δt between pulses is more than 10⁶ ns. Circuit activity need occur only when pulses appear.

The waveform capture capability is realized through a custom Application Specific Integrated Circuit (ASIC) designed at Lawrence Berkeley National Laboratory (LBNL), the Analog Transient Waveform Digitizer (ATWD). The ATWD has four channels, each with 128 samples, that synchronously record different input waveforms. For IceCube, three of the channels will capture the PMT signal presented at three different gain settings. This arrangement provides an elegant multi-range linear PMT signal capture method. With three channels operating at ×15, ×3, and ×0.5 gains, the ATWD provides the rough equivalent of a ≤14 bit 200–800 MHz ADC. Due to schedule pressures, the DOM prototypes in string 18 were equipped with only ×10 and ×2 gain channels. The third channel was, instead, routed to an analog multiplexer that could connect to various internal signals for diagnostic purposes.

The fourth ATWD channel in string 18 DOMs is permanently connected to the DOM internal clock, to calibrate/monitor ATWD sampling speed. Experience with string 18 has shown that the ATWD sample speed is extremely stable. In IceCube, the fourth channel will be connected to an analog multiplexer to permit that channel to share more than one function. The ATWD sampling speed needs to be calibrated and monitored, but only rarely after initial calibrations are completed and the system is stable. As noted above, some other internal signals that have diagnostic value also need only infrequent interrogation. Most of the time, ≤99.9%, the fourth channel is available to capture a dynode tap signal that could increase the PMT signal effective dynamic range by at least another order of magnitude. The goal is to reach at least 5000 PE, a capability that will enhance the reconstruction of EHE events, should they occur within the volume of IceCube.

The capture process is initiated by a "launch" signal derived from a discriminator connected to the high-gain signal path. The capture process stores waveforms as analog voltages on four internal linear capacitor arrays. The sampling action is generated by an active delay line within the ATWD, with "look-ahead" to create a sampling gate of width sufficient to obtain adequate settling on a capacitor before the gate switch opens, effectively closing that sampling window.

With the ATWD, waveform capture occurs at sampling speeds that may be varied at will from ∼200 Msps to more than 1 Gsps, corresponding to capture intervals of 640–128 ns. It appears that capture at ∼300 MHz (3.3 ns/sample) is appropriate for our scientific goals. The higher speed capture is useful for characterizing PMT signal shape and other diagnostic purposes, but is not needed to extract timing information from a typical waveform. A single DC current controls the sampling speed. All channels within an ATWD sample synchronously, with aperture variations between channels measured to be less than 10 ps.

The ATWD performance combination of ∼GHz sampling and very low power dissipation, ≥100 mW is unmatched by any single commercial device for transient waveform capture. The sampling action is controlled by an active delay line internal to the ATWD. No ultra-high speed logic/clocks are needed in the DOM, a highly desirable engineering situation. The maximum frequency in the string 18 DOMs is the local clock frequency, a very comfortable 33.6 MHz. For IceCube, this frequency may be higher, but will likely not exceed 40 MHz.

Two ATWDs, arranged with alternate selection logic, permit capture of longer waveforms, and also serve to reduce dead-time. For waveforms exceeding the ATWD capture interval, sampling at a lower frequency is sufficient to capture relevant information at later times. For this purpose, The DOM is equipped with a 10-bit low-power FADC, operating at the DOM local clock frequency. The PMT signal in this path is reshaped to match the lower sampling rate.

The ATWD has two distinct functions{transient waveform capture, and digitization of captured signal. These two functions are separately controlled, so that a captured signal subsequently judged uninteresting may be ignored. This feature minimizes dead-time, and also facilitates a true local coincidence capability (see sec. 7.2.9) between neighboring DOMs, in which only time- and space-associated signals are digitized. The immediate conversion to a digital format preserves data quality during subsequent DAQ data flow processes.

For interesting events, waveform capture in the ATWD is followed by 10-bit simultaneous conversion of all 128 samples/channel by an internal common-ramp Wilkinson ADC. The internal scalers count on both positive and negative clock transitions. At 33.6 MHz, complete digitization requires about 15 μs. Because of natural CMOS manufacturing variations that create offsets in the internal comparators, there exists fluctuations in the baseline, or pedestal, of the captured waveform. These pedestal variations are in the range of 5–15 counts; they are quite stable and are easily subtracted away during data processing.

A single ATWD digitizing waveforms at the nominal noise trigger rate of 300 Hz, will introduce deadtime of ∼0.5%. To this, an additional deadtime of about 0.1% must be included for readout of data via a 10-bit parallel bus. With two ATWDs operating in ping-pong mode, deadtime due to random noise hits is less than 1 × 10^-4.

The ATWD is currently designed for a 1.2 μm CMOS process with double polysilicon capacitors. This manufacturing process may not be available indefinitely, as new CMOS processes with ever-smaller feature size and other characteristics supplant older ones. The ATWD should be updated by not only choosing an appropriate one of these processes, but by increasing sample depth to perhaps 256 instead of 128, and adding new features such as a built-in DAC to reduce ADC pedestal offsets to an insignificant level. For IceCube, no aggressive design steps would be contemplated, only those for which substantial experience already exists in other ASICs.

Local Time Generation Each PMT hit must be time-stamped such that correlated hits throughout IceCube meet the relative timing accuracy requirement of 5 ns rms. This is accomplished within the DOM using a two-stage method that produces a coarse time-stamp and a fine time-stamp, both in local time units. Once data reach the surface, the local time units are transformed to master clock units, and ultimately linked to GPS.

Although timing accuracy and resolution are in the few nanosecond range, no high-speed clocks are needed anywhere in the system for the time measurement process. The current string 18 DOM design maintains a 54-bit local clock running at the DOM 33.6 MHz clock frequency. The 33.6 MHz is obtained by frequency-doubling a free-running Toyocom 16.8 MHz quartz oscillator. For IceCube, as noted earlier, the local clock frequency may be higher.

This oscillator product displays exceptionally stable behavior: measured drifts (Allan variance) taken in the laboratory are typically δt ∼ 0.06 ns/s, i.e. δƒ/ƒ ∼ 60 × 10^-12, as shown by the histogram of 46 of 49 tested oscillators in fig. 66. The three oscillators not included in the histogram were markedly worse, but still within the manufacturer's specifications. A test program will select the best 90–95% of the devices. Each point in the histogram represents the analysis of 720 measurements taken consecutively over an hour (5 s intervals). The samples in this histogram were taken after a stabilization period of a few days, as it is well known that quartz oscillators will display asymptotically better stability after an initial run-in to shake off dust, relieve strain, etc. This extraordinary stability obviates the need (and perhaps even the practical possibility!) for a phase-locked loop connecting the local and master clocks. Such short-term stability is only an order of magnitude or so worse than that of high quality commercial rubidium stabilized clocks. The cost of the Toyocom oscillator is ∼$25.

The coarse time-stamp process begins as a PMT signal triggers the discriminator. The discriminator pulse is then resynchronized to the next DOM clock transition edge. The resynchronized pulse provides the ATWD "launch" signal. In other words, the synchronous launching of the ATWD causes the PMT signal to arrive within a one-cycle wide region of the ATWD capture window. The clock value at the instant of ATWD launch is recorded in an appropriate FPGA register. The "coarse time-stamp" is defined as the clock value recorded in this register. At 33.6 MHz, this coarse time-stamp provides a time quantization Δτ of ∼29.76 ns (at 40 MHz, Δτis 25 ns). Because the register is loaded by the same signal that launches the ATWD, the connection between coarse-time stamp and ATWD launch is robust.

Clearly, the coarse time-stamp does not provide the desired ∼5 ns resolution. However, the leading edge of the PMT signal waveform within the ATWD record will occur somewhere within this particular 29.76 ns interval. The "fine time-stamp" is defined as the waveform position within the ATWD record. The measure of the fine time-stamp must be extracted from the ATWD record by an algorithmic method. A simple extrapolation to baseline of the waveform leading edge can serve as the measure of the fine time-stamp, although more sophisticated methods can be easily envisaged. Even with the small SPE signals, the contribution to timing resolution due to all electronic noise sources and digitization is expected to be less than 1 ns rms, smaller than the PMT transit time jitter of 2.5 ns rms.

For all this to work, the PMT signal must be delayed sufficiently such that the leading edge of the signal always appears well after the ATWD sampling action has begun. An overall delay of about 75 ns is needed to accommodate both the coarse time-stamp Δτ random delay and the propagation delay of the circuitry. In string 18 DOM prototypes, a coil of coaxial cable (∼50 ft) was employed for this purpose, an effective but bulky solution. For IceCube DOMs, a lumped delay line may be chosen. These provide adequate performance and offer an attractively compact solution; of course, reliability of this device (as well as all other parts) at low temperatures must be demonstrated.

Figure 66: Histogram of measured Allen variance for 46 Toyocom oscillators, measured at 5 s intervals relative to a good frequency standard. Some of hte outliers above 100 ppt later stabilized to fall within the trend.

Our observations confirm that standard commercial GPS clocks are subject to much larger short-term drifts than the Toyocom devices themselves. Figure 67 shows measured drifts of one DOM local clock (in the ice) showing a 3.5 ns short-term variance between measurements taken at 2.6 s intervals relative to GPS. In comparison, the Toyocom oscillators displayed about 50 times better stability than this during laboratory measurements taken relative to a good frequency standard (a Stanford Research Systems Loran system). Clearly, standard commercial rubidium clocks, which typically offer δƒ/ƒ ≥2 χ 10^-12 per 100 s, are a much superior choice for the master clock. The link from master clock to GPS can be easily made once per second with standard hardware and software.

7.2.6 Local/Global Time Transformation

All DOM data, recorded in association with the free-running local clocks, must be transformed to IceCube master clock units in order to reconstruct events. Because drift is negligibly small over a period of seconds, a simple linear transformation between local and master time is sufficient. This could be performed within the DOM or, more likely, at the surface within the DAQ. In either case, this transformation requires continuously updated knowledge of local clock frequency and phase, relative to the master clock. Because the Toyocom oscillators are so stable, the local/master clock calibration process needs to be invoked only a few times/min per DOM. Consequently, this vital function requires only a tiny fraction of the network bandwidth.

The master clock units are linked to GPS time once per second in the online DAQ to permit the connection of IceCube events with other detectors, and offline comparison with potentially interesting astrophysical occurrences. Interpolation within the one second interval is expected to be better than ±1 ns rms. Synchronization of event time with other laboratory detectors, terrestrial or non-terrestrial, will require detailed accounting of delays throughout the as-installed IceCube DAQ and master clock system. The ultimate accuracy for inter-laboratory timing cannot now be specified but may reasonably be expected to fall in the range of ±10 ns.

The calibration process involves a periodic transmission to a DOM by the surface DAQ of a simple bipolar time-mark signal, perhaps 5 μs between edges, synchronized precisely to the master clock. These time-mark signals, however sharply defined at the moment of generation, experience degradation while propagating down the twisted pair cables due to dispersion and attenuation. Propagation over 2–3 km of cable results in rise-times of roughly 1.8–2.4 μs. These values depend somewhat on the definition of rise-time, since the 1/t asymptotic approach is very slow. The propagation of signals such as step functions over cables such as these is very well understood. In most respects, they can be modeled as delay lines with ∼10:1 ratio for delay/rise-time. This is schematically illustrated in figure 68.

Figure 56: Measurement of Toyocom/GPS frequency ratio at 2.6 second intervals for 15 minutes. The ∼3.4 ns rms is the product of the distribution sigma value times the measurement interval, i.e., 1.3 x 10^-9 x 2.6 seconds dominated by short term drifts in the GPS unit.

Figure 68: Transmission of a bipolar time-mark signal synchronized to the master clock by the surface DAQ to the DOM provides the connection between local and master clock time.

The challenge is to exploit the very stable behavior of the cables and clocks to extract a timing measure with a relative accuracy not exceeding 5 ns. The term "relative" is essential, since an offset due to method, common to all DOMs within the required resolution, is irrelevant. The other essential conceptual element is a simple relationship between voltage noise during measurement and time resolution:

δt = δV/(dV/dt) (5)

The dV/dt of signals arriving at the DOM fall in the range of ∼ 10⁶ V/s. If voltage measurement noise is on the order of 1 mV, then timing resolution will be in the range of 1 ns. Clearly, environmental noise can significantly degrade time resolution, and must be controlled for this to work well. In the laboratory, using a simple discriminator and filter, resolutions less than 1 ns are easily achieved. However, this simple approach contains an undesirable flaw due to random errors in device thresholds, which may be several millivolts. According to the above equation, shifts of several nanoseconds may occur among an ensemble of DOMs because of unknown threshold offsets. The threshold offsets can in principle be measured by pulse injection, but questions linger as to validity of method due to concerns such as the integrative behavior of the discriminator coupling pulse rise-time and width to effective threshold.

On the other hand, it is completely straightforward to employ a conventional FADC and measure the entire waveform of interest, including baseline just before the pulse arrives. Digital offsets from one FADC to another become irrelevant. Most noise sources are very effectively suppressed because the baseline is measured just before the pulse appears. Defective or distorted waveforms, for whatever reason, may be recognized and rejected. A 10-bit FADC running at the local clock frequency samples the leading edge and the immediately preceding baseline, as well as subsequent features such as the zero-crossing point. About 16 samples of each feature is more than sufficient. Figure 69 shows the actual received waveform after ≤ 2 km of cable. The measure of timing may be chosen in various ways by focussing on various parts (or perhaps all) of the waveform, with differing systematic impact. The two most illustrative approaches focus on the leading edge or the zero-crossing point of the received bipolar time-mark waveform. Although dV/dt is higher for the zero-crossing point, this measure is prone to greater error since the extrapolation to baseline is less immediate, and the instant of zero-crossing depends on circuit component values and on the risetime of the pulse, which depends on cable length. The zero-crossing method, although offering somewhat better precision for tracking oscillator frequency, is not as suitable for establishing the local-master time transformation, which also includes phase. A leading-edge method is a simpler and more appropriate choice.

The presently used leading-edge algorithm simply extrapolates to baseline using a few samples above a threshold. Benefits from using more sophisticated fitting to the waveforms surely exist, but remain to be explored. The leading edge measurement algorithm, in whatever form it takes, must be sophisticated enough to avoid excursions into the very first part of the waveform during which the first and second derivatives have the same sign, or it must permit a sufficiently complex spline form that spans accurately the somewhat asymmetric point of inflection. In any case, the results show that the desired time resolution can be obtained, even with the most simple approach.

Figure 69: Time-mark waveforms measured at the digital optical module, after traveling through 2+km of twisted pair.

The extrapolation-to-baseline algorithm defines the instant that the waveform was received, in local DOM time units. At the surface, the moment of transmission is precisely defined by the master clock time distribution subsystem. Repeated measurements of this type serve to measure the frequency and phase of the DOM local clock very accurately. The stability of the Toyocom oscillators will permit, with beneficial effect, fits that include measurements made over several minutes.

Figure 70 presents the results obtained with DOMs deployed in the ice. This figure needs some explanation to extract the intrinsic resolutions of the DOM and Test-Board DAQ. Each point in the one-way histograms is based on two consecutive measurements; the rms of each measurement is 1/√2 smaller. The differences between up-going resolution (7.8 ns) and downgoing resolution (4.5 ns) is thought to be due to a noisier environment in the counting room (where the Test-Board DAQ is located) than in the DOM. The round trip resolution involves one up-going and one down-going measurement, and is observed to be approximately the sum of the individual resolutions taken in quadrature. In the future, resolutions should be better, since a GPS clock would not be used directly as a master clock, and steps would be taken to reduce or eliminate environmental noise in IceCube. The time resolution measured in the laboratory using a Toyocom oscillator as reference was indeed significantly better, ∼3 ns rms, supporting this hypothesis.

The measurement of the local clock phase, however, contains an Offset due to the differing physical lengths of the twisted quads throughout the detector. The differing physical lengths, twists, and stretches introduce differences in the propagation time for signals to pass from one end of a cable to the other. The signal propagation time, or cable electrical length, can vary by a large amount, up to ∼5000 ns from the top of the IceCube string to the bottom.

The traditional way to measure cable electrical lengths is known as Time Domain Reflectometry (TDR). Simply put, the TDR method sends a short pulse into a cable and measures the time between input and a return signal from the end of the cable; however, the TDR method requires either an unterminated or a shorted end point to reflect electrical energy. Neither condition is permitted in our case, since the cable must serve as a fairly high-bandwidth network and deliver power as well. In addition, the return pulse shape is very different due to dispersion and attenuation in the cable during the round trip, complicating the definition of time measurement at the level of accuracy that we require. More relevant than the electrical lengths themselves, however, are the differences in these electrical lengths. The cable electrical lengths must be measured in a way that provides accurate relative values of the cable lengths. There is an easy way to do this, known as the Reciprocal Active Pulsing (RAP) method (see next section) [158].

Since our approach is novel, we present in some detail how timing precision and accuracy of ∼3–5 ns is obtained throughout a simple copper-based km-scale network. We emphasize that every critical functionality regarding time-base integrity has already been demonstrated in the AMANDA string 18 DOM ensemble.

Time resolution results obtained with the digital optical modules currently deployed in the ice. The resolutions displayed here are ∼ √2 larger than the individual resolutions because two consecutive measurements are used to generate each point in the histograms. GPS clock drift, digitization noise, and environmental noise in the counting house dominate the resolutions.

Figure 71: General concept of the RAP method for measurement of the electrical length of the twisted quads.

7.2.7 Cable Electrical Length Measurement

The RAP method employs a timing strategy identical to the local/master clock time-base calibration, but adds symmetric round-trip measurement with identical bipolar time-mark pulses that are sent/received in both directions. The return pulse is separated by a known, programmable wait interval. The wait interval is chosen to allow the pulse energy in the cable to dissipate, so that the first pulse has a negligible impact on measurement of the second. By using circuitry, firmware, and software identical to that used for local time calibration in both DOM and Surface DAQ, the measurement of cable length provides high relative accuracy in cable length measurements, even though the rise-times vary somewhat due to differing cable lengths. The RAP method may be used with either the DOM or the DAQ as the initiator of the first pulse; consistent results are obtained in either case. The RAP method is, of course, invoked by software, with no need for manual intervention. The general idea is illustrated in fig. 71.

The definition of cable length is simply one half of the total delay between initiating pulse and received responding pulse, minus one half of the wait interval. Although a common offset exists in this method, taking the relative differences among the ensemble of cable lengths reduces this source of error to a negligible level.

Measurements of the deployed AMANDA string 18 cables were made from the northern hemisphere using the satellite link to the South Pole to download relevant code and commands. The results, shown in fig. 72, are completely consistent with expectations. The statistical error bars, ≥1 ns, are much smaller than the points. The RAP process may be invoked repeatedly, leading to a measurement value for each DOM with negligible statistical errors; it may also be invoked automatically at periodic intervals, to monitor system stability, without interfering with normal data acquisition.

7.2.8 Data Flow and Feature Extraction

The raw representation of a digitized waveform would include 128 10-bit samples and the 54-bit time stamp, requiring at least 170 bytes in an unreduced format. Waveforms exceeding the maximum digitization value of 1023 would initiate digitization of the next (lower) gain channel, resulting in additional 1280 bits of data, etc. Without filtering or compression, a DOM with 300 Hz noise would generate 50 kB/s, or more, most of which is uninteresting noise hits with long pedestals. Such a data rate would overtax the network capacity.

Feature extraction, local coincidence, and zero-suppression offer useful methods to reduce the bandwidth burden on the network. Feature extraction offers a powerful means to reduce transmission rate since most of the hits, perhaps more than 95%, are simple SPE pulses. The feature extraction process for SPE events reduces the waveform information to amplitude and timing information only. After extraction, an SPE hit should require only 8 bytes to characterize, including coarse and fine time-stamps. What is critical is that the algorithmic procedure that makes the determination whether a given waveform is an SPE or not be robust, since detailed information is permanently lost after extraction. It seems reasonable to set the cut such that typical SPE waveforms pass occasionally to the surface.

Figure 72: Measured electrical lengths of twisted quad cables in string 18 using the RAP method.

In any case, the bandwidth requirement for SPE waveforms will drop by a factor of 20 with feature extraction. This leads to a data rate/DOM for SPE pulses of ∼2.3 kB/s, assuming a 300 Hz noise rate. Feature extraction is included in the baseline DOM capabilities because it provides such a large data compression factor.

For multiple photoelectron events, the "non-SPE" waveforms, perhaps the full 170 bytes/event are needed. Assuming 5% of the 300 Hz hit rate fall into this category, another 2.5 kB/s are generated. While zero-suppression can reduce the unneeded pedestal data for this category, the factor of reduction here may be only a factor of ∼2. Zero-suppression may also complicate event formatting, and its role in DOM data transmission may not be prominent.

7.2.9 Local Coincidence

Conventional experimental design traditions bring all real-time signals to the counting-house, within which various trigger logic arrays impose filters to retain only events of interest. For IceCube, the architecture is quite different, and real-time functions play a limited role in the DAQ. Information arrives at the "counting house" in the form of time-stamped data packets. Since the vast majority of PMT pulses are noise hits, it is reasonable to ask what is the impact of noise on system design and cost, and on physics.

The noise rates of the PMTs are interesting for scientific purposes such as SN/GRB searches and for diagnostic purposes such as PMT gain stability. However, the DOM noise rate is approximately 100 times greater than the rate induced by high-energy particles. The question naturally arises whether it is necessary to transmit all the noise pulses individually to the surface DAQ. Unlike hits due to high energy particles, noise hits appear randomly, primarily as isolated hits, with no correlation to neighboring or nearby DOMs. Isolated hits are defined as those hits falling within a loose timing window, ∼1 μs, but separated by two or more unhit DOMs between. In AMANDA, such isolated hits have been relentlessly "cleaned out" during an early stage of analysis. From the standpoint of event fitting, an isolated hit is more than 100 times more likely to be noise than signal.

Given that ∼5% of the PMT hits are non-SPE events, and that feature extraction is in operation for the remainder, the data rate per DOM is ∼5 kB/s, without local coincidence. Evidence from string 18 suggests that ∼15% of the coincidence events are non-SPE in character. The expected coincidence rate is ∼5 Hz, including random coincidences. At this rate, each DOM would generate only ∼500 bytes/s, about one order of magnitude lower than without coincidence. With local coincidence, data processing resources needed at the surface become smaller by a comparable factor as well.

The question turns next to whether some simple local coincidence requirement, imposed in the ice, selects essentially all interesting hits for transmission to surface. Early work on this topic, including both analytical and Monte Carlo studies based on an IceCube-like geometry, but using an AMANDA-like trigger criterion, suggest that a local coincidence requirement does eliminate a small fraction, ∼1–3%, of real events [159, 160, 161], depending on the tightness of timing cuts. These isolated hits may add some useful information about energy, but likely add very little to tracking precision.

It should be noted that event filtering in IceCube is likely to be quite different than that of AMANDA. The digital system architecture permits much greater flexibility in the design of filters, due to software implementation rather than hardware. Studies have not yet been made to determine whether IceCube would or would not suffer a loss of efficiency, and hence whether the instrument would be operated with some form of local coincidence. If the conclusion is yes, local coincidence will be used, then isolated hits would not be present in the output data, except as they contribute to the noise rate histograms for SN/GRB searches. In either case, the DOM design will include the local coincidence capability. IceTop, discussed elsewhere, will employ local coincidence operation to mitigate the uninteresting but copious flux of low-energy muons at the surface. For IceCube, the use of local coincidence is an option that can be turned on or off.

Each DOM communicates with its nearest neighbors by means of a dedicated copper wire pair about 20 m in length. This link is thus capable of propagating short pulses, ∼50 ns, with good edge characteristics. The DOM is capable of sending and receiving these short signals in a full duplex mode. It is therefore straightforward to arrange logic within the FPGA such that digitization occurs only when some coincidence requirement has been met. Once a DOM has triggered an ATWD, it sends signals to each of its neighbors and also starts a count-down from a predetermined number of clock cycles. While the count-down is active, a DOM is receptive to pulses from either or both of its neighbors. Should this occur, the DOM will digitize, store, and subsequently transmit the time-stamped data to surface. Meanwhile, the neighboring DOM will have started a count-down period to initiate a period of receptivity, even if it has not yet been triggered. If the neighboring DOM had been triggered prior to receiving a pulse, it will initiate digitization as well. This logic ensures that the local coincidence will capture events independent of which DOM is hit first.

In this way, a local coincidence requirement of programmable width has been implemented in AMANDA string 18 and has worked flawlessly. For the future, it is intended to augment this capability such that a DOM which receives pulses from each of its neighbors during the programmed interval, which itself has not yet been triggered, will transmit pulses back to both so that they will initiate digitization. This will realize next-to-nearest neighbor coincidence. A somewhat more sophisticated approach is also possible, in which a triggered DOM transmits a pulse to each of its neighbors, as before, but which is now a pulse of predetermined length, say three clock cycles. A neighboring DOM that has not triggered will observe this pulse and transmit it onward but with only two cycles in length; each untriggered DOM in this chain will transmit the pulse onward, shortening it until it reaches zero length. In this manner, a correlation length for the local coincidence requirement can be introduced that extends beyond next-to-nearest neighbor. It remains to be determined whether such a scheme adds to physical content.

The cost for implementation is fairly modest, as a minimal amount of circuitry is needed to generate and receive these pulses, in any scenario. The penetrator must have six pins, rather than two. The cable connecting DOMs is more complicated since it must support six wires. However, by using a short cable to connect four DOMs to just one main cable breakout, a simplification has been achieved in the main cable construction; costs for the main cable are reduced significantly.

The PMT noise rate can be recorded properly by the DOM, and transmitted periodically to the surface DAQ as time-ordered histograms, so that the supernova/GRB search capability is retained. Programmable dead-time for the histogramming scalers can be included to alleviate statistical degradation due to after-pulsing; this requires only a small amount of FPGA logic to implement. Assuming that the histogramming intervals for detection are 10 ms, the average count is only 3. Allocating 8-bit (maximum count: 255) scalers for the GRB/SN search adds only 100 bytes/s, an insignificant burden. These 10 ms, 8-bit, scalers are unlikely to overflow, even for a very nearby SN.

In addition, however, the DOM can maintain a circular buffer of many seconds (even minutes, if needed) depth, filled with the stream of coarse time-stamps, rounded off to ∼0.1 ms. This provides a detailed record of any anomalous rate development, should a nearby SN occur. The interval of interest could be requested in the event that a global sum of the 10 ms data indicates the onset of a significant upward rate fluctuation.

7.2.10 System Design Aspects

Control Logic An Altera 10K50 FPGA provides the glue logic between circuit elements and handles all real-time activities, i.e., those operations for which nanosecond timing is required. The FPGA handles the synchronized launch of the ATWD, loads data into memory, receives messages from the surface and transmits the requested response message. The FPGA will be the site for the SPE feature extraction process. For an FPGA implementation, algorithms involving only integer arithmetic are desirable; this does not appear to impose a limitation for accuracy, robustness or execution speed.

ARM CPU The string 18 DOM state is controlled by a low-power Cirrus logic 32-bit CPU targeted toward the hand-held market. The choice of an ARM device was deliberate, in the reasonable hope that this general architecture would remain viable in the market-place at least for the duration of the IceCube production phase. The particular CPU employed in AMANDA string 18 is certain to be replaced by similar but more powerful devices on a yearly time-scale; nevertheless, software compatibility is expected to be high. A product from another vendor may be chosen for IceCube.

The CPU runs a Real-Time Executive, a thin version of a Real-Time Operating System (RTOS). This facilitates interrupt response and message handling. The CPU boots from flash memory upon power-up, and runs at 16.8 MHz in string 18. It offers the rough performance equivalent of a 31 MHz 486, with a power dissipation of only 50 mW. Higher level functions requiring code unsuitable for FPGA implementation will be realized in the ARM CPU. Since the string 18 DOMs were designed, new FPGA products that offer embedded ARM CPUs (hard or soft) have become available. Such integrated devices may provide an attractive simplification through the elimination of one complex component.

Slow Control, Calibration, and Self-Test Features Each DOM is responsible for controlling critical data taking parameters. These include setting local PMT high voltage levels, discriminator threshold levels, ATWD digitizer control voltages, and even selection and loading of appropriate FPGA programs. Since none of these operations is performed during actual data acquisition, they are deemed part of the slow control system. All adjustments capable of affecting the quality and quantity of data produced by the DOM are controlled from within the DOM itself. Therefore, all slow control requests, regardless of their origin (user interface, automated experiment control sequences, etc.) are finally serviced within the DOM and its resident software.

Each DOM is also capable of carrying out extensive self-test sequences. In addition to verifying correct operation of the digital platform and peripherals within the DOM, these programs can test analog electronics for the presence of increased noise, or spurious discriminator or ATWD behavior. Upon initiation, these tests are carried out under the control of the local DOM processor and results are reported back to surface electronics upon their completion.

One of the most important of these self-test features is uniqueness. Each DOM has its own serial number, most likely and elegantly implemented via a preprogrammed serial memory chip on the PC board. Upon booting up, this identification number is transmitted to surface DAQ, which can then verify that the channel number in the DAQ corresponds correctly to the deployed sequence along the string. Thus, cockpit errors that can easily lead to reversed connections or misplaced channels are instantly located.

There are many DOM parameters, measurable either with scheduled tasks in the DOM CPU, or via command messages from anywhere, that must be included in the calibration database. The primary scheduled task is the local-master clock transformation calibration process. Other tasks that should not require frequent activation are the ATWD capture speed calibration, the ATWD trigger discriminator threshold scan, the PMT SPE pulse-height distribution/average, and ATWD transfer characteristics such as sample pedestal values, gain and linearity. The transit time of photoelectrons through the PMT can be measured using a built-in triggerable weak light source (not the optical beacon board).

Table of Contents