2005 Bad OM/File Cleaning

2005 Bad OM List (1/18/05 updated with adcpeak analysis by Michael Walter)
2005 GOOD File List (1534 bad files, 3.6%) DOES NOT ACCOUNT FOR ICECUBE FLASHER RUNS!
2005 BAD File List (1534 bad files, 3.6%; 2675 files including those for which there is no online filter output) DOES NOT ACCOUNT FOR ICECUBE FLASHER RUNS!
2005 short run list (Feb 16 - Oct. 31) (n <= 6)
NEW! 2005 String-21 flasher coincidence files (add to bad file list) and dates
NEW! Combined list of all bad files not included in the final L3 sample

Methodology

OM noise rates are calculated file-by-file using the number of events with TDC < 7us. The number of counts in this TDC time window should follow a poisson distribution throughout the year. Poisson parameters are calculated for each OM using all files, and then for each file and each OM there is a poissonian likelihood. Each OM is assigned a 'quality factor' Q, given by the sum of log likelihood for all files for that OM divided by the number of files. Similarly, each file is assigned a quality Q given by the sum of log likelihood for all OMs for that file divided by the number of OMs. Files and OMs are selected simultaneously by eliminating the 'Q object' (OM or file) with the highest value and recomputing Q for the remaining objects. The procedure should result in a more efficient cleaning since the interdependence of problematic OMs and problematic files is virtually eliminated. It is also a 'cleaner' procedure, as it doesn't require arbitrary parameters (e.g. dead and noisy thresholds), except a requirement of <10% files where the OM produces no noise hits and a requirement that the mean noise rate is less than some value (~8kHz). The latter was never applied in this analysis, as noisy OMs typically do not behave in a poisson fashion.

NOTE that in addition to this list based on the rates in the PMTs, another bunch of OMs are excluded due to geometrical reasons (OMs 81-86, upper and lower parts of strings 11-13 and string 17).

Files outside the date window Feb 16 - Oct. 31 are marked as bad and removed.

Feb. 16 - Oct. 31 Analysis

	Each matrix element (file, OM) is filled if the noise rate for that element is out of range (83Hz < NR < 8.3kHz). If the OM has been rejected, no elements for that OM are filled. Rejected files are excluded.	Mean detector noise rate, by file At any given quality level, rejected OMs and files are excluded.	Count of OMs outside noise rate range, by file At any given quality level, rejected OMs and files are excluded.	Distribution of files, by number of OMs in file outside noise rate range, a rough measure of file quality. The position of the large peak yields an estimate of the typical number of bad OMs in any given file. At any given quality level, rejected OMs and files are excluded.	Distribution of OMs by number of files outside noise rate range, a rough measure of OM quality At any given quality level, rejected OMs and files are excluded.
Q < infinity
Q < 6
Q < 5.5
Q < 5
Q < 4.75

At Q < 6, the majority of bad OMs and files are eliminated. A few unstable OMs (e.g OM 41, see below) remain, as well as a set of files at the beginning of the season missing ~30 otherwise healthy OMs. At Q < 5, the unstable OMs are eliminated along with the problematic set of files. The Q < 5 quality level also produces similar numbers of bad OMs and bad files as a more standard approach (described below), leading us to select Q < 5 as the final quality level.

Final Q value distribution of OMs (Q=5)

The most unstable OMs (mostly dead and very noisy modules) have very large Q values and are not shown.

Final Q value distribution of Files (Q=5)

Very bad files (HV failures, etc.) have large Q values and are not shown

Comparison to a more standard approach, similar to 2000 OM/file selection

1. Cut dead (noise rate < 83Hz for >90% of files) (58 OMs) & noisy OMs (noise rate > 8.3kHz for >90% of files) (2 OMs)
2. Cut files with >=50 remaining OMs with noise rate < 83Hz or noise rate > 8.3kHz and runs before 9153 (1279 files, 3.2%)
3. Cut OMs according to stability criteria
- >3% of files at >8300Hz noise rate (4 OMs)
- >10% of files at 0Hz (5 OMs)
- >850Hz noise rate RMS (5 OMs)
4. Cut files with >=15 remaining OMs with noise rate < 83Hz or noise rate > 8.3kHz (50 files, 0.1%)

Standard bad OM list and bad file list.

Bad Files

The Q < 5 bad file list (1537 files) contains all files rejected by the standard approach (1329 files). Of the 208 files not eliminated by the standard approach but contained in the Q < 5 bad file list, 144 are 3 sigma or greater outliers in mean noise rate.

Bad OMs

Modules cut Q < 5, and not cut in the standard approach:
41, 376

Modules cut in the standard approach, and not cut in Q < 5:
288

OMs 41 and 376 appear unstable and should be removed; however, the standard approach is not sensitive to this behavior since noise rates stay within the 83Hz < NR < 8.3kHz range.

288 appears unstable with large noise spikes. However, a small sample (N ~10) of OMs show similar behavior in the same subset of files; thus, the algorithm correctly eliminates the offending files rather than OM 288.

Lists at other quality levels

Q < 6	Bad OM List Bad File List
Q < 5.5	Bad OM List Bad File List
Q < 5	Bad OM List Bad File List
Q < 4.75	Bad OM List Bad File List

Typical noise rates

Notable bad OMs

TDC LE Distribution, Run 9363 (Use 0-7us window for noise rate measurement)