2004 CDF E-Log -- Day shift. Thu Feb 26, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Beate Heinemann Tom Schwartz Natasha Miladinovic Silvia Amerio J. J. Schmidt


Start of Shift Notes:  

- store 3256 going at L21.8E30 
- COT in particularly unusual configuration: SL1 with 8% reduced HV 
plan: 
- Silicon is being cooled down: will take about 2 hours. Will then be turned on 
- COT will be brought back into the state it was in 1 week ago: 
   o no nitrogen flow (need HV off for 1 hour) 
   o SL 1,2 off and 3,4,5 with reduced gains 
- keep testing trigger table PHYSICS_TEST_2_02[10,429,434]. Will test this also when Silicon is back
on
Thu Feb 26 08:37:03 Run 179380 Terminated at 2004.02.26 08:35:58 - RunControl
Thu Feb 26 08:37:04 Run 179380 TERMINATE: Nitrogren Flow Stopped Into COT. Detector Config Changed. - Tom x2080
Thu Feb 26 08:58:10 Run 179381 Activated at 2004.02.26 08:57:55 - RunControl
Thu Feb 26 08:59:10 Run 179381 ACTIVATE: Taking some junk data for some folks at CO station while COT is down. ALL COT inhibits masked. - Tom x2080
Thu Feb 26 09:12:18 COT work done: Bob Wagner is turning it back on - Beate
Thu Feb 26 09:15:00 Run 179381 Terminated at 2004.02.26 09:14:02 - RunControl
Thu Feb 26 09:15:01 Run 179381 TERMINATE: COT Back up.  - Tom x2080
Thu Feb 26 09:17:10 Run 179382 Activated at 2004.02.26 09:16:58 - RunControl
Thu Feb 26 09:17:39 Run 179382 ACTIVATE: COT now up. Running with it included. - Tom x2080
Thu Feb 26 09:34:39 On Silicon Revival

Once Andy Hocker declares nominal temperature for silicon, please page SPL 722-7483 and David Clark 722-8740. - Rainer

Thu Feb 26 09:39:18 Run 179382 Terminated at 2004.02.26 09:38:53 - RunControl
Thu Feb 26 09:39:19 Run 179382 TERMINATE: Request End Run By Expert. - Tom x2080
Thu Feb 26 09:47:21 MCR called: they want to do collimator studies for half an hour. We turn the HV off/standby and leave CLC on. - Beate
Thu Feb 26 09:49:05 Run 179383 Activated at 2004.02.26 09:48:41 - RunControl
Thu Feb 26 09:50:55 Run 179383 ACTIVATE: xtrp firmware for track ORing. sim track marino. voltage down on detector. - Tom x2080
Thu Feb 26 09:51:14 While bringing HVs down: Had a PSM alarm (1RR18E), CMP trip (part of South Wall), CMU took a while to get to standby. Everything fine now. - natasha
Thu Feb 26 10:14:04 Run 179383 Terminated at 2004.02.26 10:13:44 - RunControl
Thu Feb 26 10:14:25 Run 179383 TERMINATE: Data with HV off for use by experts. - Tom x2080
Thu Feb 26 10:16:26 
- MCR called: they finished studies and rescraperd for us. So, we are turning HV back on. 
- Silicon is cool
 - Beate
Thu Feb 26 10:18:58 Run 179384 Activated at 2004.02.26 10:18:46 - RunControl
Thu Feb 26 10:25:49 Run 179384 ACTIVATE: COT HV Up. SILICON OFF. Running with PHYSICS_TEST_2_02[10,429,433] - Tom x2080
Thu Feb 26 10:40:16 Consumer (LumMon, BeamMon and YMon )are already running with the ONLINE DQM Expert System code implemented. For the moment they are runing without problems. - Olga :: (run 179384)
-- Thu Feb 26 11:31:02 comment by...kaori --  We made a new cdfsoftb0 release 5.3.1pre4c to be current default and all the monitor related processes are running under this release. Main difference from previous release is DQM implementation in YMon, LumMon and BeamMon.
Thu Feb 26 10:53:15 
Lester Miller is done with Silicon SRC tests so we call  
David Clark to power on the Silicon.
 - JJ
Thu Feb 26 11:04:39 
As requested by expert: 

L1 Accept Rate:  14.7 kHz 
L2 Accept Rate:  122  Hz 
L3 Accept Rate:  22 Hz 

Deadtime: 0.62 % 

All looks normal.
 - Tom
Thu Feb 26 11:07:28
 - natasha
Thu Feb 26 11:09:59
 - Silvia
-- Thu Feb 26 11:11:30 comment by...Silvia --  COT SL1 and SL2 are off
Thu Feb 26 11:11:30 Run 179384 Terminated at 2004.02.26 11:11:17 - RunControl
Thu Feb 26 11:12:28 Run 179384 TERMINATE: Integrating Silicon. - Tom x2080
Thu Feb 26 11:23:14 Run 179385 Activated at 2004.02.26 11:22:41 - RunControl
Thu Feb 26 11:23:29 Run 179385 ACTIVATE: Silicon Now Included. Running With PHYSICS_TEST_2_02[10,429,433] - Tom x2080
Thu Feb 26 11:47:29 SiliMon crashes (current version) at the first event with silicon in read-out. Fix should be in soon. We put back old old SiliMon (4.11.2j). - kaori :: (run 179385)
Thu Feb 26 12:00:13 Rainer called: wants us to keep running with silicon. No DMOD calibrations until further notice from experts. - Beate
Thu Feb 26 12:07:21 Run 179385 Terminated at 2004.02.26 12:06:42 - RunControl
Thu Feb 26 12:07:47 Run 179385 TERMINATE: Quench. Ending Run. - Tom x2080
Thu Feb 26 12:09:24 Tevatron quenched - Beate
Thu Feb 26 12:15:43 Performed Post-Quench Checkout. Run Config in DaqTest/SVX/ColdStart (kept Proxy the same). Got through coldstart fine. Unmarked Silicon Cells. No new pinkies. - Tom
Thu Feb 26 12:17:44 Since yesterday, I have experienced 3 times, netscape crashed after making entries in e-log, hitting preview, and instant after hitting 'close' in the preview window. Netscape I was using is the very old version running in the control room pc's. - kaori
Thu Feb 26 12:18:17 Run 179381 RUNSTATUS: 
Marked Bad, explanation:
COT 
SVX 
ISL 
L00 
SVT
 - cdfscico
Thu Feb 26 12:23:15 Run 179382 RUNSTATUS: 
Marked Bad, explanation:
SVX 
ISL 
L00 
SVT
 - cdfscico
Thu Feb 26 12:25:05 Run 179384 RUNSTATUS: 
Marked Bad, explanation:
SVX 
ISL 
L00 
SVT
 - cdfscico
Thu Feb 26 12:25:22
DateTimeBLMDose
2004.02.2612:24:40W Inner BLM776.37RADS
2004.02.2612:24:40W Outer BLM6.49RADS
2004.02.2612:24:40E Inner BLM81.48RADS
2004.02.2612:24:40E Outer BLM739.91RADS
Integrated dosage - natasha
Thu Feb 26 12:30:37
blm summary
 - natasha
-- Thu Feb 26 13:28:22 comment by...Rainer --  The correct plot is here
Thu Feb 26 12:34:27 Run 179386 ACTIVATE: Cal QIE. ALL HV ON. - Tom x2080
Thu Feb 26 12:34:48 Run 179386 TERMINATE: Cal QIE. ALL HV On. - Tom x2080
Thu Feb 26 12:41:52 post quench checkout procedure completed. Silicon went through config without problems. I will not unmark the pink cells (there is a number of them) since David told me not to worry about them. - natasha
Thu Feb 26 12:45:14 Just after quench it took CMP 3 tries to get back up. - natasha
Thu Feb 26 12:50:27 Run 179387 ACTIVATE: TOF QIE Calib. - Tom x2080
Thu Feb 26 12:51:11 Run 179387 TERMINATE: TOF QIE Calib. - Tom x2080
Thu Feb 26 12:54:12 Run 179388 ACTIVATE: SMX QIE Calib. - Tom x2080
Thu Feb 26 12:54:19
 - natasha
-- Thu Feb 26 12:55:22 comment by...natasha --  Plots showing quench-time losses. Proton abort gap losses spiked quite a bit.
Thu Feb 26 12:55:29 plug laser on anticipating calib. - Tom
Thu Feb 26 12:55:56 Run 179388 TERMINATE: SMX QIE Calib. - Tom x2080
Thu Feb 26 12:57:13 COT, muon detectors and Plug all on standby. TOF off. Silicon still on (we did not get instructions from experts about this). - natasha
-- Thu Feb 26 13:01:03 comment by...rainer --  on ? I hope you mean standby.
Thu Feb 26 12:58:06 Run 179389 ACTIVATE: COT Calib. - Tom x2080
Thu Feb 26 12:58:07 Run 179389 TERMINATE: COT Calib. - Tom x2080
Thu Feb 26 13:00:13 Run 179390 ACTIVATE: Muon Calib. - Tom x2080
Thu Feb 26 13:00:59
BLM summary...
 - natasha
-- Thu Feb 26 13:27:31 comment by...rainer --  so there is significant radiation dose in the last bin of the BLM plot. questions:
  • something happened on the 200 mu s scale. this something usually indicates trouble with the abort kicker system.
  • abort gap loss rates were below 10 kHz at the time of the abort. can we trust the rates for clean aborts ?

    the circumstances of this quench should be thoroughly explained.
    -- Thu Feb 26 15:30:27 comment by...rainer --  A0 has been very hot in this quench - see MCR elog

    Thu Feb 26 13:01:09 Run 179390 TERMINATE: Muon Calib. - Tom x2080
    Thu Feb 26 13:03:48 Run 179391 ACTIVATE: TOF TAC Calib. - Tom x2080
    Thu Feb 26 13:05:41 Run 179391 TERMINATE: TOF TAC Calib. - Tom x2080
    Thu Feb 26 13:12:23 Run 179392 Activated at 2004.02.26 13:11:49 - RunControl
    Thu Feb 26 13:12:31 Run 179392 ACTIVATE: PMT Spike run Calib. - Tom x2080
    Thu Feb 26 13:13:58 MCR called: they will be down for at least 1-1.5 hours and are going into access. They still don't know the reason for the quench. We are investigating whether we want an access too. - Beate
    Thu Feb 26 13:21:35 Run 179392 Terminated at 2004.02.26 13:21:05 - RunControl
    Thu Feb 26 13:21:36 Run 179392 TERMINATE: PMT Spike Calib. - Tom x2080
    Thu Feb 26 13:23:39 PMT Spike Calib: L1 Rate 52.6 Hz. - Tom
    Thu Feb 26 13:25:48 Run 179393 ACTIVATE: CLC QIE Calib. - Tom x2080
    Thu Feb 26 13:27:31 Run 179393 TERMINATE: CLC QIE Calib. Note: Expert just worked on CLC crate. - Tom x2080
    Thu Feb 26 13:29:01 Run 179394 ACTIVATE: CLC QIE Calib. NOTE: Expert just finished working on clc crate. - Tom x2080
    Thu Feb 26 13:31:10 Run 179394 TERMINATE: CLC QIE Calib. - Tom x2080
    Thu Feb 26 13:31:15 PTM alarm -- sensor failure. I went upstairs and checked the temperatures themselves -- both central and plug cal look good so far. Tried calling the expert, will try later. - natasha
    Thu Feb 26 13:32:36 Reset CLC Admems. - Tom
    -- Thu Feb 26 13:54:44 comment by...Tom --  Hung on Reset using VisionDemo.
    -- Thu Feb 26 13:55:30 comment by...Tom --  TEVMON warning came on during calib. Proton Losses too low...possible lost of inputs.
    -- Thu Feb 26 13:55:44 comment by...Tom --  experts working on it.
    Thu Feb 26 13:36:20 Run 179395 ACTIVATE: CEM LED Calib. - Tom x2080
    Thu Feb 26 13:37:31 Run 179395 TERMINATE: CEM LED Calib. - Tom x2080
    Thu Feb 26 13:54:42 Run 179396 ACTIVATE: CLC QIE Calib. - Tom x2080
    Thu Feb 26 13:55:40 Made sure that MCR will definitely call us before beam is put into the machine! - Beate
    Thu Feb 26 14:19:51 Run 179397 ACTIVATE: BSC QIE Calib. - Tom x2080
    Thu Feb 26 14:20:46 Run 179397 TERMINATE: BSC QIE Calib. - Tom x2080
    Thu Feb 26 14:32:52 Run 179398 TERMINATE: changing run config. - Tom x2080
    Thu Feb 26 14:32:52 I fixed the PTM sensor failure by taking the problem sensor [PE10, Dallas sensor 1] out of the configuration, and the alarm is gone. The sensor was reading too high a value and as a results the RMS of the 9 Dallas sensors was out of range. - natasha
    Thu Feb 26 14:33:41 All HVs up in preparation for cosmics run. - natasha
    Thu Feb 26 14:36:42
    Plug Laser Calib had to be scrapped. Laser off.
     - Tom
    Thu Feb 26 14:39:42 Run 179399 Activated at 2004.02.26 14:39:30 - RunControl
    Thu Feb 26 14:40:37 Run 179399 ACTIVATE: COSMICS_SILICON_TAPE COSMICS[12,391,403] - Tom x2080
    Thu Feb 26 14:45:25 b0svt06 crate hung (H and U went red in Vxworks system monitor). Shepherded twice and went back to normal. - Tom
    Thu Feb 26 14:56:58 
    COT calibration results: 
NumBad cells = 3 (>1) 
NumBad cables = 2 (>1)
     - Silvia
    Thu Feb 26 15:35:18 We have swapped the 2300 CPU in crate b0clc00 for a 2400. This crate has one of the longer readout times and this should help speed it up. We found that the new CPU hangs VISIONdemo (if it is run on a PC as opposed to the crate itself) when doing the crate reset as the last step in the CLC QIE calibration. In this case the reset is actually done correctly but you have to control-c out of VISIONdemo. We have switched the instructions on the web page to include a vxlogin to the crate, in which case VISIONdemo will work the same for all CPU's. Problems to DAQ pager.  - Ray Culbertson and Ming Wang :: (run 179399)
    Thu Feb 26 15:35:48
     - Silvia
    -- Thu Feb 26 15:48:08 comment by...Silvia --  
    SMX calibration results (run 179388)
    Thu Feb 26 15:47:45 Run 179399 Terminated at 2004.02.26 15:47:22 - RunControl
    Thu Feb 26 15:47:46 Run 179399 TERMINATE: Expert wants to remove some crates. - Tom x2080
    Thu Feb 26 15:54:26 Run 179400 Activated at 2004.02.26 15:53:27 - RunControl
    Thu Feb 26 15:54:27 Run 179400 ACTIVATE: Running COSMICS_SILICON_TAPE COSMICS[12,391,403]. No VRB or FIB crates are being used as requested by expert. - Tom x2080
    Thu Feb 26 15:55:30
    Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
    179379 x2BCB3 BEAM PHYSICS_TEST_2_02 [10,429,433] 05:06:24 08:06:27 02:54:41 141,264,440 1,454,423 266,250 244.877 0 1 1
    179380 x2BCB4 BEAM PHYSICS_TEST_2_02 [10,429,433] 08:23:51 08:35:58 00:00:00 0 0 0.000 1 0
    179381 x2BCB5 BEAM PHYSICS_TEST_2_02 [10,429,433] 08:57:55 09:14:02 00:16:03 1,612,204 27,566 13,192 20.255 0 0 1
    179382 x2BCB6 BEAM PHYSICS_TEST_2_02 [10,429,433] 09:16:58 09:38:53 00:21:43 14,020,377 132,042 28,734 27.222 1 1 1
    179383 x2BCB7 BEAM PHYSICS_TEST_2_02 [10,429,433] 09:48:41 10:13:44 00:01:44 119,749,071 19,523 13,266 0.000 1 0
    179384 x2BCB8 BEAM PHYSICS_TEST_2_02 [10,429,433] 10:18:46 11:11:17 00:52:09 40,535,609 351,436 70,111 62.010 1 1 1
    179385 x2BCB9 BEAM PHYSICS_TEST_2_02 [10,429,433] 11:22:41 12:06:42 00:42:34 31,103,353 412,640 91,172 48.938 1 1 1
    Totals 15:55:02 05:08:57 348,285,054 2,397,630 482,725 403.302
     - End of Shift Report
    Thu Feb 26 15:58:39 Shift Summary: 
    history: 
======= 
- took data in store 3256 
- COT put back into "normal" configuration: SL1,2 off, 345 reduced gains 
- Silicon cooled down and was turned on 
- just after Silicon was back in Tevatron quenched!   
- took all calibrations except Plug Laser 
- PTM sensor failure fixed by our ACE Natasha 

ongoing work: 
============ 
- are in quiet time: Tevatron will call us before they put beam in machine 
- PCAL11 looks bad in calibrations: Mike L. will powercycle East Plug 
- Lester working on Silicon (SRC) 
- Rick Tesarek spike run after power cycling

    End of Shift Numbers
    CDF Run II 

    Runs                   179379, 179381, 179382, 179384, 179385
Delivered Luminosity   131.7  
Acquired Luminosity    109.8  
Efficiency             83.3
     - Beate