2004 CDF E-Log -- Owl shift. Tue Mar 9, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Rick Field Tom Schwarz Susana Cabrera Andrei Loginov Rob Harr


Start of Shift Notes:  

Doing quiet time calabrations and testing 3 trigger  
tables.  Expect shot set-up in about an hour.

Tue Mar 9 01:10:30 Run 179743 Terminated at 2004.03.09 01:09:51 - RunControl
Tue Mar 9 01:10:31 Run 179743 TERMINATE: PHYSICS_2_02[2,424,437] - Tom x2080
Tue Mar 9 01:11:55
After www server debacle most of evening shift yesterday and early owl shift are in day shift
feb 12 2004.  no idea why.   


of note.  all calibs done except plug so far.  still in quiet time
 - Tom
-- Tue Mar 9 06:06:59 comment by...Rob Harr (operations manager) --  

The shift e-logs and silicon e-logs (and therefore possibly all other e-logs) are missing between Feb. 12 and about 23:00 on Mar. 8. Is it possible to restore the missing parts??????


Tue Mar 9 01:13:08 We have been "flying blind". The WEBserver has been down. It just cam back up at 1:05am.  - Rick Field
Tue Mar 9 01:14:46
Calorim QIE: run 179736 
CLC QIE: run 179730 
TOF QIE: run 179731 
COT: run 179733 
MUON: 179734 
BSC QIE: 179737 
TOF TAC: 179739 
CEM LED: 179738 
SMX QIE: 179741 
SPIKE: 179742 L1 Rate 57.6 Hz
 - Tom
Tue Mar 9 01:16:51 Run 179744 Activated at 2004.03.09 01:15:57 - RunControl
Tue Mar 9 01:16:52 Run 179744 ACTIVATE: Another Test of PHYSICS_2_02[2,424,437] - Tom x2080
Tue Mar 9 01:16:53 Run 179744 Terminated at 2004.03.09 01:16:38 - RunControl
Tue Mar 9 01:17:54 Run 179744 TERMINATE: PHYSICS_2_02[2,424,437] - Tom x2080
Tue Mar 9 01:20:18 We have made a sucessfully run using the trigger table PHYSICS_2_02[2,424,437].
It was just bad luck last Friday, because there was an additional problem at the same time. - Guillelmo/Guilles :: (run 179744)
Tue Mar 9 01:25:24 Run 179745 Activated at 2004.03.09 01:24:53 - RunControl
Tue Mar 9 01:25:25 Run 179745 ACTIVATE: TEST_ALPHA_CLUSTERING[5,438,403] - Tom x2080
Tue Mar 9 01:29:57 Run 179745 Terminated at 2004.03.09 01:29:34 - RunControl
Tue Mar 9 01:31:06 Run 179745 TERMINATE: TEST_ALPHA_CLUSTERING_NOSPIKES[5,438,403] Done. All seemed fine. - Tom x2080
Tue Mar 9 01:36:22
New PINK cell in ISL , as predicted by SI experts before leaving: I_AVDD_B4W1L3 Current: 1085 Good: 1100-1125
 - Susana.
Tue Mar 9 01:38:29 Run 179746 ACTIVATE: Plug Laser Calib. BOTHPLUGS_NORMAL - Tom x2080
Tue Mar 9 01:45:46 Run 179746 TERMINATE: Plug Calib BOTHPLUGS_NORMAL - Tom x2080
Tue Mar 9 01:48:26 Run 179747 ACTIVATE: Plug laser callib. BOTHPLUGS_MM. bad order. ending this. - Tom x2080
Tue Mar 9 01:48:27 Run 179747 TERMINATE: ending due to incorrect order. - Tom x2080
Tue Mar 9 01:50:50 Run 179748 ACTIVATE: Plug Laser Calib. BOTHPLUGS_MM_C  - Tom x2080
Tue Mar 9 01:55:01 Run 179748 TERMINATE: Plug Laser Calib. BOTHPLUGS_MM_C - Tom x2080
Tue Mar 9 01:55:02 Run 179749 ACTIVATE: Plug Laser Calib. BOTHPLUGS_MM - Tom x2080
Tue Mar 9 01:56:33 Run 179749 TERMINATE: Plug Laser Calib. BOTHPLUGS_MM - Tom x2080
Tue Mar 9 02:04:51 Run 179750 Activated at 2004.03.09 02:03:52 - RunControl
Tue Mar 9 02:05:12 Run 179750 ACTIVATE: Taking Cosmic Data. COSMICS_ALL with COSMICS[12,391,403] - Tom x2080
Tue Mar 9 02:06:16
Table PHYSICS_TEST_2_02[11,437,438] could not be tested because it was not available to run
control.
 - Tom
Tue Mar 9 02:07:47 All Calibrations performed between late evening shift 3/8/4 and owl shift 3/9/4. thanks to Alison and her amazing memory of calibrations w/out the www server. - Tom
Tue Mar 9 02:09:54 Run 179750 Terminated at 2004.03.09 02:09:39 - RunControl
Tue Mar 9 02:10:03 Run 179750 TERMINATE: ending run. - Tom x2080
Tue Mar 9 02:14:09 Run 179751 ACTIVATE: Plug Laser Calib. BOTHPLUGS_NORMAL. Note to myself: Plug Laser Calibration much more useful when laser actually activated. - Tom x2080
Tue Mar 9 02:17:34
I checked "CDF II Consumer Summary" page for two runs 179728, 179727: 
179728 SiliMon 10136  PROBLEM (Errors)
No_clusters_detected_in_layer_No_ADC_counts_detected_in_layer_ 

179727 SiliMon 7482   PROBLEM (Errors)
No_clusters_detected_in_layer_No_ADC_counts_detected_in_layer_ 

179728 ObjectMon 5645 PROBLEM (Errors)
EM:OK;JET:OK;MET:OK;MUON:PROBLEM;SVT:UNDEFINEDSILICON:OK;TAU:OK;NoneTRACK:OK; 

 - Andrei
-- Tue Mar 9 02:29:01 comment by...Andrei --  
for run 179726:

179726 ObjectMon 159671 BAD (Errors) EM:OK;JET:OK;MET:BAD;MUON PROBLEM;SVT:UNDEFINEDSILICON:BAD;TAU
OK;NoneTRACK:OK;



179726 SiliMon 148193 PROBLEM
(Errors)No_clusters_detected_in_layer_No_ADC_counts_detected_in_layer_

-- Tue Mar 9 02:29:31 comment by...Andrei --  
stage0 is ok for all runs above.

Tue Mar 9 02:20:36 Run 179751 TERMINATE: Plug Laser Calib. BOTHPLUGS_NORMAL - Tom x2080
Tue Mar 9 02:22:56 Run 179752 ACTIVATE: Plug Laser Calib. BOTHPLUGS_MM_C - Tom x2080
Tue Mar 9 02:25:08 Run 179752 TERMINATE: Plug Laser Calib. BOTHPLUGS_MM_C - Tom x2080
Tue Mar 9 02:26:47 Run 179753 ACTIVATE: Plug Laser Calib. BOTHPLUGS_MM - Tom x2080
Tue Mar 9 02:28:59 Run 179753 TERMINATE: Plug Laser Calib. BOTHPLUGS_MM - Tom x2080
Tue Mar 9 02:35:12 Run 179754 Activated at 2004.03.09 02:33:00 - RunControl
Tue Mar 9 02:35:13 Run 179754 ACTIVATE: Taking Cosmics. COSMIC_ALL COSMICS[12,391,403] - Tom x2080
Tue Mar 9 02:39:04 Run 179754 Terminated at 2004.03.09 02:37:08 - RunControl
Tue Mar 9 02:39:05 Run 179754 TERMINATE: ending run. - Tom x2080
Tue Mar 9 02:42:55 Run 179755 Activated at 2004.03.09 02:42:11 - RunControl
Tue Mar 9 02:42:56 Run 179755 ACTIVATE: COSMICS-ALL COSMICS[12,391,403] - Tom x2080
Tue Mar 9 03:07:36
MCR is having a problem with the driver bias 
supply for LRF2.  Expert is working on it.
 - Rick Field
Tue Mar 9 04:30:50 Tom cannot find trigger table PHYSICS_TEST_2_02 to run. Rob Harr instructed us to page Jonathan Lewis which I did. - Rick Field
-- Tue Mar 9 05:03:02 comment by...jdl --  As usual, we forgot to set DAQVALID when fixing the Level 2 build after an initial failure. Done now.
Tue Mar 9 05:01:31 Run 179755 Terminated at 2004.03.09 05:01:00 - RunControl
Tue Mar 9 05:14:16 Run 179755 TERMINATE: all hell broke loose. power fermilab wide went out. - Tom x2080
Tue Mar 9 05:18:15 We had a power failure through all of Fermilab. Except for the antiprotons... the store is still there. The beam was off at the time of the outage. When the power returned the Silicon cooling alarm was going (along with many other alarms). The ACE hit the Silicon Rack Crash Buttton. I called Silicon and we talked with Rainer and we paged Andy (cooling) and he is on his way down. I called Rob Harr and he just arrived. - Rick Field
-- Tue Mar 9 05:40:23 comment by...Rick Field --  Feeder 35 fault/site-wide glitch!
Tue Mar 9 05:25:49 Cleared Clock Errors. All 3 lights were off. - Tom
Tue Mar 9 06:08:17
The PE Pickup bipolar signal is gone on the oscilloscope in the clock crate.  went through
procedure to start clock after power outage but didn't solve the problem.  expert paged.
 - Tom
Tue Mar 9 06:09:34 We have a clock problem. Problem with the PE pickup bipolar signal (no green line sync pulse). I paged the clock man. - Rick Field
-- Tue Mar 9 06:25:07 comment by...Rick Field --  I called Steve Hahn at home and he is working on the problem.
-- Tue Mar 9 07:24:52 comment by...Tom --  
Both clock errors and sequencer errors had to be reset.  all seems to work now. 

Tue Mar 9 06:25:02 Silicon cooling and rack powere recovered. - Andy H.
Tue Mar 9 06:50:18
We had two hear-beat alarms with the TOF, and re-started the TOF 
PC twice: at 00:15 and at ~6:30.  
 
 - Susana.
Tue Mar 9 06:53:27
CEM-CHA-WHA hear beat alarms: the hot towers are: 
           Phi-Wedge    Eta-Towers   
CEM east    5RL          8 & 9 
CHA east    5R           4,5,6,7 
WHA west     3R          -8 

Expert working on it.
 - Susana.
Tue Mar 9 07:07:19
BMU in yellow (WARNING) in the CDF GLOBAL ALARM. 
All the detector is in software Over/Under Voltage.
 - Susana.
-- Tue Mar 9 07:14:38 comment by...Susana. --  
Following expert instruccions, I press again STANDBY 
and the BMU comes back to the normal "standby" status
in terms of voltages and currents and green in the CDF
global alarm pannel.


Tue Mar 9 07:09:07
L3 does not make it past partitioning.  goes red.  cleaned up EVB and tried to restart proxy. 
still doesn't work.  trying one more time.
 - Tom
-- Tue Mar 9 07:12:42 comment by...Tom --  
L3 Paged.

Tue Mar 9 07:44:04 Run 179756 ACTIVATE: COT Calib.  - Tom x2080
Tue Mar 9 07:44:05 Run 179756 TERMINATE: COT Calib. - Tom x2080
Tue Mar 9 07:46:07 Run 179757 ACTIVATE: SMX QIE Calib. - Tom x2080
Tue Mar 9 07:46:57
Apparently after power outage its' a good idea to take calibrations (COT and Shower first).  so
on to that.
 - Tom
Tue Mar 9 07:48:08 Run 179757 TERMINATE: SMX QIE Calib. - Tom x2080
Tue Mar 9 07:49:31 Run 179758 ACTIVATE: Cal QIE Calib. - Tom x2080
Tue Mar 9 07:49:43 Shift Summary:
** WEBserver Back Up ** 
Flying blind until around 1am.  WEBserver is 
now back but is is missing many days.  I 
had to "end shift" for Feb 12 and then start 
the shift for today.  The days in between are 
missing?? 

** Quiet Time Calibrations ** 
Did quiet time calibrations. 

** Three Trigger Table Tests ** 
We did trigger (1) and (3) but Tom 
could not find trigger (2). 

** Were Waiting for Shot Set-Up ** 

** Yellow ISL-SVX Chips on SvxMon ** 
As soon as the database came up the 
silicon experts said that there really 
was no proplem with the yellow chips 
seen on SvxMon during the previous  
shift.  Apparently when the server went  
down SvxMon did not work properly and 
was accessing the wrong information. 
The "yellow" chips should have been 
"blue"?? 

** Runs 179726-179727-179728 ** 
Marked these runs as good. 

** LRF2 Problem ** 
MCR is having a problem with the driver bias  
supply for LRF2.  Expert is working on it. 
We are taking cosmics.  

** Feeder 35 Fault/Site-Wide Glitch ** 
We had a power failure through all of Fermilab.  
Except for the antiprotons... the store is still  
there. The beam was off at the time of the outage.  
When the power returned the Silicon cooling alarm was  
going (along with many other alarms). The ACE hit the 
Silicon Rack Crash Buttton. I called Silicon and 
OPS Manager.  Rainer, Andy, and Rob Harr have 
arrived and we are working on recovering. 
The solonoid went into a slow dump down to 
zero and then was restarted by Cryo.  The 
solonoid is now back up and looks stable. 

** Calabrations ** 
Need to do calabrations when able.

End of Shift Numbers
CDF Run II

Runs                   None
Delivered Luminosity   zero  
Acquired Luminosity    zero  
Efficiency             100%

 - Rick Field
Tue Mar 9 07:50:37 Run 179758 TERMINATE: Calorim QIE Calib. - Tom x2080
-- Tue Mar 9 07:56:28 comment by...Andrei --  
Cot calibration:

COT Calibration 07:42:13 on 2004-03-09
aveStart:                        304.60
total numBad:                   2113     - Too Many Bad channels
   numBad  cells:                  2
   numBad cables:                  2
   numBad slots:                  17
aveDrift:                       0.068
max number drifted in crate:    91
total numDrifted:              657
   number of drifted  cells       12
   number of drifted cables        7
   number of drifted  slots        0
Too many bad channels in calibration.

experts have been notified.


Tue Mar 9 07:55:23
Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
179743 x2BE1F BEAM PHYSICS_2_02 [2,424,437] 01:00:26 01:09:51 00:09:18 4,879 1,674 1,464 0.000 1 0
179744 x2BE20 BEAM PHYSICS_2_02 [2,424,437] 01:15:57 01:16:38 00:00:35 314 106 96 0.000 1 0
179745 x2BE21 BEAM TEST_ALPHA_CLUSTERING_NOSPIKES [5,438,403] 01:24:53 01:29:34 00:04:36 605 604 604 0.000 1 0
Totals 07:55:02 00:14:30 5,798 2,384 2,164 0.000
-End of Shift Report