2004 CDF E-Log -- Day shift. Fri Mar 5, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Rainer W. Jan E. Anadi C. Franco S./Sergei B. Mary C.


Start of Shift Notes:  

store #3273 in progress. inst lumi 12.31E30. meeting about one shift access request by D0 in
progress - therefore no plan.
 Fri Mar 5 08:29:18 
spurious alarm from COT, HV bar went briefly red but no trigger inhibit. run continuous
smoothly.
 - rainer
Fri Mar 5 08:29:58 COT software trip. It recovered in sec without any action.  - Anadi :: (run 179644)
Fri Mar 5 08:34:54 leak in the roof. can't fix it today so will have to wait until monday. cryo shifter will regulary check (info by craig olson). - rainer
Fri Mar 5 08:55:02 
I unmark two cells: B4W3L2 and B5W1L1. 
They were pink since AVDD current was out of range in both cases: 
- B4W3L2 I=1085, caution [1100,1190], good [1105,1185] 
- B5W1L1 I=1558, caution [1120,1180], good [1125,1175]
 - Anadi
-- Fri Mar 5 08:56:09 comment by...Anadi --  Sorry, both ladders ar ISL.
Fri Mar 5 08:57:34
Proton losses and the position of the E0 collimator
 - R.J. Tesarek
-- Fri Mar 5 09:03:10 comment by...R.J. Tesarek --  
Proton Losses and E0 Collimator:
Proton losses(cyan) and position of the E0 collimator(green) for the last four stores. Over the past two stores, the proton losses were significantly larger than usual as was their fractional width (RMS of the cyan line divided by the mean value) of the proton losses is considerably larger.
Fri Mar 5 08:57:40
CO team spots yellow in SVXMon.
 - rainer
-- Fri Mar 5 09:09:59 comment by...rainer --  silicon is paged (well, sort of). IB5W1L1 not giving any data, consistent with Anadi's and owl shift's observation. expert investigating.
Fri Mar 5 09:00:02 
Done Timeout detected in crate: 
PCAL_05 

had to be shepherded! 

MLE) b0pcal05:Messenger:8:53:54 AM->VISIONread/write failed in slot 15 
(MLE) b0pcal05:Messenger:8:53:55 AM->Event 5372753: Bunchcounters in slot 15 (BC=0) and slot 17
(BC=18) disagree (L2B=0)  

L2B 15:   0   9  15  77 
L2B 17:  18  15   6  33 
EHx: 00010126 001D0126 00EF0126 003A0126
 - Jan :: (run 179644)
Fri Mar 5 09:01:58 
the usual HIGH_DEADTIME(without any consequence) message occured
 - Jan :: (run 179644)
Fri Mar 5 09:09:01
 - Anadi
-- Fri Mar 5 09:10:00 comment by...Anadi --  Hourly Plots: 8:00-9:00 Proton losses are oscillating in time. Spike in proton losses, almost 13 kHz.
Fri Mar 5 09:23:18 Unmark cell SVX B2W7L4 being pink. AVDD current right at the limit of the "good" range. - current AVDD I = 1080 - good range [1065,1080]  - Anadi
Fri Mar 5 09:30:21 it appears that there is an issue with the run control monitoring of SVXMon explaining behaviour (no alarm) described here . Bill Badgett and Kaori investigating.  - rainer :: (run 179644)
-- Fri Mar 5 10:40:01 comment by...rainer --  Diagnosis here
Fri Mar 5 09:34:43 News from the 8am integration meeting RE D0 access
  • store will be dropped some time before or around noon
  • D0 will be granted a several hours access to do diagnostics to locate source of noise in the toroid magnet  - rainer
    Fri Mar 5 09:50:40 MCR is planning orbit movement at the end of the store. for safer abort at A0. we have been reminded to call them at 10am to make sure they will give us a warning before they go ahead.  - rainer
    -- Fri Mar 5 10:00:26 comment by...rainer --  gave them a call and asked for warning. they will do it gladly.
    Fri Mar 5 09:56:32 its very cold in here. it might start snowing soon ... - rainer
    Fri Mar 5 10:02:22 
    Found that there was an errant curly-brace  
introduced in the rc.mon.MessageWatcher class (in the  
fer package) that prevented the SVXMon heartbeat missing  
messages from being reported.    I have fixed this  
problem, but to pick the change up you must restart  
runcontrol. 

"For want of a Curly Brace, the Silicon was lost..."
     - W.Badgett
    -- Fri Mar 5 10:10:09 comment by...rainer --  well, the Mariner 1 Venus probe cost over $80 million lost because of a hyphen in the FORTRAN code - compared to that the $20 million for the silicon sounds affordable ....
    Fri Mar 5 10:06:49 overlap CO Franco S. relieved. Sergei taking over. - rainer
    Fri Mar 5 10:08:40
     - anadi
    -- Fri Mar 5 10:09:27 comment by...anadi --  Hourly Plots: 9:00-10:00
    Fri Mar 5 10:14:32 Kaori noticed electric smell. send Monitoring ace and silicon expert investigating to third floor. Also notified process system tech. building manager noticed it, too.  - rainer
    -- Fri Mar 5 10:15:29 comment by...rainer --  monitoring ace reports back that L3 room is cool and no smell up there.
    -- Fri Mar 5 10:29:58 comment by...rainer --  smell might be related to water look. has disappeared for now.
    -- Fri Mar 5 10:30:27 comment by...rainer --  make that water 'leak'. don't have new water look.
    Fri Mar 5 10:27:49 
    L2 decision tomeout (RXPT0
     - Jan :: (run 179644)
    Fri Mar 5 10:27:58 MCR informs us about plan to drop store between 11:00am and 11:15am. they will start beam studies at 10:45am at which time we need to have the HV in the 'before store' configuration. CLC can be OFF as well.  - rainer
    -- Fri Mar 5 10:33:19 comment by...rainer --  calling guillelmo.
    Fri Mar 5 10:33:00 Run 179644 Terminated at 2004.03.05 10:32:43 - RunControl
    Fri Mar 5 10:33:19 Run 179644 TERMINATE: beam studies start at 10:45 - Jan x2080
    Fri Mar 5 10:33:36 Bringing voltages down to Standby (before store condition) for beam studies  - Anadi
    Fri Mar 5 10:38:41 Alarm for PSM: the button went red. Problem in 1RR18D 1 It cleared itself in less than 1 min.  - Anadi
    -- Fri Mar 5 12:55:08 comment by...Anadi --  Same problem at 12:54.
    Fri Mar 5 10:42:32 CDF at 'before-store-configuration' (BeStoCo) ready for MCR beam studies. informed MCR. - rainer
    Fri Mar 5 10:44:51
    for Kaori
     - convery
    Fri Mar 5 10:49:33 Mark Mattson wants to claim pcal,ccal,wcal crates for checkout after dumping this store. - rainer
    Fri Mar 5 10:50:21 I've asked the sci-co if I could do a checkout of the Cal ADMEMs. This will take about 20 minutes, from the time beam stops. I've been given the okay. I will require the cal crates to be released from run control, but I do not need any other part of DAQ. - m mattson
    -- Fri Mar 5 11:28:10 comment by...m mattson --  
    Crates have been released. I have called the control room to let them know. During the spike run, there may be problems on the very 1st event (symptom: bunchcounter mismatch from a majority of the cal crates), which should be resolved with a manual HRR.
    -- Fri Mar 5 11:45:45 comment by...m mattson --  
    Routine Weekly Cal ADMEM Checkout
    Since we are starting a long access, I thought it would be useful to see if there were any problems with the ADMEMs. No obvious problems seen. There are 2 PHA channels which are acting funny, but these are PIN diode channels, not used in data taking. The odd behavior may be a temporary legacy of the laser runs done a few days ago (??). For this access, it is not worth opening up the torroids for these 2 channels.
    Fri Mar 5 10:53:29 
    all CAL crates (CCAL, PCAL, WCAL) are out for studies!
     - Jan
    Fri Mar 5 10:54:06 MCR inquires about our plans to take advantage of the upcoming access. right now we don't have any requests. It's ok to later go in as long as we stay within the time limits given to D0, which is a 5hours max. - rainer
    Fri Mar 5 10:58:02
     - Anadi
    -- Fri Mar 5 11:00:35 comment by...Anadi --  Last plots before dumping the store: 10:00-10:36 Spike in proton losses at 10:21, 30 kHz.
    Fri Mar 5 10:59:20

    We'll be testing shortly a new version of SCPU code for the hardware event builder. It should allow us to monitor a few new performance variables using a new GUI: the time each SCPU spends polling the VRBs for a new event (millisec), the SCPU data size from all VRBs in KB, and the time each SCPU spends waiting for an ATM packet buffer for event transmission (millisec). The median, 25th percentile, and 75th percentile of each quantity is calculated and displayed.

     - Steve Tether

    Fri Mar 5 11:14:29 from MCR elog: Tev quench: F48U, BA, BB, DA & DB. our BLMs are clean. - rainer
    -- Fri Mar 5 11:15:43 comment by...rainer --  update from MCR elog: -- Low Beta's did not quench. False indication on T25, but the buffers are still spinning.
    Fri Mar 5 11:29:32 Run 179645 Activated at 2004.03.05 11:28:41 - RunControl
    Fri Mar 5 11:29:33 Run 179645 ACTIVATE: EVB test - Guillelmo x2080
    Fri Mar 5 11:44:59 Run 179645 Terminated at 2004.03.05 11:44:27 - RunControl
    Fri Mar 5 11:45:00 Run 179645 TERMINATE: no good luck... - Guillelmo x2080
    Fri Mar 5 11:56:53 Disabled lower limit for COT temperature alarm ENDPLATE_ID_LNW. Readout (or sensor) seems to have glitches. Will probably wait till the Mar. 15 access to attempt repairs. - Aseet Mukherjee
    Fri Mar 5 12:09:59 burkhard requests L2 muon trigger crate to do light yield measurement. might impact on running the system (might need L2 autoaccept). Needs about 1 hour. - rainer
    -- Fri Mar 5 13:35:08 comment by...convery --  Burkard reports he is done.
    -- Fri Mar 5 13:55:55 comment by...Burkard --  results of the measurement can be found here
    Fri Mar 5 12:10:19 It was found that very recently YMon has been leaking a lot. The memory overuse is quite likely coming from the newly added DQM part in YMon. I turned of DQM check in YMon in YMon tcl and notified relevant people. - kaori
    Fri Mar 5 12:11:21 
    sergei burov and ray culbertson would like to re-export a file system on b0dau30 - normally
should go smooth but some chance of glitch possible. Need to wait for EVB people to finish, time
estimate for that was ~1 hour.
     - rainer
    Fri Mar 5 12:13:08 Run 179647 Activated at 2004.03.05 12:11:52 - RunControl
    Fri Mar 5 12:13:09 Run 179647 ACTIVATE: another tests - Guillelmo x2080
    Fri Mar 5 12:27:50 Run 179647 Terminated at 2004.03.05 12:27:26 - RunControl
    Fri Mar 5 12:27:51 Run 179647 TERMINATE: SCPUs Display works ^_^ - Guillelmo x2080
    Fri Mar 5 12:45:38

    After a few false starts the test was successful. Now we're putting things back the way they were.

     - Steve Tether

    Fri Mar 5 12:47:47 Run 179648 Activated at 2004.03.05 12:46:16 - RunControl
    Fri Mar 5 12:48:57 Run 179648 Terminated at 2004.03.05 12:48:50 - RunControl
    Fri Mar 5 12:50:04 EVB builder guys done. Sergei will go ahead with b0dau30 export. - rainer
    Fri Mar 5 12:54:27 
    Ray Culbertson requests b0muca00 out of readout for studies.
     - rainer
    -- Fri Mar 5 13:44:08 comment by...Ray Culbertson --  Testing done, crate restored to original state
    Fri Mar 5 12:55:14 taka wants to have all SVT crates for standalone study. - rainer
    Fri Mar 5 12:55:46 silicon guys investigating f511 which was yellow. - rainer
    Fri Mar 5 13:02:05 Run 179649 ACTIVATE: CLC calibs - Jan X2080
    Fri Mar 5 13:06:15 Run 179650 ACTIVATE: BSC calibs - Jan X2080
    Fri Mar 5 13:11:23 Run 179651 ACTIVATE: TOF calibs - Jan X2080
    Fri Mar 5 13:13:46 Sergei done with b0dau30 work. - rainer
    Fri Mar 5 13:19:59 Run 179652 ACTIVATE: SMX QIE calibs - Jan X2080
    Fri Mar 5 13:24:04 Run 179653 ACTIVATE: calorimeter calibs - Jan X2080
    -- Fri Mar 5 13:48:03 comment by...V Tiwari --  
    Several black (bad) channels from PHA all in ch 0 and ch1 which are PIN diodes. These are *not* used
for data taking. I suspect that Plug Laser was ON while taking calibrations. Anyways, this should
not be a cause of concern.
    Fri Mar 5 13:26:28 Run 179654 ACTIVATE: take CLC again to reset ADMEM!!! - Jan X2080
    Fri Mar 5 13:56:24 Run 179655 Activated at 2004.03.05 13:56:05 - RunControl
    Fri Mar 5 13:56:46 Run 179655 ACTIVATE: Rick Run - Jan X2080
    -- Fri Mar 5 14:16:36 comment by...Jan --  
    take the next run for calibs!!!
    Fri Mar 5 14:00:37 Run 179655 Terminated at 2004.03.05 14:00:12 - RunControl
    Fri Mar 5 14:01:05 Run 179655 TERMINATE: to redo spike run - Jan X2080
    Fri Mar 5 14:04:49 Run 179656 Activated at 2004.03.05 14:04:42 - RunControl
    Fri Mar 5 14:05:44 Run 179656 ACTIVATE: Rick Run again (this time with DAQ enabled) - Jan X2080
    Fri Mar 5 14:05:57 SVT GB new firmware is installed to count number of good tracks. Corresponding hwset and svtvme codes are also installed. To check the functionality, we still have pararel (split) data stream. Once we confirmed the functionality, this stream will be immidiately removed. We checked the functionality with 2000 events already and it is fine (no error). We are pretty sure it works but please page SVT just before the shot just in case.  - Taka Maruyama and Jahred Adelman
    Fri Mar 5 14:07:41 
    L1 rate around 60 Hz (standard condition, no beam)
     - AAA_Anadi, Jan :: (run 179656)
    Fri Mar 5 14:13:38 Run 179656 Terminated at 2004.03.05 14:12:53 - RunControl
    Fri Mar 5 14:15:45 Run 179657 ACTIVATE: LED calibs - Jan X2080
    Fri Mar 5 14:16:44 Heartbeat alarm (H box in ICI went purple) but iFix is running.  - Anadi
    Fri Mar 5 14:21:10 Run 179658 ACTIVATE: COT calibs - Jan X2080
    Fri Mar 5 14:21:44 Run 179659 ACTIVATE: muon calibs - Jan X2080
    Fri Mar 5 14:23:53 Run 179660 ACTIVATE: TOF TAC calibs - Jan X2080
    Fri Mar 5 14:48:36 Restart CSL to kill dangling connections to consumers.  - Tony Vaiciulis
    Fri Mar 5 14:51:11 Run 179661 ACTIVATE: start PLUG Laser calib BOTHPLUGS_NORMAL "stabilization run"  - Jan X2080
    Fri Mar 5 15:03:21 Run 179662 ACTIVATE: start Laser Plug Calib BOTHPLUGS_MM_C  - Jan X2080
    Fri Mar 5 15:03:44 CSL restarted by Tony V. - rainer
    Fri Mar 5 15:08:18 Run 179663 ACTIVATE: Plug Laser Calib BOTHPLUGS_MM  - Jan X2080
    Fri Mar 5 15:16:42 
    Last tuesday I did a scan of laser timing for both plug lasers. 
The results are: 
1) the West laser timing is rock-firm; 
2) the East laser timing drifted upwards (the less dangerous direction) by about 20ns.  
I changed the timing for this laser in the RC calibration  
files (BOTHPLUGS_MM, BOTHPLUGS_MM_C, BOTHPLUGS_NORMAL),  
bringing it from 835 to 815 ns. 
3) PIN # 2 in the electromagnetic COW # 5 of the East Plug  
(the one on the left of PMTBOX # 4) has become 
(starting from between Feb 26th and Mar 3rd) extremely noisy 
and should be disconnected.  
Can the ops-manager, please, keep me informed of when there is a possibility to make an access for
that?
     - diego
    Fri Mar 5 15:24:27 Before starting the new test run we have to start iCicle.  - Anadi
    -- Fri Mar 5 15:39:03 comment by...Anadi --  Same problem at 15:38 before starting a new L3 tag test run.
    Fri Mar 5 15:34:08 Run 179667 TERMINATE: l3 failed to config - Jan X2080
    Fri Mar 5 15:37:42 
    In starting test run PHYSICS_2_02[2,424,437] no beam, L3 failed to config. From the error log:



MLE) b0l3pcom2.fnal.gov:main:3:32:26 PM->Error on L3 node b0l3pcom2.fnal.gov (partition 1) Fri Mar 5
15:32:26 2004 l3proxy  

in Config (L3ProxyPartition.cc:1698) 
@L3_TRANS_FAIL - timeout:      b0l3003 b0l3011  b0l3002 b0l3006   b0l3023   b0l3048 

We abort and reset again.
     - Jan and Anadi
    Fri Mar 5 15:46:25 Ramping voltages up for TOF, CMU, CMP, CMX, BMU, CES, CPR, CCR, Plug and MNP. Si and COT still in standby and CLC off. I have to turn on the individual SE CPR. - Anadi
    Fri Mar 5 15:55:29
    Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
    179644 x2BDBC BEAM PHYSICS_2_02 [2,424,431] 21:35:04 10:32:43 12:24:37 424,637,420 5,985,680 1,490,810 661.043 1 1 1
    Totals 15:55:03 12:24:37 424,637,420 5,985,680 1,490,810 661.043
     - End of Shift Report
    Fri Mar 5 15:57:47 Shift Summary: 
    took over shift with store #3273 in progress with inst
lumi 12.31E30. 


History: 
- leak in roof with water dripping into 3rd floor which cannot be fixed until monday 
- one incident of 'electric smell' which might have been caused 
  by water dripping in 3rd floor, but inconclusive. 
- 10:45am CDF at 'before store' HV configuration for beam studies to move  orbit at A0 
- 11:15am beam dumped - TeV quenched but our BLMs are clean. 
- 11:30am D0 access starts for up to 5 hours. we did not go in. 
- EVB people  checked some code changes to monitor readout time.
- b0dau30 re-exported a file system without problems.  
- SVT firmware upgraded 
- bug in rc fixed concerning communication between rc and  
  svxmon. effective since last restart of rc. 
- silicon people working on IB5W1L1 which turned yellow during 
  last shift and produced pinky alarm. 
- L2 people working on muon fibres. 
- Bill Badgett working on ccal crates. 
- all quiet time calibrations not on special request taken.  
  spike run taken as well. whiteboard updated. 
- L3 fix for DSP code trigger table tested but failed due to L3 problems. 
  being investigated. 
- TOF HV being worked on by Gerry Bauer.  

Plan: 
- wait for end of access and shot setup.

    End of Shift Numbers
    CDF Run II 

    Runs                   179644
Delivered Luminosity   108.8  
Acquired Luminosity    105.4  
Efficiency             96.9
     - rainer
    Fri Mar 5 16:06:30
    Summary Plot
     - Anadi