2004 CDF E-Log -- Eve shift. Tue Mar 2, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Stephan Lammel Ian Vollrath Chris Marino Diego Cauz Mary Convery


Start of Shift Notes:  

Tevatron turnon in progress

Tue Mar 2 16:42:45 Run 179550 ACTIVATE: XTRP - 2TT - test for firmware to not send short tracks to L2 -- wpipee_98x12.mcs,
wpipeo_98x12.mcs, and simtracks_short.dat - Christopher x2080
Tue Mar 2 16:42:46 Run 179550 TERMINATE: XTRP - 2TT - test fails due to L2 decision work -- try later - Christopher x2080
Tue Mar 2 17:04:27 Taka will "borrow" SVT crates for 15 min - Stephan
-- Tue Mar 2 17:14:55 comment by...Stephan --  Taka reports work on SVT crate is complete
Tue Mar 2 17:10:30 Run 179552 ACTIVATE: Start the run for SVT readout. (for coming real data test) - maruyama
Tue Mar 2 17:10:56 Run 179552 TERMINATE: end the run for SVT SVDD bank readout - maruyama
Tue Mar 2 17:11:31
what time is it? we don't know - got some "fatal clock errors". talked to some L2 to guys to
see if it was related to what they are doing - they don't know. cleared errors on the advice of bill
badgett. will keep watch as we go into shot setup.
 - ian
-- Tue Mar 2 18:03:50 comment by...ronmoore --  The Tevatron Low-Level RF processor was rebooted at that time. (CDF Clock is derived from the Tevatron RF.)
Tue Mar 2 17:22:52
 I borrow the SVT crates to test the data readout with SVDD  
readout since this is needed to read out split real data for 
new GB firmware test. It looks OK. We can directly compare the 
split two data output.  

 We wait for the real data coming.
 - Taka
Tue Mar 2 17:37:43 I updated the XMon constants with two run ranges where the COT was in the new standard compromised setup. (runs 179103-179132 and runs 179463-179505). - Charles Plager
Tue Mar 2 17:47:13 MCR called, they will be putting beam into the Tevatron in a few minutes... - Stephan
Tue Mar 2 18:07:27
icicle crashed (2nd time in last hour or so). restarted it.
 - ian
Tue Mar 2 18:37:58
brief (~3s) cot systems alarm and gas alarm. called cyro, they are working. ignore until
further notice. 


VOLUME FROM SYSTEMS ALARMS IS QUITE LOW. HOW DO WE TURN IT UP?
 - ian
-- Tue Mar 2 18:44:38 comment by...ian --  
jj to the rescue! just the standard volume control on a windows pc.

Tue Mar 2 18:41:33 Run 179554 ACTIVATE: XTRP - 2TT - test for firmware to not send short tracks to L2 -- wpipee_98x12.mcs, wpipeo_98x12.mcs, and simtracks_short.dat (II) - Christopher x2080
-- Tue Mar 2 18:51:12 comment by...Christopher --  Started fine, but dropped to 0 Hz at about 100 events
Tue Mar 2 19:42:03 Run 179555 ACTIVATE: XTRP - 2TT - test for firmware to not send short tracks to L2 -- wpipee_98x12.mcs, wpipeo_98x12.mcs, and simtracks_short.dat (III)
L2 work done so this should work - Christopher x2080
Tue Mar 2 19:44:05 Run 179556 ACTIVATE: XTRP - 2TT - wpipee_98x12.mcs, wpipeo_98x12.mcs, and simtracks_short2.dat - Christopher x2080
Tue Mar 2 20:15:59
I have finished working on debugging the cable testing  
program using the CCAL_## and LEVEL#_CAL_## crates.  They  
are in their original configuration again. 
Have given them back to DAQ Ace to run tests.
 - W.Badgett :: (run 179557)
Tue Mar 2 20:23:13 IFIX reports alarm for SVX and L00:
There are no obvious trips in the Silicon GUI and we are at standby
Before there is time to investigate details in IFIX alarm clears  - Christopher
Tue Mar 2 20:26:20 Run 179557 TERMINATE: end cosmics - Ian x2080
Tue Mar 2 20:30:44
There is something seriously wrong with the ADMEM  
in crate b0ccal05, slot 20: 

(MLE) b0ccal05:Messenger:8:27:59 PM->VISIONread/write failed in slot 20 
MLE) b0ccal05:Messenger:8:28:09 PM->Runtime Error 1, Event 1: Bunch counter mismatch, mismatch
count = 1 

(MLE) b0ccal05:Messenger:8:28:10 PM->Event 2: Bunchcounters in slot 17 (BC=90) and slot 20
(BC=255) disagree (L2B=0)  

L2B 17:  90   3  55   9 
L2B 20: 255 255 255 255 

The VME readout registers simply aren't responding. 
(However, the IDPRom does seem to work.) 

NEed to contact ADMEM experts. 
 - W.Badgett :: (run 179558)
-- Tue Mar 2 20:34:47 comment by...STephan --  paged CCAL pager 266-8338
-- Tue Mar 2 21:04:42 comment by...Stephan --  no reply from CCAL pager, page general cal SLP 218-9500
-- Tue Mar 2 21:33:13 comment by...Stephan --  
Larry Nodulman (general cal SPL pager) called back. He suggests
to call people at home:
   - Vivek Tiwari no answer office x2032 and home x4760
   - Mark Mattson no answer office x3596 no listed home number
   - Robin Erbacher no answer office 3442 and home (left message)
Larry also suggested Rick Tesarek, who we reach at home
Mark Mattson (CCAL pager) calles in after about an hour. Bill is
working with Rick on the phone.

-- Tue Mar 2 21:57:05 comment by...m mattson --  
That's BS. The pager says you called at 8:45, and I called back at 9:15. That's half an hour, not an hour. (It was also the earliest I could get to a pay phone.)
Tue Mar 2 20:32:37 Run 179559 ACTIVATE: cosmics_notracks[6,392,403] without ccal05 - Ian x2080
Tue Mar 2 20:36:35 Run 179559 TERMINATE: end cosmics - Ian x2080
Tue Mar 2 20:38:55 MCR (Joel) called they are starting shot setup - Stephan
Tue Mar 2 20:47:28 Run 179560 ACTIVATE: COSMICS_NOTRACKS[6,392,403] without ccal05 - Ian x2080
Tue Mar 2 20:47:28
Have marked ADMEM in slot 20, crate b0ccal05 as  
offline, so readout of the rest of the crate can continue. 

ADMEM experts should investigate;   is a power-cycle  
warranted in this case? 
 - W.Badgett :: (run 179561)
Tue Mar 2 21:00:02 Run 179560 TERMINATE: end cosmics - Ian x2080
Tue Mar 2 21:17:23 Run 179562 ACTIVATE: COSMICS_NOTRACKS[6,392,403] with ccal05 - Ian x2080
Tue Mar 2 21:22:06 Run 179562 TERMINATE: end cosmics - Ian x2080
Tue Mar 2 21:25:33 Run 179563 ACTIVATE: Start cosmic runs - Ian x2080
Tue Mar 2 21:26:05 Run 179563 TERMINATE: End cosmic - Ian x2080
Tue Mar 2 21:26:14 Please do a no-beam test of TEST_MET15_DECOUPLED[1,434,403] - jdl
-- Tue Mar 2 21:57:55 comment by...jdl --  Beam-on request is for 100k events during period with no COT. Silicon on please.
-- Tue Mar 2 22:07:23 comment by...rainer --  
silicon will only go on by expert permission (SPL and/or pager carrier) after the stability of the trigger rates have been demonstrated in a run w/o silicon beforehand. that's what you might have implied. please refrain from direct instruction of shift crews to switch on silicon for non-standard data taking. in these circumstances this is always done by experts.

-- Tue Mar 2 22:41:55 comment by...convery --  We have a request for this trigger table as well as the Jet3 table. We will run without silicon to determine rates before including silicon (while the silicon experts are present).
Tue Mar 2 21:53:53 Run 179567 TERMINATE: b0ccal05 has been recalibrated (alone!) and seems to be functioning fine with the slot=20 ADMEM included. Under the mandate of Rick Tesarek, I downloed the L1 Trigger algorithm via the stand-alone qietest program, and that seemed to do the trick.

Note that to make sure the calibration constants are aligned across the calorimeter, and that ccal05 in particular has new quiet-time constants, Rick requests that a full QIE Calibration be done during the next downtime, with an accompanying CALIB_LOAD supervised by ADMEM experts.

 - W.Badgett
-- Tue Mar 2 21:57:41 comment by...Stephan --  Thanks go to Rick Tesarek who worked with Bill 30 min on the phone (covering for the CCAL team).
-- Wed Mar 3 07:38:05 comment by...R.J. Tesarek --  
ADMEM problems:
The problem observed with the ADMEM is consistent with the Trigger/Readout FPGA becoming corrupt. We've had 4-5 instances of this type of problem in the last couple of months. No instances of this problem prior to summer 2003. The solution is to reload the FPGA program and recovery seems to be complete. Bill reloaded the entire crate of ADMEMs (not just the offending module). He then took a QIE calibration with just ccal05 and loaded the constants/FRAMS to ensure that everything was ok. The pedestal problems noted below are consistent with beam in the machine.

While this specific problem is fixed, this type of problem seems to be recurrant and does not bode well for the future. Something has corrupted the program in the FPGA. I am suspicious of the software in the crate rather than an external effect (eg, radiation, power, etc).


Tue Mar 2 21:54:45
One more note on the beam-on calibration: 

Did a QIE calibration in crate b0ccal05 under the  
orders of Rick Tesarek.  There were three channels/caps  
with too big RMS, as listed below.   Rick is confident  
that these are due to beam in the Tevatron and should  
not cause problems. 

This can be corrected by taking another QIE Calibration  
during the next quiet-time period.  Please contact ADMEM  
expert (Rick, Mark, or Vivek) when this is possible. 


*******************   Start of FAILURE messages   ************** VALUE    CAP RANGE** 
* ADMEM=17 Chan=14 Failure  6: Raw pedestal RMS TOO big       (   21.5842) ( 0) ( 0) * 
* ADMEM=19 Chan= 0 Failure  6: Raw pedestal RMS TOO big       (   15.3082) ( 2) ( 0) * 
* ADMEM=19 Chan=15 Failure  6: Raw pedestal RMS TOO big       (   16.2651) ( 2) ( 0) * 
 - W.Badgett :: (run 179565)
-- Tue Mar 2 22:01:02 comment by...m mattson --  
The odd slots are all CHA. It is possible that the LED was still on, in which case the wide RMS is understood. Like Rick says, it should not cause problems, and will be corrected next quiet time.
Tue Mar 2 22:01:02 Run 179566 ACTIVATE: more cosmics - Ian x2080
Tue Mar 2 22:05:14 Run 179566 TERMINATE: End cosmic - Ian x2080
Tue Mar 2 22:11:37 Run 179568 TERMINATE: end run - Ian x2080
Tue Mar 2 22:15:12 Run 179569 ACTIVATE: start yet another cosmic run - Ian x2080
Tue Mar 2 22:16:05 Run 179569 TERMINATE: end cosmic - Ian x2080
Tue Mar 2 22:20:06 From MCR e-log: Linac RF water pump has tripped off...water coming from beneath the Filament Rectifier. - Stephan
Tue Mar 2 22:22:52 Run 179570 ACTIVATE: start cosmic - Ian x2080
Tue Mar 2 22:41:40 Run 179570 TERMINATE: end run - Ian x2080
Tue Mar 2 22:53:18 Run 179571 ACTIVATE: start cosmic - Ian x2080
Tue Mar 2 23:20:50 Run 179571 TERMINATE: end cosmic - Ian x2080
Tue Mar 2 23:29:32 Run 179572 ACTIVATE: start cosmic - Ian x2080
Tue Mar 2 23:30:14 Run 179572 TERMINATE: end cosmic - Ian x2080
Tue Mar 2 23:39:29 Store 3271: Final Protons being loaded - Stephan
Tue Mar 2 23:43:30
DateTimeBLMDose
2004.03.0223:39:45W Inner BLM1.98RADS
2004.03.0223:39:45W Outer BLM1.29RADS
2004.03.0223:39:45E Inner BLM17.46RADS
2004.03.0223:39:45E Outer BLM713.01RADS
Integrated dosage - Christopher
Tue Mar 2 23:48:44 Store 3271: Final antiprotons being loaded - Stephan
Tue Mar 2 23:56:11
Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
Totals 23:55:02 ::
 - End of Shift Report
Wed Mar 3 00:00:48 Shift Summary:
Store 3271 in preparation 
   - final protons are loaded 
   - final antiprotons being loaded 

Plan is to take data with JET_ST3_DECOUPLED[1,433,403] with Silicon 
   - test TEST_MET15_DECOUPLED[1,434,403]

End of Shift Numbers
CDF Run II

Runs                   
Delivered Luminosity   0  
Acquired Luminosity    0  
Efficiency             100

 - Stephan