2004 CDF E-Log -- Day shift. Sun Feb 29, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Rainer W. Natasa M. Vadim K. Franco S. JJ


Start of Shift Notes:  

Store #3261 ongoing, Inst Lum 12e30, Stack 209, stacking around 4.2m/h  
Run 179480 in progress.  
COT in compromised setup (HV SL12 off, SL345 reduced gain)  

Trigger table is new one physics: PHYSICS_2_03 [1,431,435]  
Plan: - drop store at 9:30am
      - D0 access afterwards for ~1hr 
      - take data until then.
 Sun Feb 29 08:33:36 MCR calls: will drop store in 1 hr sharp.  - Rainer
Sun Feb 29 08:48:17
 - Hourlyplotter, Vadim
Sun Feb 29 09:10:48 MCR calls at 9:00am - 30 min heads up for dropping the store.  - Rainer
-- Sun Feb 29 09:23:04 comment by...Rainer --  MCR gives 10min warning - taking down HV.
Sun Feb 29 09:21:15 Run 179480 Terminated at 2004.02.29 09:20:36 - RunControl
Sun Feb 29 09:26:15 Run 179480 TERMINATE: End of store in 10 minutes... - Natasa x2080
-- Sun Feb 29 09:33:32 comment by...rainer --  9:29am HV at standby. called MCR to tell them we are ready.
-- Sun Feb 29 09:44:23 comment by...Rainer --  9:44am beam is dropped.
-- Sun Feb 29 10:26:00 comment by...Rainer --  from MCR elog: The store integrated a record luminosity of 3937 nb-1.
Sun Feb 29 09:34:23
 - The last hour of the store. Vadim
Sun Feb 29 09:52:52 Run 179481 ACTIVATE: Cal QIE calibration - Natasa x2080
Sun Feb 29 09:54:08 Run 179481 TERMINATE: stuck while activating - Natasa x2080
Sun Feb 29 09:54:26
 - Fancy Store Long Plots. Vadim.
Sun Feb 29 09:54:59 Run 179482 ACTIVATE: try again: Cal QIE calibration - Natasa x2080
Sun Feb 29 09:58:30 called MCR - another 30min of quiet from now on. - Rainer
Sun Feb 29 09:59:13 Run 179482 TERMINATE: failed to activate again - Natasa x2080
Sun Feb 29 10:00:40 Run 179483 ACTIVATE: Cal QIE calib - Natasa x2080
Sun Feb 29 10:06:56 Run 179483 TERMINATE: failed to activate again.  - Natasa x2080
Sun Feb 29 10:09:07 Run 179484 ACTIVATE: cal qie calib... - Natasa x2080
Sun Feb 29 10:10:44 Run 179484 TERMINATE: failed to activate - Natasa x2080
Sun Feb 29 10:13:04 Run 179485 ACTIVATE: CLC qie calib - Natasa x2080
Sun Feb 29 10:13:06 Run 179485 TERMINATE: failed to activate - Natasa x2080
Sun Feb 29 10:15:46 Run 179486 ACTIVATE: muon calib - Natasa x2080
Sun Feb 29 10:16:46 Run 179486 TERMINATE: ending muon cal - Natasa x2080
Sun Feb 29 10:17:37 failing QIE calibrations in CLC and Calorimeter. Muon Calibration works fine. paged calorimeter front end electronics expert. - Rainer
-- Sun Feb 29 10:34:47 comment by...m mattson --  
Responding to Cal ADMEM pager. This situation is new to me. The error log doesn't show anything unusual. Results are in DBANA, although ACE (and CO?) say results may be funny... I'll look into this Monday. Try taking the calibrations, but don't worry about it if anything goes strange.
-- Sun Feb 29 10:37:43 comment by...Rainer, Natasa --  
  • mark mattson (calo electronics pager carrier) got back to us - runs seem to make it fine into DB (checking via DBANA on expert suggestion.)

  • turns out run control was not configured correctly - most likely did not report back correctly (I spied on b0pcal01 and it saw all the RC commands) 
    
>>> ADMEM_init: Finished...
Client id: 0
FER_initEVB: Connection established. Node b0dap83 Port = 6501
Sent ack to Run Control: 1 READY SUCCESS
Messenger: processStatechange Activate [0]
Starting run 179488 at 1078072312
...1
val = 65535 200
Calibration started.
NULL = 0
Nevent = 20 Npoint = 20 Total Cards = 8 Chan = 0 to 19.
...1
val = 65535 200
iadmem 14 qie  0 np   0 ns    0 val 0x   6 raw 0xc06e cap 2 range 0 data  110 buffer 0
(...)
Size of the buff = 964
Sent ack to Run Control: 1 DONE SUCCESS
Generic ML stack sent.

b0pcal01-> Messenger: processStatechange End [0]
ADMEM: Closing MyADMEM 0
ADMEM: Closing MyADMEM 1
(...)
  • redoing the QIE calibrations just to be sure.
    -- Sun Feb 29 15:12:05 comment by...rainer --  DBANA verification of the Calo QIE calibration produced a number of failed channels

    PEM 15W channel#1 cap 0-3, see CO entry below.

    Sun Feb 29 10:17:58 
    silicon guys working on silicon.
     - Rainer
    Sun Feb 29 10:19:25 Run 179487 TERMINATE: cot cal - Natasa x2080
    Sun Feb 29 10:19:26 Run 179487 ACTIVATE: cot cal - Natasa x2080
    Sun Feb 29 10:32:35 Run 179488 ACTIVATE: cal qie - Natasa x2080
    Sun Feb 29 10:34:44 Run 179488 TERMINATE: succesful calQIE calib - Natasa x2080
    Sun Feb 29 10:36:32 Run 179489 ACTIVATE: clc qie - Natasa x2080
    Sun Feb 29 10:43:22 Run 179490 ACTIVATE: bsc qie calib - Natasa x2080
    Sun Feb 29 10:44:30 Run 179490 TERMINATE: ending bcs qie calib - Natasa x2080
    Sun Feb 29 10:46:58 Run 179491 ACTIVATE: tof qie calib - Natasa x2080
    Sun Feb 29 10:48:40 Run 179491 TERMINATE: tof qie calib - Natasa x2080
    Sun Feb 29 10:51:36 Run 179492 ACTIVATE: sm qie calib - Natasa x2080
    Sun Feb 29 10:51:37 Run 179492 TERMINATE: sm qie calib - Natasa x2080
    Sun Feb 29 10:53:50 Run 179493 ACTIVATE: cem led calib - Natasa x2080
    Sun Feb 29 10:55:48 Run 179493 TERMINATE: cem led - Natasa x2080
    Sun Feb 29 11:00:00 Run 179494 TERMINATE: cem xef calib (??) - Natasa x2080
    Sun Feb 29 11:00:01 Run 179494 ACTIVATE: cem xef calib - Natasa x2080
    Sun Feb 29 11:02:02 Run 179495 ACTIVATE: tof tac calib - Natasa x2080
    Sun Feb 29 11:03:31 Run 179495 TERMINATE: tof tac calib - Natasa x2080
    Sun Feb 29 11:04:15 called MCR - ahave another 15 min of quiet time. - Rainer
    Sun Feb 29 11:12:11 Run 179496 Activated at 2004.02.29 11:12:05 - RunControl
    Sun Feb 29 11:12:59 Run 179496 ACTIVATE: PMT spike run (SUMET5[4,395,405]) - Natasa x2080
    Sun Feb 29 11:21:20 Run 179496 Terminated at 2004.02.29 11:21:10 - RunControl
    Sun Feb 29 11:21:33 Run 179496 TERMINATE: ending spike run - Natasa x2080
    Sun Feb 29 11:26:08
    There are some differences in the pedestals of this BSCQIE calibration graph compared to the reference one: http://www-cdfonline.fnal.gov/ace2help/calibration/fwdcalib.html I don't know if these are big differences...
     - Franco
    Sun Feb 29 11:31:41 While partitioning and configuring in order to test new trigger table, we got a message that Consumer Monitor is not running. It turned out that was true. We started it from RC. - natasha
    -- Sun Feb 29 11:32:27 comment by...natasha --  Procmon was not complaining, though.
    Sun Feb 29 11:33:00 Run 179497 Activated at 2004.02.29 11:32:36 - RunControl
    Sun Feb 29 11:33:29 Run 179497 ACTIVATE: testing new trigger table: PHYSICS_HIGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 11:37:28 Run 179497 Terminated at 2004.02.29 11:37:23 - RunControl
    Sun Feb 29 11:37:55 MCR calls - we are in shot setup now. - Rainer
    -- Sun Feb 29 11:38:42 comment by...Rainer --  MCR calls again - they will inject beam.
    Sun Feb 29 11:38:22 Run 179497 TERMINATE: l3 failed to activate - Natasa x2080
    -- Sun Feb 29 11:45:37 comment by...Rainer --  this is the PHYSICS_HIGHLUM_2_03[1,432,436] trigger table test

    subfarms c8/c14 report 'old'/orange status - ensuing L3_RELAY_FAILURE Error suggests doing so. first activate was done erroneously with AAA_CURRENT - now switch to AAA_SHOTSETUP, excluding offending subfarms and start all over.
    -- Sun Feb 29 11:52:39 comment by...rainer --  paged L3 pager to look into c8/c14.
    -- Sun Feb 29 12:26:54 comment by...Nuno --  All farm nodes seem fine. Converters c08 and c15 also seem okay; I killed leftover relay processes which were still running. After cleanup level3 these two subfarms are ready to be included back online. Guillelmo said he's comming to the CR just in case.

    -- Sun Feb 29 12:32:48 comment by...Guillelmo --  I do not see any error related to c8 and/or c14. There was a relay failure on two nodes (they are already fine), but nothing else. Anyway, all processor and converter nodes are working now.

    Sun Feb 29 11:40:13 Run 179498 TERMINATE: l3 relay failure! - Natasa x2080
    Sun Feb 29 11:47:23 Run 179499 ACTIVATE: testing PHYSICS_HIGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 11:53:08 Run 179499 TERMINATE: ending to check that l3 goes through transitions - Natasa x2080
    Sun Feb 29 11:56:14 Run 179500 ACTIVATE: PHYSICS_HIGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 12:01:33 Run 179500 TERMINATE: exercizing again... - Natasa x2080
    Sun Feb 29 12:04:22 Run 179501 ACTIVATE: PHYSICS_HIGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 12:16:12 Run 179501 TERMINATE: expert looking at l3... - Natasa x2080
    Sun Feb 29 12:23:48 Run 179502 ACTIVATE: PHYSICS_HIGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 12:27:33 Run 179502 TERMINATE: ending at expert's request. - Natasa x2080
    Sun Feb 29 12:30:32 Run 179503 ACTIVATE: PHYSICS_HIGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 12:35:36 final protons being loaded. - Rainer
    -- Sun Feb 29 12:42:29 comment by...rainer --  9885e9 protons loaded.
    Sun Feb 29 12:38:43
    DateTimeBLMDose
    2004.02.2912:34:53W Inner BLM0.01RADS
    2004.02.2912:34:53W Outer BLM0.00RADS
    2004.02.2912:34:53E Inner BLM0.19RADS
    2004.02.2912:34:53E Outer BLM0.56RADS
    Integrated dosage - Vadim
    Sun Feb 29 13:00:00 antiprotons loaded. - Rainer
    -- Sun Feb 29 13:00:22 comment by...rainer --   1534e9 pbars loaded.
    Sun Feb 29 13:02:02 Run 179503 TERMINATE: preparing for AAA_CURRENT.... - Natasa x2080
    Sun Feb 29 13:22:33 Run 179504 Activated at 2004.02.29 13:21:42 - RunControl
    Sun Feb 29 13:22:34 Run 179504 ACTIVATE: AAA_CURRENT: PHYSICS_HGHLUM_2_03[1,432,436] - Natasa x2080
    Sun Feb 29 13:25:17 Scraping complete at 13:15. Store # 3263, proton 8816E9 pbars 1247E9, initial luminosity 50.65E10.  - Rainer
    Sun Feb 29 13:27:18 From Run Coordinator's eLog:

    "Sun Feb 29 13:15:47-
    We tried most of the same pbar strategy on this shot with a similar
    (large) sized stack. Everything looked promising, although a bit
    lower than Friday's shot. Then the C17 separator sparked during the
    Tevatron ramp, blowing up the beam. Disappointing luminosity,
    but it should have a better than normal lifetime (because the beam
    is bigger). - JPM"

    Initial lumniosity was 51.1E30.  - JJ
    Sun Feb 29 13:32:55
     - Shotsetup shots. Vadim
    Sun Feb 29 14:02:55 SVX07 trigger inhibit: CAEN crate#14 glitched. the glitched crate resumed communication right away (i.e. no stuck communication failure), but we hockerized it anyway to be sure. after resetting trips continued smooth running. got a brief ISL02 trigger inibit alarm, but no box turns red. suspect correlation with hockerization of other crates. - Rainer
    Sun Feb 29 14:38:47
     - Last hour in pictures.
    Sun Feb 29 15:02:31 have second time: 
     
16'25" b0dap84.fnal.gov:ConsumerErrorRe:2:16:26 PM->Runtime Error 1, Event 2176, RunNum 179504:
SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B4/W6/L3/C0-5 . --> 

 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!! 
59'4" b0dap84.fnal.gov:ConsumerErrorRe:2:59:06 PM->Runtime Error 2, Event 4527, RunNum 179504:
SvxMon Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B4/W6/L3/C0-5 . --> 

 Attention !!!. CER_SVXMON_HALT_RECOVER_RUN_ERROR !!!
    expert is looking into it. - Rainer
    Sun Feb 29 15:09:44
    Some values out of tolerance in calqie, calibration run 179488
     - Franco
    Sun Feb 29 15:23:12 Run 179504 ACTIVE: 
    We had the following error:
Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data in 
 SCPU b0eb19. 
HRR fixed the problem.
     - Natasa x2080
    Sun Feb 29 15:30:52 Run 179475 Terminated at 2004.02.28 20:56:44 - RunControl
    Sun Feb 29 15:31:18 
    Marked run 179475 "Potentially Good" by hand as there  
was a RunControl problem Saturday night that prevented  
DAQ Ace from checking the box.
     - W.Badgett :: (run 179475)
    Sun Feb 29 15:44:45
     - H.P. -- Vadim
    Sun Feb 29 15:55:16
    Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
    179480 x2BD18 BEAM PHYSICS_2_03 [1,431,435] 21:39:09 09:20:36 09:54:07 409,176,969 5,173,568 1,175,310 493.856 1 1 1
    179504 x2BD30 BEAM PHYSICS_HIGHLUM_2_03 [1,432,436] 13:21:42 02:21:34 50,999,693 2,300,776 368,121 376.283 1
    Totals 15:55:02 12:15:42 460,176,662 7,474,344 1,543,431 870.139
     - End of Shift Report
    Sun Feb 29 16:00:32 Shift Summary: 
    Started shift with store #3261 colliding at Inst Lum
12e30. stack  

was 209. new trigger table PHYSICS_2_03 in effect. 

History: 
- dumped beam cleanly at 09:30am 
- Quiet time calibrations 
- Highlum trigger table PHYSICS_HGHLUM_2_03[1,432,436] test during shot setup 
- Silicon diagnostics work 
- Store #3263 started smoothly with inital luminosity 50.65E30 
  and new trigger table PHYSICS_HGHLUM_2_03[1,432,436] 
- Spark on C17 separator during ramp makes this an underachiever store 
  in terms of luminosity, but lifetime should be good. 
- Silicon crate#14 acting up, hockerized and reset fine. 

Plan: 

- UDPS test at the end of this store. 
- take data with new 2_03 sets of trigger tables (needs to be still set 
  manually in the run configurations) 
- if there is another incident of consumer monitor CPUs going down  
  like on last owl, page JJ to arbitrate expert help if needed.

    End of Shift Numbers
    CDF Run II 

    Runs                   179480,179504
Delivered Luminosity   475.3   
Acquired Luminosity    425.3   
Efficiency             89.5
     - Rainer