2004 CDF E-Log -- Eve shift. Sat Feb 14, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Doug Glenzinski Natasha Milandinovic Vadim Khotilovich Sunghyun Chang Mary Convery


Start of Shift Notes:  

 
Run in progress.  Plan is to continue taking data.

Sat Feb 14 16:19:05 Had a CSL Calibrations Consumer reporting an error to ProcMon. I did all the checks listed on the web pages and everything seemed fine. After a while Procmon reported everything fine. Must be that the problem lasted for a very short time. - natasha
Sat Feb 14 16:36:00
 - (hrl.plts) Vadim
Sat Feb 14 16:39:25 Run 179104 ACTIVE: Had L2 decision timeout: (MLE) b0wcal03:Messenger:3:19:57 PM->Runtime Error 2, Event 7650292: Bunch counter mismatch, mismatch count = 1 L2 Decision Timeout.[RXPT] Auto HRR worked. - natasha x2080
Sat Feb 14 16:48:41
Subfarm 9 is fixed now and may be included in the next run  
after Level3 cleanup 

The problem was in converter 09 

 - Arkadiy
Sat Feb 14 17:36:32
 - (Hourlies'R'Us) Vadim
Sat Feb 14 18:38:57
 - Vadim
-- Sat Feb 14 18:40:19 comment by...Vadim --  Hourly plots. B0PBMS shows a 2x spike around 17:42 but it might be a fake (too narrow).
Sat Feb 14 19:09:15 MCR are going to move collimators. Put Si, COT, CMU/P/X, BMU, CES, CPR, CCR into standby. - Vadim :: (run 179104 )
Sat Feb 14 19:27:34
MCR calls back.  they're done playing w/ collimators.  AG losses 
actually a bit worse (13k before they played, now 14k).  They 
wanted to try to reduce these losses a bit before shift change 
(at 20:00) so that they could avoid having to call-in experts. 
We'll keep an eye on it.  It was quite flat for the first 3 hours 
of our shift, they expect it to stay this flat. 

Also, next shot will likely take place when avg lumi=10E30 and 
stack big enough.  MCR guessed around 00:00, but said they'd have 
the next crew chief call once they'd decided on a shift plan
 - Doug :: (run 179104)
Sat Feb 14 19:35:50 Put HVs back to on. The SVX cell B1 W8 L1 turned pink when we we going to standby. Unmark + HRR made it greeeen again. - Vadim :: (run 179104 )
Sat Feb 14 19:43:44
 - /plots/ Vadim
Sat Feb 14 20:17:04
Dr. Pitts calls with a trigger table plan for the start of the 
next store: 

	- start w/ the HIGHLUM table on the whiteboard, 
	  DPS disabled 

	- once lumi < 40 E30, switch to regular table 
	  DPS enabled 

He suggests we take a short run w/ the HIGHLUM table towards the 
end of this store since this table hasn't been used since Jan  
(he's worried that perhaps a tagset has changed). 
 - doug
Sat Feb 14 20:18:09
AG losses have fallen back to their previous ~13 kHz.
 - doug
Sat Feb 14 20:49:06
 - Vadim
-- Sat Feb 14 20:51:07 comment by...Vadim --  Hr.Pl.: Around 20:08 there was an 1 min long drop in SBDMS (RF bucket length) but we didn't have any problems.
Sat Feb 14 21:37:50
 - Vadim
-- Sat Feb 14 21:38:54 comment by...Vadim --  Hr.Pl.: had a narrow spike in B0PBSM @20:57
Sat Feb 14 21:47:33
Phoned MCR.  They say they'll keep store until lumi is 10E30, 
which they guess will occur sometime between 0400-0600.  Stack  
is presently 154 mA. 

 - doug
Sat Feb 14 21:52:11 Run 179104 Terminated at 2004.02.14 21:51:51 - RunControl
Sat Feb 14 21:52:34 Run 179104 TERMINATE: ending in order to see trigger rates with another trigger table. - natasha x2080
Sat Feb 14 21:53:43
talked to Mary C (ops manager tonight) and she agreed that  
kevin's plan sounded like a good one. 

we'll pause now to perform this test w/ HIGHLUM table.  that way 
kevin can check the rates before going to sleep.  if it goes well 
then the owl shift can "relax" and ride-out the rest of the 
store.
 - Doug
Sat Feb 14 22:08:51
now the fun begins... 

natasha tries to bring-in L3 subfarm 9 as per arkady's request. 
it doesn't work.  things get worse as aces try to recover. 
L3 paged.  Jeff M here working.
 - Doug
-- Sat Feb 14 22:38:09 comment by...Jeff / Ilya --  Before beginning a new run, it was necessary to cleanup L3 because
subfarm 09 had been marked offline for the previous run(s).
Thus, the nodes of this subfarm were not yet running the client
processes necessary to communicate with run control.
All that was necessary to get going again was the clean-up;
Arkadiy's message above indicates this, but not very emphatically.
Sat Feb 14 22:21:52 Run 179105 Activated at 2004.02.14 22:21:45 - RunControl
Sat Feb 14 22:23:17 Run 179105 ACTIVATE: PHYSIC2_HIGHLUM_2_01[1,405,420] - natasha x2080
Sat Feb 14 22:29:53 Run 179105 ACTIVE:
Had the following messages in the error display, when starting the run:

(MLE) b0l3pcom2.fnal.gov:main:10:19:04 PM->Error on L3 node b0l3157 (partition 1) Sat Feb 14
22:19:02 2004 l3_node 

  in main (l3_node.c:1740)
  @L3_EXE_INIT_FAIL:  Error initializing filter 0

b0dau32.fnal.gov:csl_mon_send:10:20:17 PM->CSL received a "BAD EVENT" in partition 1.[MLW]
(MLE) b0l3pcom2.fnal.gov:main:10:20:18 PM->Error on L3 node b0l3137 (partition 1) Sat Feb 14
22:20:15 2004 l3_node 

  in exe_exchange (l3_ana_exe.c:567)
  @L3_EXE_TERM:  Filter 1 died

(MLE) b0l3pcom2.fnal.gov:main:10:20:18 PM->Error on L3 node b0l3145 (partition 1) Sat Feb 14
22:20:15 2004 l3_node 

  in exe_exchange (l3_ana_exe.c:567)
  @L3_EXE_TERM:  Filter 0 died

(MLE) b0l3pcom2.fnal.gov:main:10:20:18 PM->Error on L3 node b0l3145 (partition 1) Sat Feb 14
22:20:15 2004 l3_node 

  in exe_exchange (l3_ana_exe.c:567)
  @L3_EXE_TERM:  Filter 1 died
 - natasha x2080
Sat Feb 14 22:31:03
 - Andrei Loginov
-- Sat Feb 14 22:37:58 comment by...Andrei Loginov --  
Event Display pictures - yellow circle consists of xft hits.
Picture #1 - stadard COT view with yellow circle
Picture #2 - close-up view of the arc with "SetDisplayCells" on,
one can see number of xft hits.
You can see the question about this circle here:  here Time information for XFT is provided by XTC card.
-- Sat Feb 14 22:40:32 comment by...a. --  
What I know about coloring scheme for XFT hits is:

 green  : prompt hit
 blue   : delayed hit
 yellow : both

-- Sat Feb 14 22:44:35 comment by...doug --  
the xft SL2 is masked "on" in order to account for the reduced
COT voltge on the inner SL (in the case of SL2, reduced==off)

-- Sun Feb 15 03:15:26 comment by...Cheng-Ju --  Any COT wire masked on in the XFT will show up in the event display as a small yellow circle at the location of the COT wire. We have masked on entire XFT SL2 (over 2000 wires) --> yellow circular band. The little red square box in the center of a cot cell (above right plot) indicates the location of XFT segment as found by the XFT. Since all wires in SL2 are turned on, XFT is finding a segment for every XFT phi bin.
Sat Feb 14 22:33:25
From the MCR elog: 


The Plan:   
Terminate store when the stack reaches 190E10 or the average   
luminosity reaches 10E30 (whichever comes first). After store,   
stack to 50E10 and transfer to Recycler. Shot strategy, Protons   
265-275E9 per bunch, pbars per guidelines.  
 - JPM   
 - Doug
Sat Feb 14 22:36:37 Trigger cross sections for run 179105 (PHYSICS_HIGHLUM_2_01_v1) look good. We should run this table at the beginning of the next store. Thanks for taking the test run early. - KPitts :: (run 179105)
Sat Feb 14 22:40:53 Run 179105 Terminated at 2004.02.14 22:40:42 - RunControl
Sat Feb 14 22:41:08 Run 179105 TERMINATE: ending the trig rates test run. - natasha x2080
Sat Feb 14 22:41:21
 - /H.P./ Vadim
Sat Feb 14 22:58:05 Run 179106 Activated at 2004.02.14 22:57:37 - RunControl
Sat Feb 14 22:58:38 Run 179106 ACTIVATE: AAA_CURRENT:PHYSICS_2_01[4,416,424] - natasha x2080
Sat Feb 14 23:03:01
 Level3 failed Partition several times complaining about 
transition failure on the processor node b0l3157. It  
turned out that the node ran out of disk space because of 
enourmous number of core.xxxxx files. I cleaned up  
the disk, now the node is ok, so is level3. I will 
suggest Guillelmo to check why cores accumulate.There is 
no immediate danger for other all nodes, as far as I can see, 
there is plenty of disk space. 
 - Ilya
Sat Feb 14 23:14:01
Irina comes-in to update version of pb_hv6 that's monitoring 
pisaboxes.  When she/steve fixed yesterday's problems they  
apparantly downloaded an old version.
 - Doug
Sat Feb 14 23:15:18
AG losses have poked-up just above 15k a couple times.  both 
times they went back below w/i about a minute.  Should keep an 
eye on it.
 - Doug
Sat Feb 14 23:42:57
 - /H.P./ Vadim
Sat Feb 14 23:55:45
Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
179104 x2BBA0 BEAM PHYSICS_2_01 [4,416,424] 05:40:12 21:51:51 15:15:13 809,302,272 11,555,300 2,244,223 1115.789 1 1 1
179105 x2BBA1 BEAM PHYSICS_HIGHLUM_2_01 [1,405,420] 22:21:45 22:40:42 00:16:36 1,602,986 93,762 17,241 14.033 1 1 1
179106 x2BBA2 BEAM PHYSICS_2_01 [4,416,424] 22:57:37 00:56:04 37,970,620 477,844 104,423 45.385 1
Totals 23:55:02 16:27:55 848,875,878 12,126,906 2,365,887 1175.206
 - End of Shift Report
Sun Feb 15 00:04:53 Shift Summary:
  * run ;  TeV is planning to squeeze store 3231 for a 
    good while longer. 

        - at ~20:00, ramp silicon+wire chambers to stby for MCR 
          to play w/ collimators 

        - at ~22:00 took a short run (179105) w/ HIGHLUM table 

                > K Pitts said rates looked ok 
                > also got L3 subfarm 9 back 

        - at ~23:00 went back to dflt table 

  * plan: 
        - continue taking data w/ dflt table until end of store 
                > watch abort gap losses 

                > if collimator gynastics required, don't forget 
                  to reduce voltages as per whiteboard 

        - between stores: 
                > checkout, calibrations, L2 torture 

        - run plan for new store: 

                > begin w/ HIGHLUM table, DPS disabled 

                > once lumi < 4E31, switch to dflt table, DPS 		 
                  enabled 


End of Shift Numbers
CDF Run II

Runs                   179104,05,06(still going)
Delivered Luminosity   445  
Acquired Luminosity    374  
Efficiency             84%

 - doug