Eve shift. Fri Feb 27, 2004 (174/176)

2004 CDF E-Log -- Eve shift. Fri Feb 27, 2004
SciCo	DAQ Ace	Monitoring Ace	CO	(Operations Manager)
St. Lammel/R. Harris	Jan Ehlers	Anadi Canepa	Diego Cauz/Josh Tuttle	JJ. Schmidt

Start of Shift Notes:

final proton bunches loaded for store 3261 
shooting from stack of 220 mA 
silicon SRC problems, Reiner is working on the issue and will advise later if silicon is available 
plan to take data with COT SL-1/2 offf, SL-3/4/5 reduced HV/gain

Fri Feb 27 16:17:30 Run 179461 ACTIVATE: test of test - Jan x2080

Fri Feb 27 16:17:46 Run 179461 TERMINATE: end test of test - Jan x2080

Fri Feb 27 16:20:16 paged online monitor consumer operator cannot use ssh on online monitoring systems to restart the online monitors - Stephan/Robert
-- Fri Feb 27 17:12:07 comment by...josh -- kaori called back and helped fix the ticket problem

Fri Feb 27 16:25:33 final ani-protons loaded - Stephan/Robert

Fri Feb 27 16:27:41 Run 179462 ACTIVATE: AAA_SHOTSETUP - Jan x2080

Fri Feb 27 16:33:47 Run 179462 TERMINATE: end shortsetup - Jan x2080

Fri Feb 27 16:43:43 store #3261 scraping complete TevPR at 8560 E09 TevPB at 1430 E09 - Stephan/Robert

Fri Feb 27 16:45:36 store #3261 initial luminosity 72 E30 - Stephan/Robert

Fri Feb 27 16:49:01 run will start without Silicon, when COT is up... - Stephan/Robert

Fri Feb 27 16:50:23 Run 179463 Activated at 2004.02.27 16:49:54 - RunControl

Fri Feb 27 16:52:57 Run 179463 ACTIVATE: record initial luminosity physics run ;-)!! [PHYSICS_2_02[1,420,427] - Jan x2080
-- Fri Feb 27 16:54:52 comment by...Guillelmo -- But NO silicon... sniff sniff

Fri Feb 27 16:54:39 manually overcome trigger inhibit from COT (since the two innermost layers are switched off SILICONS OUT - 179463 :: (run Jan)

Fri Feb 27 17:08:26

- Anadi
-- Fri Feb 27 17:08:56 comment by...Anadi --

Shot setup plots 15:00-16:50

Fri Feb 27 17:17:10

L2 Decision Done Timeout (TWICE followed by the LOOP) but 
manual HRR helped!

- Anadi & Jan :: (run 179463)

Fri Feb 27 18:04:52

Heap Corrupt Fatal Error: 
automatic request to shepherd COT04 
Done -> Fine

- Anadi'sche & Jan :: (run 179463)

Fri Feb 27 18:11:19

- Diego
-- Fri Feb 27 18:12:48 comment by...diego -- PEM and especially PHA high-eta towers look hot

Fri Feb 27 18:15:53

- Anadi & Run Jan
-- Fri Feb 27 18:16:40 comment by...Anadi -- Hourly Plots : 17:00-18:00

Fri Feb 27 18:32:23

due to frequent (every few minutes) CLIST errors L2 experts got paged

- Jan :: (run 179463)
-- Fri Feb 27 18:36:49 comment by...Vadim/Burkard -- We pulled back the LVDS splitter today to look for a potential effect. The current configuration is identical to what we had before the LVDS splitter went in. The average CLIST error rate is similar to what we had over the last weeks. The higher L2A rate made it more obvious, but I don't think this is out of the ordinary.

Fri Feb 27 18:47:42

DONE TIMEOUT: 

(MLE) b0l3pcom1.fnal.gov:main:6:42:42 PM->Host b0eb14.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 2294539. 
The VRB had no event. 
(MLE) b0l3pcom1.fnal.gov:main:6:42:43 PM->Host b0eb14.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 2294540. 
The VRB had no event. 
(MLE) b0dap73.fnal.gov:Thread-404:6:42:49 PM->Done Timeout: COT_13

- Jan :: (run 179463)

Fri Feb 27 18:55:21 spike in proton losses at 6:33 pm, called MCR and ask to investigate - Stephan

Fri Feb 27 18:55:43 Changed Stage0 and SiliMon version to newly built cdfsoftb0 version 5.3.1pre5a. Other monitors are still running with 5.3.1pre4c. Memory usage growth of Stage0 should be in control for this new Stage0. Alexy is monitoring the memory usage remotely. Please do not restart Stage0 at beginning of a shift. New SiliMon still leaks a lot. I turned off trackmonitor to stop the leak. YMon did not accept any events (5.3.1pre4c). I turned of DQM part, then started accepting events. - kaori :: (run 179463)

Fri Feb 27 18:56:01

Spike in proton losses. Called MCR to investigate

- Anadi
-- Fri Feb 27 18:57:20 comment by...Anadi --

Spike: almost 35 kHz (LOSTP)

Fri Feb 27 19:14:26

- Anadi
-- Fri Feb 27 19:14:49 comment by...Anadi --

Hourly Plots: 18:00-19:00

Fri Feb 27 19:18:51 MCR called regarding proton spike at 6:33 pm: its correlated with spike in quake meter in tunnel - Stephan

Fri Feb 27 19:21:10 crate PCAL_00 caused an 'eventbuilder error' - Jan :: (run 179463)

Fri Feb 27 19:22:13

there are red plots here!

- diego
-- Fri Feb 27 23:54:29 comment by...Larry Nodulman -- probably need new standards for xft at 6 oclock

Fri Feb 27 19:24:04

again red plots

- diego
-- Fri Feb 27 20:25:59 comment by...ps -- fewer tracks = fewer muons, that's the phi plot; charge/pt is the trigger mix

Fri Feb 27 19:52:20 From the Run Coordinator's eLog:

"17:37:51-
A couple of things I neglected to pass on to Dan. Ron Moore would
like to open the helix to 115% after the store has been in for
16+ hours. This is fine with me, we hope to implement it next
week if all goes well. Also, Keith G. is ready for more reverse
proton studies in pbar. If we have a store with some life left
in it and a stack 150E10+, we should let him have 4-8 hours.
- JPM"

Comments by JJ:
0) Luminosity should be about 40E30 aroud 11pm. We switch from
HighLum to regular table at this point.
1) 16 hours into the store would be 0700-0800 Saturday.
Opening the helix implies taking CDF HV (except for
CLC) to standby/off state while change is made.
2) It should take about 19 hours to stack to 150 mA. I expect
this store to still have "life" at that point (25E30). 3) If lifetime is reasonable for this store, luminosity should
be about 17E30 at 24 hours and 14E30 at 30 hours. - JJ

Fri Feb 27 19:53:14

Timeout on global TDC 
Done for crate B0COT18

- Anadi'sche & Jan :: (run 179463)

Fri Feb 27 20:10:37 Run 179463 Terminated at 2004.02.27 20:10:18 - RunControl

Fri Feb 27 20:10:57 Run 179463 TERMINATE: end due to silicon monkey work - Jan x2080

Fri Feb 27 20:19:20

- Anadi
-- Fri Feb 27 20:19:42 comment by...Anadi --

Hourly Plots: 19:00-20:00

Fri Feb 27 20:21:54 We have PSM alarm twice. The problem is originated in: 1RR21F_2 and 1RR21G_2. No action taken. Everything OK. - Anadi

Fri Feb 27 20:27:50 Run 179466 Activated at 2004.02.27 20:27:28 - RunControl

Fri Feb 27 20:32:07 Run 179466 ACTIVATE: SILICON included - Jan x2080

Fri Feb 27 20:43:56 Run 179463 RUNSTATUS:

Marked Bad, explanation:
SVX  \
ISL   | Silicon not in due to SRC problems being worked on
L00  /

- cdfscico

Fri Feb 27 21:00:53 l3_REFORMAT_ERROR (reformatter message decoder doesn't reveal crate): Error during scanning MINI structure! Last Unit prob. worked on (all counts from 0): SCPU: 3 VRB : 1 LINK: 10 MINI: 0 - Miss Anadi and The Jan :: (run 179466)

Fri Feb 27 21:28:18

- Anadi
-- Fri Feb 27 21:28:41 comment by...Anadi --

Hourly Plots: 20:00-21:00

-- Fri Feb 27 21:54:54 comment by...Anadi --

Proton abort gap losses summed over the beam halo counters are low compared to previous shifts
but unstable in time.

Fri Feb 27 22:06:59

- Anadi
-- Fri Feb 27 22:07:30 comment by...Anadi -- Hourly Plots: 21:00-22:00

Fri Feb 27 22:26:43 Silicon would like to power cycle another crate. We'll stop run when they are ready, so they can power cycle. Estimate 10 to 15 minutes. - Stephan

Fri Feb 27 22:32:20 Run 179466 Terminated at 2004.02.27 22:31:45 - RunControl

Fri Feb 27 22:33:25 Run 179466 TERMINATE: L3_REF_ERROR_HIGH_RATE from PCAL08 AND silicon issues - Jan x2080

Fri Feb 27 22:46:11

since we started 179466 we had terrible problems with silicon: 

approx. 40(!) times Level 1 DONE TIMEOUT!! 
approx. INFINITE(!!!) L3_REFORMATTER_ERORRS mostly from FIB_05 and also PCAL08 

now: run stopped to power cycle FIB00

- Jehlers supported fabulously by Anadi'sche

Fri Feb 27 22:56:39 Run 179467 Activated at 2004.02.27 22:56:24 - RunControl

Fri Feb 27 22:59:15 Run 179467 ACTIVATE: silicons power cycled -> trigger table [2,424,431] - Jan x2080

Fri Feb 27 23:03:11 SVXMON_HALT_RECOVER_RUN_ERROR: MLE) b0xft04:Messenger:10:56:37 PM->Event 693: Bunchcounters in slot 13 (BC=15) and slot 7 (BC=80) disagree (L2B=0) (MLE) b0xft04:Messenger:10:56:37 PM->Runtime Error 1, Event 693: Bunch counter mismatch, mismatch count = 1 b0dap84.fnal.gov:ConsumerErrorRe:10:56:53 PM->Runtime Error 1, Event 1406, RunNum 179467: SvxMon Halt Recover Run: Pipeline out of synch in 130 silicon readout chips . - The sad Aces ;-( :: (run 179467)

Fri Feb 27 23:09:41 switch to low luminosity table for run 179467 - Stephan

Fri Feb 27 23:12:14

L1 Done Timeouts persist: 
Slot: 10:e120 
Slot: 11:e200 
Slot: 10:e420 
Slot: 16:e460

- Jan :: (run 179467)

Fri Feb 27 23:15:26

- Anadi
-- Fri Feb 27 23:16:16 comment by...Anadi -- Hourly Plots: 22:00-23:00. Proton losses are slightly increasing in time but low (below 5kHz)

Fri Feb 27 23:17:30

also reformatter eroors persist: 
FIB_05 and FIB_00

- ace :: (run 179456)

Fri Feb 27 23:28:03

REFORMATTER ERROR in FIB_06 
followed by thousands of CLIST errors 
->data taking stopped ->HRR recovered data taking

- ace :: (run 179467)
-- Fri Feb 27 23:29:19 comment by...Jan -- HAPPEND TWICE IN A ROW

Fri Feb 27 23:38:07 Run 179466 RUNSTATUS:

Marked Bad, explanation:
L3T  \
SVX  /  Level 3 reports about 9% reformatter error

- cdfscico

Fri Feb 27 23:39:13

Some additional FIB_05 errors

- Jan :: (run 179467)

Fri Feb 27 23:49:37 Run 179467 Terminated at 2004.02.27 23:49:12 - RunControl

Fri Feb 27 23:50:42 Run 179467 TERMINATE: end for silicon problem - Jan x2080

Fri Feb 27 23:53:01 Run 179467 RUNSTATUS:

Marked Bad, explanation:
L3T  \
SVX  /  SVX reformatter errors as previous run

- cdfscico

Fri Feb 27 23:55:35

Run Number	Data Type	Physics Table	Begin Time	End Time	Live Time	L1 Accepts	L2 Accepts	L3 Accepts	Live Lumi, nb-1	GR	SC	RC
179463 x2BD07	BEAM	PHYSICS_HIGHLUM_2_02 [1,420,427]	16:49:54	20:10:18	03:11:36	105,562,502	3,712,808	580,234	669.362	1	1	1
179466 x2BD0A	BEAM	PHYSICS_HIGHLUM_2_02 [1,420,427]	20:27:28	22:31:45	01:46:52	40,232,574	1,633,095	254,380	295.244	0	1	1
179467 x2BD0B	BEAM	PHYSICS_2_02 [2,424,431]	22:56:24	23:49:12	00:31:43	32,246,084	562,541	98,886	77.998	0	1	1
Totals				23:55:01	05:30:12	178,041,160	5,908,444	933,500	1042.604

- End of Shift Report

Fri Feb 27 23:56:29 Shift Summary:

 
Store #3261 came in with new record instantaneous luminosity of 7.2*10^31 
   - start without Silicon 
   - include Silicon but Silicon timeout at rate of one every 3 to 4 minutes 
   - switch to low luminosity table when Silicon team power cycled a crate 
   - switch to new low luminosity trigger table PHYSICS_2_03[1,431,435] when 
        silicon team swapped a board 

Plan to continue data taking 
   - check trigger rates and fall back to old trigger table if in doubt 
   - Tevatron may open helix Saturday morning 
   - if store terminates page JJ/Kaori

End of Shift Numbers

CDF Run II

Runs                   179463, 179466, 179467
Delivered Luminosity   1.3303 pb^-1  
Acquired Luminosity    1.0444 pb^-1  
Efficiency             78.5

- Stephan