Day shift. Wed Mar 3, 2004 (188/191)

2004 CDF E-Log -- Day shift. Wed Mar 3, 2004
SciCo	DAQ Ace	Monitoring Ace	CO	(Operations Manager)
Rainer W.	A.Canepa	T.Schwarz	F.Semeria	M.Convery

Start of Shift Notes:

Store # 3271 in progress. 
PLAN: Test of jet_St3_decoupled

Wed Mar 3 08:32:15

- (hourlies) Tom

Wed Mar 3 09:25:31 taking over from mary and stefano M. who kindly allowed me an extra 1.5 hours of sleep. thanks a lot. - rainer :: (run 179578)

Wed Mar 3 09:35:12

- (hourlies) Tom

Wed Mar 3 09:42:24 Run 179578 Terminated at 2004.03.03 09:41:53 - RunControl
-- Wed Mar 3 10:41:43 comment by...rainer -- current L1 trigger rates after prescale (Fred Prescaled)

 
L1_JET3_PS1[1]   / 5       12188 Hz 
L1_MB_CLCPS50K[1]/50000       15 Hz 
L1_MB_XING_PS1M[2]/1000003   1.8 Hz 
L1_JET3_PS1M[1]   /1000003   0.2 Hz 

Total L1 12256 Hz 
      L2    94 Hz

Wed Mar 3 09:43:02 Run 179578 TERMINATE: end run to switch to TEST_METIS_DECOUPLED[1,434,403] - Anadi x2080
-- Wed Mar 3 09:53:07 comment by...Anadi -- I am sorry: TEST_MET15_DECOUPLED[1,434,403]

Wed Mar 3 09:49:51 Run 179579 Activated at 2004.03.03 09:49:39 - RunControl

Wed Mar 3 09:50:48 Run 179579 ACTIVATE: start test run with TEST_MET15_DECOUPLED[1,434,403]. Si in NOT in. COT is NOT in. - Anadi x2080

Wed Mar 3 09:51:40

cdfsoftb0 current status: 

ObjectMon is running special tcl. 
SVXMon is running cosmic tcl since there is no streamA. 
All other Monitors should be good.

- ryo

Wed Mar 3 09:58:07 Run 179579 Terminated at 2004.03.03 09:57:55 - RunControl

Wed Mar 3 09:58:38 Run 179579 TERMINATE: end test run to integrate the Silicon - Anadi x2080

Wed Mar 3 10:03:33

pinkie

- Tom
-- Wed Mar 3 10:03:47 comment by...Tom --

A pinkie.  Unmarked.  Current low I believe.  

Ranges :
  Nominal  : 1110.0 (Not used) 
  Variance : 5.0
  Good     : 1105.0 <-> 1185.0
  Caution  : 1100.0 <-> 1190.0

Latest Values :
  Current  : 1085.0
  Voltage  : 7.510000228881836

Status : MARKED, BAD
  At 02:40:07
  Current at mark : 1085.0
  Voltage at mark : 7.510000228881836
PS_ID=2004

Wed Mar 3 10:03:37 Run 179580 Activated at 2004.03.03 10:03:31 - RunControl

Wed Mar 3 10:04:41 Run 179580 ACTIVATE: start test run with TEST_MET15_DECOUPLED[1,434,403]. Si is IN. COT is NOT in. - Anadi x2080

Wed Mar 3 10:20:08

Error without warning window. From the errorlog:. 
Error: Startload is still low. ModDone bits 0x4007f5bf for event 94733 
TIMEOUT: Mod done not set for slot(s) 14 (Clist) 12 (SVTList) 9 (XtrpList) Raw: (0x0007f5bf, cnt 1).


No action taken.

- Anadi :: (run 179580)

Wed Mar 3 10:24:31

Drastic Drop In Current For A silicon cell

- Tom
-- Wed Mar 3 10:25:21 comment by...Tom --

Physical location :
  SVX AVDD  Barrel 1  Wedge 4  Layer 2

Ranges :
  Nominal  : 820.0 (Not used) 
  Variance : 0.0
  Good     : 790.0 <-> 850.0
  Caution  : 790.0 <-> 850.0

Latest Values :
  Current  : 825.0
  Voltage  : 7.909999847412109

Status : OK, GOOD.
PS_ID=511

-- Wed Mar 3 10:43:35 comment by...rainer -- a few chips on the z-side dropped readout. we power cycled the ladder, to no avail.
-- Wed Mar 3 10:43:41 comment by...Tom --

whoops.  wrong info.  here's the right stuff.  my bad.

Physical location :
  SVX AVDD  Barrel 1  Wedge 4  Layer 4

Ranges :
  Nominal  : 1110.0 (Not used) 
  Variance : 2.74
  Good     : 1085.0 <-> 1120.0
  Caution  : 1082.26 <-> 1122.74

Latest Values :
  Current  : 930.0
  Voltage  : 7.909999847412109

Status : MARKED, BAD
  At 10:41:19
  Current at mark : 930.0
  Voltage at mark : 7.920000076293945
PS_ID=505

-- Wed Mar 3 11:10:30 comment by...Anadi -- See strip mean charge and strip occupancy plots at: plot

Wed Mar 3 10:25:22 test entry - R. Vidal

Wed Mar 3 10:40:38 CMX Trip. Turned channels back on. All is fine. - Tom
-- Wed Mar 3 15:22:57 comment by...ps -- 3 channels trip due to loss spike at 1035 seen below

Wed Mar 3 10:44:05 test entry2 - R. Vidal

Wed Mar 3 10:44:46 elog test to see whether we need more cookies. - rainer

Wed Mar 3 10:45:35

- Anadi

Wed Mar 3 10:45:39

- (hourlies) Tom

Wed Mar 3 10:52:59 Run 179580 Terminated at 2004.03.03 10:52:49 - RunControl

Wed Mar 3 10:53:32 Run 179580 TERMINATE: stop the test run to swith trigger table - Anadi x2080

Wed Mar 3 10:57:36 test entry - R. Vidal

Wed Mar 3 11:04:26

Two Huge proton spikes in the last 30 minutes. Note LOSTP scale max is 150,000. Abort Gap is plotted too.

- Tom
-- Wed Mar 3 15:23:33 comment by...ps -- the first one tripped cmx

Wed Mar 3 11:07:25 Run 179581 TERMINATE: we abort and config again to make sure the Si is IN after ahving worked on that - Anadi x2080

Wed Mar 3 11:09:26 Run 179582 Activated at 2004.03.03 11:09:12 - RunControl

Wed Mar 3 11:10:30 Run 179582 ACTIVATE: resuming JETS_ST3_DECOUPLED[1,433,403]. Si is IN. COT is NOT in. - Anadi x2080
-- Wed Mar 3 11:12:16 comment by...rainer -- silicon handbrake limit was moved from 22kHz to 35kHz for a step-by-step exercise for higher L1 rates (pre-test for UDPS)

Wed Mar 3 11:12:30 test entry - R. Vidal

Wed Mar 3 11:14:23 While configuring we had to shepherd crate pcal07. While configuring b0fib02 became red due to Chan 4 B1W4. Rainer worked on that. B1W4L4 is NOT in the run configuration at the moment. - Anadi :: (run 179582)

Wed Mar 3 11:15:17 The preview function for the CDF shift e-log has been modified. It now shows how the entry or comment will look in the context of the page. If there are any problems or suggestions please send me e-mail... - R. Vidal

Wed Mar 3 11:17:26 current L1 trigger rates after prescale (Fred Prescaled)

  
L1_JET3_PS1[1]   / 5       11209 Hz  
L1_MB_CLCPS50K[1]/50000       14 Hz  
L1_MB_XING_PS1M[2]/1000003     0 Hz  
L1_JET3_PS1M[1]   /1000003   0.2 Hz  

Total L1 11255 Hz  
      L2    92 Hz

- rainer :: (run 179582)

Wed Mar 3 11:17:33 Before activating the run, we change the prescaling for L1_JET3_PS1[1] to 5 - Anadi

Wed Mar 3 11:36:11

- (hourlies) Tom

Wed Mar 3 12:02:27 Kevin Lannon reports that SL2 is no longer masked on for XFT (as requested by TDWG) in preparation for the new COT configuration after gas flow reversal (SL1,2,3 on full, SL4 reduced gain). - convery
-- Wed Mar 3 12:10:40 comment by...convery -- SL5 also at reduced gain

Wed Mar 3 12:03:21 Automatic HRR issued to fix a L2 TO associated to: L1Mon: saw 210 L1 DMA transfers, expect 1 (buffer number 1) L1Mon: Dumping data for 1 word. - Anadi
-- Wed Mar 3 12:17:09 comment by...Anadi -- Same error at 12:16

Wed Mar 3 12:30:40 Run 179582 Terminated at 2004.03.03 12:30:32 - RunControl

Wed Mar 3 12:31:20 Run 179582 TERMINATE: end the run for fiber work - Anadi x2080
-- Wed Mar 3 12:36:21 comment by...Rainer -- ratelimit back to 22kHz. # of kicks moved from 10 to 8 for SRC e string resonance protection.

Wed Mar 3 12:39:30

- (hourlies) Tom

Wed Mar 3 12:40:29 Run 179583 Activated at 2004.03.03 12:40:11 - RunControl

Wed Mar 3 12:41:02 Run 179583 ACTIVATE: start new run JET_ST3_DECOUPLED[1,433,403]. Si is IN, COT is IN. - Anadi x2080

Wed Mar 3 12:43:35 Run 179578 RUNSTATUS:

Marked Bad, explanation:
COT crates out of readout, COT OFF due to gas reversal

- cdfscico

Wed Mar 3 12:44:06 Run 179583 Terminated at 2004.03.03 12:44:01 - RunControl

Wed Mar 3 12:44:24 Run 179579 RUNSTATUS:

Marked Bad, explanation:
COT crates out of readout, COT OFF due to gas reversal

- cdfscico

Wed Mar 3 12:44:25 Run 179583 TERMINATE: end to run to remove the COT crates - Anadi x2080
-- Wed Mar 3 12:48:50 comment by...rainer -- RXPT errors and DONE timeouts. maybe. but didn't sound good.

Wed Mar 3 12:45:46 Run 179580 RUNSTATUS:

Marked Bad, explanation:
COT  crates out of readout, COT OFF due to gas reversal

- cdfscico

Wed Mar 3 12:49:57 Run 179584 Activated at 2004.03.03 12:49:16 - RunControl

Wed Mar 3 12:50:19 Run 179584 ACTIVATE: start new run JETS_ST3_DECOUPLED[1,433,403]. Si is IN, COT is NOT in. - Anadi x2080

Wed Mar 3 12:51:48 Run 179584 Terminated at 2004.03.03 12:51:12 - RunControl

Wed Mar 3 12:52:00 Run 179584 TERMINATE: end run to include COT crates again - Anadi x2080

Wed Mar 3 12:52:24 Trigger Plan

Kevin and I consulted on the trigger plan. The data when the COT HV returns will be yet different from any other data we have taken. It will have SL1,2,3 on, SL4,5 at reduced gain. This will be quite different in L3 from the low-gain config of last week. Therefore, we would like to run the system with the best possible configuration.

Restore SL2 for XFT.

SL4 remains in 3-miss mode to compensate for the reduced gain.

Trigger table be PHYSICS_2_02_v2. With SL1 and 3 at full voltage, Level 3 performance should be close to normal.

- jdl

Wed Mar 3 12:56:39 The Liunatics and I have installed new fibers to run from the isopick cards to the splitter box we installed yesterday and connected them to the isopicks. We have connected the Isolist card to the output of the splitters. If the Isolist canvas in trigmon shows no significant errors, we will leave this in. C-list is still using the old fibers to the new splitter. - jdl :: (run 173583)

Wed Mar 3 12:58:31 Run 179585 Activated at 2004.03.03 12:57:58 - RunControl

Wed Mar 3 12:58:39 Run 179585 ACTIVATE: start new run with JET_ST3_DECOUPLED[1,433,403] with the COT in - Anadi x2080

Wed Mar 3 13:00:47

Slow Control Icicle has a purple heartbeat in GLOBAL ALARMS.  checked terminal and seems to be
running fine.  I'll hold off on this hoping it clears up on its' own.

- Tom

Wed Mar 3 13:02:56 Run 179585 Terminated at 2004.03.03 13:02:42 - RunControl

Wed Mar 3 13:03:13 Run 179585 TERMINATE: end run for COT problems - Anadi x2080
-- Wed Mar 3 13:05:22 comment by...Anadi -- Frank will be back in 30 min to work on the crates that will NOT be included in the next run: cot07 cot08 cot11 cot18

Wed Mar 3 13:09:14 Run 179586 Activated at 2004.03.03 13:08:52 - RunControl

Wed Mar 3 13:17:52 Starting the run we have the following errors: 1) Error: Startload is still low. ModDone bits 0x4006ffff for event 1 2) TIMEOUT: Mod done not set for slot(s) 19 (IsoList) Raw: (0x0006ffff, cnt 2) XTRP: (BC = 109, Buffer no = 0) SVT: (BC = 109, Buffer no = 0) Several automatic HRR issued J. Lewis is currently working on that. - Anadi

Wed Mar 3 13:18:16 COT H20 alarm set off by experts. - Tom

Wed Mar 3 13:20:19 Run 179586 Terminated at 2004.03.03 13:20:13 - RunControl

Wed Mar 3 13:21:19 Run 179586 TERMINATE: L2 Decisions Timeouts from Isolist. Put back fibers. - Anadi x2080

Wed Mar 3 13:21:21 Run 179586 TERMINATE: L2 Decision Timeouts. New fibers AFU. - Anadi x2080

Wed Mar 3 13:24:27

From the errorlog: 

MLE) b0l2de00:SpyAlpha:1:15:36 PM-> 
TIMEOUT: Mod done not set for slot(s) 19 (IsoList)  
 Raw: (0x0006ffff, cnt 18) 
XTRP: (BC = 130, Buffer no = 1) 
SVT: (BC = 130, Buffer no = 1) 

CT: 2004.03.03 13:15:40 
15'40" L2 Decision Timeout.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-28:1:15:40 PM->Requested Halt-Recover-Run issued [errmon] 
(RC)   13:15:43 Halt -> HALTED 
(RC)   13:15:49 Recover -> RECOVERED 
(MLE) b0l2de00:SpyAlpha:1:15:51 PM-> 
Error: Startload is still low. ModDone bits 0x4006ffff for event 19

- Anadi

Wed Mar 3 13:30:27 Run 179587 Activated at 2004.03.03 13:30:21 - RunControl

Wed Mar 3 13:32:25 Run 179587 ACTIVATE: start run JET_ST3_DECOUPLED[1,433,4034]after all fibers are back - Anadi x2080

------------- 1 --x5 -- current L1 trigger rates after prescale (Fred Prescaled)

 
L1_JET3_PS1[1]   / 5        9992 Hz 
L1_MB_CLCPS50K[1]/50000      13.6 Hz 
L1_MB_XING_PS1M[2]/1000003   1.8 Hz 
L1_JET3_PS1M[1]   /1000003   0.0 Hz 

Total L1 10231 Hz 
      L2    91 Hz

------------- 41506 -- x4 --------------- current L1 trigger rates after prescale (Fred Prescaled)

 
L1_JET3_PS1[1]   / 4        12442 Hz 
L1_MB_CLCPS50K[1]/50000      13.6 Hz 
L1_MB_XING_PS1M[2]/1000003   1.7 Hz 
L1_JET3_PS1M[1]   /1000003   0.0 Hz 

Total L1 12436 Hz 
      L2    91 Hz

------------- 117778 -- x3 --------------- current L1 trigger rates after prescale (Fred Prescaled)

 
L1_JET3_PS1[1]   / 3        16388 Hz 
L1_MB_CLCPS50K[1]/50000       13.8 Hz 
L1_MB_XING_PS1M[2]/1000003   1.8 Hz 
L1_JET3_PS1M[1]   /1000003   0.0 Hz 

Total L1 16443 Hz 
      L2    95 Hz

------------- 172105 -- x2 --------------- current L1 trigger rates after prescale (Fred Prescaled)

 
L1_JET3_PS1[1]   / 2        24420 Hz 
L1_MB_CLCPS50K[1]/50000      13.4 Hz 
L1_MB_XING_PS1M[2]/1000003   1.8 Hz 
L1_JET3_PS1M[1]   /1000003   0.0 Hz 

Total L1 24301 Hz 
      L2    96 Hz

------------- 222331 -- x3 --------------- test ended.
-- Wed Mar 3 18:22:27 comment by...rainer -- evaluation see silicon elog

Wed Mar 3 13:34:52

- (hourlies) Tom

Wed Mar 3 13:46:01

RXPT b0svx06 error and FERML_SRC_FATALITY error. From the error log:  

2'28" 1 crate/s: b0svx06(16),  in error.[RXPT] 
(MLE) b0dap73.fnal.gov:Thread-28:1:42:28 PM->FrontEnd Crate Error Condition from: 
VRB_ISL_06 

(MLE) b0dap73.fnal.gov:Thread-28:1:42:28 PM->FrontEnd Error: VRB_ISL_06  
CT: 2004.03.03 13:42:29 
42'29" 1 crate/s: b0svx06(16),  in error.[RXPT]b0svx06:Messenger:1:42:29 PM->SRC Fatal Error:Sl 5
Too Many L1A 2 L1A to Buff  

 -->   
 Additional Information:  

 Attention !!!. FERML_SRC_FATALITY ERROR !!!  

 SRC Fatal Error from b0svx06: Sl 5 Too Many L1A 2 L1A to Buff  

 - HALT the run                    
 - record any relevant information in e-log      
 - RECOVER - RUN      
[MLE] 
(RC)   13:42:31 Halt -> HALTED 
CT: 2004.03.03 13:42:36 
42'36" 1 crate/s: b0svx06(16),  in error.[RXPT] 
(RC)   13:42:37 Recover -> RECOVERED 
(RC)   13:42:39 Run -> ACTIVE

- Anadi :: (run 179587)

Wed Mar 3 14:20:01 Run 179587 Terminated at 2004.03.03 14:19:34 - RunControl

Wed Mar 3 14:20:02 Run 179587 TERMINATE: end run to include COT crates - Anadi x2080

Wed Mar 3 14:25:19 Run 179589 Activated at 2004.03.03 14:24:45 - RunControl

Wed Mar 3 14:31:32 Run 179589 Terminated at 2004.03.03 14:31:18 - RunControl

Wed Mar 3 14:31:48 Run 179589 TERMINATE: end to run to check the COT crate behavior - Anadi x2080

Wed Mar 3 14:34:54 Run 179592 Activated at 2004.03.03 14:33:46 - RunControl

Wed Mar 3 14:34:55 Run 179592 ACTIVATE: start run JET_ST3_DECOUPLED[1,433,403], COT in. - Anadi x2080

Wed Mar 3 14:38:42 Frank resets the COT crates: cot07, cot08, cot11, cot18 logging into each single crate. We then shepherd them and everything is working fine now. - Anadi :: (run 179591)
-- Wed Mar 3 14:50:45 comment by...Anadi --

The run number was acually 179589. From the error log:
CT: 2004.03.03 14:25:37
25'37" 3 crate/s: b0cot08(56), b0cot11(59), b0cot18(66),  Done TO.[RXPT]
(MLE) b0tsi02:Messenger:2:25:36 PM->DONE timeout
(MLE) b0dap73.fnal.gov:Thread-28:2:25:37 PM->Done Timeout: COT_08 COT_11 COT_18

Wed Mar 3 14:43:40

- (hourlies) Tom

Wed Mar 3 14:59:50

This plot is for the people who were testing splitting for ISOlist.

- Kaori
-- Wed Mar 3 15:15:20 comment by...Burkard -- Thanks Kaori, plots look fine, unfortunately we are not splitting right now :(

Wed Mar 3 15:50:46

- (hourlies) Tom

Wed Mar 3 15:56:05

Run Number	Data Type	Physics Table	Begin Time	End Time	Live Time	L1 Accepts	L2 Accepts	L3 Accepts	Live Lumi, nb-1	GR	SC	RC
179578 x2BD7A	BEAM	JET_ST3_DECOUPLED [1,433,403]	02:09:52	09:41:53	07:28:38	422,462,300	2,611,396	2,611,393	876.233	0	1	1
179579 x2BD7B	BEAM	TEST_MET15_DECOUPLED [1,434,403]	09:49:39	09:57:55	00:07:01	75,050	39,577	39,577	10.410	0	1	1
179580 x2BD7C	BEAM	TEST_MET15_DECOUPLED [1,434,403]	10:03:31	10:52:49	00:42:45	435,037	236,440	236,439	62.525	0	1	1
179581 x2BD7D	BEAM	JET_ST3_DECOUPLED [1,433,403]		11:05:17	00:00:00	0	0				1	0
179582 x2BD7E	BEAM	JET_ST3_DECOUPLED [1,433,403]	11:09:12	12:30:32	01:20:42	52,721,840	449,082	449,082	109.856		1	1
179583 x2BD7F	BEAM	JET_ST3_DECOUPLED [1,433,403]	12:40:11	12:44:01	00:00:00	9	3		0.000		1	0
179584 x2BD80	BEAM	JET_ST3_DECOUPLED [1,433,403]	12:49:16	12:51:12	00:00:00	0	0		0.000		1	0
179587 x2BD83	BEAM	JET_ST3_DECOUPLED [1,433,403]	13:30:21	14:19:34	00:44:29	41,242,171	256,021	256,021	54.759		1	1
179589 x2BD85	BEAM	JET_ST3_DECOUPLED [1,433,403]	14:24:45	14:31:18	00:00:53	515,259	4,882	4,880	0.859		1	0
179592 x2BD88	BEAM	JET_ST3_DECOUPLED [1,433,403]	14:33:46		01:19:12	44,189,436	434,680	435,641	92.995			1
Totals				15:55:02	11:43:43	561,641,102	4,032,081	4,033,033	1207.637

- End of Shift Report

Wed Mar 3 15:58:55 Run 179582 RUNSTATUS:

Marked Bad, explanation:
COT is OFF, not all COT crates in.

- cdfscico

Wed Mar 3 16:00:56 Run 179587 RUNSTATUS:

Marked Bad, explanation:
COT off, not all COT crates in.

- cdfscico

Wed Mar 3 16:01:04 Run 179592 Terminated at 2004.03.03 16:00:58 - RunControl

Wed Mar 3 16:01:57 Run 179592 TERMINATE: Switch to MET test table - Christopher x2080

Wed Mar 3 16:03:07 Run 179592 RUNSTATUS:

Marked Bad, explanation:
COT off (COT crates are in)

- cdfscico

Wed Mar 3 16:04:01 Run 179599 Activated at 2004.03.03 16:03:54 - RunControl

Wed Mar 3 16:04:11 Run 179599 Terminated at 2004.03.03 16:04:03 - RunControl

Wed Mar 3 16:04:36 Shift Summary:

 



Started shift with store #3271 , now luminosity 15.6e30 
Stack 59, stacking around 9.5m/h. Running ST3 MB trigger setup. 

History: 
- L2 people tried to finish up fibre splitting  but was unsuccessful 
- COT crates were re-integrated 
- Silicon took L1 rate scan 
- performed TEST_MET15 220K 
- TOF trigger inhibit still needs masking (somebody working on it?) 
Plan: 
- wait for COT re-integration when gas is ok.

End of Shift Numbers

CDF Run II

Runs                   179578,79,80,82,87,92
Delivered Luminosity   656  
Acquired Luminosity    499  
Efficiency             76.1

- Rainer

Wed Mar 3 16:06:33 Run 179599 TERMINATE: failed to load MET table -- no data taken - Christopher x2080

Wed Mar 3 16:07:34 Run 179600 Activated at 2004.03.03 16:07:17 - RunControl

Wed Mar 3 16:09:15 Run 179600 ACTIVATE: TEST_MET15_DECOUPLED[1,434,403] - Christopher x2080