Eve shift. Wed Feb 25, 2004 (168/169)

2004 CDF E-Log -- Eve shift. Wed Feb 25, 2004
SciCo	DAQ Ace	Monitoring Ace	CO	(Operations Manager)
Robert Harris	Chris Marino	Simon Sabik	Josh Tuttle	JJ Schmidt

Start of Shift Notes:

Inherit store 3256 with luminosity of 52E30 and running with degraded COT (SL 1 HV reduced by
8%) and silicon off.  Expect 

to run this way for about 3 1/2 hours more. After that there 
is no plan currently.

Wed Feb 25 16:12:51 Run 179358 ACTIVE: BUSY Timeout caused by evb error: b0eb15

Busy Condition in VRB/EVB forwarded by crates: CCAL_10 Busy conditions are produced by the VRB and are forwarded by the front end crates to the busy crosspoint. Busy conditions do not represent a problem with the front end crates themselves, but with the EVB system. - Christopher x2080
-- Wed Feb 25 16:13:25 comment by...Christopher -- HRR Recovered

Wed Feb 25 16:31:48 Jim the Cryo tech called and said he will be rebooting S3 and it might cause some IFIX errors. We should ignore them. - Robert Harris

Wed Feb 25 17:21:08

Checked following calibrations from Day shift (DBANA): 
- BSCQIE (Run 179351) = fine 
- CALLED (Run 179337) = fine 
- CALQIE (Run 179347) = fine 
- CLCQIE (Run 179335) = fine

- josh

Wed Feb 25 17:27:33

- Simon
-- Wed Feb 25 17:28:29 comment by...Simon -- B0ABSM spiked to 85000 and came back to normal.

Wed Feb 25 17:36:19

Big lost proton spike.

- Simon

Wed Feb 25 17:37:45 CMX tripped. South West channels 11 and 12. Ramped back up. This happened at the same time as the lost proton spike. - Simon :: (run 179358)

Wed Feb 25 17:40:07 Carla Grosso-Pilcher indicates that a tower in the WHA is a little "warm" due to a problem with a single channel of it's ADMEM. She says that it seems to be cooling down now . . . - Robert Harris

Wed Feb 25 17:49:51 Vadim tells us that he is going to power down the pulser crate, which is not in the data taking, but that we should see it on the Global Alarms. We should ignore the alarm. - Robert Harris
-- Wed Feb 25 17:56:41 comment by...Robert Harris -- He does this, and this causes a trigger inhibit halting the run. I tell him to put it back, he does, and we are running again. The board was a Pulsar board (not pulser) for the L2 system and Vadim Rusu says that they are not read out and should not be in a trigger inhibit. I tell him he should consult with the rest of L2 trigger system personnel.
-- Wed Feb 25 18:15:56 comment by...Burkard -- So I being part of the rest of the L2 trigger system personnel came over to B0. We caught Bill Badgett, who helped us to mask the crate (2RR29C) in the PSM.

Wed Feb 25 18:08:03 Bill Badgett helps out on the Pulsar crate problem. They mask off the crate in IFIX and then they are able to pull the board without creating either an alarm or a trigger inhibit. Vadim Rusu is now going to do work in that crate. - Robert Harris

Wed Feb 25 18:47:21 Run 179358 ACTIVE: Received another scalar monitor message:
Attention!!!. FERML_HIGH_DEADTIME !!!. - Christopher x2080

Wed Feb 25 19:07:50

- Simon
-- Wed Feb 25 19:08:58 comment by...Simon -- Big proton loss and abort gap loss rate at the same time (17:30).

Wed Feb 25 20:29:50 We paused the run so that Takasumi Maruyama could pull out a board from SVT06 crate. - Robert Harris

Wed Feb 25 20:43:46 Run 179358 Terminated at 2004.02.25 20:43:19 - RunControl

Wed Feb 25 20:44:05 Run 179358 TERMINATE: switch to lower lum table
Lum 37.5 - Christopher x2080

Wed Feb 25 20:46:22 restarted Stage0 consumer monitor during run change - josh

Wed Feb 25 20:48:35 Run 179361 Activated at 2004.02.25 20:48:02 - RunControl

Wed Feb 25 20:48:36 Run 179361 ACTIVATE: Physics 2_02[2,424,431] - Christopher x2080

Wed Feb 25 20:49:35

- Simon

Wed Feb 25 21:18:43 JJ gives us a plan for the remainder of the store. The first step is to stop the current running with the COT at 10 p.m. and let Vadim and Burhard perform their isolist fiber tests for 30-60 minutes. I call Vadim and Burhard and they agree to come in at 10 p.m. for the tests. - Robert Harris

Wed Feb 25 21:28:36 Run 179358 RUNSTATUS:

Marked Bad, explanation:
Silicon is off, SL1 of COT is at 8% reduced HV for Aseets test.
COT 
SVX 
ISL 
L00

- cdfscico

Wed Feb 25 21:53:37 Run 179361 ACTIVE: I noticed the VxWorks node monitoring showed b0cot12 as red with code U.
Within a few minutes L2 timeout was received adn b0cot12 failed HALT transition
I rebooted and shepherded crate before proceeding with HRR, adnd the system recovered. - Christopher x2080

Wed Feb 25 22:08:58 Run 179361 Terminated at 2004.02.25 22:08:48 - RunControl

Wed Feb 25 22:09:22 Run 179361 TERMINATE: to bring down COT HV and run some tests - Christopher x2080

Wed Feb 25 22:14:18 Run 179362 Activated at 2004.02.25 22:14:08 - RunControl

Wed Feb 25 22:15:07 Run 179362 ACTIVATE: Physics_2_02[2,424,431] - Christopher x2080

Wed Feb 25 22:24:22 JJ informs us that Aseet is coming in to put the COT back in the state it was in one week ago. - Robert Harris
-- Wed Feb 25 22:33:10 comment by...JJ --

JJ also called XFT pager (Ben answered) to verify that
XFT was set in correct mode with SL2 masked on and
SL4 set to 3 misses.  Ben says XFT is still in this mode.

Wed Feb 25 22:31:36

- Simon

Wed Feb 25 22:44:07 Aseet arrives and starts putting the COT in it's old configuration. JJ says that when he is finished we can bring up the COT HV. Burhard and Vadim are currently doing their isolist fiber test run. - Robert Harris

Wed Feb 25 22:46:47 Run 179361 RUNSTATUS:

Marked Bad, explanation:
Silicon off, COT on with SL1 at 8% reduced HV for Aseets test
COT 
SVX 
ISL 
L00

- cdfscico

Wed Feb 25 22:52:12 Asset is now in his office and we are supposed to call him over when we bring up the COT high voltage. - Robert Harris

Wed Feb 25 23:05:48 Run 179362 Terminated at 2004.02.25 23:05:22 - RunControl

Wed Feb 25 23:06:52 Run 179362 TERMINATE: end to give COT and MUON crates to BB - Christopher x2080

Wed Feb 25 23:12:02 Run 179363 Activated at 2004.02.25 23:11:34 - RunControl

Wed Feb 25 23:12:08 Run 179363 ACTIVATE: Physics 2_02[2,424,431] -- tests ongoing - Christopher x2080

Wed Feb 25 23:26:42 Run 179362 RUNSTATUS:

Marked Bad, explanation:
Silicon off and COT off. Isolist fiber test run.
COT 
SVX 
ISL 
L00

- cdfscico

Wed Feb 25 23:30:09 Run 179363 Terminated at 2004.02.25 23:29:39 - RunControl

Wed Feb 25 23:30:10 Run 179363 TERMINATE: end of L2 pulsar testing - Christopher x2080

Wed Feb 25 23:31:47 Bringing up the COT HV in compromised mode (SL1,2 off, SL3,4,5 reduced). - Robert Harris

Wed Feb 25 23:40:34

- Simon

Wed Feb 25 23:48:17

Have programmed all TDCs in the system to DSP V45, and  
marked the bankVersions as 1 in the hdwdb. 

We are running with PHYSICS_TEST_2_02 [10,429,434], and  
l3Tag=434 means there is a small patch to work around  
a problem in the L3DaqErrorFilter, which was causing  
every event to pass. 

Also, we are running with updated code in the FrontEnd  
to provide more information on the low-level bc mismatch  
errors with this DSP version.

- W.Badgett :: (run 179364)

Wed Feb 25 23:54:34 Run 179364 Activated at 2004.02.25 23:53:45 - RunControl

Wed Feb 25 23:55:36

Run Number	Data Type	Physics Table	Begin Time	End Time	Live Time	L1 Accepts	L2 Accepts	L3 Accepts	Live Lumi, nb-1	GR	SC	RC
179358 x2BC9E	BEAM	PHYSICS_HIGHLUM_2_02 [1,420,427]	14:42:20	20:43:19	05:42:44	146,805,399	5,799,818	909,603	951.325	0	1	1
179361 x2BCA1	BEAM	PHYSICS_2_02 [2,424,431]	20:48:02	22:08:48	01:17:27	78,853,498	943,027	169,545	166.168	0	1	1
179362 x2BCA2	BEAM	PHYSICS_2_02 [2,424,431]	22:14:08	23:05:22	00:43:48	6,390,992	113,256	48,093	88.209	0	0	1
179363 x2BCA3	BEAM	PHYSICS_2_02 [2,424,431]	23:11:34	23:29:39	00:14:45	1,857,933	34,548	15,088	28.546		0	1
Totals				23:55:02	07:58:45	233,907,822	6,890,649	1,142,329	1234.249

- End of Shift Report

Wed Feb 25 23:55:37 Run 179364 ACTIVATE: Physics_Test_2_02[10,429,434] -- Bill Badgett's test - Christopher x2080

Thu Feb 26 00:10:16 Shift Summary:

- Took data with Silicon off and COT on with SL1 at 8%
reduced HV 

  for Assets test.  Brought down the COT HV afterwards. 
- Vadim and Burhard did Isolist Fiber Test Runs with SVX and COT  
  off. Bill Badgett simultaneously did tests with a couple of  
  crates. 
- Asset put the COT back in its stable configuration: SL1-2 off,  
  SL3,4,5 reduced. We bring the COT HV back up. 
- Bill Badgett started a test run for the TDC DSP microcode at  
  end of shift. 
  JJ has allotted him 1-2 hours for this run.

- Robert Harris
-- Thu Feb 26 00:18:27 comment by...Guram --

Luminosity summary shows following:

Begin shift or beam at 16:00:10
End shift or beam at 23:58:24
Delivered luminosity: 36.74
Acquired luminosity: 0.04

         Totals
 Date:                2004.02.25
 Shift:               eve
 Delivered luminosity:36.7 nb-1
 Acquired luminosity: 0.0 nb-1
 Efficiency:          0.1

I guess something is wrong with luminosity summary, since "End of
Shift Report" above shows 1.2 pb^-1 of total live luminosity, and 
from the acnet plot we know that about 1.4 pb^-1 were delivered.

-- Thu Feb 26 08:21:07 comment by...j. patrick -- The crate b0clc01 was rebooted at 23:38. This crate is the source of the integrated luminosity, rebooting the crate will zero the value.

Thu Feb 26 00:18:58

Got TOF heart bit. SMACS crashed. Had to kill and restart it. 
Things are back to normal.

�(entry outside this shift's time range ) - Andrew