Owl shift. Mon Mar 1, 2004 (181/182)

2004 CDF E-Log -- Owl shift. Mon Mar 1, 2004
SciCo	DAQ Ace	Monitoring Ace	CO	(Operations Manager)
S.Miscetti	A.Ivanov	S.Sabik	B.Mohr	JJ.Schmidt

Start of Shift Notes:

Store # 3263 in progress. 
Initial Lum: 2.7E31

Mon Mar 1 00:28:08 TOF heart beat. The PC had to be restarted. - Simon :: (run 179505)

Mon Mar 1 00:48:35

Event Builder Error: 

b0l3pcom1.fnal.gov:main:12:44:39 AM->Host b0eb12.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 7260587 from VRB in slot 14: 
0x00000000 0x00002580 0x000d00cd 0x03e403e4 0x03e403e4 0x03e403e4 0x04cc0000 0x0 
000091c 
CT: 2004.03.01 00:44:46 
44'46" b0l3pcom1.fnal.gov:main:12:44:39 AM->Host b0eb12.fnal.gov, task tRec_0 
SCPU-P1-E-BadVrbByteCount: Event 7260588 total byte count is lt 32, gt 65528, or 
 not divisible by 8 for VRB in slot 10. 
Count = 1028128. --> 
 Additional Information: 

 Attention!!!. SCPU_BAD_VRB_BYTE_COUNT Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb12. 
 Datataking is paused: events are not being processed. 

 AUTO HRR will be issued. 
 N.B. an automatic HRR for HEVB errors will be re-issued (if still needed) 
 in 30 secs. [MLE] 
(MLE) b0l3pcom1.fnal.gov:main:12:44:39 AM->Host b0eb12.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 7260588 from VRB in slot 10: 
0x0003f03f 0x020fb020 0x00680100 0x00c00800 0x00000000 0x02800000 0x00e00400 0x0 
0380100 

Auto-HRR recovered.

- Andrew :: (run 179505)

Mon Mar 1 01:20:40

L2 decision timeout. 
Lots of CLIST errors. 
Auto-HRR recovered.

- Andrew :: (run 179505)

Mon Mar 1 01:20:48 Silicon trip. Channels B3 W8 L0 and B3 W8 L1. For unknown reasons, B3 W8 L1 became black instead of red during trip. The wedge was ramped back up and the channels were unmarked. - Simon :: (run 179505)
-- Mon Mar 1 05:32:48 comment by...Simon -- B3W8L0 also tripped at 18:00.

Mon Mar 1 01:35:31

- Simon

Mon Mar 1 01:56:02

TOMCAT turned red on PROCMON with PROCMON error reported and became 
back green before I took any action

- Andrew :: (run 179505)

Mon Mar 1 02:28:50

Info: Bottom left plot of L1 Sum Et shows some small deviations from unity in the ratio of simulation to hardware. Bottom right plot is constant at unity, so hardware is OK.

- B. Mohr

Mon Mar 1 04:06:56

- Simon
-- Mon Mar 1 04:07:39 comment by...Simon -- A few periodical spikes in the proton abort gaps. But they are small.

Mon Mar 1 05:19:43

Event Builder Error: 

b0l3pcom1.fnal.gov:main:5:17:01 AM->Host b0eb19.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 16 for event 11014073. 
VRB_BUS_ERROR: Alignment or VME bus error. 
Transferred 0 bytes before the error. 
(MLE) b0l3pcom1.fnal.gov:main:5:17:01 AM->Host b0eb19.fnal.gov, task tRec_0 
SCPU-P1-E-EventIdMismatch: VRB ID for event 11014074 is 0 instead of 10 (slot 16 
). 
(MLE) b0l3pcom1.fnal.gov:main:5:17:01 AM->Host b0eb19.fnal.gov, task tRec_0 
SCPU-P1-E-EventIdMismatch: VRB ID for event 11014075 is 0 instead of 11 (slot 16 
). 
(MLE) b0l3pcom1.fnal.gov:main:5:17:01 AM->Host b0eb19.fnal.gov, task tRec_0 
SCPU-P1-E-EventIdMismatch: VRB ID for event 11014076 is 0 instead of 12 (slot 16 
). 
(MLE) b0l3pcom1.fnal.gov:main:5:17:01 AM->Host b0eb19.fnal.gov, task tRec_0 
SCPU-P1-E-EventIdMismatch: VRB ID for event 11014077 is 0 instead of 13 (slot 16 
). 
(MLE) b0tsi02:Messenger:5:17:06 AM->BUSY timeout 

Auto-HRR recovered.

- Andrew :: (run 179505)

Mon Mar 1 05:22:15

Level 2 Decision Timeout:  

L1Mon: saw 210 L1 DMA transfers, expect 1 (buffer number 1) 

Auto-HRR recovered.

- Andrew :: (run 179505)

Mon Mar 1 05:35:31

- Simon

Mon Mar 1 06:13:07 PSM red alarm. Recovered by itself. - Simon :: (run 179505)

Mon Mar 1 06:19:20

Level 2 Decision Timeout: 

b0l2de00:SpyAlpha:6:18:01 AM-> 
Error: Startload is still low. ModDone bits 0x4007fdff for event 721628496 

(MLE) b0l2de00:SpyAlpha:6:18:01 AM-> 
TIMEOUT: Mod done not set for slot(s) 12 (SVTList) 
 Raw: (0x0007fdff, cnt 14) 
XTRP: (BC = 21, Buffer no = 1) 

(MLE) b0l2de00:SpyAlpha:6:18:01 AM-> 
Error: Startload is still low. ModDone bits 0x4007fdff for event 721628497 

(MLE) b0l2de00:SpyAlpha:6:18:02 AM-> 
TIMEOUT: Mod done not set for slot(s) 12 (SVTList) 
 Raw: (0x0007fdff, cnt 15) 
XTRP: (BC = 30, Buffer no = 0) 


Auto-HRR recovered.

- Andrew :: (run 179505)

Mon Mar 1 06:54:43

Event Builder Error: 


b0l3pcom1.fnal.gov:main:6:53:07 AM->Host b0eb12.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 12225006 from VRB in slot 14 
: 
0x00000000 0x00002568 0x003d00bd 0x03e403e4 0x03e403e4 0x03e403e4 0x04d40000 0x0 
00008fc 
CT: 2004.03.01 06:53:17 
53'17" b0l3pcom1.fnal.gov:main:6:53:07 AM->Host b0eb12.fnal.gov, task tRec_0 
SCPU-P1-E-BadChannelCounts: In event 12225007 the sum of channel counts is incon 
sistent with the total count, VRB slot 10. 

Auto-HRR recovered.

- Andrew :: (run 179505)

Mon Mar 1 07:03:39

- Simon

Mon Mar 1 07:15:26

Event Builder SCPU_CANT_RESET_VRB Error !!! 
 THE BAD VRB CRATE is: b0eb16 
 THE BAD VRB MODULE is in SLOT: 10 

Halted the Run, Resetted VRB module. 
Returned to data taking.

- Andrew :: (run 179505)

Mon Mar 1 07:26:51

Proton abort gap peak

- Simon

Mon Mar 1 07:55:37

Run Number	Data Type	Physics Table	Begin Time	End Time	Live Time	L1 Accepts	L2 Accepts	L3 Accepts	Live Lumi, nb-1	GR	SC	RC
179505 x2BD31	BEAM	PHYSICS_2_03 [1,431,435]	16:48:33		14:06:06	794,204,992	12,897,570	2,444,664	1388.648			1
Totals				07:55:02	14:06:06	794,204,992	12,897,570	2,444,664	1388.648

- End of Shift Report

Mon Mar 1 07:59:41 Shift Summary:

Store 3263  in progress.  
Luminosity degraded from 2.7E31 to 2.0E31. 

- Quiet data taking, running with new low lum table PHYSICS_2_03[1,431,435] 
- 1 silicon trip at the beginning of the shift, a large peak on B0PBSM counter at 7:22 am.

End of Shift Numbers

CDF Run II

Runs                   179505
Delivered Luminosity   665 nb-1  
Acquired Luminosity    625 nb-1  
Efficiency             94

- S.Miscetti