Start of Shift Notes:  

Taking data.  B0Lum 35E30, stack 0.

halted run earlier in this shift due to p loss spike. cmx, cmp tripped. see previous shift elog
for plot of spike. 

got error: 

Host b0eb12.fnal.gov, task tRec_0 
SCPU-P1-E-TracerEventId: Event 2573753, crate 0, channel 3 has either bad Tracer ID or bad markers
around Tracer word 


Attention!!!. SCPU_TRACER_EVENT_ID Error !!! 
 Hardware EVB has detected a problem with data quality in 
 SCPU b0eb12 ( FER crate : NOT_AVAILABLE). 


second time this has happened. in both cases hrr worked. 

same error once more

again:

0l3pcom1.fnal.gov:main:3:14:07 PM->Host b0eb18.fnal.gov, task tRec_0
SCPU-P1-E-VrbHeader: Dump of header words for event 6591439 from VRB in slot 20:

There was a PSM alarm on 1RR12G_2 on tuesday evening. See original entry. According to VOLTMAN, channel 2 (-6V) flucuated out of tolerance and hence the alarm. Dervin told me that the ace on shift checked the front panel of the power supply which read a proper value. My suspicion is that this read out was for channel 1 (+6) which was fine at the time. However, I'm not ruling out a communication failure between the power supply and VOLTMAN. I'm leaving the power supply masked, but am emailing Dervin and Roberto.

twice got the error: 

TIMEOUT: Mod done not set for slot(s) 14 (Clist)  Raw: (0x0007f7ff, cnt 4)XTRP: (BC = 130, Buffer no
= 1)SVT: (BC = 130, Buffer no = 1) 


hrr worked as expected.

again but now for slot 12

slot 12 again

got error: 

Host b0eb14.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 3908032 from VRB in slot 10: 
0x00000000 0x00002a40 0x002f00ef 0x08ec02f0 0x02f00580 0x06e80548 0x037c0000 0x03100310 

hrr worked. 

over the past few hours 3 L3 filter states have died: nodes 097, 123, 225. 

 

2 more: 050, 078

Abort gap losses are gradually rising

12:13:13 PM->Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 

hrr worked.

Got CMP trip in South Wall, when turning all voltages back on.
Recovered.

Abort gap losses plot during the collimator movement.

Got a trigger inhibit and further PHA and PEM trips. 
PHA and PEM iFix pages had a "diagnostic" button red, but 
by clicking on it, nothing happened.  
It turns out that software for Plug HV Monitoring running 
on the computer on the 3rd floor crashed.  
Willis restarted it and fixed the problem. 

MLE) b0l3pcom1.fnal.gov:main:3:22:25 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 6697334. 
The VRB had no event. 
first got si timeout: 

(MLE) b0l3pcom1.fnal.gov:main:3:22:26 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbReadFailed: Error reading VRB in slot 10 for event 6697335. 
The VRB had no event. 
(MLE) b0tsi02:Messenger:3:22:30 PM->DONE timeout 

then pcal05 done timeout: 

(MLE) b0dap73.fnal.gov:Thread-110125:3:22:31 PM->Requested Halt-Recover-Run issued [errmon] 
(MLE) b0dap73.fnal.gov:Thread-110125:3:22:31 PM->Done Timeout: PCAL_05 

successfully shepherded pcal05. 

notice that b0clc00, b0fcal00, and now (after shepherding) b0pcal05 become red intermittently
on vxworks with the U (Update) error condition. don't see any other errors corresponding to this. i
believe b0fcal00 was shepherded during the owl shift last night ... what is causing this ...? 

3:38:05 PM->Busy Timeout: VRB_ISL_06 

hrr worked.

3:40:15 PM->Silicon Timeout:BUSY- Slots:  08:fa00 10:fa20 12:fa40 16:f800 18:f820 20:f840 

hrr worked.

Still taking data smoothly with store 3228. 
B0lum 23E30, pbar stack 40 mA.  Plan to continue 
until at least 6am or 7am tomorrow morning 
at which time there will be 4 hours of TeV studies  
or an access to fix two TeV roughing pumps or both, 
then another store. 

Had trouble with the PEM and PHA HV, otherwise no  
significant CDF problems.

Runs                   179055
Delivered Luminosity   821.4 nb-1  
Acquired Luminosity    720.6 nb-1  
Efficiency             87.7%