2004 CDF E-Log -- Day shift. Sun Mar 7, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Larry N. Jan Ehlers Anadi Canepa Sergei B. Mary C.


Start of Shift Notes:  

Access at 10 to hopefully finish recovery
 Sun Mar 7 08:10:27 
BUSY TIMEOUT: 

Host b0eb19.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 2002693 from VRB in slot 16: 
0xa11ca01c 0xa11ca01c 0x6f09700a 0x7112720e 0x730c7413 0x7518760b 0x7714780d 0x7901860f 
1 crate/s: b0svx06(16),  busy.[RXPT]
 - Jan :: (run 179704)
Sun Mar 7 08:14:55 There will be e-lense studies at 9:30, we should turn off  - Larry
Sun Mar 7 08:14:58

Don't know why, but WHA wedges 03E 04E 05E are now reading out correctly and are green in the HV display.

 - Steve Hahn
-- Sun Mar 7 13:02:42 comment by...Steve Hahn --  

Turned out Irina had masked all these channels in WHA so they appear green with 0 HV diff for every channel. I unmasked them and tried cycling the Pisabox without success.

Sun Mar 7 08:20:14 Last night we have done extensively studies (in completely standalone teststand modes) to understand why the L2 muon fiber splitting didn't work in the beam data taking condition two nights ago(two channels failed out of 16 total). We believe that we have found the cause for that, and to fully prove it, we will need to use ~1 hour non-beam time to do a test with the L2 muon Pulsar board inside the L2 decision crate. The details are described in the Pulsar e-log (see all entries since yesterday). We will stay around, and wait for non-beam time. would very much appreciate if we can get some non-beam time this morning after the end of this store... what we need is the L2 decision crate. - Burkard and Ted
-- Sun Mar 7 08:29:04 comment by...Burkard and Ted --  To make it more clear, the test we want to do is rather simple. We will not do any actual muon fiber splitting for this test. all we need is to use the L2 decision crate. We have a muon transimitter Pulsar board in the Pulsar crate which can send data into the L2 muon Pulsar board inside the L2 decision crate. This test should not introduce problem to the L2 decision crate. we do need to book both the L2 decision crate and Pulsar crate in one paritition for this test and we don't need any other crate. The test should only take <~ 1 hour.
Sun Mar 7 08:24:43 Dervin and George Wyatt are coming in to fix/bypass the bad ODH sensor so that we can go in the collision hall for access (otherwise it would be an ODH area). We also have techs lined up to move steel for the access. - convery
Sun Mar 7 08:54:03 PSM alarm: PSM box turned red and soon after a trigger inhibit came. It cleared itself in few sec. The problem is related to 1RR18D_1. No time to check which channel caused the problem - Anadi :: (run 179704)
Sun Mar 7 09:02:04 Silicon trip: we got a hardware trigget inhibit followed by the silicon inhibit status box going red. The problematic ladder was: SVX B4W3L0. We put the run in halt and recovered the trip following the procedure (powering OFF and ON the whole wedge). Everything is fine now.  - Anadi and Jan :: (run 179704 )
Sun Mar 7 09:05:29 
FERML_HIGH_DEADTIME: just message appeared
 - Jan :: (run 179704)
Sun Mar 7 09:12:05
 - Anadi
-- Sun Mar 7 09:14:13 comment by...Anadi --  Hourly Plots: 8:00-9:00 Proton losses (LOSTPB) are still "negative". Steve explained me it's just the result of a fixed value pedestal substraction.
Sun Mar 7 09:22:38 Run 179704 Terminated at 2004.03.07 09:22:19 - RunControl
Sun Mar 7 09:22:58 Run 179704 TERMINATE: terminate for access - Jan x 2080
Sun Mar 7 09:23:30 Bringing down the voltages for store studies followed by store termination.  - Anadi
-- Sun Mar 7 09:30:33 comment by...Anadi --  
Usual "before store" configuration:
Standby: COT, Silicon, CMU SMP SMX BMU, CES, CPR, CCR, PEM PHA PSH
Off: TOF, CLC, MNP
On: everything else 
I had to bring CMU to Standby by selecting "all of CMU" and "Standby" from the Global Alarms panel.
Sun Mar 7 09:39:57
Kill All Consumer Displays/Monitors, is going to reconnect power of Consumer/DAQ servers in Computing Room to the new power strip.
 - Sergei
Sun Mar 7 09:42:57 Wireless turned on in the hall - Larry
Sun Mar 7 09:54:00 MCR calls - 10:30 instead of 10 - Larry
Sun Mar 7 10:02:31 Run 179705 ACTIVATE: Plug SMX calibration. Pluh HV at standby. - Jan x 2080
Sun Mar 7 10:03:02 Run 179705 TERMINATE: End SMX calib for plug. - Jan x 2080
-- Sun Mar 7 10:21:16 comment by...Roseanne Roseannadanna --  And it looks ok!
Sun Mar 7 10:17:35 The b0dap78,79,80 are connected to the new power strip - Sergei
Sun Mar 7 10:21:52 YMon ObjectMon, BeamMon , SiliMon AND Stage0 tcl are now set for cosmic run also SVXMon reads ALL streams  - Sergei
Sun Mar 7 10:30:40
 - Anadi
-- Sun Mar 7 10:31:36 comment by...Anadi --  Hourly Plots: 9:00-10:30. Run until 9:23, store studies afterwards.
Sun Mar 7 10:34:05 Run 179706 ACTIVATE: CAL QIE to checkout CCAL, WCAL and PCAL ADMEMs. - Vivek
Sun Mar 7 10:34:06 Run 179706 TERMINATE: End CAL QIE. - Vivek
Sun Mar 7 10:35:12 Store is dropped - Larry
-- Sun Mar 7 10:35:57 comment by...Larry --  or will be
Sun Mar 7 10:38:05 Run 179707 ACTIVATE: CAL QIE for central and wall and a possible FRAM download. - Vivek
Sun Mar 7 10:48:12 Run 179707 TERMINATE: End CAL QIE and FRAM download for central and wall crates. - Vivek
Sun Mar 7 10:56:31 Run 179708 ACTIVATE: CLC calibs  - Anadi & Jan
Sun Mar 7 10:58:24 Run 179709 ACTIVATE: BSCQIE calibs - Anadi & Jan
Sun Mar 7 11:01:15 Start All Consumer Monitors with partion_id #1 - Sergei
Sun Mar 7 11:04:48 please check the 'cleanliness' of the abort by checking the BLM fifo plots. go to page E2, select 'list dates' or smothing like that and see whether there is an entry compatible with today's abort. if it is, put the BLM summary plot into the elog, if it's not, the abort was clean. thanks. - rainer
-- Sun Mar 7 11:21:23 comment by...Anadi and Jan --  No entry for today's abort (last entry, March 2nd). Abort was clean.
Sun Mar 7 11:22:38 Run 179710 ACTIVATE: Plug Laser Calib BOTHPLUGS_NORMAL stabilization normal run  - Anadi & Jan
Sun Mar 7 11:28:29 Monitoring issue during access: PSM went red for EndWall/TOF_W. This button stayed purple even if the PSM alarm cleared out (I don't know whether this is supposed to be purple).  - Anadi
Sun Mar 7 11:28:46 Run 179711 ACTIVATE: calibs in Consumer Myron Mode (BOTHPLUGS_MM_C) - Anadi & Jan
Sun Mar 7 11:32:41 Run 179712 ACTIVATE: second D-mode data logging gjgjgkigk (Myron Mode: BOTHPLUGS_MM) - Anadi & Jan
Sun Mar 7 11:38:51 Run 179713 ACTIVATE: TOF calibs - Anadi & Jan
Sun Mar 7 11:41:07 After Mitch and Steve swapped the tracer in cot17, this crate seems to be fine. The tracer they used was in the diagnostic crate. - Jane Nachtman
Sun Mar 7 11:43:18 Run 179714 ACTIVATE: SMXQIE calibs - Anadi & Jan
Sun Mar 7 11:47:37 Run 179715 ACTIVATE: CEM LED calibs - Anadi & Jan
Sun Mar 7 11:50:22 Run 179716 ACTIVATE: TOF TAC calibs - Anadi & Jan
Sun Mar 7 11:56:36 CEM and WHA went grey in the GLOBAL ALARMS. The process PB_V6 was not updating in the PISA box so I closed the window and restarted. Sorry if I interfered with someone working on that.  - Anadi
Sun Mar 7 12:04:40 
slowcontrol ICICLE heart beat not recieved several times during access
 - Anadi and Jan
Sun Mar 7 12:05:30 Run 179717 Activated at 2004.03.07 12:05:05 - RunControl
Sun Mar 7 12:05:47 Run 179717 ACTIVATE: PMT spike run - Anadi & Jan
Sun Mar 7 12:10:30 
L1 around 60 Hz
 - Anadi and Jan
Sun Mar 7 12:14:09 Run 179717 Terminated at 2004.03.07 12:13:08 - RunControl
Sun Mar 7 12:15:35 Run 179718 ACTIVATE: COT calibs  - Anadi & Jan
-- Sun Mar 7 12:19:11 comment by...Larry --  this is a good sign!
Sun Mar 7 12:20:32 
	During access this morning, I reloaded the flash memory on the TDC in slot 17 of b0cot01.  I
marked this board back online in the hardware DB, as well as taking the crate out of SpyMode. 

	I also swapped a tracer in b0cot17.  After these fixes, the cot crates all seem to be back to
normal.  The tracer I removed from b0cot17 is now in the diagnostic crate in the trigger room.  Aces
should keep an eye out for these two crates during Cosmic runs today to make sure they are working
correctly.
 - Mitch Soderberg
Sun Mar 7 12:24:49 Run 179719 ACTIVATE: MUON calibs - Anadi & Jan
Sun Mar 7 12:29:09 Dervin reports head changed, no longer bypassed. - Larry
Sun Mar 7 12:37:05 Run 179720 ACTIVATE: COSMICS_NOTRACK - Anadi & Jan
Sun Mar 7 12:41:35 Run 179704 RUNSTATUS: 
Marked Bad, explanation:
PCAL bad ppr calibration
COT bad TDCs
 - cdfscico
Sun Mar 7 12:45:47 Run 179720 TERMINATE: for one more calib - Anadi & Jan
Sun Mar 7 12:49:35 Run 179721 ACTIVATE: another calorimeter calib - Anadi & Jan
Sun Mar 7 12:57:17 Done CheckDBANAcalib, bscqie calibration - run #179709; called calibration run #179715; calqie calibration - run #179707; clcqie calibration - #179708; smx calibration - run #179714. The calxef calibration/rpaqie calibration are still run #179494/178414. - Sergei
Sun Mar 7 12:58:15

Actually succeeded in cycling power on UNE-T Pisabox, but after waiting mandatory half hour for readout to cycle through CEM CHA and WHA, find readout still is not working for 03E 04E 05E.

I think we should not mask this big chunk of WHA so we do not forget this problem (or fool ourselves like I did this morning).

 - Steve Hahn
Sun Mar 7 13:05:25

CANWT-1 crate is going in and out of alarm in PSM for the +5 Digital supply. Sometimes is reads out as low as 3.9 V. However, we have no errors from the crate and QIE calibrations also look fine (Jan just took another calibration at our request). So I am fairly certain this is just a monitoring problem.

 - Steve Hahn
-- Sun Mar 7 13:20:12 comment by...convery --  This is in the area where TDC work was done on access.
Sun Mar 7 13:12:59 Run 179722 ACTIVATE: COSMICS_NOTRACK - Anadi & Jan
Sun Mar 7 13:15:53 Shot setup - Larry
Sun Mar 7 13:55:13 Starting test for Muon fiber splitting: working on Fiber 1 (MB02) - Burkard :: (run 179722)
Sun Mar 7 14:01:52 Run 179722 TERMINATE: end this run, record some data with fiber splitter problem - Anadi & Jan
Sun Mar 7 14:01:52 Run 179722 TERMINATE: end this run, record some data with fiber splitter problem - Anadi & Jan
Sun Mar 7 14:04:01 Run 179723 ACTIVATE: test run for muon fiber splitter - Anadi & Jan
-- Sun Mar 7 15:45:01 comment by...Burkard --  Unfortunately I did not get enough data to draw any firm conclusions
Sun Mar 7 14:08:18 YMon crashed and could not re-started it (always become "red") after start. Stopped All Consumer Displays/ Consumer Monitors, run Cosmic.sh; started All Monitors -still problem with YMon. Contacted with Consumer expert (Kaori) Did all as was told from scratch - still YMon crashes. Kaori will be in ControlRoom in three hours to help.  - Sergei
Sun Mar 7 14:09:25 Run 179723 TERMINATE: I hope I got some events - Anadi & Jan
Sun Mar 7 14:16:24 Run 179724 ACTIVATE: AAA_SHOTSETUP - Anadi & Jan
Sun Mar 7 14:17:25
Final protons have been injected
 - Anadi
Sun Mar 7 14:22:56 
for the ACEs, concerning WHA: 

the problem couldn't be solved so the GLOBAL ALARM box and the HV SUMMARY bar are yellow for this
run.
 - Anadi
Sun Mar 7 14:27:00 Pool: Burkard 66, Jan 63, Larry 60, Anadi 62, Serguei 80, Mary 67 - Larry
Sun Mar 7 14:29:40 YMon ObjectMon, BeamMon ,SiliMon AND Stage0 tcl are now set for physics run also SVXMon reads StreamA  - Sergei
-- Sun Mar 7 14:39:58 comment by...Sergei --  
NOW YMon works with set for physics run
Sun Mar 7 14:34:35 Run 179724 TERMINATE: get ready ... - Anadi & Jan
Sun Mar 7 14:46:35 Run 179725 TERMINATE: changed mind ... low lumi table - Anadi & Jan
Sun Mar 7 14:47:36 From MCR elog:
Sun Mar 7 14:34:24 comment by...ollie -- Tev at flattop, high pbar losses on ramp. So much for any kind of record.

We will prepare to run with non-HIGHLUM trigger table. - convery

Sun Mar 7 14:52:35 store 3277 scraping complete p 8610 pbar 1168 lostp 6.8k lostpbar 0.7k l=6.13 (Anadi wins) - Larry
Sun Mar 7 14:57:07 Run 179726 Activated at 2004.03.07 14:56:35 - RunControl
Sun Mar 7 14:59:42 Run 179726 ACTIVATE: trigger table PHYSICS_2_02 [2,424,431] - Anadi & Jan
-- Sun Mar 7 15:03:44 comment by...Larry --  Start with normal table, 15% DT better than change run DT
Sun Mar 7 15:00:18
 - Anadi
-- Sun Mar 7 15:00:40 comment by...Anadi --  Shot setup Plots
Sun Mar 7 15:03:53 
L=60 
15% deadtime 
L1 14kHz 
L2 380 Hz 
L3  66 Hz
 - convery :: (run 179726)
-- Sun Mar 7 15:08:25 comment by...rainer --  is DPS ON ?
-- Sun Mar 7 15:10:27 comment by...Larry --  coming up soon
-- Sun Mar 7 15:13:06 comment by...rainer --  just be aware that at 22 kHz you'll run into the L1 trigger scaler handbrake to protect the silicon, manifesting itself as done timeouts from the scaler rate.
Sun Mar 7 15:03:59 
busy timeout: 

Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-TracerEventId: Event 17051, crate 73, channel 6 has either bad Tracer ID or bad markers
around Tracer word. 


Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 17051 from VRB in slot 12
 - Jan :: (run 179726)
Sun Mar 7 15:06:44 
again busy timeout after EB error: 

Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 97855 from VRB in slot 12 

Hardware EVB has detected a problem with data quality in  
 SCPU b0eb16 (forwarded by FER crate COT_17
 - Jan :: (run 179726)
Sun Mar 7 15:09:07 CMX Trip: hardware followed by GLOBAL ALARM box turning red. SW 16,17 tripped. Trip recovered by bringing them to ON once.  - Anadi & Jan
Sun Mar 7 15:15:52 
same error message: 

Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 230394 from VRB in slot 12: 
0x00000000 0x000050e0 0x002f00af 0x0cf007ec 0x06a80d70 0x089006e8 0x18500000 0x00000000 


[IN CASE IT HAPPENS AGAIN WE END THE RUN AND RESTART]
 - J&A :: (run 179726)
Sun Mar 7 15:17:41 LOSTP is close to 18 kHz. Called MCR.  - Anad and Jan
Sun Mar 7 15:18:57 LOSTP rising rapidly. Larry called MCR, they were aware, but are hoping it will turn over. - convery
Sun Mar 7 15:24:50 Proton losses at 20 kHz (LOSTP). we halt the run and put the Si to Standby. - Anadi and Jan
-- Sun Mar 7 15:26:30 comment by...Larry --  MCR says expert is working on it- we will wait
-- Sun Mar 7 15:28:49 comment by...convery --  MCR elog: 15:26:53- LostP has rolled over ~20K. CDF has turned off thier silicon. Ron Moore is here for other reasons. He is working on bringing it down. - ollie
Sun Mar 7 15:30:17 Spurious PSM warning (box turned yellow but cleared immediately). The problem seemed to be related to SVX_SET_1 (b0fib05). No action taken. Everything looks fine now (system in halt).  - Anadi
Sun Mar 7 15:40:06 Proton losses <~ 19 kHz. Turn the Si ON (after calling MCR) - Anadi
Sun Mar 7 15:41:05 6 minute lostp <19.5k, MCR says not playing, go for it - Larry :: (run 179726)
Sun Mar 7 15:56:34
Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
179704 x2BDF8 BEAM PHYSICS_2_02 [2,424,431] 02:19:04 09:22:19 06:07:31 166,329,709 2,428,970 622,217 287.229 0 1 1
179726 x2BE0E BEAM PHYSICS_2_02 [2,424,431] 14:56:35 00:30:12 27,768,646 720,029 139,423 108.827 1
Totals 15:55:02 06:37:44 194,098,355 3,148,999 761,640 396.056
 - End of Shift Report
Sun Mar 7 16:00:33 
same coupled error (EB & Tracer) again: 

Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 744794 from VRB in slot 12: 
0x00000000 0x00005d78 0x00340034 0x16ec0770 0x079017e8 0x06440758 0x11dc0000 0x00000000 

SCPU_TRACER_EVENT_ID Error !!!  
 Hardware EVB has detected a problem with data quality in  
 SCPU b0eb16 (forwarded by FER crate CMP_00
 - J&A :: (run 179726)
Sun Mar 7 16:01:54 Shift Summary: 
Access fixed COT problems.  ShowerMax problems had fixed
themselves and Mike broght donuts.  Even better.  ODH sensor head replaced and we are no longer
bypassed.  WHA bad readout channels not fixed by cycling the Pisa box, no longer masked, consider
that a "known" problem, data is ok.   

Store 3277 came in at 6.1 and we had to turn off for a while when lostp exceeded the silicon
standard.  All in all, leaving things in much better shape than yesterday.

End of Shift Numbers
CDF Run II 

Runs                   179704-26
Delivered Luminosity   294  
Acquired Luminosity    172  
Efficiency             58.5%
 - Larry
Sun Mar 7 16:02:33 
here we go again: 

Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-VrbHeader: Dump of header words for event 766009 from VRB in slot 12: 
0x00000000 0x000071b8 0x001500d5 0x19080a28 0x0918165c 0x0a7409a8 0x1acc0000 0x00000000 
b0l3pcom1.fnal.gov:main:3:55:31 PM->Host b0eb16.fnal.gov, task tRec_0 
SCPU-P1-E-TracerEventId: Event 766010, crate 85, channel 6 has either bad Tracer ID or bad markers
around Tracer word. 

 Tracer word event ID is 5, should be 10.
 - J&A :: (run 179726)
-- Sun Mar 7 16:04:36 comment by...J&A --  crate 85 channel 6 (FER: CMP_00) or crate 66 channel 6 (FER: COT_17) <- only once
-- Sun Mar 7 16:05:50 comment by...J&A --  
always EB16