2004 CDF E-Log -- Day shift. Wed Feb 25, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Beate Heinemann Susana Cabrera Tom Schwartz Silvia Amerio J. J. Schmidt


Start of Shift Notes:  

MCR have just dropped the store. We have a 90 minute controlled access: 
- Mark Mattsen: PCAL ADMEM 
- Mary and Koji: FCAL 
- Dan Cyr: BMU but did not go in in the end due to lack of time
- Bob Wagner: COT  
- Rob Roser, Lou, Derwin, Casey 

We will try and take calibrations before the next shot setup. 
Shot setup is planned to start directly after this access.

Anyes is working on XTRP crate.
Wed Feb 25 08:26:38 Run 179335 ACTIVATE: Calib/QIE/calib_CLC_qie - Susanax2080
Wed Feb 25 08:28:59 Run 179335 TERMINATE: Calib/QIE/CALIB_CLC_QIE - Susanax2080
Wed Feb 25 08:33:12 Run 179336 ACTIVATE: XTRP test with trig_X10.mcs for track OR. (simtracks_marino.dat) - anyes
-- Wed Feb 25 08:40:05 comment by...A --  
typo trig_x11.mcs
Wed Feb 25 08:34:34 Run 179337 ACTIVATE: Calib/LED/ACE_LED_ALL - Susanax2080
Wed Feb 25 08:36:23 Run 179337 TERMINATE: Calib/LED/ACE_LED_ALL - Susanax2080
Wed Feb 25 08:36:53 Run 179338 ACTIVATE: XTRP test with trig_X10.mcs for track OR. (simtracks_max.dat) - anyes
-- Wed Feb 25 08:40:22 comment by...A --  
typo trig_x11.mcs
Wed Feb 25 08:39:20 Lou and Derwin were crying that Beate left them off the access list. Also, the "IMU Movement" folder has nice laminated pages. It lacks a grease pencil, so you can write on those pages. At some future time, this should be corrected. - m mattson, lou, and derwin
-- Wed Feb 25 08:41:07 comment by...Beate --  I am terribly sorry! Will change this now!
-- Wed Feb 25 09:58:18 comment by...m mattson --  
Now Lou and Derwin are upset that I made Beate upset... they were never crying.
-- Wed Feb 25 10:47:11 comment by...Beate --  I am happy now that noone is or has been crying.
Wed Feb 25 08:41:27 Run 179340 ACTIVATE: Calib/QIE/ACE_QIE - Susanax2080
Wed Feb 25 08:42:29 Run 179340 TERMINATE: Calib/QIE/ACE_QIE - Susanax2080
-- Wed Feb 25 08:49:38 comment by...m mattson --  
Do not be surprised about problems with pcal07, slot 20. This calibration was taken before repairs.
Wed Feb 25 08:45:28 Run 179341 ACTIVATE: XTRP quick run after putting things back to default - anyes
Wed Feb 25 08:47:45 XTRP trest of new trig file (trig_x11.mcs) for OR tracks related to UDPS. See XTRP 2TT new firmware  - Anyes
Wed Feb 25 09:04:41 Dervin powercycled the fcal crates for us. This fixed the calibration problem in fcal01 ADMEM slot 6.
The smxr in fcal00 which is used to read out the roman pots appears to be dead (we were hoping that powercycling would fix it). Not a big problem since we aren't reading the pots out. I marked it back offline. We can try swapping the smxr during the shutdown.
 - terashi, convery
Wed Feb 25 09:11:47 Run 179344 ACTIVATE: Calib/QIE/SMQIECALIB - Susanax2080
Wed Feb 25 09:16:00 Run 179344 TERMINATE: Calib/QIE/SMQIECALIB - Susanax2080
Wed Feb 25 09:20:19 Run 179345 Activated at 2004.02.25 09:20:09 - RunControl
Wed Feb 25 09:21:42 Run 179345 Terminated at 2004.02.25 09:21:23 - RunControl
Wed Feb 25 09:22:05 Cryo-guy will start repowering the solenoid. - Beate
Wed Feb 25 09:29:23 Run 179346 Activated at 2004.02.25 09:29:14 - RunControl
Wed Feb 25 09:29:58 Run 179346 ACTIVATE: Ace/ACE_CALORIMETER_MINIMAL (PMT spike run) - Susanax2080
Wed Feb 25 09:30:17
 - Silvia
-- Wed Feb 25 09:30:40 comment by...Silvia --  Smx calibration plots
Wed Feb 25 09:33:55 Run 179346 Terminated at 2004.02.25 09:33:02 - RunControl
Wed Feb 25 09:34:37 Run 179346 TERMINATE: Ace/ACE_CALORIMETER_MINIMAL (PMT spike run) L1 rate ~ 57 Hz - Susanax2080
Wed Feb 25 09:38:54 Run 179347 ACTIVATE: QIE calib and download for PCAL crates - Susanax2080
Wed Feb 25 09:49:59 Run 179347 TERMINATE: QIE calib download for PCAL crates - Susanax2080
Wed Feb 25 09:53:41 Run 179349 Activated at 2004.02.25 09:53:33 - RunControl
Wed Feb 25 09:54:09 Run 179348 ACTIVATE: XTRP test trig_x11.mcs with physics table and simtracks_marino.dat - anyes
Wed Feb 25 09:54:10 Run 179349 ACTIVATE: PMT spike run: (Ace/ACE_CALORIMETER_MINIMAL) - Susanax2080
Wed Feb 25 09:55:00 Run 179349 Terminated at 2004.02.25 09:54:22 - RunControl
Wed Feb 25 09:55:01 Run 179349 TERMINATE: ACE/ACE_CALORIMETER_MINIMAL (pmt SPIKE RUN) L1 HZ ~56 HZ - Susanax2080
Wed Feb 25 09:58:45 
A recommendation from Mark Mattson to ACE DAQ: 

If we are taking a PMT spike run, that is a test run for the L1, 
and the L3 has problems, we can chose the software event builder 
and do not use the l3 subfarms.
 - Susana.
-- Wed Feb 25 10:02:39 comment by...m mattson --  As the ACE noticed, the spike run is to test the L1 system, and only uses L3 for the consumers (like TrigMon and YMon). The software EVB, although slower, will also work.
Wed Feb 25 10:00:25 
During the access, when I was going to take the PMT SPIKE RUN 
I had a problem with one SCPU l3, RUN CONTROL told me to clean 
the event buider .
 - Susana.
Wed Feb 25 10:01:01 The ADMEM in pcal07 slot 20 was successfully replaced. It will be taken to the test stand for an evaluation.

The showermax cards in pcal07 are behaving normally, and the spike run L1 rate is good. The pcal crates had a calibration and FRAM download. CO should expect that the next QIE calibration will show "bad" channels for pcal07. The values are good, but different from the current reference values.

I will unplug the wireless system when I head back to the new trailer. - m mattson

Wed Feb 25 10:03:24 XTRP: Took another quick run(179348) with physics table & simtrack. for UPDB track ORing. Looks good. - Anyes :: (run 179348)
Wed Feb 25 10:11:43 Run 179351 ACTIVATE: Calib/QIE/CALIB_QIE_FWD (BSC cal) - Susanax2080
Wed Feb 25 10:12:39 Run 179351 TERMINATE: CALIB/QIE/Calib_qie_FWD (BSC calib) - Susanax2080
Wed Feb 25 10:14:46 Unfortunately? the HV channel in BMU wouldnt trip at all this morning, so Phil and I decided not to go into the hall and perhaps make matters worse in a futile efffort to find an intermittent problem. Last time something similar happened we went for 10days before the problem appeared again. - Dan & Phil
Wed Feb 25 10:16:46 Run 179352 ACTIVATE: Calib/Muon/CALIB_MUON_ALL1 - Susanax2080
Wed Feb 25 10:17:13 Run 179352 TERMINATE: Calib/Muon/CALIB_MUON_ALL1 - Susanax2080
Wed Feb 25 10:25:21 MCR called: they will be injecting beam in a few minutes. They will start with a wet squeeze rather than a dry squeeze. - Beate
-- Wed Feb 25 11:02:45 comment by...Beate --  They were not able to do the wet squueze in the end.
Wed Feb 25 10:34:36 COT SL1 lowered by 8%, for tracking purposes it is effectively off. We need five hours (minimum) with beam in this state. The run should (probably) be marked "bad". The goal is to test whether the initial transient in SL2 current is driven by gas heating. At the lowered setting SL1 has a current density similar to SL8 (and therefore should not age) but is in closer thermal contact with SL2. If the SL2 to SL1 current ratio is flat, it would say imply the transient is in fact from gas heating. - Aseet Mukherjee
Wed Feb 25 10:41:23 Run 179353 ACTIVATE: central laser calibration - B on - Susanax2080
Wed Feb 25 10:43:09 Solenoid is ramped up again. We're ready to go! MCR have started proton injection. - Beate
Wed Feb 25 10:49:28 Silicon Maintenance done, see silicon elog  - Florencia, Pete,Rainer
Wed Feb 25 10:50:36 TOF PC has a system Alert Message: 2/25/04 08:23:55.0 !SystemAlert! The alarm queue 'SUMMARY LIST' has overflowed. At least one alarm may have been lost. The PC is not hung and smacs seems to be running fine. - Tom
-- Wed Feb 25 11:11:52 comment by...Tom --  
Talked To Expert about this.  Doesn't seem to be a problem.  Just a log that might have
overflown and needs to be reset.  Nothing to worry about from ace side.

-- Wed Feb 25 12:10:39 comment by...Tom --  
On second call, expert suggested this message could be an indication of memory,swap memory
filling up.  Recommended Restart PC.  PC Restarted.


SMACS FILE: TofCrate1_2_3_Fix.smc
HV Setup: Default 040218_today_b14kG_g30K.suf
Wed Feb 25 10:52:14 Run 179353 TERMINATE: Terminate central hadron laser calibration - Susanax2080
Wed Feb 25 11:19:00 stack is at 203E10. - Beate
Wed Feb 25 11:29:45 MCR called: they are starting shot setup now - Beate
Wed Feb 25 11:37:28 
Icicle heart beat not received, I have re-started icicle
 - Susana.
Wed Feb 25 11:46:30 Run 179354 ACTIVATE: AAA_SHOTSETUP run: new trigger table: PHYSICS_TEST_02_2[2,429,433] and plug calorimeter crates and trigger inhibits removed because expert working with them. - Susanax2080
Wed Feb 25 11:47:42 
b0COT09 TDC crate timeout on global TDC, it has been reset 
and shepherded successfully.
 - Susana
Wed Feb 25 12:19:29 Run 179354 TERMINATE: AAA_SHOTSETUP test trigger table PHYSICS_TEST_2[10,429,433] no pcal and trigger inhibit crates. - Susanax2080
Wed Feb 25 12:25:11 I restarted the CSL code with one change which will reduce the frequency of csl Level3 messages from every 10 seconds to every 60 seconds. This should reduce the rate of a memory leak in the csl by a factor of six so it is no longer a potential operational problem. The precise cause of the leak is still unknown. - Tony Vaiciulis
Wed Feb 25 12:31:51 interim scico on duty (rainer for beate). - Rainer
-- Wed Feb 25 12:37:27 comment by...Tom --  Is the interim scico supposed to bring the aces food ??
-- Wed Feb 25 12:38:39 comment by...Rainer --  hmm.... good point. interim scico still learning ...
Wed Feb 25 12:34:26 Monitoring ACE PC with acnet plots was flipping out. The cursor wouldn't hold still on any menu items (no idea why). Restarted PC (ACNET Comes up automatically). Things are back to normal. - Tom
Wed Feb 25 12:57:25 sighted a 2nd proton bunch! things progressing. - excited interim scico
Wed Feb 25 13:21:04 Guillelmo working on L3 - Steve V. still working on trigger inhibit. still only two lonely proton bunches in the machine ... - Rainer
-- Wed Feb 25 13:53:32 comment by...Guillelmo --  Don't worry Mr. SiCo, I was only changing the l3proxy version. It's running the official current version now.
Wed Feb 25 13:30:11 MCR elog - Ready to load final protons. - djinn  - rainer
-- Wed Feb 25 13:39:29 comment by...rainer --  first protons going in.
-- Wed Feb 25 13:54:28 comment by...Rainer --  MCR started with 290's, but re-considered and put in 270's.
Wed Feb 25 13:39:12 Run 179355 ACTIVATE: AAA_SHOTSETUP PHYSICS_TEST_2_02[10,429,433] no plug and trigger inhibit crates - Susanax2080
Wed Feb 25 13:40:48
DateTimeBLMDose
2004.02.2513:39:34W Inner BLM0.41RADS
2004.02.2513:39:34W Outer BLM0.13RADS
2004.02.2513:39:34E Inner BLM1.24RADS
2004.02.2513:39:34E Outer BLM1.83RADS
Integrated dosage - Tom
Wed Feb 25 13:44:45 Run 179355 TERMINATE: AAA_SHOTSETUP - Susanax2080
Wed Feb 25 13:47:18 procmon, tevmon, etc-mon were not running (expert allegedly killed them, without restarting.) Being restarted. - Rainer
-- Wed Feb 25 13:47:48 comment by...Rainer --  never mind, was only iconized.
Wed Feb 25 13:52:39 Run 179356 TERMINATE: AAA_SHOTSETUP - Susanax2080
-- Wed Feb 25 13:53:51 comment by...Rainer --  to do trigger table test with full DAQ (some was taken by Steve V. test of trigger inhibits)
Wed Feb 25 13:53:20 called aseet to verify COT procedure for this upcoming store. he said it's ok to proceed as usually (COT to standby before anti-proton injection). he will watch from his office. - Rainer
Wed Feb 25 13:53:52 Expert stopped by about the VOICE ALARM showing red on the global alarms screen. Apparently if there is no update within a particular amount of time the alarm goes red. so if PC is real busy this will happen. update time was lengthened. if you see the VOI alarm go red again please email yum@fnal.gov about it. - Tom
Wed Feb 25 13:55:33 COT at standby. MCR beginning antiproton injection. - Tom
Wed Feb 25 13:57:07 9718E09 protons loaded. page ops manager. - rainer
Wed Feb 25 13:59:06 Run 179357 ACTIVATE: AAA_SHOTSETUP PHYSICS_TEST_02_2[10,429,433] all crates in. - Susanax2080
-- Wed Feb 25 14:05:45 comment by...rainer --  this is the test of the trigger table. we will switch on flattop to the AAA_NOSILICON w/ PHYSICS_HIGHLUM_2_02[1,420,427] in anticipation of lots of lumi ....
Wed Feb 25 14:19:11 1377E09 pbars loaded. - Rainer
-- Wed Feb 25 14:22:10 comment by...rainer --  squeeze.
-- Wed Feb 25 14:31:24 comment by...rainer --  scraping ongoing ...
-- Wed Feb 25 14:36:22 comment by...rainer --  scraping done.
Wed Feb 25 14:20:41 Run 179357 TERMINATE: AAA_SHOTSETUP PHYSICS_TEST_02_2[10,429,433] ALL CRATES - Susanax2080
Wed Feb 25 14:40:00 store #3256 colliding w/ 61.68E30, 8685 protons, 1204 pbars. - Rainer
Wed Feb 25 14:40:24 PEM TRIP Crate 1. Standby to On. - Tom
Wed Feb 25 14:41:08 VOICE Global Alarm. emailing expert. - Tom
Wed Feb 25 14:42:36 Run 179358 Activated at 2004.02.25 14:42:20 - RunControl
-- Wed Feb 25 14:45:06 comment by...Rainer --  dead time 3.3kHz L1 10 kHz, L2=400 Hz.
-- Wed Feb 25 14:49:00 comment by...Rainer --  losses are sugar. 8.6kHz proton losses, 4K proton abort gap losses. not that any silicon would take advantage of that ...
Wed Feb 25 14:43:58 Run 179358 ACTIVATE: AAA_NOSILICON -> PHYSICS_HIGHLUM_2_02[1,420,427] svx, isl, l00 masked off - Susanax2080
Wed Feb 25 14:48:52 
Done timeout error for crate b0pcal10. Run halted. 
b0pcal10 crate reset and shepherded succesfully.
 - Susana.
Wed Feb 25 14:49:47
DateTimeBLMDose
2004.02.2514:44:35W Inner BLM3.53RADS
2004.02.2514:44:35W Outer BLM3.20RADS
2004.02.2514:44:35E Inner BLM3.53RADS
2004.02.2514:44:35E Outer BLM3.53RADS
Integrated dosage - Tom
-- Wed Feb 25 14:50:43 comment by...Tom --  After scraping.
Wed Feb 25 14:56:45 CMX trip. Interim scico stepping down, beate takes over. she also brought some donuts. - Rainer :: (run )
Wed Feb 25 15:00:21 
Trigger inhibit HV:CMX. Run paused and resummed when it cleared off.
 - Susana.
Wed Feb 25 15:04:52
 - Tom
-- Wed Feb 25 15:05:52 comment by...Tom --  hourly plots through shot.
Wed Feb 25 15:06:47 
CMX Trip Channels 16 and 17.
 - Tom
-- Wed Feb 25 15:07:04 comment by...Tom --  standby to on fixed it.
Wed Feb 25 15:46:43 
L2 decission timeout, out-of-sinc problem, the run was haletd, 
Alpha rebooted successfully, recovered and run.
 - Susana.
Wed Feb 25 15:55:07 Shift Summary: 
access from 8-9.30am:  
===================== 
- fcal01 ADMEM fixed due to crate power cycling 
- pcal ADMEM was fixed by M. Mattsen 
- Silicon guys swapped VRB 
- Anyes did some UDPS related tests of XTRP 
- we took all calibrations 

otherwise 
========= 
- tested new trigger table PHYSICS_TEST_02_2[2,429,433] 
- "system alerts" of TOF PC were reset by restarting the PC 

running: 
======= 
- store 3256 is going: initial lumi was 62E30 
- COT SL1 degraded: 8% lower HV 
- one trip of CMX, else running great

End of Shift Numbers
CDF Run II 

Runs                   179358
Delivered Luminosity   266.3  
Acquired Luminosity    215.2  
Efficiency             80.8
 - Beate
Wed Feb 25 15:55:26
Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
179358 x2BC9E BEAM PHYSICS_HIGHLUM_2_02 [1,420,427] 14:42:20 01:04:20 37,493,166 1,333,863 201,532 217.183 1
Totals 15:55:02 01:04:20 37,493,166 1,333,863 201,532 217.183
 - End of Shift Report
Wed Feb 25 15:56:20
 - (hourlies) Tom