2004 CDF E-Log -- Eve shift. Mon Feb 23, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
Robert Harris Chris Marino Simon Sabik Josh Tuttle JJ Schmidt


Start of Shift Notes:  

Inherit Store 3252 with lum 13E30 and a cosmic ray trigger  
test run in progress.  Plan was to drop the store at 4 p.m. 
and go into shot setup, but this still hasn't happened yet. 

Mon Feb 23 16:09:42 MCR calls and says they are ready to drop store 3252. We lower the CLC voltage and tell them to go ahead and drop the store. - Robert Harris
Mon Feb 23 16:16:40 Run 179292 Activated at 2004.02.23 16:16:23 - RunControl
Mon Feb 23 16:17:08 Run 179292 ACTIVATE: Testing UDPS - Christopher 2080
Mon Feb 23 16:22:21 Run 179292 Terminated at 2004.02.23 16:21:29 - RunControl
Mon Feb 23 16:22:23 Run 179292 TERMINATE: Pretty boring, seeing as there is no beam left.... - Christopher 2080
Mon Feb 23 16:22:24 Run 179292 TERMINATE: Boring without Beam - Christopher 2080
-- Mon Feb 23 16:29:43 comment by...SCN, Marcel --   We saved a configuration called UDPS_TEST (based on AAA_NOSILICON) in the top Run Configuration directory (i.e. not under Ace or anything.) I'd recommend trying to get it tested with beam this store, but that is up to ops.
Mon Feb 23 16:26:56 Notice pilot protons in the Tevatron already, and call MCR to see if we are going to get any quiet time. MCR confirms that they are in shot setup and we will not get any quite time. Therefore we take no calibrations and go right into cosmic running. - Robert Harris
Mon Feb 23 16:31:26 RC asks for EVB cleanup -- cleanup successful - Christopher
Mon Feb 23 16:33:34 Run 179294 Activated at 2004.02.23 16:33:20 - RunControl
Mon Feb 23 16:33:45 Steve Nahn indcates they didn't get enough rate for the UDPS test and need to do it again. He highly recommend that we do another UDPS test run with this store, and in particular before we take data with the Silicon in.  - Robert Harris
Mon Feb 23 16:34:45 Run 179294 ACTIVATE: COSMICS[12,391,403] - Christopher x2080
Mon Feb 23 16:35:41 Since we already have pilot protons, I give MCR a call to see when they think we will have final protons. MCR estimates we will have final protons sometime within the next 90 minutes, which doesn't sound much faster then usual. - Robert Harris
Mon Feb 23 16:41:10 Rob Roser says the plan for this next store is to take data with the silicon off and the COT on so they can have one more data point on their COT aging test w/o silicon cooling. Rob doesn't see anything wrong with Steve Nahn doing a UDPS test once the luminosity from the next store has decreased substantially, towards the end of the store. - Robert Harris
Mon Feb 23 16:55:23 Run 179294 Terminated at 2004.02.23 16:54:55 - RunControl
Mon Feb 23 16:55:26 Run 179294 TERMINATE: to give bill imu crates for DSP test - Christopher x2080
Mon Feb 23 16:58:37 Bill Badgett requests to take a couple of crates during our current cosmic ray run to test a new verion of the TDC DSP microcode. He indicates he can have the system back with a 5 minute warning. I agree. - Robert Harris :: (run 179297)
Mon Feb 23 17:00:11 Run 179297 Activated at 2004.02.23 16:59:22 - RunControl
Mon Feb 23 17:00:37 MCR calls and tells us they are in pre shot setup squeeze,with one coalesced bunch of protons in the machine. They are concucting a tevatron study where the bunch is going to be pinged. The warn us that there may be some additional losses as a result. They say that they should be done in 15 minutes and will go back into shot setup. - Robert Harris :: (run 179297)
Mon Feb 23 17:01:16 Run 179297 ACTIVATE: return to cosmics minus imu crates - Christopher x2080
Mon Feb 23 17:43:50 Run 179297 Terminated at 2004.02.23 17:43:29 - RunControl
Mon Feb 23 17:44:09 Run 179297 TERMINATE: return to imu crates to partition - Christopher x2080
Mon Feb 23 17:44:36 Back in shot setup with two pilot proton bunches in the Tevatron. - Robert Harris
Mon Feb 23 17:46:54
Looking at the low-level bc mismatch errors that  
occur when reading out the TDCs in "dual-fifo" mode.  IMU 
crates 00 and 01 have been programmed in DSP V45 (which  
uses dual-fifo-accecss mode) 

Can reproduce the errors at a low rate (once per 15 minutes  
or so).   Will try to include crates in the main partition, too. 
 - W.Badgett :: (run 179301)
Mon Feb 23 17:52:51 Run 179301 ACTIVATE: L2_Torture to test TDC DSP code in IMU00 and IMU01 - Christopher x2080
Mon Feb 23 17:56:54
BC errors in IMU crates are happening at a higher  
rate than when I was using the SEVB, presumably due to  
the higher possible L2Accept rate (measured in clock time) 
Perhaps once every minute or two. 
 - W.Badgett :: (run 179301)
-- Mon Feb 23 17:59:03 comment by...WB --  
A very strange clue is that BC errors come in 
bursts during this run -- IMU BC mismatches 
correlate in time with XFT bc mismatches.   The correlation 
is not exact, as the event numbers disagree.


Mon Feb 23 18:00:28
Have put both IMU crates into "spy-mode" readout,  
with dmaChain=true. 
 - W.Badgett :: (run 179302)
-- Mon Feb 23 18:03:04 comment by...W.Badgett --  
This run was a disaster, with every 
event in the short run with errors.

Mon Feb 23 18:04:57 We get the first two full size bunches of protons in the Tevatron, and Bill Badgett returns the TDC microcode to its original state. - Robert Harris
Mon Feb 23 18:12:32 Voice scheduler alarm (red). Lasted a few minutes and became green again. - Simon
Mon Feb 23 18:15:32 Voice Scheduler alarm instructions from JJ: Voice scheduler problem isnew. When VOI box is red in the upper monitor the shift should check if the scheduler screen is still present on the lower monitor. The voice scheduler is still running if the scheduler screen is present on the lower monitor. Steve Hahn and JC Yun were consulted and indicate this is a new problem they do not yet understand. We should make an e-log entry whenever it happens. - Robert Harris
Mon Feb 23 18:16:05 All the final proton bunches are now in the Tevatron. - Robert Harris
Mon Feb 23 18:18:29
IMU crates have been deprogrammed back to their  
original configuration, including TDC DSP=37, dmaChain=false,  
and spyMode=false.   Must scrutinize the error logs in  
detail to extract some conclusion. 

Shot-setup-run starting with IMU crates back to normal. 

 - W.Badgett :: (run 179303)
Mon Feb 23 18:19:46
DateTimeBLMDose
2004.02.2318:14:29W Inner BLM0.03RADS
2004.02.2318:14:29W Outer BLM0.01RADS
2004.02.2318:14:29E Inner BLM0.02RADS
2004.02.2318:14:29E Outer BLM0.33RADS
Integrated dosage - Simon
Mon Feb 23 18:22:52 Run 179303 ACTIVATE: AAA_SHOTSETUP - Christopher x2080
Mon Feb 23 18:26:47 They are shooting from an antiproton stack of 180 mA and have gotten about 85% and 90% pbar transfer efficiency on the first two transfers: pretty good. - Robert Harris
Mon Feb 23 18:41:34 All pbar bunches are in, varying in size from 1.9 to 4.6 mA/bunch. - Robert Harris
Mon Feb 23 18:46:26 JJ informs us that until further notice the CO is supposed to restart stage 0 once per shift due to a memory leak. - Robert Harris
Mon Feb 23 18:49:25 Run 179303 TERMINATE: prepare for running without silicon - Christopher x2080
Mon Feb 23 18:54:15 Run 179305 TERMINATE: forgot to change table to high lum - Christopher x2080
Mon Feb 23 19:06:17 Store #3254 scraping complete. - Robert Harris
Mon Feb 23 19:07:40 Proton and anti-proton currents: TevPR=8.68E12, TevPB=1.07E12  - Robert Harris
Mon Feb 23 19:10:43 Initial Luminosity 52.8E30. - Robert Harris
Mon Feb 23 19:11:10 Run 179307 Activated at 2004.02.23 19:10:58 - RunControl
Mon Feb 23 19:11:49 Run 179307 ACTIVATE: Physics_highlum_2_02[1,420,427] without silicon - Christopher x2080
Mon Feb 23 19:14:42
 - Simon
-- Mon Feb 23 19:15:07 comment by...Simon --  Losses and SVRAD during shot.
Mon Feb 23 19:35:12 JJ gives us instructions for the night shift before leaving: plan is to call Jonathan Lewis around 10:15 to say we are ready to test the UDPS table. DAQ ace will switchover to use the UDPS trigger table. Jonathan will give us guidance as to how long to run the UDPS table. When that is done, we should switch to the regular low lum physics table. Those are the last instructions for the evening shift and the owl shift.  - Robert Harris
Mon Feb 23 20:04:07 Run 179307 ACTIVE: b0cot08 done timeout -- shepherd successful - Christopher x2080
Mon Feb 23 20:11:37
 - Simon
Mon Feb 23 20:38:47 Run 179307 ACTIVE: Received Scaler Monitor error at RC warning of high deadtime. Window suggests looking for
hot triggers, but I see no evidence of high deadtime, and CO finds no unusual triggers. - Christopher x2080
Mon Feb 23 21:21:37 ICICLE crashed. Restarted it. - Simon
Mon Feb 23 21:23:16
 - Simon
Mon Feb 23 21:56:38 The D0 control room calls to let us know they want to put in there forward detectors, the Roman pots, and ask if we have any objections. I tell them we have no objections, in fact, now is a good time to do that since our silicon is off.  - Robert Harris
Mon Feb 23 22:03:40 Run 179307 Terminated at 2004.02.23 22:03:12 - RunControl
Mon Feb 23 22:03:41 Run 179307 TERMINATE: ending run for UDPS test - Christopher x2080
Mon Feb 23 22:03:41 Luminosity drops belows 40E30, so I figure it is a good time to run the UDPS trigger table. I page Jonathan Lewis and he indicates that we should run the test for about an hour assuming that we don't get any Silicon resonance errors.  - Robert Harris
Mon Feb 23 22:04:45
Two things of note from MCR elog.... 

1)  Mon Feb 23 19:06:39 comment by...BME --  It looks like 
the F49H collimator did not move all the way in. The plot  
on the left shows the final position during the last number 
of stores (~+400 mils). Currently the collimator sits at 
~+76mils. We are attempting to contact Dean Still.  

2) Remaining pbar stack (28 mA) was lost due to problems with 
A:IB. (Note sure what that is but problem is with transformer 
that regulates this device.) Stack was lost at 1943 and has 
still not returned. 
 - jj
-- Mon Feb 23 22:05:24 comment by...jj --  Stacking has not returned. The original stack is of course gone forever.
Mon Feb 23 22:09:16 Run 179307 RUNSTATUS:
Marked Bad, explanation:
Silicon is off.
SVX 
ISL 
L00 
 - cdfscico
Mon Feb 23 22:09:53 Run 179312 Activated at 2004.02.23 22:09:36 - RunControl
Mon Feb 23 22:10:39 Run 179312 ACTIVATE: UDPS TEST with PHYSICS_TEST_2_02[9,428,432] - Christopher x2080
-- Mon Feb 23 22:14:24 comment by...Christopher --  plan to run this test/table for ~1hr
as expected L1 accept at ~28 kHz, L2 accept ~310 Hz
deadtime < 5%
Mon Feb 23 22:24:04 killed and restarted Stage0 after Run 179307 terminated (due to memory leak, CO's have been asked to restart this monitor once per shift) - josh
Mon Feb 23 22:33:55 Jonathan Lewis calls and remarks "this run is a beautiful thing!" - Robert Harris :: (run 179312)
Mon Feb 23 22:40:47
 - 
-- Mon Feb 23 22:42:14 comment by...Christopher --  Run 179312 Trigger Rates (UDPS TEST)
Mon Feb 23 22:41:25
 - Simon
Mon Feb 23 22:45:10 Run 179312 Terminated at 2004.02.23 22:44:38 - RunControl
Mon Feb 23 22:45:11 Run 179312 TERMINATE: SM_SCPU Timeout - Christopher x2080
Mon Feb 23 22:50:16 Had to cleanup evb; also b0eb12 needed to be manually reset - Christopher
Mon Feb 23 22:51:39
simulated FRED bits 36 and 43 differ from TFRD bank, bottom difference plot should be empty
 - josh
-- Mon Feb 23 23:12:27 comment by...jdl --  These are the UPS inputs. One is nBuffers>=1, the other is nBuffers>=3. If you look at the run summary for Prefred inputs you see things called PREFRED_BIT_24 and _36. If you compare the ratios of the counts of these with MB_XING, you get fractions that are similar to the buffer usage stats in the screen shot above. This is all very good. The only problem here is that Trigmon does not know about these bits and is using old information to similate them. This error can be ignored for this test. Also, the error described in the next plot below is the UPS trigger and the normalization trigger. (More details for thes strong-stomached on request.) That is another error that can be ignored for this test table only.
Mon Feb 23 22:53:40 Run 179313 Activated at 2004.02.23 22:53:19 - RunControl
Mon Feb 23 22:54:19
...also FRED unprescaled trigger bits 35 and 43 differ substantially from TFRD
 - josh
Mon Feb 23 22:55:46 Run 179313 ACTIVATE: Resume UDPS Test PHYSICS_TEST_2_02[9,428,432] - Christopher x2080
Mon Feb 23 22:57:30 restarted icicle as it had crashed - Christopher
Mon Feb 23 23:16:58 Run 179313 Terminated at 2004.02.23 23:15:58 - RunControl
Mon Feb 23 23:16:59 Run 179313 TERMINATE: returning to normal data taking - Christopher x2080
Mon Feb 23 23:23:08 Run 179314 Activated at 2004.02.23 23:22:04 - RunControl
Mon Feb 23 23:23:09 Run 179314 ACTIVATE: AAA_NOSILICON PHYSICS_2_02[2,424,431] - Christopher x2080
Mon Feb 23 23:31:30
 - Simon
Mon Feb 23 23:35:42 Run 179312 RUNSTATUS:
Marked Bad, explanation:
Silicon off during this UDPS test run
SVX 
ISL 
L00 
 - cdfscico
Mon Feb 23 23:35:43 Run 179313 RUNSTATUS:
Marked Bad, explanation:
Silicon off during this UDPS test run.
SVX 
ISL 
L00 
 - cdfscico
Mon Feb 23 23:39:46 I looked a little more at scaler values. The ratio of the counts for the L1_TWO_TRK2_DPHI90_UPS trigger to the counts for the similar UPSNORM trigger was 0.497. That means the 49.7% of the time that there was at least one buffer free when we had a two track trigger, we took one because there were at least 3 buffers free. The ratio for the prefred inputs that determine if there are >=1 or >=3 is 0.498. The agreement is darn good. I would guess that difference is an artifact of the long integration time and agreement would be better using run sections. The next thing to check is the actual buffer usage scalers to see how well they agree with the UPS prefred bits. - jdl :: (run 179312)
Mon Feb 23 23:56:30
Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
179307 x2BC6B BEAM PHYSICS_HIGHLUM_2_02 [1,420,427] 19:10:58 22:03:12 02:49:04 69,632,105 2,767,833 444,260 456.596 1 1 1
179312 x2BC70 BEAM PHYSICS_TEST_2_02 [9,428,432] 22:09:36 22:44:38 00:32:19 56,606,901 620,551 116,049 75.108 1 1 1
179313 x2BC71 BEAM PHYSICS_TEST_2_02 [9,428,432] 22:53:19 23:15:58 00:21:39 36,740,591 401,040 75,608 48.094 1 1 1
179314 x2BC72 BEAM PHYSICS_2_02 [2,424,431] 23:22:04 00:30:41 30,786,068 376,837 66,659 64.104 1
Totals 23:55:01 04:13:44 193,765,665 4,166,261 702,576 643.902
 - End of Shift Report
Tue Feb 24 00:00:06 CCR tripped - Simon :: (run 179313)
Tue Feb 24 00:01:15 Shift Summary:
* 16:10 Store 3252 is dropped and the Tevatron goes into
shot setup with a stack of around 200 mA. We have *no* quiet time, so we don't take calibration
runs. 


* 17:00 Bill Badgett works on TDC DSP microcode for about an hour using two crates during a cosmic
ray run while Tevatron in shot setup. 


* 19:00 Store 3254 from a pbar stack of 180 mA.has intial luminosity of 53E30. We take ~500 nb^-1 of
good data at an efficiency of around 93%. 


* 22:00 Luminosity drops below 40E30 and we begin an hour-long UDPS test run. Jonathan Lewis happy
with it: "this run is a beautiful thing!"  


* 23:20 Resume taking physics data with low luminosity table at 35E30. 

During the shift there are two instructions from JJ: 
  1) Watch for and log problems with the Voice Scheduler (see e-log). 
  2) Restart Stage 0 once per shift to avoid memory leak problem. 

End of Shift Numbers
CDF Run II

Runs                   
Delivered Luminosity   735.8   
Acquired Luminosity    651.5  
Efficiency             88.5

 - Robert Harris