2004 CDF E-Log -- Eve shift. Tue Feb 17, 2004
SciCo DAQ Ace Monitoring Ace CO (Operations Manager)
doug glenzinski tom schwartz susana cabrera Sughyun chang rob harr


Start of Shift Notes:  

 
* --- plan: 

	> wait for COT HV to come back (it's on its way) 
	> take physics w/ dflt table, dps enabled
 Tue Feb 17 16:13:20 Run 179152 Activated at 2004.02.17 16:12:53 - RunControl
Tue Feb 17 16:13:21 Run 179152 ACTIVATE: COT back on. - Tom x2080
Tue Feb 17 16:24:08 pause run for a minute so that eric james can mask-off the hot CMU trigger tower. - doug :: (run 179152)
Tue Feb 17 16:29:27 
B0PAGC almost at ~19K in fast time plots. Ops manager informs about new MCR procedure: MCR will
use the collimators only when b0pagc>20k. When this will happen, we will put the Silcion in standby
and call MCR.
 - Susana.
-- Tue Feb 17 16:45:43 comment by...doug --  also put cot, ces, cpr, ccr, cmu, cmp, cmx, bmu at stdy as per the whiteboard
Tue Feb 17 16:33:37 
Masked off CMU wire pair 12-14 in east wedge 0 from  
the trigger.  This brings trigger rates back down to 
their expected levels.  Since the hot spots in this  
wedge seem to come and go, we'll probably just leave  
this pair disabled until fixes can be made.  This  
should not have a big effect on trigger efficiency  
as we have two wire pairs per stack.
 - Eric James :: (run 179152)
-- Tue Feb 17 17:09:39 comment by...ps --  the TrigMon trigger data plots still show an excess in wedge 0, but the cross section is where it's supposed to be so we believe we got the pair that was causing the problem. Simulated triggers are way out of whack because now there are 2 masked pairs it doesn't know about, both with large rates.
Tue Feb 17 17:10:00
 - Susana.
Tue Feb 17 17:41:24 Run 179152 Terminated at 2004.02.17 17:40:42 - RunControl
Tue Feb 17 17:41:25 Run 179152 TERMINATE: mcr improving abort gap and changing trig table - Tom x2080
Tue Feb 17 17:48:13 ramp-down appropriate chambers in preparation of collimator moves to clean-up AG. take this opportunity to prepare to do a test run w/ new trigger table using 5 series offline in L3 - doug
-- Tue Feb 17 18:01:17 comment by...doug --  MCR does a very nice job and lowers the PAGC losses from 19.5kHz to 10 kHz. bringing voltages back up.
Tue Feb 17 17:52:01 
Silicon, COT,  CES,CPR,CCR,  CMU,CMP,CMX,BMU in standby. 
In particular, for CMU we hit standby botton twice.  After the first hit CMU was in yellow (warning
state, hardware overcurrent). After the second hit, all CMU went to standby mode.
 - Susana.
Tue Feb 17 18:03:59
 - Susana.
-- Tue Feb 17 18:05:10 comment by...Susana. --  
After scraping, proton AG losses back down to ~10K.

-- Tue Feb 17 18:05:59 comment by...Susana, --  
Silicon COT CES-CPR-CCR CMU-CMP-CMX-BMU back on
Tue Feb 17 18:04:13 Run 179153 Activated at 2004.02.17 18:03:49 - RunControl
Tue Feb 17 18:04:28 Run 179153 ACTIVATE: New Fancy Trigger Table PHYSICS_2_02[1,418,426] - Tom x2080
-- Tue Feb 17 18:05:57 comment by...Tom --  Started With a luminosity of 24.3E30
-- Tue Feb 17 18:08:25 comment by...Tom --  Also MCR just finished moving collimators.
Tue Feb 17 18:14:01 w/ new table (i think it's only really different at L3), we have L1A=11kHz L2A=180 Hz L3A=35 Hz @ 24E30 - doug :: (run 179153)
Tue Feb 17 18:23:08 Run 179152 RUNSTATUS: 
Marked Bad, explanation:
L00 west all red in silimon.  experts fixed, but suggested to mark this bad on
account of this problem.
 - cdfscico
-- Tue Feb 17 18:30:27 comment by...doug --  note, the run isn't marked bad, just the L00 status bit
-- Wed Feb 18 08:03:44 comment by...SCN --  Explanation in Si Elog. 95% sure ISL should be marked bad too, but will get pager carrier/SPL (same victim) to check.
Tue Feb 17 18:29:24 Run 179151 RUNSTATUS: 
Marked Bad, explanation:
COT HV was off while they added 5% N to gas mix;  this was a test run
 - cdfscico
Tue Feb 17 18:45:05 Busy Timeout VRB_ISL_06. HRR Recovered it. - Tom
Tue Feb 17 18:53:26 Run 179153 Terminated at 2004.02.17 18:53:18 - RunControl
Tue Feb 17 18:55:01 Run 179153 TERMINATE: End of using new Trigger Table. Switching back to default. - Tom x2080
Tue Feb 17 18:58:11 Run 179154 Activated at 2004.02.17 18:57:51 - RunControl
Tue Feb 17 18:59:22 Run 179154 ACTIVATE: Back To Default Physics_2_01[4,416,424]. - Tom x2080
Tue Feb 17 19:15:03
grey in imon can be ignored - they are known problems.
 - rainer
Tue Feb 17 19:26:02
 - Susana.
Tue Feb 17 19:48:12 
Stuck CellID in 5 events in Silicon SVX B1W5 L4.  HRR recovered.  Here's full error message: 

b0dap84.fnal.gov:ConsumerErrorRe:7:45:35 PM->Runtime Error 1, Event 10962, RunNum 179154: SvxMon
Halt Recover Run: Stuck Cellid in 5 events in Silicon/S/B1/W5/L4/C7-13 .[MLE] 

(MLE) b0dap73.fnal.gov:Thread-10648:7:45:35 PM->Requested Halt-Recover-Run issued [errmon] 
(RC)   19:45:38 Halt -> HALTED 
(RC)   19:45:44 Recover -> RECOVERED 
(RC)   19:45:55 Run -> ACTIVE
 - Tom
Tue Feb 17 19:55:45 Run 179154 Terminated at 2004.02.17 19:55:34 - RunControl
Tue Feb 17 19:55:59 Run 179154 TERMINATE: Switching Trigger Table to Highlum - Tom x2080
Tue Feb 17 20:03:58 Run 179155 Activated at 2004.02.17 20:03:20 - RunControl
Tue Feb 17 20:05:02 Run 179155 ACTIVATE: New Trigger Table PHYSICS_HIGHLUM_2_02[1,420,427] - Tom x2080
Tue Feb 17 20:23:48 
L1_Jet5_PS20 changed prescale from 20 to 4.  We were getting a fair amount of L2 Timeouts so we
tried to raise the L1 accept rate.
 - Tom/Rob
-- Tue Feb 17 20:49:50 comment by...Tom --  
No More L2 Timeouts.  Seemed To Be a nice little ad-hoc fix.
Tue Feb 17 20:38:21
 - Susana.
Tue Feb 17 20:50:28 Run 179155 Terminated at 2004.02.17 20:50:02 - RunControl
Tue Feb 17 20:50:50 Run 179155 TERMINATE: Going Back To Regular Table. - Tom x2080
Tue Feb 17 20:57:15 Run 179156 Activated at 2004.02.17 20:54:29 - RunControl
Tue Feb 17 20:57:16 Run 179156 ACTIVATE: Back To Default Trigger Table PHYSICS_2_02[1,418,426] - Tom x2080
-- Tue Feb 17 20:59:26 comment by...Tom --  Sorry typed in wrong trigger table. We're using: PHYSICS_2_01[4,416,424]
Tue Feb 17 21:07:16
.
 - S.H. Chang
-- Tue Feb 17 21:31:41 comment by...doug --  this is a known problem as mentioned here.
-- Wed Feb 18 09:25:02 comment by...ps --  also on the co's fix list
Tue Feb 17 21:38:46 
SPM alarm, it cleared off after ~2 seconds. (1RR18D-1, CHANNEL 2)
 - sUSANA.
Tue Feb 17 21:45:40
 - Susana.
Tue Feb 17 21:55:28 
CCR Trip while we're dealing with an apparent loss of communication with an SVX crate.  on to
standby to on fixed the CCR Trip.
 - Tom
-- Tue Feb 17 21:58:54 comment by...Tom --  Sorry this is a little out of order. Loss of communication i mention above is same problem on ifix SVX trip i mention below.
Tue Feb 17 21:56:37 
IFIX:SVX HV Trip.  A bunch of cells on the Silicon monitoring screen went blue.  Silicon Expert
on phone with Monitoring Ace.
 - Tom
-- Tue Feb 17 21:57:00 comment by...Tom --  
Run is in Halt.

-- Tue Feb 17 22:03:13 comment by...Tom --  
Something odd is that the IFIX Global alarm did not go off even though the Trigger Inhibit page
is showing a red inhibit on ifix:SVX HV.
Tue Feb 17 22:22:47 Started Taking Data again with IFIX: L00 masked. IFIX tripped on SVX, L00, and ISL then all recovered except L00. Expert is coming. Seems like an IFIX issue right now. - Tom
-- Tue Feb 17 23:02:37 comment by...rainer --   CAEN weirdness

  • crate#1 dropped out and killed 10 wedges. looks like a hocker-type incident except for the fact that the CAEN remained responsive. re-ramp brought the wedges back up
  • L00 B1W3L3 kept a trigger inhibit - Vbias was set to 0 volts, either a fallout of above or due to the same reason (single event upset ?). resetting the correct bias voltage cleared the inhibit. Details see silicon elog.
    Tue Feb 17 22:35:19 Run 179153 RUNSTATUS: 
    Marked Bad, explanation:
this was a test run w/ new trigger table physics_2_02 that uses 5.x offline
in L3, but w/ the old silicon alignment.  jonathan suggested i mark it bad.
     - cdfscico
    Tue Feb 17 22:38:12 Run 179155 RUNSTATUS: 
    Marked Bad, explanation:
i marked L3T bad b/c this was a test run w/ a new HIGHLUM trigger table that uses
5.x offline in L3 but w/ the old silicon alignment.  jonathan suggested i mark it bad.
     - cdfscico
    Tue Feb 17 22:41:31 
    b0cot05 crate was unresponsive.  Showed red in VXWorks System Monitor with an update flag (red
U) but did not go red in replies and acknowledgements.  Shepherded the crate and it seems to work
now.
     - Tom
    Tue Feb 17 22:43:23 
    L00 is now unmasked.  No Inhibit set for IFIX:L00.  Rainer is still working on it.
     - Tom
    Tue Feb 17 22:47:18
     - Susana.
    Tue Feb 17 23:03:00 
    b0pcal05 crate problem on an HRR requested because of Stuck Cellid SVX B1W5L4/C7-13 came from
b0dap84.  seemed to be another communication update loss on the cal crate. shepherding seemed to
fix.
     - Tom
    Tue Feb 17 23:06:33 i am unable to start netscape from b0dap29. annoying b/c the bookmarks at the top were very useful. i don't get them w/ mozilla. - doug
    Tue Feb 17 23:40:11 DAQ had two Silicon Inhibits: 1) SVX B2 W11 LAYER0 2) L00 B0 W3 LAYER3 No information in the CDF GLOBAL ALARMS Ifix, all silicon in green. 1) Silicon Power Supply control showed a set of wedges in grey: no communication. 2) Silicon Power Supply control did not show anything. For more on the subject, please go to the link to Silicon E-log, from Rainer.  - Susana
    Tue Feb 17 23:47:35
     - Susana
    -- Tue Feb 17 23:48:28 comment by...Susana. --  
    B0PAGC twice above ~15K Hz, TevMon in warning status.
    Tue Feb 17 23:55:57
    Run Number Data Type Physics Table Begin Time End Time Live Time L1 Accepts L2 Accepts L3 Accepts Live Lumi, nb-1 GR SC RC
    179151 x2BBCF BEAM PHYSICS_2_01 [4,416,424] 15:22:51 16:01:20 00:35:50 4,378,324 78,829 34,900 57.957 0 1 1
    179152 x2BBD0 BEAM PHYSICS_2_01 [4,416,424] 16:12:53 17:40:42 01:19:23 72,420,202 1,173,655 211,714 121.815 1 1 1
    179153 x2BBD1 BEAM PHYSICS_2_02 [1,418,426] 18:03:49 18:53:18 00:47:51 39,821,527 628,235 119,285 68.879 0 1 1
    179154 x2BBD2 BEAM PHYSICS_2_01 [4,416,424] 18:57:51 19:55:34 00:55:56 45,072,024 706,920 134,069 76.635 1 1 1
    179155 x2BBD3 BEAM PHYSICS_HIGHLUM_2_02 [1,420,427] 20:03:20 20:50:02 00:44:59 8,700,084 420,974 88,104 59.089 0 1 1
    179156 x2BBD4 BEAM PHYSICS_2_01 [4,416,424] 20:54:29 02:16:07 119,286,632 1,652,401 325,724 166.047 1
    Totals 23:55:02 06:40:09 289,678,793 4,661,014 913,796 550.421
     - End of Shift Report
    Wed Feb 18 00:05:17 Shift Summary: 
     * at 16:20, start run 179152 w/ 5% N in COT and using 
    dflt trig table, dps enabled 

        - lumi = 26E30, lostp=4kHz, lostpb<1kHz, b0pagc=18kHz 
                 L1A=11 kHz, L2A=200 Hz, L3A=38 Hz 

  * at 17:40, stop 179152 to scrape AG losses and load new  
    trigger table physics_2_02[1,418,426] for testing 

  * at 19:00, stop 179153 to return to old trigger table for      
    comparison w/ above run (at similar luminosities) 

  * at 20:00, stop 179154 to test new HIGHLUM table 

  * at 21:00, stop 179155 to return to dflt table (for rest of  
    shift) 

  * Plan: 
        > take data w/ dflt table 
        > separator scans at ~0800, read whiteboard

    End of Shift Numbers
    CDF Run II 

    Runs                   179151-156
Delivered Luminosity   646 nb-1  
Acquired Luminosity    498 nb-1  
Efficiency             77%
     - doug