|
2004 CDF E-Log -- Eve shift. Sat Feb 14, 2004 |
| SciCo |
DAQ Ace |
Monitoring Ace |
CO |
(Operations Manager) |
| Doug Glenzinski |
Natasha Milandinovic |
Vadim Khotilovich |
Sunghyun Chang |
Mary Convery |
Start of Shift Notes:  
Run in progress. Plan is to continue taking data.
Sat Feb 14 16:19:05
Had a CSL Calibrations Consumer reporting an error to ProcMon. I did all the checks listed on the web pages and everything seemed fine. After a while Procmon reported everything fine. Must be that the problem lasted for a very short time. - natasha
Sat Feb 14 16:36:00



- (hrl.plts) Vadim
Sat Feb 14 16:39:25
Run 179104
ACTIVE:
Had L2 decision timeout:
(MLE) b0wcal03:Messenger:3:19:57 PM->Runtime Error 2, Event 7650292: Bunch counter mismatch, mismatch count = 1
L2 Decision Timeout.[RXPT]
Auto HRR worked. - natasha x2080
Sat Feb 14 16:48:41
Subfarm 9 is fixed now and may be included in the next run
after Level3 cleanup
The problem was in converter 09
- Arkadiy
Sat Feb 14 17:36:32



- (Hourlies'R'Us) Vadim
Sat Feb 14 18:38:57




- Vadim
-- Sat Feb 14 18:40:19 comment by...Vadim -- Hourly plots. B0PBMS shows a 2x spike around 17:42 but it might be a fake (too narrow).
Sat Feb 14 19:09:15
MCR are going to move collimators. Put Si, COT, CMU/P/X, BMU, CES, CPR, CCR into standby. - Vadim :: (run 179104 )
Sat Feb 14 19:27:34
MCR calls back. they're done playing w/ collimators. AG losses
actually a bit worse (13k before they played, now 14k). They
wanted to try to reduce these losses a bit before shift change
(at 20:00) so that they could avoid having to call-in experts.
We'll keep an eye on it. It was quite flat for the first 3 hours
of our shift, they expect it to stay this flat.
Also, next shot will likely take place when avg lumi=10E30 and
stack big enough. MCR guessed around 00:00, but said they'd have
the next crew chief call once they'd decided on a shift plan
- Doug :: (run 179104)
Sat Feb 14 19:35:50
Put HVs back to on. The SVX cell B1 W8 L1 turned pink when we we going to standby. Unmark + HRR made it greeeen again. - Vadim :: (run 179104 )
Sat Feb 14 19:43:44



- /plots/ Vadim
Sat Feb 14 20:17:04
Dr. Pitts calls with a trigger table plan for the start of the
next store:
- start w/ the HIGHLUM table on the whiteboard,
DPS disabled
- once lumi < 40 E30, switch to regular table
DPS enabled
He suggests we take a short run w/ the HIGHLUM table towards the
end of this store since this table hasn't been used since Jan
(he's worried that perhaps a tagset has changed).
- doug
Sat Feb 14 20:18:09
AG losses have fallen back to their previous ~13 kHz.
- doug
Sat Feb 14 20:49:06




- Vadim
-- Sat Feb 14 20:51:07 comment by...Vadim -- Hr.Pl.: Around 20:08 there was an 1 min long drop in SBDMS (RF bucket length) but we didn't have any problems.
Sat Feb 14 21:37:50



- Vadim
-- Sat Feb 14 21:38:54 comment by...Vadim -- Hr.Pl.: had a narrow spike in B0PBSM @20:57
Sat Feb 14 21:47:33
Phoned MCR. They say they'll keep store until lumi is 10E30,
which they guess will occur sometime between 0400-0600. Stack
is presently 154 mA.
- doug
Sat Feb 14 21:52:11
Run 179104
Terminated at 2004.02.14 21:51:51 - RunControl
Sat Feb 14 21:52:34
Run 179104
TERMINATE: ending in order to see trigger rates with another trigger table. - natasha x2080
Sat Feb 14 21:53:43
talked to Mary C (ops manager tonight) and she agreed that
kevin's plan sounded like a good one.
we'll pause now to perform this test w/ HIGHLUM table. that way
kevin can check the rates before going to sleep. if it goes well
then the owl shift can "relax" and ride-out the rest of the
store.
- Doug
Sat Feb 14 22:08:51
now the fun begins...
natasha tries to bring-in L3 subfarm 9 as per arkady's request.
it doesn't work. things get worse as aces try to recover.
L3 paged. Jeff M here working.
- Doug
-- Sat Feb 14 22:38:09 comment by...Jeff / Ilya -- Before beginning a new run, it was necessary to cleanup L3 because
subfarm 09 had been marked offline for the previous run(s).
Thus, the nodes of this subfarm were not yet running the client
processes necessary to communicate with run control.
All that was necessary to get going again was the clean-up;
Arkadiy's message above indicates this, but not
very emphatically.
Sat Feb 14 22:21:52
Run 179105
Activated at 2004.02.14 22:21:45 - RunControl
Sat Feb 14 22:23:17
Run 179105
ACTIVATE: PHYSIC2_HIGHLUM_2_01[1,405,420] - natasha x2080
Sat Feb 14 22:29:53
Run 179105
ACTIVE: Had the following messages in the error display, when starting the run:
(MLE) b0l3pcom2.fnal.gov:main:10:19:04 PM->Error on L3 node b0l3157 (partition 1) Sat Feb 14
22:19:02 2004 l3_node
in main (l3_node.c:1740)
@L3_EXE_INIT_FAIL: Error initializing filter 0
b0dau32.fnal.gov:csl_mon_send:10:20:17 PM->CSL received a "BAD EVENT" in partition 1.[MLW]
(MLE) b0l3pcom2.fnal.gov:main:10:20:18 PM->Error on L3 node b0l3137 (partition 1) Sat Feb 14
22:20:15 2004 l3_node
in exe_exchange (l3_ana_exe.c:567)
@L3_EXE_TERM: Filter 1 died
(MLE) b0l3pcom2.fnal.gov:main:10:20:18 PM->Error on L3 node b0l3145 (partition 1) Sat Feb 14
22:20:15 2004 l3_node
in exe_exchange (l3_ana_exe.c:567)
@L3_EXE_TERM: Filter 0 died
(MLE) b0l3pcom2.fnal.gov:main:10:20:18 PM->Error on L3 node b0l3145 (partition 1) Sat Feb 14
22:20:15 2004 l3_node
in exe_exchange (l3_ana_exe.c:567)
@L3_EXE_TERM: Filter 1 died
- natasha x2080
Sat Feb 14 22:31:03

- Andrei Loginov
-- Sat Feb 14 22:37:58 comment by...Andrei Loginov --
Event Display pictures - yellow circle consists of xft hits.
Picture #1 - stadard COT view with yellow circle
Picture #2 - close-up view of the arc with "SetDisplayCells" on,
one can see number of xft hits.
You can see the question about this circle here:
here
Time information for XFT is provided by XTC card.
-- Sat Feb 14 22:40:32 comment by...a. -- What I know about coloring scheme for XFT hits is:
green : prompt hit
blue : delayed hit
yellow : both
-- Sat Feb 14 22:44:35 comment by...doug -- the xft SL2 is masked "on" in order to account for the reduced
COT voltge on the inner SL (in the case of SL2, reduced==off)
-- Sun Feb 15 03:15:26 comment by...Cheng-Ju -- Any COT wire masked on in the XFT will show up in the event
display as a small yellow circle at the location of the COT wire.
We have masked on entire XFT SL2 (over 2000 wires) --> yellow
circular band. The little red square box in the center of
a cot cell (above right plot) indicates the location of XFT
segment as found by the XFT. Since all wires in SL2 are
turned on, XFT is finding a segment for every XFT phi bin.
Sat Feb 14 22:33:25
From the MCR elog:
The Plan:
Terminate store when the stack reaches 190E10 or the average
luminosity reaches 10E30 (whichever comes first). After store,
stack to 50E10 and transfer to Recycler. Shot strategy, Protons
265-275E9 per bunch, pbars per guidelines.
- JPM
- Doug
Sat Feb 14 22:36:37
Trigger cross sections for run 179105 (PHYSICS_HIGHLUM_2_01_v1) look good. We should run this table at the beginning of the next store. Thanks for taking the test run early. - KPitts :: (run 179105)
Sat Feb 14 22:40:53
Run 179105
Terminated at 2004.02.14 22:40:42 - RunControl
Sat Feb 14 22:41:08
Run 179105
TERMINATE: ending the trig rates test run. - natasha x2080
Sat Feb 14 22:41:21



- /H.P./ Vadim
Sat Feb 14 22:58:05
Run 179106
Activated at 2004.02.14 22:57:37 - RunControl
Sat Feb 14 22:58:38
Run 179106
ACTIVATE: AAA_CURRENT:PHYSICS_2_01[4,416,424] - natasha x2080
Sat Feb 14 23:03:01
Level3 failed Partition several times complaining about
transition failure on the processor node b0l3157. It
turned out that the node ran out of disk space because of
enourmous number of core.xxxxx files. I cleaned up
the disk, now the node is ok, so is level3. I will
suggest Guillelmo to check why cores accumulate.There is
no immediate danger for other all nodes, as far as I can see,
there is plenty of disk space.
- Ilya
Sat Feb 14 23:14:01
Irina comes-in to update version of pb_hv6 that's monitoring
pisaboxes. When she/steve fixed yesterday's problems they
apparantly downloaded an old version.
- Doug
Sat Feb 14 23:15:18
AG losses have poked-up just above 15k a couple times. both
times they went back below w/i about a minute. Should keep an
eye on it.
- Doug
Sat Feb 14 23:42:57



- /H.P./ Vadim
Sat Feb 14 23:55:45
| Run Number |
Data Type |
Physics Table |
Begin Time |
End Time |
Live Time |
L1 Accepts |
L2 Accepts |
L3 Accepts |
Live Lumi, nb-1 |
GR |
SC |
RC |
|
179104
x2BBA0 |
BEAM |
PHYSICS_2_01 [4,416,424] |
05:40:12 |
21:51:51 |
15:15:13 |
809,302,272 |
11,555,300 |
2,244,223 |
1115.789 |
1 |
1 |
1 |
|
179105
x2BBA1 |
BEAM |
PHYSICS_HIGHLUM_2_01 [1,405,420] |
22:21:45 |
22:40:42 |
00:16:36 |
1,602,986 |
93,762 |
17,241 |
14.033 |
1 |
1 |
1 |
|
179106
x2BBA2 |
BEAM |
PHYSICS_2_01 [4,416,424] |
22:57:37 |
|
00:56:04 |
37,970,620 |
477,844 |
104,423 |
45.385 |
|
|
1 |
| Totals |
|
|
|
23:55:02 |
16:27:55 |
848,875,878 |
12,126,906 |
2,365,887 |
1175.206 |
|
|
|
- End of Shift Report
Sun Feb 15 00:04:53
Shift Summary: * run ; TeV is planning to squeeze store 3231 for a
good while longer.
- at ~20:00, ramp silicon+wire chambers to stby for MCR
to play w/ collimators
- at ~22:00 took a short run (179105) w/ HIGHLUM table
> K Pitts said rates looked ok
> also got L3 subfarm 9 back
- at ~23:00 went back to dflt table
* plan:
- continue taking data w/ dflt table until end of store
> watch abort gap losses
> if collimator gynastics required, don't forget
to reduce voltages as per whiteboard
- between stores:
> checkout, calibrations, L2 torture
- run plan for new store:
> begin w/ HIGHLUM table, DPS disabled
> once lumi < 4E31, switch to dflt table, DPS
enabled
End of Shift Numbers
|
CDF Run II
Runs 179104,05,06(still going)
Delivered Luminosity 445
Acquired Luminosity 374
Efficiency 84%
|
- doug