CDF Flash RAM Disaster Recovery

CDF VxWorks Flash RAM Disaster Recovery

Described below is the procedure that can be followed in case the flash ram in a VME processor becomes corrupted.

Each processor contains two flash rams. The boards are normally configured to boot from the 4MB "soldered" flash ram, which contains a VxWorks boot image. The 1MB "socketed" flash ram contains PPCbug, a basic CPU monitor program from Motorola.

If the soldered flash becomes corrupted, the strategy is to boot the board to PPCbug, and use various monitor commands to install a basic VxWorks boot image into the 4MB flash ram. Then the board is booted to the basic VxWorks kernel, and the full CDF flash ram image installed.

To do this, perform the following steps:

Connect to the appropriate vxcom (after setup fer) port for that processor and verify that it functions correctly. If it doesn't work, do not go any further until a terminal connection to the serial port is available.
PPCbug on older MVME2301s can only download files on a 10MBit ethernet link, it WILL NOT WORK connected to a fast ethernet link. Hence the first step is to connect it to a slow ethernet link:
- Obtain the emergency 10MBit hub and ethernet cross cable from the third floor computer room. See JJ Schmidt.
- Go the the counting room or detector rack with the appropriate ethernet patch panel.
- Identify the correct cable from the processor of interest.
- Unplug this cable and plug it into the hub.
- Plug one end of the cross cable into the hub, and the other end into the just vacated patch panel socket. Also make sure the power cord for the hub is installed.
- This 10MBit ethernet step is generally not necessary for most MVME processors.
  It is not necessary on all of the 5500's, 2400's, 2304's, and newer 2301's. It is necessary for the first batch of 2301's, the issue is that the version of PPCbug in them cannot deal with being connected to a fast ethernet-capable port. It is possible to update PPCbug but this can only be done from PPCbug, it cannot easily be done from VxWorks; it is in a different flash ram than the VxWorks kernel.
- In general, the MVME2301s in the collision hall do not have this problem. (modulo latter day swapping)
- It is possible to load a newer version of PPCbug (5.2) to eliminate this problem − see instructions here − and we recommend that this be done whenever feasible. If the MVME2301 has PPCbug version 4.1, or later, the 100baseT network connection will work fine. As MVME2301s are swapped out, they should be reprogrammed, and eventually this 10baseT procecdure will no longer be necessary at all.
Configure the VME processor to boot from the socketed flash ram:
- Power off the crate.
- Pull out the processor a few inches from the crate. It normally is not necessary to unplug any cables.
- Find the J15 set of pins at the bottom front of the board.
- Move the red jumper so it stradles the rear two pins of J15.
- Reinsert the processor into the crate. For 2400 series boards the jumper is labeled J8.
Power on the crate. Press the "reset" button on the CPU.
You should then see the PPC1-Bug prompt.
Issue the "TIME" command. If the response is "RTC clock not running", or something besides the time, then issue the command
```
SET mmddyyhhmm
```
substituting the correct date and time.

Issue the "niot" command and answer the questions to set the network configuration:

PPC1-Bug>niot
Controller LUN =00?
Device LUN     =00?
Node Control Memory Address =07F9E000?
Client IP Address      =131.225.81.115? 131.225.237.142
Server IP Address      =131.225.80.191? 131.225.237.20
Subnet IP Address Mask =255.255.0.0?
Broadcast IP Address   =255.255.255.255?
Gateway IP Address     =131.225.87.200? 131.225.237.200
Boot File Name ("NULL" for None)     =? .

Update Non-Volatile RAM (Y/N)? y

Substitute the correct IP address for the processor for "Client IP Address" above.

Issue the "niop" command to fetch the VxWorks boot image:
```
PPC1-Bug>niop
Controller LUN =00?
Device LUN     =00?
Get/Put        =G? g
File Name      =? vxWorks-2301.st_rom.bin
Memory Address =00004000?
Length         =00000000?
Byte Offset    =00000000?

Bytes Received =&310508, Bytes Loaded =&310508
Bytes/Second   =&103502, Elapsed Time =3 Second(s)
```
If the board is a 2304, use "vxWorks-2305.st_rom.bin" for the "File Name" above; if the board is a 2400, use "vxWorks_romCompress-2400.bin". The files are located in the /cdf/onln/data/vxworks/tftpboot directory, accessible from the online linux cluster. Alternatively, a very minimal VxWorks boot image can be found in the boot.bin (and similarly named) files.

Load the boot image into the soldered flash ram:

PPC1-Bug>pflash 4000:FFF00 ff000100
Source Starting/Ending Addresses      =00004000/00103EFF
Destination Starting/Ending Addresses =FF000100/FF0FFFFF
Number of Effective Bytes             =000FFF00 (&1048320)

Program FLASH Memory (Y/N)? y
FLASH Memory Programming Complete

Power down the crate, and move the J15 jumper back to the front two pins of J15.
Remove the 10 MBit hub and RETURN TO THE THIRD FLOOR CABINET. Restore the original ethernet connection.
Power on the crate. You should see the VxWorks boot process beginning. Hit any character to get the VxWorks boot prompt.
Fill in the nominal boot parameters. Use boot/front-end/startup.cdf for the startup script.
Reboot the board to the VxWorks shell prompt.
Now install the full CDF flash ram image using the normal procedure. Set the correct startup script.
Reboot again, and you are done. If this procedure does not work for some reason, it will be necessary to take the board to prep. They will have to manually install a set of socketed flash rams with PPC bug, and then copy PPC bug to the soldered flash.

Flashing MVME5500s from MOTload

The MVME5500 comes with the MOTload package installed at the factory, rather than PPCbug. Lots of informative messages can be accessed from the MOTload prompt by typing help. Here are some useful MOTload commands:

# setting time in format mmddyyhhmmss
set -t030304134500

# copy MOTLoad from /dev/flash1 (8MB) to /dev/flash0 (32MB) (not normally necessary)
dla = malloc -x100000
bmb -a0xfff00000 -bffffffff -cdla
flashProgram -o0x1f00000 -sdla -v

# copy MOTLoad from /dev/flash0 (32MB) to /dev/flash1 (8MB) (not normally necessary)

dla = malloc -x100000
bmb -a0xf3f00000 -bf3ffffff -cdla
flashProgram -d/dev/flash1 -o0x700000 -sdla -v


# general commands

MVME5500> time
MON MAR 22 16:52:42.00 2004

MVME5500> netShow
Interface                  EAddress      Status   Speed    Duplex
/dev/enet0                 0001AF0BD569  Up       100MBS   Full
/dev/enet1                 0001AF0BD56A  Down     Unknown  Unknown

Initially (from the factory) both /dev/flash0 and /dev/flash1 are installed with MOTload. The VxWorks kernel should be written to /dev/flash0 via tftp (trivial file transfer protocol). You need to use /dev/flash0 since the kernel expects to reside there. At CDF, b0server00, b0server01 and b0server02 have tftp enabled, with the /cdf/onln/data/vxworks/tftpboot directory accessible to remote tftp clients. To load the minimal boot image:

tftpGet -c131.225.237.5 -g131.225.237.200 -s131.225.237.20 -d/dev/enet0 -fboot-5500.bin
flashProgram -o0x1f00100 -n0xfff00 -v

To load the full kernel into Flash RAM: (recommended)

tftpGet -c131.225.237.5 -g131.225.237.200 -s131.225.237.20 -d/dev/enet0 -fvxWorks_romCompress-5500.bin
flashProgram -o0x1f00100 -n0xfff00 -v

In this example, 131.225.237.5 (b0diag01) is the target client running MOTload, 131.225.237.200 is the subnet gateway, and 131.225.237.20 is one of the tftp servers (b0vxboot02, also known as b0server02 on the 236 subnet). The device /dev/enet0 is the 10/100T ethernet interface. The file boot-5500.bin is specific to the MVME5500. After this binary file is loaded, unplug the MVME5500 and make sure the jumper J8 is installed between pins 1 and 2 (for flashRam0). If the jumper was already on flashRam0, then press and hold the front panel reset button for three seconds, until the BFL front panel LED lights up. To install the full CDF VxWorks kernal in flashRam0, replace boot-5500.bin file name with vxWorks_romCompress.bin.5500, as noted above.* Unlike the 2000 series MVME processors, you do not have to go through the intermediary step of loading boot-5500.bin first.

If you loaded the minimal boot image boot-5500.bin , then at the [VxWorks Boot]: prompt, you can type c to change the boot parameters according the the CDF Boot Parameters web page. If you loaded the full kernel, type bootChange to alter the boot parameters.

* NB 1: The Trigger Supervisor crate b0tsi00 uses both the 100BaseT and the Gigabit Ethernet spigots, and requires a kernel with gigabit ethernet enabled: vxWorks_romCompress.bin.gigE.5500

v* NB 2: When loading a kernel from VxWorks, you must choose the vxWorks.romCompress file without the .bin suffix. The VxWorks based flash loader cannot handle the binary file.

Last modified by W.Badgett 2006.04.24