Experimental Physics and
Industrial Control System

Hello all,

at NSLS-II, we have several dozen MVME3100 VME crates running RTEMS with EPICS driver for various diagnostics and MRF timing hardware. The entire setup was handily created quite some time ago by combining RTEMS 4.9 with Till Straumann's Cexp and GeSys to provide convenient booting and debugging capabilities. Results of that combination are Michael Davidsaver's RTEMS Source Builder freeze, and Debian packages rtems-cexp and rtems-gesys available in EPICS Debian packaging on GitHub.

Quite a while ago I have attempted an effort to update the boot image to a more recent version of RTEMS. Alas, results were not satisfactory. The process I went through is described in detail in my post on RTEMS mailing list. Despite the valuable feedback, I didn't procure any good solution for booting issues with the new image.

Recently the project was rekindled with a new effort when Tomin, our junior Controls member, went to revise the building procedure. One realization made was that I happened to overlook building libbspExt which is required for the board we use, MVME3100 (duh). Addressing that cleared some errors, but the main behavior of the system remained the same - the startup script cannot be located after the system boots, and the Cexp shell is suspiciously laggy.

In short, the current procedure to assemble RTEMS+Cexp+GeSys did not change in terms of configuring the source:

BUILD RTEMS:
../configure --enable-networking --enable-cxx --enable-rdbg --enable-posix --prefix=$HOME/src/RTEMS/4.10 --target=powerpc-rtems4.10 --enable-rtemsbsp=mvme3100

BUILD CEXP:
../cexpsh/configure --enable-std-rtems-installdirs --with-rtems-top=$HOME/src/RTEMS/4.10 --host=powerpc-rtems4.10 --enable-rtemsbsp=mvme3100

BUILD GESYS:
../rtems-gesys/configure --enable-std-rtems-installdirs --with-rtems-top=$HOME/src/RTEMS/4.10 --host=powerpc-rtems4.10 --enable-rtemsbsp=mvme3100

For libbspExt, we tried 1.5 (manually with RTEMS_MAKEFILE_PATH) and 1.6 (via ./configure) without notable difference. We have tried building RTEMS both via rtems-source-builder and manually. We have tried 4.10 and 4.11. Depending on a particular combination, the result is either the startup script not being found, or there is an unrecoverable exception unless GeSys initialization is not skipped. One thing to note is that if I use our existing 4.9 boot image, everything works in exactly the same setup.

Same as with my previous attempts, I can skip the initialization and perform manual NFS mounting from Cexp to load the script directly. It doesn't work nicely from there, but at least I can see that the network works. Cexp shell remains laggy at all times.

I would like to ask for some input from those who happen to run similar setups (RTEMS+Cexp+GeSys on MVME3100), and for any ideas on what we might be missing. Given that it "almost works", I have a feeling there is just some bit which we overlooked - possibly an option for RTEMS build, or some change in DHCP setup, etc. Perhaps someone has a comprehensive procedure of getting the RTEMS+Cexp+GeSys combo built?

Below are boot processes for 4.10 (bad) and 4.9 (good) images for reference.

Thanks,

Anton.

4.10.2 boot (doesn't work):

Copyright(C)2008-2013,Emerson Network Power-Embedded Computing,Inc.

All Rights Reserved
Copyright Motorola Inc. 1999-2007, All Rights Reserved
MOTLoad RTOS Version 2.0,  PAL Version 1.2 RM04
Wed Nov  6 17:17:43 MST 2013

MPU-Type             =MPC8540
MPU-Int Clock Speed  =666MHz
MPU-CCB Clock Speed  =333MHz
MPU-DDR Clock Speed  =166MHz
MPU-PCI Clock Speed  =66MHz, PCI, 64-bit
MPU-Int Cache(L2) Enabled, 256KB, L2CTL =A8000300

Reset/Boot Vector    =Flash0

Local Memory Found   =10000000 (&268435456)
User Download Buffer =0159D000:0179CFFF

MVME3100> bootmem=malloc 0x500000                                             
return = 0193D000 (&26464256)
errno  = 00000000
MVME3100> netBoot -d/dev/enet0 -v -abootmem -p -h -u
Network Loading from: /dev/enet0

Locating BOOTP Server... Found It!

Client IP Address      = 10.0.1.104
Server IP Address      = 10.0.1.1
Gateway IP Address     = 0.0.0.0
Subnet IP Address Mask = 255.255.255.0
Boot File Name         = rtems-4.9/rtems.boot
Load Address           = 0193D000
Buffer Size = ffffffff 

Network Boot File Load Start - Press <ESC> to Bypass, <SPC> to Continue

Network Boot File Load in Progress - Press <CTRL-C> to Abort[]>...<

Bytes Received =&2870020, Bytes Loaded =&2870020
Bytes/Second   =&478336, Elapsed Time =6 Second(s)

Execution Halted on Request
MV-----------------------------------------
Welcome to rtems-4.10.2(PowerPC/Generic (no FPU)/mvme3100)
BSP: mvme3100, CVS Release ($Name: rtems-4-10-2 $)
CPU 0x8020 - rev 0x20
Additionnal boot options are H
Initial system stack at 12D7DF8
Software IRQ stack starts at 12D7F00 with size 16384
Going to start PCI buses scanning and initialization
Board Type: MVME3100-1152 (S/N E10EA0D)
External (=PCI Bus) Clock Freq   :   66666666 Hz
Core Complex Bus (CCB) Clock Freq:  333333330 Hz
CPU Clock Freq:                     133333332 Hz
Ethernet 0                  EC:9E:CD:0E:16:5B
Ethernet 1                  EC:9E:CD:0E:16:5C
Ethernet 2                  EC:9E:CD:0E:16:5D
Number of PCI buses found is : 2
BUS:SLOT:FUN  VENDOR-DEV_ID: COMMAND STATUS BASE_ADDR0 BASE_ADDR1 IRQ_PIN -> IRQ_LINE
  0:0x00:0    0x1057-0x0008:  0x0006 0x20B0 0x80000000 0x00000000       0 ->   0 (=0x00)
  0:0x11:0    0x10E3-0x0148:  0x0146 0x02B0 0x80100004 0x00000000       1 ->   0 (=0x00)
  0:0x12:0    0x10B5-0x6520:  0x0147 0x02B0 0x00000000 0x00000000       0 ->   0 (=0x00)
  0:0x14:0    0x1095-0x3124:  0x01C7 0x02B0 0x80300004 0x00000000       1 ->   2 (=0x02)
  1:0x00:0    0x10B5-0x9030:  0x0142 0x0290 0x80200000 0x00000000       1 ->   4 (=0x04)
Memory:                             268435456 bytes
Cleared PCI errors: pci_stat was 0x20B0
OpenPIC Version 1.2 (1 CPUs and 56 IRQ sources) at 0xE1040000
OpenPIC Vendor 0 (Unknown), Device 0 (Unknown), Stepping 0
OpenPIC timer frequency is 41666666 Hz
MSR is 0x00001000, TCR 0x04000000
-----------------------------------------
Exit from bspstart
Tundra Tsi148 PCI-VME bridge detected at 0x80100000, IRQ 0
Tsi148 Outbound Ports:
Port  VME-Addr   Size       PCI-Adrs   Mode:
0:    0x20000000 0x0E000000 0xC0000000 A32, SUP, D32, SCT
1:    0x00000000 0x00FF0000 0xCF000000 A24, SUP, D32, SCT
2:    0x00000000 0x00010000 0xCFFF0000 A16, SUP, D32, SCT
7:    0x00000000 0x01000000 0xCE000000 CSR, SUP, D32, SCT
Tsi148 Inbound Ports:
Port  VME-Addr   Size       PCI-Adrs   Mode:
0:    0xC0000000 0x10000000 0x00000000 A32, PGM, DAT, SUP, USR, MBLT, BLT
vmeTsi148 IRQ manager: looking for registers on VME...
Trying to find CSR on VME...
vmeTsi148 - IRQ manager using VME CSR to flush FIFO
6 I2C devices registered
Registering /dev/console as minor 0 (==/dev/ttyS0)
libbspExt - Warning: it seems that MCP support is not available on your CPU
                     or not implemented by your board. Address probing must
                     be performed in polling mode with interrupts disabled
Welcome to RTEMS 4.10.2 GeSys
This system  was built on 20180710EDT15:29:08
Installing TIOCGWINSZ line discipline: ok.
To skip initialization, press a key now...
bootpc_init: using network interface 'tse1'
bootpc hw address is ec:9e:cd:e:16:5b
My ip address is 10 .0 .1 .104 
Time Server is 10 .0 .1 .1 
Domain name is cs.nsls2.local
Boot file is rtems-4.9/rtems.boot
Command line is IOCNAME=a_timing2
Subnet mask is 255 .255 .255 .0 
Server ip address is 10 .0 .1 .1 
Gateway ip address is 0 .0 .0 .0 
Log server ip address is 10 .0 .1 .1 
bootpc_adjust_interface: add net route, error=114
Trying to synchronize NTP...OK
Trying symfile 'BUILTIN', system script 'st.sys'
'st.sys':
opening script file: No such file or directory
Type 'cexpsh.help()' for help (no quotes)
Cexp>

4.9 image (works):

Copyright(C)2008-2013,Emerson Network Power-Embedded Computing,Inc.
All Rights Reserved
Copyright Motorola Inc. 1999-2007, All Rights Reserved
MOTLoad RTOS Version 2.0,  PAL Version 1.2 RM04
Wed Nov  6 17:17:43 MST 2013

MPU-Type             =MPC8540
MPU-Int Clock Speed  =666MHz
MPU-CCB Clock Speed  =333MHz
MPU-DDR Clock Speed  =166MHz
MPU-PCI Clock Speed  =66MHz, PCI, 64-bit
MPU-Int Cache(L2) Enabled, 256KB, L2CTL =A8000300

Reset/Boot Vector    =Flash0

Local Memory Found   =10000000 (&268435456)
User Download Buffer =0159D000:0179CFFF

MVME3100> bootmem=malloc 0x500000                                             
return = 0193D000 (&26464256)
errno  = 00000000
MVME3100> netBoot -d/dev/enet0 -v -abootmem -p -h -u
Network Loading from: /dev/enet0

Locating BOOTP Server... Found It!

Client IP Address      = 10.0.1.104
Server IP Address      = 10.0.1.1
Gateway IP Address     = 0.0.0.0
Subnet IP Address Mask = 255.255.255.0
Boot File Name         = rtems-4.9/rtems.boot
Load Address           = 0193D000
Buffer Size = ffffffff 

Network Boot File Load Start - Press <ESC> to Bypass, <SPC> to Continue

Network Boot File Load in Progress - Press <CTRL-C> to Abort[]>...<

Bytes Received =&2200688, Bytes Loaded =&2200688
Bytes/Second   =&550172, Elapsed Time =4 Second(s)

Execution Halted on Request
MV-----------------------------------------
Welcome to rtems-4.9.6(PowerPC/Generic (no FPU)/mvme3100)
BSP: mvme3100, CVS Release ($Name: rtems-4-9-6 $)
CPU 0x8020 - rev 0x20
Additionnal boot options are H
Initial system stack at 241AA8
Software IRQ stack starts at 241BA0 with size 8192
Going to start PCI buses scanning and initialization
PCI A Status: 09 : 66MHz PCI 64-bit
PCI B Status: B8 :  33MHz PCI 64-bit
    PMC1: ready PMC2: ready  3.3V
PCI C Status: 09 : 66MHz PCI 64-bit
PMC1: empty
PMC2: empty
PMCSpan: empty
Build Date: Thu Oct 24 16:11:26 UTC 2013
Board Type: MVME3100-1152 (S/N E10EA0D)
External (=PCI Bus) Clock Freq   :   66666666 Hz
Core Complex Bus (CCB) Clock Freq:  333333330 Hz
CPU Clock Freq:                     133333332 Hz
Ethernet 0                  EC:9E:CD:0E:16:5B
Ethernet 1                  EC:9E:CD:0E:16:5C
Ethernet 2                  EC:9E:CD:0E:16:5D
Number of PCI buses found is : 2
BUS:SLOT:FUN  VENDOR-DEV_ID: COMMAND STATUS BASE_ADDR0 BASE_ADDR1 IRQ_PIN -> IRQ_LINE
  0:0x00:0    0x1057-0x0008:  0x0006 0x20B0 0x80000000 0x00000000       0 ->   0 (=0x00)
  0:0x11:0    0x10E3-0x0148:  0x0146 0x02B0 0x80100004 0x00000000       1 ->   0 (=0x00)
  0:0x12:0    0x10B5-0x6520:  0x0147 0x02B0 0x00000000 0x00000000       0 ->   0 (=0x00)
  0:0x14:0    0x1095-0x3124:  0x01C7 0x02B0 0x80300004 0x00000000       1 ->   2 (=0x02)
  1:0x00:0    0x10B5-0x9030:  0x0142 0x0290 0x80200000 0x00000000       1 ->   4 (=0x04)
Memory:                             268435456 bytes
Cleared PCI errors: pci_stat was 0x20B0
Configuration.work_space_size = 500000
OpenPIC Version 1.2 (1 CPUs and 56 IRQ sources) at 0xE1040000
OpenPIC Vendor 0 (Unknown), Device 0 (Unknown), Stepping 0
OpenPIC timer frequency is 41666666 Hz
MSR is 0x00001000, TCR 0x04000000
-----------------------------------------
Exit from bspstart
Tundra Tsi148 PCI-VME bridge detected at 0x80100000, IRQ 0
Tsi148 Outbound Ports:
Port  VME-Addr   Size       PCI-Adrs   Mode:
0:    0x20000000 0x0E000000 0xC0000000 A32, SUP, D32, SCT
1:    0x00000000 0x00FF0000 0xCF000000 A24, SUP, D32, SCT
2:    0x00000000 0x00010000 0xCFFF0000 A16, SUP, D32, SCT
7:    0x00000000 0x01000000 0xCE000000 CSR, SUP, D32, SCT
Tsi148 Inbound Ports:
Port  VME-Addr   Size       PCI-Adrs   Mode:
0:    0xC0000000 0x10000000 0x00000000 A32, PGM, DAT, SUP, USR, MBLT, BLT
vmeTsi148 IRQ manager: looking for registers on VME...
Trying to find CSR on VME...
vmeTsi148 - IRQ manager using VME CSR to flush FIFO
6 I2C devices registered
Registering /dev/console as minor 0 (==/dev/ttyS0)
libbspExt - Warning: it seems that MCP support is not available on your CPU
                     or not implemented by your board. Address probing must
                     be performed in polling mode with interrupts disabled
Welcome to RTEMS GeSys
This system $Name: GeSys_2_4 $ was built on 20131114UTC13:46:29
$Id: init.c,v 1.43 2008/03/22 20:36:13 guest Exp $
Installing TIOCGWINSZ line discipline: ok.
To skip initialization, press a key now...
bootpc_init: using network interface 'tse1'
bootpc hw address is ec:9e:cd:e:16:5b
My ip address is 10 .0 .1 .104 
Time Server is 10 .0 .1 .1 
Domain name is cs.nsls2.local
Boot file is rtems-4.9/rtems.boot
Command line is IOCNAME=a_timing2
Subnet mask is 255 .255 .255 .0 
Server ip address is 10 .0 .1 .1 
Gateway ip address is 0 .0 .0 .0 
Log server ip address is 10 .0 .1 .1 
bootpc_adjust_interface: add net route, error=114
Trying to synchronize NTP...OK
yr:2018 m:7 d:13 hr:17 min:5 sec:14
Change Dir to 'rtems-4.9/'
Trying symfile 'BUILTIN', system script 'st.sys'
Type 'cexpsh.help()' for help (no quotes)
'st.sys':

<startup script loading commences>