EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024  Index 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RTEMS Test Failures on GHA
From: "Johnson, Andrew N. via Core-talk" <core-talk at aps.anl.gov>
To: EPICS core-talk <core-talk at aps.anl.gov>
Date: Sat, 4 Sep 2021 04:17:48 +0000
I’ve been looking a little more closely at what’s different in the initial output from the RTEMS tap files that pass vs failing on GHA. In all cases I’m using epicsTypesTest. The results aren’t earth-shattering, but they might tickle some thoughts.

This was from the last GHA build which passed all its tests on RTEMS-4.10.2, the 7.0.6 tagged release:

Startup.

***** RTEMS Version: rtems-4.10.2(Intel i386/i386 with i387/pc386) *****

***** Initializing network *****
3C509: attach() called.
3C509: isa_probe() looking for a card...
3C509: isa_probe() fail to find a board.
bootpc_init: using network interface 'ne1'
bootpc hw address is 52:54:0:12:34:57
My ip address is 10 .0 .2 .15 
Ignoring BOOTP/DHCP option code 53
Ignoring BOOTP/DHCP option code 51
Subnet mask is 255 .255 .255 .0 
Server ip address is 10 .0 .2 .2 
Gateway ip address is 0 .0 .0 .0 
Log server ip address is 10 .0 .2 .2 
bootpc_adjust_interface: add net route, error=114

***** Setting up file system *****
***** Using compiled in file data *****
-> /iocshTestBadArgIndirect.cmd - ok
-> /iocshTestBadArg.cmd - ok
-> /iocshTestSuccessIndirect.cmd - ok
-> /iocshTestSuccess.cmd - ok
***** Initializing NFS *****
Warning: No timezone information, times will be displayed in UTC.
***** Preparing EPICS application *****
chdir("/")
***** Starting EPICS application *****

The above output is character-wise identical to a test I just ran on my RHEL-7 box using the latest Base commit.

The GHA output from the latest commit looks like this instead:

Startup.

***** RTEMS Version: rtems-4.10.2(Intel i386/i386 with i387/pc386) *****

***** Initializing network *****
3C509: attach() called.
3C509: isa_probe() looking for a card...
3C509: isa_probe() fail to find a board.
bootpc_init: using network interface 'ne1'
bootpc hw address is 52:54:0:12:34:57
BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255
BOOTP call failed -- error 116BOOTP timeout for server 255.255.255.255

#### Test stopped by epicsTypesTest.t after 500 seconds

There are 32 “BOOTP timeout” lines above so my guess is that the BOOTP timeout is about 15 seconds long.

Here’s the GHA output from the RTEMS-4.9.6 build of the latest commit. These tests that we have full results for all pass, but the total time needed to run the tests exceeds the 20 minutes we allow for the testing phase.

Startup.

***** RTEMS Version: rtems-4.9.6(Intel i386/i386 with i387/pc386) *****

***** Initializing network *****
3C509: attach() called.
3C509: isa_probe() looking for a card...
3C509: isa_probe() fail to find a board.
could not find 82559ER device
Probing for NE2000 on PCI (aka. Realtek 8029)
Found 0:4.0
Using port=0xC000 irq=11
network device 'ne2' <52:54:00:12:34:57> initialized on port 0xC000, irq 11
bootpc_init: using network interface 'ne1'
bootpc hw address is 52:54:0:12:34:57
BOOTP timeout for server 0xffffffff
BOOTP call failed -- error 116
***** Setting up file system *****
***** Using compiled in file data *****
-> /iocshTestBadArgIndirect.cmd - ok
-> /iocshTestBadArg.cmd - ok
-> /iocshTestSuccessIndirect.cmd - ok
-> /iocshTestSuccess.cmd - ok
***** Initializing NFS *****
RTEMS-RPCIOD $Release$, Till Straumann, Stanford/SLAC/SSRL 2002, See LICENSE file for licensing info.
RTEMS-NFS $Release$, Till Straumann, Stanford/SLAC/SSRL 2002, See LICENSE file for licensing info.
Warning: No timezone information, times will be displayed in UTC.
***** Preparing EPICS application *****
chdir("/")
***** Starting EPICS application *****


We don’t currently run the tests on GHA for RTEMS-5 builds because the OS is so slow to start. This is the output on my RHEL-7 box, note that RTEMS was built with the new network stack:

i386: isr=0 irr=1
i386: isr=0 irr=1

 initConsole --- Info ---
stdin: fileno: 0, ttyname: /dev/console
stdout: fileno: 1, ttyname: /dev/console
stderr: fileno: 2, ttyname: /dev/console
time set to : 04/14/14 07:30:06.000301343 UTC
Startup.
epicsThreadSetPriority called by non epics thread

***** RTEMS Version: rtems-5.0.0 (Intel i386/Pentium/pc686) *****

***** Initializing network (libbsd, dhcpcd) *****

Getting this far takes about 3 seconds, followed by a 45 second pause before:

***** ifconfig lo0 *****

***** add dhcpcd hook *****

***** Start default network dhcpcd *****

--Info (hpj)-- bsd task prio IRQS: 96  -----

--Info (hpj)-- bsd task prio TIME: 98  -----

 ---- Waiting for DHCP ...

At 48 seconds there’s a 2 second pause here, then:

***** Primary Network interface : em0 *****
all out ---> interface = em0
all out ---> reason = PREINIT
all out ---> pid = 1
all out ---> ifmetric = 201
all out ---> ifwireless = 0
all out ---> ifflags = 4294936643
all out ---> ifmtu = 1500
all out ---> interface_order = em0
all out ---> if_up = false
all out ---> if_down = true

***** Primary Network interface : em0 *****
all out ---> interface = em0
all out ---> reason = CARRIER
all out ---> pid = 1
all out ---> ifmetric = 201
all out ---> ifwireless = 0
all out ---> ifflags = 4294936643
all out ---> ifmtu = 1500
all out ---> interface_order = em0
all out ---> if_up = false
all out ---> if_down = true

***** Primary Network interface : em0 *****
all out ---> interface = em0
all out ---> reason = NOCARRIER
all out ---> pid = 1
all out ---> ifmetric = 201
all out ---> ifwireless = 0
all out ---> ifflags = 34818
all out ---> ifmtu = 1500
all out ---> interface_order = em0
all out ---> if_up = false
all out ---> if_down = true

***** Primary Network interface : em0 *****
all out ---> interface = em0
all out ---> reason = CARRIER
all out ---> pid = 1
all out ---> ifmetric = 201
all out ---> ifwireless = 0
all out ---> ifflags = 4294936643
all out ---> ifmtu = 1500
all out ---> interface_order = em0
all out ---> if_up = false
all out ---> if_down = true

50 seconds in; a short delay here of about 1-2 seconds, then the remaining output over the next 3-4 seconds:

***** Primary Network interface : em0 *****
all out ---> interface = em0
all out ---> reason = BOUND
Interface em0 bounded
all out ---> pid = 1
all out ---> ifmetric = 201
all out ---> ifwireless = 0
all out ---> ifflags = 4294936643
all out ---> ifmtu = 1500
all out ---> interface_order = em0
all out ---> if_up = true
all out ---> if_down = false
all out ---> new_ip_address = 10.0.2.15
all out ---> new_subnet_cidr = 24
all out ---> new_broadcast_address = 10.0.2.255
all out ---> new_network_number = 10.0.2.0
all out ---> new_subnet_mask = 255.255.255.0
all out ---> new_dhcp_lease_time = 86400
all out ---> new_dhcp_message_type = 5
all out ---> new_dhcp_server_identifier = 10.0.2.2
-------------- IFCONFIG -----------------
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 52:54:00:12:34:56
        inet6 fe80::5054:ff:fe12:3456%em0 prefixlen 64 scopeid 0x1
        inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xffffff00
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
-------------- NETSTAT ------------------
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
10.0.2.0/24        link#1             U           em0
10.0.2.15          link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::1                               link#2                        UH          lo0
fe80::%em0/64                     link#1                        U           em0
fe80::5054:ff:fe12:3456%em0       link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0

***** Until now no NTP support in RTEMS 5 with rtems-libbsd *****

***** Ask ntp server once... *****
epicsNtpGetTime: Network is unreachable
***** Can't get time from ntp ...

***** Setting up file system *****
***** Using compiled in file data *****
-> /iocshTestBadArgIndirect.cmd - ok
-> /iocshTestBadArg.cmd - ok
-> /iocshTestSuccessIndirect.cmd - ok
-> /iocshTestSuccess.cmd - ok
***** Initializing NFS *****
 check for time registered , C++ initialization ...
***** Preparing EPICS application *****
chdir("/")
***** Starting EPICS application *****

We reach here at about 55 seconds.


Finally I built the pc686 BSP using the legacy network stack instead of the libbsd one and recompiled EPICS against that. The build failed in the network configuration code:

/local/anj/RTEMS-5.1/rtems-5.1/bin/i386-rtems5-gcc --pipe -B/local/anj/RTEMS-5.1/rtems-5.1/i386-rtems5/pc686/lib/ -specs bsp_specs -qrtems   -fasm   -mtune=pentiumpro -march=pentium       -D_GNU_SOURCE -D_DEFAULT_SOURCE            -DUNIX      -O2 -g -ffunction-sections -fdata-sections   -Wall -Werror-implicit-function-declaration          -D__LINUX_ERRNO_EXTENSIONS__ -DRTEMS_LEGACY_STACK  -I. -I../O.Common -I. -I. -I.. -I../posix -I../../../../include/compiler/gcc -I../../../../include/os/RTEMS -I../../../../include         -c ../rtems_netconfig.c
../rtems_netconfig.c:74:12: error: conflicting types for 'rtems_3c509_driver_attach'
 extern int rtems_3c509_driver_attach (struct rtems_bsdnet_ifconfig *, int);
            ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../rtems_netconfig.c:17:0:
/local/anj/RTEMS-5.1/rtems-5.1/i386-rtems5/pc686/lib/include/bsp.h:98:12: note:previous declaration of 'rtems_3c509_driver_attach' was here
 extern int rtems_3c509_driver_attach(struct rtems_bsdnet_ifconfig *config);
            ^~~~~~~~~~~~~~~~~~~~~~~~~

Oops, something changed… This patch may be wrong, but allows the build to complete (in modules/libcom at least):

diff --git a/modules/libcom/RTEMS/rtems_netconfig.c b/modules/libcom/RTEMS/rtems_netconfig.c
index 38fb6bfce..7ac85b87c 100644
--- a/modules/libcom/RTEMS/rtems_netconfig.c
+++ b/modules/libcom/RTEMS/rtems_netconfig.c
@@ -71,7 +71,7 @@ static struct rtems_bsdnet_ifconfig fxp_driver_config = {
     rtems_fxp_attach,                   /* attach function */
     &ne2k_driver_config,             /* link to next interface */
 };
-extern int rtems_3c509_driver_attach (struct rtems_bsdnet_ifconfig *, int);
+// extern int rtems_3c509_driver_attach (struct rtems_bsdnet_ifconfig *, int);
 static struct rtems_bsdnet_ifconfig e3c509_driver_config = {
     "ep0",                              /* name */
     rtems_3c509_driver_attach,          /* attach function */

With that the tests connect to the ne2k interface and get an IP address, but they hang before actually running the test code:

i386: isr=0 irr=1
i386: isr=0 irr=1

 initConsole --- Info ---
stdin: fileno: 0, ttyname: /dev/console
stdout: fileno: 1, ttyname: /dev/console
stderr: fileno: 2, ttyname: /dev/console
time set to : 04/14/14 07:30:06.000165660 UTC
Startup.
epicsThreadSetPriority called by non epics thread

***** RTEMS Version: rtems-5.1.0 (Intel i386/Pentium/pc686) *****

***** Initializing network (Legacy Stack)  *****
3C509: attach() called.
3C509: isa_probe() looking for a card...
3C509: isa_probe() fail to find a board.
could not find 82559ER device
Probing for NE2000 on PCI (aka. Realtek 8029)
Found 0:3.0
Using port=0xc000 irq=11
network device 'ne2' <52:54:00:12:34:56> initialized on port 0xc000, irq 11
bootpc_init: using network interface 'ne1'
bootpc hw address is 52:54:0:12:34:56
My ip address is 10 .0 .2 .15 
Ignoring BOOTP/DHCP option code 53
Ignoring BOOTP/DHCP option code 51
Subnet mask is 255 .255 .255 .0 
Server ip address is 10 .0 .2 .2 
Gateway ip address is 0 .0 .0 .0 
Log server ip address is 10 .0 .2 .2 
bootpc_adjust_interface: add net route, error=114

***** Network Status  *****
************ INTERFACE STATISTICS ************
***** lo0 *****
Address:127.0.0.1       Net mask:255.0.0.0       
Flags: Up Loopback Running Multicast
Send queue limit:50   length:0    Dropped:0       
***** ne1 *****
Ethernet Address: 52:54:00:12:34:56
Address:10.0.2.15       Broadcast Address:10.0.2.255      Net mask:255.255.255.0   
Flags: Up Broadcast Running Simplex Multicast
Send queue limit:50   length:0    Dropped:0       
    Received packets: 1        Transmitted packets: 2       
        Receive acks: 1              Transmit acks: 2       
     Packet overruns: 0               Frame errors: 0       
          CRC errors: 0             Missed packets: 0       
          Interrupts: 2       

************ MBUF STATISTICS ************
mbufs:1440    clusters: 175    free: 175
drops:   0       waits:   0  drains:   0
      free:1439          data:1           header:0           socket:0       
       pcb:0           rtable:0           htable:0           atable:0       
    soname:0           soopts:0           ftable:0           rights:0       
    ifaddr:0          control:0          oobdata:0       

Destination     Gateway/Mask/Hw    Flags     Refs     Use Expire Interface
10.0.2.0        255.255.255.0      U           0        0      1 ne1
127.0.0.1       127.0.0.1          UH          0        0      0 lo0
************ IP Statistics ************
             total packets received           2
 datagrams delivered to upper level           2
    total ip packets generated here           1

************ ICMP Statistics ************

************ UDP Statistics ************
                total input packets           2
     of above, arrived as broadcast           1
               total output packets           1

************ TCP Statistics ************

qemu: terminating on signal 2

With that I’m done for the long weekend.

- Andrew


-- 
Complexity comes for free, simplicity you have to work for.


Navigate by Date:
Prev: epics-7.0 » linux32 - Build # 360 - Unstable! APS Jenkins via Core-talk
Next: Build failed: EPICS Base 7 base-7.0-386 AppVeyor via Core-talk
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024 
Navigate by Thread:
Prev: epics-7.0 » linux32 - Build # 363 - Fixed! APS Jenkins via Core-talk
Next: Build failed: EPICS Base 7 base-7.0-386 AppVeyor via Core-talk
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  <20212022  2023  2024 
ANJ, 01 Oct 2021 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·