Experimental Physics and
Industrial Control System

Till Straumann <[email protected]> · Tue, 18 Jan 2005 11:14:18 -0800

Hi.
Eric Norum initiated a discussion about this around the end of 2004/11
resulting in an investigation. My conclusions are appended. I could
identify two major reasons for poor results:
a) epics/rtems frequently uses 'malloc_free_space' which can be
   a very expensive operation under RTEMS (designed to be a diagnostic).
   AFAIK, this has been addressed and a more efficient implementation is
   available with the development version (4.7).
   An unofficial patch is available for 4.6 but my usual workaround
   is to hack
 libCom/osi/os/RTEMS/osdPoolStatus.c:osiSufficientSpaceInPool()
 to always return 1.
b) half/full duplex ethernet mismatch (SSRL policy is to disable
   autonegotiation) resulted in lost packets with tcp performance
   becoming really bad.
After addressing these issues, I could report the following results
for EPICS/RTEMS vs. vxWorks on a Synergy VGM series board (300MHz ppc):
           RTEMS-svgm  vxworks-svgm    RTEMS-svgm  vxworks-svgm
              (1000 channels)              (10000 channels)
search       4.8 Mbps     3.0 Mbps      14.5 Mbps    11.1 Mbps
pend       215.7 It/s   215.8 It/s     211.1 It/s   206.2 It/s
async put    5.5 Mbps     6.2 Mbps      11.0 Mbps    11.5 Mbps
async get   12.0 Mbps    10.6 Mbps      25.6 Mbps    19.7 Mbps
sync  get    0.7 Mbps     0.4 Mbps       0.2 Mbps     0.2 Mbps

If you are unable to obtain similar performance it could be a driver
issue. The RTEMS kernel and the BSD networking stack performance
have been proven be equivalent.

HTH
-- Till
wnolan wrote:

Hi all,
I was curious what sort or rates are being observed using the catime

utility for RTEMS installations. 
On a mvme2307 under RTEMS 4.6.2 running EPICS 3.14.6 I am seeing a

extream slowdown from our VX works installations and was wondering if it

was tyipical. 
--- Test data
Same Hardware

MVME2307

1x OMS VME58

1x Jerger VSC16

1x ACROMAG AVME9440

and process database( about 30 records ), no state programs running.
targeting a calc input field for 10000 connections.
VX-works gives
channel connect test

Elapsed Per Item = 0.00031412 sec, 3183.5 Items per sec, 1.8 Mbps

Search tries per chan - mean=1.010000 std dev=0.099499 min=1.000000

max=2.000000
RTEMS gives
channel connect test

Elapsed Per Item = 0.00358250 sec, 279.1 Items per sec, 0.2 Mbps

Search tries per chan - mean=2.210500 std dev=1.631377 min=1.000000

max=8.000000
-------------
I am also seeing the slowdown on ASYN Get tests
VX works seems to run about 6 Mbps where in the same test RTEMS is

running at about 1 Mbps 
A factor of 10 on channel connect and a factor of 6 on Asyn Get's 
Sync gets are running at about the same rate under both systems , just

under 1 Mbps 
Any help, Pointers or comparative data would be a help,
Bill Nolan
[email protected]

Original message:
Here's an update/synopsis of what I know, so far:
SYSTEM:  RTEMS-4.6.2-svgm / EPICS 3.14.6 (server and client)
NETWORK: 100-TX-full
A) old (rtems-4.6.2) malloc_free_space() kills
   EPICS/channel access performance on fragmented-memory
   systems as reported previously.
B) relatively poor and non-reproducable 'catime' results
   were due to lost packets.
   REASON: our networking folks like to disable autonegotiation
           on a full-duplex network. The driver in this case
           defaults to half-duplex and misses packets if the
           link is heavily used both ways.
           Enforcing full-duplex in the driver yielded much
           better performance (see below).
   SECONDARY EFFECT: In the case of lost packets, TCP response
           was VERY bad because 'fast retransmission' doesn't
           work well.
           Defining TCP_ACK_HACK in
             cpukit/libnetworking/netinet/tcp_input.c
           remedies that somewhat.
RESULTS:
A) tcpecho
'tcpecho' is a simple application that runs on the RTEMS target
and echoes TCP traffic back to the sender (or, optionally discards
it which is the 'unidirectional' mode below).
I measured the time it took to transfer a 16MB file from a
linux-x86 host.
a) unidirectional, driver in half-duplex mode:         ~2s
   (not many packets lost)
b) true echo, driver in half-dup, no TCP_ACK_HACK    ~347s !!
   (packets lost, poor retransmission)
c) true echo, driver in half-dup, with TCP_ACK_HACK   ~45s
   (packets lost, fair retransmission)
d) true echo, driver in full-dup,                      ~2s
B) catime
catime to epics example app. The IOC under test ran with
no calls to malloc_free_space() [rtems only] and with
full-duplex enabled [both, under vxWorks and rtems]. Since
the figures for the different EPICS data types are very similar,
the table presents an average:
           RTEMS-svgm  vxworks-svgm    RTEMS-svgm  vxworks-svgm
              (1000 channels)              (10000 channels)
search       4.8 Mbps     3.0 Mbps      14.5 Mbps    11.1 Mbps
pend       215.7 It/s   215.8 It/s     211.1 It/s   206.2 It/s
async put    5.5 Mbps     6.2 Mbps      11.0 Mbps    11.5 Mbps
async get   12.0 Mbps    10.6 Mbps      25.6 Mbps    19.7 Mbps
sync  get    0.7 Mbps     0.4 Mbps       0.2 Mbps     0.2 Mbps
==> RTEMS doesn't look bad at all !!
-- Till
PS: vxworks suffers from the 'duplex' mismatch problem also.

Here's the tcpecho code:

#include <rtems.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

int tcpecho(int bufsz, int discard)
{
int   rval = 0;
int   arg = 1, i;
int   sd  = -1, sd1 = -1;
char *buf = 0, *bufp;
struct sockaddr_in addr;
	if ( !bufsz )
		bufsz = 10000; /* default */
	if ( (sd = socket(PF_INET, SOCK_STREAM, 0)) < 0 ) {
		perror("socket");
		goto cleanup;
	}
    if ( setsockopt(sd, SOL_SOCKET, SO_KEEPALIVE, &arg, sizeof(arg)) )
		perror("setting SO_KEEPALIVE");
    if ( setsockopt(sd, SOL_SOCKET, SO_REUSEADDR, &arg, sizeof(arg)) )
		perror("setting SO_REUSEADDR");
	memset(&addr,0,sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_port   = htons(4445);
	arg             = sizeof(addr);
	if ( bind(sd, (struct sockaddr*)&addr, arg) < 0 ) {
		perror("bind");
		goto cleanup;
	}
	if ( listen(sd,1) ) {
		perror("listen");
		goto cleanup;
	}
	arg = sizeof(addr);
	if ( (sd1=accept(sd, (struct sockaddr*)&addr, &arg)) < 0 ) {
		perror("accept");
		goto cleanup;
	}
	if ( !(buf = malloc(bufsz)) ) {
		fprintf(stderr,"Unable to allocate buffer; not enough memory\n");
		goto cleanup;
	}
	while ( (arg = read(sd1,buf,bufsz)) > 0 ) {
		i = 0;
		rval++;
		if ( discard )
			continue;
		bufp = buf;
		while ( arg > 0 ) {
			if ( (i=write(sd1, bufp, arg)) < 0 ) {
				perror("write");
				goto cleanup;
			}
			arg  -= i;
			bufp += i;
			rval--;
		}
	}
cleanup:
	if ( sd >= 0 )
		close(sd);
	if ( sd1 >= 0 )
		close(sd1);
	free(buf);
	return rval;
}

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System