Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
<== Date ==> <== Thread ==>

Subject: Re: CA Connect rates under RTEMS
From: Till Straumann <strauman@slac.stanford.edu>
To: wnolan@bnl.gov
Cc: tech-talk@aps.anl.gov
Date: Tue, 18 Jan 2005 11:14:18 -0800
Hi.

Eric Norum initiated a discussion about this around the end of 2004/11
resulting in an investigation. My conclusions are appended. I could
identify two major reasons for poor results:

a) epics/rtems frequently uses 'malloc_free_space' which can be
   a very expensive operation under RTEMS (designed to be a diagnostic).
   AFAIK, this has been addressed and a more efficient implementation is
   available with the development version (4.7).
   An unofficial patch is available for 4.6 but my usual workaround
   is to hack

libCom/osi/os/RTEMS/osdPoolStatus.c:osiSufficientSpaceInPool()

to always return 1.

b) half/full duplex ethernet mismatch (SSRL policy is to disable
   autonegotiation) resulted in lost packets with tcp performance
   becoming really bad.

After addressing these issues, I could report the following results
for EPICS/RTEMS vs. vxWorks on a Synergy VGM series board (300MHz ppc):

           RTEMS-svgm  vxworks-svgm    RTEMS-svgm  vxworks-svgm
              (1000 channels)              (10000 channels)
search       4.8 Mbps     3.0 Mbps      14.5 Mbps    11.1 Mbps
pend       215.7 It/s   215.8 It/s     211.1 It/s   206.2 It/s
async put    5.5 Mbps     6.2 Mbps      11.0 Mbps    11.5 Mbps
async get   12.0 Mbps    10.6 Mbps      25.6 Mbps    19.7 Mbps
sync  get    0.7 Mbps     0.4 Mbps       0.2 Mbps     0.2 Mbps


If you are unable to obtain similar performance it could be a driver issue. The RTEMS kernel and the BSD networking stack performance have been proven be equivalent.


HTH -- Till

wnolan wrote:
Hi all, I was curious what sort or rates are being observed using the catime
utility for RTEMS installations.


On a mvme2307 under RTEMS 4.6.2 running EPICS 3.14.6 I am seeing a
extream slowdown from our VX works installations and was wondering if it
was tyipical.


--- Test data Same Hardware
MVME2307
1x OMS VME58
1x Jerger VSC16
1x ACROMAG AVME9440
and process database( about 30 records ), no state programs running. targeting a calc input field for 10000 connections. VX-works gives channel connect test
Elapsed Per Item = 0.00031412 sec, 3183.5 Items per sec, 1.8 Mbps
Search tries per chan - mean=1.010000 std dev=0.099499 min=1.000000
max=2.000000


RTEMS gives channel connect test
Elapsed Per Item = 0.00358250 sec, 279.1 Items per sec, 0.2 Mbps
Search tries per chan - mean=2.210500 std dev=1.631377 min=1.000000
max=8.000000


-------------

I am also seeing the slowdown on ASYN Get tests VX works seems to run about 6 Mbps where in the same test RTEMS is
running at about 1 Mbps


A factor of 10 on channel connect and a factor of 6 on Asyn Get's

Sync gets are running at about the same rate under both systems , just
under 1 Mbps


Any help, Pointers or comparative data would be a help,

Bill Nolan wnolan@bnl.gov






Original message:


Here's an update/synopsis of what I know, so far:

SYSTEM:  RTEMS-4.6.2-svgm / EPICS 3.14.6 (server and client)
NETWORK: 100-TX-full

A) old (rtems-4.6.2) malloc_free_space() kills
   EPICS/channel access performance on fragmented-memory
   systems as reported previously.

B) relatively poor and non-reproducable 'catime' results
   were due to lost packets.

   REASON: our networking folks like to disable autonegotiation
           on a full-duplex network. The driver in this case
           defaults to half-duplex and misses packets if the
           link is heavily used both ways.

           Enforcing full-duplex in the driver yielded much
           better performance (see below).

   SECONDARY EFFECT: In the case of lost packets, TCP response
           was VERY bad because 'fast retransmission' doesn't
           work well.
           Defining TCP_ACK_HACK in
             cpukit/libnetworking/netinet/tcp_input.c
           remedies that somewhat.

RESULTS:

A) tcpecho

'tcpecho' is a simple application that runs on the RTEMS target
and echoes TCP traffic back to the sender (or, optionally discards
it which is the 'unidirectional' mode below).
I measured the time it took to transfer a 16MB file from a
linux-x86 host.

a) unidirectional, driver in half-duplex mode:         ~2s
   (not many packets lost)
b) true echo, driver in half-dup, no TCP_ACK_HACK    ~347s !!
   (packets lost, poor retransmission)
c) true echo, driver in half-dup, with TCP_ACK_HACK   ~45s
   (packets lost, fair retransmission)
d) true echo, driver in full-dup,                      ~2s

B) catime

catime to epics example app. The IOC under test ran with
no calls to malloc_free_space() [rtems only] and with
full-duplex enabled [both, under vxWorks and rtems]. Since
the figures for the different EPICS data types are very similar,
the table presents an average:

           RTEMS-svgm  vxworks-svgm    RTEMS-svgm  vxworks-svgm
              (1000 channels)              (10000 channels)
search       4.8 Mbps     3.0 Mbps      14.5 Mbps    11.1 Mbps
pend       215.7 It/s   215.8 It/s     211.1 It/s   206.2 It/s
async put    5.5 Mbps     6.2 Mbps      11.0 Mbps    11.5 Mbps
async get   12.0 Mbps    10.6 Mbps      25.6 Mbps    19.7 Mbps
sync  get    0.7 Mbps     0.4 Mbps       0.2 Mbps     0.2 Mbps

==> RTEMS doesn't look bad at all !!

-- Till

PS: vxworks suffers from the 'duplex' mismatch problem also.


Here's the tcpecho code:



#include <rtems.h>


#include <sys/socket.h>
#include <netinet/in.h>

#include <string.h>
#include <stdlib.h>
#include <stdio.h>


int tcpecho(int bufsz, int discard) { int rval = 0; int arg = 1, i; int sd = -1, sd1 = -1; char *buf = 0, *bufp; struct sockaddr_in addr;

	if ( !bufsz )
		bufsz = 10000; /* default */

	if ( (sd = socket(PF_INET, SOCK_STREAM, 0)) < 0 ) {
		perror("socket");
		goto cleanup;
	}
    if ( setsockopt(sd, SOL_SOCKET, SO_KEEPALIVE, &arg, sizeof(arg)) )
		perror("setting SO_KEEPALIVE");
    if ( setsockopt(sd, SOL_SOCKET, SO_REUSEADDR, &arg, sizeof(arg)) )
		perror("setting SO_REUSEADDR");

	memset(&addr,0,sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_port   = htons(4445);
	arg             = sizeof(addr);

	if ( bind(sd, (struct sockaddr*)&addr, arg) < 0 ) {
		perror("bind");
		goto cleanup;
	}

	if ( listen(sd,1) ) {
		perror("listen");
		goto cleanup;
	}

	arg = sizeof(addr);
	if ( (sd1=accept(sd, (struct sockaddr*)&addr, &arg)) < 0 ) {
		perror("accept");
		goto cleanup;
	}

	if ( !(buf = malloc(bufsz)) ) {
		fprintf(stderr,"Unable to allocate buffer; not enough memory\n");
		goto cleanup;
	}

	while ( (arg = read(sd1,buf,bufsz)) > 0 ) {
		i = 0;
		rval++;
		if ( discard )
			continue;
		bufp = buf;
		while ( arg > 0 ) {
			if ( (i=write(sd1, bufp, arg)) < 0 ) {
				perror("write");
				goto cleanup;
			}
			arg  -= i;
			bufp += i;
			rval--;
		}
	}

cleanup:
	if ( sd >= 0 )
		close(sd);
	if ( sd1 >= 0 )
		close(sd1);
	free(buf);

	return rval;
}


References:
CA Connect rates under RTEMS wnolan

Navigate by Date:
Prev: RE: CA Connect rates under RTEMS Jeff Hill
Next: Re: CA Connect rates under RTEMS (vxWorks, too) Kate Feng
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
Navigate by Thread:
Prev: RE: CA Connect rates under RTEMS (vxWorks, too) Jeff Hill
Next: RE: CA Connect rates under RTEMS Feng, Shuchen
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  <20052006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020 
ANJ, 02 Sep 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·