Experimental Physics and Industrial Control System
Hi.
Eric Norum initiated a discussion about this around the end of 2004/11
resulting in an investigation. My conclusions are appended. I could
identify two major reasons for poor results:
a) epics/rtems frequently uses 'malloc_free_space' which can be
a very expensive operation under RTEMS (designed to be a diagnostic).
AFAIK, this has been addressed and a more efficient implementation is
available with the development version (4.7).
An unofficial patch is available for 4.6 but my usual workaround
is to hack
libCom/osi/os/RTEMS/osdPoolStatus.c:osiSufficientSpaceInPool()
to always return 1.
b) half/full duplex ethernet mismatch (SSRL policy is to disable
autonegotiation) resulted in lost packets with tcp performance
becoming really bad.
After addressing these issues, I could report the following results
for EPICS/RTEMS vs. vxWorks on a Synergy VGM series board (300MHz ppc):
RTEMS-svgm vxworks-svgm RTEMS-svgm vxworks-svgm
(1000 channels) (10000 channels)
search 4.8 Mbps 3.0 Mbps 14.5 Mbps 11.1 Mbps
pend 215.7 It/s 215.8 It/s 211.1 It/s 206.2 It/s
async put 5.5 Mbps 6.2 Mbps 11.0 Mbps 11.5 Mbps
async get 12.0 Mbps 10.6 Mbps 25.6 Mbps 19.7 Mbps
sync get 0.7 Mbps 0.4 Mbps 0.2 Mbps 0.2 Mbps
If you are unable to obtain similar performance it could be a driver
issue. The RTEMS kernel and the BSD networking stack performance
have been proven be equivalent.
HTH
-- Till
wnolan wrote:
Hi all,
I was curious what sort or rates are being observed using the catime
utility for RTEMS installations.
On a mvme2307 under RTEMS 4.6.2 running EPICS 3.14.6 I am seeing a
extream slowdown from our VX works installations and was wondering if it
was tyipical.
--- Test data
Same Hardware
MVME2307
1x OMS VME58
1x Jerger VSC16
1x ACROMAG AVME9440
and process database( about 30 records ), no state programs running.
targeting a calc input field for 10000 connections.
VX-works gives
channel connect test
Elapsed Per Item = 0.00031412 sec, 3183.5 Items per sec, 1.8 Mbps
Search tries per chan - mean=1.010000 std dev=0.099499 min=1.000000
max=2.000000
RTEMS gives
channel connect test
Elapsed Per Item = 0.00358250 sec, 279.1 Items per sec, 0.2 Mbps
Search tries per chan - mean=2.210500 std dev=1.631377 min=1.000000
max=8.000000
-------------
I am also seeing the slowdown on ASYN Get tests
VX works seems to run about 6 Mbps where in the same test RTEMS is
running at about 1 Mbps
A factor of 10 on channel connect and a factor of 6 on Asyn Get's
Sync gets are running at about the same rate under both systems , just
under 1 Mbps
Any help, Pointers or comparative data would be a help,
Bill Nolan
[email protected]
Original message:
Here's an update/synopsis of what I know, so far:
SYSTEM: RTEMS-4.6.2-svgm / EPICS 3.14.6 (server and client)
NETWORK: 100-TX-full
A) old (rtems-4.6.2) malloc_free_space() kills
EPICS/channel access performance on fragmented-memory
systems as reported previously.
B) relatively poor and non-reproducable 'catime' results
were due to lost packets.
REASON: our networking folks like to disable autonegotiation
on a full-duplex network. The driver in this case
defaults to half-duplex and misses packets if the
link is heavily used both ways.
Enforcing full-duplex in the driver yielded much
better performance (see below).
SECONDARY EFFECT: In the case of lost packets, TCP response
was VERY bad because 'fast retransmission' doesn't
work well.
Defining TCP_ACK_HACK in
cpukit/libnetworking/netinet/tcp_input.c
remedies that somewhat.
RESULTS:
A) tcpecho
'tcpecho' is a simple application that runs on the RTEMS target
and echoes TCP traffic back to the sender (or, optionally discards
it which is the 'unidirectional' mode below).
I measured the time it took to transfer a 16MB file from a
linux-x86 host.
a) unidirectional, driver in half-duplex mode: ~2s
(not many packets lost)
b) true echo, driver in half-dup, no TCP_ACK_HACK ~347s !!
(packets lost, poor retransmission)
c) true echo, driver in half-dup, with TCP_ACK_HACK ~45s
(packets lost, fair retransmission)
d) true echo, driver in full-dup, ~2s
B) catime
catime to epics example app. The IOC under test ran with
no calls to malloc_free_space() [rtems only] and with
full-duplex enabled [both, under vxWorks and rtems]. Since
the figures for the different EPICS data types are very similar,
the table presents an average:
RTEMS-svgm vxworks-svgm RTEMS-svgm vxworks-svgm
(1000 channels) (10000 channels)
search 4.8 Mbps 3.0 Mbps 14.5 Mbps 11.1 Mbps
pend 215.7 It/s 215.8 It/s 211.1 It/s 206.2 It/s
async put 5.5 Mbps 6.2 Mbps 11.0 Mbps 11.5 Mbps
async get 12.0 Mbps 10.6 Mbps 25.6 Mbps 19.7 Mbps
sync get 0.7 Mbps 0.4 Mbps 0.2 Mbps 0.2 Mbps
==> RTEMS doesn't look bad at all !!
-- Till
PS: vxworks suffers from the 'duplex' mismatch problem also.
Here's the tcpecho code:
#include <rtems.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
int tcpecho(int bufsz, int discard)
{
int rval = 0;
int arg = 1, i;
int sd = -1, sd1 = -1;
char *buf = 0, *bufp;
struct sockaddr_in addr;
if ( !bufsz )
bufsz = 10000; /* default */
if ( (sd = socket(PF_INET, SOCK_STREAM, 0)) < 0 ) {
perror("socket");
goto cleanup;
}
if ( setsockopt(sd, SOL_SOCKET, SO_KEEPALIVE, &arg, sizeof(arg)) )
perror("setting SO_KEEPALIVE");
if ( setsockopt(sd, SOL_SOCKET, SO_REUSEADDR, &arg, sizeof(arg)) )
perror("setting SO_REUSEADDR");
memset(&addr,0,sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(4445);
arg = sizeof(addr);
if ( bind(sd, (struct sockaddr*)&addr, arg) < 0 ) {
perror("bind");
goto cleanup;
}
if ( listen(sd,1) ) {
perror("listen");
goto cleanup;
}
arg = sizeof(addr);
if ( (sd1=accept(sd, (struct sockaddr*)&addr, &arg)) < 0 ) {
perror("accept");
goto cleanup;
}
if ( !(buf = malloc(bufsz)) ) {
fprintf(stderr,"Unable to allocate buffer; not enough memory\n");
goto cleanup;
}
while ( (arg = read(sd1,buf,bufsz)) > 0 ) {
i = 0;
rval++;
if ( discard )
continue;
bufp = buf;
while ( arg > 0 ) {
if ( (i=write(sd1, bufp, arg)) < 0 ) {
perror("write");
goto cleanup;
}
arg -= i;
bufp += i;
rval--;
}
}
cleanup:
if ( sd >= 0 )
close(sd);
if ( sd1 >= 0 )
close(sd1);
free(buf);
return rval;
}
- References:
- CA Connect rates under RTEMS wnolan
- Navigate by Date:
- Prev:
RE: CA Connect rates under RTEMS Jeff Hill
- Next:
Re: CA Connect rates under RTEMS (vxWorks, too) Kate Feng
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
<2005>
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
RE: CA Connect rates under RTEMS (vxWorks, too) Jeff Hill
- Next:
RE: CA Connect rates under RTEMS Feng, Shuchen
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
<2005>
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024