Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Thu, 23 Jul 2009 14:52:07 -0600

Ernest,

I did commit a change yesterday evening so that the cvtFast.h functions are
used when/where they can help. However after running the new
"cvtFastPerform" performance tests I see that, on windows vista 64 and an
ancient Linux, the cvtFast functions are maybe only twice as fast as
snPrintf. 

So it probably takes from around one to five seconds to convert
YAGS:DMP1:500:IMAGE to a string. During this conversion the GW will not
respond to or update any of its clients. Its real messy to service other
clients while in the middle of the conversion because PCAS would need to
remember where it was in the array conversion so it could resume the
conversion process later, and these activities would need to be properly
scheduled. It will probably be better to change over to a multi-threaded
server implementation, and before that option arrives find the clients that
ask for gonzo arrays as a string and change them to use client side
conversion.

> By the way, why don't we suffer from these issues with the IOC 
> core implementation of the CA Server (i.e. rsrv)?	

In rsrv there are multiple threads in the server implementation so if one
thread is busy converting/copying a double array into a string array the
other threads, dedicated to other clients, are still active. The end result
is that, for other clients the IOC remains responsive. A single server is
much harder to implement correctly, but when we decided on that approach it
appeared to be necessary in order to get portability (at that time most of
the workstation OS didn?t have native threads).

> One of the work-arounds we are considering for the purposes of IOC-Load
> balancing is to use a softIOC
> in front of the source IOC.  Then most of our CA clients would make a
> connection to the softIOC
> for a subscription to large arrays such as camera images.
> What do you think?

You could also have more than one gateways in parallel, and dedicate one of
them to camera images. The soft IOC code base is of course has the highest
levels of stability when compared to the gateway. 

Independent of which workarounds are available, we need to eliminate any
issues that you are seeing with the GW. The exception possibly being if your
issues are caused by string conversion  of arrays. Maybe that can only be
fixed safely by speeding up the conversion function.

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: Ernest L. Williams Jr. [mailto:[email protected]]
> Sent: Thursday, July 23, 2009 1:59 PM
> To: Jeff Hill
> Cc: 'Dirk Zimoch'; 'Ralph Lange'; 'Amedeo Perazzo'; 'Core-Talk'; Zhou,
> Jingchen; Ernest L. Williams Jr.
> Subject: Re: epicsSnPrintf is very slow when converting string arrays in
> GDD's aitConvert.cc
> 
> Hi Jeff,
> 
> Here is an example of some the large arrays we have running around at
> SLAC --LCLS:
> 
> FEE system
> -------
> DIAG:FEE1:481:DISPLAYWF.NELM   262144
> DIAG:FEE1:482:DISPLAYWF.NELM   262144
> CAMR:FEE1:455:IMAGE.NELM       307200
> 
> E- system
> ---------------
> YAGS:DMP1:500:IMAGE.NELM       1.44768e+06
> CAMR:IN20:186:IMAGE.NELM       307200
> ==========================================================================
> ================
> 
> By the way, why don't we suffer from these issues with the IOC core
> implementation of the CA Server (i.e. rsrv)?
> Or maybe, there are some subtle issues with large arrays even for iocCore?
> 
> 
> One of the work-arounds we are considering for the purposes of IOC-Load
> balancing is to use a softIOC
> in front of the source IOC.  Then most of our CA clients would make a
> connection to the softIOC
> for a subscription to large arrays such as camera images.
> What do you think?
> 
> 
> 
> Thanks,
> Ernest
> 
> 
> 
> 
> 
> Jeff Hill wrote:
> >
> > Ø 0.000132 seconds per conversion - is epicsSnPrintf() really that slow?
> >
> > To answer my own question; no. I just finished adding
> > cvtFastPerform.cpp to libCom/tests and it looks like double to string
> > conversion take more like 3 uSec worst case which is about 44 times
> > faster than calculated.
> >
> > Nevertheless, I am curious. What is the largest array size being
> > communicated via the gateways at SLAC and at PSI? This might become a
> > big issue for arrays exceeding say one million elements (that are
> > converted to strings by the gateway).
> >
> > Still bug hunting.
> >
> > Jeff
> > ______________________________________________________
> > Jeffrey O. Hill Email [email protected] <mailto:[email protected]>
> > LANL MS H820 Voice 505 665 1831
> > Los Alamos NM 87545 USA FAX 505 665 5107
> >
> > Message content: TSPA
> >
> > *From:* Jeff Hill [mailto:[email protected]]
> > *Sent:* Wednesday, July 22, 2009 6:13 PM
> > *To:* 'Dirk Zimoch'; 'Ralph Lange'; 'Ernest L. Williams Jr.'; 'Amedeo
> > Perazzo'
> > *Cc:* 'Core-Talk'
> > *Subject:* epicsSnPrintf is very slow when converting string arrays in
> > GDD's aitConvert.cc
> >
> > All,
> >
> > I have discovered that a PCAS hang can occur when clients are fetching
> > very large arrays as strings. Just to convert the number of elements
> > that fit in a very large EPICS_CA_MAX_ARRAY_SIZE buffer with
> > epicsSnPrintf can take a very long time. It appears that this required
> > about 33 seconds during testing with EPICS_CA_MAX_ARRAY_SIZE of
> > 10,000,000. Based on that I calculate 33 / ( 10,000,000 /
> > MAX_STRING_SIZE ) is 0.000132 seconds per conversion. Is
> > epicsSnPrintf() really that slow?
> >
> > The solution appears to be to use the cvtFast library where it is in
> > range in GDD?s aitConvert.cc. The cvtFast functions are a bit scary in
> > today?s world - because a buffer size isn?t passed in ? BTW. The
> > current workaround is to just not use them for buffer sizes below some
> > threshold.
> >
> > Jeff
> > ______________________________________________________
> > Jeffrey O. Hill Email [email protected] <mailto:[email protected]>
> > LANL MS H820 Voice 505 665 1831
> > Los Alamos NM 87545 USA FAX 505 665 5107
> >
> > Message content: TSPA
> >

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System