RE: ARM - EPICS performance evaluation

["Hill, Jeffrey O" <johill@lanl.gov>]

Ø  I have used the catime tool in EPICs for the client/server network performance

Ø  (i.e channel access).But the results that i am getting with this board when

Ø  compared to a linux-x86 system (intel i3 processor) are very poor.

I would look first at CPU consumption when the tests are being run.

 

If cpu utilization is low then I would verify that the timer interrupt delay (the clock tick interrupt period) is properly calibrated on your ARM board. There are tests in the acctst regression tests, as I recall, that verify that delays scheduled with ca_pend_event are reasonable, and this should also be a good predictor for proper delays in the UDP scheduler in the ca client library, and the delays in the TCP scheduling in the IP kernel. There are also time, timer,  and thread tests in the libCom regression test in base that are relevant (see “<EPICS base>/src/libCom/test”). Low cpu utilization might also point to issues with the proper scheduling of interrupts for the send and receive queues in the network interface driver.

 

If cpu utilization is high then I would verify that the client library is built optimized as this can have some impact on the efficiency of C++ code. Otherwise, high cpu use could also be caused by not performing enough labor each time that an interrupt occurs in the network interface driver.

Ø  Get Latency - mean = 6060.9 uS, std dev = 24137.5 uS, min = 2256.0 uS max = 176482.1 uS

This round trip latency should be considered a red flag if your processor doesn’t also run at very low clock speed. As I recall a 20 MHz 68020 had better round trip latency than this. If cpu utilization is low, then this is maybe a red flag that something is wrong with proper scheduling of interrupts for the send and receive queues in the network interface driver, or something is wrong with the time tick interrupts.

 

Jeff

 

From: ssahoo@vecc.gov.in [mailto:ssahoo@vecc.gov.in]
Sent: Tuesday, March 20, 2012 12:53 AM
To: Hill, Jeffrey O; tech-talk@aps.anl.gov; Emmanuel Mayssat
Subject: ARM - EPICS performance evaluation

 

Thank you, for explaining what the free channel and pend event means..

Can anyone tell me what is the reason for drastically different results for channel connect test , Async and synch get and put test for the two systems used ( one x86 platform and another ARM 9 platform , both running Linux) . But it may be noted that the pend event test and channel free test gives similar results for both of them. The Performance of the ARM board deteriorates only in those tests where network is utilized (i.e. when send and recieve Mbps is not 0.00).

 

Regards,

Shantonu

To: "ssahoo@vecc.gov.in" <ssahoo@vecc.gov.in>, "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
From: "Hill, Jeffrey O" <johill@lanl.gov>
Date: 03/19/2012 08:40PM
Subject: RE: ARM - EPICS performance evaluation

     {mso-style-name:"Table Normal";      mso-tstyle-rowband-size:0;      mso-tstyle-colband-size:0;      mso-style-noshow:yes;      mso-style-priority:99;      mso-style-qformat:yes;      mso-style-parent:"";      mso-padding-alt:0in 5.4pt 0in 5.4pt;      mso-para-margin:0in;      mso-para-margin-bottom:.0001pt;      mso-pagination:widow-orphan;      font-size:11.0pt;      font-family:"Calibri","sans-serif";      mso-ascii-font-family:Calibri;      mso-hansi-font-family:Calibri;      mso-bidi-font-family:"Times New Roman";}

Ø  What is meant by free channel test and pend event test?

 

The pend event measures the time to call ca_pend_event(1e-9) and the free channel test measures the time to call ca_clear_channel.

 

Jeff

 

From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of ssahoo@vecc.gov.in
Sent: Saturday, March 17, 2012 9:58 AM
To: tech-talk@aps.anl.gov
Subject: Re: ARM - EPICS performance evaluation

 

 I am using the TS7500 (ARM 9) board. By performance evaluation i meant the network and latency performance measurement. I have used the catime tool in EPICs for the client/server network performance (i.e channel access).But the results that i am getting with this board when compared to a linux-x86 system (intel i3 processor) are very poor. I am attaching the two reports generated. Have u performed such measurement for your boards. It would be great if u could suggest something regarding this or how to perform the performance measurements.

 

ARM 9 Board :

 

Testing with 10000 channels named VEC:Start

Channel Connect Test

--------------------

Per Op, 980.5962 uS (   0.0010 MHz ),   0.5221 snd Mbps,   0.2611 rcv Mbps

Search tries per chan - mean = 1.0 std dev = 0.1 min = 1.0 max = 2.0

channel name=VEC:Start, native type=5, native count=1

Pend Event Test

----------------

Per Op,   2.9166 uS (   0.3429 MHz )

DBR_FLOAT Test

--------------

   ### async put test ###

Per Op, 162.5500 uS (   0.0062 MHz ),   1.1812 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op, 193.4481 uS (   0.0052 MHz ),   0.6617 snd Mbps,   0.9925 rcv Mbps

   ### synch get test ###

Per Op, 5874.1800 uS (   0.0002 MHz ),   0.0218 snd Mbps,   0.0327 rcv Mbps

DBR_DOUBLE Test

---------------

   ### async put test ###

Per Op, 171.7263 uS (   0.0058 MHz ),   1.1181 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op, 188.0780 uS (   0.0053 MHz ),   0.6806 snd Mbps,   1.0209 rcv Mbps

   ### synch get test ###

Per Op, 5855.1620 uS (   0.0002 MHz ),   0.0219 snd Mbps,   0.0328 rcv Mbps

DBR_STRING Test

---------------

   ### async put test ###

Per Op, 121.9442 uS (   0.0082 MHz ),   3.6738 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op, 122.7867 uS (   0.0081 MHz ),   1.0425 snd Mbps,   3.6486 rcv Mbps

   ### synch get test ###

Per Op, 10904.0876 uS (   0.0001 MHz ),   0.0117 snd Mbps,   0.0411 rcv Mbps

DBR_INT Test

------------

   ### async put test ###

Per Op, 100.5193 uS (   0.0099 MHz ),   1.9101 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op, 124.0433 uS (   0.0081 MHz ),   1.0319 snd Mbps,   1.5478 rcv Mbps

   ### synch get test ###

Per Op, 5841.4275 uS (   0.0002 MHz ),   0.0219 snd Mbps,   0.0329 rcv Mbps

Get Latency Test

----------------

Get Latency - mean = 6060.9 uS, std dev = 24137.5 uS, min = 2256.0 uS max = 176482.1 uS

Free Channel Test

-----------------

Per Op,   0.4478 uS (   2.2330 MHz ),   0.0000 snd Mbps,   0.0000 rcv Mbps

 

 

x86 system: 

 

Testing with 10000 channels named VEC:Start

Channel Connect Test

--------------------

Per Op,  12.3947 uS (   0.0807 MHz ),  41.3081 snd Mbps,  20.6540 rcv Mbps

Search tries per chan - mean = 1.1 std dev = 0.2 min = 1.0 max = 2.0

channel name=VEC:Start, native type=5, native count=1

Pend Event Test

----------------

Per Op,   2.5694 uS (   0.3892 MHz )

DBR_FLOAT Test

--------------

   ### async put test ###

Per Op,   0.8990 uS (   1.1123 MHz ), 213.5624 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op,   1.1567 uS (   0.8645 MHz ), 110.6575 snd Mbps, 165.9863 rcv Mbps

   ### synch get test ###

Per Op, 223.9892 uS (   0.0045 MHz ),   0.5715 snd Mbps,   0.8572 rcv Mbps

DBR_DOUBLE Test

---------------

   ### async put test ###

Per Op,   0.6781 uS (   1.4746 MHz ), 283.1310 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op,   1.0460 uS (   0.9560 MHz ), 122.3730 snd Mbps, 183.5595 rcv Mbps

   ### synch get test ###

Per Op, 235.0035 uS (   0.0043 MHz ),   0.5447 snd Mbps,   0.8170 rcv Mbps

DBR_STRING Test

---------------

   ### async put test ###

Per Op,   0.9966 uS (   1.0034 MHz ), 449.5136 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op,   1.0918 uS (   0.9159 MHz ), 117.2383 snd Mbps, 410.3341 rcv Mbps

   ### synch get test ###

Per Op, 284.6693 uS (   0.0035 MHz ),   0.4496 snd Mbps,   1.5738 rcv Mbps

DBR_INT Test

------------

   ### async put test ###

Per Op,   0.6753 uS (   1.4809 MHz ), 284.3340 snd Mbps,   0.0000 rcv Mbps

   ### async get test ###

Per Op,   1.0411 uS (   0.9606 MHz ), 122.9506 snd Mbps, 184.4258 rcv Mbps

   ### synch get test ###

Per Op, 244.8803 uS (   0.0041 MHz ),   0.5227 snd Mbps,   0.7841 rcv Mbps

Get Latency Test

----------------

Get Latency - mean = 214.1 uS, std dev = 51.4 uS, min = 164.8 uS max = 3700.0 uS

Free Channel Test

-----------------

Per Op,   0.4586 uS (   2.1806 MHz ),   0.0000 snd Mbps,   0.0000 rcv Mbps

 

 

What is meant by free channel test and pend event test?

 

 

Regards,

Shantonu sahoo.

 

To: "ssahoo@vecc.gov.in" <ssahoo@vecc.gov.in>
From: Emmanuel Mayssat <emayssat@yahoo.com>
Date: 03/17/2012 05:44AM
Cc: epics <tech-talk@aps.anl.gov>
Subject: Re: ARM - EPICS performance evaluation

 

Well, I am not sure what you call performance evaluation.

But as far as running epics/seq/etc, that's no problem.

(I used 3.14.11 with minimal patching)

 

The issue you may be facing is I/O access.

On the ts7370 (ARM9), it takes me 4ms to get a AI reading.

The main issues on this board is

1/ no buffering

2/ you are running a full blown linux OS

 

For higher througput, I am looking at microcontrollers (arduino megas or more advanced)

Read this to get a flavor of the issues ;)

http://sites.google.com/site/measuringstuff/the-arduino

 

To have a full blown arm production deployment (as we are strongly considering),

there are practical issues:

1/ management of SD images (SD card)

2/ controlled upgrade/change of SD image

 

To bypass those issues, we are going in the direction of

beagleboard-xm (full blown linux desktop environment, with epics IOC but no direct critical I/O)

attached to microcontrollers (for critical I/O)

Cost of parts ~ $200

 

One of my coworkers called this design 'smart chassis': with the spare IO, you can monitor the environment (temp, vibration, etc)

I call it 'epics plug-and-play': You plug your chassis to your network and the epics PV are immediately available.

For maintenance, when the hardware is swapped/changed, the ioc software is changed with it.

For troubleshooting on the bench, engineers/technicians can attach a keyboard and a monitor to a chassis and immediately start working. (Qt works great with embedded systems ;)

We design a lot of our electronics, so by using open hardware solutions, we can *in theory* integrate the ARM+ucontroller directly in our electronics design (maybe at 2nd generation design)

...etc...

 

In short, they are several advantages to this design which fit *OUR* needs.

 

Looking forward ARM cpu are becoming more and more powerful, epics/qt is on the way...

You see, the tide is also going that direction...

 

 

---

From: "ssahoo@vecc.gov.in" <ssahoo@vecc.gov.in>
To: Emmanuel Mayssat <emayssat@yahoo.com>
Sent: Friday, March 16, 2012 9:39 AM
Subject: Re: EPICS performance evaluation

 

Yes, I am using TS7500 from embeddedarm (ARM9).. 

 

So, have u performed any performance evaluation test for ur arm platform.

 

Thanking you,