Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020 
<== Date ==> <== Thread ==>

Subject: Re: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux
From: "Johannes Spinneken (EGI)" <johannes@evergreeninnovations.co>
To: Mark Rivers <rivers@cars.uchicago.edu>, "'Mooney, Tim M.'" <mooney@aps.anl.gov>, "'tech-talk@aps.anl.gov'" <tech-talk@aps.anl.gov>
Date: Fri, 9 Feb 2018 15:31:10 +0000

Mark (and all)

 

Many thanks for looking into this and providing such detailed feedback – really appreciated.

 

I have been looking into this further and have now (most likely) established the source of the latency. As a simple additional test, I have implemented direct UDP traffic between the NI cRIO and a Linux RT machine. Based on direct UDP, I see return traffic of order <300 microseconds, which is far less than the 10ms using EPICS on the cRIO.

 

I had some fairly long conversations with National Instruments support staff, and also looked further into their implementation of the SVE (Shared Variable Engine) and I/O Servers. Some relevant links are

 

http://www.ni.com/white-paper/4679/en/

http://www.ni.com/white-paper/13865/en/

http://www.ni.com/white-paper/12079/en/

 

I attached the most relevant data flow diagram and a table outlining the relative merits of various network protocols in LabVIEW.

 

Using LabVIEW, the NI way of accessing EPICS (for example on a cRIO) is by binding a Network Published Shared Variable to an EPICS PV. This subsequently involves the following communication cycles:

 

  • Shared Variable is updated
  • Shared Variable communicates to Shared Variable Engine (SVE)
  • SVE publishes Shared Variable on the EPICS network
  • The same in reverse on returning traffic

 

The Shared Variable (network published version) and I/O Servers are NOT designed for low latency communication, but rather for high data throughput. As a result, any communication protocol in LabVIEW that relies on Shared Variables and I/O Serves (that is: EPICS and Modbus TCP) are inherently affected by the latency introduced through NI I/O servers. It seems that 10ms round-trip latency is all that can be achieved using this architecture.

 

For true real-time communication, the NI cRIO supports EtherCAT. For low latency communication, TCP/UDP seems most suitable.

 

It is unfortunate in my view that EPICS suffers from the NI introduced latencies, as this pretty much renders EPICS unusable as a (soft) real time protocol on a cRIO. Given Mark’s timing trials, I am convinced that this is not in any way related to the EPICS structure, but rather introduced through the LabVIEW I/O server.

 

Thanks again for your help!

 

Best wishes

Johannes

 

 

Johannes Spinneken PhD FRSA

Principal Systems Engineer

 

Evergreen Innovations UK & USA

+1 916 266 3709 (US cell)

+44 7868 923583 (UK mobile)

www.evergreeninnovations.co

 

 

 

 

 

From: Mark Rivers <rivers@cars.uchicago.edu>
Date: Friday, 9 February 2018 at 06:03
To: "'Mooney, Tim M.'" <mooney@aps.anl.gov>, "Johannes Spinneken (EGI)" <johannes@evergreeninnovations.co>, "'tech-talk@aps.anl.gov'" <tech-talk@aps.anl.gov>
Subject: RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux

 

Curiosity continues …

 

This time I ran the 2 IOCS on the same machine.  So it removes the network latency, it is now inter-process communication on the same machine.

 

camonitor over 10 ms time period again (most values deleted to save space).  Calc1 incremented by 470, so there were 235 round trips.  The round trip time was thus 43 microseconds.  Comparing with the 92 microseconds when the IOCs are on different machines the round trip network overhead appears to be about 49 microseconds.  In this case each IOC was using 66% of a core, so it is not CPU bound.

 

Calc1                          2018-02-09 07:36:42.930026 723056

Calc2                          2018-02-09 07:36:42.930046 723057

Calc1                          2018-02-09 07:36:42.930074 723058

Calc2                          2018-02-09 07:36:42.930093 723059

Calc1                          2018-02-09 07:36:42.930109 723060

Calc2                          2018-02-09 07:36:42.930128 723061

Calc1                          2018-02-09 07:36:42.930146 723062

Calc1                          2018-02-09 07:36:42.939897 723518

Calc1                          2018-02-09 07:36:42.939933 723520

Calc2                          2018-02-09 07:36:42.939916 723519

Calc1                          2018-02-09 07:36:42.939971 723522

Calc2                          2018-02-09 07:36:42.939952 723521

Calc1                          2018-02-09 07:36:42.940010 723524

Calc2                          2018-02-09 07:36:42.939992 723523

Calc1                          2018-02-09 07:36:42.940051 723526

 

Finally I ran a test where both Calc1.db and Calc2.db are running in the same IOC.  This then eliminates both the network latency and the inter-process communication.  In this case Calc1 incremented by 748, so there were 374 round trips.  The round-trip time was thus 27 microseconds.  In this case I believe it is limited by CPU usage, the CPU use for the IOC is 130%, meaning it is using 1.3 cores.

 

Calc1                          2018-02-09 07:48:12.513015 406467

Calc2                          2018-02-09 07:48:12.513027 406468

Calc1                          2018-02-09 07:48:12.513038 406469

Calc1                          2018-02-09 07:48:12.522906 407209

Calc2                          2018-02-09 07:48:12.522922 407210

Calc1                          2018-02-09 07:48:12.522938 407211

Calc2                          2018-02-09 07:48:12.522954 407212

Calc1                          2018-02-09 07:48:12.522970 407213

Calc2                          2018-02-09 07:48:12.522985 407214

Calc1                          2018-02-09 07:48:12.523002 407215

 

Mark

 

 

From: Mark Rivers
Sent: Friday, February 09, 2018 7:14 AM
To: 'Mooney, Tim M.' <mooney@aps.anl.gov>; 'Johannes Spinneken (EGI)' <johannes@evergreeninnovations.co>; 'tech-talk@aps.anl.gov' <tech-talk@aps.anl.gov>
Subject: RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux

 

Hi Johannes,

 

I was curious about what could be obtained in a test that is more similar to what you are asking, where there is an exchange of messages between an EPICS client and server.

 

I did another test, this time with IOCs running on 2 different Linux machines.  Each IOC is both a Channel Access client and a Channel Access server.

 

Machine 1 runs this database:

 

record(calc, "Calc1") {

  field(INPA, "Calc2 CP")

  field(CALC, "A+1")

}

 

Machine 2 runs this database

 

record(calc, "Calc2") {

  field(INPA, "Calc1 CP")

  field(CALC, "A+1")

}

 

Note that the INPA field has the CP property, meaning that calc record will process whenever it receives a Channel Access monitor on its input PV.  So the way it works is that Calc1 increments its value, Calc2 receives a monitor for that and increments its value, which in turn causes Calc1 to receive a monitor and increment its value, etc.  There is thus an exchange of messages as fast as possible.  It will be limited by the overhead in CA writes, CA monitor reception, processing the records, and network latency.

 

Here is the result of running camonitor in both records over a 10 ms time interval.  The Calc1 record incremented from 910732 to 910948 in this time, or a difference of 216.  It increments by 2 for each round-trip exchange, so there were 108 round trips.  Since this was over a period of 10 ms that means that the round-trip time was about 92 microseconds.

 

So the 10 ms time you are seeing is definitely not limited by EPICS Channel Access, record processing, or network overhead, it is something specific to your system!

 

corvette> camonitor Calc1 Calc2

Calc1                          2018-02-09 06:51:09.779757 910732

Calc2                          2018-02-09 06:51:09.780078 910733

Calc1                          2018-02-09 06:51:09.779837 910734

Calc2                          2018-02-09 06:51:09.780161 910735

Calc1                          2018-02-09 06:51:09.779919 910736

Calc2                          2018-02-09 06:51:09.780242 910737

Calc1                          2018-02-09 06:51:09.780001 910738

Calc2                          2018-02-09 06:51:09.780325 910739

Calc1                          2018-02-09 06:51:09.780082 910740

Calc2                          2018-02-09 06:51:09.780409 910741

Calc1                          2018-02-09 06:51:09.780175 910742

Calc2                          2018-02-09 06:51:09.780494 910743

Calc1                          2018-02-09 06:51:09.780250 910744

Calc2                          2018-02-09 06:51:09.780576 910745

Calc1                          2018-02-09 06:51:09.780331 910746

Calc1                          2018-02-09 06:51:09.780411 910748

Calc2                          2018-02-09 06:51:09.780655 910747

Calc1                          2018-02-09 06:51:09.780495 910750

Calc2                          2018-02-09 06:51:09.780739 910749

Calc2                          2018-02-09 06:51:09.780818 910751

Calc1                          2018-02-09 06:51:09.780578 910752

Calc2                          2018-02-09 06:51:09.780903 910753

Calc1                          2018-02-09 06:51:09.780661 910754

Calc2                          2018-02-09 06:51:09.780983 910755

Calc1                          2018-02-09 06:51:09.780744 910756

Calc2                          2018-02-09 06:51:09.781065 910757

Calc1                          2018-02-09 06:51:09.780824 910758

Calc2                          2018-02-09 06:51:09.781148 910759

Calc1                          2018-02-09 06:51:09.780902 910760

Calc2                          2018-02-09 06:51:09.781229 910761

Calc1                          2018-02-09 06:51:09.780984 910762

Calc2                          2018-02-09 06:51:09.781307 910763

Calc1                          2018-02-09 06:51:09.781066 910764

Calc2                          2018-02-09 06:51:09.781388 910765

Calc1                          2018-02-09 06:51:09.781148 910766

Calc2                          2018-02-09 06:51:09.781473 910767

Calc1                          2018-02-09 06:51:09.781232 910768

Calc2                          2018-02-09 06:51:09.781556 910769

Calc1                          2018-02-09 06:51:09.781315 910770

Calc2                          2018-02-09 06:51:09.781634 910771

Calc1                          2018-02-09 06:51:09.781394 910772

Calc2                          2018-02-09 06:51:09.781719 910773

Calc1                          2018-02-09 06:51:09.781479 910774

Calc2                          2018-02-09 06:51:09.781802 910775

Calc1                          2018-02-09 06:51:09.781559 910776

Calc2                          2018-02-09 06:51:09.781883 910777

Calc1                          2018-02-09 06:51:09.781642 910778

Calc2                          2018-02-09 06:51:09.781965 910779

Calc1                          2018-02-09 06:51:09.781726 910780

Calc2                          2018-02-09 06:51:09.782060 910781

Calc1                          2018-02-09 06:51:09.781825 910782

Calc2                          2018-02-09 06:51:09.782160 910783

Calc1                          2018-02-09 06:51:09.781917 910784

Calc2                          2018-02-09 06:51:09.782239 910785

Calc1                          2018-02-09 06:51:09.781997 910786

Calc2                          2018-02-09 06:51:09.782321 910787

Calc1                          2018-02-09 06:51:09.782078 910788

Calc2                          2018-02-09 06:51:09.782401 910789

Calc1                          2018-02-09 06:51:09.782158 910790

Calc2                          2018-02-09 06:51:09.782479 910791

Calc1                          2018-02-09 06:51:09.782235 910792

Calc2                          2018-02-09 06:51:09.782580 910793

Calc1                          2018-02-09 06:51:09.782338 910794

Calc2                          2018-02-09 06:51:09.782705 910795

Calc1                          2018-02-09 06:51:09.782461 910796

Calc2                          2018-02-09 06:51:09.782827 910797

Calc1                          2018-02-09 06:51:09.782591 910798

Calc2                          2018-02-09 06:51:09.782953 910799

Calc1                          2018-02-09 06:51:09.782712 910800

Calc2                          2018-02-09 06:51:09.783078 910801

Calc1                          2018-02-09 06:51:09.782836 910802

Calc2                          2018-02-09 06:51:09.783204 910803

Calc1                          2018-02-09 06:51:09.782962 910804

Calc2                          2018-02-09 06:51:09.783328 910805

Calc1                          2018-02-09 06:51:09.783086 910806

Calc2                          2018-02-09 06:51:09.783451 910807

Calc1                          2018-02-09 06:51:09.783209 910808

Calc2                          2018-02-09 06:51:09.783576 910809

Calc1                          2018-02-09 06:51:09.783333 910810

Calc2                          2018-02-09 06:51:09.783699 910811

Calc1                          2018-02-09 06:51:09.783456 910812

Calc2                          2018-02-09 06:51:09.783826 910813

Calc1                          2018-02-09 06:51:09.783590 910814

Calc2                          2018-02-09 06:51:09.783951 910815

Calc1                          2018-02-09 06:51:09.783713 910816

Calc2                          2018-02-09 06:51:09.784081 910817

Calc1                          2018-02-09 06:51:09.783859 910818

Calc2                          2018-02-09 06:51:09.784212 910819

Calc1                          2018-02-09 06:51:09.783990 910820

Calc2                          2018-02-09 06:51:09.784325 910821

Calc1                          2018-02-09 06:51:09.784084 910822

Calc2                          2018-02-09 06:51:09.784448 910823

Calc1                          2018-02-09 06:51:09.784206 910824

Calc2                          2018-02-09 06:51:09.784570 910825

Calc1                          2018-02-09 06:51:09.784333 910826

Calc2                          2018-02-09 06:51:09.784695 910827

Calc1                          2018-02-09 06:51:09.784453 910828

Calc2                          2018-02-09 06:51:09.784819 910829

Calc1                          2018-02-09 06:51:09.784583 910830

Calc2                          2018-02-09 06:51:09.784944 910831

Calc1                          2018-02-09 06:51:09.784701 910832

Calc2                          2018-02-09 06:51:09.785074 910833

Calc1                          2018-02-09 06:51:09.784831 910834

Calc2                          2018-02-09 06:51:09.785222 910835

Calc1                          2018-02-09 06:51:09.785000 910836

Calc2                          2018-02-09 06:51:09.785338 910837

Calc1                          2018-02-09 06:51:09.785100 910838

Calc2                          2018-02-09 06:51:09.785456 910839

Calc1                          2018-02-09 06:51:09.785213 910840

Calc2                          2018-02-09 06:51:09.785565 910841

Calc1                          2018-02-09 06:51:09.785320 910842

Calc2                          2018-02-09 06:51:09.785690 910843

Calc1                          2018-02-09 06:51:09.785444 910844

Calc2                          2018-02-09 06:51:09.785816 910845

Calc1                          2018-02-09 06:51:09.785572 910846

Calc2                          2018-02-09 06:51:09.785941 910847

Calc1                          2018-02-09 06:51:09.785697 910848

Calc2                          2018-02-09 06:51:09.786072 910849

Calc1                          2018-02-09 06:51:09.785862 910850

Calc2                          2018-02-09 06:51:09.786195 910851

Calc1                          2018-02-09 06:51:09.785971 910852

Calc2                          2018-02-09 06:51:09.786303 910853

Calc1                          2018-02-09 06:51:09.786063 910854

Calc2                          2018-02-09 06:51:09.786385 910855

Calc1                          2018-02-09 06:51:09.786142 910856

Calc2                          2018-02-09 06:51:09.786463 910857

Calc1                          2018-02-09 06:51:09.786218 910858

Calc2                          2018-02-09 06:51:09.786544 910859

Calc1                          2018-02-09 06:51:09.786303 910860

Calc2                          2018-02-09 06:51:09.786627 910861

Calc1                          2018-02-09 06:51:09.786392 910862

Calc2                          2018-02-09 06:51:09.786714 910863

Calc1                          2018-02-09 06:51:09.786473 910864

Calc2                          2018-02-09 06:51:09.786796 910865

Calc1                          2018-02-09 06:51:09.786554 910866

Calc2                          2018-02-09 06:51:09.786875 910867

Calc1                          2018-02-09 06:51:09.786635 910868

Calc2                          2018-02-09 06:51:09.786956 910869

Calc1                          2018-02-09 06:51:09.786716 910870

Calc2                          2018-02-09 06:51:09.787037 910871

Calc1                          2018-02-09 06:51:09.786801 910872

Calc2                          2018-02-09 06:51:09.787122 910873

Calc1                          2018-02-09 06:51:09.786883 910874

Calc2                          2018-02-09 06:51:09.787204 910875

Calc1                          2018-02-09 06:51:09.786965 910876

Calc2                          2018-02-09 06:51:09.787285 910877

Calc1                          2018-02-09 06:51:09.787047 910878

Calc2                          2018-02-09 06:51:09.787373 910879

Calc1                          2018-02-09 06:51:09.787133 910880

Calc2                          2018-02-09 06:51:09.787455 910881

Calc1                          2018-02-09 06:51:09.787214 910882

Calc2                          2018-02-09 06:51:09.787538 910883

Calc1                          2018-02-09 06:51:09.787299 910884

Calc2                          2018-02-09 06:51:09.787623 910885

Calc1                          2018-02-09 06:51:09.787379 910886

Calc2                          2018-02-09 06:51:09.787701 910887

Calc1                          2018-02-09 06:51:09.787458 910888

Calc2                          2018-02-09 06:51:09.787782 910889

Calc1                          2018-02-09 06:51:09.787538 910890

Calc2                          2018-02-09 06:51:09.787863 910891

Calc1                          2018-02-09 06:51:09.787624 910892

Calc2                          2018-02-09 06:51:09.787948 910893

Calc1                          2018-02-09 06:51:09.787705 910894

Calc2                          2018-02-09 06:51:09.788025 910895

Calc1                          2018-02-09 06:51:09.787786 910896

Calc2                          2018-02-09 06:51:09.788112 910897

Calc1                          2018-02-09 06:51:09.787869 910898

Calc2                          2018-02-09 06:51:09.788192 910899

Calc1                          2018-02-09 06:51:09.787949 910900

Calc2                          2018-02-09 06:51:09.788273 910901

Calc1                          2018-02-09 06:51:09.788030 910902

Calc2                          2018-02-09 06:51:09.788352 910903

Calc1                          2018-02-09 06:51:09.788108 910904

Calc2                          2018-02-09 06:51:09.788432 910905

Calc1                          2018-02-09 06:51:09.788189 910906

Calc2                          2018-02-09 06:51:09.788513 910907

Calc1                          2018-02-09 06:51:09.788270 910908

Calc2                          2018-02-09 06:51:09.788594 910909

Calc1                          2018-02-09 06:51:09.788352 910910

Calc2                          2018-02-09 06:51:09.788673 910911

Calc1                          2018-02-09 06:51:09.788429 910912

Calc2                          2018-02-09 06:51:09.788754 910913

Calc1                          2018-02-09 06:51:09.788510 910914

Calc2                          2018-02-09 06:51:09.788841 910915

Calc1                          2018-02-09 06:51:09.788599 910916

Calc2                          2018-02-09 06:51:09.788921 910917

Calc1                          2018-02-09 06:51:09.788682 910918

Calc2                          2018-02-09 06:51:09.789001 910919

Calc1                          2018-02-09 06:51:09.788757 910920

Calc2                          2018-02-09 06:51:09.789081 910921

Calc1                          2018-02-09 06:51:09.788837 910922

Calc2                          2018-02-09 06:51:09.789162 910923

Calc1                          2018-02-09 06:51:09.788919 910924

Calc2                          2018-02-09 06:51:09.789245 910925

Calc1                          2018-02-09 06:51:09.789000 910926

Calc2                          2018-02-09 06:51:09.789324 910927

Calc1                          2018-02-09 06:51:09.789080 910928

Calc2                          2018-02-09 06:51:09.789404 910929

Calc1                          2018-02-09 06:51:09.789162 910930

Calc2                          2018-02-09 06:51:09.789485 910931

Calc1                          2018-02-09 06:51:09.789241 910932

Calc2                          2018-02-09 06:51:09.789566 910933

Calc1                          2018-02-09 06:51:09.789328 910934

Calc2                          2018-02-09 06:51:09.789645 910935

Calc1                          2018-02-09 06:51:09.789401 910936

Calc2                          2018-02-09 06:51:09.789725 910937

Calc1                          2018-02-09 06:51:09.789483 910938

Calc2                          2018-02-09 06:51:09.789806 910939

Calc1                          2018-02-09 06:51:09.789562 910940

Calc2                          2018-02-09 06:51:09.789886 910941

Calc1                          2018-02-09 06:51:09.789646 910942

Calc2                          2018-02-09 06:51:09.789965 910943

Calc1                          2018-02-09 06:51:09.789722 910944

Calc2                          2018-02-09 06:51:09.790046 910945

Calc1                          2018-02-09 06:51:09.789803 910946

Calc2                          2018-02-09 06:51:09.790127 910947

Calc1                          2018-02-09 06:51:09.789882 910948

Calc2                          2018-02-09 06:51:09.790208 910949

 

Mark

 

 

From: Mark Rivers
Sent: Tuesday, February 06, 2018 2:40 PM
To: 'Mooney, Tim M.' <mooney@aps.anl.gov>; Johannes Spinneken (EGI) <johannes@evergreeninnovations.co>; tech-talk@aps.anl.gov
Subject: RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux

 

Hi Johannes,

 

Here is one quick measurement.

 

  • I have an areaDetector simDetector running on a Linux machine.  It has a small image size (50x50) so it can run very fast.  It is doing about 5,500 frames/s.
  • There is an image counter PV that updates each time there is a new image, i.e. at 5,500 Hz.
  • I ran “camonitor” on a different Linux machine to see how fast it received the monitor updates on that PV.  I used –tcs so we see both the CA client timestamps and the CA server timestamps. I sent the camonitor output to a file so the terminal I/O would not slow it down.
  • Both of these machines have 10GB Ethernet connections.

 

This is the result, covering a time period of about 10 ms.

 

Viper:~>camonitor -tcs 13SIM1:cam1:ArrayCounter_RBV > test.out

^C

Viper:~>more test.out

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.602116(2018-02-06 14:34:30.602287) 4182302

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.602320(2018-02-06 14:34:30.602489) 4182303

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.602505(2018-02-06 14:34:30.602611) 4182304

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.602696(2018-02-06 14:34:30.602888) 4182305

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.602899(2018-02-06 14:34:30.603005) 4182306

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.603127(2018-02-06 14:34:30.603238) 4182307

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.603344(2018-02-06 14:34:30.603470) 4182308

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.603536(2018-02-06 14:34:30.603642) 4182309

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.603727(2018-02-06 14:34:30.603838) 4182310

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.603899(2018-02-06 14:34:30.604019) 4182311

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.604109(2018-02-06 14:34:30.604228) 4182312

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.604318(2018-02-06 14:34:30.604440) 4182313

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.604506(2018-02-06 14:34:30.604606) 4182314

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.604692(2018-02-06 14:34:30.604842) 4182315

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.604884(2018-02-06 14:34:30.605005) 4182316

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.605076(2018-02-06 14:34:30.605206) 4182317

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.605260(2018-02-06 14:34:30.605363) 4182318

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.605427(2018-02-06 14:34:30.605539) 4182319

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.605613(2018-02-06 14:34:30.605725) 4182320

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.605795(2018-02-06 14:34:30.605919) 4182321

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.605966(2018-02-06 14:34:30.606132) 4182322

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.606149(2018-02-06 14:34:30.606265) 4182323

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.606319(2018-02-06 14:34:30.606431) 4182324

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.606479(2018-02-06 14:34:30.606594) 4182325

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.606611(2018-02-06 14:34:30.606727) 4182326

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.606820(2018-02-06 14:34:30.606917) 4182327

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.607015(2018-02-06 14:34:30.607116) 4182328

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.607186(2018-02-06 14:34:30.607287) 4182329

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.607363(2018-02-06 14:34:30.607459) 4182330

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.607542(2018-02-06 14:34:30.607643) 4182331

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.607732(2018-02-06 14:34:30.607841) 4182332

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.607914(2018-02-06 14:34:30.608016) 4182333

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.608100(2018-02-06 14:34:30.608202) 4182334

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.608252(2018-02-06 14:34:30.608355) 4182335

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.608432(2018-02-06 14:34:30.608532) 4182336

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.608620(2018-02-06 14:34:30.608731) 4182337

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.608801(2018-02-06 14:34:30.608909) 4182338

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.608983(2018-02-06 14:34:30.609098) 4182339

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.609164(2018-02-06 14:34:30.609279) 4182340

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.609341(2018-02-06 14:34:30.609449) 4182341

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.609526(2018-02-06 14:34:30.609625) 4182342

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.609701(2018-02-06 14:34:30.609817) 4182343

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.609877(2018-02-06 14:34:30.609990) 4182344

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.610057(2018-02-06 14:34:30.610185) 4182345

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.610268(2018-02-06 14:34:30.610392) 4182346

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.610454(2018-02-06 14:34:30.610568) 4182347

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.610620(2018-02-06 14:34:30.610739) 4182348

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.610800(2018-02-06 14:34:30.610908) 4182349

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.610988(2018-02-06 14:34:30.611097) 4182350

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.611168(2018-02-06 14:34:30.611289) 4182351

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.611355(2018-02-06 14:34:30.611454) 4182352

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.611536(2018-02-06 14:34:30.611644) 4182353

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.611728(2018-02-06 14:34:30.611834) 4182354

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.611915(2018-02-06 14:34:30.612013) 4182355

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.612084(2018-02-06 14:34:30.612204) 4182356

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.612265(2018-02-06 14:34:30.612388) 4182357

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.612447(2018-02-06 14:34:30.612543) 4182358

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.612619(2018-02-06 14:34:30.612721) 4182359

13SIM1:cam1:ArrayCounter_RBV   2018-02-06 14:34:30.612806(2018-02-06 14:34:30.612915) 4182360

 

You can see that it does not appear to lose any monitors, and the time between monitors is just under 200 microseconds.  So at least in this setup we have not yet hit the EPICS protocol or network speed limits.

 

Mark

 

 

From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Mooney, Tim M.
Sent: Tuesday, February 06, 2018 2:12 PM
To: Johannes Spinneken (EGI) <johannes@evergreeninnovations.co>; tech-talk@aps.anl.gov
Subject: Re: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux

 

Hi Johannes,

 

With an interrupt service routine, vxWorks can read from or write to an IndustryPack FPGA at around 10 kHz, but I would not try to process an EPICS record at that rate - at least not a record that isn't prepared to throttle the rate at which it posts monitors.  I don't usually process general purpose EPICS records faster than around 100 Hz.  I think it's pretty common practice to keep stuff faster than around 100 Hz restricted to driver code, which doesn't really have to know about EPICS.

 

Tim Mooney (mooney@aps.anl.gov) (630)252-5417
Beamline Controls Group (www.aps.anl.gov)
Advanced Photon Source, Argonne National Lab


From: tech-talk-bounces@aps.anl.gov <tech-talk-bounces@aps.anl.gov> on behalf of Johannes Spinneken (EGI) <johannes@evergreeninnovations.co>
Sent: Tuesday, February 6, 2018 1:33:13 PM
To: tech-talk@aps.anl.gov
Subject: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux

 

Hi

 

I am fairly new to EPICS and have been testing an application with the following setup:

 

  • National Instruments cRIO running an EPICS server
  • EPICS client on RT Linux (fully preemptible Kernel 4.9.76) using EPICS base libs (C application)
  • PV write on cRIO in LabVIEW RT, with new data at 1ms intervals (using 1ms timed loop)
  • Read of EPICS PV in RT Linux with loop times <1ms
  • Posting data back to another EPICS PV to then be read as “round trip” timing on cRIO

 

The setup works in principle, but I am seeing higher latencies than I was expecting. I was hoping to achieve latencies in the low ms order (say 1 – 2ms data round trip from cRIO to RT Linux back to cRIO). In practice, I observe round-trip timings of order 10ms. Two questions:

 

(i)                   What is the best timing (latency) that can generally be achieved using EPICS? Is there something fundamental in the protocol that prohibits data exchange in the low ms order? I understand that this will of course depend on target hardware / software, but a ballpark figure would be good as “expectation management”. A typical ping between cRIO and Linux RT takes approx. 0.2ms, so the hardware seems clearly capable of <1ms timings.

 

(ii)                 Does anyone have experience with running EPICS on LabVIEW RT / cRIO? I suspect some (most) of the latency is due to the way in which NI handles TCP traffic. I already discovered that data is normally buffered for 10ms (or 8 kB). This can be avoided by using the Shared Variable Data Flush VI:

 

http://zone.ni.com/reference/en-XX/help/371361H-01/lvcomm/flushsharedvar/

 

which indeed reduced round-trip timings by around 8-10ms. However, I seem to be unable to push timings down further below the 10ms round-trip mark. Any guidance would be much appreciated.

 

Many thanks!

 

Best wishes

Johannes

 

Johannes Spinneken PhD FRSA

Principal Systems Engineer

 

Evergreen Innovations UK & USA

+1 916 266 3709 (US cell)

+44 7868 923583 (UK mobile)

www.evergreeninnovations.co

 

 

 

 

Attachment: Screen Shot 2018-02-09 at 07.13.31.png
Description: Screen Shot 2018-02-09 at 07.13.31.png

Attachment: Screen Shot 2018-02-09 at 07.17.05.png
Description: Screen Shot 2018-02-09 at 07.17.05.png


References:
Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Johannes Spinneken (EGI)
Re: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Mooney, Tim M.
RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Mark Rivers
RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Mark Rivers
RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Mark Rivers

Navigate by Date:
Prev: Re: 回复: Question about values of historic PVs dislpayed in CSS XY Graph widget Shen, Guobao
Next: Re: Where is histogram? Southern, Tim
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020 
Navigate by Thread:
Prev: RE: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Mark Rivers
Next: Re: Latency between NI LabVIEW RT (cRIO) and EPICS on RT Linux Gabriel de Souza Fedel
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020 
ANJ, 09 Feb 2018 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·