Experimental Physics and
Industrial Control System

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Manoussakis, Adamandios via Tech-talk
Sent: Thursday, April 1, 2021 6:10 PM
To: tech-talk at aps.anl.gov
Subject: RE: Question on Multiple Instruments in one IOC or Multiple IOCs for each

From: Manoussakis, Adamandios
Sent: Wednesday, March 31, 2021 10:45 PM
To: D Willimoto <dwilliams at kiesbar.com>
Subject: RE: Question on Multiple Instruments in one IOC or Multiple IOCs for each

Hey Dave,

 

Thanks for the explanations every little bit helps me get a better understanding on deploying them.  If you tend to keep your IOCs fairly light on processing, have you stacked more than two devices (I know you mentioned the raspi) in an IOC like 8 dmms.  I think I plan to try it out and see if it breaks down but still wasn’t sure if there was a proper way to deploy all of the same devices in IOCs.

 

Thanks

Adam 

Sent from my iPhone

 

On Mar 31, 2021, at 1:05 AM, D Willimoto <dwilliams at kiesbar.com> wrote:



Hi Adam,

 

I am not replying to the group.  Forgive me, but I'm not a very public person.

 

I use st.cmd.  Can't help you with procServ.  (Having trouble remembering anything at the moment...)  We used procServ at my last job and it wasn't any problem.  I think.

 

I do not think you should have problems with 30 IOCs.  Depending on a lot of other things.  But I'm not sure where the boundaries are--as in what is platform dependent and what is EPICS dependent.  Update period and general communication structure is key.  

 

With your last question, I may not understand correctly.  If my answer is off-target, sorry about that.

 

My general design approach is to push all polling to the digitizer level.  To the FPGA/ADC.  To the hardware.  I write gateware or software scripts on the digitizer level to only update the EPICS CPU when a change occurs.  I also make user-definable thresholds for the change.  For instance, I only care if a power supply changes output by 0.01V.  So, I have a cheap 80MHz FPGA running a C program scanning the incoming ADC.  If the output changes by 0.01V, the FPGA updates a shared memory buffer with the new value and sends an interrupt to the EPICS CPU.  The EPICS CPU updates the IOC.  Anyone with camonitor gets notified.  Including archivers.

 

This pushes all the communication intensive and laggy processing to the hardware level.  Basically, the IOCs never call anything.  They are entirely event-driven.  I do record processing at the FPGA level also.  And just push the calculations up in the same way.

 

I'm not sure if this answers your question.  But summarized:

 

1.  Push processing and polling to hardware.

2.  Eliminate inter-IOC requests and calculations.

3. Update and record only changes, not time-series data for every time period.

4.  Use shared memory buffers for data exchange between digitizer and CPU.  This changes millisecond call-and-receive to microsecond available-if-requested.

 

Best of luck,

Dave

 

 

On Wed, Mar 31, 2021 at 7:02 AM Manoussakis, Adamandios via Tech-talk <tech-talk at aps.anl.gov> wrote:

Hey Dave,

 

Thanks for the information, for platform we will be using Debian flavors of linux as the host for the IOCs and so far have not encountered running multiple IOCs of the epics 3/7.  I was concerned with spawning out so many IOC instances if I did go with the design of one ioc one device (looking at say 30+ IOCs on one host vs combining down to ~5).  I can see the benefit of efficiency when records need to communicate between each other but in this case the digitizers/dmms do not have any dependencies on each other. 

 

Couple other thoughts

 

I was looking at starting to use procServ but was a little confused from reading about the documentation on its use.  What is the difference between spawning multiple IOCs through running your st.cmd for each and instead spawning them using procServ?  Is the IOC treated differently in linux when using procServ to start them?

 

Will I run into resource issues (on a highend desktop) spawning so many IOCs (~30) on one host (I did implement Ralphs script already to solve the multiple ioc ca issue)?

 

Are there concerns when running say 8 devices using streamdevice in a single IOC, I would assume there might be some delay when streamdevice has to make multiple calls in the same IOC?

 

 

 

 

 

 

 

From: D Willimoto <dwilliams at kiesbar.com>
Sent: Tuesday, March 30, 2021 1:52 PM
To: Manoussakis, Adamandios <manoussakis1 at llnl.gov>
Subject: Re: Question on Multiple Instruments in one IOC or Multiple IOCs for each

 

Hi Adam,

 

The platform is one dependency.  Some platforms have trouble instantiating multiple IOCs of certain EPICS versions.  It's difficult to do on a RaspPi for instance.

 

I use RaspiPi to connect to a vacuum and power supply system and I combine them into one IOC for this reason.

 

From a design perspective, there are two contrary forces.  First, encapsulation of device as an object and address.  

 

Second is the power of records to be actions and perform modifications.  For instance a record can be a calculation from two other records. If so, you want those two source records in the same IOC to eliminate constant communication lag.  This factor also exists with other database systems like Elastic, Splunk, etc.

 

There are other perspectives but these are the main practical ones in my view.

 

Best,

Dave

 

 

On Wed, Mar 31, 2021 at 5:44 AM Manoussakis, Adamandios via Tech-talk <tech-talk at aps.anl.gov> wrote:

Hello,

 

From a design perspective, is it better to have one device per IOC or would it be more efficient to group same devices into one IOC?  I have multiple same instruments (digitizers, DMMs) and was thinking to create one IOC with say 8 Digitizers or 8 DMMs but wasn’t sure if this was the way to go about this.  If its better to go the other direction 1 device 1 IOC, is there a better way to spawn out the IOCs (I would guess a bash script with variables for any of the values that need to be changed eg. IP address for each)?

 

Thanks,

Adam