Experimental Physics and
Industrial Control System

Ben Franksen via Tech-talk <tech-talk at aps.anl.gov> · Fri, 19 Jun 2020 15:19:58 +0200

Am 18.06.20 um 00:59 schrieb Konrad, Martin via Tech-talk:
>> Just curious, why do you need procServ at all? You can run tmux (and
>>  presumably Screen) as a systemd service if you want. You can run 
>> tmux commands to start, stop, and restart an IOC.
> Compared to screen and tmux, procServ is lighter and less complex. It
> can also restart a crashed IOC process and it prevents users who are in
> the habit of pressing Ctrl-C + Ctrl-D for logging out from accidentally
> shutting down the IOC.

Yup, these are exactly the reasons we are using it.

(Sorry for calling procServ the "standard" way to run soft IOCs. That
was what I thought but obviously people use a variety of tools.)

> While we're at it I would like to propose a forth solution: How about
> beefing up the IOC itself so it can listen on a socket for connections
> to the IOC shell? This would come with a few advantages (some of them
> might only apply to systems running systemd):
> 
> - No need to use procServ and friends anymore (less moving parts)

Indeed procServ implements some things (namely process supervision) that
systemd does as well, so there is some redundancy here.

On the other hand, separation of concerns is also important. What you
propose for the IOC core must also be maintained in the long run and has
to work on all supported operating systems. This is not trivial.

I think the best way to go about this would be to implement this as a
support module instead of the IOC core.

> - Moving the IOC shell to a socket would free up stdout/stderr for
> logging errors. We could use systemd to connect these file descriptors
> to syslog and leverage more reliable protocols for transporting the data
> to remote log servers like LogStash. This would put an end to annoying
> messages emitted by some unhappy support module that floods the IOC
> shell just because one of us forgot a "printf" somewhere in the code
> which in the worst case might render the IOC shell useless.

IMO this fixes the problem at the wrong end. A driver that spews out
messages with high frequency is buggy and should be fixed. Preventing
such a broken driver from being deployed in a production system is a
matter of QA.

I personally /do/ want to see error messages on the console, just not
repeatedly on every attempt to process a record.

> - It would be transparent to the calling process if the IOC process
> failed to start. In contrast if the IOC runs in procServ all the calling
> process sees is that procServ started happily. That's not enough to
> prevent systemd from starting other services which depend on the IOC or
> to raise a red flag in a deployment tool.

I understand the motivation here but I guess this will not be as useful
in practice as you imagine it to be. Knowing whether an IOC is "running"
or not is very little information. Any service depending on an IOC will
probably depend on a number of PVs existing and having the right
status/severity etc., which indirectly includes information about the
devices controlled by the IOC. There is also the question when you
regard an IOC as "running". Is that after the process was started? After
iocInit? Or after all CA connections have connected? And how do you
signal readiness to systemd? Make the IOC core depend on the sd-notify API?

Cheers
Ben

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System