EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Initial value readback from hardware into output records
From: Russell Redman <[email protected]>
To: Ralph Lange <[email protected]>, Mark Rivers <[email protected]>
Cc: <[email protected]>, <[email protected]>
Date: Thu, 09 Dec 2004 11:20:55 -0800
Title: Re: Initial value readback from hardware into output records
This is a difficult problem, no two ways about it.  I have fairly regularly thought that initialization was one of the weaker aspects of EPICS.  There are also some really nasty differences between hardware and simulation mode I/O that need to be considered.  

Mark Rivers has pointed out that my existing database does not use drivers based on the generic asynDriver interface, and does not need to be changed to use them.  Notwithstanding, there do appear to be some real advantages to using the new drivers and the database should probably be rewritten to use them if and when it gets ported to 3.14
.  

Most of the control variables that I have to work with are not controlled directly by the IOC. We have a large set of peripheral controllers that are connected by CAN bus and serial lines to the IOC.  Thankfully, we were able to avoid GPIB, but many of the same issues arise for all three media.  
My impression is that the existing CAN bus drivers do not use asynDriver, but the warm reboot option seems sufficiently appealing that “we” (i.e. not me right now) might want to rewrite the CAN drivers to use a similar interface.

Mark also suggested that a solution might be to write an asyn simulation driver that looks like a real driver but does not attempt to access the hardware.  I am VERY reluctant to attempt this solution, partly because it sounds like a lot of work, but mostly because it ignores completely the SIMM and SIML fields that should really be used to control simulation mode.  Instead, we would have to boot the whole database in either hardware or simulation mode and it would be impossible to switch between them on-the-fly.  I prefer Ralph’s implied approach of understanding, and possibly correcting, the way records manage the SIML field during iocInit.

I fear it is impractical to detect the presence or absence of hardware when it sits on the other end of a cable.  All our serial connections, for example, are connected to a terminal server.  The terminal server itself does not detect whether there is a cable attached to a port, much less whether the cable is plugged in at the other end and the peripheral controller is turned on.  The only test available is to read a value back from the hardware and wait to see whether the reply times out.  This test would have to be completed for each serial line before any dependent records could complete their initialization, and I do not see any generic way to do it.

In simulation mode, the record should still read back its initial value during init_record, but from the simulator instead of from the hardware.  Whatever the hardware does, the simulator should do too, although it may use a very different mechanism.  Note that this immediately raises very serious issues about the order in which records are initialized, which I will come back to below.

The correct solution might be to allow allow records to be initialized during operation and to do this every time SIMM changes.  This would allow the operator to switch a module in and out of simulation mode safely and repeatedly.  I am not sure what the side effects might be however, beyond the obvious problem that the value of the record is likely to change unpredictably during the switch between modes, and that different records will change their modes and their values at different times.  In operation, I would start the database by default in simulation modem, to avoid accessing possibly nonexistent hardware, and would then switch the mode to hardware by hand (and possibly separately for each hardware package) or by setting a record in the startup script.

Note that this is not the same as creating and deleting records dynamically, although dynamic records could probably be used to address the problem.

The order and timing in which records are initialized is a difficult problem that has given me a lot of grief.  I have been assuming that a record that reads back a value during iocInit will not process, and will not cause other records linked to it through FLNK, CPP INLINKs or monitors to process.  Otherwise, mayhem would result as records attempt to process before they have been initialized.  I suspect I am missing something here because I have not read through the code.  If I recall correctly, iocInit runs in two passes.  The first sets up all the data structures and device support, and the second process all those records that have PINI=YES.  It sounds like the value is being read back during the initialization of the device support.  Surely, the correct time to read a value back from hardware is during the second pass.  By this point it should be OK for the record to process, and for it to cause any linked records or monitors to be processed as well.

Some related issues: If the normal asynchronous callback is used to fetch the value from the hardware, would that cause the record to process?  What happens if the hardware sends back its reply before the database has completed iocInit?  Would data be lost?  Many peripherals do not use flow control and will continue to ram bytes into a port regardless of whether the IOC is listening.  A really thorny problem arises for output records whose values are read back from the hardware in groups.  Status words are often of this kind, and I have several examples in my IOC of CAN messages that return four real values as 12-bit integers for the last-set values of a collection of DACs.  This is hard enough to deal with using ai records in “I/O Intr” mode, where the logic in simulation mode is utterly, utterly different from what happens in hardware mode.  It will be much worse for ao records during iocInit, which will not be configures for SCAN=”I/O Intr”..  I do not have a solution for this, except to disable the read-during-initialization for such records.

Setting global values is also a bit tricky.  You cannot use dbpf until after iocInit, by which time it is too late to set the simulation mode or initialization policy.  Default starting values can be set hard-coded into the database, or configured using templates in the startup script.  I suspect that templates are the correct tool, although I have not yet had any reason to try them in my own database.

Enough musing for now.  I have probably confused everyone who bothers to read this far.  I know my own head is spinning.

Cheers,
Dr. Russell O. Redman
Tel: (250) 363-6917 | Fax: (250) 363-0045
<mailto:[email protected]>
National Research Council Canada | Conseil national de recherches Canada
5071 West Saanich Road           | 5071 West Saanich Road
Victoria, B. C. V9E 2E7          | Victoria, C.-B. V9E 2E7
Government of Canada             | Gouvernement du Canada
--



On 2004-12-09 3:39 AM, "Ralph Lange" <[email protected]> wrote:

Interesting ...

Looking at the ao record I realized:

While the record support's process() usually checks for the simulation
mode by reading the SIML link and only calls device support in "real
mode", its init_record() only sets SIMM if SIML is a constant and calls
device support anyway.

For a record coming up in simulation mode after a reboot, whatever you
do seems to be wrong:
 - If you skip device support's initialization, the record cannot be
   switched to real mode later, as the device is not initialized.
 - If you do the initialization, you will access hardware, which might
   be against the intention for rebooting the record in simulation mode
   in the first place.

It looks as if a proper solution will not be available before EPICS is
capable of adding records on-line - then there have to be ways for late
hardware initialization, which can also be used for this case.


For the time being ...

Here's an idea to make a database work more smoothly without the
hardware being present:

In its init routine, the device support should be able to detect if the
hardware is missing. Instead of raising an error and maybe blocking the
record (e.g. done by setting PACT=1) it could internally mark the
hardware as missing and force the record into simulation mode instead.

If your database doesn't use the simulation mode mechanism or has been
set up to come up in simulation mode, everything will be just fine.

What if the record later gets switched to real mode (or gets this from
its SIML location)? Check for the missing hardware flag in the device
support's write routine and raise an error (set the record INVALID). The
overhead for this additional check (done at every processing) should be
tolerable.

Zero-tolerance mode: On production systems (where the hardware should
please really exist) device support should still raise errors and maybe
block the record at boot time. So you might want to add a common global
switch (variable to be set from the startup script) for all your device
supports to enable the fault tolerant procedure (disabled by default).


What do you think? Doesn't really look perfectly elegant, I agree.
Should the case of "missing hardware at boot time" be handled in a more
generic way? (I.e.: what would a future device support interface need to
handle this properly?)

Cheers,
Ralph


>>>>> "Mark" == Mark Rivers <[email protected]> writes:

  > Russell,
  > The features I was discussing only apply to generic device support for
  > drivers that use the new asynDriver interfaces.  Since your drivers are
  > not based on asyn, and you are not using the asyn generic device
  > support, then this will not be a problem for you.  Nothing in EPICS base
  > has changed in this regard.
 
  > If you begin to use asyn-based drivers, and you don't want the records
  > to set the value during init_record you can:
  > - Use non-generic device support
  > - Write an asyn SIMULATION driver, that looks just like the real driver,
  > but does not talk to hardware
 
  > Mark
 

  >     -----Original Message-----
  >     From: Russell Redman [mailto:[email protected]]
  >     Sent: Tuesday, December 07, 2004 10:44 AM
  >     To: Mark Rivers; [email protected];
  > [email protected]
  >     Subject: Re: Initial value readback from hardware into output
  > records
        
        
  >     I am curious about a closely related issue that I fear will
  > arise when  we try to upgrade our version of EPICS from 3.13.8 to use
  > the newer software.  My code runs in both hardware and simulation modes
  > to allow the software to be tested without having the hardware attached.
  > When the IOC boots, it does not know which mode it is supposed to be in.
  > The operator sets HARDWARE/SIMULATION mode flag, then runs an INITIALISE
  > action to open communications with hardware or with the simulator, as
  > appropriate.  Very bad things happen if the EPICS drivers attempt to
  > read back values from nonexistent hardware, and I have put a
  > considerable effort into the startup code to ensure that I/O records do
  > not process on startup.
        
  >     So here is my question: How can I PREVENT an output record from
  > reading back a value during init_record from hardware that may or may
  > not exist?
        
  >     Cheers,
  >     Dr. Russell O. Redman



Replies:
Re: Initial value readback from hardware into output records Steven . Banks
Re: Initial value readback from hardware into output records Benjamin Franksen
References:
RE: Initial value readback from hardware into output records Ralph Lange

Navigate by Date:
Prev: Re: EPICS problem Andrew Johnson
Next: Re: Initial value readback from hardware into output records Steven . Banks
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Initial value readback from hardware into output records Ralph Lange
Next: Re: Initial value readback from hardware into output records Steven . Banks
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·