Re: Modbus

[Zenon Szalata <zms@slac.stanford.edu>]

Hi Mark,
I am using unmodified modbusInterpose.c; the timeout gets copied in both 
writeIt and readIt routines.
I have put print statements in drvAsynIPPort:readIt and I see that 
poll() is used.  I also verified that the timeout in this routine has 
the 1 sec value.  So, it looks like all is fine and the read should not 
be failing and yet it does.  This is getting more confusing.  I have no 
idea what is going on.
Here are a few "facts":

I run the IOC in question on my desktop computer which has a second NIC 
configured to 192.168.1.xx private sub-net.  Beckhoff system is 
connected to the private sub-net and I have not observed a single IO 
failure over a long time; there are no failures.
I ran the same IOC on two different computers which have access to the 
same "public restricted" sub-net where the Beckhoff is connected.  In 
both cases the computers are more powerful and IO fails sometimes very 
often at other time not so often.  "public restricted" is a sub-net 
configured such that it is only accessible from two public sub-nets here 
at SLAC.  Other than that, it is supposed to be a normal sub-net.
When the IOC fails to do IO, it is the read in the write-read that 
fails.  As best as I can tell, the read fails immediately after the 
write completes.  This is based on observing the print statements in 
both the write and read routines, which makes rather subjective.
The error printed is:

2013/08/05 08:55:14.982 modbusAsyn::doModbusIO port BKH15 error
calling writeRead, error=sioc-esb-bkh15:502 timeout: Resource
temporarily unavailable, nwrite=6/6, nread=0


Is it some kind of networking issue?
What else can I look at?
Any suggestions are most welcome,
Zen


On 08/06/13 06:26, Mark Rivers wrote:
> The timeout value gets passed from my writeIt and readIt routines in my modbusIbterpose.c to the underlying IP driver in the pasynUser structure via these 2 statements:
>
> static asynStatus writeIt(void *ppvt, asynUser *pasynUser,
>                            const char *data, size_t numchars,
>                            size_t *nbytesTransfered)
> {
> ...
>      pasynUser->timeout = pPvt->timeout;
>
> and
>
>
> static asynStatus readIt(void *ppvt, asynUser *pasynUser,
>                           char *data, size_t maxchars, size_t *nbytesTransfered,
>                           int *eomReason)
>
> ...
>      pasynUser->timeout = pPvt->timeout;
>
>
> So pPvt->timeout gets copied to pasynUser->timeout, and the pasynUser is passed to the underlying IP driver as follows:
>
>      /* Set number read to 0 in case of errors */
>      *nbytesTransfered = 0;
>
>      switch(pPvt->linkType) {
>          case modbusLinkTCP:
>              nRead = maxchars + mbapSize + 1;
>              status = pPvt->pasynOctet->read(pPvt->octetPvt, pasynUser,
>                                              pPvt->buffer, nRead,
>                                              &nbytesActual, eomReason);
>
>
> Maybe your driver is not doing that copy to the pasynUser structure?
>
> Mark
>
>
> ________________________________________
> From: Zenon Szalata [zms@slac.stanford.edu]
> Sent: Tuesday, August 06, 2013 12:05 AM
> To: Mark Rivers
> Cc: Williams Jr., Ernest L.; tech-talk@aps.anl.gov
> Subject: Re: Modbus
>
> After the call to pPvt->pasynOctet->read in modbusInterpose:readIt the
> values are:
> pPvt->timeout=1.0,
> status=1,
> eomReason=0.
>
> I have simplified considerably the drvModbusAsynConfigure routine in my
> version of the modbus driver.  It must be that I have removed some code
> fragment that is needed so that the drvAsynIPPort driver gets the
> timeout value.  I have to confess that I tried a few times to understand
> how information is passed from one component to another in the asyn
> module and each time I walked away from the problem without getting to
> the bottom of it.
> I wonder if you could outline how the pPvt->timeout value gets passed on
> to the IPPort driver.  This might help me see what I am missing in my
> simplified version.
> In a nutshell, I kept only two routines from the drvModbusAsyn.c, the
> configure and doModbusIO routines.   I made only slight modifications to
> doModbusIO routine.  On the other hand I simplified the configure
> routine quite a lot.  I removed all device support from it and more.
> Sorry for troubling you with this,
> Zen
>
> On 08/05/13 17:45, Mark Rivers wrote:
> > The timeout used  by the drvAsynIPPort driver is the one passed in pasynUser->timeout, which is taken from pPvt->timeout in modbusInterpose.c.
> >
> > The pasynOctet->read() routine is the readIt function in drvAsynIPPort.c.
> >
> > When readIt fails what is the return code, and what is eomReason?
> >
> > Mark
> >
> > ________________________________________
> > From: Zenon Szalata [zms@slac.stanford.edu]
> > Sent: Monday, August 05, 2013 3:01 PM
> > To: Mark Rivers
> > Cc: Williams Jr., Ernest L.; tech-talk@aps.anl.gov
> > Subject: Re: Modbus
> >
> > Hi Mark,
> > I think that the timeout you refer to is controlled via the third
> > argument to modbusInterposeConfig routine.  This is set to 1000 for all
> > Beckhoff IOCs.
> > I have not used the asyn trace facility because there are large number
> > of IOs happening in one second.  I will try it.
> > I verified using a print statement in the readIt routine in
> > modbusInterpose.c that the pPvt->timeout is 1.0, so, if this is the real
> > timeout, then it is not used in the IO request.
> > Is there a timeout in the drvAsynIPPort driver?
> >
> > I am looking at modbusInterpose.c and I find it difficult to tell what
> > timeout the read routine is using.  Here is the line from modbusInterpose.c:
> >
> >          status = pPvt->pasynOctet->read(pPvt->octetPvt, pasynUser,
> >                           pPvt->buffer, nRead, &nbytesActual, eomReason);
> >
> > I can't tell how the pasyOctet->read routine is coded; not sure where it
> > is defined and what timeout is used in the routine which ultimately
> > issues the modbus read request.
> >
> > Now, in my new code, I put print statements in the writeIt and readIt
> > routines in the modbusInterpose.c file and I see that the readIt fails
> > immediately after writeIt completes with no perceivable delay.
> > So I am confused.
> >
> > In the case of the IOCs using your modbus support module, there could be
> > some other effect causing the error.
> > For better or worse in these IOCs, I am setting up the IOCs such that
> > drvModbusAsynConfigure is called for each Beckhoff bus terminal type.  I
> > use 100 ms period for the poller loop in each.  I think each thread
> > loops asynchronously with respect other threads. Therefore it is
> > conceivable that often a write is issued from one thread when the
> > coupler is not ready as it is processing some IO from another thread.  I
> > did not test this to see if this is indeed happening.
> >
> > To avoid this I have created a separate modbus IO class object in my new
> > code, which uses a message queue to coordinate orderly execution of IO
> > requests, which is done in a poller loop in this class. Requests for IO
> > from multiple threads are scheduled for execution via the message queue.
> > Thanks Mark,
> > Zen
> >
> >
> > On 08/05/13 10:23, Mark Rivers wrote:
> > > Hi Zen,
> > >
> > > What timeout are you using for your TCP read?  The read does not actually put anything on the Ethernet, it just waits for a message to arrive from the Beckoff.  So if you use a timeout of 1 second or so then it should never time out, assuming the module really does answer within 20 ms or so.
> > >
> > > You need to turn on asynTrace in the underlying drvAsynIPPort driver to see exactly the timing of what is being sent and received by that driver.
> > >
> > > Mark
> > >
> > > ________________________________________
> > > From: Zenon Szalata [zms@slac.stanford.edu]
> > > Sent: Monday, August 05, 2013 11:33 AM
> > > To: Mark Rivers; Ernest Williams; tech-talk@aps.anl.gov
> > > Subject: Modbus
> > >
> > > Hi Mark,
> > > I am nearly done with my version of modbus TCP for use with Beckhoff
> > > hardware.  I have a few IOCs using this support module and all seems to
> > > be working well in my test area.
> > > I have deployed an IOC for a couple of stepper motors controlled by
> > > Beckhoff devices in a production area and discovered that the IOC is
> > > generating many errors.
> > > The error is:
> > >
> > >      2013/08/05 08:55:14.982 modbusAsyn::doModbusIO port BKH15 error
> > > calling writeRead, error=sioc-esb-bkh15:502 timeout: Resource
> > > temporarily unavailable, nwrite=6/6, nread=0
> > >
> > > Now, modbusAsyn is the name I used for the file which has the two
> > > routines I took from your file drvModbusAsyn.  Looking at the log files
> > > of the IOCs used in the same production area which control Beckhoff
> > > devices and which are built with your modbus support module, I see that
> > > the same errors are generated.  Here is an example:
> > >
> > > 2013/06/16 13:19:25.918 drvModbusAsyn::doModbusIO port BKH10_CPL_R0
> > > error calling writeRead, error=sioc-esb-bkh10:502 timeout: Resource
> > > temporarily unavailable, nwrite=6/6, nread=0
> > >
> > > People who use these IOCs ignored these errors because mostly they did
> > > not matter.
> > >
> > > In the case of the stepper motor controller IOC it does matter. First of
> > > all I realized that the difference between my testing environment and
> > > that of production is the computer speed and a different networking
> > > environment.  The production computer used for the Beckhoff IOCs is much
> > > faster.  So it seems that there is some timing issue at work here.
> > > After putting print statements in various places, I see that the routine
> > > writeIt in modbusInterpose.c file always succeeds but the routine readIt
> > > in the same file fails from time to time.  I reread the manual for the
> > > Beckhoff BK9000 Ethernet coupler and came across a statement saying that
> > > the coupler takes from 13 to 15 ms to process a modbus message.  So it
> > > seems that the read comes too soon after the write.  In my version of
> > > the modbus support module for Beckhoff, I have added a 20 ms delay after
> > > the pasynOctet->write in modbusInterpose.c and this has mostly solved
> > > the problem.  I say mostly, because I still see this error message but
> > > very infrequently.
> > > I am writing all this because I don't understand why this error is
> > > generated so I am not sure if the direction I took to fix the problem is
> > > correct.  I also think that your modbus support module may need to be
> > > modified if it is to be used with Beckhoff devices.
> > > I would like to hear what you think about this.
> > > Thanks,
> > > Zen