EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Modbus
From: Mark Rivers <[email protected]>
To: "'Zenon Szalata'" <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Fri, 9 Aug 2013 19:07:41 +0000
Hi Zen,

> It is rather clear that the it is not caused by a timeout, 
> so the word timeout in the error message is very misleading.

I'm not sure I agree with that statement.  What appears to be happening is that the write to the Beckhoff succeeds, but the read fails because the device did not send anything back.  As far as I know there are really only 2 errors conditions on a read:

1) The device has closed the socket connection
2) The device does not send any response within the timeout period.

I don't think the device has closed the socket connection, that is not the error that the drvAsynIPPort is reporting.  So the only alternative is that we expected the device to send a response, and it did not, at least not within the timeout period.  That is by definition a timeout error.

As far as I know the TCP client really cannot generate a more meaningful error condition than "timeout" when devices are behaving like the Beckhoff does.  Someone please correct me if I am wrong.

Mark


-----Original Message-----
From: Zenon Szalata [mailto:[email protected]] 
Sent: Wednesday, August 07, 2013 6:03 PM
To: Mark Rivers
Cc: Williams Jr., Ernest L.; [email protected]
Subject: Re: Modbus

Hi Mark,
I don't yet have a resolution to this problem yet.  Further evidence 
points to some environmental condition that is causing this.  Two days 
ago the laboratory and our experimental area entered a summer shutdown 
mode.  Since then I have not seen a single error generated by any IOC 
controlling Beckhoff devices.  This period will end in September.  In 
the meantime we are discussing ways to diagnose this, how to analyze the 
network, etc. but in absence of errors probably not much can be done.

So, for now, I am closing this thread.

One more comment about the error.  It is rather clear that the it is not 
caused by a timeout, so the word timeout in the error message is very 
misleading.
I have looked at the code in drvAsynIPPort:readIt and I see that the 
return value from the poll() routine is not checked and recv() is called 
regardless of the return status from poll, and in this case most likely 
poll() already failed and recv() should not even be called.  Anyway, 
timeout is encoded into the error message string. Anyway, I am not 
really complaining about this, although it is true that the word timeout 
put me on a wrong track.  Oh well, life is tough.
Thanks for all the help,
Zen
On 08/06/13 06:26, Mark Rivers wrote:
> The timeout value gets passed from my writeIt and readIt routines in my modbusIbterpose.c to the underlying IP driver in the pasynUser structure via these 2 statements:
>
> static asynStatus writeIt(void *ppvt, asynUser *pasynUser,
>                            const char *data, size_t numchars,
>                            size_t *nbytesTransfered)
> {
> ...
>      pasynUser->timeout = pPvt->timeout;
>
> and
>
>
> static asynStatus readIt(void *ppvt, asynUser *pasynUser,
>                           char *data, size_t maxchars, size_t *nbytesTransfered,
>                           int *eomReason)
>
> ...
>      pasynUser->timeout = pPvt->timeout;
>
>
> So pPvt->timeout gets copied to pasynUser->timeout, and the pasynUser is passed to the underlying IP driver as follows:
>
>      /* Set number read to 0 in case of errors */
>      *nbytesTransfered = 0;
>
>      switch(pPvt->linkType) {
>          case modbusLinkTCP:
>              nRead = maxchars + mbapSize + 1;
>              status = pPvt->pasynOctet->read(pPvt->octetPvt, pasynUser,
>                                              pPvt->buffer, nRead,
>                                              &nbytesActual, eomReason);
>
>
> Maybe your driver is not doing that copy to the pasynUser structure?
>
> Mark
>
>
> ________________________________________
> From: Zenon Szalata [[email protected]]
> Sent: Tuesday, August 06, 2013 12:05 AM
> To: Mark Rivers
> Cc: Williams Jr., Ernest L.; [email protected]
> Subject: Re: Modbus
>
> After the call to pPvt->pasynOctet->read in modbusInterpose:readIt the
> values are:
> pPvt->timeout=1.0,
> status=1,
> eomReason=0.
>
> I have simplified considerably the drvModbusAsynConfigure routine in my
> version of the modbus driver.  It must be that I have removed some code
> fragment that is needed so that the drvAsynIPPort driver gets the
> timeout value.  I have to confess that I tried a few times to understand
> how information is passed from one component to another in the asyn
> module and each time I walked away from the problem without getting to
> the bottom of it.
> I wonder if you could outline how the pPvt->timeout value gets passed on
> to the IPPort driver.  This might help me see what I am missing in my
> simplified version.
> In a nutshell, I kept only two routines from the drvModbusAsyn.c, the
> configure and doModbusIO routines.   I made only slight modifications to
> doModbusIO routine.  On the other hand I simplified the configure
> routine quite a lot.  I removed all device support from it and more.
> Sorry for troubling you with this,
> Zen
>
> On 08/05/13 17:45, Mark Rivers wrote:
>> The timeout used  by the drvAsynIPPort driver is the one passed in pasynUser->timeout, which is taken from pPvt->timeout in modbusInterpose.c.
>>
>> The pasynOctet->read() routine is the readIt function in drvAsynIPPort.c.
>>
>> When readIt fails what is the return code, and what is eomReason?
>>
>> Mark
>>
>> ________________________________________
>> From: Zenon Szalata [[email protected]]
>> Sent: Monday, August 05, 2013 3:01 PM
>> To: Mark Rivers
>> Cc: Williams Jr., Ernest L.; [email protected]
>> Subject: Re: Modbus
>>
>> Hi Mark,
>> I think that the timeout you refer to is controlled via the third
>> argument to modbusInterposeConfig routine.  This is set to 1000 for all
>> Beckhoff IOCs.
>> I have not used the asyn trace facility because there are large number
>> of IOs happening in one second.  I will try it.
>> I verified using a print statement in the readIt routine in
>> modbusInterpose.c that the pPvt->timeout is 1.0, so, if this is the real
>> timeout, then it is not used in the IO request.
>> Is there a timeout in the drvAsynIPPort driver?
>>
>> I am looking at modbusInterpose.c and I find it difficult to tell what
>> timeout the read routine is using.  Here is the line from modbusInterpose.c:
>>
>>          status = pPvt->pasynOctet->read(pPvt->octetPvt, pasynUser,
>>                           pPvt->buffer, nRead, &nbytesActual, eomReason);
>>
>> I can't tell how the pasyOctet->read routine is coded; not sure where it
>> is defined and what timeout is used in the routine which ultimately
>> issues the modbus read request.
>>
>> Now, in my new code, I put print statements in the writeIt and readIt
>> routines in the modbusInterpose.c file and I see that the readIt fails
>> immediately after writeIt completes with no perceivable delay.
>> So I am confused.
>>
>> In the case of the IOCs using your modbus support module, there could be
>> some other effect causing the error.
>> For better or worse in these IOCs, I am setting up the IOCs such that
>> drvModbusAsynConfigure is called for each Beckhoff bus terminal type.  I
>> use 100 ms period for the poller loop in each.  I think each thread
>> loops asynchronously with respect other threads. Therefore it is
>> conceivable that often a write is issued from one thread when the
>> coupler is not ready as it is processing some IO from another thread.  I
>> did not test this to see if this is indeed happening.
>>
>> To avoid this I have created a separate modbus IO class object in my new
>> code, which uses a message queue to coordinate orderly execution of IO
>> requests, which is done in a poller loop in this class. Requests for IO
>> from multiple threads are scheduled for execution via the message queue.
>> Thanks Mark,
>> Zen
>>
>>
>> On 08/05/13 10:23, Mark Rivers wrote:
>>> Hi Zen,
>>>
>>> What timeout are you using for your TCP read?  The read does not actually put anything on the Ethernet, it just waits for a message to arrive from the Beckoff.  So if you use a timeout of 1 second or so then it should never time out, assuming the module really does answer within 20 ms or so.
>>>
>>> You need to turn on asynTrace in the underlying drvAsynIPPort driver to see exactly the timing of what is being sent and received by that driver.
>>>
>>> Mark
>>>
>>> ________________________________________
>>> From: Zenon Szalata [[email protected]]
>>> Sent: Monday, August 05, 2013 11:33 AM
>>> To: Mark Rivers; Ernest Williams; [email protected]
>>> Subject: Modbus
>>>
>>> Hi Mark,
>>> I am nearly done with my version of modbus TCP for use with Beckhoff
>>> hardware.  I have a few IOCs using this support module and all seems to
>>> be working well in my test area.
>>> I have deployed an IOC for a couple of stepper motors controlled by
>>> Beckhoff devices in a production area and discovered that the IOC is
>>> generating many errors.
>>> The error is:
>>>
>>>      2013/08/05 08:55:14.982 modbusAsyn::doModbusIO port BKH15 error
>>> calling writeRead, error=sioc-esb-bkh15:502 timeout: Resource
>>> temporarily unavailable, nwrite=6/6, nread=0
>>>
>>> Now, modbusAsyn is the name I used for the file which has the two
>>> routines I took from your file drvModbusAsyn.  Looking at the log files
>>> of the IOCs used in the same production area which control Beckhoff
>>> devices and which are built with your modbus support module, I see that
>>> the same errors are generated.  Here is an example:
>>>
>>> 2013/06/16 13:19:25.918 drvModbusAsyn::doModbusIO port BKH10_CPL_R0
>>> error calling writeRead, error=sioc-esb-bkh10:502 timeout: Resource
>>> temporarily unavailable, nwrite=6/6, nread=0
>>>
>>> People who use these IOCs ignored these errors because mostly they did
>>> not matter.
>>>
>>> In the case of the stepper motor controller IOC it does matter. First of
>>> all I realized that the difference between my testing environment and
>>> that of production is the computer speed and a different networking
>>> environment.  The production computer used for the Beckhoff IOCs is much
>>> faster.  So it seems that there is some timing issue at work here.
>>> After putting print statements in various places, I see that the routine
>>> writeIt in modbusInterpose.c file always succeeds but the routine readIt
>>> in the same file fails from time to time.  I reread the manual for the
>>> Beckhoff BK9000 Ethernet coupler and came across a statement saying that
>>> the coupler takes from 13 to 15 ms to process a modbus message.  So it
>>> seems that the read comes too soon after the write.  In my version of
>>> the modbus support module for Beckhoff, I have added a 20 ms delay after
>>> the pasynOctet->write in modbusInterpose.c and this has mostly solved
>>> the problem.  I say mostly, because I still see this error message but
>>> very infrequently.
>>> I am writing all this because I don't understand why this error is
>>> generated so I am not sure if the direction I took to fix the problem is
>>> correct.  I also think that your modbus support module may need to be
>>> modified if it is to be used with Beckhoff devices.
>>> I would like to hear what you think about this.
>>> Thanks,
>>> Zen
>
>



References:
Modbus Zenon Szalata
RE: Modbus Mark Rivers
Re: Modbus Zenon Szalata
RE: Modbus Mark Rivers
Re: Modbus Zenon Szalata
RE: Modbus Mark Rivers
Re: Modbus Zenon Szalata

Navigate by Date:
Prev: RE: motor record: continuously update limit switches Mark Rivers
Next: motor record & PC6K: HOMF/HOMR broken Martin Konrad
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: Modbus Zenon Szalata
Next: fix for bug 1179642 Hill, Jeff
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 20 Apr 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·