RE: Slow TCP/IP Modbus and data lose

["Engbretson, Mark S. via Tech-talk" <tech-talk at aps.anl.gov>]

It certainly made a difference . . . 

Thanks!

-----Original Message-----
From: Mark Rivers <rivers at cars.uchicago.edu> 
Sent: Thursday, September 10, 2020 1:41 PM
To: Engbretson, Mark S. <engbretson at anl.gov>; tech-talk at aps.anl.gov (tech-talk at aps.anl.gov) <tech-talk at aps.anl.gov>
Subject: RE: Slow TCP/IP Modbus and data lose 

Hi Mark,

What is the hardware connection?  Is it serial, with or without an Ethernet terminal server?

If so, I think that Nicholas DiMonte at the APS recently had the same problem.  This is what I told him, and it fixed it.

*************************************
Hi Nicholas,

I have looked more closely at the trace debug with an without the delay, and I'm pretty sure I know what the problem is.  It is not the read timeout I mentioned previously.  I need to see your st.cmd (renamed to .txt) to be sure.

I believe your drvAsynIPPortConfigure command has the final argument, noProcessEos=1.  That is normally what one wants for Modbus TCP because it does not use EOS terminators.

However, the asynInterposeEos processing actually does 2 things on read operations:

- It terminates the read if the EOS terminator is found.
- It terminates the read when the requested number of characters has been received.  If the lower-level device (TCP or serial port) returns fewer bytes than requested, then the interpose layer keeps issuing read operations to the lower level driver until it gets the requested number of characters, or the EOS, or a timeout.

The low-level TCP driver always returns immediately with whatever data is read from the socket with recv().  That may be fewer than the requested number of bytes.

With "real" Modbus TCP devices you don't need to use the interposeEos layer, because the device always returns the data in a single TCP packet.  There is no need for the interposeEos layer to buffer the data and issue multiple read requests.

However, your don't have a real TCP device, you have a Lantronix terminal server.  The problem is that the Lantronix is not sending the response from the device in a single TCP packet, it is breaking it up into multiple TCP packets.  So you do need the asynInterposeEos layer to buffer the input and issue multiple read requests to the TCP driver.

The solution to the problem is simple.  Just change noProcessEos from 1 to 0 in drvAsynIPPortConfigure.  That should then work with writeDelay=0, and without your hack to the code.

*************************************

If you are using a real serial port with drvAsynSerialPortConfigure you will have the same problem.  It will return whenever any characters are available, so you need to use noProcessEos=0 to enable EOS processing.  That will wait until the expected number of characters has been received, or a timeout occurs.

Mark


-----Original Message-----
From: Engbretson, Mark S. <engbretson at anl.gov> 
Sent: Thursday, September 10, 2020 1:26 PM
To: tech-talk at aps.anl.gov (tech-talk at aps.anl.gov) <tech-talk at aps.anl.gov>
Cc: Mark Rivers <rivers at cars.uchicago.edu>
Subject: Slow TCP/IP Modbus and data lose 

Mark? Question

I have an Teledyne D-Series Syringe Pump that speaks modbus. It has 142 read/write data bits , 80 read/write 2 register Floats, and 22 read only 2 register floats.

Since the data size is larger than modbus supports, I have broken it up into smaller reads and writes that should be of legal size. Doing tests on any given object seems to work, most of the time.

What happens is that the expected data tends to return 'short', and I see a flush happening to discard  bytes that are either appearing out of order or delayed. So the data that I am trying to decode tends to be missing data off the end.

If I make the request small enough, I might get more packets that survive intact, but then I also have to do more reads to get all the data. At some point, I start losing bytes again in specific requests; flushing more data, and if this goes on for a while, I will end up dropping the connection to the hardware , but I never reach a point where I am sure that all of the data that this hardware exposes is 'real',  much less trusting trying to write settings back to it.

You have an idea of what might be causing this data to be returned out of order or what might be a work around? I am assuming that this hardware just does not have the processing power  to keep up with the requests and I don't know how to dumb the code down to a request and data rate that it can keep up with. Suggestions?

The basic command template  that I am using is some variation of this

drvModbusAsynConfigure("$(PORT)_Reg_In_1",  "$(PORT)", 1,  3, 0, 20, 8, 1000, "Teledyne")

with a substitution table like

file "$(MODBUS)/db/aiFloat64.template" { pattern
{      P,           R,            PORT,  OFFSET,   DATA_TYPE,  LOPR,  HOPR, PREC,   EGU,       SCAN,   DESC}
{16bmSP:, reg:0_RBV,     SP1_Reg_In_1,    0,     FLOAT32_BE,   0,    10,    4,  egu, "I/O Intr", "Register"}
. . .
}

Trace info

2020/09/09 14:38:20.270 164.54.105.209:502 flushed 21 bytes
2020/09/09 14:38:20.280 164.54.105.209:502 write 12

00 02 00 00 00 06 01 03 00 00 00 32
2020/09/09 14:38:20.316 164.54.105.209:502 read 80

00 02 00 00 00 67 01 03 64 43 4f 00 00 43 4f 00 00 43 4f 00
00 43 4f 00 00 43 4f 00 00 43 4f 00 00 41 b0 00 00 41 b0 00
00 38 d1 b7 17 38 d1 b7 17 41 b0 00 00 38 d1 b7 17 41 e8 00
00 41 e8 00 00 00 00 00 00 00 00 00 00 48 49 fe c0 48 49 fe
2020/09/09 14:38:24.316 164.54.105.209:502 flushed 29 bytes
2020/09/09 14:38:24.327 164.54.105.209:502 write 12

00 03 00 00 00 06 01 03 00 00 00 32
2020/09/09 14:38:24.366 164.54.105.209:502 read 80

00 03 00 00 00 67 01 03 64 43 4f 00 00 43 4f 00 00 43 4f 00
00 43 4f 00 00 43 4f 00 00 43 4f 00 00 41 b0 00 00 41 b0 00
00 38 d1 b7 17 38 d1 b7 17 41 b0 00 00 38 d1 b7 17 41 e8 00
00 41 e8 00 00 00 00 00 00 00 00 00 00 48 49 fe c0 48 49 fe
2020/09/09 14:38:28.366 164.54.105.209:502 flushed 29 bytes
2020/09/09 14:38:28.376 164.54.105.209:502 write 12

00 04 00 00 00 06 01 03 00 00 00 32
2020/09/09 14:38:28.416 164.54.105.209:502 read 80

00 04 00 00 00 67 01 03 64 43 4f 00 00 43 4f 00 00 43 4f 00
00 43 4f 00 00 43 4f 00 00 43 4f 00 00 41 b0 00 00 41 b0 00
00 38 d1 b7 17 38 d1 b7 17 41 b0 00 00 38 d1 b7 17 41 e8 00
00 41 e8 00 00 00 00 00 00 00 00 00 00 48 49 fe c0 48 49 fe
2020/09/09 14:38:32.416 164.54.105.209:502 flushed 29 bytes
2020/09/09 14:38:32.426 164.54.105.209:502 write 12

00 05 00 00 00 06 01 03 00 00 00 32
2020/09/09 14:38:32.465 164.54.105.209:502 read 80

00 05 00 00 00 67 01 03 64 43 4f 00 00 43 4f 00 00 43 4f 00
00 43 4f 00 00 43 4f 00 00 43 4f 00 00 41 b0 00 00 41 b0 00
00 38 d1 b7 17 38 d1 b7 17 41 b0 00 00 38 d1 b7 17 41 e8 00
00 41 e8 00 00 00 00 00 00 00 00 00 00 48 49 fe c0 48 49 fe
2020/09/09 14:38:36.466 164.54.105.209:502 flushed 29 bytes
2020/09/09 14:38:36.476 164.54.105.209:502 write 12

00 06 00 00 00 06 01 03 00 00 00 32
2020/09/09 14:38:36.515 164.54.105.209:502 read 72

00 06 00 00 00 67 01 03 64 43 4f 00 00 43 4f 00 00 43 4f 00
00 43 4f 00 00 43 4f 00 00 43 4f 00 00 41 b0 00 00 41 b0 00
00 38 d1 b7 17 38 d1 b7 17 41 b0 00 00 38 d1 b7 17 41 e8 00
00 41 e8 00 00 00 00 00 00 00 00 00
2020/09/09 14:38:40.515 164.54.105.209:502 flushed 37 bytes
2020/09/09 14:38:40.526 164.54.105.209:502 write 12

00 07 00 00 00 06 01 03 00 00 00 32
2020/09/09 14:38:40.565 164.54.105.209:502 read 72

00 07 00 00 00 67 01 03 64 43 4f 00 00 43 4f 00 00 43 4f 00
00 43 4f 00 00 43 4f 00 00 43 4f 00 00 41 b0 00 00 41 b0 00
00 38 d1 b7 17 38 d1 b7 17 41 b0 00 00 38 d1 b7 17 41 e8 00
00 41 e8 00 00 00 00 00 00 00 00 00