Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <2020 Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <2020
<== Date ==> <== Thread ==>

Subject: RE: Ethernet question
From: Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>
To: 'Heinz Junkes' <junkes@fhi-berlin.mpg.de>, 'Michael Davidsaver' <mdavidsaver@gmail.com>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Wed, 1 Jan 2020 17:42:19 +0000

 

Michael wrote;

 

Ø  Do these devices use Jumbo frames other features which are not not standard Ethernet?

 

They are very old devices (early 1990’s).  They are 10-Mbit with AUI connectors.  They do not use Jumbo packets.

 

Ø  However, what you describe (no reply) for some messages sounds like problems I've have with jumbo frames.  The short replies get through, but the long replies are ignored.

 

Heinz wrote:

Ø  I thought also about the MTU right away, just like Michael. With 10Gbit/s Ethernet this is 9000 bytes as default and not 1500 bytes as with the 1Gbit/s Ethernet.

I don’t think the problem is MTU.  Here are the reasons:

 

-          I know the protocol well enough to know that the device never sends packets over ~1500 bytes.

-          I can communicate with the device fine over a 1 Gbit NIC on Linux with MTU set to 1500 or 9000.  So setting MTU to 9000 on a 1 Gbit NIC does not prevent it from working.

-          I cannot communicate with the device over a 10 Gbit NIC on Linux with MTU set to 1500.

 

 

This is the part of the communication that works on a 1 Gbit NIC.  It is requesting information on the attached Instrument Control Bus devices.

 

First it sends a 108 byte packet (130 bytes with header) and gets a 94 byte response.

 

It then sends a 172 byte packet (194 bytes with header) and gets a 78 byte response.

 

nmc_sendcmd): enter module=2

nmc_check_module enter

(nmc_sendcmd): calling nmc_putmsg to write 108 bytes packet=

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f2 66 03 af 01 00 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4c 00 00 00 00 00 00 00 00 00 44 00 00 00 01 00 16 00 20 00 00 00 00 01 10 11 20 21 30 31 40 41 50 51 60 61 70 71 80 81 90 91 a0 a1 b0 b1 c0 c1 d0 d1 e0 e1 f0 f1 ff 7f 00 00 45 2a 9e 29 1b 7f

(nmc_putmsg): wrote 130 bytes of 130

(nmcEtherGrab): got a 94 byte packet from AIM

(nmcEtherGrab): ...response packet

nmc_findmod_by_addr enter

(nmcEtherGrab): sending 94 bytes to module 2 (0x7f1b1c006290)

(nmc_getmsg): message length:94 (0x7f1b1c006290)

(nmc_sendcmd): received 32 bytes message=

ff ff ff ff 02 f4 02 f7 ff ff 02 f6 ff ff ff ff 02 f2 ff ff ff ff ff ff ff ff ff ff ff ff ff ff

../icb_control_subs2.c(110):icb_poll_controller: address=NI3ED:2, type=2 (2)

nmc_findmod_by_addr enter

(nmc_sendcmd): enter module=2

nmc_check_module enter

(nmc_sendcmd): calling nmc_putmsg to write 172 bytes packet=

00 00 00 00 00 00 00 00 00 00 00 00 00 00 aa aa 03 00 00 af 90 51 f2 66 03 af 01 00 02 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 8c 00 00 00 00 00 00 00 00 00 84 00 00 00 01 00 15 00 01 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

(nmc_putmsg): wrote 194 bytes of 194

(nmcEtherGrab): got a 78 byte packet from AIM

(nmcEtherGrab): ...response packet

nmc_findmod_by_addr enter

(nmcEtherGrab): sending 78 bytes to module 2 (0x7f1b1c006290)

(nmc_getmsg): message length:78 (0x7f1b1c006290)

 

This is the same part of the communication on a 10 Gbit NIC with MTU=1500 that does not work.

 

First it sends a 108 byte packet (130 bytes with header) and gets a 94 byte response. This is correct, and is the same as the working 1 Gbit NIC above.

 

It then sends a 172 byte packet (194 bytes with header) but does not receive any response.  It tries again because the first attempt timed out, but still gets no response.

 

(nmc_sendcmd): enter module=0

nmc_check_module enter

(nmc_sendcmd): calling nmc_putmsg to write 108 bytes packet=

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f2 66 03 af 01 00 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4c 00 00 00 00 00 00 00 00 00 44 00 00 00 01 00 16 00 20 00 00 00 00 01 10 11 20 21 30 31 40 41 50 51 60 61 70 71 80 81 90 91 a0 a1 b0 b1 c0 c1 d0 d1 e0 e1 f0 f1 00 00 00 00 00 00 00 00 00 00

(nmc_putmsg): wrote 130 bytes of 130

(nmcEtherGrab): got a 94 byte packet from AIM

(nmcEtherGrab): ...response packet

nmc_findmod_by_addr enter

(nmcEtherGrab): sending 94 bytes to module 0 (0x7f22f0000950)

(nmc_getmsg): message length:94 (0x7f22f0000950)

(nmc_sendcmd): received 32 bytes message=

ff ff ff ff 02 f4 02 f7 ff ff 02 f6 ff ff ff ff 02 f2 ff ff ff ff ff ff ff ff ff ff ff ff ff ff

../icb_control_subs2.c(110):icb_poll_controller: address=NI3ED:2, type=2 (2)

nmc_findmod_by_addr enter

nmc_findmod_by_addr enter

(nmc_sendcmd): enter module=0

nmc_check_module enter

(nmc_sendcmd): calling nmc_putmsg to write 172 bytes packet=

00 00 00 00 00 00 00 00 00 00 00 00 00 00 aa aa 03 00 00 af 50 64 f2 66 03 af 01 00 02 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 8c 00 00 00 00 00 00 00 00 00 84 00 00 00 01 00 15 00 01 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

(nmc_putmsg): wrote 194 bytes of 194

(nmc_getmsg): message length:-1 (0x7f22f0000950)

(nmc_getmsg): timeout while waiting for message

nmc_signal called by 'nmc_getmsg' with error 0 = 0x0

(nmc_sendcmd): tries=0, max_tries=3, timeout_time=1000.000000

(nmc_sendcmd): calling nmc_putmsg to write 172 bytes packet=

00 00 00 00 00 00 00 00 00 00 00 00 00 00 aa aa 03 00 00 af 50 64 f2 66 03 af 01 00 02 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 8c 00 00 00 00 00 00 00 00 00 84 00 00 00 01 00 15 00 01 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

(nmc_putmsg): wrote 194 bytes of 194

(nmc_getmsg): message length:-1 (0x7f22f0000950)

(nmc_getmsg): timeout while waiting for message

nmc_signal called by 'nmc_getmsg' with error 0 = 0x0

(nmc_sendcmd): tries=1, max_tries=3, timeout_time=1000.000000

 

This is ifconfig on the 10Gbit NIC that is not working:

 

p5p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

       inet 164.54.160.82  netmask 255.255.255.0  broadcast 164.54.160.255

        inet6 fe80::3efd:feff:fea3:f258  prefixlen 64  scopeid 0x20<link>

        ether 3c:fd:fe:a3:f2:58  txqueuelen 1000  (Ethernet)

        RX packets 142371854664  bytes 104792989809154 (95.3 TiB)

        RX errors 0  dropped 920  overruns 0  frame 217

        TX packets 99116666961  bytes 29259207664411 (26.6 TiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

tcpdump agrees with the driver debugging printout, the second packet was sent multiple times but no response was seen. Here is the tcpdump output.  The first packet is the 108 byte message sent from Linux to the device, and the second packet is the 72 byte response received about 50 ms later.  Then there are 3 172-byte messages sent by Linux with no reply.  These messages are sent 1.00 seconds apart because that is the timeout.

 

11:30:18.680030 3c:fd:fe:a3:f2:58 (oui Unknown) > 00:00:af:00:03:ed (oui Unknown), 802.3, length 116: LLC, dsap SNAP (0xaa) Individual, ssap

SNAP (0xaa) Command, ctrl 0x03: oui Unknown (0x0000af), pid Unknown (0x5034), length 108:

        0x0000:  aaaa 0300 00af 5034 f266 03af 0100 0101  ......P4.f......

        0x0010:  0000 0000 0000 0000 0000 0000 0000 4c00  ..............L.

        0x0020:  0000 0000 0000 0000 4400 0000 0100 1600  ........D.......

        0x0030:  2000 0000 0001 1011 2021 3031 4041 5051  .........!01@APQ

        0x0040:  6061 7071 8081 9091 a0a1 b0b1 c0c1 d0d1  `apq............

        0x0050:  e0e1 f0f1 0000 0000 0000 0000 0000 0000  ................

        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0070:  ffff ffff                                ....

11:30:18.733989 00:00:af:00:03:ed (oui Unknown) > 3c:fd:fe:a3:f2:58 (oui Unknown), 802.3, length 80: LLC, dsap SNAP (0xaa) Individual, ssap S

NAP (0xaa) Command, ctrl 0x03: oui Unknown (0x0000af), pid Unknown (0x5034), length 72:

        0x0000:  aaaa 0300 00af 5034 f266 03af 0100 0101  ......P4.f......

        0x0010:  3cfd fea3 f258 636f 7276 6574 7465 2800  <....Xcorvette(.

        0x0020:  0000 0000 0000 0000 2000 0000 0200 0900  ................

        0x0030:  ffff ffff 02f4 02f7 ffff 02f6 ffff ffff  ................

        0x0040:  02f2 ffff ffff ffff ffff ffff ffff ffff  ................

11:30:18.734415 3c:fd:fe:a3:f2:58 (oui Unknown) > 00:00:af:00:03:ed (oui Unknown), 802.3, length 180: LLC, dsap SNAP (0xaa) Individual, ssap

SNAP (0xaa) Command, ctrl 0x03: oui Unknown (0x0000af), pid Unknown (0x5034), length 172:

        0x0000:  aaaa 0300 00af 5034 f266 03af 0100 0201  ......P4.f......

        0x0010:  0000 0000 0000 0000 0000 0000 0000 8c00  ................

        0x0020:  0000 0000 0000 0000 8400 0000 0100 1500  ................

        0x0030:  0100 0000 2000 0000 0000 0000 0000 0000  ................

        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x00b0:  0000 0000                                ....

11:30:19.735073 3c:fd:fe:a3:f2:58 (oui Unknown) > 00:00:af:00:03:ed (oui Unknown), 802.3, length 180: LLC, dsap SNAP (0xaa) Individual, ssap

SNAP (0xaa) Command, ctrl 0x03: oui Unknown (0x0000af), pid Unknown (0x5034), length 172:

        0x0000:  aaaa 0300 00af 5034 f266 03af 0100 0201  ......P4.f......

        0x0010:  0000 0000 0000 0000 0000 0000 0000 8c00  ................

        0x0020:  0000 0000 0000 0000 8400 0000 0100 1500  ................

        0x0030:  0100 0000 2000 0000 0000 0000 0000 0000  ................

        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x00b0:  0000 0000                                ....

11:30:20.735792 3c:fd:fe:a3:f2:58 (oui Unknown) > 00:00:af:00:03:ed (oui Unknown), 802.3, length 180: LLC, dsap SNAP (0xaa) Individual, ssap

SNAP (0xaa) Command, ctrl 0x03: oui Unknown (0x0000af), pid Unknown (0x5034), length 172:

        0x0000:  aaaa 0300 00af 5034 f266 03af 0100 0201  ......P4.f......

        0x0010:  0000 0000 0000 0000 0000 0000 0000 8c00  ................

        0x0020:  0000 0000 0000 0000 8400 0000 0100 1500  ................

        0x0030:  0100 0000 2000 0000 0000 0000 0000 0000  ................

        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x00b0:  0000 0000                                ....

 

I’m stumped.  Clearly some communication is working, because we got a response to the first message.  However, we did not receive a response to the second message, even though the transmitted packet appears to be the same (except for expected differences).  These packets are very small, not near the size where MTU should make a difference.

 

Thanks,

Mark

 

 

From: Heinz Junkes <junkes@fhi-berlin.mpg.de>
Sent: Tuesday, December 24, 2019 5:49 AM
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: tech-talk@aps.anl.gov
Subject: Re: Ethernet question

 

Hello, Mark,


I thought also about the MTU right away, just like Michael. With 10Gbit/s Ethernet this is 9000 bytes 

as default and not 1500 bytes as with the 1Gbit/s Ethernet.
We had similar problems with a small Omron PLC. It only supported 10Mbit/s and it could handle
UDP even only with a MTU of 768 bytes. Omron's internal buffer was simply designed too small. 

Here we had to connect a router to make it work.

Heinz  

 

P.S.

 

I found this in 82599-datasheet-v3-4.pdf:

 

 



On 23. Dec 2019, at 23:46, Mark Rivers via Tech-talk <tech-talk@aps.anl.gov> wrote:

Folks,
 
I am having a strange problem which I am wondering if someone might shed some light on.
 
I have some old Ethernet data acquisition devices, the Canberra AIM MCAs.  These are not TCP/IP, but rather use the IEEE 802.2 Extended SNAP protocol.  I have been talking to these devices for many years from vxWorks, Windows (using WinPCAP) and Linux (using libnet and libpcap).  I recently found that I was not able to talk to them from a number of my Linux systems.  It partly works: I can build a list of the devices on the network, I can take “ownership” of the device, and I can read what auxiliary Instrument Control Bus devices are connected to it (ADCs, amplifiers, HVPS, etc.).  However, some messages I send to the device never get answered.  When I look with Wireshark and compare the packet sent with a Windows machine and the non-functioning Linux machine the packets look the same.  But the Windows machine gets a reply and Linux does not.
 
On further investigation I found that the problem only occurs when using 10 Gbit Ethernet adapters.  When using a 1 Gbit adapter it works fine.  I was able to see this in a single machine that has both 10 Gbit and 1 Gbit adapters.  If I use the 1 Gbit adapter it works fine, if I use the 10 Gbit adapter it fails (but kind of works as described above).  On 3 separate systems 1 Gbit works, and 3 other systems 10 Gbit fails.
 
This does not make sense to me.  The Canberra device itself is 10 Mbit Ethernet.  It is plugged into a 1 Gbit switch.  The packets are all “slowed-down” in the switch to 10 Mbit, so I don’t understand why it matters if the network card is 1 Gbit or 10 Gbit.
 
I also used ethtool to lower the speed/duplex of the 10000T interface to 1000/Full and then to 100/Full and that did not help.  I could still see the module, but much functionality does not work.  So a 1 Gbit adapter works, but a 10 Gbit adapter slowed down to 1 Gbit or even 100 Mbit does not work.
 
Any ideas what can cause these symptoms?
 
Thanks,
Mark

 


Replies:
RE: Ethernet question Mark Rivers via Tech-talk

Navigate by Date:
Next: RE: Ethernet question Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <2020
Navigate by Thread:
Next: RE: Ethernet question Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  <2020
ANJ, 01 Jan 2020 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·