Experimental Physics and
Industrial Control System

Michael Davidsaver via Core-talk <core-talk at aps.anl.gov> · Thu, 15 Dec 2022 10:14:14 -0800

On 12/14/22 23:48, Torsten Bögershausen wrote:

Hej again,

sorry for the noise.

It seems as if my script picked up the wrong library.
Not sure why, that is another issue.

The effect was, that I had 2 "virtual env" -
after updating both, it looks as if the problem has gone.
Re-testing on Mac to be done.

Thanks for the help and the patience, Michael.

No problem.  And thank you for reporting this issue.
I'll be pushing out PVXS 1.1.1 soon.

... I have been running tests with p4p for years. ...

I'm wondering about the circumstances as well.  This
was not a small bug.  Although it won't manifest
when the peer is also PVXS (or Kay's core.pva I think)
due to implementation specific decisions on which
structure definitions are "cached".

Perhaps this hasn't been reported previously because,
this error happens to be recoverable.  The decode error
occurs at a point in the protocol where PVXS can safely
retry the operation over a new TCP connection, with an
empty "cache", where it may succeed (as happened during
the packet capture you provided).

Still, the idea that people have not been reporting "CRIT"
errors with stack traces gives me pause.  So again,
thank you for bringing this up.

/Torsten

On 12/15/22 06:10, Michael Davidsaver wrote:

On 12/14/22 20:57, Torsten Bögershausen wrote:

Hej Michael,
I am not sure, what is going on:

Were you able to update pvxslibs to 1.1.1a1?

pip install pvxslibs==1.1.1a1
pip freeze | grep pvxs

$ find . -type f -name libpvxs* | xargs file
./lib/python3.7/site-packages/pvxslibs/lib/libpvxs.so:     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb7ba08c0db385deef78ba0dd1114b8e2d1c5183, not stripped
./lib/python3.7/site-packages/pvxslibs/lib/libpvxs.so.1.1: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb7ba08c0db385deef78ba0dd1114b8e2d1c5183, not stripped
===========================================
And the dump in the log file:
2022-12-15T05:37:37.695799013 DEBUG pvxs.client.io UDP search Rx 53 from 192.168.209.75:43759
2022-12-15T05:37:37.695808594 DEBUG pvxs.client.io Search reply for IOC:m1.CNEN
2022-12-15T05:37:37.695814768 DEBUG pvxs.client.io Server 192.168.209.75:5075 creating channel 'IOC:m1.CNEN' (305419897)
2022-12-15T05:37:37.695820391 DEBUG pvxs.client.io UDP search processed 1/40
0000 : CA024007 09000000
2022-12-15T05:37:37.739627299 DEBUG pvxs.tcp.io Server 192.168.209.75:5075 Receive header
2022-12-15T05:37:37.739653998 DEBUG pvxs.client.io Server 192.168.209.75:5075 active channel to 'IOC:m1.CNEN' 305419897:302003202
2022-12-15T05:37:37.739664504 DEBUG pvxs.client.io Server 192.168.209.75:5075 channel 'IOC:m1.CNEN' op0a INIT
0000 : CA02400A 3E000000
2022-12-15T05:37:37.739864291 DEBUG pvxs.tcp.io Server 192.168.209.75:5075 Receive header
2022-12-15T05:37:37.739970136 CRIT pvxs.tcp.io Server Error while processing cmd 0x0a: vector::_M_range_check
Dumping a stack trace of thread 'PVXCTCP':
[    0x7fb706845b6b]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../epicscorelibs/lib/libCom.so.7.0.7.99.0(epicsStackTrace+0x4b)
[    0x7fb706b324b5]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.1(_ZN4pvxs6detail11_log_printfEjPKcz+0xa5)
[    0x7fb706af4cf6]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.1(_ZN4pvxs4impl8ConnBase7bevReadEv+0x696)
[    0x7fb706af5031]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.1(_ZN4pvxs4impl8ConnBase8bevReadSEP11buffereventPv+0x21)
[    0x7fb706465432]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/./libevent_core.so.2.2.0(bufferevent_run_deferred_callbacks_locked+0x82)
[    0x7fb70645edb6]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/./libevent_core.so.2.2.0(event_process_active_single_queue.isra.33+0x446)
[    0x7fb70645f75f]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/./libevent_core.so.2.2.0(event_base_loop+0x49f)
[    0x7fb706b68e00]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.1(_ZN4pvxs4impl6evbase3Pvt3runEv+0x160)
[    0x7fb70685ef69]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../epicscorelibs/lib/libCom.so.7.0.7.99.0(epicsThreadCallEntryPoint+0x69)
[    0x7fb70684a8da]: .../ethercatmc/test/motionTest/venvpython3.7/lib/python3.7/site-packages/p4p/../epicscorelibs/lib/libCom.so.7.0.7.99.0(start_routine+0xda)
[    0x7fb7093d2fa3]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xf3)
[    0x7fb70911a06f]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)
2022-12-15T05:37:37.742282270 DEBUG pvxs.client.io Server 192.168.209.75:5075 detach channel 'IOC:m1.RBV' to re-search
2022-12-15T05:37:37.742290971 DEBUG pvxs.client.io Server 192.168.209.75:5075 detach channel 'IOC:m1.CNEN' to re-search
2022-12-15T05:37:37.742293949 DEBUG pvxs.client.io Cleaning connection to 192.168.209.75:5075
2022-12-15T05:37:38.699371065 DEBUG pvxs.client.io Search tick 2
0000 : CA028003 00000050 66696E64 00000000
0010 : 00000000 00000000 00000000 00000000
0020 : CE6D0103 74637000 03123456 790B494F
0030 : 433A6D31 2E434E45 4E123456 780A494F

Thanks
/Torsten

On 14 Dec 2022, at 23:14, Michael Davidsaver <mdavidsaver at gmail.com> wrote:

On 12/14/22 09:14, Torsten Bögershausen wrote:

Can you collect a packet capture?

There is even a packet trace, done on the Mac.
Don't be confuse about the IP-addresses, its all on localhost.
See the run from this morning:
ls -1 2022-12-14-1*
2022-12-14-1038.pcapng
2022-12-14-1038.txt

 From the packet capture, I find a message decoding bug.

https://github.com/mdavidsaver/pvxs/issues/32

I'm uploading pvxslibs==1.1.1a1 which should
include a fix.  Please re-test.

https://pypi.org/project/pvxslibs/1.1.1a1/

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System