Experimental Physics and
Industrial Control System

Michael Davidsaver via Core-talk <core-talk at aps.anl.gov> · Fri, 16 Apr 2021 08:25:54 -0700

On 4/16/21 5:29 AM, Ben Franksen via Core-talk wrote:
> Am 16.04.21 um 12:59 schrieb Zimoch Dirk (PSI) via Core-talk:
>> Today I pushed a merge request which caused several tests to fail at locations which are unrelated to my change.
>> https://github.com/dirk-zimoch/epics-base/actions/runs/754804587
>> Is there something wrong with the test setup? Is the current 7.0 head OK?
> 
> This came up in the recent meeting... the notes merely quote MAD as
> stating "We should document the tests that are regularly failing,
> suggest adding a GH issue for this." My preference would be to fix them
> (either our code, or the tests if the latter are at fault) ...

To be clear.  Tests which always fail are fixed or removed.

However, there are some tests which occasionally fail due to
incomplete or incorrect synchronization.  I think that these
have some value, though it is diminished, and shouldn't be
removed unless they become too noisy.

There are also few inexplicable crash on exit of otherwise successful
test runs.  eg. I've seen epicsTypeTest do this a few times, which
certainly has nothing to do the test itself.

I've been (slowly) adding instrumentation to try to get some more
information on what is going on.  This appears in the GHA builds
under "Post Automatic core dumper analysis".

https://github.com/mdavidsaver/ci-core-dumper

eg. In a failing windows test which Dirk links testCaProvider has crashed

>   00 000000a4`b335f810 00007ff9`166bed8e Com!freeListFree+0x14
>   01 000000a4`b335f840 00007ff9`177ae0a1 dbCore!cvtRawToEngBpt+0x226e
>   02 000000a4`b335f870 00007ff9`28531c04 ca!ca_clear_channel+0x201
>   03 000000a4`b335f8d0 00007ff9`28529a7b pvAccessCA!epics::pvAccess::ca::CAClientFactory::start+0x8f14
>   04 000000a4`b335f9c0 00007ff9`28521594 pvAccessCA!epics::pvAccess::ca::CAClientFactory::start+0xd8b
>   05 000000a4`b335fa50 00007ff7`fabdbfc0 pvAccessCA!epics::pvAccess::ca::NotifierConveyor::NotifierConveyor+0x134
>   06 000000a4`b335fa80 00007ff9`47ff1cf6 testCaProvider+0xbfc0
...

https://github.com/dirk-zimoch/epics-base/runs/2360148130?check_suite_focus=true#step:19:93

> ... but I have no
> idea how to do that.

Pick a failing test, roll up your virtual sleeves and dig it!

The idea of documenting which tests have problems is to help
with the first part.  Currently the list of troublesome tests
mostly exists in the minds of those of us who routinely look
at test results.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System