Experimental Physics and
Industrial Control System

"J. Lewis Muir via Core-talk" <core-talk at aps.anl.gov> · Fri, 16 Apr 2021 11:03:55 -0500

On 04/16, Michael Davidsaver via Core-talk wrote:
> On 4/16/21 5:29 AM, Ben Franksen via Core-talk wrote:
> > Am 16.04.21 um 12:59 schrieb Zimoch Dirk (PSI) via Core-talk:
> >> Today I pushed a merge request which caused several tests to fail at locations which are unrelated to my change.
> >> https://github.com/dirk-zimoch/epics-base/actions/runs/754804587
> >> Is there something wrong with the test setup? Is the current 7.0 head OK?
> > 
> > This came up in the recent meeting... the notes merely quote MAD as
> > stating "We should document the tests that are regularly failing,
> > suggest adding a GH issue for this." My preference would be to fix them
> > (either our code, or the tests if the latter are at fault) ...
> 
> To be clear.  Tests which always fail are fixed or removed.
> 
> However, there are some tests which occasionally fail due to
> incomplete or incorrect synchronization.  I think that these
> have some value, though it is diminished, and shouldn't be
> removed unless they become too noisy.

If they occasionally fail, they shouldn't be enabled IMO unless someone
is intending to fix them.  If someone is intending to fix them in the
near future and leaving them enabled helps to reproduce the failure,
that's fine.  But if no one is intending to fix them soon, then they
should be disabled.

> There are also few inexplicable crash on exit of otherwise successful
> test runs.  eg. I've seen epicsTypeTest do this a few times, which
> certainly has nothing to do the test itself.

This is sad and would be good to track down.

> I've been (slowly) adding instrumentation to try to get some more
> information on what is going on.  This appears in the GHA builds
> under "Post Automatic core dumper analysis".

This is much appreciated!

> The idea of documenting which tests have problems is to help
> with the first part.  Currently the list of troublesome tests
> mostly exists in the minds of those of us who routinely look
> at test results.

Documenting tests that sometimes fail is madness.  That's like having
an application that emits error messages and then telling users to
ignore certain error messages.  So now not all error messages are really
errors.  Instead of seeing an error message and knowing there's a
problem, now the user has to check to see whether it's an error message
that should be ignored.  That's a terrible path to go down.

Lewis

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System