Re: Base CI tests on MacOS failing

[Torsten Bögershausen via Core-talk <core-talk at aps.anl.gov>]

On 4/27/20 3:01 PM, Ralph Lange via Core-talk wrote:
> Dear all,
>
> The 3.15 and 7.0 branches are now switched over to use the shared 
> ci-scripts module for their Travis-CI and AppVeyor builds.
> On Travis, the MacOS builds fail their tests. The epicsTimerTest fails 
> the 5% error threshold in some of its cases. Typically like this:
>     epicsTimerTest.tap ........... 1/41
>     not ok  4 - 5.604376 < 5.000000, delay = 1.100000 s, error =
>     0.061648 s (5.6 %)
>     not ok  5 - 12.089890 < 5.000000, delay = 1.200000 s, error =
>     0.145079 s (12.1 %)
>     not ok  7 - 9.455878 < 5.000000, delay = 1.400000 s, error =
>     0.132382 s (9.5 %)
>     not ok 16 - 5.582973 < 5.000000, delay = 2.300000 s, error =
>     0.128408 s (5.6 %)
>     not ok 20 - 5.313693 < 5.000000, delay = 2.700000 s, error =
>     0.143470 s (5.3 %)
>     epicsTimerTest.tap ........... Failed 5/41 subtests
>
>
> This seems to have gotten worse as the shared ci-scripts now run the 
> tests on Travis-CI using parallel make (as on AppVeyor/Windows) - the 
> former scripts were running the tests serially.
> How should we proceed? Timing is notoriously weird on CI builder 
> machines - should this test be run with higher thresholds on CI? Or be 
> moved to the performance tests (not run as part of the build at all)?
Salut Ralph,
My spontaneous feeling, with the risk of being wrong: Try to run them.

What is the difference between a highly-loaded (soft) IOC and a machine
running ci scripts, probably highly loaded as well ?
May be not so much.

As we have 25% VAT here, taking half of it as the threshold could work ?
12.5 % ?
/Torsten


> Cheers,
> ~Ralph