Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  Index 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022 
<== Date ==> <== Thread ==>

Subject: ringPointerTest
From: Andrew Johnson <anj@aps.anl.gov>
To: EPICS core-talk <core-talk@aps.anl.gov>
Date: Mon, 6 Oct 2014 12:09:46 -0500
The instability that Jenkins has been reporting on the 3.15 branch is
due to test failures in ringPointerTest, and I've just run the tests on
Solaris (various unrelated commits to come that fix building here) and
it's failing there too (with more failures than Jenkins is seeing).
There were changes to this test program in the parallel-cbthreads-3
branch, so I'm guessing that's what introduced the problem.

Note that our Jenkins server still has only one CPU, as does our Solaris
box venus, so this might be related to that. Full Solaris results are
given below.

BTW I'm not very happy about test programs that produce a variable
number of results in the event of test failures, as this one does.
Please don't write tests that call testPass() or testFail() within a
loop that might finish early (or late) resulting in fewer (more) results
than your plan said it should.

Giving a different number of test results than planned won't always flag
a test failure (the Jenkins TAP parser isn't perfect either). Consider
the possibility that one loop might finish early and another later than
expected, with the result being the correct number of tests passing even
though the software is not behaving as it should.

It's much better to introduce loop counters and check the value of those
counters at the end of the loop (this also makes the test program output
less verbose and easier to find the failures in, especially on
VxWorks/RTEMS).

- Andrew


venus% ./ringPointerTest
1..256
# ******************************************************
# ** Test 1: local ring pointer, check size and order **
# ******************************************************
ok  1 - Ring empty
ok  2 - ring filled, 10 values
ok  3 - Pop test: 0 == 0
ok  4 - Pop test: 1 == 1
ok  5 - Pop test: 2 == 2
ok  6 - Pop test: 3 == 3
ok  7 - Pop test: 4 == 4
ok  8 - Pop test: 5 == 5
ok  9 - Pop test: 6 == 6
ok 10 - Pop test: 7 == 7
ok 11 - Pop test: 8 == 8
ok 12 - Pop test: 9 == 9
ok 13 - Ring empty
# **************************************************************
# ** Test 2: unlocked ring pointer, one consumer, check order **
# **************************************************************
# consumer starting
ok 14 - Pushing 0, ring not full
ok 15 - Pushing 1, ring not full
ok 16 - Pushing 2, ring not full
ok 17 - Pushing 3, ring not full
ok 18 - Pushing 4, ring not full
ok 19 - Pushing 5, ring not full
ok 20 - Pushing 6, ring not full
ok 21 - Pushing 7, ring not full
ok 22 - Pushing 8, ring not full
ok 23 - Pushing 9, ring not full
ok 24 - consumer: (got) 0 == 0 (expected)
ok 25 - consumer: (got) 1 == 1 (expected)
ok 26 - consumer: (got) 2 == 2 (expected)
ok 27 - consumer: (got) 3 == 3 (expected)
ok 28 - Pushing 10, ring not full
ok 29 - Pushing 11, ring not full
ok 30 - Pushing 12, ring not full
ok 31 - Pushing 13, ring not full
ok 32 - consumer: (got) 4 == 4 (expected)
ok 33 - consumer: (got) 5 == 5 (expected)
ok 34 - consumer: (got) 6 == 6 (expected)
ok 35 - consumer: (got) 7 == 7 (expected)
ok 36 - Pushing 14, ring not full
ok 37 - Pushing 15, ring not full
ok 38 - Pushing 16, ring not full
ok 39 - Pushing 17, ring not full
ok 40 - consumer: (got) 8 == 8 (expected)
ok 41 - consumer: (got) 9 == 9 (expected)
ok 42 - consumer: (got) 10 == 10 (expected)
ok 43 - consumer: (got) 11 == 11 (expected)
ok 44 - Pushing 18, ring not full
ok 45 - Pushing 19, ring not full
ok 46 - consumer: (got) 12 == 12 (expected)
ok 47 - consumer: (got) 13 == 13 (expected)
ok 48 - consumer: (got) 14 == 14 (expected)
ok 49 - consumer: (got) 15 == 15 (expected)
ok 50 - consumer: (got) 16 == 16 (expected)
ok 51 - consumer: (got) 17 == 17 (expected)
ok 52 - consumer: (got) 18 == 18 (expected)
ok 53 - consumer: (got) 19 == 19 (expected)
ok 54 - Ring empty
ok 55 - Value test: 0 was processed
ok 56 - Value test: 1 was processed
ok 57 - Value test: 2 was processed
ok 58 - Value test: 3 was processed
ok 59 - Value test: 4 was processed
ok 60 - Value test: 5 was processed
ok 61 - Value test: 6 was processed
ok 62 - Value test: 7 was processed
ok 63 - Value test: 8 was processed
ok 64 - Value test: 9 was processed
ok 65 - Value test: 10 was processed
ok 66 - Value test: 11 was processed
ok 67 - Value test: 12 was processed
ok 68 - Value test: 13 was processed
ok 69 - Value test: 14 was processed
ok 70 - Value test: 15 was processed
ok 71 - Value test: 16 was processed
ok 72 - Value test: 17 was processed
ok 73 - Value test: 18 was processed
ok 74 - Value test: 19 was processed
#
*************************************************************************************
# ** Test 3: locked ring pointer, many consumers, many producers, check
no of copies **
#
*************************************************************************************
ok 75 - Ring empty
# consumer0 starting
# consumer1 starting
# consumer2 starting
# consumer3 starting
# producer0 starting
ok 76 - producer0: Pushing 0, ring not full
ok 77 - producer0: Pushing 1, ring not full
ok 78 - producer0: Pushing 2, ring not full
ok 79 - producer0: Pushing 3, ring not full
ok 80 - producer0: Pushing 4, ring not full
ok 81 - producer0: Pushing 5, ring not full
ok 82 - producer0: Pushing 6, ring not full
ok 83 - producer0: Pushing 7, ring not full
ok 84 - producer0: Pushing 8, ring not full
ok 85 - producer0: Pushing 9, ring not full
# producer1 starting
# producer2 starting
# producer3 starting
ok 86 - consumer3: got a 0 (0 times seen before)
ok 87 - consumer3: got a 1 (0 times seen before)
ok 88 - consumer3: got a 2 (0 times seen before)
ok 89 - consumer3: got a 3 (0 times seen before)
ok 90 - producer0: Pushing 10, ring not full
ok 91 - producer0: Pushing 11, ring not full
ok 92 - producer0: Pushing 12, ring not full
ok 93 - producer0: Pushing 13, ring not full
ok 94 - consumer2: got a 4 (0 times seen before)
ok 95 - consumer3: got a 5 (0 times seen before)
ok 96 - consumer2: got a 6 (0 times seen before)
ok 97 - consumer3: got a 7 (0 times seen before)
ok 98 - consumer2: got a 8 (0 times seen before)
ok 99 - consumer3: got a 9 (0 times seen before)
ok 100 - consumer2: got a 10 (0 times seen before)
ok 101 - consumer3: got a 11 (0 times seen before)
ok 102 - producer0: Pushing 14, ring not full
ok 103 - producer0: Pushing 15, ring not full
ok 104 - producer0: Pushing 16, ring not full
ok 105 - producer0: Pushing 17, ring not full
ok 106 - producer0: Pushing 18, ring not full
ok 107 - producer0: Pushing 19, ring not full
ok 108 - producer1: Pushing 0, ring not full
ok 109 - producer1: Pushing 1, ring not full
ok 110 - consumer1: got a 12 (0 times seen before)
ok 111 - consumer2: got a 13 (0 times seen before)
ok 112 - consumer3: got a 14 (0 times seen before)
ok 113 - consumer1: got a 15 (0 times seen before)
ok 114 - consumer2: got a 16 (0 times seen before)
ok 115 - consumer3: got a 17 (0 times seen before)
ok 116 - consumer0: got a 18 (0 times seen before)
ok 117 - consumer1: got a 19 (0 times seen before)
ok 118 - consumer2: got a 0 (1 times seen before)
ok 119 - consumer3: got a 1 (1 times seen before)
ok 120 - producer1: Pushing 2, ring not full
ok 121 - producer1: Pushing 3, ring not full
ok 122 - producer1: Pushing 4, ring not full
ok 123 - producer1: Pushing 5, ring not full
ok 124 - producer1: Pushing 6, ring not full
ok 125 - producer1: Pushing 7, ring not full
ok 126 - producer1: Pushing 8, ring not full
ok 127 - producer1: Pushing 9, ring not full
ok 128 - producer1: Pushing 10, ring not full
ok 129 - producer1: Pushing 11, ring not full
ok 130 - consumer3: got a 2 (1 times seen before)
ok 131 - consumer3: got a 3 (1 times seen before)
ok 132 - consumer3: got a 4 (1 times seen before)
ok 133 - consumer3: got a 5 (1 times seen before)
ok 134 - producer1: Pushing 12, ring not full
ok 135 - producer1: Pushing 13, ring not full
ok 136 - producer1: Pushing 14, ring not full
ok 137 - producer1: Pushing 15, ring not full
ok 138 - consumer3: got a 6 (1 times seen before)
ok 139 - consumer2: got a 7 (1 times seen before)
ok 140 - consumer3: got a 8 (1 times seen before)
ok 141 - consumer2: got a 9 (1 times seen before)
ok 142 - consumer3: got a 10 (1 times seen before)
ok 143 - consumer2: got a 11 (1 times seen before)
ok 144 - consumer3: got a 12 (1 times seen before)
ok 145 - consumer2: got a 13 (1 times seen before)
ok 146 - producer1: Pushing 16, ring not full
ok 147 - producer1: Pushing 17, ring not full
ok 148 - producer1: Pushing 18, ring not full
ok 149 - producer1: Pushing 19, ring not full
ok 150 - producer2: Pushing 0, ring not full
ok 151 - producer2: Pushing 1, ring not full
ok 152 - producer2: Pushing 2, ring not full
ok 153 - producer2: Pushing 3, ring not full
ok 154 - consumer1: got a 14 (1 times seen before)
ok 155 - consumer3: got a 15 (1 times seen before)
ok 156 - consumer2: got a 16 (1 times seen before)
ok 157 - consumer1: got a 17 (1 times seen before)
ok 158 - consumer3: got a 18 (1 times seen before)
ok 159 - consumer2: got a 19 (1 times seen before)
ok 160 - consumer1: got a 0 (2 times seen before)
ok 161 - consumer3: got a 1 (2 times seen before)
ok 162 - consumer2: got a 2 (2 times seen before)
ok 163 - consumer1: got a 3 (2 times seen before)
ok 164 - producer2: Pushing 4, ring not full
ok 165 - producer2: Pushing 5, ring not full
ok 166 - producer2: Pushing 6, ring not full
ok 167 - producer2: Pushing 7, ring not full
ok 168 - producer2: Pushing 8, ring not full
ok 169 - producer2: Pushing 9, ring not full
ok 170 - producer2: Pushing 10, ring not full
ok 171 - producer2: Pushing 11, ring not full
ok 172 - producer2: Pushing 12, ring not full
ok 173 - producer2: Pushing 13, ring not full
ok 174 - consumer2: got a 4 (2 times seen before)
ok 175 - consumer1: got a 5 (2 times seen before)
ok 176 - consumer2: got a 6 (2 times seen before)
ok 177 - consumer1: got a 7 (2 times seen before)
ok 178 - consumer2: got a 8 (2 times seen before)
ok 179 - consumer1: got a 9 (2 times seen before)
ok 180 - consumer2: got a 10 (2 times seen before)
ok 181 - consumer1: got a 11 (2 times seen before)
ok 182 - producer2: Pushing 14, ring not full
ok 183 - producer2: Pushing 15, ring not full
ok 184 - producer2: Pushing 16, ring not full
ok 185 - producer2: Pushing 17, ring not full
ok 186 - producer2: Pushing 18, ring not full
ok 187 - producer2: Pushing 19, ring not full
ok 188 - producer3: Pushing 0, ring not full
ok 189 - producer3: Pushing 1, ring not full
ok 190 - consumer3: got a 12 (2 times seen before)
ok 191 - consumer2: got a 13 (2 times seen before)
ok 192 - consumer1: got a 14 (2 times seen before)
ok 193 - consumer3: got a 15 (2 times seen before)
ok 194 - consumer2: got a 16 (2 times seen before)
ok 195 - consumer1: got a 17 (2 times seen before)
not ok 196 - Ring empty
not ok 197 - Value test: 0 was processed 4 times
not ok 198 - Value test: 1 was processed 4 times
not ok 199 - Value test: 2 was processed 4 times
not ok 200 - Value test: 3 was processed 4 times
not ok 201 - Value test: 4 was processed 4 times
not ok 202 - Value test: 5 was processed 4 times
not ok 203 - Value test: 6 was processed 4 times
not ok 204 - Value test: 7 was processed 4 times
not ok 205 - Value test: 8 was processed 4 times
not ok 206 - Value test: 9 was processed 4 times
not ok 207 - Value test: 10 was processed 4 times
not ok 208 - Value test: 11 was processed 4 times
not ok 209 - Value test: 12 was processed 4 times
not ok 210 - Value test: 13 was processed 4 times
not ok 211 - Value test: 14 was processed 4 times
not ok 212 - Value test: 15 was processed 4 times
not ok 213 - Value test: 16 was processed 4 times
not ok 214 - Value test: 17 was processed 4 times
not ok 215 - Value test: 18 was processed 4 times
not ok 216 - Value test: 19 was processed 4 times
ok 217 - consumer3: got a 18 (2 times seen before)
ok 218 - consumer2: got a 19 (2 times seen before)
ok 219 - consumer1: got a 0 (3 times seen before)
ok 220 - consumer3: got a 1 (3 times seen before)

Planned 256 tests but only ran 220

    Results
    =======
       Tests: 220
      Passed: 199 = 90.45%
      Failed:  21 =  9.55%

Replies:
Re: ringPointerTest Michael Davidsaver

Navigate by Date:
Prev: Build failed in Jenkins: epics-base-3.15-win64 #90 APS Jenkins
Next: Assertion failure in dbLock.c Andrew Johnson
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022 
Navigate by Thread:
Prev: Jenkins build is back to normal : epics-base-3.15-win64 #92 APS Jenkins
Next: Re: ringPointerTest Michael Davidsaver
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022 
ANJ, 17 Nov 2014 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·