Re: Sequence monitor not getting callback

[Benjamin Franksen <benjamin.franksen@helmholtz-berlin.de>]

Hi Michael

Am Dienstag, 27. September 2011, um 00:02:59 schrieb Zelazny, Michael Stanley:
> Any idea what the "numMonitoredChans 1 firstMonitorCount 0 assignCount 1
> firstConnectCount 1" message is trying to tell me?

It says:

numMonitoredChans 1
        you have 1 monitored channel

firstMonitorCount 0
        but 0 channels have received a monitor event

assignCount 1
        of your channels, 1 is currently assigned to a PV

firstConnectCount 1
        and one is actually connected

> Background:
>
> I'm trying to reproduce the "sequence not getting its monitor callback"
> problem.
>
> I have 2600 sequences running:

Wow. You are certainly driving the system to its limits...

> lcls-zelazny bash:~/epics/ioc/SeqSupport/SeqSupportApp/src>cat
> SeqChecker.st program SeqChecker
> [...snip test code...]

Ok, thanks for the simple test setup. I could indeed reproduce your problem.

There are actually two separate problems here.

The first one is that the +c option is broken in 2.1.2: I only check that all
channels are connected, but not that each monitored channel got an initial
monitor event. This was easy to fix.

The second one is much harder. It appears that if the program does not wait
for an initial monitor event (either because -c option is in effect, or because
of problem number one above), then (for reasons that are still unclear to me)
one sometimes *never* gets a monitor event.

I have verified this down to the level of basic CA calls: I see that
ca_add_masked_array_event gets called, returns a non-NULL event id, but
strangely no event ever gets received in spite of the PV obviously changing
its value (as can be seen with e.g. camonitor). This only happens reliably if
I start at least 2000 programs, I cannot reproduce this with only a few
hundred.

It might be a hint that the error disappears if I run the IOC under valgrind.
(BTW, I had to re-compile valgrind from source, as it has a hard-coded limit
of 500 threads, which I increased to 3000 for this test. Also had to increase
the limit for the number of memory segments.) Whether this is because
everything becomes much slower or because memory allocation is different under
valgrind I could not yet establish.

I am not sure, but it /could/ be caused by a problem in base, i.e. the CA
client or server. I would be surprised if anyone ever tested EPICS with 2000
threads in the same IOC. However, the problem appears with 3.14.12 and with
3.14.8.2. Maybe Jeff has an idea...

Until this deeper problem gets solved, I recommend avoiding the -c option
(i.e. always wait for connects & monitors, which is the default). For 2.1.2 I
will create a new release with the easy fix for the +c option bug. (This will
have to wait until I am back from the EPICS meeting.)

Cheers
Ben

________________________________

Helmholtz-Zentrum Berlin für Materialien und Energie GmbH

Mitglied der Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren e.V.

Aufsichtsrat: Vorsitzender Prof. Dr. Dr. h.c. mult. Joachim Treusch, stv. Vorsitzende Dr. Beatrix Vierkorn-Rudolph
Geschäftsführer: Prof. Dr. Anke Rita Kaysser-Pyzalla, Dr. Ulrich Breuer

Sitz Berlin, AG Charlottenburg, 89 HRB 5583

Postadresse:
Hahn-Meitner-Platz 1
D-14109 Berlin

http://www.helmholtz-berlin.de