Fwd: Re: GigE cameras stops

[Bruce Hill <bhill@slac.stanford.edu>]

We're having a problem with 'CAS: UDP recv errors" that's causing
our GigE camera IOC's to stop updating images overnight.

We're running EPICS 3.14.9, asyn R4.10.1, and areaDetector R1-2.

When the images stop, we get these errors on our console:

17_14:39:08:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:08:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:09:CAS: UDP recv error (errno=Interrupted system call)
...

Googling "Interrupted system call" seems to indicated that repeating the
system call, in this case, recv_from, may be needed.   It appears that the CA
code is already doing that.

I'm hoping that Jeff or Mark, or anyone, may have already seen this problem.
We've been working on upgrading to 3.14.11 and newer modules, so if this
problem has already been fixed we may just need to refocus our efforts on
the upgrade.

Thanks,
- Bruce


-------- Original Message --------

Subject:	Re: GigE cameras stops
Date:	Thu, 17 Feb 2011 14:45:52 -0800
From:	Pavel Stoffel <pstoffel@slac.stanford.edu>
To:	Hill, Bruce <bhill@slac.stanford.edu>

Hi Bruce,

Turned on tracing for cameras:

It looks like the camera stops working when we get these 'CAS: UDP recv 
errors (errno=Interrupted system call)'.
Everything before it, repeats for every frame.  In this case, I 
collected 54335 frames before it stopped.

Here is the tail end of the trace file:

17_14:38:22:2011/02/17 14:38:22.888 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45326
17_14:38:23:2011/02/17 14:38:23.088 prosilica:frameCallback: 
frameId=45329, timeStamp=763787.037673
17_14:38:23:2011/02/17 14:38:23.088 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45327
17_14:38:23:2011/02/17 14:38:23.288 prosilica:frameCallback: 
frameId=45330, timeStamp=763787.237673
17_14:38:23:2011/02/17 14:38:23.288 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45328
17_14:38:23:2011/02/17 14:38:23.488 prosilica:frameCallback: 
frameId=45331, timeStamp=763787.437673
17_14:38:23:2011/02/17 14:38:23.488 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45329
17_14:38:23:2011/02/17 14:38:23.688 prosilica:frameCallback: 
frameId=45332, timeStamp=763787.637673
17_14:38:23:2011/02/17 14:38:23.688 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45330
17_14:38:23:2011/02/17 14:38:23.888 prosilica:frameCallback: 
frameId=45333, timeStamp=763787.837673
17_14:38:23:2011/02/17 14:38:23.888 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45331
17_14:38:24:2011/02/17 14:38:24.088 prosilica:frameCallback: 
frameId=45334, timeStamp=763788.037673
17_14:38:24:2011/02/17 14:38:24.088 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45332
17_14:38:24:2011/02/17 14:38:24.288 prosilica:frameCallback: 
frameId=45335, timeStamp=763788.237673
17_14:38:24:2011/02/17 14:38:24.288 XPP:USR:GIGE1:cam1:ImageCounter_RBV 
devAsynInt32::interruptCallbackInput new value=45333
17_14:38:24:2011/02/17 14:38:24.456 PS1 addr 0 queueRequest priority 0 
not lockHolder
17_14:38:24:2011/02/17 14:38:24.456 PS1 callback
17_14:39:08:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:08:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:09:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:09:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:10:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:17:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:17:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:18:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:18:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:19:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:32:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:33:CAS: UDP recv error (errno=Interrupted system call)
17_14:39:55:CAS: UDP recv error (errno=Interrupted system call)




Pavel Stoffel wrote:
> Hi Bruce,
>
> The cameras stopped after about 30-40K frames.  IOC 1 had 104 frame 
> errors, IOC 2 1:
>
> $ ls-cam1
> XPP:USR:GIGE1:cam1:ImageRate_RBV 0
> XPP:USR:GIGE1:cam1:ImageCounter_RBV 42931
> XPP:USR:GIGE1:cam1:PSBadFrameCounter_RBV 104
>
> $ ls-cam2
> XPP:USR:GIGE2:cam1:ImageRate_RBV 0
> XPP:USR:GIGE2:cam1:ImageCounter_RBV 29353
> XPP:USR:GIGE2:cam1:PSBadFrameCounter_RBV 1
>
> Don't see any errors in the log file :-(
>
> Thanks,
> Pavel
>
>
>