RE: Roper Quad-RO locks up with areaDetector/ccd apps

["Mark Rivers" <rivers@cars.uchicago.edu>]

Hi Ru,
 
I was just able to test areaDetector R1-5beta2 with a Princeton Instruments MicroMAX 5MHz, which uses the PI PCI card.
 
I used the EPICS "sscan" record to trigger the detector Acquire PV.  I used a 0.1 second acquire time, and was collecting a 450x515 region of the detector.  I was able to do a 2000 point scan twice with no problems, which thus saved 4000 images to disk.  This was with NumExposures, NumImages, and NumAcquisitions all set to 1.  There was 0 second detector settling time and positioner settling time in my sscan record, so it was immediately starting the next acquisition when the previous one completed.
 
So I think my conclusion is that there is not a fundamental problem with areaDetector or WinView.  I suspect the problem is specific to your Firewire camera, unless you are doing something differently in your client.
 
Mark
 

________________________________

From: Mark Rivers
Sent: Sat 8/8/2009 8:37 AM
To: Ru Igarashi; tech-talk@aps.anl.gov
Subject: RE: Roper Quad-RO locks up with areaDetector/ccd apps


Ru,
 
> We essentially run a loop triggering the
> detector via the ccd/areaDetector's acquire PV, wait
> for a monitor from the acquire status, make sure the
> status is "Done", do a few other tasks, then trigger
> again.  There's at least 1.5 seconds between the end
> of the trigger "Done" change-of-state and the subsequent
> software trigger.
 
Just to be clear:  you should wait for the $(P)$(R)Acquire to go back to to 0 ("Done"), not the $(P)$(R)DetectorState_RBV PV.  But I think this is what you are doing.
 
I have not seen the problem you are seeing, but I will do a test next week to be sure, because I am not sure I have tested >1000 exposures in a loop.
 
The cameras I can test are the MicroMAX 5MHz, which uses the older Princeton Instruments PCI card, and the CoolSnap HQ2 which uses the Photometrics PCI card.  I don't have any Princeton Instruments Firewire cameras I can test.  
 
It could be a problem that is specific to the Firewire interface.  One thing to look at is the Windows Task Manager/Performance screen.  Look to see if the number of Handles or Commit Charge Total (virtual memory) are increasing monotonically as the application runs.  If so there is a resource leak.
 
Note that ccd and areaDetector use a completely different way of talking to the Microsoft COM interface which is used to control WinView.  ccd uses a small Visual Basic socket server that I wrote, and the COM calls are done from Visual Basic, which Princeton Instruments documents and "supports".  areaDetector calls COM directly from C++, which is not documented by Princeton, but works fine. 
 
What you need to do is figure out if this is a problem in the Princeton Instruments software or my ccd or areaDetector software.  Because ccd and areaDetector are so different I doubt if they both have the same bug, but it's always possible.  In order to see if the problem is in the Princeton software what you need to do is run a Visual Basic Script (.vbs file, not Visual Basic itself) that does the same loop that EPICS is doing and see if you can reproduce the problem.  That is what Princeton did to locate the resource leak in XP SP2 that was causing WinView to die after ~1600 images.  I am appending the VBS script that Princeton sent me that reproduced that problem.  It seems to me that this should work for you, just name this file test.vbs and double-click on it in Windows Explorer.  If you can get it to fail with this VBS script then you need to send it to Princeton, because they support VBS and will recognize it as a valid test.
 
Mark
 
test.vbs
**********
 
Option Explicit

Dim ExpVB: Set ExpVB = CreateObject("WinX32.ExpSetup")
Dim wins: set wins = CreateObject("WinX32.DocWindows")
dim win
dim doc
Dim I
dim numExps
dim myCheck
dim frame


wins.closeall

numExps = inputbox("enter number of experiments to run")
myCheck = isnumeric( numExps ) 
if myCheck then 
    ExpVB.SetParam 827, false

    For I = 1 to numExps
        set doc = ExpVB.Start2
        ExpVB.WaitForExperiment
        if i=1 then
             'create new display window
            dim docDisplay: set docDisplay = CreateObject("WinX32.DocFile")
            dim info: set info = CreateObject("WinX32.DocInfo")
            info.Name = "Display"
            info.dataType = doc.getParam( 9 )
            info.x = doc.getParam( 6 )
            info.y = doc.getParam( 7 )
            info.z = doc.getParam( 24 )
            info.bShowWindow = true
            info.bAppend = false
            info.filetype = 1
            docDisplay.OpenNew "", info
       end if
       doc.GetFrame 1, frame 
       doc.save
       doc.close
 
       docDisplay.PutFrame 1, frame
       docDisplay.Update
 
    Next

    ExpVB.SetParam 827, true

else
    msgbox "Please enter a valid number"
end 
 
 
________________________________

From: tech-talk-bounces@aps.anl.gov on behalf of Ru Igarashi
Sent: Fri 8/7/2009 7:55 PM
To: tech-talk@aps.anl.gov
Subject: Roper Quad-RO locks up with areaDetector/ccd apps



One of the beamlines at the Canadian Light Source (CLS)
uses a Princeton Instruments (Roper) Quad-RO CCD detector
(2084x2084). We have used the synApps 'ccd' application
in the past, and have recently attempted to use the
'areaDetector' app instead.  In both cases, however,
when we run multiple image sequential scans, the hardware
locks up.  We essentially run a loop triggering the
detector via the ccd/areaDetector's acquire PV, wait
for a monitor from the acquire status, make sure the
status is "Done", do a few other tasks, then trigger
again.  There's at least 1.5 seconds between the end
of the trigger "Done" change-of-state and the subsequent
software trigger. The sequence is typically set for
2500-4000 exposures.

With 'ccd' we could get about 2500-3000 exposures before
the hardware locked up.  At least we think it is the
hardware because WinView still operates (e.g. close image,
start acquire, empty image frame appears, but no image),
restarting WinView does not solve the problem, and only
a hardware power cycle (long wait) solves the problem.
With 'ccd', there was a proxy app that displayed comm
traffic, and at the onset of the problem, it was clear
that communications were out of sync (command acks too
early, missing data packets) and remained out of sync.

We were hoping a switch to 'areaDetector' would resolve
this issue but it seems to have actually made the situation
worse: we now only get at most a few hundred exposures
before locking up.  Adding more time between exposures
(in case there was some sort of interference with
communication before or after an exposure) did not help,
either.

It looks as though areaDetector holds the status flag
"busy" until after the data is written to file, and we
don't think we see an overlap with disk or network activity
(haven't thought through caching issues properly), so it
doesn't seem to be a problem with triggering during the
previous image's file I/O.

The CCD hardware is linked to a Windows XP box via firewire.
XP is Service Pack 3 (so it isn't the old SP 2 bug).
areaDetector is V1.4 (prebuilt), ccd was V1.6.  WinView
is V2.5.22.0.

Has anyone else experienced the same kind of problem
with their Roper CCD detector?  What can we do to either
further diagnose the problem or resolve it?  Perhaps
a Windows app that that can replace and perform the same
scan operations a few thousand times?

ru

--
Ru Igarashi                   Software & Instrumentation Specialist
e-mail: ru.igarashi@usask.ca  Canadian Light Source
ph: 306 657 3751              University of Saskatchewan
fax: 306 657 3535             Saskatoon SK S7N 0X4