Re: ADPilatus writing cbf strangeness

[John Dobbins via Tech-talk <tech-talk at aps.anl.gov>]

Thanks. I will investigate the Grimsel behavior.

John

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, December 10, 2021 1:27:01 PM
To: John Dobbins <john.dobbins at cornell.edu>
Cc: Jacob Ruff <jpr243 at cornell.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: ADPilatus writing cbf strangeness

 

Hi John,

 

Ø  I was surprised that this happens even if all plug-ins are disabled.

 

It is not the plugins that are consuming CPU, it is reading the decompressing the CBF file.

 

Ø  If I disable PIL10:cam1:ArrayCallbacks then it works. Of course in this case the IOC is simply initiating acquisition and reporting when it is complete (?)

 

Correct, in that case it does not read the CBF files at all.

 

Ø  Since it is happy with tiff files can I safely assume the issue is the decompression of the cbf file by the IOC?

 

Yes.

 

Ø  I am also wondering if the IOC falls slightly behind and Grimsel removes the next file (having written it to its final destination) before the IOC has a chance to read it? 

 

Grimsel should not be automatically removing files from the ramdisk when it writes them to the final destination.  I am quite sure that only happens when Grimsel changes to running under a different UID (which it never does on our system, so we need to delete them manually).

 

Mark

 

 

From: John Dobbins <john.dobbins at cornell.edu>
Sent: Friday, December 10, 2021 11:56 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: Jacob Ruff <jpr243 at cornell.edu>; tech-talk at aps.anl.gov
Subject: Re: ADPilatus writing cbf strangeness

 

Mark,

 

The IOC seems to be choking on the images.  Although the update rate of 'top' is slow it is clear the IOC process goes to 100% of CPU.

 

I was surprised that this happens even if all plug-ins are disabled.  If I disable PIL10:cam1:ArrayCallbacks then it works. Of course in this case the IOC is simply initiating acquisition and reporting when it is complete (?)

 

Since it is happy with tiff files can I safely assume the issue is the decompression of the cbf file by the IOC?

 

I am also wondering if the IOC falls slightly behind and Grimsel removes the next file (having written it to its final destination) before the IOC has a chance to read it? 

 

John

 

 

 

 

---

From: John Dobbins <john.dobbins at cornell.edu>
Sent: Friday, December 10, 2021 11:29 AM
To: Mark Rivers <rivers at cars.uchicago.edu>
Cc: Jacob Ruff <jpr243 at cornell.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: ADPilatus writing cbf strangeness

 

Mark,

 

Yes, the filepath for the IOC is PPU ramdisk. I will investigate per your suggestion.

 

John

---

From: Mark Rivers <rivers at cars.uchicago.edu>
Sent: Friday, December 10, 2021 11:22 AM
To: John Dobbins <john.dobbins at cornell.edu>
Cc: Jacob Ruff <jpr243 at cornell.edu>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: RE: ADPilatus writing cbf strangeness

 

Hi John,

 

What is the FilePath you are using for ADPilatus?  Is it reading from the PPU ramdisk? 

 

I suggest you run camonitor on one of the PVs that updates each time ADPilatus reads a CBF file and does NDArray callbacks.  Is it gradually falling behind the detector so that after N images the allowed timeout has been reached?  Have you looked at “top” to see if the ADPilatus task is CPU bound, perhaps because the time to read the CBF files is longer than the time between when they are created?

 

Mark

 

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of John Dobbins via Tech-talk
Sent: Friday, December 10, 2021 10:15 AM
To: tech-talk at aps.anl.gov
Cc: Jacob Ruff <jpr243 at cornell.edu>
Subject: ADPilatus writing cbf strangeness

 

All,

 

I am hoping you can help me understand what is happening when I try to write cbf files from a PILATUS 6M.

 

ADCore 3.9 ADPilatus Driver 2.9 

 

If I write tiff files everything seems fine.

 

When I write cbf files it works for slow acquisition rate, say 1 Hz.

 

At ~ AcquirePeriod = 0.35, ExposureTime =0.09, NumImages =20

internal trigger

 

PIL10:cam1:StatusMessage_RBV reports a series of messages like

 

"Reading Image File ....."

 

But after ~ 13 or 14 images I get a message

 

"Timeout waiting for file to be created"

 

All 20 images appear to be written normally to disk. 

 

I tried changing PIL10:cam1:ImageFileTmot from 5 to 10 to 20 but this doesn't seem to have any effect other than delaying the appearance of the error message.

 

 

Note our set-up is:

 

DCU runs camserver, collects images from detector which it writes to local ramdisk

 

PPU hosts IOC which communicates with camserver

 

Furka copies images from DCU ramdisk to PPU rammdisk, Grimsel copies images from PPU ramdisk to final destination (a network location)

 

Any guidance would be appreciated.

 

John Dobbins

 

Research Support Specialist

Cornell High Energy Synchrotron Source

Cornell University

 

www.chess.cornell.edu