RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

["Heesterman, Peter J via Tech-talk" <tech-talk@aps.anl.gov>]

Thank you, Mark, you are quite right.

 

I’m able to scroll through the 3 colours as separate ‘frames’ (0..2), with this selection of dimension mapping.

 

Also, if i select the dimension mapping 1, 2 and 0, I can scroll through the time (frame) dimension (but only for one colour).

 

Many thanks for your help,

 

Peter.

 

 

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: 27 November 2018 15:01
To: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>; Engbretson, Mark S. <engbretson@anl.gov>
Cc: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Peter,

 

Your HDFView problem is that it is not slicing the data the way you want.  Your dataset is 160 (frames) x 360 (height) x 640 (width) x 3 (colors).

 

By default it is displaying 160x360 images, which is not what you want, since that is frame # x height.  You need to right-click on “data” and select Open As.  Then you can select Height=Dim1, Width=Dim2.  I don’t think it will display RGB color, so you can only view one color plane at a time.  See my screen shot, which is correct for the simDetector.

 

Mark

 

 

From: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Sent: Tuesday, November 27, 2018 6:11 AM
To: Mark Rivers <rivers@cars.uchicago.edu>; Engbretson, Mark S. <engbretson@anl.gov>
Cc: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

>> NDPluginStdArrays is not optional, it is always built as part of ADCore.

 

Sorry for the misunderstanding.

I (incorrectly) connected your comment about “NDPluginStdArrays” with:

#NDDRIVERSTDARRAYS=$(AREA_DETECTOR)/NDDriverStdArrays

 

>> hdfview, you are running 2.11.0, I am running 2.14.0

 

HDVView 2.11.0 is the version that installs on Ubuntu 18.04, by means of apt-get.

May I ask which Linux version you’re using?

 

In order to get the newer version, I have built 2.14 from sources.

(I eschewed the Centos distribution, that the HDF group provide).

 

Now I’m able to open the data from a recorded data file (this from the URL module), but the image isn’t recognisable.

See the screen shot attached (you can also see the ImageJ view across my desk).

 

Do you have any ideas about this?

 

>> NDPluginDriver::connectToArrayPort Error calling pasynManager->connectDevice to array port FFT1 address 0, status=3, error=asynManager:connectDevice port FFT1 not found.

 

I still have the error in the SimDetector module, both on Ubuntu and Windows.

 

I’m not sure I fully understand the name binding and correlation between these lines of the script?

            dbLoadRecords("NDStdArrays.template", "P=13SIM1:,R=image2:,PORT=Image2,ADDR=0,TIMEOUT=1,NDARRAY_PORT=SIM2,TYPE=Float64,FTVL=DOUBLE,NELEMENTS=12000000")

            NDStdArraysConfigure("Image2", 3, 0, "FFT1", 0)

            NDFFTConfigure("FFT1", 3, 0, "SIM1", 0, 0, 0, 0, 0, 5)

 

Many thanks,

 

Peter.

 

 

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: 20 November 2018 18:22
To: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>; Engbretson, Mark S. <engbretson@anl.gov>
Cc: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

• The module is commented out in EXAMPLE_RELEASE.local, and I hadn’t appreciated that it was a dependency, when I carried out the install.
• I have corrected that.

 

I’m not sure what you mean.  What module is commented out in EXAMPLE_RELEASE.local?  NDPluginStdArrays is not optional, it is always built as part of ADCore.

 

•  Does this suggest that I’m missing some aspect of hdfview, where I am still unable to view data...?

 

The fact that h5dump works fine on your HDF5 file suggests that perhaps your version of hdfview is the problem.  I am running 2.14.0, you are running 2.11.0.  Try updating.

 

Mark

 

 

 

 

From: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Sent: Tuesday, November 20, 2018 12:11 PM
To: Mark Rivers <rivers@cars.uchicago.edu>; Engbretson, Mark S. <engbretson@anl.gov>
Cc: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

>> I think your problem with the ImageJ display is that you have not enabled NDPluginStdArrays. 

Ah .. OK, thank you.

 

The module is commented out in EXAMPLE_RELEASE.local, and I hadn’t appreciated that it was a dependency, when I carried out the install.

I have corrected that.

 

I am now able to view the simulated image (13SIM:image1).

 

>> You error about the FFT is coming before the IOC has really even started

Yes, this is odd.

I’m assuming that it originates with either of:

            NDFFTConfigure("FFT1", 3, 0, "SIM1", 0, 0, 0, 0, 0, 5)

            dbLoadRecords("NDFFT.template", "P=13SIM1:, R=FFT1:, PORT=FFT1, ADDR=0, TIMEOUT=1, NDARRAY_PORT=SIM1, NAME=FFT1, NCHANS=1024")

 

I’m not sure how much importance can be attached to the sequence of log messages in a threaded application.

 

In any case, I still have this error, do you have any ideas?

 

>> h5dump on the HDF5 file, and put it into test5_001.txt.  What does that file look like for you? 

It is attached.

It looks pretty much like yours, to me.

 

Does this suggest that I’m missing some aspect of hdfview, where I am still unable to view data...?

 

Many thanks,

 

Peter.

 

 

 

 

 

 

 

 

 

 

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: 19 November 2018 21:25
To: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>; Engbretson, Mark S. <engbretson@anl.gov>
Cc: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

medmInit: Unable to load font widgetDM_48

  Trying default (fixed) instead

 

medmInit: Unable to load font widgetDM_60

  Trying default (fixed) instead

2018/11/19 18:31:31.558 NDPluginDriver::connectToArrayPort Error calling pasynManager->connectDevice to array port FFT1 address 0, status=3, error=asynManager:connectDevice port FFT1 not found

< envPaths

epicsEnvSet("IOC","iocHDF5Test")

 

You error about the FFT is coming before the IOC has really even started, right after your medm widget messages.  I have no idea how that can happen,  since it has not even called dbLoadDatabase yet.  How can the pluginDriver possibly be generating an error message, it is not even running yet?  Is this message perhaps from another IOC, or the previous IOC as it exits?

 

For your HDF5 problem, this is test5.sh:

********************************

corvette:simDetectorIOC/iocBoot/iocHDF5Test>more test5.sh

caput 13SIM1:cam1:SizeX 10

caput 13SIM1:cam1:SizeY 10

caput 13SIM1:cam1:AcquireTime .001

caput 13SIM1:cam1:AcquirePeriod 0.1

caput 13SIM1:cam1:ImageMode "Continuous"

caput 13SIM1:cam1:ArrayCallbacks "Enable"

caput -S 13SIM1:cam1:NDAttributesFile "./test5_attributes.xml"

caput 13SIM1:cam1:Acquire 1

caput 13SIM1:HDF1:EnableCallbacks Enable

caput -S 13SIM1:HDF1:FilePath "./"

caput -S 13SIM1:HDF1:FileName "test5"

caput -S 13SIM1:HDF1:FileTemplate "%s%s_%3.3d.h5"

caput -S 13SIM1:HDF1:XMLFileName "./test5_layout.xml"

caput 13SIM1:HDF1:AutoIncrement 1

caput 13SIM1:HDF1:FileNumber 1

caput 13SIM1:HDF1:FileWriteMode "Stream"

caput 13SIM1:HDF1:NumCapture "0"

caput 13SIM1:HDF1:Capture 1

sleep 3

caput 13SIM1:HDF1:Capture 0

sleep 5

h5dump test5_001.h5 > test5_001.txt

********************************

 

Note that the last thing it does is to run h5dump on the HDF5 file, and put it into test5_001.txt.  What does that file look like for you?  Mine starts like this:

 

HDF5 "test5_001.h5" {

GROUP "/" {

   GROUP "entry" {

      ATTRIBUTE "NX_class" {

         DATATYPE  H5T_STRING {

            STRSIZE 7;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

         DATASPACE  SCALAR

         DATA {

         (0): "NXentry"

         }

      }

      ATTRIBUTE "signal" {

         DATATYPE  H5T_STRING {

            STRSIZE 4;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

         DATASPACE  SCALAR

         DATA {

         (0): "data"

         }

      }

      DATASET "AD_template_ID" {

         DATATYPE  H5T_STRING {

            STRSIZE 58;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }

         DATA {

         (0): "$Id: pilatus_layout.xml 8015 2015-02-24 18:28:27Z jemian $"

         }

      }

…

 

Please send me the version created on your system.

 

Mark

 

 

 

 

 

From: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Sent: Monday, November 19, 2018 1:06 PM
To: Mark Rivers <rivers@cars.uchicago.edu>; Engbretson, Mark S. <engbretson@anl.gov>
Cc: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

>> It looks like from your log file that the FFT error message is actually part of a comment in your startup script.  I don’t see how this can be a real error message, since no asyn ports have been created yet at that point in the startup script.

 

I assume this is a threading artifact of simultaneous logging, which jumbles and obfuscates the text output, particularly when stdout and stderr are redirected to a file.

I’ve removed the comments from the start-up script, (as best I can) where this happens, so as to make the error clearer to see.

 

>> Did you look at the ImageJ log file?

Yes, there are no errors shown there.

Just no image display...

 

I attach screen shots with the requested information, as best I can.

Also screen shot of hdvview showing the resulting file, no attribute nodes (quite unlike the screen shot you sent me!).

 

Cheers,

 

Peter.

 

 

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: 19 November 2018 15:51
To: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>; Engbretson, Mark S. <engbretson@anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

• NB, data visualisation with ImageJ works for me with the ADURL module with the values given, but fails for the SimDetector module (with or without attempting to use HDF5).

 

Did you look at the ImageJ log file?

 

Please send screen shots of the following:

 

• simDetector main screen
• simDetector setup screen
• commonPlugins screen
• EPICS_AD_Viewer ImageJ screen

 

Mark

 

 

From: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Sent: Monday, November 19, 2018 9:31 AM
To: Engbretson, Mark S. <engbretson@anl.gov>; Mark Rivers <rivers@cars.uchicago.edu>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark(s),

 

I am using simDetector 2-8, which I believe is the most recent.

My comment about the st.cmd script referred to the iocHDF5Test sub-folder (which I was seeking to exercise), and not the main iocSimDetector folder (where the script is correct).

- I’m sorry that was unclear.

 

Are you talking about EPICS_CA_MAX_ARRAY_BYTES?

It is set as an environment variable which is applied to shells through /etc/profile.d/custom.sh.

 

NB, data visualisation with ImageJ works for me with the ADURL module with the values given, but fails for the SimDetector module (with or without attempting to use HDF5).

 

Cheers,

 

Peter.

 

 

From: Engbretson, Mark S. <engbretson@anl.gov>
Sent: 19 November 2018 15:04
To: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

But did it make any difference? You might have to set this both in the ADURL and SIMDetector soft IOC st,cmd *AND* in for environment for IMAGEJ to work.

 

From: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Sent: Monday, November 19, 2018 8:59 AM
To: Engbretson, Mark S. <engbretson@anl.gov>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

I have the latest sources.

My comment about the paths refers to the iocHDF5Test sub-folder that I was working with, not the iocSimDetector folder (where it’s correct).

I’m sorry this wasn’t clear.

 

I has set EPICS_CA_MAX_ARRAY_BYTES to 30000000 which is overkill I think, so I reduced it to 10000000.

 

Cheers,

 

Peter.

 

 

From: Engbretson, Mark S. <engbretson@anl.gov>
Sent: 19 November 2018 14:44
To: Heesterman, Peter J <Peter.Heesterman@ukaea.uk>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

I don’t think that you have the latest sources for ADSimDetector since the paths are correct on the sources that I have.

 

If ImageJ   doesn’t work, chances are the value for

 

EPICS_CA_MAX_ARRAY_BYTES

 

Is either not set or set to be a value too small for the actual image data.

 

 

From: tech-talk-bounces@aps.anl.gov <tech-talk-bounces@aps.anl.gov> On Behalf Of Heesterman, Peter J via Tech-talk
Sent: Monday, November 19, 2018 8:24 AM
To: EPICS Tech-Talk (tech-talk@aps.anl.gov) <tech-talk@aps.anl.gov>
Subject: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi,

 

I’m seeking to use these modules with HDF5 data archiving.

Accordingly, I’ve been looking at the iocHDF5Test within ADSimDetector, and focused on test5.sh.

 

I’m working with an Ubuntu 18.04 VM.

I have:

    areaDetector and ADCore R3-3-2

    ADSupport R1-5

  ADViewers R1-2

    ADURL R2-2

    Asyn4-33

    Base 3.15.6

 

There are some issues that I need to raise:

 

a. st.cmd in ADSimDetector hasn’t been updated to reflect module paths – in particular lines like

        dbLoadRecords("$(ADCORE)/db/simDetector.template","P=$(PREFIX),R=cam1:,PORT=$(PORT),ADDR=0,TIMEOUT=1"), where the file is no longer in ADCore.

    This is easily corrected. Actually, I believe st_base.cmd from iocSimDetector can just be used.

 

b. When I run the sim detector and try to visualise with ImageJ, I get green on both 13SIM1:image1 and 13SIM1:image2.

    But, I don’t get any image displayed for either.

 

c. I note an error:

    NDPluginDriver::connectToArrayPort Error calling pasynManager->connectDevice to array port FFT1 address 0, status=3, error=asynManager:connectDevice port FFT1 not found.

 

    Could my (b) have anything to do with this error?

    I understand that image2 would be the FFT of the simulated data generated for image1?

 

d. I am able to create a HDF5 file (for either ADSimDetector or ADURL) as requested by the test5.sh script (I cloned the logic of this for the ADURL module).

    These files are large (>~100MB), and clearly contain the simulated and URL data, respectively.

    I’m able to open the file(s) with the hdfview application (after closing the IOC, to avoid any file write/read share issues).

    But, I’m unable to view any of the data because there are no attribute nodes under the ‘test5_001.h5’ root, in the viewer.

 

    Viewing the ‘properties’ on the ‘test5_001.h5’ node in hdfview, shows no attributes and no datasets.

 

    I refer to http://cars9.uchicago.edu/software/epics/NDFileHDF5.html , which I’ve studied in some detail.

 

I am able to visualise data from the ADURL module, quite normally.

 

What is it that I’m missing?

 

Many thanks,

 

Peter.