EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5
From: "Heesterman, Peter J via Tech-talk" <[email protected]>
To: Mark Rivers <[email protected]>, "Engbretson, Mark S." <[email protected]>
Cc: "EPICS Tech-Talk \([email protected]\)" <[email protected]>
Date: Tue, 27 Nov 2018 15:21:27 +0000

Thank you, Mark, you are quite right.

 

I’m able to scroll through the 3 colours as separate ‘frames’ (0..2), with this selection of dimension mapping.

 

Also, if i select the dimension mapping 1, 2 and 0, I can scroll through the time (frame) dimension (but only for one colour).

 

Many thanks for your help,

 

Peter.

 

 

From: Mark Rivers <[email protected]>
Sent: 27 November 2018 15:01
To: Heesterman, Peter J <[email protected]>; Engbretson, Mark S. <[email protected]>
Cc: EPICS Tech-Talk ([email protected]) <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Peter,

 

Your HDFView problem is that it is not slicing the data the way you want.  Your dataset is 160 (frames) x 360 (height) x 640 (width) x 3 (colors).

 

By default it is displaying 160x360 images, which is not what you want, since that is frame # x height.  You need to right-click on “data” and select Open As.  Then you can select Height=Dim1, Width=Dim2.  I don’t think it will display RGB color, so you can only view one color plane at a time.  See my screen shot, which is correct for the simDetector.

 

Mark

 

 

From: Heesterman, Peter J <[email protected]>
Sent: Tuesday, November 27, 2018 6:11 AM
To: Mark Rivers <[email protected]>; Engbretson, Mark S. <[email protected]>
Cc: EPICS Tech-Talk ([email protected]) <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

>> NDPluginStdArrays is not optional, it is always built as part of ADCore.

 

Sorry for the misunderstanding.

I (incorrectly) connected your comment about “NDPluginStdArrays” with:

#NDDRIVERSTDARRAYS=$(AREA_DETECTOR)/NDDriverStdArrays

 

>> hdfview, you are running 2.11.0, I am running 2.14.0

 

HDVView 2.11.0 is the version that installs on Ubuntu 18.04, by means of apt-get.

May I ask which Linux version you’re using?

 

In order to get the newer version, I have built 2.14 from sources.

(I eschewed the Centos distribution, that the HDF group provide).

 

Now I’m able to open the data from a recorded data file (this from the URL module), but the image isn’t recognisable.

See the screen shot attached (you can also see the ImageJ view across my desk).

 

Do you have any ideas about this?

 

>> NDPluginDriver::connectToArrayPort Error calling pasynManager->connectDevice to array port FFT1 address 0, status=3, error=asynManager:connectDevice port FFT1 not found.

 

I still have the error in the SimDetector module, both on Ubuntu and Windows.

 

I’m not sure I fully understand the name binding and correlation between these lines of the script?

            dbLoadRecords("NDStdArrays.template", "P=13SIM1:,R=image2:,PORT=Image2,ADDR=0,TIMEOUT=1,NDARRAY_PORT=SIM2,TYPE=Float64,FTVL=DOUBLE,NELEMENTS=12000000")

            NDStdArraysConfigure("Image2", 3, 0, "FFT1", 0)

            NDFFTConfigure("FFT1", 3, 0, "SIM1", 0, 0, 0, 0, 0, 5)

 

Many thanks,

 

Peter.

 

 

From: Mark Rivers <[email protected]>
Sent: 20 November 2018 18:22
To: Heesterman, Peter J <[email protected]>; Engbretson, Mark S. <[email protected]>
Cc: EPICS Tech-Talk ([email protected]) <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

  • The module is commented out in EXAMPLE_RELEASE.local, and I hadn’t appreciated that it was a dependency, when I carried out the install.
  • I have corrected that.

 

I’m not sure what you mean.  What module is commented out in EXAMPLE_RELEASE.local?  NDPluginStdArrays is not optional, it is always built as part of ADCore.

 

  •  Does this suggest that I’m missing some aspect of hdfview, where I am still unable to view data...?

 

The fact that h5dump works fine on your HDF5 file suggests that perhaps your version of hdfview is the problem.  I am running 2.14.0, you are running 2.11.0.  Try updating.

 

Mark

 

 

 

 

From: Heesterman, Peter J <[email protected]>
Sent: Tuesday, November 20, 2018 12:11 PM
To: Mark Rivers <
[email protected]>; Engbretson, Mark S. <[email protected]>
Cc: EPICS Tech-Talk (
[email protected]) <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

>> I think your problem with the ImageJ display is that you have not enabled NDPluginStdArrays. 

Ah .. OK, thank you.

 

The module is commented out in EXAMPLE_RELEASE.local, and I hadn’t appreciated that it was a dependency, when I carried out the install.

I have corrected that.

 

I am now able to view the simulated image (13SIM:image1).

 

>> You error about the FFT is coming before the IOC has really even started

Yes, this is odd.

I’m assuming that it originates with either of:

            NDFFTConfigure("FFT1", 3, 0, "SIM1", 0, 0, 0, 0, 0, 5)

            dbLoadRecords("NDFFT.template", "P=13SIM1:, R=FFT1:, PORT=FFT1, ADDR=0, TIMEOUT=1, NDARRAY_PORT=SIM1, NAME=FFT1, NCHANS=1024")

 

I’m not sure how much importance can be attached to the sequence of log messages in a threaded application.

 

In any case, I still have this error, do you have any ideas?

 

>> h5dump on the HDF5 file, and put it into test5_001.txt.  What does that file look like for you? 

It is attached.

It looks pretty much like yours, to me.

 

Does this suggest that I’m missing some aspect of hdfview, where I am still unable to view data...?

 

Many thanks,

 

Peter.

 

 

 

 

 

 

 

 

 

 

From: Mark Rivers <[email protected]>
Sent: 19 November 2018 21:25
To: Heesterman, Peter J <
[email protected]>; Engbretson, Mark S. <[email protected]>
Cc: EPICS Tech-Talk (
[email protected]) <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

medmInit: Unable to load font widgetDM_48

  Trying default (fixed) instead

 

medmInit: Unable to load font widgetDM_60

  Trying default (fixed) instead

2018/11/19 18:31:31.558 NDPluginDriver::connectToArrayPort Error calling pasynManager->connectDevice to array port FFT1 address 0, status=3, error=asynManager:connectDevice port FFT1 not found

< envPaths

epicsEnvSet("IOC","iocHDF5Test")

 

You error about the FFT is coming before the IOC has really even started, right after your medm widget messages.  I have no idea how that can happen,  since it has not even called dbLoadDatabase yet.  How can the pluginDriver possibly be generating an error message, it is not even running yet?  Is this message perhaps from another IOC, or the previous IOC as it exits?

 

For your HDF5 problem, this is test5.sh:

********************************

corvette:simDetectorIOC/iocBoot/iocHDF5Test>more test5.sh

caput 13SIM1:cam1:SizeX 10

caput 13SIM1:cam1:SizeY 10

caput 13SIM1:cam1:AcquireTime .001

caput 13SIM1:cam1:AcquirePeriod 0.1

caput 13SIM1:cam1:ImageMode "Continuous"

caput 13SIM1:cam1:ArrayCallbacks "Enable"

caput -S 13SIM1:cam1:NDAttributesFile "./test5_attributes.xml"

caput 13SIM1:cam1:Acquire 1

caput 13SIM1:HDF1:EnableCallbacks Enable

caput -S 13SIM1:HDF1:FilePath "./"

caput -S 13SIM1:HDF1:FileName "test5"

caput -S 13SIM1:HDF1:FileTemplate "%s%s_%3.3d.h5"

caput -S 13SIM1:HDF1:XMLFileName "./test5_layout.xml"

caput 13SIM1:HDF1:AutoIncrement 1

caput 13SIM1:HDF1:FileNumber 1

caput 13SIM1:HDF1:FileWriteMode "Stream"

caput 13SIM1:HDF1:NumCapture "0"

caput 13SIM1:HDF1:Capture 1

sleep 3

caput 13SIM1:HDF1:Capture 0

sleep 5

h5dump test5_001.h5 > test5_001.txt

********************************

 

Note that the last thing it does is to run h5dump on the HDF5 file, and put it into test5_001.txt.  What does that file look like for you?  Mine starts like this:

 

HDF5 "test5_001.h5" {

GROUP "/" {

   GROUP "entry" {

      ATTRIBUTE "NX_class" {

         DATATYPE  H5T_STRING {

            STRSIZE 7;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

         DATASPACE  SCALAR

         DATA {

         (0): "NXentry"

         }

      }

      ATTRIBUTE "signal" {

         DATATYPE  H5T_STRING {

            STRSIZE 4;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

         DATASPACE  SCALAR

         DATA {

         (0): "data"

         }

      }

      DATASET "AD_template_ID" {

         DATATYPE  H5T_STRING {

            STRSIZE 58;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }

         DATA {

         (0): "$Id: pilatus_layout.xml 8015 2015-02-24 18:28:27Z jemian $"

         }

      }

 

Please send me the version created on your system.

 

Mark

 

 

 

 

 

From: Heesterman, Peter J <[email protected]>
Sent: Monday, November 19, 2018 1:06 PM
To: Mark Rivers <
[email protected]>; Engbretson, Mark S. <[email protected]>
Cc: EPICS Tech-Talk (
[email protected]) <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

>> It looks like from your log file that the FFT error message is actually part of a comment in your startup script.  I don’t see how this can be a real error message, since no asyn ports have been created yet at that point in the startup script.

 

I assume this is a threading artifact of simultaneous logging, which jumbles and obfuscates the text output, particularly when stdout and stderr are redirected to a file.

I’ve removed the comments from the start-up script, (as best I can) where this happens, so as to make the error clearer to see.

 

>> Did you look at the ImageJ log file?

Yes, there are no errors shown there.

Just no image display...

 

I attach screen shots with the requested information, as best I can.

Also screen shot of hdvview showing the resulting file, no attribute nodes (quite unlike the screen shot you sent me!).

 

Cheers,

 

Peter.

 

 

From: Mark Rivers <[email protected]>
Sent: 19 November 2018 15:51
To: Heesterman, Peter J <
[email protected]>; Engbretson, Mark S. <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

  • NB, data visualisation with ImageJ works for me with the ADURL module with the values given, but fails for the SimDetector module (with or without attempting to use HDF5).

 

Did you look at the ImageJ log file?

 

Please send screen shots of the following:

 

  • simDetector main screen
  • simDetector setup screen
  • commonPlugins screen
  • EPICS_AD_Viewer ImageJ screen

 

Mark

 

 

From: Heesterman, Peter J <[email protected]>
Sent: Monday, November 19, 2018 9:31 AM
To: Engbretson, Mark S. <
[email protected]>; Mark Rivers <[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark(s),

 

I am using simDetector 2-8, which I believe is the most recent.

My comment about the st.cmd script referred to the iocHDF5Test sub-folder (which I was seeking to exercise), and not the main iocSimDetector folder (where the script is correct).

- I’m sorry that was unclear.

 

Are you talking about EPICS_CA_MAX_ARRAY_BYTES?

It is set as an environment variable which is applied to shells through /etc/profile.d/custom.sh.

 

NB, data visualisation with ImageJ works for me with the ADURL module with the values given, but fails for the SimDetector module (with or without attempting to use HDF5).

 

Cheers,

 

Peter.

 

 

From: Engbretson, Mark S. <[email protected]>
Sent: 19 November 2018 15:04
To: Heesterman, Peter J <
[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

But did it make any difference? You might have to set this both in the ADURL and SIMDetector soft IOC st,cmd *AND* in for environment for IMAGEJ to work.

 

From: Heesterman, Peter J <[email protected]>
Sent: Monday, November 19, 2018 8:59 AM
To: Engbretson, Mark S. <
[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi Mark,

 

I have the latest sources.

My comment about the paths refers to the iocHDF5Test sub-folder that I was working with, not the iocSimDetector folder (where it’s correct).

I’m sorry this wasn’t clear.

 

I has set EPICS_CA_MAX_ARRAY_BYTES to 30000000 which is overkill I think, so I reduced it to 10000000.

 

Cheers,

 

Peter.

 

 

From: Engbretson, Mark S. <[email protected]>
Sent: 19 November 2018 14:44
To: Heesterman, Peter J <
[email protected]>
Subject: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

I don’t think that you have the latest sources for ADSimDetector since the paths are correct on the sources that I have.

 

If ImageJ   doesn’t work, chances are the value for

 

EPICS_CA_MAX_ARRAY_BYTES

 

Is either not set or set to be a value too small for the actual image data.

 

 

From: [email protected] <[email protected]> On Behalf Of Heesterman, Peter J via Tech-talk
Sent: Monday, November 19, 2018 8:24 AM
To: EPICS Tech-Talk (
[email protected]) <[email protected]>
Subject: ADSimDetector and ADURL - runtime issues with FFT and HDF5

 

Hi,

 

I’m seeking to use these modules with HDF5 data archiving.

Accordingly, I’ve been looking at the iocHDF5Test within ADSimDetector, and focused on test5.sh.

 

I’m working with an Ubuntu 18.04 VM.

I have:

    areaDetector and ADCore R3-3-2

    ADSupport R1-5

  ADViewers R1-2

    ADURL R2-2

    Asyn4-33

    Base 3.15.6

 

There are some issues that I need to raise:

 

a. st.cmd in ADSimDetector hasn’t been updated to reflect module paths – in particular lines like

        dbLoadRecords("$(ADCORE)/db/simDetector.template","P=$(PREFIX),R=cam1:,PORT=$(PORT),ADDR=0,TIMEOUT=1"), where the file is no longer in ADCore.

    This is easily corrected. Actually, I believe st_base.cmd from iocSimDetector can just be used.

 

b. When I run the sim detector and try to visualise with ImageJ, I get green on both 13SIM1:image1 and 13SIM1:image2.

    But, I don’t get any image displayed for either.

 

c. I note an error:

    NDPluginDriver::connectToArrayPort Error calling pasynManager->connectDevice to array port FFT1 address 0, status=3, error=asynManager:connectDevice port FFT1 not found.

 

    Could my (b) have anything to do with this error?

    I understand that image2 would be the FFT of the simulated data generated for image1?

 

d. I am able to create a HDF5 file (for either ADSimDetector or ADURL) as requested by the test5.sh script (I cloned the logic of this for the ADURL module).

    These files are large (>~100MB), and clearly contain the simulated and URL data, respectively.

    I’m able to open the file(s) with the hdfview application (after closing the IOC, to avoid any file write/read share issues).

    But, I’m unable to view any of the data because there are no attribute nodes under the ‘test5_001.h5’ root, in the viewer.

 

    Viewing the ‘properties’ on the ‘test5_001.h5’ node in hdfview, shows no attributes and no datasets.

 

    I refer to http://cars9.uchicago.edu/software/epics/NDFileHDF5.html , which I’ve studied in some detail.

 

I am able to visualise data from the ADURL module, quite normally.

 

What is it that I’m missing?

 

Many thanks,

 

Peter.

 

 


References:
ADSimDetector and ADURL - runtime issues with FFT and HDF5 Heesterman, Peter J via Tech-talk
RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Heesterman, Peter J via Tech-talk
RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Mark Rivers via Tech-talk
RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Heesterman, Peter J via Tech-talk
RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Mark Rivers via Tech-talk
RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Mark Rivers via Tech-talk

Navigate by Date:
Prev: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Mark Rivers via Tech-talk
Next: Question about this mailing list Mark Rivers via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: ADSimDetector and ADURL - runtime issues with FFT and HDF5 Mark Rivers via Tech-talk
Next: Controls Software Engineer at Berkeley Lab Robert Gunion via Tech-talk
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  <20182019  2020  2021  2022  2023  2024 
ANJ, 28 Nov 2018 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·