Re: areaDetector and HDF5 file saving segafaults

[Mark Rivers <rivers@cars.uchicago.edu>]

Hi Hinko,


Since you are using busy 1-6-1 I don't think the crash is due to the busy record.  It does not have the bug which was causing crashes for me.


This is the code where it is crashing in H5Pdcpl.c

        /* Set the fill value */
        if(NULL == (fill.type = H5T_copy(type, H5T_COPY_TRANSIENT)))
            HGOTO_ERROR(H5E_PLIST, H5E_CANTCOPY, FAIL, "can't copy datatype")
        fill.size = (ssize_t)H5T_get_size(type);
        if(NULL == (fill.buf = H5MM_malloc((size_t)fill.size)))
            HGOTO_ERROR(H5E_RESOURCE, H5E_CANTINIT, FAIL, "memory allocation failed for fill value")
        HDmemcpy(fill.buf, value, (size_t)fill.size);   CRASHES HERE

That code is being called from here in NDFileHDF5AttributeDataset::createHDF5Dataset():


asynStatus NDFileHDF5AttributeDataset::createHDF5Dataset()
{
  asynStatus status = asynSuccess;

  cparm_ = H5Pcreate(H5P_DATASET_CREATE);

  H5Pset_fill_value(cparm_, datatype_, ptrFillValue_);  CRASHES IN CALL FROM HERE



This is the private data in the NDFileHDF5AttributeDataset class:

private:
  asynStatus createHDF5Dataset();
  asynStatus configureDims(int user_chunking);
  asynStatus configureDimsFromDataset(bool multiframe, int extradimensions, int *extra_dims, int *user_chunking);
  asynStatus typeAsHdf();
  void extendDataSet();
  void extendDataSet(hsize_t *offsets);
  void extendIndexDataSet(hsize_t offset);

  std::string      name_;            // Name of the attribute
  std::string      dsetName_;        // Name of the dataset to store
  hid_t            file_;            // File handle
  NDAttrDataType_t type_;            // NDAttribute type
  std::string      groupName_;       // Name of the parent group
  hid_t            dataset_;         // Dataset handle
  hid_t            dataspace_;
  hid_t            memspace_;
  hid_t            datatype_;
  hid_t            cparm_;
  hid_t            filespace_;
  void             *ptrFillValue_;
  hsize_t          *dims_;
  hsize_t          *offset_;
  hsize_t          *chunk_;
  hsize_t          *maxdims_;
  hsize_t          *virtualdims_;
  hsize_t          *elementSize_;
  bool             isUndefined_;
  int              rank_;            // number of dimensions
  int              nextRecord_;
  int              extraDimensions_;
  hdf5::When_t     whenToSave_;

};

I have 2 suggestions to track down the problem:

- Use gdb to examine the values of the variables in H5Pdcpl.c and the NDFileHDF5AttributeDataset private member data when it crashes

or

- Use asynPrint to print the values every time that function is called, and route the asynTrace output to a file, since it may be large.


Have you tried to reproduce it with the simDetector?


Mark





________________________________
From: Hinko Kocevar <hinkocevar@gmail.com>
Sent: Monday, November 13, 2017 2:32 AM
To: Mark Rivers
Cc: tech-talk@aps.anl.gov
Subject: Re: areaDetector and HDF5 file saving segafaults

Hi Mark,

On Thu, Nov 9, 2017 at 1:06 PM, Mark Rivers <rivers@cars.uchicago.edu> wrote:
> Hi Hinko,
>
>> I'm using Single saving mode, saving one image per acquisition at ~10 images per second.
>
> How is the file saving being done?  Are you setting the AutoSave PV to 1?   If so, what version of the "busy" module are you using?  I have recently seen a segfault when using AutoSave at much higher frame rates with the TIFF file plugin, and the same problem might well occur with the HDF5 plugin. I understand the problem, it has to do with the busy record asyn device support not correctly handling callbacks when they occur rapidly.  However, I don't think this happens with any official release of the busy module, only with changes made on the master branch. I am actively working on a fix for that problem.

Yes, I'm using autosave=1. Busy version is 1-6-1.

>
> It seems unlikely that hardware problems would cause the error with the stack trace you sent, but it's always possible I suppose.
>
> There was also a change in ADCore R3-0 to fix a problem storing NDAttributes in HDF5 files when those attributes were coming from EPICS PVs that were disconnected.  You are using that fixed version.  Is it possible that one of your attributes is coming from a disconnected EPICS PV?  If so, perhaps I introduced a bug.

I think all PVs should be connected - I'm only accessing PVs local to the IOC.

Thanks,
Hinko

>
> Mark
>
>
>
>
>
> ________________________________
> From: Hinko Kocevar <hinkocevar@gmail.com>
> Sent: Thursday, November 9, 2017 1:52 AM
> To: Mark Rivers
> Cc: tech-talk@aps.anl.gov
> Subject: Re: areaDetector and HDF5 file saving segafaults
>
> Hi Mark,
>
> Image size is 1936 x 1216, data type is uint16 (~2.3MB). Camera is AlliedVision Manta G-235B, controlled with aravis+aravisGigE.
> I'm saving some attributes - see the attached XML file.
> I'm using Single saving mode, saving one image per acquisition at ~10 images per second.
> It happens quite often, if I leave if running for couple of minutes I get the segfault.
> I have not yet tried reproducing with simDetector - will do.
> I'm using HDF5 ver 1.10.1 that comes with ADSupport.
>
> The thing is that I have not noticed this problems before, this now happens on a remote machine at an experiment we are conducting. It could be the hardware..
>
> Thank you,
> Hinko
>
> On Tue, Nov 7, 2017 at 2:59 PM, Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>> wrote:
> Hi Hinko,
>
>
> Also, what version of HDF5 are you using, the one from ADSupport (and if so what version) or a system library version?
>
>
> Mark
>
>
>
> ________________________________
> From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov>> on behalf of Hinko Kocevar <hinkocevar@gmail.com<mailto:hinkocevar@gmail.com>>
> Sent: Tuesday, November 7, 2017 3:58 AM
> To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
> Subject: areaDetector and HDF5 file saving segafaults
>
> Hi,
>
> I'm experiencing segmentation faults that trace to HDF5 file saving plugin. This is with ADAravis camera, 10 images per second.
>
> EPICS base is 3.15.4, AreaDetector is 3-1.
>
> epics>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7fffed55a700 (LWP 27283)]
> 0x00007ffff35d7d89 in __memcpy_ssse3_back () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007ffff35d7d89 in __memcpy_ssse3_back () at /lib64/libc.so.6
> #1  0x00007ffff6371211 in H5Pset_fill_value (plist_id=<optimized out>, type_id=216172782117023038, value=0x7ffd67ff2fb0) at ../H5Pdcpl.c:3217
> #2  0x00007ffff776bff7 in NDFileHDF5AttributeDataset::createHDF5Dataset() (this=this@entry=0x7ffd5c007460) at ../NDFileHDF5AttributeDataset.cpp:99
> #3  0x00007ffff776c648 in NDFileHDF5AttributeDataset::createDataset(int) (this=this@entry=0x7ffd5c007460, user_chunking=<optimized out>)
>     at ../NDFileHDF5AttributeDataset.cpp:75
> #4  0x00007ffff776529e in NDFileHDF5::createAttributeDataset(NDArray*) (this=this@entry=0x939530, pArray=pArray@entry=0x7fff10968800)
>     at ../NDFileHDF5.cpp:2535
> #5  0x00007ffff776717f in NDFileHDF5::openFile(char const*, int, NDArray*) (this=0x939530, fileName=0x7fffed5595d0 "/home/dev/DTUdata/image1_20896.h5", openMode=<optimized out>, pArray=0x7fff10968800) at ../NDFileHDF5.cpp:268
> #6  0x00007ffff775820d in NDPluginFile::openFileBase(int, NDArray*) (this=this@entry=0x939530, openMode=openMode@entry=2, pArray=pArray@entry=0x7fff10968800) at ../NDPluginFile.cpp:73
> #7  0x00007ffff7758ee4 in NDPluginFile::writeFileBase() (this=this@entry=0x939530) at ../NDPluginFile.cpp:230
> #8  0x00007ffff77597f8 in NDPluginFile::processCallbacks(NDArray*) (this=0x939530, pArray=0x7fff10968800) at ../NDPluginFile.cpp:751
> #9  0x00007ffff770871e in NDPluginDriver::processTask() (this=0x939530) at ../NDPluginDriver.cpp:467
> #10 0x00007ffff48d44c9 in epicsThreadCallEntryPoint(void*) (pPvt=0x7fffd4001400) at ../../../src/libCom/osi/epicsThread.cpp:85
> #11 0x00007ffff48d9f6c in start_routine (arg=0x7fffd4001310) at ../../../src/libCom/osi/os/posix/osdThread.c:403
> #12 0x00007ffff2f66e25 in start_thread () at /lib64/libpthread.so.0
> #13 0x00007ffff358534d in clone () at /lib64/libc.so.6
>
> Thanks,
> Hinko
>
> --
> .. the more I see the less I believe.., AE AoR
>
>
>
> --
> .. the more I see the less I believe.., AE AoR



--
.. the more I see the less I believe.., AE AoR