Experimental Physics and
Industrial Control System

Mark Engbretson <[email protected]> · Tue, 26 Jun 2018 12:15:49 -0500

Ok, thanks for the explanation.

-----Original Message-----
From: Mark Rivers <[email protected]> 
Sent: Tuesday, June 26, 2018 11:41 AM
To: 'Mark Engbretson' <[email protected]>; [email protected]
Subject: RE: Question about displaying process memory allocation on Linux

> Hmmm,  I have an AD-3-2 Area Detector that when I set a large queue size,
more than available computer memory, and it gets filled, if someone is
attempting to acquire 10-20 FPS faster than the hardware can support, it
aborts/crashes.

ADCore does not look at the available computer memory, it looks at the
parameter maxMemorySize that was passed to the driver constructor.  If that
value is 0 then there is no memory checking done, and it may bring the
system to its knees if the IOC tries to allocate too much memory.

In the case that maxMemorySize is a non-zero value then NDArrayPool::alloc()
will return a NULL pointer if it could not allocate the array.  It is then
the responsibility of the driver or plugin to check for the NULL pointer and
take the appropriate action to avoid a crash.  I don't guarantee that all
drivers and plugins have been written to properly handle a NULL pointer
return, so the IOC may still crash.  The main goal of the memory checking is
to prevent the areaDetector IOC from consuming all of the memory on the
system, and bringing the entire computer down.  Graceful handling of the
memory shortage in the IOC would be nice, but has been a secondary
consideration.

Mark

-----Original Message-----
From: Mark Engbretson <[email protected]> 
Sent: Tuesday, June 26, 2018 10:10 AM
To: Mark Rivers <[email protected]>; [email protected]
Subject: RE: Question about displaying process memory allocation on Linux

Hmmm,  I have an AD-3-2 Area Detector that when I set a large queue size,
more than available computer memory, and it gets filled, if someone is
attempting to acquire 10-20 FPS faster than the hardware can support, it
aborts/crashes. 

Ok, will give 3-3 a go. Thanks.

-----Original Message-----
From: Mark Rivers <[email protected]>
Sent: Tuesday, June 26, 2018 9:28 AM
To: Mark Engbretson <[email protected]>; [email protected]
Subject: Re: Question about displaying process memory allocation on Linux

Hi Mark,

> but do you have a suggested place where you might suggest putting
NDArrayPool::emptyFreeList() so that memory will always be flushed, when it
can be?

That's a tricky question.  The point of the free list is that it should be
more efficient than calling malloc() and free() for the large NDArrays each
time a new one is allocated and freed.  If the free list were regularly
emptied it would defeat this optimization.  That being said I have not
actually benchmarked the time to use NDArrayPool->alloc vs calling new
NDArray(ndims, dims, dataType, 0, NULL).

> And perhaps a place where it can be done automagically when memory is
about to be exceeded so that AD just starts to dump frames instead of
crashing because there is no free memory left?

If by "no free memory left" you mean exceeding maxMemorySize then that is
already the case, and has been for a long time.  If NDArrayPool->alloc()
would exceed the maxMemorySize then it deletes NDArrays from the free list
until enough memory is available (if possible).

However, prior to R3-3 each driver and plugin had its own NDArrayPool, and
so it was difficult to use maxMemorySize effectively.  It is the total
memory in the IOC that matters, but previously one had to set the
maxMemorySize for each driver and plugin.  In R3-3 all plugins get their
NDArrays from the driver's NDArrayPool, so a single setting of maxMemorySize
is all that is needed, and it controls that driver and all of its plugins.

Mark

________________________________
From: Mark Engbretson <[email protected]>
Sent: Tuesday, June 26, 2018 9:03 AM
To: Mark Rivers; [email protected]
Subject: RE: Question about displaying process memory allocation on Linux

Not an answer to your Linux issues . . . sorry . . . . but do you have a
suggested place where you might suggest putting NDArrayPool::emptyFreeList()
so that memory will always be flushed, when it can be? And perhaps a place
where it can be done automagically when memory is about to be exceeded so
that AD just starts to dump frames instead of crashing because there is no
free memory left?

From: [email protected] <[email protected]> On
Behalf Of Mark Rivers
Sent: Tuesday, June 26, 2018 8:52 AM
To: [email protected]
Subject: Question about displaying process memory allocation on Linux

Folks,

In areaDetector the NDArrays are allocated from a free list in NDArrayPool.
Previously once a large number of arrays had been allocated on the free list
there was no way to free that memory back to the OS without restarting the
IOC.  I have now added an NDArrayPool::emptyFreeList() method and an
EmptyFreeList record that results in a call to that method.

When I test on Windows and I look at the Commit Size for the simDetector
process in Task Manager I see exactly what I expect.  If I have large queues
and the detector is acquiring faster than the plugins can process the free
list contains NDArrays totaling over 3GB.   The Commit Size is ~3.2 GB.
When I process the EmptyFreeList record the CommitSize immediately drops to
0.2 GB.

I am trying to see if I see the same thing on Linux, using the ps -o
trs,vsz,drs command.

This is what I see when I first start the simDetector IOC, before I collect
any images.  Note that the virtual memory size (VSZ) is already 5GB.  This
seems strange.  Can anyone explain?

corvette:dxpSITORO/iocBoot/iocFalconX1>ps -p 115038 -o trs,vsz,drs

TRS    VSZ   DRS

15838 5080984 5065145

This is what I see after collecting enough images that there is a total of
3GB of NDArrays allocated.  VSZ has increased by 3GB (from 5GB to 8GB),
which is what I expect.

corvette:dxpSITORO/iocBoot/iocFalconX1>ps -p 115038 -o trs,vsz,drs

TRS    VSZ   DRS

15838 8171424 8155585

This is what I see when acquisition has stopped and all plugins have
competed.  The NDArrays are now all on the free list, VSZ remains at 8 GB,
which is what I expect.

corvette:dxpSITORO/iocBoot/iocFalconX1>ps -p 115038 -o trs,vsz,drs

TRS    VSZ   DRS

15838 8171424 8155585

This is what I see after I process the EmptyFreeList record.  Note that VSZ
has only decreased by about 0.2GB, not by 3GB which I would have expected.
Can anyone explain?

corvette:dxpSITORO/iocBoot/iocFalconX1>ps -p 115038 -o trs,vsz,drs

TRS    VSZ   DRS

15838 7974816 7958977

Is there another statistic I can look at to see that my process has indeed
returned all 3GB to the OS?

Thanks,

Mark

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System