Re: Problem with huge waveforms in EPICS 7

[Michael Davidsaver via Core-talk <core-talk at aps.anl.gov>]

On 6/28/22 00:13, Zimoch Dirk (PSI) wrote:
> Confirmed with my test setup.
Applied as https://github.com/epics-base/epics-base/commit/4df48c91f4e7202899bd859312a2b68148bf0bad

I also made a note on the PR

https://github.com/epics-base/epics-base/pull/99#discussion_r910230977


> The real cameras on Windows will be tested soon but I am confident.
>
> Dirk
>
> On Mon, 2022-06-27 at 20:53 +0000, Mark Rivers wrote:
> > I tested Michael's patch and it fixes the problem when I run the ADSimDetector test.
> >   
> > (base) J:\epics\devel\asyn\iocBoot\ioctestAsynPortDriver>python monitor_test_ad.py
> > 2022-06-27 15:50:43.877772: Adding BIGARRAY callback function
> > 2022-06-27 15:50:43.878761: Sleeping for 5 seconds
> > 2022-06-27 15:50:45.787905: pv_callback counter=1, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=99
> > 2022-06-27 15:50:48.880187: Starting acquisition
> > 2022-06-27 15:50:48.901216: Waiting for acquisition to complete
> > 2022-06-27 15:50:50.290341: pv_callback counter=2, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=100
> > 2022-06-27 15:50:51.699526: pv_callback counter=3, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=112
> > 2022-06-27 15:50:52.709903: pv_callback counter=4, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=126
> > 2022-06-27 15:50:53.127619: pv_callback counter=5, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=136
> > 2022-06-27 15:50:53.545305: pv_callback counter=6, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=140
> > 2022-06-27 15:50:53.956352: pv_callback counter=7, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=144
> > 2022-06-27 15:50:54.371361: pv_callback counter=8, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=148
> > 2022-06-27 15:50:55.201308: pv_callback counter=9, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=152
> > 2022-06-27 15:50:55.626239: pv_callback counter=10, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=161
> > 2022-06-27 15:50:56.240083: pv_callback counter=11, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=165
> > 2022-06-27 15:50:56.654982: pv_callback counter=12, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=171
> > 2022-06-27 15:50:57.071780: pv_callback counter=13, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=175
> > 2022-06-27 15:50:57.495975: pv_callback counter=14, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=179
> > 2022-06-27 15:50:57.909298: pv_callback counter=15, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=183
> > 2022-06-27 15:50:58.536483: pv_callback counter=16, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=188
> > 2022-06-27 15:50:59.158556: pv_callback counter=17, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=194
> > 2022-06-27 15:50:59.574255: pv_callback counter=18, pvName=13SIM1:image1:ArrayData, size=20000000, value=[0]=199
> > 2022-06-27 15:50:59.576321: Acquisition complete
> > 2022-06-27 15:50:59.576321: Sleeping for 60 seconds
> > 2022-06-27 15:51:59.579580: Exiting
> >   
> > There are no callbacks after acquisition is complete.
> >   
> > Mark
> >   
> >   
> > -----Original Message-----
> > From: Core-talk <core-talk-bounces at aps.anl.gov> On Behalf Of Michael Davidsaver via Core-talk
> > Sent: Monday, June 27, 2022 9:51 AM
> > To: Zimoch Dirk (PSI) <dirk.zimoch at psi.ch>
> > Cc: core-talk at aps.anl.gov
> > Subject: Re: Problem with huge waveforms in EPICS 7
> >   
> > On 6/27/22 06:56, Zimoch Dirk (PSI) via Core-talk wrote:
> > > On Mon, 2022-06-27 at 15:12 +0200, Zimoch Dirk wrote:
> > > > The problem appears between commit 4ab98081 and 56f05d72. (The two
> > > > commits in between do not produce stable code.)
> > > Ben did some code refactoring of db_field_log which changed array handling.
> > >
> > > In particular in dbEvent.c the following snipped disappeared from
> > > db_queue_event_log()
> > >
> > >       /*
> > >        * if we have an event on the queue and both the last
> > >        * event on the queue and the current event are emtpy
> > >        * (i.e. of type dbfl_type_rec), simply ignore duplicate
> > >        * events (saving empty events serves no purpose)
> > >        */
> > >       if (pevent->npend > 0u &&
> > >           (*pevent->pLastLog)->type == dbfl_type_rec &&
> > >           pLog->type == dbfl_type_rec) {
> > >           db_delete_field_log(pLog);
> > >           UNLOCKEVQUE (ev_que);
> > >           return;
> > >       }
> > >
> > > I have added a debug printf in this code and see that it triggers
> > > whenever a frame is dropped. That is about 3 out of 4 frames in my test case.
> > >
> > > It is not trivial to put back the code because dbfl_type_rec disappeared from enum dbfl_type too.
> >   
> > I think the (untested) equivalent would now be:
> >   
> > > diff --git a/modules/database/src/ioc/db/dbEvent.c
> > > b/modules/database/src/ioc/db/dbEvent.c
> > > index 0164f55b9..480784d1e 100644
> > > --- a/modules/database/src/ioc/db/dbEvent.c
> > > +++ b/modules/database/src/ioc/db/dbEvent.c
> > > @@ -789,6 +789,18 @@ static void db_queue_event_log (evSubscrip
> > > *pevent, db_field_log *pLog)
> > >
> > >       LOCKEVQUE (ev_que);
> > >
> > > +    /* if we have an event on the queue and both the last
> > > +     * event on the queue and the current event are empty,
> > > +     * simply ignore duplicate events.
> > > +     */
> > > +    if (pevent->npend > 0u
> > > +            && !dbfl_has_copy(*pevent->pLastLog)
> > > +            && !dbfl_has_copy(pLog)) {
> > > +        db_delete_field_log(pLog);
> > > +        UNLOCKEVQUE (ev_que);
> > > +        return;
> > > +    }
> > > +
> > >       /*
> > >        * add to task local event que
> > >        */