EPICS Home EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

Format page
for printing


Search Tech-talk
2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <2022>  2023  2024  Index 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <2022>  2023  2024 
<== Date ==> <== Thread ==>
Subject: RE: Problem with huge waveforms in EPICS 7
From: Mark Rivers via Core-talk <core-talk at aps.anl.gov>
To: "Zimoch Dirk (PSI)" <dirk.zimoch at psi.ch>, Andrew Johnson <anj at anl.gov>
Cc: "core-talk at aps.anl.gov" <core-talk at aps.anl.gov>
Date: Sat, 25 Jun 2022 19:14:19 +0000

Hi Dirk,

 

I just tried to reproduce your problem using the database you sent

 

Bottom line: I reproduced your problem when the client that is monitoring DZ:BIGARRAY is running on another machine and the link between them does not support the full data rate.  The client continues to receive monitors for a long time after STOP has been sent.  You did not say in your message whether the client was on the same machine as the IOC, or was on another machine?

 

Andrew suggested this might be the issue:

 

The normal Linux scheduler tends to maximize throughput, not fairness, so it could be delaying the threads which process your STOP message while the threads handling image data can continue to make progress.

 

I don’t think that can be the problem because another client on the remote machine that is only monitoring the first 10 elements of DZ:BIGARRAY does not see the issue, i.e. as soon as the STOP is sent that client stops receiving monitor events.  This says that DZ:BIGARRAY did stop processing as soon as Python sent the STOP message.  However, somewhere in the CA callback system monitors are continuing to be sent to the bandwidth-limited client.

 

Details:

I had to modify it a little because I don’t have the sequence device support, and so if the INP field is DZ:COUNT then NORD is just 1, which is not a useful test.

 

This is my version:

(base) corvette:asyn/iocBoot/ioctestAsynPortDriver>more big_array.db

record (waveform, "DZ:BIGARRAY")

{

    field(FTVL, "SHORT")

    field(NELM, "20000000")

#    field(DTYP, "sequence")

    field(SCAN, ".1 second")

    field(SDIS, "DZ:STOP")

#    field(INP,  "DZ:COUNT")

    field(FLNK, "DZ:COUNT")

}

 

record (calc, "DZ:COUNT")

{

    field(CALC, "VAL+1")

}

 

record(bo, "DZ:STOP")

{

    field(ZNAM,"GO")

    field(ONAM,"STOP")

}

 

I then wrote a little Python program that does the following:

-          Writes 20000000 values to the waveform record so that NORD is the entire array

-          Enables monitors on the waveform record and specifies a callback function to be called when a new monitor arrives.  The monitor callback function prints the time and the size of the PV.

-          Sets the STOP record to GO

-          Sleeps for 10 seconds during which time the callbacks occur

-          Sets the STOP record to STOP

-          Sleeps for 10 seconds waiting for any additional callbacks

-          Exits

 

This is the Python program:

 

(base) corvette:asyn/iocBoot/ioctestAsynPortDriver>more monitor_test.py

from epics import PV

import numpy as np

import time

from datetime import datetime

 

def current_time():

    return datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')

 

def pv_callback(pvname=None, value=None, timestamp=None, **kw):

    print('%s: pv_callback pvName=%s, size=%d' % (current_time(), pvname, len(value)))

 

big_array = PV('DZ:BIGARRAY', auto_monitor=True)

print('%s: Writing BIGARRAY values' % (current_time()))

big_array.put(np.arange(1, 20000000, dtype=np.int16))

print('%s: Adding BIGARRAY callback function' % (current_time()))

big_array.add_callback(pv_callback)

start_stop = PV('DZ:STOP')

print('%s: Setting STOP to GO' % (current_time()))

start_stop.put(0)

 

print('%s: Sleeping for 10 seconds' % (current_time()))

time.sleep(10)

 

print('%s: Stopping acquisition' % (current_time()))

start_stop.put(1)

 

print('%s: Sleeping for 10 seconds' % (current_time()))

time.sleep(10)

 

print('%s: Exiting' % (current_time()))

 

 

This is the output of the Python program when it is running on the local IOC machine, so there is no low-bandwidth link between them.

 

(base) corvette:asyn/iocBoot/ioctestAsynPortDriver>python monitor_test.py

2022-06-25 13:25:41.214100: Writing BIGARRAY values

2022-06-25 13:25:44.763057: Adding BIGARRAY callback function

2022-06-25 13:25:44.798053: Setting STOP to GO

2022-06-25 13:25:44.815973: Sleeping for 10 seconds

2022-06-25 13:25:44.998700: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.067772: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.180062: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.290525: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.376936: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.471849: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.569029: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.669601: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.768242: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.878744: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:45.966175: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.069236: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.183709: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.266726: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.367657: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.469932: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.571696: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.667975: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.770982: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.880141: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:46.970064: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.074405: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.170555: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.267894: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.366129: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.477814: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.576328: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.666130: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.765597: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.868785: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:47.976573: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.076846: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.174300: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.268280: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.365140: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.466902: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.566294: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.677551: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.768485: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.866027: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:48.985653: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.082891: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.191633: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.265853: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.368805: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.491465: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.571327: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.672883: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.787401: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.875892: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:49.966585: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.076082: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.168870: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.293178: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.378965: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.483342: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.566776: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.676284: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.768054: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.865984: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:50.980998: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.065416: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.182154: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.265362: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.365551: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.463559: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.564197: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.667975: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.785107: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.868070: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:51.968567: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.065875: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.166434: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.268779: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.365338: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.465865: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.571419: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.670184: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.773308: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.865806: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:52.972999: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.069405: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.166225: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.265075: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.368060: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.481217: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.570139: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.662794: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.767850: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.891931: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:53.969974: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.075642: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.175603: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.292386: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.369205: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.479541: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.568244: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.674201: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.782411: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.824534: Stopping acquisition

2022-06-25 13:25:54.824791: Sleeping for 10 seconds

2022-06-25 13:25:54.881902: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:25:54.982720: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:26:04.827368: Exiting

 

The Python program was running on the same Linux machine as the IOC.  It was getting the callbacks at 10 Hz, so it was able to keep up at the full data rate.   It only received 2 callbacks after setting DZ:STOP to STOP.

 

At the same time that the Python program was running I was running camonitor of DZ:STOP and the first 10 elements of DZ:BIGARRAY.  This was also running on the same machine as the IOC and the Python program.

 

corvette:asyn/iocBoot/ioctestAsynPortDriver>camonitor -#10 DZ:STOP DZ:BIGARRAY

DZ:STOP                        2022-06-25 13:25:20.242761 1 STOP

DZ:BIGARRAY 2022-06-25 13:25:20.179685 10 1 2 3 4 5 6 7 8 9 10 DISABLE NO_ALARM

DZ:STOP                        2022-06-25 13:25:44.815972 1 GO

DZ:BIGARRAY 2022-06-25 13:25:44.879700 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:44.979683 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.079684 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.179702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.279674 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.379706 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.479680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.579676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.679719 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.779688 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.879708 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:45.979684 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.079676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.179680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.279714 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.379680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.479683 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.579672 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.679720 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.779690 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.879718 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:46.979708 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.079675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.179677 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.279674 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.379715 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.479702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.579669 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.679695 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.779724 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.879683 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:47.979712 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.079678 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.179683 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.279717 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.379721 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.479703 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.579736 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.679675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.779710 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.879710 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:48.979708 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.079697 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.179682 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.279705 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.379707 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.479701 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.579673 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.679684 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.779699 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.879725 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:49.979701 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.079680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.179710 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.279713 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.379704 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.479723 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.579685 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.679697 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.779687 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.879708 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:50.979721 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.079675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.179703 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.279705 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.379714 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.479712 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.579690 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.679679 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.779691 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.879676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:51.979748 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.079686 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.179716 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.279686 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.379681 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.479678 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.579680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.679680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.779686 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.879711 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:52.979690 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.079685 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.179687 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.279683 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.379697 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.479706 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.579718 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.679700 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.779689 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.879742 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:53.979720 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.079702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.179707 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.279689 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.379686 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.479731 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.579710 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.679732 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:25:54.779687 10 1 2 3 4 5 6 7 8 9 10

DZ:STOP                        2022-06-25 13:25:54.824925 1 STOP

DZ:BIGARRAY 2022-06-25 13:25:54.779687 10 1 2 3 4 5 6 7 8 9 10 DISABLE NO_ALARM

 

camonitor only got one callback after STOP was set to STOP.  The monitor for STOP is at 13:25:54.824925  which is within 1 ms of the time when Python set that value, 13:25:54.824534

 

Conclusion: there is no problem when the client is running on the same machine as the IOC and can receive monitors at the full rate.

 

I then tested again with the Python program and camonitor running on a Windows machine that is connected to the IOC machine with a 1 Gbit/s link.

 

(base) J:\epics\devel\asyn\iocBoot\ioctestAsynPortDriver>python monitor_test.py

2022-06-25 13:49:05.835287: Writing BIGARRAY values

2022-06-25 13:49:10.575048: Adding BIGARRAY callback function

2022-06-25 13:49:10.940239: Setting STOP to GO

2022-06-25 13:49:10.964259: Sleeping for 10 seconds

2022-06-25 13:49:11.450059: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:11.908468: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:12.319079: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:12.821216: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:13.239216: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:13.656039: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:14.089180: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:14.505642: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:14.914650: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:15.329280: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:15.760579: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:16.170148: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:16.580252: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:16.987538: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:17.391743: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:17.807907: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:18.221297: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:18.631483: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:19.037221: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:19.446158: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:19.851599: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:20.257247: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:20.679623: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:20.964807: Stopping acquisition

2022-06-25 13:49:20.966808: Sleeping for 10 seconds

2022-06-25 13:49:21.087319: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:21.492251: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:21.902952: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:22.311530: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:22.722542: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:23.130133: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:23.537294: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:23.953661: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:24.368631: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:24.797462: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:25.214898: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:25.624171: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:26.043720: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:26.444878: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:26.863146: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:27.267655: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:27.674763: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:28.078046: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:28.483430: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:28.890307: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:29.298299: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:29.717302: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:30.133419: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:30.541612: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:30.959208: pv_callback pvName=DZ:BIGARRAY, size=19999999

2022-06-25 13:49:30.967216: Exiting

 

Note that is only gets monitors every 400 ms, about 2.5 per second because of the network bandwidth limit.

 

Conclusion: This reproduces Dirk’s problem.  Monitors continue to be received for the full 10 seconds after STOP was sent.  I increased the second sleep to 20 seconds, and the monitors were still coming after 20 seconds.

 

This is the output of camonitor on the Windows machine at the same time the above Python output was being generated:

 

J:\epics\devel\mca\iocBoot\iocWindows>camonitor -#10 DZ:STOP DZ:BIGARRAY

DZ:STOP                        2022-06-25 13:46:43.034423 1 STOP

DZ:BIGARRAY 2022-06-25 13:46:42.979677 10 1 2 3 4 5 6 7 8 9 10 DISABLE NO_ALARM

DZ:STOP                        2022-06-25 13:49:10.966713 1 GO

DZ:BIGARRAY 2022-06-25 13:49:10.979706 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.079679 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.179703 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.279702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.379702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.479680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.579681 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.679694 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.779679 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.879662 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:11.979749 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.079672 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.179708 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.279681 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.379676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.479672 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.579689 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.679675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.779697 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.879691 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:12.979679 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.079698 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.180442 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.279703 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.379696 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.479702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.579705 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.679680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.779696 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.879677 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:13.979700 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.079627 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.179698 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.279694 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.379713 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.479704 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.579716 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.679677 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.779679 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.879703 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:14.979709 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.079683 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.179675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.279675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.379684 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.479675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.579687 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.679678 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.779721 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.879677 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:15.979695 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.079676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.179699 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.279671 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.379702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.479677 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.579711 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.679693 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.779680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.879673 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:16.979698 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.079670 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.179678 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.279669 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.379684 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.479702 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.579693 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.679748 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.779750 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.879680 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:17.979701 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.079673 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.179706 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.279697 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.379675 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.479698 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.579677 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.679684 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.779668 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.879695 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:18.979682 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.079676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.179710 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.279713 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.382961 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.479686 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.579712 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.679714 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.779692 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.879706 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:19.979707 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.079670 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.179709 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.279678 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.379699 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.479721 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.579676 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.679716 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.779695 10 1 2 3 4 5 6 7 8 9 10

DZ:BIGARRAY 2022-06-25 13:49:20.879697 10 1 2 3 4 5 6 7 8 9 10

DZ:STOP                        2022-06-25 13:49:20.969702 1 STOP

DZ:BIGARRAY 2022-06-25 13:49:20.879697 10 1 2 3 4 5 6 7 8 9 10 DISABLE NO_ALARM

 

It works fine.  It receives the monitors at 10 Hz because it only monitors 10 elements.  

 

The Python program sent the STOP at 13:49:20.964807.  That is within 5 ms of when camonitor got the callback for STOP, and stopped receiving arrays.

 

Only 1 additional monitors was received after the STOP.

 

Mark

 

From: Core-talk <core-talk-bounces at aps.anl.gov> On Behalf Of Zimoch Dirk (PSI) via Core-talk
Sent: Saturday, June 25, 2022 4:03 AM
To: Andrew Johnson <anj at anl.gov>
Cc: core-talk at aps.anl.gov
Subject: Re: Problem with huge waveforms in EPICS 7

 

Hi Andrew,

 

The cameras run on Windows. I did my test on Linux but not as root, thus I had no RT scheduling. I will repeat the test on Monday running as root.

 

The STOP message gets processed in time! A client that does not monitor the array sees the change immediately. The counter stops. But CA keeps sending!

 

I had expected that the IOC would drop frames if CA cannot send fast enough. Not trying for minutes to work through a pile of unsent frames. And then not even sending updates but simply repeating the last frame.

 

Dirk



Am 24.06.2022 um 18:10 schrieb Andrew Johnson via Core-talk <core-talk at aps.anl.gov>:

 Hi Dirk,

What OS is the IOC running on — I'm guessing Linux but you didn't say. If so is it built for and using priority thread scheduling? If the OSSPRI field from epicsThreadShowAll is all zeros it isn't, and enabling that might help. The normal Linux scheduler tends to maximize throughput, not fairness, so it could be delaying the threads which process your STOP message while the threads handling image data can continue to make progress. However this is just a guess.

- Andrew


General musings: The setpriority(2) manpage on RHEL-7 says:

BUGS
       According  to  POSIX, the nice value is a per-process setting.  However, under
       the current Linux/NPTL implementation of POSIX threads, the nice  value  is  a
       per-thread attribute: different threads in the same process can have different
       nice values.  Portable applications should avoid relying on the  Linux  behav‐
       ior, which may be made standards conformant in the future.


I wonder whether we should look at setting nice values for Linux threads when the process doesn't have the ability to use SCHED_FIFO?


On 6/24/22 10:39 AM, Zimoch Dirk (PSI) via Core-talk wrote:

Hi folks,
 
Some of or users complained that a camera server became less responsive since it had been upgraded from EPICS 3.14.12.6
to 7.0.6.1.
 
The camera sends image data as arrays of 20000000 SHORTs (5000x4000 pixels). When the user presses the "STOP" button on
the client which displays the image, it takes a long time to stop. The more active clients, the longer it takes.
But even sending stop from a different client (e.g. command line caput) takes a long time before the GUI clients update.
 
I have set up a simple simulation and run it with 'var CADEBUG 3'
Here is what I see: on EPICS 7.0.6.1
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: TCP Request from 129.129.130.117:47142 => cmmd=4 (CA_PROTO_WRITE) cid=0x4 type=0 count=1 postsize=8 version=13
CAS: Request from 129.129.130.117:47142 =>   available=0x2    N=1 paddr=0x7efcb800db80
CAS: Request from 129.129.130.117:47142 =>   Wrote string "STOP"
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
[>80 times the same!]
CAS: Sending a message of 40000056 bytes <---- I think this one contains the update of the STOP button
CAS: Sending a message of 40000032 bytes
[eventually stops many seconds later]
 
The IOC obviously gets the STOP message immediately when I press the button on the client. But the client (and any other
client showing the image) does not see the button change. The GUI appears "frozen". But a command line camonitor
monitoring the stop button (and a counter that counts the number of created images but not the image itself) show that
the records stop immediately.
Nevertheless the IOC keeps sending images. But the images do not change any more on the clients. So it seems that the
IOC keeps sending the same array data over and over again.
 
On 3.14.12, the output looks similar, but the "send after stop" consists of only a few messages:
CAS: Request from 129.129.130.117:47184 => cmmd=4 cid=0x1 type=0 count=1 postsize=8
CAS: Request from 129.129.130.117:47184 =>   available=0x2    N=1 paddr=0x7f0768010b28
CAS: Request from 129.129.130.117:47184 =>   Wrote string "STOP"
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000032 bytes
CAS: Sending a message of 40000056 bytes <---- update of the STOP button
 
What can be wrong here? 
The IOC consists of a counting calc, a bo for the stop switch and a waveform record with a driver that simply fills the
waveform with a sequence starting at the counter value. Nothing fancy.
 
Here is my db:
 
record (waveform, "DZ:BIGARRAY")
{
    field(FTVL, "SHORT")
    field(NELM, "20000000")
    field(DTYP, "sequence")
    field(SCAN, ".1 second")
    field(SDIS, "DZ:STOP")
    field(INP,  "DZ:COUNT")
    field(FLNK, "DZ:COUNT")
}
 
record (calc, "DZ:COUNT")
{
    field(CALC, "VAL+1")
}
 
record(bo, "DZ:STOP")
{
    field(ZNAM,"GO")
    field(ONAM,"STOP")
}
 
I suspect this happens when record produces new waveforms faster than they can be sent.
The IOC has no problem processing the waveform at 10 Hz, but I see only about 3 CAS messages per second.
I had to slow down the waveform processing to ".5 second" to improves responsiveness. That is when the monitor updates
can be sent as quickly as being produced. But opening a second client again spoils everything.
 
Dirk
 
 



-- 
Complexity comes for free, Simplicity you have to work for.
Replies:
RE: Problem with huge waveforms in EPICS 7 Mark Rivers via Core-talk
RE: Problem with huge waveforms in EPICS 7 Mark Rivers via Core-talk
References:
Problem with huge waveforms in EPICS 7 Zimoch Dirk (PSI) via Core-talk
Re: Problem with huge waveforms in EPICS 7 Andrew Johnson via Core-talk
Re: Problem with huge waveforms in EPICS 7 Zimoch Dirk (PSI) via Core-talk
Navigate by Date:
Prev: Re: Problem with huge waveforms in EPICS 7 Mark Rivers via Core-talk
Next: RE: Problem with huge waveforms in EPICS 7 Mark Rivers via Core-talk
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <2022>  2023  2024 
Navigate by Thread:
Prev: Re: Problem with huge waveforms in EPICS 7 Zimoch Dirk (PSI) via Core-talk
Next: RE: Problem with huge waveforms in EPICS 7 Mark Rivers via Core-talk
Index: 2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  <2022>  2023  2024 
ANJ, 14 Sep 2022 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·