On 2/26/25 02:19, Matt Rippa via Tech-talk wrote:
...
When the issue shows up the PVAccess updates slow way down but don't completely stop. The 2 healthy TCP-rx and TCP-tx threads are properly running from what I can see and they're properly set with FIFO and assigned to cores 7 and 6:
This is likely another occurrence of a reference loop leading to a resource leak.
https://urldefense.us/v3/__https://github.com/epics-base/pvAccessCPP/issues/190__;!!G_uCfscf7eWS!bwqcecbJ-jvM8kvsnHSujanwRl5yZwNVM15Db3kmfPZBuKOMOaHydCRHZQJRTiv9ev5AIM3uGS758xPgmeis2Daf7Q$
My immediate recommendation is to migrate to PVXS and QSRV2, which should not have issues of this sort.
TCP-tx 0x7f79e40220f0 2132240 20 20 OK *FIFO* 6
TCP-rx 0x7f79e4021cc0 2132239 20 20 OK *FIFO* 7
The full mcoreThreadShowAll is below. Anything we should pay attention to?
Thanks!
-Matt
4.18.0-513.18.1.rt7.320.el8_9.x86_64 #1 SMP PREEMPT_RT
[matt.rippa@hbfm2ts-lp2 epics-base]$ git submodule
+dead44c3cbe70b134766af13a56b571dbb40a278 .ci (v3.4.1-3-gdead44c)
7a2d264f2cb107bfd10adb23bc2b73d8323a79e4 modules/normativeTypes (6.0.1-3-g7a2d264)
+f1268adb8ecbacbd74bb66c172d02d9d427bedfd modules/pvAccess (7.1.7-2-gf1268adb)
+144f0228cc412d2dc1eaad7e09e310697d18532f modules/pvData (8.0.6-2-g144f022)
+f207e512d67addab79e33a00b712e3444228ba7c modules/pvDatabase (4.7.1-4-gf207e51)
949b3f63c2387bb92c1c22ca2f80f8d320805117 modules/pva2pva (1.4.1-1-g949b3f6)
8ed07fef96e41d35d47ab61276e29eb1a81e7fec modules/pvaClient (4.8.0-1-g8ed07fe)
[matt.rippa@hbfm2ts-lp2 epics-base]$ git lg -5
* 5dfc6caf3- (1 year, 1 month ago)Accept should return SOCKET rather than int- Freddie Akeroyd(HEAD -> upstream-7, upstream/7.0)
* cb49bd013- (12 months ago)Update ci-scripts to 3.4.1- Ralph Lange
* 4720b61c1- (1 year, 1 month ago)Move call to setThreadName()- Freddie Akeroyd
* 4383cf291- (1 year ago)allow macros with defaults in dbLoadRecords without substitutions- Dirk Zimoch
* a6977ae73- (1 year ago)Fix issue where VSCode makefile extension can delete files- Simon Rose
#!../../bin/linux-x86_64/m2ts
< envPaths
epicsEnvSet("IOC","iocm2ts")
epicsEnvSet("TOP","/home/matt.rippa/work/m2-cem-project/m2ts2")
epicsEnvSet("EPICS_BASE","/home/matt.rippa/work/vendor/epics-base")
epicsEnvSet("MCUtils","/home/matt.rippa/work/m2-cem-project/m2ts2/MCoreUtils")
epicsEnvSet("GCBRec","/home/matt.rippa/work/m2-cem-project/m2ts2/gcbCommandRecord")
cd "/home/matt.rippa/work/m2-cem-project/m2ts2"
epicsEnvSet("IOCSH_PS1","m2ts2> ")
epicsEnvSet("EPICS_PVA_ADDR_LIST","10.1.2.173")
epicsEnvSet("EPICS_PVA_SERVER_PORT","41366")
## Register all support components
dbLoadDatabase "dbd/m2ts.dbd"
m2ts_registerRecordDeviceDriver pdbbase
MCoreUtils version 1.2.3-SNAPSHOT
MCoreUtils: Read 26 thread rule(s) from /etc/rtrules
MCoreUtils: Read 0 thread rule(s) from /root/.rtrules
## Load record instances
#dbLoadTemplate "db/user.substitutions"
#dbLoadRecords "db/m2tsVersion.db", "user=mrippa"
dbLoadRecords "db/AP323.db", "m2top="
#var mySubDebug 1
traceIocInit
iocInit will be traced
M2TSStartup
cd "/home/matt.rippa/work/m2-cem-project/m2ts2/iocBoot/iocm2ts"
iocInit
iocInit: Reached initHookAtIocBuild
Starting iocInit
iocInit: Reached initHookAtBeginning
#!../../bin/linux-x86_64/m2ts
< envPaths
epicsEnvSet("IOC","iocm2ts")
epicsEnvSet("TOP","/home/matt.rippa/work/m2-cem-project/m2ts2")
epicsEnvSet("EPICS_BASE","/home/matt.rippa/work/vendor/epics-base")
epicsEnvSet("MCUtils","/home/matt.rippa/work/m2-cem-project/m2ts2/MCoreUtils")
epicsEnvSet("GCBRec","/home/matt.rippa/work/m2-cem-project/m2ts2/gcbCommandRecord")
cd "/home/matt.rippa/work/m2-cem-project/m2ts2"
epicsEnvSet("IOCSH_PS1","m2ts2> ")
epicsEnvSet("IOCSH_PS1","m2ts2> ")
epicsEnvSet("EPICS_PVA_ADDR_LIST","10.1.2.173")
epicsEnvSet("EPICS_PVA_SERVER_PORT","41366")
## Register all support components
dbLoadDatabase "dbd/m2ts.dbd"
m2ts_registerRecordDeviceDriver pdbbase
MCoreUtils version 1.2.3-SNAPSHOT
MCoreUtils: Read 26 thread rule(s) from /etc/rtrules
MCoreUtils: Read 0 thread rule(s) from /root/.rtrules
## Load record instances
#dbLoadTemplate "db/user.substitutions"
#dbLoadRecords "db/m2tsVersion.db", "user=mrippa"
dbLoadRecords "db/AP323.db", "m2top="
#var mySubDebug 1
traceIocInit
iocInit will be traced
M2TSStartup
iocInit
iocInit: Reached initHookAtIocBuild
Starting iocInit
iocInit: Reached initHookAtBeginning
############################################################################
## EPICS R7.0.8.1-DEV
## Rev. R7.0.8-16-g5dfc6caf3c898b213c84-dirty
## Rev. Date Git: 2024-03-06 09:48:26 -0600
############################################################################
iocInit: Reached initHookAfterCallbackInit
iocInit: Reached initHookAfterCaLinkInit
iocInit: Reached initHookAfterInitDrvSup
iocInit: Reached initHookAfterInitRecSup
iocInit: Reached initHookAfterInitDevSup
iocInit: Reached initHookAfterInitDatabase
iocInit: Reached initHookAfterFinishDevSup
initPeriodic: Scan rate '.015 second' is not achievable.
iocInit: Reached initHookAfterScanInit
iocInit: Reached initHookAfterInitialProcess
iocInit: Reached initHookAfterCaServerInit
iocInit: Reached initHookAfterIocBuilt
iocInit: Reached initHookAtIocRun
iocInit: Reached initHookAfterDatabaseRunning
iocInit: Reached initHookAfterInterruptAccept
iocInit: Reached initHookAfterCaServerRunning
iocInit: Reached initHookAtEnd
iocRun: All initialization complete
iocInit: Reached initHookAfterIocRunning
m2ts2> mcoreThreadShowAll
NAME EPICS ID LWP ID OSIPRI OSSPRI STATE POLICY CPUSET
_main_ 0x75b1d0 2989179 0 0 OK ? ?
errlog 0x775510 2989181 10 10 OK FIFO 0-2
TMpage1Writer 0x836ae0 2989182 87 86 OK FIFO 5
TMwaveformWriter 0x831040 2989183 86 85 OK FIFO 5
TMTemps 0x8311e0 2989184 81 80 OK FIFO 5
TMTempEnable 0x831430 2989185 70 69 OK FIFO 5
AP323_ISR_1 0x8316f0 2989186 91 90 OK FIFO 11
AP323_ISR_2 0x831950 2989187 91 90 OK FIFO 12
AP323_ISR_0 0x831c00 2989188 91 90 OK FIFO 10
M2MirrorControlT 0x831f80 2989189 90 89 OK FIFO 13
HSDataThread 0x832460 2989190 70 69 OK FIFO 5
M2VibrationContr 0x832790 2989191 90 89 OK FIFO 14
SafetyShutdown 0x832c80 2989192 90 89 OK FIFO 8
StatusManager 0x832f50 2989193 89 88 OK FIFO 5
m2Init 0x833290 2989194 80 79 OK FIFO 5
taskwd 0x8488b0 2989195 10 10 OK FIFO 0-2
timerQueue 0x833df0 2989196 70 69 OK FIFO 0-2
cbLow 0x8440a0 2989197 59 58 OK FIFO 0-2
cbMedium 0x8336f0 2989198 64 63 OK FIFO 0-2
cbHigh 0x844fc0 2989199 71 70 OK FIFO 0-2
dbCaLink 0x8484d0 2989200 50 50 OK FIFO 0-2
PVAL 0x848580 2989201 50 50 OK FIFO 0-2
PDB-event 0x834f60 2989202 19 19 OK FIFO 0-2
pvAccess-client 0x835ac0 2989203 35 35 OK FIFO 3
UDP-rx 0.0.0.0:0 <https://urldefense.us/v3/__http://0.0.0.0:0__;!!G_uCfscf7eWS!chyDZiPx_MJlvtvi1JKbrXm5N3_nuYHJROJpo37qY9RG7upFgvG0107Ng1DWkPTsh3_xuQ56s6dQwKdQQnR8_XStNzXo$> 0x8432d0 2989204 50 50 OK FIFO 6
UDP-rx 10.26.70. 0x843de0 2989205 50 50 OK FIFO 6
UDP-rx 10.26.70. 0x83f2b0 2989206 50 50 OK FIFO 6
UDP-rx 224.0.0.1 0x83fa20 2989207 50 50 OK FIFO 6
scanOnce 0x8407b0 2989208 68 67 OK FIFO 0-2
scan-10 0x840af0 2989209 65 64 OK FIFO 5
scan-5 0x840d40 2989210 66 65 OK FIFO 5
scan-2 0x840f90 2989211 67 66 OK FIFO 5
scan-1 0x8411e0 2989212 68 67 OK FIFO 5
scan-0.5 0x841430 2989213 69 68 OK FIFO 5
scan-0.2 0x841680 2989214 70 69 OK FIFO 5
scan-0.1 0x9c2500 2989215 71 70 OK FIFO 5
scan-0.015 0x9c2750 2989216 72 71 OK FIFO 5
CAS-TCP 0x9cb050 2989217 16 16 OK FIFO 7
CAS-UDP 0x9cb2a0 2989218 12 12 OK FIFO 7
CAS-beacon 0x9cb4f0 2989219 14 14 OK FIFO 7
ipToAsciiProxy 0x7fb8c400f7d0 2989220 10 10 OK FIFO 0-2
PVAS timers 0x9cbb20 2989221 25 25 OK FIFO 0-2
timerQueue 0x7fb8c400ff30 2989222 52 51 OK FIFO 0-2
TCP-acceptor 0x9cc6a0 2989223 50 50 OK FIFO 6
CAC-UDP 0x7fb8c40110b0 2989224 54 53 OK FIFO 6
UDP-rx 0.0.0.0:0 <https://urldefense.us/v3/__http://0.0.0.0:0__;!!G_uCfscf7eWS!chyDZiPx_MJlvtvi1JKbrXm5N3_nuYHJROJpo37qY9RG7upFgvG0107Ng1DWkPTsh3_xuQ56s6dQwKdQQnR8_XStNzXo$> 0x9ed1f0 2989225 50 50 OK FIFO 6
UDP-rx 10.26.70. 0xa2dca0 2989226 50 50 OK FIFO 6
UDP-rx 10.26.70. 0xa2e0b0 2989227 50 50 OK FIFO 6
UDP-rx 224.0.0.1 0xa4e7a0 2989228 50 50 OK FIFO 6
m2tsStateTelemTh 0xa82520 2989229 90 89 OK FIFO 3
gcbCommandM 0xa8a890 2989230 81 80 OK FIFO 9
fastguider 0xaa7d90 2989231 89 88 OK FIFO 13
GCBProcessor 0xaa7ff0 2989233 84 83 OK FIFO 3
TCP-tx 0x7fb8680220f0 2989269 20 20 OK FIFO 6
TCP-rx 0x7fb868021cc0 2989268 20 20 OK FIFO 7
UDP-rx 224.0.0.1 0x7fb848058a30 2990167 50 50 OK FIFO 6
UDP-rx 10.26.70. 0x7fb848027c50 2990166 50 50 OK FIFO 6
UDP-rx 10.26.70. 0x7fb848006a60 2990165 50 50 OK FIFO 6
UDP-rx 0.0.0.0:0 <https://urldefense.us/v3/__http://0.0.0.0:0__;!!G_uCfscf7eWS!chyDZiPx_MJlvtvi1JKbrXm5N3_nuYHJROJpo37qY9RG7upFgvG0107Ng1DWkPTsh3_xuQ56s6dQwKdQQnR8_XStNzXo$> 0x7fb848068700 2990164 50 50 OK FIFO 6
pvAccess-client 0x7fb8480062f0 2990163 35 35 OK FIFO 3
...
/etc/rtrules:
# Format of each line: name:policy:priority:affinity:pattern
#
# name distinguishing tag
# policy scheduling policy (first letter suffices, case independent, * = don't change)
# priority scheduling priority (OSI units, + or - defines a relative change, * = don't change)
# affinity CPU set (use , and - to specify ranges, * = don't change)
# pattern regular expression to match thread names against
#
# Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
errlog:f:10:4:errlog
m2Init:f:80:5:m2Init
HSDataT:f:70:5:HSDataThread
TMTempE:f:70:5:TMTempEnable
TMTemps:f:81:5:TMTemps
TMwfW:f:86:5:TMwaveformWriter
TMp1W:f:87:5:TMpage1Writer
StatMan:f:89:5:StatusManager
TCP-acc:f:*:6:TCP-acceptor.*
TCP-tx:f:*:6:TCP-tx.*
TCP-rx:f:*:7:TCP-rx.*
SSDTask:f:90:8:SafetyShutdown
gcbCommandM:f:81:9:gcbCommandM*
AP323_ISR_0:f:91:10:AP323_ISR_0
AP323_ISR_1:f:91:11:AP323_ISR_1
AP323_ISR_2:f:91:12:AP323_ISR_2
fastguider:f:89:13:fastguider
MCT1:f:90:13:M2MirrorControlT1
VCT1:f:90:14:M2VibrationControlT1
# The gcbCommandQueue is the GCBProcessor
gcbProc:f:84:3:GCBProcessor
# The Gui Status Record has is m2tsStateTelem
m2tsStateTelemTh:f:90:3:m2tsStateTelemTh
# pvaccess-client
pva-c:f:*:3:pvAccess-client
# set CAS threads to SCHED_RR on CPU 7
CAS-all:f:*:7:CAS-.*
# set CAC threads to SCHED_RR on CPU 6
CAC-all:f:*:6:CAC-.*
UDP-all:f:*:6:UDP-.*
# increase priority of all scan tasks by 5
scan:*:+5:5:scan-.*
- References:
- Run-away TCP-tx and TCP-rx Matt Rippa via Tech-talk
- Navigate by Date:
- Prev:
Re: Re: History of EPICS Nonn, Patrick via Tech-talk
- Next:
Re: [EXTERNAL] History of EPICS Johnson, Andrew N. via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
<2025>
- Navigate by Thread:
- Prev:
Run-away TCP-tx and TCP-rx Matt Rippa via Tech-talk
- Next:
Help interpreting CA protocol Nicholas Devenish via Tech-talk
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
<2025>
| 
·
Home
·
News
·
About
·
Base
·
Modules
·
Extensions
·
Distributions
·