RE: quadEM FFP plugin update issues

[Mark Rivers via Tech-talk <tech-talk@aps.anl.gov>]

Hi Kiman,

 

Canmonitor shows that XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue is still updating at 10 Hz, it is not stopped.  However, the data is all NAN, so the problem appears to be bad data from the BPM.

 

Note that Vertical Current 4 FFT is fine, but Current 3 FFT is not.  That looks like perhaps Current 3 stopped working.  You should include Current3 and Current4 scalar PVs your CA monitor to see if perhaps Current3 stopped returning valid values.

 

When this happens also grab a screenshot of the Current3 time series.

 

Mark

 

 

From: Ha, Kiman <kha@bnl.gov>
Sent: Monday, September 29, 2025 2:19 PM
To: Mark Rivers <rivers@cars.uchicago.edu>; tech-talk@aps.anl.gov
Cc: Gavrilov, Dmitri <dgavrilov@bnl.gov>; Wlodek, Jakub <jwlodek@bnl.gov>
Subject: RE: quadEM FFP plugin update issues

 

Hi Mark,

 

After ~20 hours later it happened again PosY.FFTAbs stopped:

I changed FFT length 1024.

 

After stop waveform data shows “NaN“, however array count is still running.

 

 

 

[kha@xf03idc-ws1 ~]$ camonitor -#2 XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:20.680943 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:20.780961 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:20.880687 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:20.981006 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.082019 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.180937 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.281095 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.381122 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.481108 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.580926 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.681254 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.781217 2 0 nan 

XF:03IDC-BI{EM:1}FFT:PosY:FFTAbsValue 2025-09-29 15:06:21.881178 2 0 nan 

^C

[kha@xf03idc-ws1 ~]$ camonitor -#2 XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.080831 2 0 308.386 

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.181127 2 0 136.118 

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.281480 2 0 473.483 

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.380783 2 0 205.761 

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.481021 2 0 191.585 

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.581142 2 0 258.867 

XF:03IDC-BI{EM:1}FFT:PosX:FFTAbsValue 2025-09-29 15:08:20.681860 2 0 216.273

 

$ ps -eL | grep FFT

   5138   10557 pts/65   00:00:00 FFT1_updateQueu

   5138   12216 pts/65   00:00:00 FFT1_Plugin_1

   5146   10581 pts/67   00:00:00 FFT1_updateQueu

   5146   12045 pts/67   00:00:00 FFT1_Plugin_1

   5151    9999 pts/58   00:00:00 FFT1_updateQueu

   5151   12645 pts/58   00:00:00 FFT1_Plugin_1

   5339   10931 pts/80   00:00:00 FFT1_updateQueu

   5339   11756 pts/80   00:00:00 FFT1_Plugin_1

   5342    9988 pts/82   00:00:00 FFT1_updateQueu

   5342   12128 pts/82   00:00:00 FFT1_Plugin_1

   5343   10543 pts/79   00:00:00 FFT1_updateQueu

   5343   12733 pts/79   00:00:00 FFT1_Plugin_1

   5348    9984 pts/89   00:00:00 FFT1_updateQueu

   5348   11941 pts/89   00:00:00 FFT1_Plugin_1

   5349    9996 pts/85   00:00:00 FFT1_updateQueu

   5349   11632 pts/85   00:00:00 FFT1_Plugin_1

   5352    9993 pts/83   00:00:00 FFT1_updateQueu

   5352   12824 pts/83   00:00:00 FFT1_Plugin_1

   5489    9990 pts/103  00:00:00 FFT1_updateQueu

   5489   12418 pts/103  00:00:00 FFT1_Plugin_1

   6501   10946 pts/105  00:00:00 FFT1_updateQueu

   6501   12510 pts/105  00:00:00 FFT1_Plugin_1

1574116 1574229 pts/81   00:00:00 NSLS2_EM_FFT1_u

1574116 1574232 pts/81   00:00:00 NSLS2_EM_FFT2_u

1574116 1574235 pts/81   00:00:00 NSLS2_EM_FFT3_u

1574116 1574238 pts/81   00:00:00 NSLS2_EM_FFT4_u

1574116 1574241 pts/81   00:00:00 NSLS2_EM_FFT5_u

1574116 1574244 pts/81   00:00:00 NSLS2_EM_FFT6_u

1574116 1574247 pts/81   00:00:00 NSLS2_EM_FFT7_u

1574116 1574250 pts/81   00:00:00 NSLS2_EM_FFT8_u

1574116 1574253 pts/81   00:00:00 NSLS2_EM_FFT9_u

1574116 1574256 pts/81   00:00:00 NSLS2_EM_FFT10_

1574116 1574259 pts/81   00:00:00 NSLS2_EM_FFT11_

1574116 1574360 pts/81   00:00:55 NSLS2_EM_FFT1_P

1574116 1574362 pts/81   00:00:56 NSLS2_EM_FFT2_P

1574116 1574364 pts/81   00:00:56 NSLS2_EM_FFT3_P

1574116 1574366 pts/81   00:00:57 NSLS2_EM_FFT4_P

1574116 1574368 pts/81   00:00:57 NSLS2_EM_FFT5_P

1574116 1574370 pts/81   00:00:58 NSLS2_EM_FFT6_P

1574116 1574372 pts/81   00:00:58 NSLS2_EM_FFT7_P

1574116 1574374 pts/81   00:00:57 NSLS2_EM_FFT8_P

1574116 1574376 pts/81   00:00:55 NSLS2_EM_FFT9_P

1574116 1574378 pts/81   00:00:53 NSLS2_EM_FFT10_

1574116 1574380 pts/81   00:00:50 NSLS2_EM_FFT11_

2189366 2189484 pts/55   00:00:00 FFT1_updateQueu

2189366 2189604 pts/55   00:00:00 FFT1_Plugin_1

2639210 2639311 pts/92   00:00:00 NSLS2_EM_FFT1_u

2639210 2639314 pts/92   00:00:00 NSLS2_EM_FFT2_u

2639210 2639317 pts/92   00:00:00 NSLS2_EM_FFT3_u

2639210 2639320 pts/92   00:00:00 NSLS2_EM_FFT4_u

2639210 2639323 pts/92   00:00:00 NSLS2_EM_FFT5_u

2639210 2639326 pts/92   00:00:00 NSLS2_EM_FFT6_u

2639210 2639329 pts/92   00:00:00 NSLS2_EM_FFT7_u

2639210 2639332 pts/92   00:00:00 NSLS2_EM_FFT8_u

2639210 2639335 pts/92   00:00:00 NSLS2_EM_FFT9_u

2639210 2639338 pts/92   00:00:00 NSLS2_EM_FFT10_

2639210 2639341 pts/92   00:00:00 NSLS2_EM_FFT11_

2639210 2639432 pts/92   00:01:35 NSLS2_EM_FFT1_P

2639210 2639434 pts/92   00:01:38 NSLS2_EM_FFT2_P

2639210 2639436 pts/92   00:01:42 NSLS2_EM_FFT3_P

2639210 2639438 pts/92   00:01:42 NSLS2_EM_FFT4_P

2639210 2639440 pts/92   00:01:43 NSLS2_EM_FFT5_P

2639210 2639442 pts/92   00:01:43 NSLS2_EM_FFT6_P

2639210 2639444 pts/92   00:01:42 NSLS2_EM_FFT7_P

2639210 2639446 pts/92   00:01:41 NSLS2_EM_FFT8_P

2639210 2639448 pts/92   00:01:37 NSLS2_EM_FFT9_P

2639210 2639450 pts/92   00:01:33 NSLS2_EM_FFT10_

2639210 2639452 pts/92   00:01:28 NSLS2_EM_FFT11_

2649780 2649898 pts/25   00:00:00 FFT1_updateQueu

2649780 2650015 pts/25   00:00:00 FFT1_Plugin_1

3809994 3810144 pts/61   00:00:00 FFT1_updateQueu

3809994 3810267 pts/61   00:00:00 FFT1_Plugin_1

 

 

Thanks,

Kiman

 

From: Mark Rivers <rivers@cars.uchicago.edu>
Sent: Saturday, September 27, 2025 12:22 PM
To: Ha, Kiman <kha@bnl.gov>; tech-talk@aps.anl.gov
Subject: RE: quadEM FFP plugin update issues

 

Hi Kiman, I just tested on my system, and I can’t reproduce the problem. This is not related to the issue of the plugin updates stopping, but I would suggest that you change the number of points from 1000 to 1024. I think the “beats”

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Hi Kiman,

 

I just tested on my system, and I can’t reproduce the problem.

 

This is not related to the issue of the plugin updates stopping, but I would suggest that you change the number of points from 1000 to 1024.  I think the “beats” you see in frequency are artifacts of the fact that the number of points is not a power of 2.

 

This is what my Y position FFT looks like:

 

 

The first thing to do is open the detailed screens for that FFT plugin.  This is what I see with “More/FFT plugin”.  Note that Array rate=10 Hz, the execution time is 0.474 ms, and the array counter is incrementing at 10 Hz.  Are these values updating on your screen?

 

 

This is what I see when I press the More/NDPluginBase Full”  on the screen above:

 

 

The next thing to determine is whether Channel Access updates of the waveform record are being sent.  The way to test this is with camonitor.  This is what I see with camonitor on the X and Y positions, asking only for the first 2 points in the waveform.

 

(base) [epics@corvette ~]$ camonitor -#2 13IDA:QE1:FFT:PosX:FFTAbsValue

13IDA:QE1:FFT:PosX:FFTAbsValue 2025-09-27 10:56:29.331851 2 0 1.15078

13IDA:QE1:FFT:PosX:FFTAbsValue 2025-09-27 10:56:29.431627 2 0 1.15078

13IDA:QE1:FFT:PosX:FFTAbsValue 2025-09-27 10:56:29.531824 2 0 1.15078

13IDA:QE1:FFT:PosX:FFTAbsValue 2025-09-27 10:56:29.631831 2 0 0.822881

13IDA:QE1:FFT:PosX:FFTAbsValue 2025-09-27 10:56:29.731868 2 0 0.822881

^C

(base) [epics@corvette ~]$ camonitor -#2 13IDA:QE1:FFT:PosY:FFTAbsValue

13IDA:QE1:FFT:PosY:FFTAbsValue 2025-09-27 10:56:34.831667 2 0 0.326255

13IDA:QE1:FFT:PosY:FFTAbsValue 2025-09-27 10:56:34.931805 2 0 0.326255

13IDA:QE1:FFT:PosY:FFTAbsValue 2025-09-27 10:56:35.031571 2 0 0.0809308

13IDA:QE1:FFT:PosY:FFTAbsValue 2025-09-27 10:56:35.131777 2 0 0.0809308

13IDA:QE1:FFT:PosY:FFTAbsValue 2025-09-27 10:56:35.231808 2 0 0.0809308

 

They are both updating at 10 Hz, as expected.

 

What do you see when you run this camonitor command on your X and X FFT positions?

 

The next thing to do is use “ps” to list the threads with names that contain FFT.  This is what I see:

 

(base) [epics@corvette ~]$ ps -eL | grep FFT

1737089 1737178 pts/14   00:00:00 QE1_FFT1_update

1737089 1737181 pts/14   00:00:00 QE1_FFT2_update

1737089 1737184 pts/14   00:00:00 QE1_FFT3_update

1737089 1737187 pts/14   00:00:00 QE1_FFT4_update

1737089 1737190 pts/14   00:00:00 QE1_FFT5_update

1737089 1737193 pts/14   00:00:00 QE1_FFT6_update

1737089 1737196 pts/14   00:00:00 QE1_FFT7_update

1737089 1737199 pts/14   00:00:00 QE1_FFT8_update

1737089 1737202 pts/14   00:00:00 QE1_FFT9_update

1737089 1737205 pts/14   00:00:00 QE1_FFT10_updat

1737089 1737208 pts/14   00:00:00 QE1_FFT11_updat

1737089 1737301 pts/14   00:00:03 QE1_FFT1_Plugin

1737089 1737303 pts/14   00:00:04 QE1_FFT2_Plugin

1737089 1737305 pts/14   00:00:04 QE1_FFT3_Plugin

1737089 1737307 pts/14   00:00:04 QE1_FFT4_Plugin

1737089 1737309 pts/14   00:00:04 QE1_FFT5_Plugin

1737089 1737311 pts/14   00:00:04 QE1_FFT6_Plugin

1737089 1737313 pts/14   00:00:04 QE1_FFT7_Plugin

1737089 1737315 pts/14   00:00:04 QE1_FFT8_Plugin

1737089 1737317 pts/14   00:00:04 QE1_FFT9_Plugin

1737089 1737319 pts/14   00:00:04 QE1_FFT10_Plugi

1737089 1737321 pts/14   00:00:04 QE1_FFT11_Plugi

 

Note that there are 11 FFT threads with _update in the name and 11 with _Plugin in the name.  What do you see?  Are all the threads there?

 

Let me know the results of these tests on your system.

 

Mark

 

 

From: Ha, Kiman <kha@bnl.gov>
Sent: Friday, September 26, 2025 5:59 PM
To: tech-talk@aps.anl.gov
Cc: Mark Rivers <rivers@cars.uchicago.edu>
Subject: quadEM FFP plugin update issues

 

Hi Mark,

 

We are using a new NSLS2EM hardware setup with the quadEM IOC.

 

One of our IOCs has an issue where the FFT plugin stops updating shortly after the IOC is started.

Although the FFT plugin has stopped updating, the circular buffer index continues to update.

 

Time-domain display continues to work without any problems.

Do you have any suggestions or recommendations to help resolve this issue?

Is it possible to restart the FFT plugin without rebooting the IOC?

 

See plot the PosX FFT is still running but the PosY is stopped.

 

 

 

Thanks,

Kiman