Re: Link error in MRF event system due to reference clock glitch

["Hong, Ran via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Michael,

Thanks for your information. 

It is likely that some old electronics we kept using caused this issue. After we switch the RF input to the new electronics we built, the event system hasn't experienced any issues so far.

Ran Hong

---

From: Michael Davidsaver <mdavidsaver at gmail.com>
Sent: Friday, March 14, 2025 10:24 AM
To: Hong, Ran <rhong at anl.gov>; Timo Korhonen <Timo.Korhonen at ess.eu>; Steven Hunt <Hunt at lbl.gov>
Cc: tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Link error in MRF event system due to reference clock glitch

 

Early in NSLS2 commissioning we saw occasional random loss of lock, which turned out to be due to a defective RF attenuator, which was labeled as -3 dB but measured as -20 dB. This put the input RF power level at the EVG below the minimum specified

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

 

ZjQcmQRYFpfptBannerEnd

Early in NSLS2 commissioning we saw occasional random loss of lock,
which turned out to be due to a defective RF attenuator, which was
labeled as -3 dB but measured as -20 dB.  This put the input RF
power level at the EVG below the minimum specified level.

After this was attenuator identified and replaced, I do not recall
any further glitches.

So as Steve and Timo say, look for abnormalities in the RF input.


On 3/14/25 08:02, Timo Korhonen via Tech-talk wrote:
> True; now that you Steve mentioned it, I remembered that the RF generator model that we were using made a phase jump  if the frequency was changed via software, through the network (or GPIB, it was at that time) interface. This was another reason to have two parallel generators, and the second one (for injector) followed the one being changed, but with a smooth correction without phase jumps.
> 
> Timo
> 
> *From: *Steven Hunt <Hunt at lbl.gov>
> *Date: *Friday, 14 March 2025 at 14:44
> *To: *Timo Korhonen <Timo.Korhonen at ess.eu>
> *Cc: *"Hong, Ran" <rhong at anl.gov>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
> *Subject: *Re: Link error in MRF event system due to reference clock glitch
> 
> Does this happen while your RF source is stable?   If the frequency is changing, for instance when changing frequency to correct average horizontal orbit, many RF generators are incapable of 'smooth' transition, and can cause such issues.
> 
> Best regards
> 
> Steve
> 
> On Fri, Mar 14, 2025 at 2:58 AM Timo Korhonen via Tech-talk <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>> wrote:
> 
>     Dear Ran Hong,
> 
>     Can you share a few more details: I assume you are clocking the EVM/EVG from the machine RF?
> 
>     Do you have the same RF source for the whole complex, or maybe multiple sources, for example different ones for the storage ring and the injection chain?
> 
>     Do you know where the phase glitches come from? Are they intentional? Is it a (short) glitch, or a phase change?
> 
>     To me, phase glitches sound more like a problem somewhere in the system, which would be the root cause to be fixed.
> 
>     I would expect a RF phase glitch in the storage ring to result in a beam dump. At the SLS (1.0) we intentionally used a phase glitch to dump the beam.
> 
>     Could it be that your machine protection does something like this? Just a thought.
> 
>     It is a while ago but a far as I remember, we had two RF sources that were phase locked to each other. The other one (for the injector) was kept stable.
> 
>     I cannot at least initially think of any other method than stabilizing the clock. Maybe other people have ideas.
> 
>     Best regards,
> 
>     Timo
> 
>     *From: *Tech-talk <tech-talk-bounces at aps.anl.gov <mailto:tech-talk-bounces at aps.anl.gov>> on behalf of "Hong, Ran via Tech-talk" <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>>
>     *Reply to: *"Hong, Ran" <rhong at anl.gov <mailto:rhong at anl.gov>>
>     *Date: *Friday, 14 March 2025 at 06:54
>     *To: *"tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>" <tech-talk at aps.anl.gov <mailto:tech-talk at aps.anl.gov>>
>     *Subject: *Link error in MRF event system due to reference clock glitch
> 
>     Hello Everyone,
> 
>     After commissioned the MRF event system at Advanced Photon Source at ANL for a year, we recently experienced some instabilities.
> 
>     Occasionally, clients lose important events like the machine clock (P0) event, fast beam abort event, injection events, etc. The “Link Error” count of all event receivers incremented by 1. Further investigation showed that the event link in the entire system was down for a few milliseconds, and then recover by itself.
> 
>     In my previous studies, I found that such errors can be caused by sudden changes of the reference clock phase. When investigating these recent errors, I triggered a scope with the fast beam abort signal from the machine protection system and found a glitch happened about 30 microseconds before the beam abort signal. Normally the clock period jitters about 40ps, while the glitch the clock period changed by +/-300 ps. I think it is very likely that such changes caused the event link error.
> 
>     Questions to other facilities that are using MRF event systems: have you seen similar glitches in your reference clock? If so, is there a way to make the MRF event master more resilient to such glitches?
> 
>     Thank you!
> 
>     Ran Hong
> 
>     Argonne National Lab, Advanced Photon Source
>