Re: Link error in MRF event system due to reference clock glitch

["Hong, Ran via Tech-talk" <tech-talk at aps.anl.gov>]

That stilled happened when the RF frequency was stable.

Ran Hong

---

From: Timo Korhonen <Timo.Korhonen at ess.eu>
Sent: Friday, March 14, 2025 10:02 AM
To: Steven Hunt <Hunt at lbl.gov>
Cc: Hong, Ran <rhong at anl.gov>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Link error in MRF event system due to reference clock glitch

 

True; now that you Steve mentioned it, I remembered that the RF generator model that we were using made a phase jump if the frequency was changed via software, through the network (or GPIB, it was at that time) interface. This was another reason

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

 

ZjQcmQRYFpfptBannerEnd

True; now that you Steve mentioned it, I remembered that the RF generator model that we were using made a phase jump  if the frequency was changed via software, through the network (or GPIB, it was at that time) interface. This was another reason to have two parallel generators, and the second one (for injector) followed the one being changed, but with a smooth correction without phase jumps.

 

Timo

 

From: Steven Hunt <Hunt at lbl.gov>
Date: Friday, 14 March 2025 at 14:44
To: Timo Korhonen <Timo.Korhonen at ess.eu>
Cc: "Hong, Ran" <rhong at anl.gov>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: Re: Link error in MRF event system due to reference clock glitch

 

Does this happen while your RF source is stable?   If the frequency is changing, for instance when changing frequency to correct average horizontal orbit, many RF generators are incapable of 'smooth' transition, and can cause such issues.

 

Best regards

Steve

 

On Fri, Mar 14, 2025 at 2:58 AM Timo Korhonen via Tech-talk <tech-talk at aps.anl.gov> wrote:

Dear Ran Hong,

 

Can you share a few more details: I assume you are clocking the EVM/EVG from the machine RF?

Do you have the same RF source for the whole complex, or maybe multiple sources, for example different ones for the storage ring and the injection chain?

Do you know where the phase glitches come from? Are they intentional? Is it a (short) glitch, or a phase change?

 

To me, phase glitches sound more like a problem somewhere in the system, which would be the root cause to be fixed.

I would expect a RF phase glitch in the storage ring to result in a beam dump. At the SLS (1.0) we intentionally used a phase glitch to dump the beam.

 

Could it be that your machine protection does something like this? Just a thought.

 

It is a while ago but a far as I remember, we had two RF sources that were phase locked to each other. The other one (for the injector) was kept stable.

 

I cannot at least initially think of any other method than stabilizing the clock. Maybe other people have ideas.

 

Best regards,

Timo

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of "Hong, Ran via Tech-talk" <tech-talk at aps.anl.gov>
Reply to: "Hong, Ran" <rhong at anl.gov>
Date: Friday, 14 March 2025 at 06:54
To: "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: Link error in MRF event system due to reference clock glitch

 

Hello Everyone,

 

After commissioned the MRF event system at Advanced Photon Source at ANL for a year, we recently experienced some instabilities.

 

Occasionally, clients lose important events like the machine clock (P0) event, fast beam abort event, injection events, etc. The “Link Error” count of all event receivers incremented by 1. Further investigation showed that the event link in the entire system was down for a few milliseconds, and then recover by itself.

 

In my previous studies, I found that such errors can be caused by sudden changes of the reference clock phase. When investigating these recent errors, I triggered a scope with the fast beam abort signal from the machine protection system and found a glitch happened about 30 microseconds before the beam abort signal. Normally the clock period jitters about 40ps, while the glitch the clock period changed by +/-300 ps. I think it is very likely that such changes caused the event link error.

 

Questions to other facilities that are using MRF event systems: have you seen similar glitches in your reference clock? If so, is there a way to make the MRF event master more resilient to such glitches?

 

Thank you!

 

Ran Hong

Argonne National Lab, Advanced Photon Source