RE: Link error in MRF event system due to reference clock glitch

[Abdalla Ahmad via Tech-talk <tech-talk at aps.anl.gov>]

Hi Hong

 

I wonder why are you using external electronics to divide the RF and not use the internal pre-scaler in MRF’s EVG? We are connecting the RF from the master oscillator directly to the EVG but dividing it by 4 through the internal clock source pre-scaler of the EVG.

 

Best Regards,

Abdalla Al-Dalleh

Control Engineer

SESAME

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> On Behalf Of Hong, Ran via Tech-talk
Sent: Tuesday, April 15, 2025 12:47 AM
To: Timo Korhonen <Timo.Korhonen at ess.eu>; tech-talk at aps.anl.gov
Subject: Re: Link error in MRF event system due to reference clock glitch

 

Hi Timo,

 

Thanks for your reply. We fixed the problem by changing our Rf clock source, and then I worked on another project. Sorry for the delay.

 

What we use to clock the EVM is the machine RF divided by 3. We haven't pin-point where the glitch is from. We didn't see any glitches in the machine RF. It is likely the glitch is from the modules that divides the RF by 3 and the fan-out modules between the machine RF and the MRF EVM. 

 

Now we use some new electronics we designed to divide the machine RF, and since then I haven't seen any glitches yet.

 

The machine protection system can handle the beam dump, but without the event system, our data acquisition system will not collect the postmortem data.

 

Best regards,

 

Ran Hong

---

From: Timo Korhonen <Timo.Korhonen at ess.eu>
Sent: Friday, March 14, 2025 4:57 AM
To: Hong, Ran <rhong at anl.gov>; tech-talk at aps.anl.gov <tech-talk at aps.anl.gov>
Subject: Re: Link error in MRF event system due to reference clock glitch

 

Dear Ran Hong, Can you share a few more details: I assume you are clocking the EVM/EVG from the machine RF? Do you have the same RF source for the whole complex, or maybe multiple sources, for example different ones for the storage ring and

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

 

ZjQcmQRYFpfptBannerEnd

Dear Ran Hong,

 

Can you share a few more details: I assume you are clocking the EVM/EVG from the machine RF?

Do you have the same RF source for the whole complex, or maybe multiple sources, for example different ones for the storage ring and the injection chain?

Do you know where the phase glitches come from? Are they intentional? Is it a (short) glitch, or a phase change?

 

To me, phase glitches sound more like a problem somewhere in the system, which would be the root cause to be fixed.

I would expect a RF phase glitch in the storage ring to result in a beam dump. At the SLS (1.0) we intentionally used a phase glitch to dump the beam.

 

Could it be that your machine protection does something like this? Just a thought.

 

It is a while ago but a far as I remember, we had two RF sources that were phase locked to each other. The other one (for the injector) was kept stable.

 

I cannot at least initially think of any other method than stabilizing the clock. Maybe other people have ideas.

 

Best regards,

Timo

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of "Hong, Ran via Tech-talk" <tech-talk at aps.anl.gov>
Reply to: "Hong, Ran" <rhong at anl.gov>
Date: Friday, 14 March 2025 at 06:54
To: "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: Link error in MRF event system due to reference clock glitch

 

Hello Everyone,

 

After commissioned the MRF event system at Advanced Photon Source at ANL for a year, we recently experienced some instabilities.

 

Occasionally, clients lose important events like the machine clock (P0) event, fast beam abort event, injection events, etc. The “Link Error” count of all event receivers incremented by 1. Further investigation showed that the event link in the entire system was down for a few milliseconds, and then recover by itself.

 

In my previous studies, I found that such errors can be caused by sudden changes of the reference clock phase. When investigating these recent errors, I triggered a scope with the fast beam abort signal from the machine protection system and found a glitch happened about 30 microseconds before the beam abort signal. Normally the clock period jitters about 40ps, while the glitch the clock period changed by +/-300 ps. I think it is very likely that such changes caused the event link error.

 

Questions to other facilities that are using MRF event systems: have you seen similar glitches in your reference clock? If so, is there a way to make the MRF event master more resilient to such glitches?

 

Thank you!

 

Ran Hong

Argonne National Lab, Advanced Photon Source