Link error in MRF event system due to reference clock glitch

["Hong, Ran via Tech-talk" <tech-talk at aps.anl.gov>]

Hello Everyone,

 

After commissioned the MRF event system at Advanced Photon Source at ANL for a year, we recently experienced some instabilities.

 

Occasionally, clients lose important events like the machine clock (P0) event, fast beam abort event, injection events, etc. The “Link Error” count of all event receivers incremented by 1. Further investigation showed that the event link in the entire system was down for a few milliseconds, and then recover by itself.

 

In my previous studies, I found that such errors can be caused by sudden changes of the reference clock phase. When investigating these recent errors, I triggered a scope with the fast beam abort signal from the machine protection system and found a glitch happened about 30 microseconds before the beam abort signal. Normally the clock period jitters about 40ps, while the glitch the clock period changed by +/-300 ps. I think it is very likely that such changes caused the event link error.

 

Questions to other facilities that are using MRF event systems: have you seen similar glitches in your reference clock? If so, is there a way to make the MRF event master more resilient to such glitches?

 

Thank you!

 

Ran Hong

Argonne National Lab, Advanced Photon Source