Experimental Physics and
Industrial Control System

Michael Davidsaver <[email protected]> · Wed, 25 Jul 2018 21:55:17 -0700

On 07/25/2018 06:52 PM, Matt Rippa via Tech-talk wrote:
> 
> We commissioned a new RTEMS IOC here in Hawaii. The system ran
> for 2 days, 18 hours and 47 minutes before it simply halted. No exception was generated therefore no stack trace was seen. Also no console, no iocsh, no log messages. A system reset recovered our system before halting in the same manner 2 hours later. This occurred 4 times before the night ended.
> 
> We speculate bad access or stack corruption but without a stack trace
> we're at a complete loss as to how to diagnose this. This *same software release* has run for several weeks at our other site with no issues.
> 
> Is there a way to force an exception (or stack trace), for example with watchdog?
> 
> Many thanks for your insight!

The only time I can recall truly seeing nothing printed during a crash was
when a hardware fault occurred.  This was accompanied by a red fault LED
on the underside of the cards.

In every other case I can think of, including mistakes I made during driver
development, something was printed.

NSLS2 had a batch of mvme3100 cards suffering from silver creep corrosion,
which is related to the particular lead-free solder process used.  In the
case of the mvme3100, we were told this only involved a small number of boards
assembled in 2011.  We had boards assembled both earlier, and later,
which were not effected.

https://www.google.com/search?q=silver+creep+corrosion

This condition is very obvious under small magnification.

https://www.google.com/search?q=silver+creep+corrosion&source=lnms&tbm=isch&sa=X

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System