Re: Linux scan thread suspended

[Pilar Gil Jaldo <pilar.gil@sevensols.com>]

Thank you Mark for the advices!

At the moment is not a reproducible error. I have seen it twice and with changes the PV values for the IOC is not possible to reproduce the behavior.

I will try to debug deeper using gdb when it happens next time :)

Pilar

On 05/09/18 16:32, Mark Rivers wrote:

Is the error reproducible? Hopefully yes, so the next time it happens you should use gdb to get a stack trace on the task that generates the malloc error.  You can also examine the input string to epicsStrDup.

The error may be a side effect of the network issues but the direct cause is the malloc failure in epicsStrDup.

Mark


Sent from my iPhone

On Sep 5, 2018, at 9:22 AM, Pilar Gil Jaldo <pilar.gil@sevensols.com<mailto:pilar.gil@sevensols.com>> wrote:


Hello Mark,

Thank you for your reply. I'm going to give a little bit more information about the system.

It is a 64 bit machine running EPICS 3.13.7  with..

             total       used       free     shared    buffers     cached
Mem:          7812        700       7112          0         49        410

I was speculating with a network problem because I have the same system with the same configuration running outside the network and I have never seen this error before. This error is just happing since the other devices are requesting messages to the CA Network without getting an answer from the offline devices.

The possibility of a non zero terminated string would be also strange because in the other running system everything is working fine.

Thank you very much in advance!

Pilar

On 05/09/18 16:02, Mark Rivers wrote:

Hi Pilar,

I think your case is different from the tech-talk thread.  In the tech-talk thread they said there was no IOC error mesage, but in  your case you have a very clear IOC error message:



epicsStrDup: mallocMustSucceed(1) - malloc failed Thread scan2 (0x9c2b038) suspending.


This says to me that your IOC has run out of memory.  I suspect there is a memory leak somewhere.

Is this a 32 or 64 bit machine, and how much memory does it have?

You could restart the IOC and use "top" to look at the memory usage with time.

What version of EPICS are you using, and what devices does this IOC control?

Mark


-----Original Message-----
From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> <tech-talk-bounces@aps.anl.gov><mailto:tech-talk-bounces@aps.anl.gov> On Behalf Of Pilar Gil Jaldo
Sent: Wednesday, September 5, 2018 8:36 AM
To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Linux scan thread suspended

Hello everybody,

I have a question regarding the reasons why a scan thread can be suspended. I have a running IOC and at a certain moment the IOC breaks with following message:

epicsStrDup: mallocMustSucceed(1) - malloc failed Thread scan2 (0x9c2b038) suspending.

I have found in the tech-talk messages following reason which seems to be the same problem that mine.

https://epics.anl.gov/tech-talk/2013/msg00154.php

The IOC is on a network where many devices are sending packages to the CA network, but nobody is answering because other devices are switched off (the traffic in the CA is very high).

Is it possible that the segmentation fault is caused by a network congestion? Can the scan thread be suspended by a higher network traffic?

Thank you very much in advance for your answers.
Pilar Gil Jaldo


--

[http://sevensols.com]<http://sevensols.com>
Pilar Gil Jaldo

Embedded SW Engineer

This e-mail and any materials attached hereto, including, without limitation, all content hereof and thereof are confidential and proprietary to Seven Solutions,S.L and/or its affiliates, and are *confidential*. Without the prior written consent of Seven Solutions, this content may not (i) be disclosed to any third party, (ii) be reproduced, modified or otherwise used by anyone other than current employees of Seven Solutions or its affiliates. If you are not the intended recipient or if this message has been received in error, please immediately notify us via e-mail and delete it.

--

Pilar Gil Jaldo

Embedded SW Engineer

This e-mail and any materials attached hereto, including, without limitation, all content hereof and thereof are confidential and proprietary to Seven Solutions,S.L and/or its affiliates, and are *confidential*. Without the prior written consent of Seven Solutions, this content may not (i) be disclosed to any third party, (ii) be reproduced, modified or otherwise used by anyone other than current employees of Seven Solutions or its affiliates. If you are not the intended recipient or if this message has been received in error, please immediately notify us via e-mail and delete it.