Experimental Physics and
Industrial Control System

Latest update:

It seems the exception is thrown when in pendEvent(), it tries to destruct the callback guard.

Jeff and other guys, do you have any thoughts?

Let me know.

Thanks a lot.

Lorna Zhang
-----Lorna Zhang/SHA/NIC wrote: -----

To: "Jeff Hill" <[email protected]>
From: Lorna Zhang/SHA/NIC
Date: 04/29/2011 04:50PM
cc: "'Andrew Johnson'" <[email protected]>, [email protected]
Subject: RE: Does EPICS Base support multi-thread on vxWorks 6.3?

Jeff,

Thank you very much for your detailed guidance, and sorry for the delayed reply, I was fixing some other issues during the week.

I checked the result that ca_pend_event returns, and found it IS  ECA_INTERNAL(142)！I also saw exception related stuff in my profiler (attached below). What does this tell us? (All this is based on preemptive callback mode being turned off.)



I have no idea about the  "old file descriptor registration features of the CA Client API" , but I saw some suspicious comments on the function ca_client_context::pendEvent(), which appears in the call stack in profiler. Are they relevant?



I also tried stop/resume thread and check the call stack, I found that it is the first EPICS thread that has eaten up the CPU. It hangs at semMGive(), and before that there's an allocate exception?? For the second EPICS thread, it gets stuck in semMTake().



Here comes the most weird thing, if I add a printf after calling ca_pend_event() to print out the return value of the function, then after deploy a second EPICS client, the CPU usage will stay low, while the return value is still ECA_INTERNAL (before deploy the second one, ca_pend_event() returns 80, which is ECA_TIMEOUT).

Any thoughts??

BTW, Alex Chen sits just behind me :-p

Lorna Zhang


"Jeff Hill" ---04/20/2011 12:53:33 AM---Ø  The delay passed to ca_pend_event is 0.5s.  Ø

From:	"Jeff Hill" <[email protected]>
To:	<[email protected]>
Cc:	"'Andrew Johnson'" <[email protected]>, <[email protected]>
Date:	04/20/2011 12:53 AM
Subject:	RE: Does EPICS Base support multi-thread on vxWorks 6.3?

---

Ø   The delay passed to ca_pend_event is 0.5s.  
Ø    
Ø   I changed to call ca_context_create with ca_enable_preemptive_callback
Ø   (without any other change... and still calling ca_pend_event afterwards),
Ø   and found out the CPU 100% usage issue was gone. Weird!  

That is definitely weird, because changing to preemptive callback shouldn’t have that impact. I am suspicious that there is a bug, but I had a look at the ca client code and I don’t (yet) see an issue which would be impacted by preemptive callback mode being turned on/off.

 

Please suspend the thread that is using CPU, and take some samples of its stack trace so we can see where the CPU is spending time. On vxWorks, with the target shell, one can suspend the thread with “ts <task id>”, print a stack trace with “tt <task id >”, and resume the thread with “tr <task id>”. If you repeat that sequence several times we should receive a good indication of where the CPU is being consumed.

 

Considering this further, if there was an unexpected C++ exception occurring in ca_pend_event it would be caught by a try catch block at the membrane between C and C++. In that situation ca_pend_event would return ECA_INTERNAL and one could imagine that some CPU would be consumed, because the epicsThreadSleep in ca_pend_event might not be executed. Does ca_pend_event return ECA_INTERNAL?

 

Are you using the old file descriptor registration features of the CA Client API? I suspect not. Knowing that you are not using that API eliminates some of the code in ca_pend_event from further scrutiny.

 

Presumably this isn’t an SMP system (because it is ppc-603)?

 

Note also that Alex Chen, also of NI, has some bug entries against EPICS at launch pad related to glitches in the advancement of time on windows, but they shouldn’t be relevant to vxWorks.

 

Jeff

______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

 

Message content: TSPA

 

With sufficient thrust, pigs fly just fine. However, this is

not necessarily a good idea. It is hard to be sure where they

are going to land, and it could be dangerous sitting under them

as they fly overhead. -- RFC 1925

 

From:  [email protected] [ mailto:[email protected] ]
Sent:  Tuesday, April 19, 2011 3:27 AM
To:  Jeff Hill
Cc:  'Andrew Johnson'; [email protected]
Subject:  RE: Does EPICS Base support multi-thread on vxWorks 6.3?

 

The delay passed to ca_pend_event is 0.5s.  

I changed to call ca_context_create with ca_enable_preemptive_callback (without any other change... and still calling ca_pend_event afterwards), and found out the CPU 100% usage issue was gone. Weird!  

Lorna Zhang
National Instruments Shanghai
Tel: 86-21-50509810-3230  

From:	"Jeff Hill" <[email protected]>
To:	<[email protected]>, "'Andrew Johnson'" <[email protected]>
Cc:	<[email protected]>
Date:	04/19/2011 06:37 AM
Subject:	RE: Does EPICS Base support multi-thread on vxWorks 6.3?

 

---

Ø   Is there any other reason that could cause similar issues?  

   

What delay is being passed to ca_pend_event?

   

Otherwise, set a breakpoint in the function that uses CPU, to see who calls it, and how often.  

   

Jeff  

______________________________________________________
Jeffrey O. Hill           Email         [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107  

   

Message content: TSPA  

   

With sufficient thrust, pigs fly just fine. However, this is  

not necessarily a good idea. It is hard to be sure where they  

are going to land, and it could be dangerous sitting under them  

as they fly overhead. -- RFC 1925  

   

From:  [email protected] [ mailto:[email protected] ] On Behalf Of [email protected]
Sent:  Monday, April 18, 2011 2:59 AM
To:  Andrew Johnson
Cc:  [email protected]
Subject:  Re: Does EPICS Base support multi-thread on vxWorks 6.3?  

 

I'm not sharing a single CA client context between threads, I did make a call to ca_context_create() with ca_disable_preemptive_callback when a new thread is initiated.  

Is there any other reason that could cause similar issues?  

Andrew, thanks for your recommendation of using ca_enable_preemptive_callback, but that might not be my first choice.  

Lorna

From:	Andrew Johnson <[email protected]>
To:	[email protected]
Cc:	[email protected]
Date:	04/15/2011 11:07 PM
Subject:	Re: Does EPICS Base support multi-thread on vxWorks 6.3?

 

---

Hi Lorna,

On Friday 15 April 2011 02:34:53 [email protected] wrote:
>
> We developed a client based on libca (EPICS Base 3.14.9), which will
> create it's own worker thread on construction. But we found it working
> strangely on vxWorks (ppc-603). Whenever a second EPICS client is created
> (meaning there are now 2 EPICS worker threads on vxWorks), the CPU usage
> became 100%.
>
> From the profile, we found the ca_pend_event is consuming lots of the CPU.
>
> Meanwhile, it seems epicsMutex is failing to do the lock for the second
> EPICS client??

EPICS should work fine on your hardware and OS combination, this is probably
an application programming issue.

Are these two threads sharing a single CA client context?  Hopefully Jeff Hill
will respond here too, but you might want to read this and the following
sections of the CA Reference Manual for some ideas:
http://www.aps.anl.gov/epics/base/R3-14/12-docs/CAref.html#Thread
In particular if you're experienced in writing multi-threaded code you might
want to use ca_enable_preemptive_callback mode and completely avoid having to
call ca_pend_event() in your code.

- Andrew
--
An error is only a mistake if you don't learn from it.
When you learn something from it, it becomes a lesson.