RE: Gateway fails with "epicsMutex pthread_mutex_lock failed: error Invalid argument"

[Mark Bennett <Mark.Bennet@synchrotron.org.au>]

The following patch to base 3.14.10 was received from Jeff Hill:

diff -u -b -w -b -r1.92.2.16 casStrmClient.cc
--- casStrmClient.cc 21 Oct 2008 20:50:26 -0000 1.92.2.16
+++ casStrmClient.cc 8 Jan 2009 22:13:09 -0000
@@ -1413,9 +1413,7 @@
this->chanTable.remove ( *pChan->pChanI );
this->chanList.remove ( *pChan->pChanI );
pChan->pChanI->uninstallFromPV ( this->eventSys );
- casPVI * pPVI = pChan->getPV()->pPVI;
delete pChan->pChanI;
- pPVI->deleteSignal ();
}

This has solved the problem. Refer Mantis #329.

Thanks for your prompt support Jeff.

Mark
________________________________________
From: tech-talk-bounces@aps.anl.gov [tech-talk-bounces@aps.anl.gov] On Behalf Of Mark Bennett [Mark.Bennet@synchrotron.org.au]
Sent: Thursday, 8 January 2009 12:24 PM
To: tech-talk@aps.anl.gov
Subject: Gateway fails with "epicsMutex pthread_mutex_lock failed: error        Invalid argument"

One of our EPICS gateways appears to be failing when an IOC is restarted multiple times within half an hour or so. A large number of PVs (4,000) are accessed via this EPICS gateway - by a channel archiver specifically. Prior to failure, the client event and post rates are equal at ~1550Hz while the server event rate is about half the post rate (server post rate ~1550Hz).

A large number of gateways (12) are hosted on a single box in separate directories. The box is running 64-bit redhat linux but we are currently building linux-x86 binaries. The number of available threads has been increased by decreasing the stack size using "ulimit -s 2048". The gateway is started using options -no_cache and -archive. We followed Dirk Zimoch's guidelines from a previous post (thanks).

We are using EPICS base 3.14.9 and gateway 2.0.3.0.

The IOC is on network 10.135 and the archiver is on network 10.100. The IOCs IP (restarted multiple times from above) is 10.135.2.11. There are other IOCs on this subnet whose PVs are accessed via the same gateway:

    IOCs on 10.135 ---> EPICS gateway (10.135 --> 10.100) ---> Channel Archiver 10.100.0.244

Here is an example gateway error log - the gateway process does not die but becomes non-functional:

Jan 07 16:00:21 PV Gateway Version 2.0.3.0 [Dec 17 2008 18:16:17]
EPICS 3.14.9 PID=16488 ServerPID=16487
EPICS_CA_ADDR_LIST=10.135.255.255
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_SERVER_PORT=Not specified
EPICS_CA_MAX_ARRAY_BYTES=5000000
EPICS_CAS_INTF_ADDR_LIST=10.100.10.247
EPICS_CAS_SERVER_PORT=Not specified
EPICS_CAS_IGNORE_ADDR_LIST=Not specified
Running as user ics on host CR01GW04
Statistics PV prefix is SR12ID01GW04
Jan 07 16:02:21 Warning: Virtual circuit disconnect 10.135.2.11:5064
Jan 07 16:08:28 Warning: Virtual circuit disconnect 10.135.2.11:5064
Jan 07 16:29:23 Warning: Virtual circuit disconnect 10.135.2.11:5064
epicsMutex pthread_mutex_lock failed: error Invalid argument

Jan 07 16:29:33 !!! Errlog message received (message is above)
epicsMutexOsdLock
Jan 07 16:29:33 !!! Errlog message received (message is above)
Thread _main_ (0xf2f31c0) can't proceed, suspending.

Jan 07 16:29:33 !!! Errlog message received (message is above)


Can anyone help with this problem please?

Regards,

Dr Mark Bennett
Principal Controls Engineer
Australian Synchrotron
This message and any attachments may contain proprietary or confidential information. If you are not the intended recipient or you received the message in error, you must not use, copy or distribute the message. Please notify the sender immediately and destroy the original message. Thank you.
This message and any attachments may contain proprietary or confidential information. If you are not the intended recipient or you received the message in error, you must not use, copy or distribute the message. Please notify the sender immediately and destroy the original message. Thank you.