RE: Windows EPICS problem - PROBLEM SOLVED

["Mark Rivers" <rivers@cars.uchicago.edu>]

I figured out the source of the problem, but I don't understand it.
 
I had installed a packet filter from Prosilica for their GigE Vision cameras.  That filter is intercepts GigE Vision packets before they reach the Windows TCP/IP stack for efficiency, but passes everything else on.  When I just uninstalled that packet filter channel access then behaved normallly.  When I installed it again channel access went back to the behavior where I had to add localhost to EPICS_CA_ADDR_LIST.
 
What is strange is that on another Windows machine I also have the packet filter installed, and channel access works fine. That is the computer 164.54.160.38 for which I had also included the route print on my last message.  The only difference I can see is which Ethernet chip set is being used.
 
Anyway, I can work around the problem by adding local host to EPICS_CA_ADDR_LIST.  The benefits of the packet filter outweigh the annoyance.
 
Mark
 

________________________________

From: Mark Rivers
Sent: Fri 4/11/2008 2:11 PM
To: Jeff Hill
Cc: tech-talk@aps.anl.gov
Subject: RE: Windows EPICS problem


Jeff,
 
I now have some time to get back to this problem.  I've included a bunch of diagnostics, but I cannot see the problem.
 
I have established that defining EPICS_CA_ADDR_LIST=127.0.0.1 (and not defining EPICS_CA_AUTO_ADDR_LIST) in the shell running the client is sufficient to solve the problem.

If 127.0.0.1 is not in EPICS_CA_ADDR_LIST I cannot connected local PVs.  It does not help to set that in the command shell running the IOC, it only works if it is defined for the client.
 
Here is the output of ipconfig/all.  It looks OK to me.
#########################################################################################################

C:\Documents and Settings\epics>ipconfig /all
Windows IP Configuration
        Host Name . . . . . . . . . . . . : ferrari
        Primary Dns Suffix  . . . . . . . : CARS.APS.ANL.GOV
        Node Type . . . . . . . . . . . . : Hybrid
        IP Routing Enabled. . . . . . . . : No
        WINS Proxy Enabled. . . . . . . . : No
        DNS Suffix Search List. . . . . . : CARS.APS.ANL.GOV
                                            APS.ANL.GOV
                                            ANL.GOV
Ethernet adapter Local Area Connection:
        Connection-specific DNS Suffix  . :
        Description . . . . . . . . . . . : Broadcom NetXtreme 57xx Gigabit Controller
        Physical Address. . . . . . . . . : 00-18-8B-13-7D-BA
        Dhcp Enabled. . . . . . . . . . . : No
        IP Address. . . . . . . . . . . . : 164.54.160.93
        Subnet Mask . . . . . . . . . . . : 255.255.255.0
        Default Gateway . . . . . . . . . : 164.54.160.1
        DNS Servers . . . . . . . . . . . : 164.54.160.12
                                            164.54.160.45
#########################################################################################################
 
Here is the output of route print on the machine with the problem:
 
#########################################################################################################
C:\Documents and Settings\epics>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 18 8b 13 7d ba ...... Broadcom NetXtreme 57xx Gigabit Controller - Packet Scheduler Miniport
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     164.54.160.1   164.54.160.93       10
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1       1
     164.54.160.0    255.255.255.0    164.54.160.93   164.54.160.93       10
    164.54.160.93  255.255.255.255        127.0.0.1       127.0.0.1       10
   164.54.255.255  255.255.255.255    164.54.160.93   164.54.160.93       10
        224.0.0.0        240.0.0.0    164.54.160.93   164.54.160.93       10
  255.255.255.255  255.255.255.255    164.54.160.93   164.54.160.93       1
Default Gateway:      164.54.160.1
===========================================================================
Persistent Routes:
  None
#########################################################################################################
Here is the output of route print on another Windows machine that does not have any trouble connecting to local PVs. As far as I can see they look identical except for the host local host IP number.
 
#########################################################################################################

C:\Documents and Settings\rivers.CARS_NT_DOMAIN>route print
========================================================================
===
Interface List
0x1 ........................... MS TCP Loopback interface
0x20002 ...00 07 e9 6a 79 ba ...... Intel(R) PRO/1000 MT Network
Connection - Packet Scheduler Miniport
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     164.54.160.1   164.54.160.38      10
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1      1
     164.54.160.0    255.255.255.0    164.54.160.38   164.54.160.38      10
    164.54.160.38  255.255.255.255        127.0.0.1       127.0.0.1      10
   164.54.255.255  255.255.255.255    164.54.160.38   164.54.160.38      10
        224.0.0.0        240.0.0.0    164.54.160.38   164.54.160.38      10
  255.255.255.255  255.255.255.255    164.54.160.38   164.54.160.38      1
Default Gateway:      164.54.160.1
===========================================================================
Persistent Routes:
  None


#########################################################################################################

Here is the output of TCPview when EPICS_CA_ADDR_LIST is not defined and there is thus no connection to the IOC application called simDetectorApp.

#########################################################################################################

simDetectorApp.exe:856 TCP ferrari:5064 ferrari:0 LISTENING 
simDetectorApp.exe:856 UDP ferrari:1497 *:*  
simDetectorApp.exe:856 UDP ferrari:5064 *:*  
simDetectorApp.exe:856 UDP ferrari:1498 *:*  
simDetectorApp.exe:856 UDP ferrari:1499 *:*  

#########################################################################################################

Here is the output of TCPview when EPICS_CA_ADDR_LIST is defined and there is a client connection to the IOC application. The last line shows the client connection.

#########################################################################################################

simDetectorApp.exe:856 TCP ferrari:5064 ferrari:0 LISTENING 
simDetectorApp.exe:856 UDP ferrari:1497 *:*  
simDetectorApp.exe:856 UDP ferrari:5064 *:*  
simDetectorApp.exe:856 UDP ferrari:1498 *:*  
simDetectorApp.exe:856 UDP ferrari:1499 *:*  
simDetectorApp.exe:856 TCP ferrari:5064 localhost:1688 ESTABLISHED 
#########################################################################################################
Here is the output of cainfo when EPICS_CA_ADDR_LIST is not defined:
#########################################################################################################
C:\Documents and Settings\epics>set EPICS_CA_ADDR_LIST=
C:\Documents and Settings\epics>set EPICS_CA_AUTO_ADDR_LIST=
C:\Documents and Settings\epics>cainfo -s10 13SIM1:cam1:FileName_RBV
Channel connect timed out: '13SIM1:cam1:FileName_RBV' not found.
ca_client_context at 003A3D18 pndRecvCnt=0 ioSeqNo=1
Channel Access Client Context at 003A3E98 for user epics
        revision "@(#) EPICS 3.14.8.2, CA Client Library Aug  1 2006"
Hash table with 0 buckets and 0 items of type class tcpiiu installed
        connection time out watchdog period 30.000000
Datagram IO circuit (and disconnected channel repository)
        repeater port 5065
        default server port 5064
Channel Access Address List
164.54.160.255:5064
        socket identifier 1756
        bytes in xmit buffer 16
        shut down command bool 0
        recv thread exit signal:
repeater subscribe timer: attempts=0 registered=0 once=0
disconnect governor timer:
search timer delay 0.032000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.064000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.128000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.256000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.512000
0 channels with search request pending
1 channels with search response pending
Channel "13SIM1:cam1:FileName_RBV" is disconnected
        network IO pointer = 00980048
        server identifier 4294967295
        search retry number=5
        name length=25
search timer delay 1.024000
0 channels with search request pending
0 channels with search response pending
search timer delay 2.048000
0 channels with search request pending
0 channels with search response pending
search timer delay 4.096000
0 channels with search request pending
0 channels with search response pending
search timer delay 8.192000
0 channels with search request pending
0 channels with search response pending
search timer delay 16.384000
0 channels with search request pending
0 channels with search response pending
search timer delay 32.768000
0 channels with search request pending
0 channels with search response pending
search timer delay 65.536000
0 channels with search request pending
0 channels with search response pending
search timer delay 131.072000
0 channels with search request pending
0 channels with search response pending
search timer delay 262.144000
0 channels with search request pending
0 channels with search response pending
Program begin time:
epicsTime: Fri Apr 11 2008 13:24:09.008243293
epicsTime: revision "@(#) EPICS 3.14.8.2, Common Utilities Library Aug  1 2006"
Channel identifier hash table:
Hash table with 512 buckets and 1 items of type class nciu installed
Channel "13SIM1:cam1:FileName_RBV" is disconnected
        network IO pointer = 00980048
        server identifier 4294967295
        search retry number=5
        name length=25
Channel "13SIM1:cam1:FileName_RBV" is disconnected
        network IO pointer = 00980048
        server identifier 4294967295
        search retry number=5
        name length=25
entries per bucket: mean = 0.001953 std dev = 0.044151 max = 1
IO identifier hash table:
Hash table with 0 buckets and 0 items of type class baseNMIU installed
Beacon source identifier hash table:
Hash table with 0 buckets and 0 items of type class bhe installed
Timer queue:
EPICS threaded timer queue at 003A4F58
epicsTimerQueue with 16 items pending
class searchTimer, state = pending, delay = 0.004087
search timer delay 0.512000
0 channels with search request pending
1 channels with search response pending
Channel "13SIM1:cam1:FileName_RBV" is disconnected
class searchTimer, state = pending, delay = 0.007467
search timer delay 0.032000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.011513
search timer delay 1.024000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.016706
search timer delay 0.064000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.083627
search timer delay 0.128000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.238962
search timer delay 0.256000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 1.034945
search timer delay 2.048000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 3.082937
search timer delay 4.096000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 7.178929
search timer delay 8.192000
0 channels with search request pending
0 channels with search response pending
class disconnectGovernorTimer, state = pending, delay = 8.986641
disconnect governor timer:
class repeaterSubscribeTimer, state = pending, delay = 8.986631
repeater subscribe timer: attempts=0 registered=0 once=0
class searchTimer, state = pending, delay = 15.370618
search timer delay 16.384000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 31.754610
search timer delay 32.768000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 64.522361
search timer delay 65.536000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 130.058351
search timer delay 131.072000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 261.130343
search timer delay 262.144000
0 channels with search request pending
0 channels with search response pending
reschedule event
exit event
exitFlag = F, terminateFlag = F
IP address to name conversion engine:
ipAddrToAsciiEngine at 003A4928 with 0 requests pending
mutex:
epicsMutexId 003A4DA8 source ..\..\..\src\libCom\osi\epicsMutex.cpp line 197
epicsMutex: win32 critical section at 003A4D88
laborEvent:
exitFlag  boolean = 0
exit event:
Default mutex:
epicsMutexId 003A3DF0 source ..\..\..\src\libCom\osi\epicsMutex.cpp line 197
epicsMutex: win32 critical section at 003A3DD0
mutex:
epicsMutexId 003A3DF0 source ..\..\..\src\libCom\osi\epicsMutex.cpp line 197
epicsMutex: win32 critical section at 003A3DD0
        preemptive callback is disabled
        there are 0 unsatisfied IO operations blocking ca_pend_io()
        the current io sequence number is 1
IO done event:
Synchronous group identifier hash table:
Hash table with 0 buckets and 0 items of type struct CASG installed
#########################################################################################################
 
Now set the environment EPICS_CA_ADDR_LIST so it will find the PV and try again:
 
#########################################################################################################
C:\Documents and Settings\epics>set EPICS_CA_ADDR_LIST=127.0.0.1
 
C:\Documents and Settings\epics>cainfo -s10 13SIM1:cam1:FileName_RBV
ca_client_context at 003A3D48 pndRecvCnt=0 ioSeqNo=1
Channel Access Client Context at 003A3EC8 for user epics
        revision "@(#) EPICS 3.14.8.2, CA Client Library Aug  1 2006"
Hash table with 512 buckets and 1 items of type class tcpiiu installed
Virtual circuit to "localhost:5064" at version V4.11 state 1
        current data cache pointer = 009AAF98 current data cache size = 16384
        contiguous receive message count=0, busy detect bool=0, flow control bool=0
eceive thread is busy=0
        virtual circuit socket identifier 1672
        send thread flush signal:
        send thread:
        recv thread:
        echo pending bool = 0
IO identifier hash table:
Connected channels
Channel "13SIM1:cam1:FileName_RBV", connected to server localhost:5064, native type DBF_CHAR, native element count 256,
read access, write access
        network IO pointer = 009A63D8
        server identifier 10
        search retry number=1
        name length=25
Virtual circuit to "localhost:5064" at version V4.11 state 1
        current data cache pointer = 009AAF98 current data cache size = 16384
        contiguous receive message count=0, busy detect bool=0, flow control bool=0
eceive thread is busy=0
        virtual circuit socket identifier 1672
        send thread flush signal:
        send thread:
        recv thread:
        echo pending bool = 0
IO identifier hash table:
Connected channels
Channel "13SIM1:cam1:FileName_RBV", connected to server localhost:5064, native type DBF_CHAR, native element count 256,
read access, write access
        network IO pointer = 009A63D8
        server identifier 10
        search retry number=1
        name length=25
entries per bucket: mean = 0.001953 std dev = 0.044151 max = 1
        connection time out watchdog period 30.000000
Datagram IO circuit (and disconnected channel repository)
        repeater port 5065
        default server port 5064
Channel Access Address List
164.54.160.255:5064
localhost:5064
        socket identifier 1756
        bytes in xmit buffer 16
        shut down command bool 0
        recv thread exit signal:
repeater subscribe timer: attempts=0 registered=0 once=0
disconnect governor timer:
search timer delay 0.032000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.064000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.128000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.256000
0 channels with search request pending
0 channels with search response pending
search timer delay 0.512000
0 channels with search request pending
0 channels with search response pending
search timer delay 1.024000
0 channels with search request pending
0 channels with search response pending
search timer delay 2.048000
0 channels with search request pending
0 channels with search response pending
search timer delay 4.096000
0 channels with search request pending
0 channels with search response pending
search timer delay 8.192000
0 channels with search request pending
0 channels with search response pending
search timer delay 16.384000
0 channels with search request pending
0 channels with search response pending
search timer delay 32.768000
0 channels with search request pending
0 channels with search response pending
search timer delay 65.536000
0 channels with search request pending
0 channels with search response pending
search timer delay 131.072000
0 channels with search request pending
0 channels with search response pending
search timer delay 262.144000
0 channels with search request pending
0 channels with search response pending
Program begin time:
epicsTime: Fri Apr 11 2008 13:22:04.670037758
epicsTime: revision "@(#) EPICS 3.14.8.2, Common Utilities Library Aug  1 2006"
Channel identifier hash table:
Hash table with 512 buckets and 1 items of type class nciu installed
Channel "13SIM1:cam1:FileName_RBV", connected to server localhost:5064, native type DBF_CHAR, native element count 256,
read access, write access
        network IO pointer = 009A63D8
        server identifier 10
        search retry number=1
        name length=25
Channel "13SIM1:cam1:FileName_RBV", connected to server localhost:5064, native type DBF_CHAR, native element count 256,
read access, write access
        network IO pointer = 009A63D8
        server identifier 10
        search retry number=1
        name length=25
entries per bucket: mean = 0.001953 std dev = 0.044151 max = 1
IO identifier hash table:
Hash table with 0 buckets and 0 items of type class baseNMIU installed
Beacon source identifier hash table:
Hash table with 512 buckets and 1 items of type class bhe installed
CA beacon hash entry for 127.0.0.1:5064 <no period estimate>
        beacon number 0, on <undefined>
CA beacon hash entry for 127.0.0.1:5064 <no period estimate>
        beacon number 0, on <undefined>
entries per bucket: mean = 0.001953 std dev = 0.044151 max = 1
Timer queue:
EPICS threaded timer queue at 003A4F88
epicsTimerQueue with 17 items pending
class searchTimer, state = pending, delay = 0.018199
search timer delay 0.032000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.024576
search timer delay 0.064000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.088306
search timer delay 0.128000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.216296
search timer delay 0.256000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.472289
search timer delay 0.512000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 0.984283
search timer delay 1.024000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 2.008014
search timer delay 2.048000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 4.056006
search timer delay 4.096000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 8.152018
search timer delay 8.192000
0 channels with search request pending
0 channels with search response pending
class disconnectGovernorTimer, state = pending, delay = 9.959748
disconnect governor timer:
class repeaterSubscribeTimer, state = pending, delay = 9.959738
repeater subscribe timer: attempts=0 registered=0 once=0
class searchTimer, state = pending, delay = 16.343724
search timer delay 16.384000
0 channels with search request pending
0 channels with search response pending
class tcpRecvWatchdog, state = pending, delay = 29.987311
Receive virtual circuit watchdog at 009A64E8, period 30.000000
class searchTimer, state = pending, delay = 32.727444
search timer delay 32.768000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 65.495436
search timer delay 65.536000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 131.031428
search timer delay 131.072000
0 channels with search request pending
0 channels with search response pending
class searchTimer, state = pending, delay = 262.103150
search timer delay 262.144000
0 channels with search request pending
0 channels with search response pending
reschedule event
exit event
exitFlag = F, terminateFlag = F
IP address to name conversion engine:
ipAddrToAsciiEngine at 003A4958 with 0 requests pending
mutex:
epicsMutexId 003A4DD8 source ..\..\..\src\libCom\osi\epicsMutex.cpp line 197
epicsMutex: win32 critical section at 003A4DB8
laborEvent:
exitFlag  boolean = 0
exit event:
Default mutex:
epicsMutexId 003A3E20 source ..\..\..\src\libCom\osi\epicsMutex.cpp line 197
epicsMutex: win32 critical section at 003A3E00
mutex:
epicsMutexId 003A3E20 source ..\..\..\src\libCom\osi\epicsMutex.cpp line 197
epicsMutex: win32 critical section at 003A3E00
        preemptive callback is disabled
        there are 0 unsatisfied IO operations blocking ca_pend_io()
        the current io sequence number is 1
IO done event:
Synchronous group identifier hash table:
Hash table with 0 buckets and 0 items of type struct CASG installed
C:\Documents and Settings\epics>
#########################################################################################################


________________________________

From: Jeff Hill [mailto:johill@lanl.gov]
Sent: Fri 4/11/2008 10:05 AM
To: Mark Rivers
Subject: RE: Windows EPICS problem



Hi Mark,

As I recall, at a higher interest level, the show function in the CA client
library will dump the CA client's UDP search request destination address
list. That might also be interesting information.

Jeff

> -----Original Message-----
> From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
> Sent: Tuesday, April 08, 2008 6:01 PM
> To: anj@aps.anl.gov; johill@lanl.gov; tech-talk@aps.anl.gov
> Subject: Windows EPICS problem
>
> Folks,
>
> I'm having a strange problem on a Windows XP machine.  I am running a soft
> IOC on that machine, but I can't see its PVs from any local CA client
> (caget, IDL, medm, etc).
>
>  However:
> - CA clients on that Windows machine can see PVs on all other machines on
> the network
> - CA clients on other machines on the network can see PVs on the Windows
> machine
>
> So the problem is only that local Windows clients can not see the PVs from
a
> soft IOC running on the same computer.  This is only happening on this
> computer, not other Windows computers running the same IOC binary. (They
are
> not running at the same time, so it is not a duplicate PV name problem).
>
> None of the EPICS environment variables are set except:
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> EPICS_DISPLAY_PATH=P:/epics/adl
>
> The Windows Firewall is not running, nor is any other firewall that I am
> aware of.
>
> What could be wrong?  Of course I need this machine to be running an EPICS
> demo tomorrow!
>
> Help!
>
> Thanks,
> Mark
>