Experimental Physics and
Industrial Control System

Michael Davidsaver <[email protected]> · Thu, 28 Dec 2017 09:37:38 -0600

On 12/28/2017 05:53 AM, Mark Rivers wrote:
> 
>> However, in this particular case building with c++11 should avoid a crash here as initialization
>> order of local static variables is well defined in new c++ versions.
> 
> Are you suggesting to require c++11 for base 7.0.1.1?  That would mean not supporting VS2010.

No.  I was suggesting a possible workaround so that you could move forward.

> I  was in fact trying to see if the problem would go away with VS2015 when I ran into the problem I reported yesterday with VS2015 failing because of the error building Com.res.

So much for that.  I manged to miss this issue.  Time to open some tickets I think.
So far I count four distinct symptoms, and the test failures.

What I see so far:

1. Global ctor ordering issues needs to be opened against pvDataCPP and pvAccessCPP
   (any probably others).

2. The osiSockAttach() issue against pvAccessCPP.

3. The Com.res failure against Base (on launchpad)

4. The timestamp issue w/ genVersionHeader.pl against Base

Am I missing anything?

> Mark
> 
> 
> ________________________________
> From: Michael Davidsaver <[email protected]>
> Sent: Wednesday, December 27, 2017 6:02 PM
> To: Mark Rivers; 'Andrew Johnson'; '[email protected]'
> Subject: Re: Problem building example application on windows-x64
> 
> Looks like more global constructor order fun...  This has been a persistent problem
> due to an annoying "optimization" of storing pointers to factories in global variables.
> 
> eg. from clientContextImpl.cpp
> 
>> PVDataCreatePtr BaseRequestImpl::pvDataCreate = getPVDataCreate();
> 
> or responseHandlers.cpp
> 
>> static PVDataCreatePtr pvDataCreate = getPVDataCreate();
> 
> or serializationHelper.cpp
> 
>> PVDataCreatePtr SerializationHelper::_pvDataCreate(getPVDataCreate());
> 
> and that's just in pvAccessCPP.  If any of these get ordered before the initialization in pvDataCPP,
> then bang...
> 
> The DLL loading process probably ensures the necessary ordering, but not so with static linking.
> 
> Fixing this generally is time consuming and tricky as I don't know of a way to detect potential
> issues before a crash occurs.
> 
> However, in this particular case building with c++11 should avoid a crash here as initialization
> order of local static variables is well defined in new c++ versions.
> 
> 
> On 12/27/2017 04:29 PM, Mark Rivers wrote:
>> I rebuilt the windows-x64-static architecture with HOST_OPT=NO so I could get a stack trace in the example application.  Here it is:
>>
>>
>>
>>>          test.exe!epicsMutex::lock()  Line 273 + 0x5 bytes     C++
>>
>>            test.exe!epics::pvData::Lock::Lock(epicsMutex & m={...})  Line 45 + 0x46 bytes  C++
>>
>>            test.exe!epics::pvData::FieldCreate::getFieldCreate()  Line 1459 + 0x12 bytes        C++
>>
>>            test.exe!epics::pvData::getFieldCreate()  Line 1230  C++
>>
>>            test.exe!epics::pvData::PVDataCreate::PVDataCreate()  Line 358 + 0x23 bytes     C++
>>
>>            test.exe!epics::pvData::PVDataCreate::getPVDataCreate()  Line 617 + 0x21 bytes            C++
>>
>>            test.exe!epics::pvData::getPVDataCreate()  Line 1656          C++
>>
>>            test.exe!epics::pvAccess::`dynamic initializer for 'pvDataCreate''()  Line 58 + 0x1a bytes   C++
>>
>>            test.exe!_initterm(void (void)* * pfbegin=0x000000013f6ada00, void (void)* * pfend=0x000000013f6adf48)  Line 873        C
>>
>>            test.exe!_cinit(int initFloatingPrecision=1)  Line 301            C
>>
>>            test.exe!__tmainCRTStartup()  Line 262 + 0xa bytes            C
>>
>>            test.exe!mainCRTStartup()  Line 189 C
>>
>>            kernel32.dll!00000000775259cd()
>>
>>             [Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
>>
>>            ntdll.dll!000000007775a561()            ''
>>
>>
>>
>> In epicsMutex::lock this=0.
>>
>>
>>
>> That is called from this code in lock.h
>>
>>     explicit Lock(Mutex &m)
>>
>>     : mutexPtr(m), locked(true)
>>
>>     { mutexPtr.lock();}
>>
>>
>>
>> In that code the debugger tells me that "m id=???", Error CXX0030 Expression cannot be evaluated.
>>
>>
>>
>> This is Visual Studio 2010.
>>
>>
>>
>> Mark
>>
>>
>>
>>
>>
>>
>>
>> *From:*Mark Rivers
>> *Sent:* Wednesday, December 27, 2017 3:15 PM
>> *To:* 'Andrew Johnson'; [email protected]
>> *Subject:* RE: Problem building example application on windows-x64
>>
>>
>>
>> OK, that patch let me get the example application to compile.
>>
>>
>>
>> However, there are problems when running it.  The behavior I see is different on windows-x64 and windows-x64-static.
>>
>>
>>
>> - On windows-x64 the IOC application runs OK but when I try to exit the application I get the error I reported in my first message today, i.e. epicsSocketDestroy errors.  ^C will not kill the IOC application, I need to use the Task Manager.  This is a serious problem, and as I said earlier it affects other IOCs such as simDetector.
>>
>>
>>
>> J:\epics\devel\example\iocBoot\ioctest>..\..\bin\windows-x64\test.exe st.cmd
>>
>> #!../../bin/windows-x64-static/test
>>
>> < envPaths
>>
>> epicsEnvSet("IOC","ioctest")
>>
>> epicsEnvSet("TOP","J:/epics/devel/example")
>>
>> epicsEnvSet("EPICS_BASE","H:/epics-devel/base-7.0.1")
>>
>> cd "J:/epics/devel/example"
>>
>> ## Register all support components
>>
>> dbLoadDatabase "dbd/test.dbd"
>>
>> test_registerRecordDeviceDriver pdbbase
>>
>> ## Load record instances
>>
>> dbLoadTemplate "db/user.substitutions"
>>
>> dbLoadRecords "db/testVersion.db", "user=rivers"
>>
>> dbLoadRecords "db/dbSubExample.db", "user=rivers"
>>
>> #var mySubDebug 1
>>
>> #traceIocInit
>>
>> cd "J:/epics/devel/example/iocBoot/ioctest"
>>
>> iocInit
>>
>> Starting iocInit
>>
>> ############################################################################
>>
>> ## EPICS R7.0.1.1
>>
>> ## EPICS Base built Dec 16 2017
>>
>> ############################################################################
>>
>> iocRun: All initialization complete
>>
>> 2017-12-27T14:52:54.595 No broadcast addresses found or specified - empty address list!
>>
>> ## Start any sequence programs
>>
>> #seq sncExample, "user=rivers"
>>
>> epics>
>>
>> epics>
>>
>> epics>
>>
>> epics>
>>
>> epics> exit
>>
>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>
>> 2017-12-27T14:53:16.250 Receive thread for UDP socket 0.0.0.0:0 has not exited.
>>
>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>
>> 2017-12-27T14:53:21.296 Receive thread for UDP socket 164.54.160.31:5076 has not exited.
>>
>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>
>> 2017-12-27T14:53:26.333 Receive thread for UDP socket 0.0.0.0:5076 has not exited.
>>
>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>
>>
>>
>>
>>
>> On windows-x64-static the IOC crashes immediately, without even starting to execute the st.cmd file:
>>
>>
>>
>> J:\epics\devel\example>make -sj clean uninstall
>>
>>
>>
>> J:\epics\devel\example>make -sj
>>
>> devXxxSoft.c
>>
>> xxxRecord.c
>>
>> testHello.c
>>
>> dbSubExample.c
>>
>> devtestVersion.c
>>
>> initTrace.c
>>
>> testMain.cpp
>>
>> test_registerRecordDeviceDriver.cpp
>>
>> ../devtestVersion.c(27) : warning C4267: '=' : conversion from 'size_t' to 'epicsUInt32', possible loss of data
>>
>> ../xxxRecord.c(211) : warning C4244: '=' : conversion from 'epicsFloat64' to 'float', possible loss of data
>>
>> ../xxxRecord.c(211) : warning C4244: '=' : conversion from 'epicsFloat64' to 'float', possible loss of data
>>
>> Appending initTrace.obj
>>
>> Appending testHello.obj
>>
>> Appending devtestVersion.obj
>>
>> Appending dbSubExample.obj
>>
>> Appending devXxxSoft.obj
>>
>> Appending xxxRecord.obj
>>
>>
>>
>> J:\epics\devel\example>cd iocBoot\ioctest
>>
>>
>>
>> J:\epics\devel\example\iocBoot\ioctest>..\..\bin\windows-x64-static\test.exe st.cmd
>>
>>
>>
>> At that point Windows pops up a dialog box telling me that test.exe has crashed.  Opening the Visual Studio debugger tells me that it is an access violation on address 0.
>>
>>
>>
>> Is there a test in EPICS base that instantiates the example IOC?
>>
>>
>>
>> Mark
>>
>>
>>
>>
>>
>>> -----Original Message-----
>>
>>> From: [email protected] <mailto:[email protected]> [mailto:[email protected]] On Behalf
>>
>>> Of Andrew Johnson
>>
>>> Sent: Wednesday, December 27, 2017 1:04 PM
>>
>>> To: [email protected] <mailto:[email protected]>
>>
>>> Subject: Re: Problem building example application on windows-x64
>>
>>>
>>
>>> Hi Mark,
>>
>>>
>>
>>> On 12/27/2017 12:05 PM, Mark Rivers wrote:
>>
>>>> I just did this again, but running makeBaseApp.pl with Windows rather
>>
>>>> than Linux.  I get the same problem. It looks like the problem is that
>>
>>>> testVersion.h is changing because the timestamp is incrementing by 1
>>
>>>> second between the time it is created and checked?
>>
>>>
>>
>>> I agree; it seems to be a problem with the genVersionHeader.pl script
>>
>>> that Michael added in 3.16.1 and the dependency rules. This is the
>>
>>> relevant part of your build log:
>>
>>>
>>
>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/genVersionHeader.pl
>>
>>>> -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>
>>>>
>>
>>>> Updating VCS header ../O.Common/testVersion.h
>>
>>>>     testVERSION = "2017-12-27T11:39:33"
>>
>>>>
>>
>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/mkmf.pl  -m
>>
>>>> devtestVersion.d -I. -I../O.Common -I. -I. -I..
>>
>>>> -I../../../include/compiler/msvc -I../../../include/os/WIN32
>>
>>>> -I../../../include -IH:/epics-devel/base-7.0.1/include/compiler/msvc
>>
>>>> -IH:/epics-devel/base-7.0.1/include/os/WIN32
>>
>>>> -IH:/epics-devel/base-7.0.1/include        devtestVersion.obj
>>
>>>> ../devtestVersion.c
>>
>>>>
>>
>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/genVersionHeader.pl
>>
>>>> -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>
>>>>
>>
>>>> Updating VCS header ../O.Common/testVersion.h
>>
>>>>     testVERSION = "2017-12-27T11:39:34"
>>
>>>>
>>
>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/mkmf.pl  -m
>>
>>>> devtestVersion.d -I. -I../O.Common -I. -I. -I..
>>
>>>> -I../../../include/compiler/msvc -I../../../include/os/WIN32
>>
>>>> -I../../../include -IH:/epics-devel/base-7.0.1/include/compiler/msvc
>>
>>>> -IH:/epics-devel/base-7.0.1/include/os/WIN32
>>
>>>> -IH:/epics-devel/base-7.0.1/include        devtestVersion.obj
>>
>>>> ../devtestVersion.c
>>
>>>>
>>
>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/genVersionHeader.pl
>>
>>>> -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>
>>>>
>>
>>>> Updating VCS header ../O.Common/testVersion.h
>>
>>>>     testVERSION = "2017-12-27T11:39:35"
>>
>>>
>>
>>> whereas on Linux I get:
>>
>>>
>>
>>>> perl -CSD /home/phoebus/ANJ/epics/base/7.0/bin/linux-x86_64/genVersionHeader.pl
>>
>>>>  -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>
>>>>
>>
>>>> Updating VCS header ../O.Common/testVersion.h
>>
>>>>     testVERSION = "2017-12-27T12:08:23-0600"
>>
>>>>
>>
>>>> /usr/bin/gcc  -D_GNU_SOURCE -D_DEFAULT_SOURCE         -
>>
>>> DUSE_TYPED_RSET
>>
>>>> -D_X86_64_  -DUNIX  -Dlinux     -O3 -g   -Wall      -mtune=generic     -m64
>>
>>>> -Wpointer-arith -fvisibility=hidden -fPIC -I. -I../O.Common -I. -I. -I..
>>
>>>> -I../../../include/compiler/gcc -I../../../include/os/Linux
>>
>>> -I../../../include
>>
>>>> -I/home/phoebus/ANJ/epics/base/7.0/include/compiler/gcc
>>
>>>> -I/home/phoebus/ANJ/epics/base/7.0/include/os/Linux
>>
>>>> -I/home/phoebus/ANJ/epics/base/7.0/include        -MM -MF devtestVersion.d
>>
>>>> ../devtestVersion.c
>>
>>>>
>>
>>>> perl -CSD /home/phoebus/ANJ/epics/base/7.0/bin/linux-x86_64/genVersionHeader.pl
>>
>>>>  -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>
>>>>
>>
>>>> Keeping VCS header ../O.Common/testVersion.h
>>
>>>>     testVERSION = "2017-12-27T12:08:23-0600"
>>
>>>
>>
>>> There seems to be a mismatch between the assumptions that script makes
>>
>>> about file-system modification times and how GNUmake works on Windows.
>>
>>> The old DOS FAT file-system had a time-stamp granularity of 2 seconds
>>
>>> and that has resulted in some workarounds being added to GNUmake on
>>
>>> Windows, e.g. see http://www.delorie.com/djgpp/v2faq/faq22_18.html or
>>
>>> there may be a delay added at the end of the phase-1 run before GNUmake
>>
>>> re-reads the Makefiles - that might explain what's happening here.
>>
>>>
>>
>>> Michael: How about we drop the seconds field completely from the build
>>
>>> date/time-stamp? That should prevent this kind of problem from
>>
>>> recurring. It would still trigger a re-creation of the header whenever
>>
>>> there is a roll-over, but that should only happen once, not continuously
>>
>>> like Mark is seeing.
>>
>>>
>>
>>> Mark: If you want to try out that fix, edit
>>
>>> base-7.0/src/tools/genVersionHeader.pl and make this change:
>>
>>>
>>
>>>> diff --git a/src/tools/genVersionHeader.pl b/src/tools/genVersionHeader.pl
>>
>>>> index 5851ea8..8e50fec 100644
>>
>>>> --- a/src/tools/genVersionHeader.pl
>>
>>>> +++ b/src/tools/genVersionHeader.pl
>>
>>>> @@ -19,7 +19,7 @@ use POSIX qw(strftime);
>>
>>>>  use strict;
>>
>>>>
>>
>>>>  # RFC 8601 date+time w/ zone (eg "2014-08-29T09:42:47-0700")
>>
>>>> -my $tfmt = '%Y-%m-%dT%H:%M:%S';
>>
>>>> +my $tfmt = '%Y-%m-%dT%H:%M';
>>
>>>>  $tfmt .= '%z' unless $^O eq 'MSWin32'; # %z returns zone name on Windows
>>
>>>>  my $now = strftime($tfmt, localtime);
>>
>>>>
>>
>>>
>>
>>> - Andrew
>>
>>>
>>
>>> --
>>
>>> Arguing for surveillance because you have nothing to hide is no
>>
>>> different than making the claim, "I don't care about freedom of
>>
>>> speech because I have nothing to say." -- Edward Snowdon
>>
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System