Experimental Physics and
Industrial Control System

Michael Davidsaver <[email protected]> · Thu, 28 Dec 2017 09:44:57 -0600

On 12/28/2017 09:39 AM, Mark Rivers wrote:
>> Am I missing anything?
> 
> Many unit test failures.

Ok, fair enough.  These are probably (hopefully) due to the 4 enumerated
issue, but I'll open a ticket for the test failures as well.

>> -----Original Message-----
>> From: Michael Davidsaver [mailto:[email protected]]
>> Sent: Thursday, December 28, 2017 9:38 AM
>> To: Mark Rivers; 'Andrew Johnson'; '[email protected]'
>> Subject: Re: Problem building example application on windows-x64
>>
>> On 12/28/2017 05:53 AM, Mark Rivers wrote:
>>>
>>>> However, in this particular case building with c++11 should avoid a crash here as
>> initialization
>>>> order of local static variables is well defined in new c++ versions.
>>>
>>> Are you suggesting to require c++11 for base 7.0.1.1?  That would mean not supporting
>> VS2010.
>>
>> No.  I was suggesting a possible workaround so that you could move forward.
>>
>>> I  was in fact trying to see if the problem would go away with VS2015 when I ran into
>> the problem I reported yesterday with VS2015 failing because of the error building
>> Com.res.
>>
>> So much for that.  I manged to miss this issue.  Time to open some tickets I think.
>> So far I count four distinct symptoms, and the test failures.
>>
>> What I see so far:
>>
>> 1. Global ctor ordering issues needs to be opened against pvDataCPP and pvAccessCPP
>>    (any probably others).
>>
>> 2. The osiSockAttach() issue against pvAccessCPP.
>>
>> 3. The Com.res failure against Base (on launchpad)
>>
>> 4. The timestamp issue w/ genVersionHeader.pl against Base
>>
>>
>> Am I missing anything?
>>
>>
>>> Mark
>>>
>>>
>>> ________________________________
>>> From: Michael Davidsaver <[email protected]>
>>> Sent: Wednesday, December 27, 2017 6:02 PM
>>> To: Mark Rivers; 'Andrew Johnson'; '[email protected]'
>>> Subject: Re: Problem building example application on windows-x64
>>>
>>> Looks like more global constructor order fun...  This has been a persistent problem
>>> due to an annoying "optimization" of storing pointers to factories in global variables.
>>>
>>> eg. from clientContextImpl.cpp
>>>
>>>> PVDataCreatePtr BaseRequestImpl::pvDataCreate = getPVDataCreate();
>>>
>>> or responseHandlers.cpp
>>>
>>>> static PVDataCreatePtr pvDataCreate = getPVDataCreate();
>>>
>>> or serializationHelper.cpp
>>>
>>>> PVDataCreatePtr SerializationHelper::_pvDataCreate(getPVDataCreate());
>>>
>>> and that's just in pvAccessCPP.  If any of these get ordered before the initialization in
>> pvDataCPP,
>>> then bang...
>>>
>>> The DLL loading process probably ensures the necessary ordering, but not so with static
>> linking.
>>>
>>> Fixing this generally is time consuming and tricky as I don't know of a way to detect
>> potential
>>> issues before a crash occurs.
>>>
>>> However, in this particular case building with c++11 should avoid a crash here as
>> initialization
>>> order of local static variables is well defined in new c++ versions.
>>>
>>>
>>> On 12/27/2017 04:29 PM, Mark Rivers wrote:
>>>> I rebuilt the windows-x64-static architecture with HOST_OPT=NO so I could get a
>> stack trace in the example application.  Here it is:
>>>>
>>>>
>>>>
>>>>>          test.exe!epicsMutex::lock()  Line 273 + 0x5 bytes     C++
>>>>
>>>>            test.exe!epics::pvData::Lock::Lock(epicsMutex & m={...})  Line 45 + 0x46 bytes
>> C++
>>>>
>>>>            test.exe!epics::pvData::FieldCreate::getFieldCreate()  Line 1459 + 0x12 bytes
>> C++
>>>>
>>>>            test.exe!epics::pvData::getFieldCreate()  Line 1230  C++
>>>>
>>>>            test.exe!epics::pvData::PVDataCreate::PVDataCreate()  Line 358 + 0x23 bytes
>> C++
>>>>
>>>>            test.exe!epics::pvData::PVDataCreate::getPVDataCreate()  Line 617 + 0x21
>> bytes            C++
>>>>
>>>>            test.exe!epics::pvData::getPVDataCreate()  Line 1656          C++
>>>>
>>>>            test.exe!epics::pvAccess::`dynamic initializer for 'pvDataCreate''()  Line 58 + 0x1a
>> bytes   C++
>>>>
>>>>            test.exe!_initterm(void (void)* * pfbegin=0x000000013f6ada00, void (void)* *
>> pfend=0x000000013f6adf48)  Line 873        C
>>>>
>>>>            test.exe!_cinit(int initFloatingPrecision=1)  Line 301            C
>>>>
>>>>            test.exe!__tmainCRTStartup()  Line 262 + 0xa bytes            C
>>>>
>>>>            test.exe!mainCRTStartup()  Line 189 C
>>>>
>>>>            kernel32.dll!00000000775259cd()
>>>>
>>>>             [Frames below may be incorrect and/or missing, no symbols loaded for
>> kernel32.dll]
>>>>
>>>>            ntdll.dll!000000007775a561()            ''
>>>>
>>>>
>>>>
>>>> In epicsMutex::lock this=0.
>>>>
>>>>
>>>>
>>>> That is called from this code in lock.h
>>>>
>>>>     explicit Lock(Mutex &m)
>>>>
>>>>     : mutexPtr(m), locked(true)
>>>>
>>>>     { mutexPtr.lock();}
>>>>
>>>>
>>>>
>>>> In that code the debugger tells me that "m id=???", Error CXX0030 Expression cannot
>> be evaluated.
>>>>
>>>>
>>>>
>>>> This is Visual Studio 2010.
>>>>
>>>>
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:*Mark Rivers
>>>> *Sent:* Wednesday, December 27, 2017 3:15 PM
>>>> *To:* 'Andrew Johnson'; [email protected]
>>>> *Subject:* RE: Problem building example application on windows-x64
>>>>
>>>>
>>>>
>>>> OK, that patch let me get the example application to compile.
>>>>
>>>>
>>>>
>>>> However, there are problems when running it.  The behavior I see is different on
>> windows-x64 and windows-x64-static.
>>>>
>>>>
>>>>
>>>> - On windows-x64 the IOC application runs OK but when I try to exit the application I
>> get the error I reported in my first message today, i.e. epicsSocketDestroy errors.  ^C will
>> not kill the IOC application, I need to use the Task Manager.  This is a serious problem, and
>> as I said earlier it affects other IOCs such as simDetector.
>>>>
>>>>
>>>>
>>>> J:\epics\devel\example\iocBoot\ioctest>..\..\bin\windows-x64\test.exe st.cmd
>>>>
>>>> #!../../bin/windows-x64-static/test
>>>>
>>>> < envPaths
>>>>
>>>> epicsEnvSet("IOC","ioctest")
>>>>
>>>> epicsEnvSet("TOP","J:/epics/devel/example")
>>>>
>>>> epicsEnvSet("EPICS_BASE","H:/epics-devel/base-7.0.1")
>>>>
>>>> cd "J:/epics/devel/example"
>>>>
>>>> ## Register all support components
>>>>
>>>> dbLoadDatabase "dbd/test.dbd"
>>>>
>>>> test_registerRecordDeviceDriver pdbbase
>>>>
>>>> ## Load record instances
>>>>
>>>> dbLoadTemplate "db/user.substitutions"
>>>>
>>>> dbLoadRecords "db/testVersion.db", "user=rivers"
>>>>
>>>> dbLoadRecords "db/dbSubExample.db", "user=rivers"
>>>>
>>>> #var mySubDebug 1
>>>>
>>>> #traceIocInit
>>>>
>>>> cd "J:/epics/devel/example/iocBoot/ioctest"
>>>>
>>>> iocInit
>>>>
>>>> Starting iocInit
>>>>
>>>>
>> ##########################################################################
>> ##
>>>>
>>>> ## EPICS R7.0.1.1
>>>>
>>>> ## EPICS Base built Dec 16 2017
>>>>
>>>>
>> ##########################################################################
>> ##
>>>>
>>>> iocRun: All initialization complete
>>>>
>>>> 2017-12-27T14:52:54.595 No broadcast addresses found or specified - empty address
>> list!
>>>>
>>>> ## Start any sequence programs
>>>>
>>>> #seq sncExample, "user=rivers"
>>>>
>>>> epics>
>>>>
>>>> epics>
>>>>
>>>> epics>
>>>>
>>>> epics>
>>>>
>>>> epics> exit
>>>>
>>>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>>>
>>>> 2017-12-27T14:53:16.250 Receive thread for UDP socket 0.0.0.0:0 has not exited.
>>>>
>>>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>>>
>>>> 2017-12-27T14:53:21.296 Receive thread for UDP socket 164.54.160.31:5076 has not
>> exited.
>>>>
>>>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>>>
>>>> 2017-12-27T14:53:26.333 Receive thread for UDP socket 0.0.0.0:5076 has not exited.
>>>>
>>>> epicsSocketDestroy: failed to close a socket because "WINSOCK Error 10093"
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On windows-x64-static the IOC crashes immediately, without even starting to execute
>> the st.cmd file:
>>>>
>>>>
>>>>
>>>> J:\epics\devel\example>make -sj clean uninstall
>>>>
>>>>
>>>>
>>>> J:\epics\devel\example>make -sj
>>>>
>>>> devXxxSoft.c
>>>>
>>>> xxxRecord.c
>>>>
>>>> testHello.c
>>>>
>>>> dbSubExample.c
>>>>
>>>> devtestVersion.c
>>>>
>>>> initTrace.c
>>>>
>>>> testMain.cpp
>>>>
>>>> test_registerRecordDeviceDriver.cpp
>>>>
>>>> ../devtestVersion.c(27) : warning C4267: '=' : conversion from 'size_t' to 'epicsUInt32',
>> possible loss of data
>>>>
>>>> ../xxxRecord.c(211) : warning C4244: '=' : conversion from 'epicsFloat64' to 'float',
>> possible loss of data
>>>>
>>>> ../xxxRecord.c(211) : warning C4244: '=' : conversion from 'epicsFloat64' to 'float',
>> possible loss of data
>>>>
>>>> Appending initTrace.obj
>>>>
>>>> Appending testHello.obj
>>>>
>>>> Appending devtestVersion.obj
>>>>
>>>> Appending dbSubExample.obj
>>>>
>>>> Appending devXxxSoft.obj
>>>>
>>>> Appending xxxRecord.obj
>>>>
>>>>
>>>>
>>>> J:\epics\devel\example>cd iocBoot\ioctest
>>>>
>>>>
>>>>
>>>> J:\epics\devel\example\iocBoot\ioctest>..\..\bin\windows-x64-static\test.exe st.cmd
>>>>
>>>>
>>>>
>>>> At that point Windows pops up a dialog box telling me that test.exe has crashed.
>> Opening the Visual Studio debugger tells me that it is an access violation on address 0.
>>>>
>>>>
>>>>
>>>> Is there a test in EPICS base that instantiates the example IOC?
>>>>
>>>>
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>
>>>>> From: [email protected] <mailto:[email protected]>
>> [mailto:[email protected]] On Behalf
>>>>
>>>>> Of Andrew Johnson
>>>>
>>>>> Sent: Wednesday, December 27, 2017 1:04 PM
>>>>
>>>>> To: [email protected] <mailto:[email protected]>
>>>>
>>>>> Subject: Re: Problem building example application on windows-x64
>>>>
>>>>>
>>>>
>>>>> Hi Mark,
>>>>
>>>>>
>>>>
>>>>> On 12/27/2017 12:05 PM, Mark Rivers wrote:
>>>>
>>>>>> I just did this again, but running makeBaseApp.pl with Windows rather
>>>>
>>>>>> than Linux.  I get the same problem. It looks like the problem is that
>>>>
>>>>>> testVersion.h is changing because the timestamp is incrementing by 1
>>>>
>>>>>> second between the time it is created and checked?
>>>>
>>>>>
>>>>
>>>>> I agree; it seems to be a problem with the genVersionHeader.pl script
>>>>
>>>>> that Michael added in 3.16.1 and the dependency rules. This is the
>>>>
>>>>> relevant part of your build log:
>>>>
>>>>>
>>>>
>>>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/genVersionHeader.pl
>>>>
>>>>>> -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>>>
>>>>>>
>>>>
>>>>>> Updating VCS header ../O.Common/testVersion.h
>>>>
>>>>>>     testVERSION = "2017-12-27T11:39:33"
>>>>
>>>>>>
>>>>
>>>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/mkmf.pl  -m
>>>>
>>>>>> devtestVersion.d -I. -I../O.Common -I. -I. -I..
>>>>
>>>>>> -I../../../include/compiler/msvc -I../../../include/os/WIN32
>>>>
>>>>>> -I../../../include -IH:/epics-devel/base-7.0.1/include/compiler/msvc
>>>>
>>>>>> -IH:/epics-devel/base-7.0.1/include/os/WIN32
>>>>
>>>>>> -IH:/epics-devel/base-7.0.1/include        devtestVersion.obj
>>>>
>>>>>> ../devtestVersion.c
>>>>
>>>>>>
>>>>
>>>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/genVersionHeader.pl
>>>>
>>>>>> -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>>>
>>>>>>
>>>>
>>>>>> Updating VCS header ../O.Common/testVersion.h
>>>>
>>>>>>     testVERSION = "2017-12-27T11:39:34"
>>>>
>>>>>>
>>>>
>>>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/mkmf.pl  -m
>>>>
>>>>>> devtestVersion.d -I. -I../O.Common -I. -I. -I..
>>>>
>>>>>> -I../../../include/compiler/msvc -I../../../include/os/WIN32
>>>>
>>>>>> -I../../../include -IH:/epics-devel/base-7.0.1/include/compiler/msvc
>>>>
>>>>>> -IH:/epics-devel/base-7.0.1/include/os/WIN32
>>>>
>>>>>> -IH:/epics-devel/base-7.0.1/include        devtestVersion.obj
>>>>
>>>>>> ../devtestVersion.c
>>>>
>>>>>>
>>>>
>>>>>> perl -CSD H:/epics-devel/base-7.0.1/bin/windows-x64/genVersionHeader.pl
>>>>
>>>>>> -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>>>
>>>>>>
>>>>
>>>>>> Updating VCS header ../O.Common/testVersion.h
>>>>
>>>>>>     testVERSION = "2017-12-27T11:39:35"
>>>>
>>>>>
>>>>
>>>>> whereas on Linux I get:
>>>>
>>>>>
>>>>
>>>>>> perl -CSD /home/phoebus/ANJ/epics/base/7.0/bin/linux-x86_64/genVersionHeader.pl
>>>>
>>>>>>  -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>>>
>>>>>>
>>>>
>>>>>> Updating VCS header ../O.Common/testVersion.h
>>>>
>>>>>>     testVERSION = "2017-12-27T12:08:23-0600"
>>>>
>>>>>>
>>>>
>>>>>> /usr/bin/gcc  -D_GNU_SOURCE -D_DEFAULT_SOURCE         -
>>>>
>>>>> DUSE_TYPED_RSET
>>>>
>>>>>> -D_X86_64_  -DUNIX  -Dlinux     -O3 -g   -Wall      -mtune=generic     -m64
>>>>
>>>>>> -Wpointer-arith -fvisibility=hidden -fPIC -I. -I../O.Common -I. -I. -I..
>>>>
>>>>>> -I../../../include/compiler/gcc -I../../../include/os/Linux
>>>>
>>>>> -I../../../include
>>>>
>>>>>> -I/home/phoebus/ANJ/epics/base/7.0/include/compiler/gcc
>>>>
>>>>>> -I/home/phoebus/ANJ/epics/base/7.0/include/os/Linux
>>>>
>>>>>> -I/home/phoebus/ANJ/epics/base/7.0/include        -MM -MF devtestVersion.d
>>>>
>>>>>> ../devtestVersion.c
>>>>
>>>>>>
>>>>
>>>>>> perl -CSD /home/phoebus/ANJ/epics/base/7.0/bin/linux-x86_64/genVersionHeader.pl
>>>>
>>>>>>  -t ../../.. -N testVERSION -V "" ../O.Common/testVersion.h
>>>>
>>>>>>
>>>>
>>>>>> Keeping VCS header ../O.Common/testVersion.h
>>>>
>>>>>>     testVERSION = "2017-12-27T12:08:23-0600"
>>>>
>>>>>
>>>>
>>>>> There seems to be a mismatch between the assumptions that script makes
>>>>
>>>>> about file-system modification times and how GNUmake works on Windows.
>>>>
>>>>> The old DOS FAT file-system had a time-stamp granularity of 2 seconds
>>>>
>>>>> and that has resulted in some workarounds being added to GNUmake on
>>>>
>>>>> Windows, e.g. see http://www.delorie.com/djgpp/v2faq/faq22_18.html or
>>>>
>>>>> there may be a delay added at the end of the phase-1 run before GNUmake
>>>>
>>>>> re-reads the Makefiles - that might explain what's happening here.
>>>>
>>>>>
>>>>
>>>>> Michael: How about we drop the seconds field completely from the build
>>>>
>>>>> date/time-stamp? That should prevent this kind of problem from
>>>>
>>>>> recurring. It would still trigger a re-creation of the header whenever
>>>>
>>>>> there is a roll-over, but that should only happen once, not continuously
>>>>
>>>>> like Mark is seeing.
>>>>
>>>>>
>>>>
>>>>> Mark: If you want to try out that fix, edit
>>>>
>>>>> base-7.0/src/tools/genVersionHeader.pl and make this change:
>>>>
>>>>>
>>>>
>>>>>> diff --git a/src/tools/genVersionHeader.pl b/src/tools/genVersionHeader.pl
>>>>
>>>>>> index 5851ea8..8e50fec 100644
>>>>
>>>>>> --- a/src/tools/genVersionHeader.pl
>>>>
>>>>>> +++ b/src/tools/genVersionHeader.pl
>>>>
>>>>>> @@ -19,7 +19,7 @@ use POSIX qw(strftime);
>>>>
>>>>>>  use strict;
>>>>
>>>>>>
>>>>
>>>>>>  # RFC 8601 date+time w/ zone (eg "2014-08-29T09:42:47-0700")
>>>>
>>>>>> -my $tfmt = '%Y-%m-%dT%H:%M:%S';
>>>>
>>>>>> +my $tfmt = '%Y-%m-%dT%H:%M';
>>>>
>>>>>>  $tfmt .= '%z' unless $^O eq 'MSWin32'; # %z returns zone name on Windows
>>>>
>>>>>>  my $now = strftime($tfmt, localtime);
>>>>
>>>>>>
>>>>
>>>>>
>>>>
>>>>> - Andrew
>>>>
>>>>>
>>>>
>>>>> --
>>>>
>>>>> Arguing for surveillance because you have nothing to hide is no
>>>>
>>>>> different than making the claim, "I don't care about freedom of
>>>>
>>>>> speech because I have nothing to say." -- Edward Snowdon
>>>>
>>>
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System