RE: Db/Makefile infinite loop

["Pearson, Matthew via Tech-talk" <tech-talk at aps.anl.gov>]

Hi Simon,

 

Thanks, that seems to be the explanation. I wasn’t sure where to look in the RULES.Db but it’s all clear now. Your fix makes sense.

 

I also looked at the same RULES.Db for 3.14.12.8, and those same rules aren’t there, which explains why it worked for that version of base. For 3.14.12.8 I think just this rule is run:

 

$(COMMON_DIR)/%.db$(RAW): %.template

        @$(RM) $(notdir $@)$(DEP)

        @$(MKMF) -m$(notdir $@)$(DEP) $(DBFLAGS) $@ $< $(TEMPLATE_FILENAME)

        $(ECHO) "Inflating database from $<"

        @$(RM) $@ $*.tmp

        $(MSI) $(DBFLAGS) $< > $*.tmp

        $(MV) $*.tmp $@

 

Cheers,

Matt

 

From: Simon Rose <Simon.Rose at ess.eu>
Sent: Thursday, October 20, 2022 5:31 AM
To: Pearson, Matthew <pearsonmr at ornl.gov>; tech-talk at aps.anl.gov
Subject: [EXTERNAL] Re: Db/Makefile infinite loop

 

Hi Matthew –

 

I think that I understand why this is happening. Warning, this is going to be a bit technical. I will for the moment assume that we are using R7.0.7 from EPICS base.

 

Consider the simpler case of a .template file being expanded into a .db file but without the generated file.

 

In that case, when you run make we end up in the O.$(T_A) directory and we try to build example.db.d first. There are two rules from RULES.Db from EPICS base:

 

%.db$(DEP): %$(TEMPL_SUFFIX)

                @$(RM) $@

                $(MSI3_15) -D $(DBFLAGS) -o $(COMMONDEP_TARGET) $< > $@

 

%.db$(DEP): ../%$(TEMPL_SUFFIX)

                @$(RM) $@

                $(MSI3_15) -D $(DBFLAGS) -o $(COMMONDEP_TARGET) $< > $@

 

We don’t match the first one since we are currently in the O.$(T_A) path, but we do match the second one. Great! .db file built.

 

But note what happens when we add the rules

 

example.template: example.generated

 

example.generated:

                echo "# some more database logic" > example.generated

 

 

into our makefile: in that case the implicit search hits the _first_ rule above, and not the second that is, example.db.d can be built from example.template (from the makefile) and example.generated (also from the makefile). They key is that example.template _does not exist_, at least in the given directory. This is the heart of why you get an infinite loop; since the .d file has been remade, make will run again in that directory, and hit the same chain of dependencies: example.db.d -> example.template (doesn’t exist, needs to be remade) etc.

 

I believe that the correct fix is to point  at the real example.template file, and replace the new rules with

 

../example.template: example.generated

 

example.generated:

                echo "# some more database logic" > example.generated

 

This will mean that once we build the .db.d file once, we have all of our dependencies in place and we stop on the next attempt.

 

It might also be a bit better to prefix the generated file with $(COMMON_DIR) as it is not architecture dependent, but that is somewhat cosmetic.

 

Note also that this is pretty consistent with how these additional rules are generally used, see hxxps://github.com/epics-modules/busy/blob/master/busyApp/src/Makefile#L37-L56 for an example. The targets after this point are typically referencing entities in the build directory, not the source directory.

 

I hope this helps,

 

Cheers,

 

Simon

 

 

From: "Pearson, Matthew" <pearsonmr at ornl.gov>
Date: Wednesday, 19 October 2022 at 17:10
To: Simon Rose <Simon.Rose at ess.eu>, "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: RE: Db/Makefile infinite loop

 

Hi,

 

Sure, I’ve put it here, with a description of what I did:

hxxps://github.com/mp49/epics_make_loop_test

 

This is just a simple module I created that reproduces the problem.

 

For this test I used fresh downloads of base 3.14.12.8, 3.15.9 and 7.0.7.

 

If you clone that, then edit the RELEASE file to point to your base versions, and you should see the problem happen (with 3.15.9+).

 

Cheers,

Matt

 

From: Simon Rose <Simon.Rose at ess.eu>
Sent: Wednesday, October 19, 2022 8:33 AM
To: Pearson, Matthew <pearsonmr at ornl.gov>; tech-talk at aps.anl.gov
Subject: [EXTERNAL] Re: Db/Makefile infinite loop

 

Hello Matthew –

 

I have tried to run your code and I do not get the infinite loop, but I find instead the expected output.  Can you please push the exact failing “epics module” online somewhere in order to test it out?

 

Cheers,

 

Simon

 

From: Tech-talk <tech-talk-bounces at aps.anl.gov> on behalf of "Pearson, Matthew via Tech-talk" <tech-talk at aps.anl.gov>
Reply to: "Pearson, Matthew" <pearsonmr at ornl.gov>
Date: Tuesday, 18 October 2022 at 20:28
To: "tech-talk at aps.anl.gov" <tech-talk at aps.anl.gov>
Subject: Db/Makefile infinite loop

 

Hi,

 

One of our support modules that built fine with base 3.14 is failing with base 7.0.7 (and base 3.15.9). This seems to be because the Db/Makefile has additional rules that define targets that already exist, in order to trigger a script to generate database templates before MSI is executed.

 

For example, if we have a database template file called “example.template”, and we want to build an “example.db”, and “example.template” is:

 

# Some database logic

include "example.generated"

 

And “example.generated” is only built at build time via a script. The script needs to run before MSI is run.

 

I’ve reproduced the problem with this simplified Db/Makefile:

 

TOP=../..

include $(TOP)/configure/CONFIG

 

DB += example.db

 

include $(TOP)/configure/RULES

 

example.template: example.generated

 

example.generated:

              echo "some text" > example.generated

 

 

So, the “example.template” target is defined, presumably as a trick to ensure that the script is run before MSI tries to build “example.db”. But “example.template” already exists in the Db directory.

 

With based 3.14 I get this output:

 

perl /home/controls/common/base/base-3.14.12.8/bin/linux-x86_64/makeMakefile.pl O.linux-x86_64 ../../..

mkdir O.Common

make -C O.linux-x86_64 -f ../Makefile TOP=../../.. \

    T_A=linux-x86_64 install

make[1]: Entering directory '/home/controls/inst/st99/st99-example/main/st99-exampleApp/Db/O.linux-x86_64'

echo "some text" > example.generated

#cp ../example.template .

Inflating database from example.template

msi    -I. -I.. -I../O.Common -I../../../db -I/home/controls/common/base/base-3.14.12.8/db example.template > example.tmp

mv example.tmp ../O.Common/example.db

Installing created db file ../../../db/example.db

make[1]: Leaving directory '/home/controls/inst/st99/st99-example/main/st99-exampleApp/Db/O.linux-x86_64'

 

which works fine.

 

However, with base 7.0.7 and 3.15.9 I get:

 

perl -CSD /home/controls/common/base/base-3.15.9/bin/linux-x86_64/makeMakefile.pl O.linux-x86_64 ../../..

mkdir O.Common

make -C O.linux-x86_64 -f ../Makefile TOP=../../.. \

    T_A=linux-x86_64 install

make[1]: Entering directory '/home/controls/inst/st99/st99-example/main/st99-exampleApp/Db/O.linux-x86_64'

echo "some text" > example.generated

#cp ../example.template .

/home/controls/common/base/base-3.15.9/bin/linux-x86_64/msi -D    -I. -I.. -I../O.Common -I../../../db -I/home/controls/common/base/base-3.15.9/db -o ../O.Common/example.db example.template > example.db.d

/home/controls/common/base/base-3.15.9/bin/linux-x86_64/msi -D    -I. -I.. -I../O.Common -I../../../db -I/home/controls/common/base/base-3.15.9/db -o ../O.Common/example.db example.template > example.db.d

/home/controls/common/base/base-3.15.9/bin/linux-x86_64/msi -D    -I. -I.. -I../O.Common -I../../../db -I/home/controls/common/base/base-3.15.9/db -o ../O.Common/example.db example.template > example.db.d

 

and it’s an infinite loop.

 

I believe this is because we have a Makefile target with the same name as the database template file, and the timestamp on that file never changes (because the Makefile rule is not touching the template file, only generating a different file). So this logic may be suspect from the start. Is there a better way to do this? Specially, how to run a script in the Db/Makefile before MSI without resorting to the above trick?

 

I have a workaround, which works for all base versions, which is to make a new copy of the template file so that Make is happy:

 

example.generated:

              echo "some text" > example.generated

              cp ../example.template .

 

 

However, why did it work for 3.14 and not 3.15+? Is it because the db.d format changed and it now references “example.template” whereas before it didn’t? For base 3.14 the O.linux-x86_64/example.db.d file is:

 

# DO NOT EDIT: This file created by mkmf.pl,v 1.5 2002/03/25 21:33:24 jba Exp $

 

../O.Common/example.db : example.generated

 

Whereas the same file using base 3.15 is:

 

../O.Common/example.db: ../example.template \

./example.generated

 

I assume that db.d file is used as a Makefile rule, and since “example.template” is now referenced the build system will repeatedly try to execute the “example.template” rule?

 

Cheers,

Matt

 

 

 

STS Integrated Control System Lead Engineer

Spallation Neutron Source

Oak Ridge National Lab