[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Automatic dependencies are out of sync


  • To: Jan Beulich <JBeulich@xxxxxxxx>
  • From: Juergen Gross <jgross@xxxxxxxx>
  • Date: Thu, 6 Sep 2018 10:27:27 +0200
  • Autocrypt: addr=jgross@xxxxxxxx; prefer-encrypt=mutual; keydata= xsBNBFOMcBYBCACgGjqjoGvbEouQZw/ToiBg9W98AlM2QHV+iNHsEs7kxWhKMjrioyspZKOB ycWxw3ie3j9uvg9EOB3aN4xiTv4qbnGiTr3oJhkB1gsb6ToJQZ8uxGq2kaV2KL9650I1SJve dYm8Of8Zd621lSmoKOwlNClALZNew72NjJLEzTalU1OdT7/i1TXkH09XSSI8mEQ/ouNcMvIJ NwQpd369y9bfIhWUiVXEK7MlRgUG6MvIj6Y3Am/BBLUVbDa4+gmzDC9ezlZkTZG2t14zWPvx XP3FAp2pkW0xqG7/377qptDmrk42GlSKN4z76ELnLxussxc7I2hx18NUcbP8+uty4bMxABEB AAHNHkp1ZXJnZW4gR3Jvc3MgPGpncm9zc0BzdXNlLmRlPsLAeQQTAQIAIwUCU4xw6wIbAwcL CQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJELDendYovxMvi4UH/Ri+OXlObzqMANruTd4N zmVBAZgx1VW6jLc8JZjQuJPSsd/a+bNr3BZeLV6lu4Pf1Yl2Log129EX1KWYiFFvPbIiq5M5 kOXTO8Eas4CaScCvAZ9jCMQCgK3pFqYgirwTgfwnPtxFxO/F3ZcS8jovza5khkSKL9JGq8Nk czDTruQ/oy0WUHdUr9uwEfiD9yPFOGqp4S6cISuzBMvaAiC5YGdUGXuPZKXLpnGSjkZswUzY d9BVSitRL5ldsQCg6GhDoEAeIhUC4SQnT9SOWkoDOSFRXZ+7+WIBGLiWMd+yKDdRG5RyP/8f 3tgGiB6cyuYfPDRGsELGjUaTUq3H2xZgIPfOwE0EU4xwFgEIAMsx+gDjgzAY4H1hPVXgoLK8 B93sTQFN9oC6tsb46VpxyLPfJ3T1A6Z6MVkLoCejKTJ3K9MUsBZhxIJ0hIyvzwI6aYJsnOew cCiCN7FeKJ/oA1RSUemPGUcIJwQuZlTOiY0OcQ5PFkV5YxMUX1F/aTYXROXgTmSaw0aC1Jpo w7Ss1mg4SIP/tR88/d1+HwkJDVW1RSxC1PWzGizwRv8eauImGdpNnseneO2BNWRXTJumAWDD pYxpGSsGHXuZXTPZqOOZpsHtInFyi5KRHSFyk2Xigzvh3b9WqhbgHHHE4PUVw0I5sIQt8hJq 5nH5dPqz4ITtCL9zjiJsExHuHKN3NZsAEQEAAcLAXwQYAQIACQUCU4xwFgIbDAAKCRCw3p3W KL8TL0P4B/9YWver5uD/y/m0KScK2f3Z3mXJhME23vGBbMNlfwbr+meDMrJZ950CuWWnQ+d+ Ahe0w1X7e3wuLVODzjcReQ/v7b4JD3wwHxe+88tgB9byc0NXzlPJWBaWV01yB2/uefVKryAf AHYEd0gCRhx7eESgNBe3+YqWAQawunMlycsqKa09dBDL1PFRosF708ic9346GLHRc6Vj5SRA UTHnQqLetIOXZm3a2eQ1gpQK9MmruO86Vo93p39bS1mqnLLspVrL4rhoyhsOyh0Hd28QCzpJ wKeHTd0MAWAirmewHXWPco8p1Wg+V+5xfZzuQY0f4tQxvOpXpt4gQ1817GQ5/Ed/wsDtBBgB CAAgFiEEhRJncuj2BJSl0Jf3sN6d1ii/Ey8FAlrd8NACGwIAgQkQsN6d1ii/Ey92IAQZFggA HRYhBFMtsHpB9jjzHji4HoBcYbtP2GO+BQJa3fDQAAoJEIBcYbtP2GO+TYsA/30H/0V6cr/W V+J/FCayg6uNtm3MJLo4rE+o4sdpjjsGAQCooqffpgA+luTT13YZNV62hAnCLKXH9n3+ZAgJ RtAyDWk1B/0SMDVs1wxufMkKC3Q/1D3BYIvBlrTVKdBYXPxngcRoqV2J77lscEvkLNUGsu/z W2pf7+P3mWWlrPMJdlbax00vevyBeqtqNKjHstHatgMZ2W0CFC4hJ3YEetuRBURYPiGzuJXU pAd7a7BdsqWC4o+GTm5tnGrCyD+4gfDSpkOT53S/GNO07YkPkm/8J4OBoFfgSaCnQ1izwgJQ jIpcG2fPCI2/hxf2oqXPYbKr1v4Z1wthmoyUgGN0LPTIm+B5vdY82wI5qe9uN6UOGyTH2B3p hRQUWqCwu2sqkI3LLbTdrnyDZaixT2T0f4tyF5Lfs+Ha8xVMhIyzNb1byDI5FKCb
  • Cc: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxxx>, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
  • Delivery-date: Thu, 06 Sep 2018 08:27:41 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Openpgp: preference=signencrypt

On 06/09/18 10:10, Jan Beulich wrote:
>>>> On 06.09.18 at 09:34, <jgross@xxxxxxxx> wrote:
>> On 05/09/18 16:05, Juergen Gross wrote:
>>> I think we have a major problem in our build system regarding
>>> automatic dependencies.
>>>
>>> Starting with a new tree (after git clone or make clean) I have
>>> no dependency files (*.d2) anywhere:
>>>
>>> $ make clean
>>> $ find . -name '*.d2' | wc -l
>>> 0
>>>
>>> Doing "make" will produce only some of them in a very limited number of
>>> directories:
>>>
>>> $ make
>>> $ find . -name '*.d2' | wc -l
>>> 279
>>> $ find . -name '*.d2' | xargs -n 1 dirname | sort -u
>>> ./tools/firmware/xen-dir/xen-root/xen/arch/x86
>>> ./tools/firmware/xen-dir/xen-root/xen/arch/x86/x86_64
>>> ./tools/firmware/xen-dir/xen-root/xen/common
>>> ./tools/firmware/xen-dir/xen-root/xen/common/compat
>>> ./xen/arch/x86
>>> ./xen/arch/x86/x86_64
>>> ./xen/common
>>> ./xen/common/compat
>>>
>>> And only after the next "make" we have all of the *.d2 files available:
>>>
>>> $ make
>>> $ find . -name '*.d2' | wc -l
>>> 969
>>>
>>> I guess the reason for that is that the *.d2 files only depend on the
>>> *.d files which are built together with the *.o files. They are not
>>> needed for building the product files, so they seem to be always one
>>> make step behind.
>>>
>>> Now comes the weird part (that's why I started to look into this):
>>> When I now do:
>>>
>>> $ touch tools/libxc/include/xenctrl.h
>>> $ make
>>>
>>> I get build failures in tools/tests/depriv (lots of unknown types like
>>> uint64_t). The .d file in the depriv directory suddenly contains only
>>> 7 instead of 44 entries, while the .d2 file is still okay (it is one
>>> make "older", do you remember?):
>>>
>>> $ wc .*.d*
>>>    7   14  546 .depriv-fd-checker.d
>>>   44   88 1923 .depriv-fd-checker.d2
>>>
>>> And doing another make will succeed again, but now both dependency files
>>> are cut down to 7 entries (sure they are: .d2 is rebuilt from .d, while
>>> .d would only be rebuilt if .o is being built, which isn't happening as
>>> no file it depends on has changed):
>>>
>>> $ make
>>> $ wc .*.d*
>>>   7  14 546 .depriv-fd-checker.d
>>>   7  14 336 .depriv-fd-checker.d2
>>>
>>> I have no immediate idea how to solve that.
>>
>> I've setup a little example Makefile solving the problem (just to show
>> the correct dependencies, needs to be adapted for naming the .d and .d2
>> files and how to build the .d2):
>>
>> -->8 snip here 8<--
>>
>> DEPS := tst.d2
>>
>> all: tst $(DEPS)
> 
> -include $(DEPS) already ought to have the effect of such a dependency,
> since all makefiles are checked for rules of how to re-make them.

Obviously this isn't the case. Otherwise there would be .d2 files more
common after doing a make.

> 
>> %.d2: %.d
>>         cp $< $@
> 
> Such a rule already exists in ./Config.mk.

Right, that was just needed in my little test Makefile.

> 
>> %.o %.d: %.c
>>         gcc -MMD -o $(patsubst %.c,%.o,$<) -c $<
> 
> Doesn't this result in gcc to be invoked twice, perhaps resulting in
> corrupt .o and/or .d? I think %.d wants to depend on %.o, without
> a command.

No, that's perfectly fine. make will invoke the command only once, its
just not clear for which target (that's the reason I need to use the
$(patsubst %.c,%.o,$<) instead of $@, which might be the .o _or_ the .d
file).

From the make docs:

  Pattern rules may have more than one target. Unlike normal rules, this
  does not act as many different rules with the same prerequisites and
  recipe. If a pattern rule has multiple targets, make knows that the
  rule’s recipe is responsible for making all of the targets. The recipe
  is executed only once to make all the targets.

> 
>> %: %.o
>>         gcc $< -o $@
>>
>> -include $(DEPS)
>>
>> -->8 snip here 8<--
>>
>> So the basic ideas are:
>>
>> - add a rule for constructing the .d files
>> - let the build depend on the .d2 files
> 
> IOW I wonder whether this really is any different from what we
> do now (minus bugs/quirks in make itself, of course). And from this
> as well as your original mail I still don't understand what's actually
> broken with the current approach.

The main problem is that the .d2 files used for determining which object
files need to be (re-)built are based on the build before the last one.
I'm not sure this is always the case, but at least when starting with a
clean tree I need two invocations of "make" to get all .d2 files built.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.