{"thread":{"id":"54579","subject":"[RFC PATCH 0/8] speed up 'make clean'","startedAt":"2020-11-05T21:01:16Z","lastAt":"2020-11-06T01:26:55Z","messageCount":3,"participants":["Ramsay Jones","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"409220","messageId":"e1d218bb-7658-565e-0931-2411efbb561c@ramsayjones.plus.com","threadId":"54579","inReplyTo":null,"subject":"[RFC PATCH 0/8] speed up 'make clean'","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2020-11-05T21:01:08Z","receivedAt":"2020-11-05T21:01:16Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\nApproximately five years ago, I stopped building the documentation\nlocally and instead started using the 'pre-built' documentation repos\nfrom git://git.kernel.org/pub/scm/git/git-{htmldocs,manpages}.git.\nI placed them next to the git repository (ie. as siblings) and wrote\na script (mk-dist-doc.sh) to reproduce the effect of 'make dist-doc',\nusing git-archive, to place the distribution tarballs at the top of\nthe git repo worktree.\n\n[During the creation of these patches, I actually did a 'make' in\nthe Documentation directory and was surprised that it succeeded!\nHowever, it reminded me why I stopped building the docs! :-P ]\n\nThis meant that the Documentation directory was essentially unused\nfor build purposes, so that a 'make clean' should have nothing to\ndo in that directory. However, on cygwin, I noticed a long pause\nduring a 'make clean' in that directory and, further, it seemed to\nbe GEN-erating some files! I put an item on my TODO list to take a\nlook at that - during the v2.28.0 cycle, I finally found time to look\nat this issue! :-D\n\nThis patch series is the result. Note that this was mainly about\nspeeding up 'make clean' on cygwin (and the performance numbers\nin the patches reflect use on cygwin). Tonight, I did some timings\non Linux as well, so that I could show this table:\n\n CYGWIN LINUX\nv2.29.0 23.339s 1.709s\nPatch #1 12.364s -47.02% 0.877s -48.68%\nPatch #2 10.361s -16.20% 0.805s -8.21%\nPatch #3 8.430s -18.64% 0.672s -16.52%\nPatch #4 6.454s -23.44% 0.484s -27.98%\nPatch #5 6.428s -0.40% 0.503s +3.93%\nPatch #6 6.440s +0.19% 0.517s +2.78%\nPatch #7 6.430s -0.16% 0.515s -0.39%\nPatch #8 4.064s -36.80% 0.322s -37.48%\n\n 23.339/4.064 = 5.74 1.709/0.322 = 5.31 \n\nNote that patches 5-7 are just preparatory patches for the final\npatch, and are not expected to show any improvement.\n\nThis series is marked RFC because I forgot that 'git-gui' patches\nneed to be separately submitted to Pratyush Yadav, the git-gui\nmaintainer. So, I need to drop patch #4 from this series and\nsubmit that to Pratyush (the commit message needs to be re-written\nas well). Unfortunately, this means that the following commit\nmessages will need to be re-written (and timings re-done).\n\nAlso, since this series is based on v2.29.0, I had anticipated some\nconflicts with the 'rs/dist-doc-with-git-archive' branch when merging\nthis with the 'master', 'next' and 'seen' branches. I have just done\na trial merge with each branch and, to my surprise, there were no\nconflicts and they (auto) merged clean. ;)\n\nHowever, I can certainly rebase these onto the 'master' branch, if that\nwould be easier to deal with. (There has to be a v2 anyway, so ...)\n\nI should note here that the main idea used in these patches, to use\nthe $(MAKECMDGOALS) variable to conditionally include some files, can\nbe defeated by invoking make with multiple goals. For example, if you\nwere to do 'make clean all', then $(MAKECMDGOALS) would be 'clean all'\nand not 'clean'.\n\nHowever, doing so with the top-level Makefile would only negate the\neffect of the last patch, since all of the sub-makes are issued as\n'make -C clean'. So, all of the conditional includes in patches\n#1-4 will work as intended (and show the noted improvement).\n\n[Yes, 'cd Documentation; make clean all' will be slower that a doing\nseparate 'make clean; make', but the extra 10s, or so, will be swamped\nby the documentation build time! ;-) ]\n\nATB,\nRamsay Jones\n\nRamsay Jones (8):\n Documentation/Makefile: conditionally include doc.dep\n Documentation/Makefile: conditionally include ../GIT-VERSION-FILE\n gitweb/Makefile: conditionally include ../GIT-VERSION-FILE\n git-gui/Makefile: conditionally include GIT-VERSION-FILE\n Makefile: don't try to clean old debian build product\n Makefile: don't use a versioned temp distribution directory\n Makefile: don't delete dist tarballs directly by name\n Makefile: conditionally include GIT-VERSION-FILE\n\n .gitignore | 1 +\n Documentation/Makefile | 4 ++++\n Makefile | 28 +++++++++++++++++++---------\n git-gui/Makefile | 2 ++\n gitweb/Makefile | 2 ++\n 5 files changed, 28 insertions(+), 9 deletions(-)\n\n-- \n2.29.0\n"},{"id":"409235","messageId":"xmqq361niifm.fsf@gitster.c.googlers.com","threadId":"54579","inReplyTo":"e1d218bb-7658-565e-0931-2411efbb561c@ramsayjones.plus.com","subject":"Re: [RFC PATCH 0/8] speed up 'make clean'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-11-05T21:48:29Z","receivedAt":"2020-11-05T21:48:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramsay Jones writes:\n\n> [Yes, 'cd Documentation; make clean all' will be slower that a doing\n> separate 'make clean; make', but the extra 10s, or so, will be swamped\n> by the documentation build time! ;-) ]\n\nHmph, the \"all\" part in \"make clean all\" needs the information we\nread from these generated files, and time must be spent to generate\nthem whether \"make clean all\" or \"make clean; make all\" is used. In\nthe latter, we may not generate and read them in the first phase,\nbut the second one \"make all\" would need to do so anyway. So I am\npuzzled why \"make clean all\" needs to be slower---don't we generate\nand read them only once in either case?\n\n"},{"id":"409258","messageId":"f107d6be-0924-88b0-bd34-582b1eed172b@ramsayjones.plus.com","threadId":"54579","inReplyTo":"xmqq361niifm.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH 0/8] speed up 'make clean'","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2020-11-06T01:26:49Z","receivedAt":"2020-11-06T01:26:55Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 05/11/2020 21:48, Junio C Hamano wrote:\n> Ramsay Jones writes:\n> \n>> [Yes, 'cd Documentation; make clean all' will be slower that a doing\n>> separate 'make clean; make', but the extra 10s, or so, will be swamped\n>> by the documentation build time! ;-) ]\n> \n> Hmph, the \"all\" part in \"make clean all\" needs the information we\n> read from these generated files, and time must be spent to generate\n> them whether \"make clean all\" or \"make clean; make all\" is used. In\n> the latter, we may not generate and read them in the first phase,\n> but the second one \"make all\" would need to do so anyway. So I am\n> puzzled why \"make clean all\" needs to be slower---don't we generate\n> and read them only once in either case?\n> \n\nHmm, interesting. I was all ready to explain the results of the\nmoc-up of this that I did about a month ago, but thought I should\njust check again ... ;-)\n\nThis time I used the actual Makefile, rather than a moc-up, and got\ndifferent (in some sense, worse) results. What I was going to say\nwas, no the doc.dep file gets generated twice - but that is not true. :(\n\nHowever, if you run 'make clean all', you will not be pleased with\nthe results!\n\n $ make clean\n ...\n $ make clean all >zzz 2>&1\n $ grep WARNING zzz | wc -l\n 26\n $ tail -1 zzz\n make: *** [Makefile:362: git.1] Error 13\n $ \n\nIf you look at the output, you will see that, while processing the\n'clean' target, the 'doc.dep' file is created, -included, and then\nimmediately deleted. While processing the 'all' target I had expected\nthe 'doc.dep' file to be recreated - but it isn't. It seems to have\ndone the 'drop the internal data, re-read and re-parse' only the\nonce, and on the second go round (because it has already generated\nit once) does not re-create and re-read the dependency data again.\nThus, the 'all' target is executed without any 'doc.dep' data and\nfalls over in a heap. :(\n\n[BTW, just doing 'make' does not do a full build. I will look at why\nthat is later].\n\nDoing 'make clean; make all' works fine, of course.\n\nThat will teach me to cut corners! Ho-Hum. Sorry about that.\n\nThanks.\n\nATB,\nRamsay Jones\n"}]}