{"thread":{"id":"57604","subject":"[Question] .git folder file updates for changing head commit","startedAt":"2022-03-23T15:19:12Z","lastAt":"2022-03-30T10:49:07Z","messageCount":9,"participants":["John Garry","Taylor Blau","Glen Choo","Andreas Schwab","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"451996","messageId":"6fa76f28-063b-8964-c0a2-642dae88f1b3@huawei.com","threadId":"57604","inReplyTo":null,"subject":"[Question] .git folder file updates for changing head commit","fromName":"John Garry","fromEmail":"john.garry@huawei.com","sentAt":"2022-03-23T15:19:06Z","receivedAt":"2022-03-23T15:19:12Z","isPatch":false,"sender":{"key":"john.garry@huawei.com","avatar":null},"body":"Hi guys,\n\nI have a question about which files in the .git folder are updated as we \nchange the head commit. I could check the codebase myself but prob will \nmake a mistake and maybe some expert would be so kind as to just kindly \ntell me.\n\nFor building the linux perf tool we use the git head commit id as part \nof the tool version sting. To save time in re-building, the Makefile \nrule has a dependency on .git/HEAD for rebuilding. An alternative \napproach would be to compare git log output to check current versus \nprevious build head commit, but that is seen as inefficient time-wise.\n\nThe problem is that actions like git cherry-pick and git reset --hard \nHEAD^ may not update .git/HEAD (so don't trigger a rebuild).\n\nIs there some more suitable file(s) which we could use as a dependency? \n From my limited experimentation, .git/index seems to always update when \nthe changing head commit.\n\nThanks,\njohn\n"},{"id":"452028","messageId":"YjuCSpAHD6xHkb5G@nand.local","threadId":"57604","inReplyTo":"6fa76f28-063b-8964-c0a2-642dae88f1b3@huawei.com","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-03-23T20:25:46Z","receivedAt":"2022-03-23T20:25:56Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:\n> For building the linux perf tool we use the git head commit id as part of\n> the tool version sting. To save time in re-building, the Makefile rule has a\n> dependency on .git/HEAD for rebuilding. An alternative approach would be to\n> compare git log output to check current versus previous build head commit,\n> but that is seen as inefficient time-wise.\n\nHaving a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to\nme.\n\nPresumably your Makefile rules would map out which parts of your program\ndepend on each other, and would get invalidated when the source itself\nchanges, no?\n\nPerhaps you also care about the commit you're building from in order to\nembed something into your program. But it seems like you could inject\nthe output of \"git rev-parse HEAD\" when you construct the version\nidentifier whenever you do need to rebuild.\n\nThanks,\nTaylor\n"},{"id":"452062","messageId":"kl6la6dgtmv8.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57604","inReplyTo":"6fa76f28-063b-8964-c0a2-642dae88f1b3@huawei.com","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-23T20:50:03Z","receivedAt":"2022-03-23T20:50:17Z","isPatch":false,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Hi John, I'm not a big expert on Make (or even Git :p) but think I can\nanswer some of your questions.\n\nJohn Garry <john.garry@huawei.com> writes:\n\n> Hi guys,\n>\n> I have a question about which files in the .git folder are updated as we \n> change the head commit. I could check the codebase myself but prob will \n> make a mistake and maybe some expert would be so kind as to just kindly \n> tell me.\n>\n> For building the linux perf tool we use the git head commit id as part \n> of the tool version sting. To save time in re-building, the Makefile \n> rule has a dependency on .git/HEAD for rebuilding. An alternative \n> approach would be to compare git log output to check current versus \n> previous build head commit, but that is seen as inefficient time-wise.\n\nYou're on the right track because if .git/HEAD doesn't change, the HEAD\nis still pointing to either the same commit (if in detached HEAD) or\nbranch (if not in detached HEAD).\n\nThe problem is that when HEAD points to a branch, the branch's 'head\ncommit' can change, implicitly changing the commit that .git/HEAD\nactually points to.\n\n> The problem is that actions like git cherry-pick and git reset --hard \n> HEAD^ may not update .git/HEAD (so don't trigger a rebuild).\n\nThat's what's happening here. If your HEAD is pointing to\n\"refs/heads/main\", \"cherry-pick\" and \"reset --hard\" will change where\n\"refs/heads/main\" points, but HEAD still points to \"refs/heads/main\".\n\nBranches are often located in \"loose form\" under .git/refs/heads/*,\n*but* refs can also be stored in a \"packed form\" under .git/packed-refs\n(https://git-scm.com/docs/git-pack-refs). There is also a relatively\nnew ref storage format called reftable that I'm not familiar with, but\npresumably stores its refs in other locations too.\n\nAll of this is to say that depending on specific files under .git/\nexposes you to a lot of Git internals that really aren't meant for\nexternal consumption. I suggest finding a different approach than\nwatching Git-internal files for changes.\n\n> Is there some more suitable file(s) which we could use as a dependency? \n>  From my limited experimentation, .git/index seems to always update when \n> the changing head commit.\n\nI think users typically expect the index (and therefore .git/index) to\nchange when the head commit changes, but this isn't always true e.g.\n\"reset --soft\" will move the branch without changing the index.\n\nP.S. Even if you knew the value of .git/index or .git/HEAD, that\nwouldn't tell you whether or not the files in your working directory\nhave changed, so I'm not sure if you want to use either as Makefile\ndependency.\n\n>\n> Thanks,\n> john\n"},{"id":"452072","messageId":"87wngkpddp.fsf@igel.home","threadId":"57604","inReplyTo":"6fa76f28-063b-8964-c0a2-642dae88f1b3@huawei.com","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2022-03-23T21:28:34Z","receivedAt":"2022-03-23T21:38:13Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Mär 23 2022, John Garry wrote:\n\n> For building the linux perf tool we use the git head commit id as part of\n> the tool version sting. To save time in re-building, the Makefile rule has\n> a dependency on .git/HEAD for rebuilding.\n\nThere is no guarantee that .git is a directory.  In a separate worktree\n.git is actually a gitfile.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"452110","messageId":"f80bac33-7864-8f31-d81f-78cf7cb2987d@huawei.com","threadId":"57604","inReplyTo":"YjuCSpAHD6xHkb5G@nand.local","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"John Garry","fromEmail":"john.garry@huawei.com","sentAt":"2022-03-24T12:56:32Z","receivedAt":"2022-03-24T12:56:16Z","isPatch":false,"sender":{"key":"john.garry@huawei.com","avatar":null},"body":"On 23/03/2022 20:25, Taylor Blau wrote:\n> On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:\n>> For building the linux perf tool we use the git head commit id as part of\n>> the tool version sting. To save time in re-building, the Makefile rule has a\n>> dependency on .git/HEAD for rebuilding. An alternative approach would be to\n>> compare git log output to check current versus previous build head commit,\n>> but that is seen as inefficient time-wise.\n> Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to\n> me.\n> \n\nYeah, as mentioned, the idea was the we can check a rebuild on this \nrather than running a relatively time-consuming command like \"git \ndescribe\". Wasn't my idea :)\n\n> Presumably your Makefile rules would map out which parts of your program\n> depend on each other, and would get invalidated when the source itself\n> changes, no?\n\nCorrect, they do. It's just the part which constructs the app version \ntag which depends on this.\n\n> \n> Perhaps you also care about the commit you're building from in order to\n> embed something into your program. But it seems like you could inject\n> the output of \"git rev-parse HEAD\" when you construct the version\n> identifier whenever you do need to rebuild.\n> \n\nok, so I think that we may need to run a git command always one way or \nanother.\n\nThanks,\nJohn\n\n"},{"id":"452111","messageId":"5ceff005-bbe4-37f5-c453-17008c2167bc@huawei.com","threadId":"57604","inReplyTo":"kl6la6dgtmv8.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"John Garry","fromEmail":"john.garry@huawei.com","sentAt":"2022-03-24T12:59:45Z","receivedAt":"2022-03-24T12:59:29Z","isPatch":false,"sender":{"key":"john.garry@huawei.com","avatar":null},"body":"On 23/03/2022 20:50, Glen Choo wrote:\n> Hi John, I'm not a big expert on Make (or even Git :p) but think I can\n> answer some of your questions.\n> \n> John Garry <john.garry@huawei.com> writes:\n> \n>> Hi guys,\n>>\n>> I have a question about which files in the .git folder are updated as we\n>> change the head commit. I could check the codebase myself but prob will\n>> make a mistake and maybe some expert would be so kind as to just kindly\n>> tell me.\n>>\n>> For building the linux perf tool we use the git head commit id as part\n>> of the tool version sting. To save time in re-building, the Makefile\n>> rule has a dependency on .git/HEAD for rebuilding. An alternative\n>> approach would be to compare git log output to check current versus\n>> previous build head commit, but that is seen as inefficient time-wise.\n> \n> You're on the right track because if .git/HEAD doesn't change, the HEAD\n> is still pointing to either the same commit (if in detached HEAD) or\n> branch (if not in detached HEAD).\n> \n> The problem is that when HEAD points to a branch, the branch's 'head\n> commit' can change, implicitly changing the commit that .git/HEAD\n> actually points to.\n> \n\nok\n\n>> The problem is that actions like git cherry-pick and git reset --hard\n>> HEAD^ may not update .git/HEAD (so don't trigger a rebuild).\n> \n> That's what's happening here. If your HEAD is pointing to\n> \"refs/heads/main\", \"cherry-pick\" and \"reset --hard\" will change where\n> \"refs/heads/main\" points, but HEAD still points to \"refs/heads/main\".\n> \n> Branches are often located in \"loose form\" under .git/refs/heads/*,\n> *but* refs can also be stored in a \"packed form\" under .git/packed-refs\n> (https://git-scm.com/docs/git-pack-refs). There is also a relatively\n> new ref storage format called reftable that I'm not familiar with, but\n> presumably stores its refs in other locations too.\n> \n> All of this is to say that depending on specific files under .git/\n> exposes you to a lot of Git internals that really aren't meant for\n> external consumption. I suggest finding a different approach than\n> watching Git-internal files for changes.\n\nThanks for the detailed explanation.\n\nI agree with your suggestion to find a new method (which may be \nreverting to the old method to just run proper git commands).\n\n> \n>> Is there some more suitable file(s) which we could use as a dependency?\n>>   From my limited experimentation, .git/index seems to always update when\n>> the changing head commit.\n> \n> I think users typically expect the index (and therefore .git/index) to\n> change when the head commit changes, but this isn't always true e.g.\n> \"reset --soft\" will move the branch without changing the index.\n\nok, good to know. I was going to try that...\n\n> \n> P.S. Even if you knew the value of .git/index or .git/HEAD, that\n> wouldn't tell you whether or not the files in your working directory\n> have changed, so I'm not sure if you want to use either as Makefile\n> dependency.\n> \n\nok, thanks again for all this info,\nJohn\n\n"},{"id":"452112","messageId":"da0bba94-8dc2-a5c0-98f9-42182345601c@huawei.com","threadId":"57604","inReplyTo":"87wngkpddp.fsf@igel.home","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"John Garry","fromEmail":"john.garry@huawei.com","sentAt":"2022-03-24T13:00:19Z","receivedAt":"2022-03-24T13:00:04Z","isPatch":false,"sender":{"key":"john.garry@huawei.com","avatar":null},"body":"On 23/03/2022 21:28, Andreas Schwab wrote:\n> On Mär 23 2022, John Garry wrote:\n> \n>> For building the linux perf tool we use the git head commit id as part of\n>> the tool version sting. To save time in re-building, the Makefile rule has\n>> a dependency on .git/HEAD for rebuilding.\n> There is no guarantee that .git is a directory.  In a separate worktree\n> .git is actually a gitfile.\n\nok, thanks. I'll keep that in mind.\n\nJohn\n"},{"id":"452437","messageId":"220327.86tubje6bn.gmgdl@evledraar.gmail.com","threadId":"57604","inReplyTo":"YjuCSpAHD6xHkb5G@nand.local","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-27T15:48:24Z","receivedAt":"2022-03-27T15:57:54Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 23 2022, Taylor Blau wrote:\n\n> On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:\n>> For building the linux perf tool we use the git head commit id as part of\n>> the tool version sting. To save time in re-building, the Makefile rule has a\n>> dependency on .git/HEAD for rebuilding. An alternative approach would be to\n>> compare git log output to check current versus previous build head commit,\n>> but that is seen as inefficient time-wise.\n>\n> Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to\n> me.\n>\n> Presumably your Makefile rules would map out which parts of your program\n> depend on each other, and would get invalidated when the source itself\n> changes, no?\n>\n> Perhaps you also care about the commit you're building from in order to\n> embed something into your program. But it seems like you could inject\n> the output of \"git rev-parse HEAD\" when you construct the version\n> identifier whenever you do need to rebuild.\n\nOur very own build process for git.git relies on this, see how version.o\nneeds a GIT-VERSION-FILE, which we generate by shelling out and\nincluding.\n\nIt is unfortunate that we don't have an advertised way to do this, with\nthe ref backend I think trickery using $(file) to read the HEAD will\nwork to do it in pure-make, i.e. you'd need to parse it to see if it's a\nsymref, then depend on the target it points to.\n\nOr you could recursively depend on a glob of the whole refspace for\ngenerating the version file ...\n\nIt would be nice if we had a way to guarantee that we'd write some file\non HEAD updates, AFAIK not even the new reference-transaction hook will\ndo that (due to \"git reset --hard\" and friends).\n\nAnd yes, this does actually matter. There's a huge performance\ndifference between a Makefile that needs to shell out for every little\nthing.\n\nI've been optimizing our own Makefile incrementally from running in\n~500ms down to 100-200ms for noop runs over over the last few\nmonths. It's possible to get it down to 10-20ms at least by getting rid\nof the remaining shell-outs.\n\nThat makes a big difference when e.g. using \"make\" with \"git rebase -i\n-x\".\n"},{"id":"452660","messageId":"7f23ed59-c237-dce1-53a5-43f35b92c972@huawei.com","threadId":"57604","inReplyTo":"220327.86tubje6bn.gmgdl@evledraar.gmail.com","subject":"Re: [Question] .git folder file updates for changing head commit","fromName":"John Garry","fromEmail":"john.garry@huawei.com","sentAt":"2022-03-30T10:48:56Z","receivedAt":"2022-03-30T10:49:07Z","isPatch":false,"sender":{"key":"john.garry@huawei.com","avatar":null},"body":"On 27/03/2022 16:48, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Mar 23 2022, Taylor Blau wrote:\n> \n>> On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:\n>>> For building the linux perf tool we use the git head commit id as part of\n>>> the tool version sting. To save time in re-building, the Makefile rule has a\n>>> dependency on .git/HEAD for rebuilding. An alternative approach would be to\n>>> compare git log output to check current versus previous build head commit,\n>>> but that is seen as inefficient time-wise.\n>> Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to\n>> me.\n>>\n>> Presumably your Makefile rules would map out which parts of your program\n>> depend on each other, and would get invalidated when the source itself\n>> changes, no?\n>>\n>> Perhaps you also care about the commit you're building from in order to\n>> embed something into your program. But it seems like you could inject\n>> the output of \"git rev-parse HEAD\" when you construct the version\n>> identifier whenever you do need to rebuild.\n> Our very own build process for git.git relies on this, see how version.o\n> needs a GIT-VERSION-FILE, which we generate by shelling out and\n> including.\n\nI see.\n\n> \n> It is unfortunate that we don't have an advertised way to do this, with\n> the ref backend I think trickery using $(file) to read the HEAD will\n> work to do it in pure-make, i.e. you'd need to parse it to see if it's a\n> symref, then depend on the target it points to.\n> \n> Or you could recursively depend on a glob of the whole refspace for\n> generating the version file ...\n> \n> It would be nice if we had a way to guarantee that we'd write some file\n> on HEAD updates, AFAIK not even the new reference-transaction hook will\n> do that (due to \"git reset --hard\" and friends).\n> \n> And yes, this does actually matter. There's a huge performance\n> difference between a Makefile that needs to shell out for every little\n> thing.\n> \n\nI'm not sure if I mentioned it before, but the main motivation for \nadding the dependency was that \"git describe\" may be slow. We have since \nmoved away from that (using \"describe\", which I hope turns out to be \nok), but your GIT-VERSION-GEN still uses it. I just found that \"git \ndescribe\" is unreliable for us as it depends on tags being pushed to a \nclone git.\n\n> I've been optimizing our own Makefile incrementally from running in\n> ~500ms down to 100-200ms for noop runs over over the last few\n> months. It's possible to get it down to 10-20ms at least by getting rid\n> of the remaining shell-outs.\n> \n> That makes a big difference when e.g. using \"make\" with \"git rebase -i\n> -x\".\n> .\n\nok, thanks,\nJohn\n"}]}