{"thread":{"id":"58725","subject":"Consist timestamps within a checkout/clone","startedAt":"2022-10-31T19:24:50Z","lastAt":"2022-11-03T13:46:37Z","messageCount":17,"participants":["Mark Hills","Ævar Arnfjörð Bjarmason","Andreas Schwab","Taylor Blau","rsbecker@nexbridge.com","Marc Branchaud","Erik Cervin Edin","Matheus Tavares"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"466107","messageId":"2210311614160.25661@stax.localdomain","threadId":"58725","inReplyTo":null,"subject":"Consist timestamps within a checkout/clone","fromName":"Mark Hills","fromEmail":"mark@xwax.org","sentAt":"2022-10-31T19:01:20Z","receivedAt":"2022-10-31T19:24:50Z","isPatch":false,"sender":{"key":"mark@xwax.org","avatar":null},"body":"Our use case: we commit some compiled objects to the repo, where compiling \nis either slow or requires software which is not always available.\n\nSince upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) \nwe are noticing a change in build behaviour.\n\nNow, after a \"git clone\" we find the Makefile intermittently attempting \n(and failing) some builds that are not intended.\n\nIndeed, Make is acting reasonably as the source file is sometimes \nmarginally newer than the destination (both checked out by Git), example \nbelow.\n\nI've never had to consider consistency timestamps within a Git checkout \nuntil now.\n\nIt's entirely possible there's _never_ a guarantee of consistency here.\n\nBut then something has certainly changed in practice, as this fault has \ngone from never happening to now every couple of days.\n\nImaginging I can't be the first person to encounter this, I searched for \nexisting threads or docs, but overwhemingly the results were question of \nGit tracking the timestamps (as part of the commit) which this is not; \nit's consistency within one checkout.\n\n$ git clone --depth 1 file:///path/to/repo.git\n\n$ stat winner.jpeg\n  File: winner.jpeg\n  Size: 258243          Blocks: 520        IO Block: 4096   regular file\nDevice: fd07h/64775d    Inode: 33696       Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\nAccess: 2022-10-31 16:05:17.756858496 +0000\nModify: 2022-10-31 16:05:17.756858496 +0000\nChange: 2022-10-31 16:05:17.756858496 +0000\n Birth: -\n\n$ stat winner.svg\n  File: winner.svg\n  Size: 52685           Blocks: 112        IO Block: 4096   regular file\nDevice: fd07h/64775d    Inode: 33697       Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\nAccess: 2022-10-31 16:05:17.766859030 +0000\nModify: 2022-10-31 16:05:17.766859030 +0000\nChange: 2022-10-31 16:05:17.766859030 +0000\n Birth: -\n\nElsewhere in the repository, it's clear the timestamps are not consistent:\n\n$ stat Makefile\n  File: Makefile\n  Size: 8369            Blocks: 24         IO Block: 4096   regular file\nDevice: fd07h/64775d    Inode: 33655       Links: 1\nAccess: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\nAccess: 2022-10-31 16:05:51.628660212 +0000\nModify: 2022-10-31 16:05:17.746857963 +0000\nChange: 2022-10-31 16:05:17.746857963 +0000\n Birth: -\n\n-- \nMark\n"},{"id":"466114","messageId":"221031.86zgdb68p3.gmgdl@evledraar.gmail.com","threadId":"58725","inReplyTo":"2210311614160.25661@stax.localdomain","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-31T20:21:20Z","receivedAt":"2022-10-31T20:26:39Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 31 2022, Mark Hills wrote:\n\n> Our use case: we commit some compiled objects to the repo, where compiling \n> is either slow or requires software which is not always available.\n>\n> Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) \n> we are noticing a change in build behaviour.\n>\n> Now, after a \"git clone\" we find the Makefile intermittently attempting \n> (and failing) some builds that are not intended.\n>\n> Indeed, Make is acting reasonably as the source file is sometimes \n> marginally newer than the destination (both checked out by Git), example \n> below.\n>\n> I've never had to consider consistency timestamps within a Git checkout \n> until now.\n>\n> It's entirely possible there's _never_ a guarantee of consistency here.\n>\n> But then something has certainly changed in practice, as this fault has \n> gone from never happening to now every couple of days.\n>\n> Imaginging I can't be the first person to encounter this, I searched for \n> existing threads or docs, but overwhemingly the results were question of \n> Git tracking the timestamps (as part of the commit) which this is not; \n> it's consistency within one checkout.\n>\n> $ git clone --depth 1 file:///path/to/repo.git\n>\n> $ stat winner.jpeg\n>   File: winner.jpeg\n>   Size: 258243          Blocks: 520        IO Block: 4096   regular file\n> Device: fd07h/64775d    Inode: 33696       Links: 1\n> Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> Access: 2022-10-31 16:05:17.756858496 +0000\n> Modify: 2022-10-31 16:05:17.756858496 +0000\n> Change: 2022-10-31 16:05:17.756858496 +0000\n>  Birth: -\n>\n> $ stat winner.svg\n>   File: winner.svg\n>   Size: 52685           Blocks: 112        IO Block: 4096   regular file\n> Device: fd07h/64775d    Inode: 33697       Links: 1\n> Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> Access: 2022-10-31 16:05:17.766859030 +0000\n> Modify: 2022-10-31 16:05:17.766859030 +0000\n> Change: 2022-10-31 16:05:17.766859030 +0000\n>  Birth: -\n>\n> Elsewhere in the repository, it's clear the timestamps are not consistent:\n>\n> $ stat Makefile\n>   File: Makefile\n>   Size: 8369            Blocks: 24         IO Block: 4096   regular file\n> Device: fd07h/64775d    Inode: 33655       Links: 1\n> Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> Access: 2022-10-31 16:05:51.628660212 +0000\n> Modify: 2022-10-31 16:05:17.746857963 +0000\n> Change: 2022-10-31 16:05:17.746857963 +0000\n>  Birth: -\n\nI think you're almost certainly running into the parallel checkout,\nwhich is new in that revision range. Try tweaking checkout.workers and\ncheckout.thresholdForParallelism (see \"man git-config\").\n\nI can't say without looking at the code/Makefile (and even then, I don't\nhave time to dig here:), but if I had to bet I'd say that your\ndependencies have probably always been broken with these checked-in\nfiles, but they happend to work out if they were checked out in sorted\norder.\n\nAnd now with the parallel checkout they're not guaranteed to do that, as\nsome workers will \"race ahead\" and finish in an unpredictable order.\n\nBut that's all just a guess, perhaps it has nothing to do with parallel\ncheckout, such dependency issues are sensitive to all sorts of other\nthings, e.g. maybe git got slightly faster (or slower), so now files\nthat were always on different seconds (or the same) aren't in the state\nthey were in before...\n"},{"id":"466115","messageId":"878rkvychn.fsf@igel.home","threadId":"58725","inReplyTo":"2210311614160.25661@stax.localdomain","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2022-10-31T20:17:08Z","receivedAt":"2022-10-31T20:26:53Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Okt 31 2022, Mark Hills wrote:\n\n> It's entirely possible there's _never_ a guarantee of consistency here.\n\nI don't think the order in which git writes the individual files is\ndefined in any way.  Thus depending on the precision of the time stamps\nin the file system whether a file ends up newer than another one may\nvary each time due to timing differences.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"466116","messageId":"Y2Ax5XOgSOOcgo8J@nand.local","threadId":"58725","inReplyTo":"221031.86zgdb68p3.gmgdl@evledraar.gmail.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-10-31T20:36:53Z","receivedAt":"2022-10-31T20:37:00Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> I think you're almost certainly running into the parallel checkout,\n> which is new in that revision range. Try tweaking checkout.workers and\n> checkout.thresholdForParallelism (see \"man git-config\").\n>\n> I can't say without looking at the code/Makefile (and even then, I don't\n> have time to dig here:), but if I had to bet I'd say that your\n> dependencies have probably always been broken with these checked-in\n> files, but they happend to work out if they were checked out in sorted\n> order.\n>\n> And now with the parallel checkout they're not guaranteed to do that, as\n> some workers will \"race ahead\" and finish in an unpredictable order.\n\nDoesn't checkout.thresholdForParallelism only matter when\ncheckout.workers != 1?\n\nSo what you wrote seems like a reasonable explanation, but only if the\noriginal reporter set checkout.workers to imply the non-sequential\nbehavior in the first place.\n\nThat said...\n\n  - I also don't know off-hand of a place where we've defined the order\n    where Git will checkout files in the working copy. So depending on\n    that behavior isn't a safe thing to do.\n\n  - Committing build artifacts into your repository is generally\n    discouraged.\n\nSo while I'd guess that setting `checkout.workers` back to \"1\" (if it\nwasn't already) will probably restore the existing behavior, counting\non that behavior in the first place is wrong.\n\nThanks,\nTaylor\n"},{"id":"466128","messageId":"a87ebafd-c83-7a1d-d8d2-953bc9a93184@xwax.org","threadId":"58725","inReplyTo":"221031.86zgdb68p3.gmgdl@evledraar.gmail.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Mark Hills","fromEmail":"mark@xwax.org","sentAt":"2022-10-31T22:29:04Z","receivedAt":"2022-10-31T22:29:47Z","isPatch":false,"sender":{"key":"mark@xwax.org","avatar":null},"body":"On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Mon, Oct 31 2022, Mark Hills wrote:\n> \n> > Our use case: we commit some compiled objects to the repo, where compiling \n> > is either slow or requires software which is not always available.\n> >\n> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) \n> > we are noticing a change in build behaviour.\n> >\n> > Now, after a \"git clone\" we find the Makefile intermittently attempting \n> > (and failing) some builds that are not intended.\n> >\n> > Indeed, Make is acting reasonably as the source file is sometimes \n> > marginally newer than the destination (both checked out by Git), example \n> > below.\n> >\n> > I've never had to consider consistency timestamps within a Git checkout \n> > until now.\n> >\n> > It's entirely possible there's _never_ a guarantee of consistency here.\n> >\n> > But then something has certainly changed in practice, as this fault has \n> > gone from never happening to now every couple of days.\n> >\n> > Imaginging I can't be the first person to encounter this, I searched for \n> > existing threads or docs, but overwhemingly the results were question of \n> > Git tracking the timestamps (as part of the commit) which this is not; \n> > it's consistency within one checkout.\n> >\n> > $ git clone --depth 1 file:///path/to/repo.git\n> >\n> > $ stat winner.jpeg\n> >   File: winner.jpeg\n> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file\n> > Device: fd07h/64775d    Inode: 33696       Links: 1\n> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> > Access: 2022-10-31 16:05:17.756858496 +0000\n> > Modify: 2022-10-31 16:05:17.756858496 +0000\n> > Change: 2022-10-31 16:05:17.756858496 +0000\n> >  Birth: -\n> >\n> > $ stat winner.svg\n> >   File: winner.svg\n> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file\n> > Device: fd07h/64775d    Inode: 33697       Links: 1\n> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> > Access: 2022-10-31 16:05:17.766859030 +0000\n> > Modify: 2022-10-31 16:05:17.766859030 +0000\n> > Change: 2022-10-31 16:05:17.766859030 +0000\n> >  Birth: -\n> >\n> > Elsewhere in the repository, it's clear the timestamps are not consistent:\n> >\n> > $ stat Makefile\n> >   File: Makefile\n> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file\n> > Device: fd07h/64775d    Inode: 33655       Links: 1\n> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> > Access: 2022-10-31 16:05:51.628660212 +0000\n> > Modify: 2022-10-31 16:05:17.746857963 +0000\n> > Change: 2022-10-31 16:05:17.746857963 +0000\n> >  Birth: -\n> \n> I think you're almost certainly running into the parallel checkout,\n> which is new in that revision range. Try tweaking checkout.workers and\n> checkout.thresholdForParallelism (see \"man git-config\").\n\nThanks, it will be interesting to try this and I'll report back.\n \n> I can't say without looking at the code/Makefile (and even then, I don't\n> have time to dig here:), but if I had to bet I'd say that your\n> dependencies have probably always been broken with these checked-in\n> files, but they happend to work out if they were checked out in sorted\n> order.\n>\n> And now with the parallel checkout they're not guaranteed to do that, as\n> some workers will \"race ahead\" and finish in an unpredictable order.\n\nThese are very simple Makefile rules, I don't think these dependencies are \nbroken; but your theory is in good alignment with the observed behaviour.\n\nFor example, the rule from the recent case above is:\n\n  %.jpeg:         %.png\n                  convert $< $(IMFLAGS) $@\n\n  %.png:          %.svg\n                  inkscape --export-type=png --export-filename=$@ $<\n\nAs you suggest, perhaps the Git implementation previously ran checked out \nin some kind of time order then this happens to fulfil a useful behaviour.\n\nSpecificaly with build artefacts. These are likely to have been added to \nthe repo after the source file. This could have been providing some \npratical and useful tendency of ordering.\n\n> But that's all just a guess, perhaps it has nothing to do with parallel\n> checkout, such dependency issues are sensitive to all sorts of other\n> things, e.g. maybe git got slightly faster (or slower), so now files\n> that were always on different seconds (or the same) aren't in the state\n> they were in before...\n\nHopefully I'll get to some experiments to narrow this down.\n\nThanks\n\n-- \nMark"},{"id":"466129","messageId":"d4db484f-a525-f6db-1bfb-922f788dacd@xwax.org","threadId":"58725","inReplyTo":"Y2Ax5XOgSOOcgo8J@nand.local","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Mark Hills","fromEmail":"mark@xwax.org","sentAt":"2022-10-31T22:31:23Z","receivedAt":"2022-10-31T22:31:28Z","isPatch":false,"sender":{"key":"mark@xwax.org","avatar":null},"body":"On Mon, 31 Oct 2022, Taylor Blau wrote:\n\n> On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > I think you're almost certainly running into the parallel checkout,\n> > which is new in that revision range. Try tweaking checkout.workers and\n> > checkout.thresholdForParallelism (see \"man git-config\").\n> >\n> > I can't say without looking at the code/Makefile (and even then, I don't\n> > have time to dig here:), but if I had to bet I'd say that your\n> > dependencies have probably always been broken with these checked-in\n> > files, but they happend to work out if they were checked out in sorted\n> > order.\n> >\n> > And now with the parallel checkout they're not guaranteed to do that, as\n> > some workers will \"race ahead\" and finish in an unpredictable order.\n> \n> Doesn't checkout.thresholdForParallelism only matter when\n> checkout.workers != 1?\n> \n> So what you wrote seems like a reasonable explanation, but only if the\n> original reporter set checkout.workers to imply the non-sequential\n> behavior in the first place.\n> \n> That said...\n> \n>   - I also don't know off-hand of a place where we've defined the order\n>     where Git will checkout files in the working copy. So depending on\n>     that behavior isn't a safe thing to do.\n> \n>   - Committing build artifacts into your repository is generally\n>     discouraged.\n\nIf it's undefined and never implemented this is reasonable.\n\nBut \"generally\" is a caveat, so while I agree with the statement it also \nimplies there's valid cases outside of that. Ones which used to work, too.\n\nHere are some useful cases I have seen for the combination of build rule + \nchecked in file:\n\n- part of a build requires licensed software that's not always available\n\n- part of the build requires large memory that other builders generally do \n  not have available\n\n- part of the build process uses a different platform or some other system \n  requirement\n\n- to fetch data eg. from a URL, with a record of the URL/automation but \n  also a copy of the file as a record and for offline use\n\nSo it's useful, to retain repeatable automation but not always build from \nsquare one.\n\nGenerally discouraged to check in build results yes, but I've found it \nvery practical.\n \n> So while I'd guess that setting `checkout.workers` back to \"1\" (if it \n> wasn't already) will probably restore the existing behavior, counting on \n> that behavior in the first place is wrong.\n\nI think perhaps the tail is wagging the dog here, though.\n\nIt's 'wrong' because it doesn't work; but I haven't seen anything to make \nme think this is fundamentally or theoretically flawed.\n\nIf we had a transactional file system we'd reasonably expect a checkout to \nbe an atomic operation -- same timestamp on the files created in that \nstep. A discrepancy in timestamps would be considered incorrect; it would \nimply an 'order' to the checkout which, as you say, is order-less.\n\nSowhat could be the bad outcomes if Git created files stamped with the \npoint in time of the \"git checkout\"?\n\n> Thanks,\n> Taylor\n> \n> \n\n-- \nMark"},{"id":"466131","messageId":"005e01d8ed7a$020589a0$06109ce0$@nexbridge.com","threadId":"58725","inReplyTo":"d4db484f-a525-f6db-1bfb-922f788dacd@xwax.org","subject":"RE: Consist timestamps within a checkout/clone","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-10-31T22:42:02Z","receivedAt":"2022-10-31T22:42:14Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On October 31, 2022 6:31 PM, Mark Hills wrote:\n>On Mon, 31 Oct 2022, Taylor Blau wrote:\n>> On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> > I think you're almost certainly running into the parallel checkout,\n>> > which is new in that revision range. Try tweaking checkout.workers\n>> > and checkout.thresholdForParallelism (see \"man git-config\").\n>> >\n>> > I can't say without looking at the code/Makefile (and even then, I\n>> > don't have time to dig here:), but if I had to bet I'd say that your\n>> > dependencies have probably always been broken with these checked-in\n>> > files, but they happend to work out if they were checked out in\n>> > sorted order.\n>> >\n>> > And now with the parallel checkout they're not guaranteed to do\n>> > that, as some workers will \"race ahead\" and finish in an unpredictable order.\n>>\n>> Doesn't checkout.thresholdForParallelism only matter when\n>> checkout.workers != 1?\n>>\n>> So what you wrote seems like a reasonable explanation, but only if the\n>> original reporter set checkout.workers to imply the non-sequential\n>> behavior in the first place.\n>>\n>> That said...\n>>\n>>   - I also don't know off-hand of a place where we've defined the order\n>>     where Git will checkout files in the working copy. So depending on\n>>     that behavior isn't a safe thing to do.\n>>\n>>   - Committing build artifacts into your repository is generally\n>>     discouraged.\n>\n>If it's undefined and never implemented this is reasonable.\n>\n>But \"generally\" is a caveat, so while I agree with the statement it also implies\n>there's valid cases outside of that. Ones which used to work, too.\n>\n>Here are some useful cases I have seen for the combination of build rule +\n>checked in file:\n>\n>- part of a build requires licensed software that's not always available\n>\n>- part of the build requires large memory that other builders generally do\n>  not have available\n>\n>- part of the build process uses a different platform or some other system\n>  requirement\n>\n>- to fetch data eg. from a URL, with a record of the URL/automation but\n>  also a copy of the file as a record and for offline use\n>\n>So it's useful, to retain repeatable automation but not always build from square\n>one.\n>\n>Generally discouraged to check in build results yes, but I've found it very practical.\n>\n>> So while I'd guess that setting `checkout.workers` back to \"1\" (if it\n>> wasn't already) will probably restore the existing behavior, counting\n>> on that behavior in the first place is wrong.\n>\n>I think perhaps the tail is wagging the dog here, though.\n>\n>It's 'wrong' because it doesn't work; but I haven't seen anything to make me think\n>this is fundamentally or theoretically flawed.\n>\n>If we had a transactional file system we'd reasonably expect a checkout to be an\n>atomic operation -- same timestamp on the files created in that step. A\n>discrepancy in timestamps would be considered incorrect; it would imply an 'order'\n>to the checkout which, as you say, is order-less.\n>\n>Sowhat could be the bad outcomes if Git created files stamped with the point in\n>time of the \"git checkout\"?\n\nTimestamps are written based on when git modifies the file in the working directory. This actually ensures that automation does work. If intermediate contents are checked into repositories (I have people who do this for very justifiable regulatory reasons), the build has to make sure that there are appropriate separations of timestamps (a.k.a. 1 second) at a minimum on UNIX-ish systems. On some other boxes that do not even have timestamps for files (you know who you are) this is moot.\n\nHowever, there is a use case for maintaining timestamps - specifically for debuggers that check timestamps of source files. It is a big pain to make this work in git - but I script around this by setting the timestamps of files to the commit time when doing release builds, and allowing users to set the timestamp to the same for debugging. It helps but should not change the semantics of dev builds.\n\n-Randall\n\n"},{"id":"466199","messageId":"c060312e-0d35-8439-85dd-920b172c90be@xiplink.com","threadId":"58725","inReplyTo":"2210311614160.25661@stax.localdomain","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2022-11-01T13:55:11Z","receivedAt":"2022-11-01T13:55:18Z","isPatch":false,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"\nOn 2022-10-31 15:01, Mark Hills wrote:\n> Our use case: we commit some compiled objects to the repo, where compiling\n> is either slow or requires software which is not always available.\n> \n> Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)\n> we are noticing a change in build behaviour.\n> \n> Now, after a \"git clone\" we find the Makefile intermittently attempting\n> (and failing) some builds that are not intended.\n> \n> Indeed, Make is acting reasonably as the source file is sometimes\n> marginally newer than the destination (both checked out by Git), example\n> below.\n\nA fix for this was proposed in 2018 and dismissed [1].\n\nBack then, the problem was that as Git wrote files into a directory \nsometimes the clock would tick over at a bad time, and we'd end up with \nsome files being \"newer\" than others.  This would sour Make runs as you \ndescribe.\n\nNominally this is caused by putting generated files in the repo, but \nmany times that is unavoidable (e.g. you're forking an upstream that \nputs automake-generated stuff in the repo).\n\nIMHO, dismissing the problem back then was a mistake.  At the time I \nadvocated teaching Git to give all the files it touches (creates or \nmodifies) in a directory the same mtime (e.g. the time at the start of \nthe checkout operation).\n\nInstead the decision was to do nothing in Git, and instead let people \ncreate their own post-checkout hooks to touch the files.  I (and others) \nargued this was inadequate, to no avail.\n\n\t\tM.\n\n[1] https://public-inbox.org/git/20180413170129.15310-1-mgorny@gentoo.org/#r\n"},{"id":"466202","messageId":"CA+JQ7M81t0Lby=sB5GpUzJWakPgbi-ZNiQUL4va0wjDuk4v++Q@mail.gmail.com","threadId":"58725","inReplyTo":"2210311614160.25661@stax.localdomain","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2022-11-01T14:34:51Z","receivedAt":"2022-11-01T14:36:01Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"I have little to add on the underlying issue or non-issue but some\nideas on how to solve your problem\n\nOn Mon, Oct 31, 2022 at 8:39 PM Mark Hills <mark@xwax.org> wrote:\n>\n> ...\n> Indeed, Make is acting reasonably as the source file is sometimes\n> marginally newer than the destination (both checked out by Git), example\n> below.\n>\n> I've never had to consider consistency timestamps within a Git checkout\n> until now.\n>\n> It's entirely possible there's _never_ a guarantee of consistency here.\n\nIf your makefile depends on checkout, why not\n  git ls-files | xargs touch\nor if this done in an environment where there's not a fresh clone each\ntime, maybe\n  git diff HEAD --name-only --diff-filter=AM | xargs touch\nor something along those lines\n"},{"id":"466204","messageId":"221101.86fsf24qnu.gmgdl@evledraar.gmail.com","threadId":"58725","inReplyTo":"CA+JQ7M81t0Lby=sB5GpUzJWakPgbi-ZNiQUL4va0wjDuk4v++Q@mail.gmail.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T15:53:30Z","receivedAt":"2022-11-01T15:53:47Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 01 2022, Erik Cervin Edin wrote:\n\n> I have little to add on the underlying issue or non-issue but some\n> ideas on how to solve your problem\n>\n> On Mon, Oct 31, 2022 at 8:39 PM Mark Hills <mark@xwax.org> wrote:\n>>\n>> ...\n>> Indeed, Make is acting reasonably as the source file is sometimes\n>> marginally newer than the destination (both checked out by Git), example\n>> below.\n>>\n>> I've never had to consider consistency timestamps within a Git checkout\n>> until now.\n>>\n>> It's entirely possible there's _never_ a guarantee of consistency here.\n>\n> If your makefile depends on checkout, why not\n>   git ls-files | xargs touch\n> or if this done in an environment where there's not a fresh clone each\n> time, maybe\n>   git diff HEAD --name-only --diff-filter=AM | xargs touch\n> or something along those lines\n\nI believe you might be trying to re-invent \"make -B\" :)\n"},{"id":"466206","messageId":"221101.86bkpq4jan.gmgdl@evledraar.gmail.com","threadId":"58725","inReplyTo":"a87ebafd-c83-7a1d-d8d2-953bc9a93184@xwax.org","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T17:46:29Z","receivedAt":"2022-11-01T18:32:54Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 31 2022, Mark Hills wrote:\n\n> On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:\n>\n>> \n>> On Mon, Oct 31 2022, Mark Hills wrote:\n>> \n>> > Our use case: we commit some compiled objects to the repo, where compiling \n>> > is either slow or requires software which is not always available.\n>> >\n>> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) \n>> > we are noticing a change in build behaviour.\n>> >\n>> > Now, after a \"git clone\" we find the Makefile intermittently attempting \n>> > (and failing) some builds that are not intended.\n>> >\n>> > Indeed, Make is acting reasonably as the source file is sometimes \n>> > marginally newer than the destination (both checked out by Git), example \n>> > below.\n>> >\n>> > I've never had to consider consistency timestamps within a Git checkout \n>> > until now.\n>> >\n>> > It's entirely possible there's _never_ a guarantee of consistency here.\n>> >\n>> > But then something has certainly changed in practice, as this fault has \n>> > gone from never happening to now every couple of days.\n>> >\n>> > Imaginging I can't be the first person to encounter this, I searched for \n>> > existing threads or docs, but overwhemingly the results were question of \n>> > Git tracking the timestamps (as part of the commit) which this is not; \n>> > it's consistency within one checkout.\n>> >\n>> > $ git clone --depth 1 file:///path/to/repo.git\n>> >\n>> > $ stat winner.jpeg\n>> >   File: winner.jpeg\n>> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file\n>> > Device: fd07h/64775d    Inode: 33696       Links: 1\n>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n>> > Access: 2022-10-31 16:05:17.756858496 +0000\n>> > Modify: 2022-10-31 16:05:17.756858496 +0000\n>> > Change: 2022-10-31 16:05:17.756858496 +0000\n>> >  Birth: -\n>> >\n>> > $ stat winner.svg\n>> >   File: winner.svg\n>> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file\n>> > Device: fd07h/64775d    Inode: 33697       Links: 1\n>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n>> > Access: 2022-10-31 16:05:17.766859030 +0000\n>> > Modify: 2022-10-31 16:05:17.766859030 +0000\n>> > Change: 2022-10-31 16:05:17.766859030 +0000\n>> >  Birth: -\n>> >\n>> > Elsewhere in the repository, it's clear the timestamps are not consistent:\n>> >\n>> > $ stat Makefile\n>> >   File: Makefile\n>> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file\n>> > Device: fd07h/64775d    Inode: 33655       Links: 1\n>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n>> > Access: 2022-10-31 16:05:51.628660212 +0000\n>> > Modify: 2022-10-31 16:05:17.746857963 +0000\n>> > Change: 2022-10-31 16:05:17.746857963 +0000\n>> >  Birth: -\n>> \n>> I think you're almost certainly running into the parallel checkout,\n>> which is new in that revision range. Try tweaking checkout.workers and\n>> checkout.thresholdForParallelism (see \"man git-config\").\n>\n> Thanks, it will be interesting to try this and I'll report back.\n\nFWIW I was under the impression that we'd made it the default, so unless\nyou opted-in it's probably not that.\n\n>> I can't say without looking at the code/Makefile (and even then, I don't\n>> have time to dig here:), but if I had to bet I'd say that your\n>> dependencies have probably always been broken with these checked-in\n>> files, but they happend to work out if they were checked out in sorted\n>> order.\n>>\n>> And now with the parallel checkout they're not guaranteed to do that, as\n>> some workers will \"race ahead\" and finish in an unpredictable order.\n>\n> These are very simple Makefile rules, I don't think these dependencies are \n> broken; but your theory is in good alignment with the observed behaviour.\n>\n> For example, the rule from the recent case above is:\n>\n>   %.jpeg:         %.png\n>                   convert $< $(IMFLAGS) $@\n>\n>   %.png:          %.svg\n>                   inkscape --export-type=png --export-filename=$@ $<\n\nGrom a glance those don't seem broken to me, but I don't know how it\ninteracts with your built assets.\n\nSo e.g. if you are checking in your *.jpeg files those will be more\nrecent than either the *.png or source *.svn, so they won't be built.\n\nThis is fast getting out of scope of Git-specific advice, but you should\nrun \"make --debug\" (there's also sub-debug flags) to see if make's idea\nof the dependency graph matches yours.\n"},{"id":"466207","messageId":"221101.867d0e4ixy.gmgdl@evledraar.gmail.com","threadId":"58725","inReplyTo":"d4db484f-a525-f6db-1bfb-922f788dacd@xwax.org","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-01T18:34:14Z","receivedAt":"2022-11-01T18:40:34Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 31 2022, Mark Hills wrote:\n\n> On Mon, 31 Oct 2022, Taylor Blau wrote:\n>\n>> On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> > I think you're almost certainly running into the parallel checkout,\n>> > which is new in that revision range. Try tweaking checkout.workers and\n>> > checkout.thresholdForParallelism (see \"man git-config\").\n>> >\n>> > I can't say without looking at the code/Makefile (and even then, I don't\n>> > have time to dig here:), but if I had to bet I'd say that your\n>> > dependencies have probably always been broken with these checked-in\n>> > files, but they happend to work out if they were checked out in sorted\n>> > order.\n>> >\n>> > And now with the parallel checkout they're not guaranteed to do that, as\n>> > some workers will \"race ahead\" and finish in an unpredictable order.\n>> \n>> Doesn't checkout.thresholdForParallelism only matter when\n>> checkout.workers != 1?\n>> \n>> So what you wrote seems like a reasonable explanation, but only if the\n>> original reporter set checkout.workers to imply the non-sequential\n>> behavior in the first place.\n>> \n>> That said...\n>> \n>>   - I also don't know off-hand of a place where we've defined the order\n>>     where Git will checkout files in the working copy. So depending on\n>>     that behavior isn't a safe thing to do.\n>> \n>>   - Committing build artifacts into your repository is generally\n>>     discouraged.\n>\n> If it's undefined and never implemented this is reasonable.\n>\n> But \"generally\" is a caveat, so while I agree with the statement it also \n> implies there's valid cases outside of that. Ones which used to work, too.\n>\n> Here are some useful cases I have seen for the combination of build rule + \n> checked in file:\n>\n> - part of a build requires licensed software that's not always available\n>\n> - part of the build requires large memory that other builders generally do \n>   not have available\n>\n> - part of the build process uses a different platform or some other system \n>   requirement\n>\n> - to fetch data eg. from a URL, with a record of the URL/automation but \n>   also a copy of the file as a record and for offline use\n>\n> So it's useful, to retain repeatable automation but not always build from \n> square one.\n>\n> Generally discouraged to check in build results yes, but I've found it \n> very practical.\n>  \n>> So while I'd guess that setting `checkout.workers` back to \"1\" (if it \n>> wasn't already) will probably restore the existing behavior, counting on \n>> that behavior in the first place is wrong.\n>\n> I think perhaps the tail is wagging the dog here, though.\n>\n> It's 'wrong' because it doesn't work; but I haven't seen anything to make \n> me think this is fundamentally or theoretically flawed.\n>\n> If we had a transactional file system we'd reasonably expect a checkout to \n> be an atomic operation -- same timestamp on the files created in that \n> step. A discrepancy in timestamps would be considered incorrect; it would \n> imply an 'order' to the checkout which, as you say, is order-less.\n>\n> Sowhat could be the bad outcomes if Git created files stamped with the \n> point in time of the \"git checkout\"?\n\nI agree that it's practical in some scenarios, including checking in\nbuilt assets.\n\nBut those that are doing that need to be aware that combining that sort\nof thing with source control tends to upend your build system's idea of\nthe world.\n\nE.g. until recently in git.git we had a po/git.pot in-tree, which is a\n\"compiled file\" (although a plain-text one) that was checked in, and\ndealing with that in make's dependency graph was a (minor) pain\nsometimes.\n"},{"id":"466312","messageId":"20221102141609.1603860-1-matheus.bernardino@usp.br","threadId":"58725","inReplyTo":"221101.86bkpq4jan.gmgdl@evledraar.gmail.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-11-02T14:16:09Z","receivedAt":"2022-11-02T14:16:21Z","isPatch":false,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"> On Mon, Oct 31 2022, Mark Hills wrote:\n>\n> > On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:\n> >\n> >>\n> >> On Mon, Oct 31 2022, Mark Hills wrote:\n> >>\n> >> > Our use case: we commit some compiled objects to the repo, where compiling\n> >> > is either slow or requires software which is not always available.\n> >> >\n> >> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)\n> >> > we are noticing a change in build behaviour.\n> >> >\n> >> > Now, after a \"git clone\" we find the Makefile intermittently attempting\n> >> > (and failing) some builds that are not intended.\n> >> >\n> >> > Indeed, Make is acting reasonably as the source file is sometimes\n> >> > marginally newer than the destination (both checked out by Git), example\n> >> > below.\n> >> >\n> >> > I've never had to consider consistency timestamps within a Git checkout\n> >> > until now.\n> >> >\n> >> > It's entirely possible there's _never_ a guarantee of consistency here.\n> >> >\n> >> > But then something has certainly changed in practice, as this fault has\n> >> > gone from never happening to now every couple of days.\n> >> >\n> >> > Imaginging I can't be the first person to encounter this, I searched for\n> >> > existing threads or docs, but overwhemingly the results were question of\n> >> > Git tracking the timestamps (as part of the commit) which this is not;\n> >> > it's consistency within one checkout.\n> >> >\n> >> > $ git clone --depth 1 file:///path/to/repo.git\n> >> >\n> >> > $ stat winner.jpeg\n> >> >   File: winner.jpeg\n> >> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file\n> >> > Device: fd07h/64775d    Inode: 33696       Links: 1\n> >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> >> > Access: 2022-10-31 16:05:17.756858496 +0000\n> >> > Modify: 2022-10-31 16:05:17.756858496 +0000\n> >> > Change: 2022-10-31 16:05:17.756858496 +0000\n> >> >  Birth: -\n> >> >\n> >> > $ stat winner.svg\n> >> >   File: winner.svg\n> >> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file\n> >> > Device: fd07h/64775d    Inode: 33697       Links: 1\n> >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> >> > Access: 2022-10-31 16:05:17.766859030 +0000\n> >> > Modify: 2022-10-31 16:05:17.766859030 +0000\n> >> > Change: 2022-10-31 16:05:17.766859030 +0000\n> >> >  Birth: -\n> >> >\n> >> > Elsewhere in the repository, it's clear the timestamps are not consistent:\n> >> >\n> >> > $ stat Makefile\n> >> >   File: Makefile\n> >> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file\n> >> > Device: fd07h/64775d    Inode: 33655       Links: 1\n> >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> >> > Access: 2022-10-31 16:05:51.628660212 +0000\n> >> > Modify: 2022-10-31 16:05:17.746857963 +0000\n> >> > Change: 2022-10-31 16:05:17.746857963 +0000\n> >> >  Birth: -\n> >>\n> >> I think you're almost certainly running into the parallel checkout,\n> >> which is new in that revision range. Try tweaking checkout.workers and\n> >> checkout.thresholdForParallelism (see \"man git-config\").\n\nThis does look like something you would see with parallel checkout, yes.\nBut...\n\n> > Thanks, it will be interesting to try this and I'll report back.\n>\n> FWIW I was under the impression that we'd made it the default, so unless\n> you opted-in it's probably not that.\n\n... it indeed should be disabled by default. It seems Mark didn't\nmanually enable parallel checkout, as the original message only mentions\nthe git upgrade as a changing factor. And Alpine's git installation\nscript for 2.32.4 [1] doesn't seem to change our defaults either.\n\nPerhaps, it just happens that 2.32.4 changed the checkout processing\ntime slightly so that each entry is finished a bit slower (or the system\nwas overloaded at that moment?). Anyways, the creation order (based on\nthe mtimes) looks correct to me from a sequential-checkout point of\nview: first Makefile, than winner.jpeg, and finally winner.svg. That's\nthe order in which these files would appear in the index, which is the\norder followed by sequential checkout.\n\n[1]: https://git.alpinelinux.org/aports/tree/main/git/APKBUILD?h=3.14-stable&id=0f3285f2cfcb8362460002c27e219fadbf18c885\n"},{"id":"466313","messageId":"CAHd-oW5yBLfO-qTS9K57GjjYNUuu+zxon4xEEE6r7k4P8XERVw@mail.gmail.com","threadId":"58725","inReplyTo":"20221102141609.1603860-1-matheus.bernardino@usp.br","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2022-11-02T14:28:59Z","receivedAt":"2022-11-02T14:29:44Z","isPatch":false,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"[Oops, I accidentally sent this from my business account. I'm\nquote-replying it now from my correct personal account just in case\nthe original falls under spam folders for \"spoofing\".]\n\nOn Wed, Nov 2, 2022 at 11:16 AM Matheus Tavares\n<matheus.bernardino@usp.br> wrote:\n>\n> > On Mon, Oct 31 2022, Mark Hills wrote:\n> >\n> > > On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:\n> > >\n> > >>\n> > >> On Mon, Oct 31 2022, Mark Hills wrote:\n> > >>\n> > >> > Our use case: we commit some compiled objects to the repo, where compiling\n> > >> > is either slow or requires software which is not always available.\n> > >> >\n> > >> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)\n> > >> > we are noticing a change in build behaviour.\n> > >> >\n> > >> > Now, after a \"git clone\" we find the Makefile intermittently attempting\n> > >> > (and failing) some builds that are not intended.\n> > >> >\n> > >> > Indeed, Make is acting reasonably as the source file is sometimes\n> > >> > marginally newer than the destination (both checked out by Git), example\n> > >> > below.\n> > >> >\n> > >> > I've never had to consider consistency timestamps within a Git checkout\n> > >> > until now.\n> > >> >\n> > >> > It's entirely possible there's _never_ a guarantee of consistency here.\n> > >> >\n> > >> > But then something has certainly changed in practice, as this fault has\n> > >> > gone from never happening to now every couple of days.\n> > >> >\n> > >> > Imaginging I can't be the first person to encounter this, I searched for\n> > >> > existing threads or docs, but overwhemingly the results were question of\n> > >> > Git tracking the timestamps (as part of the commit) which this is not;\n> > >> > it's consistency within one checkout.\n> > >> >\n> > >> > $ git clone --depth 1 file:///path/to/repo.git\n> > >> >\n> > >> > $ stat winner.jpeg\n> > >> >   File: winner.jpeg\n> > >> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file\n> > >> > Device: fd07h/64775d    Inode: 33696       Links: 1\n> > >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> > >> > Access: 2022-10-31 16:05:17.756858496 +0000\n> > >> > Modify: 2022-10-31 16:05:17.756858496 +0000\n> > >> > Change: 2022-10-31 16:05:17.756858496 +0000\n> > >> >  Birth: -\n> > >> >\n> > >> > $ stat winner.svg\n> > >> >   File: winner.svg\n> > >> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file\n> > >> > Device: fd07h/64775d    Inode: 33697       Links: 1\n> > >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> > >> > Access: 2022-10-31 16:05:17.766859030 +0000\n> > >> > Modify: 2022-10-31 16:05:17.766859030 +0000\n> > >> > Change: 2022-10-31 16:05:17.766859030 +0000\n> > >> >  Birth: -\n> > >> >\n> > >> > Elsewhere in the repository, it's clear the timestamps are not consistent:\n> > >> >\n> > >> > $ stat Makefile\n> > >> >   File: Makefile\n> > >> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file\n> > >> > Device: fd07h/64775d    Inode: 33655       Links: 1\n> > >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)\n> > >> > Access: 2022-10-31 16:05:51.628660212 +0000\n> > >> > Modify: 2022-10-31 16:05:17.746857963 +0000\n> > >> > Change: 2022-10-31 16:05:17.746857963 +0000\n> > >> >  Birth: -\n> > >>\n> > >> I think you're almost certainly running into the parallel checkout,\n> > >> which is new in that revision range. Try tweaking checkout.workers and\n> > >> checkout.thresholdForParallelism (see \"man git-config\").\n>\n> This does look like something you would see with parallel checkout, yes.\n> But...\n>\n> > > Thanks, it will be interesting to try this and I'll report back.\n> >\n> > FWIW I was under the impression that we'd made it the default, so unless\n> > you opted-in it's probably not that.\n>\n> ... it indeed should be disabled by default. It seems Mark didn't\n> manually enable parallel checkout, as the original message only mentions\n> the git upgrade as a changing factor. And Alpine's git installation\n> script for 2.32.4 [1] doesn't seem to change our defaults either.\n>\n> Perhaps, it just happens that 2.32.4 changed the checkout processing\n> time slightly so that each entry is finished a bit slower (or the system\n> was overloaded at that moment?). Anyways, the creation order (based on\n> the mtimes) looks correct to me from a sequential-checkout point of\n> view: first Makefile, than winner.jpeg, and finally winner.svg. That's\n> the order in which these files would appear in the index, which is the\n> order followed by sequential checkout.\n>\n> [1]: https://git.alpinelinux.org/aports/tree/main/git/APKBUILD?h=3.14-stable&id=0f3285f2cfcb8362460002c27e219fadbf18c885\n"},{"id":"466316","messageId":"221102.86leot2ytm.gmgdl@evledraar.gmail.com","threadId":"58725","inReplyTo":"c060312e-0d35-8439-85dd-920b172c90be@xiplink.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-02T14:45:17Z","receivedAt":"2022-11-02T14:52:43Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 01 2022, Marc Branchaud wrote:\n\n> On 2022-10-31 15:01, Mark Hills wrote:\n>> Our use case: we commit some compiled objects to the repo, where compiling\n>> is either slow or requires software which is not always available.\n>> Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS\n>> upgrade)\n>> we are noticing a change in build behaviour.\n>> Now, after a \"git clone\" we find the Makefile intermittently\n>> attempting\n>> (and failing) some builds that are not intended.\n>> Indeed, Make is acting reasonably as the source file is sometimes\n>> marginally newer than the destination (both checked out by Git), example\n>> below.\n>\n> A fix for this was proposed in 2018 and dismissed [1].\n>\n> Back then, the problem was that as Git wrote files into a directory\n> sometimes the clock would tick over at a bad time, and we'd end up\n> with some files being \"newer\" than others.  This would sour Make runs\n> as you describe.\n>\n> Nominally this is caused by putting generated files in the repo, but\n> many times that is unavoidable (e.g. you're forking an upstream that \n> puts automake-generated stuff in the repo).\n>\n> IMHO, dismissing the problem back then was a mistake.  At the time I\n> advocated teaching Git to give all the files it touches (creates or \n> modifies) in a directory the same mtime (e.g. the time at the start of\n> the checkout operation).\n>\n> Instead the decision was to do nothing in Git, and instead let people\n> create their own post-checkout hooks to touch the files.  I (and\n> others) argued this was inadequate, to no avail.\n>\n> \t\tM.\n>\n> [1] https://public-inbox.org/git/20180413170129.15310-1-mgorny@gentoo.org/#r\n\nI think that's the wrong take-away from that thread. Maybe a patch for\nthis will get rejected in the end, but in that case it wasn't because\nthe git project is never going to take a patch like this.\n\nMaybe it won't, but:\n\n * That commit has no tests\n * It's clearly controversial behavior, so *if* we add it I think it's\n   better to make it opt-in configurable.\n * Once that's done, you'd need doc changes etc. for that.\n\nNow, maybe a sufficiently polished version would also be \"meh\" for\nwhatever reason, I just think it's premature to say that a change in\nthis direction would never be accepted.\n\nThat being said, I do wonder if software in the wild is being\nmonkeypatched to work around issues with make (or make-like tools)\nwhether such a change isn't better advocated in e.g. GNU make itself.\n\nIf it added \"B\" to \"MAKEFLAGS\" if it detected:\n\n * I'm in a git repository\n * It's the first time I'm running here, or \"nothing is built yet\"\n * My dependency graph would be different with \"-B\"\n\nWouldn't that be what people who want this feature are after?\n\nIt's not like it's SCM-agnostic, it already goes to significant trouble\nto cater to RCS and SCCS of all things, so I don't see why they'd\ncategorically reject a patch to cater to modern VCS's.\n\nAnd, unlike Gike, GNU make wouldn't need to guess that munging\ntimestamps would fix it, it can compute both versions of the dependency\ngraph, so it would know...\n"},{"id":"466400","messageId":"CA+JQ7M8g+e2y4tJ6k64ZQheS=x+HiZuB720M5bR8Y9yPFv0jZg@mail.gmail.com","threadId":"58725","inReplyTo":"221101.86fsf24qnu.gmgdl@evledraar.gmail.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2022-11-03T13:02:28Z","receivedAt":"2022-11-03T13:04:50Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"On Tue, Nov 1, 2022 at 4:53 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>\n> I believe you might be trying to re-invent \"make -B\" :)\n\nTrue, in the simple case, but if you\n  git diff HEAD --name-only --diff-filter=AM | xargs touch\nthat should consolidate the modified times on disk of the files of that commit\n\nIt needs a bit more work, something like\n  pre_checkout=$(git rev-parse HEAD)\n  git checkout XYX &&\n  git diff pre_checkout...XYZ --name-only --diff-filter=AM | xargs touch\nbut something like that can work around the inconsistent ordered\nmodified times after a checkout\n"},{"id":"466402","messageId":"bb6c6509-171e-ebdc-e251-aeba380bdda6@xiplink.com","threadId":"58725","inReplyTo":"221102.86leot2ytm.gmgdl@evledraar.gmail.com","subject":"Re: Consist timestamps within a checkout/clone","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2022-11-03T13:46:28Z","receivedAt":"2022-11-03T13:46:37Z","isPatch":false,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"\nOn 2022-11-02 10:45, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Nov 01 2022, Marc Branchaud wrote:\n>>\n>> Instead the decision was to do nothing in Git, and instead let people\n>> create their own post-checkout hooks to touch the files.  I (and others)\n>> argued this was inadequate, to no avail.\n>> >> [1] \nhttps://public-inbox.org/git/20180413170129.15310-1-mgorny@gentoo.org/#r\n> \n> I think that's the wrong take-away from that thread. Maybe a patch for\n> this will get rejected in the end, but in that case it wasn't because\n> the git project is never going to take a patch like this.\n> \n> Maybe it won't, but:\n> \n>   * That commit has no tests\n>   * It's clearly controversial behavior, so *if* we add it I think it's\n>     better to make it opt-in configurable.\n>   * Once that's done, you'd need doc changes etc. for that.\n> \n> Now, maybe a sufficiently polished version would also be \"meh\" for\n> whatever reason, I just think it's premature to say that a change in\n> this direction would never be accepted.\n\nI did not say that it would never be accepted; perhaps I should have \nsaid \"outcome\" instead of \"decision\".  That 2018 thread barely discussed \nchanges to the patch itself.  The patch's writer (not me) didn't pursue \nthe work, after its frosty initial reception.\n\nPast discussions around these proposals have been negative, which \ndiscourages people from polishing a submission as you suggest.  I hope \nthat this time is different.  (Before you suggest I submit a patch, I \nsadly don't have the time to hack on Git these days.)\n\n> That being said, I do wonder if software in the wild is being\n> monkeypatched to work around issues with make (or make-like tools)\n> whether such a change isn't better advocated in e.g. GNU make itself.\n> \n> If it added \"B\" to \"MAKEFLAGS\" if it detected:\n> \n>   * I'm in a git repository\n>   * It's the first time I'm running here, or \"nothing is built yet\"\n>   * My dependency graph would be different with \"-B\"\n> \n> Wouldn't that be what people who want this feature are after?\n> \n> It's not like it's SCM-agnostic, it already goes to significant trouble\n> to cater to RCS and SCCS of all things, so I don't see why they'd\n> categorically reject a patch to cater to modern VCS's.\n> \n> And, unlike Gike, GNU make wouldn't need to guess that munging\n> timestamps would fix it, it can compute both versions of the dependency\n> graph, so it would know...\n\nFair points about advocating for changes in make.  However, Gnu make \nisn't the only flavour out there.  Our builds use both BSD's and Gnu's \nmakes, for example.  (Also, BSD make has a completely different \ninterpretation of -B, and does not have any flag that mirrors Gnu make's \n-B.)\n\nGit is really the ideal place to solve this problem, instead of playing \nwhack-a-mole with build tools and upstream projects.  Making the \nbehaviour opt-in is perfectly reasonable.\n\n\t\tM.\n"}]}