git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Consist timestamps within a checkout/clone

From
MHMark Hills <mark@xwax.org>
Date
Oct 31, 2022, 22:29 UTC
Message-ID
<a87ebafd-c83-7a1d-d8d2-953bc9a93184@xwax.org>
In-Reply-To
<221031.86zgdb68p3.gmgdl@evledraar.gmail.com>
On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:
Show 66 quoted lines
> 
> On Mon, Oct 31 2022, Mark Hills wrote:
> 
> > Our use case: we commit some compiled objects to the repo, where compiling 
> > is either slow or requires software which is not always available.
> >
> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) 
> > we are noticing a change in build behaviour.
> >
> > Now, after a "git clone" we find the Makefile intermittently attempting 
> > (and failing) some builds that are not intended.
> >
> > Indeed, Make is acting reasonably as the source file is sometimes 
> > marginally newer than the destination (both checked out by Git), example 
> > below.
> >
> > I've never had to consider consistency timestamps within a Git checkout 
> > until now.
> >
> > It's entirely possible there's _never_ a guarantee of consistency here.
> >
> > But then something has certainly changed in practice, as this fault has 
> > gone from never happening to now every couple of days.
> >
> > Imaginging I can't be the first person to encounter this, I searched for 
> > existing threads or docs, but overwhemingly the results were question of 
> > Git tracking the timestamps (as part of the commit) which this is not; 
> > it's consistency within one checkout.
> >
> > $ git clone --depth 1 file:///path/to/repo.git
> >
> > $ stat winner.jpeg
> >   File: winner.jpeg
> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file
> > Device: fd07h/64775d    Inode: 33696       Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > Access: 2022-10-31 16:05:17.756858496 +0000
> > Modify: 2022-10-31 16:05:17.756858496 +0000
> > Change: 2022-10-31 16:05:17.756858496 +0000
> >  Birth: -
> >
> > $ stat winner.svg
> >   File: winner.svg
> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file
> > Device: fd07h/64775d    Inode: 33697       Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > Access: 2022-10-31 16:05:17.766859030 +0000
> > Modify: 2022-10-31 16:05:17.766859030 +0000
> > Change: 2022-10-31 16:05:17.766859030 +0000
> >  Birth: -
> >
> > Elsewhere in the repository, it's clear the timestamps are not consistent:
> >
> > $ stat Makefile
> >   File: Makefile
> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file
> > Device: fd07h/64775d    Inode: 33655       Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > Access: 2022-10-31 16:05:51.628660212 +0000
> > Modify: 2022-10-31 16:05:17.746857963 +0000
> > Change: 2022-10-31 16:05:17.746857963 +0000
> >  Birth: -
> 
> I think you're almost certainly running into the parallel checkout,
> which is new in that revision range. Try tweaking checkout.workers and
> checkout.thresholdForParallelism (see "man git-config").
Thanks, it will be interesting to try this and I'll report back.
 
Show 8 quoted lines
> I can't say without looking at the code/Makefile (and even then, I don't
> have time to dig here:), but if I had to bet I'd say that your
> dependencies have probably always been broken with these checked-in
> files, but they happend to work out if they were checked out in sorted
> order.
>
> And now with the parallel checkout they're not guaranteed to do that, as
> some workers will "race ahead" and finish in an unpredictable order.

These are very simple Makefile rules, I don't think these dependencies are broken; but your theory is in good alignment with the observed behaviour.

For example, the rule from the recent case above is:
  %.jpeg:         %.png
                  convert $< $(IMFLAGS) $@
  %.png:          %.svg
                  inkscape --export-type=png --export-filename=$@ $<

As you suggest, perhaps the Git implementation previously ran checked out in some kind of time order then this happens to fulfil a useful behaviour.

Specificaly with build artefacts. These are likely to have been added to the repo after the source file. This could have been providing some pratical and useful tendency of ordering.

Show 5 quoted lines
> But that's all just a guess, perhaps it has nothing to do with parallel
> checkout, such dependency issues are sensitive to all sorts of other
> things, e.g. maybe git got slightly faster (or slower), so now files
> that were always on different seconds (or the same) aren't in the state
> they were in before...
Hopefully I'll get to some experiments to narrow this down.
Thanks
-- 
Mark
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 7 of 17 in “Consist timestamps within a checkout/clone”
  1. Mark HillsOct 31, 2022
  2. Ævar Arnfjörð BjarmasonOct 31, 2022
  3. Taylor BlauOct 31, 2022
  4. Mark HillsOct 31, 2022
  5. rsbecker@nexbridge.comOct 31, 2022
  6. Ævar Arnfjörð BjarmasonNov 1, 2022
  7. Mark HillsOct 31, 2022
  8. Ævar Arnfjörð BjarmasonNov 1, 2022
  9. Matheus TavaresNov 2, 2022
  10. Matheus TavaresNov 2, 2022
  11. Andreas SchwabOct 31, 2022
  12. Marc BranchaudNov 1, 2022
  13. Ævar Arnfjörð BjarmasonNov 2, 2022
  14. Marc BranchaudNov 3, 2022
  15. Erik Cervin EdinNov 1, 2022
  16. Ævar Arnfjörð BjarmasonNov 1, 2022
  17. Erik Cervin EdinNov 3, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.