git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: non-smooth progress indication for git fsck and git gc

From
Duy Nguyen <pclouds@gmail.com>
Date
Aug 17, 2018, 14:39 UTC
Message-ID
<CACsJy8Aycxf3S9zARuv_BeKLyh667ewcB1dr3X9VY3i3meR9hg@mail.gmail.com>
In-Reply-To
<20180816210657.GA9291@sigill.intra.peff.net>
On Thu, Aug 16, 2018 at 11:08 PM Jeff King <peff@peff.net> wrote:
Show 28 quoted lines
>
> On Thu, Aug 16, 2018 at 04:55:56PM -0400, Jeff King wrote:
>
> > >  * We spend the majority of the ~30s on this:
> > >    https://github.com/git/git/blob/63749b2dea5d1501ff85bab7b8a7f64911d21dea/pack-check.c#L70-L79
> >
> > This is hashing the actual packfile. This is potentially quite long,
> > especially if you have a ton of big objects.
> >
> > I wonder if we need to do this as a separate step anyway, though. Our
> > verification is based on index-pack these days, which means it's going
> > to walk over the whole content as part of the "Indexing objects" step to
> > expand base objects and mark deltas for later. Could we feed this hash
> > as part of that walk over the data? It's not going to save us 30s, but
> > it's likely to be more efficient. And it would fold the effort naturally
> > into the existing progress meter.
>
> Actually, I take it back. That's the nice, modern way we do it in
> git-verify-pack. But git-fsck uses the ancient "just walk over all of
> the idx entries method". It at least sorts in pack order, which is good,
> but:
>
>   - it's not multi-threaded, like index-pack/verify-pack
>
>   - the index-pack way is actually more efficient than pack-ordering for
>     the delta-base cache, because it actually walks the delta-graph in
>     the optimal order
>

I actually tried to make git-fsck use index-pack --verify at one point. The only thing that stopped it from working was index-pack automatically wrote the newer index version if I remember correctly, and that would fail the final hash check. fsck performance was not a big deal so I dropped it. Just saying it should be possible, if someone's interested in that direction.

-- 
Duy
Previous: Jeff KingNext: Ulrich Windl
Message 11 of 25 in “non-smooth progress indication for git fsck and git gc”
  1. Ulrich WindlAug 16, 2018
  2. Duy NguyenAug 16, 2018
  3. Jeff KingAug 16, 2018
  4. Antw: Re: non-smooth progress indication for git fsck and git gcUlrich Windl, Aug 20, 2018
  5. Jeff KingAug 16, 2018
  6. Jeff KingAug 16, 2018
  7. Junio C HamanoAug 16, 2018
  8. Ævar Arnfjörð BjarmasonAug 16, 2018
  9. Jeff KingAug 16, 2018
  10. Jeff KingAug 16, 2018
  11. Duy NguyenAug 17, 2018
  12. Antw: Re: non-smooth progress indication for git fsck and git gcUlrich Windl, Aug 20, 2018
  13. Ævar Arnfjörð BjarmasonAug 20, 2018
  14. Ulrich WindlAug 20, 2018
  15. Jeff KingAug 21, 2018
  16. Ulrich WindlAug 21, 2018
  17. Duy NguyenAug 21, 2018
  18. Ævar Arnfjörð BjarmasonSep 1, 2018
  19. Ævar Arnfjörð BjarmasonSep 1, 2018
  20. Jeff KingSep 2, 2018
  21. Jeff KingSep 2, 2018
  22. Jeff KingSep 2, 2018
  23. Ævar Arnfjörð BjarmasonSep 3, 2018
  24. Jeff KingSep 7, 2018
  25. Junio C HamanoSep 4, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.