git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 10/10] pack-objects: improve partial packfile reuse

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Oct 11, 2019, 21:04 UTC
Message-ID
<20191011210434.140296-1-jonathantanmy@google.com>
In-Reply-To
<20191011180125.GA20601@sigill.intra.peff.net>
Show 12 quoted lines
> > This makes sense - offsets may be different when we omit objects from
> > the packfile. I think this can be computed by calculating the number of
> > zero bits between the current object's index and the nth object prior
> > (where n is the offset) in the bitmap resulting from
> > reuse_partial_packfile_from_bitmap() above, thus eliminating the need
> > for this array, but I haven't tested it.
> 
> You need to know not just the number of zero bits, but the accumulated
> offset due to those missing objects. So you'd end up having to walk over
> the revindex for that set of objects. This array is basically caching
> those accumulated offsets (for the parts we _do_ include) so we don't
> have to compute them repeatedly.

Ah...yes. For some reason I thought that the offset was a number of objects, but it is actually a number of bytes. The patch makes sense now.

> There's also a more subtle issue with entry sizes; see below.
Good point.
Show 14 quoted lines
> > > @@ -1002,6 +1132,10 @@ static int have_duplicate_entry(const struct object_id *oid,
> > >  {
> > >  	struct object_entry *entry;
> > >  
> > > +	if (reuse_packfile_bitmap &&
> > > +	    bitmap_walk_contains(bitmap_git, reuse_packfile_bitmap, oid))
> > > +		return 1;
> > 
> > Hmm...why did we previously not need to check the reuse information, but
> > we do now? I gave the code a cursory glance but couldn't find the
> > answer.
> 
> I think the original code may simply have been buggy and nobody noticed.
> Here's what I wrote when this line was added in our fork:
[snip explanation]
Thanks - I'll also take a look if I have time.
> Thanks for looking at it. I still have to take a careful pass over the
> whole split, but I've tried to at least answer your questions in the
> meantime.

Thanks for your responses. Also thanks to Christian for splitting it in the first place, making it easier to review.

Previous: Jeff KingNext: Junio C Hamano
Message 26 of 31 in “Rewrite packfile reuse code”
  1. 00/10 Rewrite packfile reuse codeChristian Couder, Sep 13, 2019
  2. 01/10 builtin/pack-objects: report reused packfile objectsChristian Couder, Sep 13, 2019
  3. 02/10 packfile: expose get_delta_base()Christian Couder, Sep 13, 2019
  4. 06/10 pack-bitmap: introduce bitmap_walk_contains()Christian Couder, Sep 13, 2019
  5. 04/10 ewah/bitmap: always allocate 2 more wordsChristian Couder, Sep 13, 2019
  6. Jonathan TanOct 10, 2019
  7. Christian CouderOct 11, 2019
  8. Jeff KingOct 11, 2019
  9. 07/10 csum-file: introduce hashfile_total()Christian Couder, Sep 13, 2019
  10. 08/10 pack-objects: introduce pack.allowPackReuseChristian Couder, Sep 13, 2019
  11. Junio C HamanoSep 13, 2019
  12. 03/10 ewah/bitmap: introduce bitmap_word_alloc()Christian Couder, Sep 13, 2019
  13. 05/10 pack-bitmap: don't rely on bitmap_git->reuse_objectsChristian Couder, Sep 13, 2019
  14. Jonathan TanOct 10, 2019
  15. Christian CouderOct 11, 2019
  16. 10/10 pack-objects: improve partial packfile reuseChristian Couder, Sep 13, 2019
  17. Junio C HamanoSep 13, 2019
  18. Jeff KingSep 14, 2019
  19. Junio C HamanoSep 14, 2019
  20. Jeff KingOct 2, 2019
  21. Junio C HamanoOct 3, 2019
  22. Christian CouderOct 3, 2019
  23. Jonathan TanOct 10, 2019
  24. Christian CouderOct 11, 2019
  25. Jeff KingOct 11, 2019
  26. Jonathan TanOct 11, 2019
  27. Junio C HamanoOct 12, 2019
  28. Jeff KingOct 13, 2019
  29. Junio C HamanoOct 17, 2019
  30. Jeff KingOct 17, 2019
  31. 09/10 builtin/pack-objects: introduce obj_is_packed()Christian Couder, Sep 13, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.