git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Keeping unreachable objects in a separate pack instead of loose?

From
Ted Ts'o <tytso@mit.edu>
Date
Jun 11, 2012, 21:14 UTC
Message-ID
<20120611211401.GA21775@thunk.org>
In-Reply-To
<20120611183414.GD20134@sigill.intra.peff.net>
On Mon, Jun 11, 2012 at 02:34:14PM -0400, Jeff King wrote:
Show 14 quoted lines
> You _could_ make a separate cruft pack for each pack that you repack. So
> if I have A.pack and B.pack, I'd pack all of the reachable objects into
> C.pack, and then make D.pack containing the unreachable objects from
> A.pack, and E.pack with the unreachable objects from B.pack. And then
> set the mtime of the cruft packs to that of their parent packs.
> 
> And then the next time you pack, repacking D and E would probably be a
> no-op that preserves mtime, but might create a new pack that ejects some
> now-reachable object.
> 
> To implement that, I think your --list-unreachable would just have to
> print a list of "<pack-mtime> <sha1>" pairs, and then you would pack
> each set with an identical mtime (or even a "close enough" mtime within
> some slop)....

How about this instead? We distinguish between cruft packs and "real" packs by the filename. So we have "cruft-<SHA1>.{idx,pack}" and "pack-<SHA1>.{idx.pack}".

Normally, git will look at any pack in the pack directory that has an .idx and .pack extension, but during repack operation, it will by only look in the pack-* packs first. If it can't find an object there, it will then fall back to trying to fetch the object from the cruft-* packs, and if it finds the object, it copies it into the new pack which is creating, thus "rescueing" an object which reappears during the expiry window. This should be a relatively rare event, and if it happens, the object will be in two packs, a pack-* pack and a cruft-* pack, but that's OK.

So since git pack-objects isn't even looking in the cruft-* packs except when it needs to rescue an object, the objects in the cruft-* packs won't get copied, and we won't need to have per-object mtimes. It also means it will go faster since it's not copying the cruft-* packs at all, and possibly not even looking at them.

Now all we need to do is delete any cruft-* packs which are older than the expiry window. We don't even need to look at their contents.

It does imply that we may accumulate a new cruft-<SHA1> pack each time we run git gc, but users shouldn't be running git gc all that often anyway. And even if they do run it all the time, it will still be more efficient than keeping the unreachable objects as loose objects.

     	       	    	    		    	    - Ted
Previous: Hallvard Breien FurusethNext: Jeff King
Message 17 of 48 in “Keeping unreachable objects in a separate pack instead of loose?”
  1. Theodore Ts'oJun 10, 2012
  2. Hallvard B FurusethJun 10, 2012
  3. Thomas RastJun 11, 2012
  4. Ted Ts'oJun 11, 2012
  5. Jeff KingJun 11, 2012
  6. Nicolas PitreJun 11, 2012
  7. Ted Ts'oJun 11, 2012
  8. Jeff KingJun 11, 2012
  9. Ted Ts'oJun 11, 2012
  10. Jeff KingJun 11, 2012
  11. Jeff KingJun 11, 2012
  12. Ted Ts'oJun 11, 2012
  13. Jeff KingJun 11, 2012
  14. Hallvard Breien FurusethJun 11, 2012
  15. Jeff KingJun 11, 2012
  16. Hallvard Breien FurusethJun 11, 2012
  17. Ted Ts'oJun 11, 2012
  18. Jeff KingJun 11, 2012
  19. Ted Ts'oJun 11, 2012
  20. Jeff KingJun 11, 2012
  21. Ted Ts'oJun 11, 2012
  22. Jeff KingJun 11, 2012
  23. Nicolas PitreJun 12, 2012
  24. Jeff KingJun 12, 2012
  25. Nicolas PitreJun 12, 2012
  26. Jeff KingJun 12, 2012
  27. Shawn PearceJun 12, 2012
  28. Jeff KingJun 12, 2012
  29. Nicolas PitreJun 12, 2012
  30. Andreas SchwabJun 12, 2012
  31. Jeff KingJun 12, 2012
  32. Nicolas PitreJun 12, 2012
  33. Jeff KingJun 12, 2012
  34. Nicolas PitreJun 12, 2012
  35. Jeff KingJun 12, 2012
  36. Nicolas PitreJun 12, 2012
  37. Nicolas PitreJun 12, 2012
  38. Jeff KingJun 12, 2012
  39. Nicolas PitreJun 12, 2012
  40. Ted Ts'oJun 12, 2012
  41. Nicolas PitreJun 12, 2012
  42. Ted Ts'oJun 12, 2012
  43. Nicolas PitreJun 12, 2012
  44. Ted Ts'oJun 12, 2012
  45. Jeff KingJun 12, 2012
  46. Martin FickJun 13, 2012
  47. Johan HerlandJun 13, 2012
  48. Junio C HamanoJun 11, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.