git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Keeping unreachable objects in a separate pack instead of loose?

From
Nicolas Pitre <nico@fluxnic.net>
Date
Jun 12, 2012, 19:15 UTC
Message-ID
<alpine.LFD.2.02.1206121509340.23555@xanadu.home>
In-Reply-To
<20120612183702.GD1803@thunk.org>
On Tue, 12 Jun 2012, Ted Ts'o wrote:
Show 20 quoted lines
> On Tue, Jun 12, 2012 at 02:25:47PM -0400, Nicolas Pitre wrote:
> > > Earlier in the thread, I outlined another scheme by which you could
> > > repack and avoid the duplicates. It does not require changes to git's
> > > object lookup process, because it would involve manually feeding the
> > > list of cruft objects to pack-objects (which will pack what you ask it,
> > > regardless of whether the objects are in other packs).
> > 
> > That might be hard to achieve good delta compression though, as the main 
> > key to sort those objects is their path name, and with unreferenced 
> > objects you might not necessarily have that information.  The ability to 
> > reuse pack data might mitigate this though.
> 
> Compared to loose objects, even not-so-great delta compression is
> manna from heaven.  Remember what originally got me to start this
> flag.  There was 4.5 megabytes worth of loose objects, that when I
> created the object id list and fed the result to git pack-object, the
> resulting pack was 244k.
> 
> OK, maybe the delta compression wasn't optimal.  Compared to the 4.5
> megabytes of loose objects --- I'll happily settle for that!  :-)

Sure. However I would be even happier if we could delete those unneeded objects outright. The official reason why they're there for two weeks should be to avoid some race conditions, and in this case two weeks is way over the top as in "normal" conditions the actual window for a race is in the order of a few seconds.. Any other use case should be considered abusive.

Show 14 quoted lines
> > So the problem is really about 'git gc' creating more data on disk which 
> > is counter productive for a garbage collecting task.  Maybe the trick is 
> > simply not to delete any of the old pack which content was repacked into 
> > a single new pack and let them age before deleting them, rather than 
> > exploding a bunch of loose objects.  But then we're back to the same 
> > issue I wanted to get away from i.e. identifying real cruft packs and 
> > making them safely deletable.
> 
> But the old packs are huge; in my case, a full set of packs was around
> 16 megabytes.  Right now, git gc *increased* my disk usage by 4.5
> megabytes.  If we don't delete the old backs, then git gc would
> increase disk usage by 16 megabytes --- which is far, far worse.
> 
> Writing a 244k cruft pack is a soooooo much preferable.

But as you might have noticed, there are a bunch of semantic problems with that as well.

Nicolas
Previous: Ted Ts'oNext: Ted Ts'o
Message 41 of 48 in “Keeping unreachable objects in a separate pack instead of loose?”
  1. Theodore Ts'oJun 10, 2012
  2. Hallvard B FurusethJun 10, 2012
  3. Thomas RastJun 11, 2012
  4. Ted Ts'oJun 11, 2012
  5. Jeff KingJun 11, 2012
  6. Nicolas PitreJun 11, 2012
  7. Ted Ts'oJun 11, 2012
  8. Jeff KingJun 11, 2012
  9. Ted Ts'oJun 11, 2012
  10. Jeff KingJun 11, 2012
  11. Jeff KingJun 11, 2012
  12. Ted Ts'oJun 11, 2012
  13. Jeff KingJun 11, 2012
  14. Hallvard Breien FurusethJun 11, 2012
  15. Jeff KingJun 11, 2012
  16. Hallvard Breien FurusethJun 11, 2012
  17. Ted Ts'oJun 11, 2012
  18. Jeff KingJun 11, 2012
  19. Ted Ts'oJun 11, 2012
  20. Jeff KingJun 11, 2012
  21. Ted Ts'oJun 11, 2012
  22. Jeff KingJun 11, 2012
  23. Nicolas PitreJun 12, 2012
  24. Jeff KingJun 12, 2012
  25. Nicolas PitreJun 12, 2012
  26. Jeff KingJun 12, 2012
  27. Shawn PearceJun 12, 2012
  28. Jeff KingJun 12, 2012
  29. Nicolas PitreJun 12, 2012
  30. Andreas SchwabJun 12, 2012
  31. Jeff KingJun 12, 2012
  32. Nicolas PitreJun 12, 2012
  33. Jeff KingJun 12, 2012
  34. Nicolas PitreJun 12, 2012
  35. Jeff KingJun 12, 2012
  36. Nicolas PitreJun 12, 2012
  37. Nicolas PitreJun 12, 2012
  38. Jeff KingJun 12, 2012
  39. Nicolas PitreJun 12, 2012
  40. Ted Ts'oJun 12, 2012
  41. Nicolas PitreJun 12, 2012
  42. Ted Ts'oJun 12, 2012
  43. Nicolas PitreJun 12, 2012
  44. Ted Ts'oJun 12, 2012
  45. Jeff KingJun 12, 2012
  46. Martin FickJun 13, 2012
  47. Johan HerlandJun 13, 2012
  48. Junio C HamanoJun 11, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.