git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC Patch] Preventing corrupt objects from entering the repository

From
Nicolas Pitre <nico@cam.org>
Date
Feb 11, 2008, 20:41 UTC
Message-ID
<alpine.LFD.1.00.0802111513360.2732@xanadu.home>
In-Reply-To
<20080211195623.GA21878@auto.tuwien.ac.at>
On Mon, 11 Feb 2008, Martin Koegler wrote:
Show 22 quoted lines
> On Sun, Feb 10, 2008 at 07:33:34PM -0500, Nicolas Pitre wrote:
> > On Sun, 10 Feb 2008, Junio C Hamano wrote:
> > 
> > > mkoegler@auto.tuwien.ac.at (Martin Koegler) writes:
> > > 
> > > > This patch adds a cache to keep the object data in memory. The delta
> > > > resolving code must also search in the cache.
> > > 
> > > I have to wonder what the memory pressure in real-life usage
> > > will be like.
> 
> > FWIW, I don't like this idea.
> >
> > I'm struggling to find ways to improve performances of 
> > pack-objects/index-pack with those large repositories that are becoming 
> > more common (i.e. GCC, OOO, Mozilla, etc.)  Anything that increase 
> > memory usage isn't very welcome IMHO.
> 
> Maybe I have missed something, but all repack problems reported on the
> git mailing list happen durring the deltifing phase. The problematic
> files are mostly bigger blobs. I'm aware of these problems, so my
> patch does not keep any blobs in memory.

This is not only repack, which is something that few people should run anyway.

It is also index-pack which takes an awful amount of time on the OOO repo (much less than the repack but still), and that's something that is visible to anyone cloning it.

So the comparison with pack-objects is useless. In absolute terms, index-pack has to inprove, not regress.

> So my conclusion is, that the memory usage of index-pack with --strict
> should not be too worse compared to pack-objects.

Well, the comparison is flawed anyway. Agressively repacking the OOO repo is reported to take around 14 GB of RAM if not bounded, whereas index-pack remained around 300MB. But the buffer cache managed by the kernel is also a big performance factor and anything that requires more memory in user space will affect that cache.

Then remember that this repo has 2411764 objects.
> Please remember, that --strict is used for pushing data.

And what exactly does it provide in terms of safetiness that index-pack doesn't already do? I'm not sure I fully understood the goal here.

Nicolas
Previous: Martin KoeglerNext: Martin Koegler
Message 5 of 23 in “Preventing corrupt objects from entering the repository”
  1. Preventing corrupt objects from entering the repositoryMartin Koegler, Feb 10, 2008
  2. Junio C HamanoFeb 11, 2008
  3. Nicolas PitreFeb 11, 2008
  4. Martin KoeglerFeb 11, 2008
  5. Nicolas PitreFeb 11, 2008
  6. Martin KoeglerFeb 11, 2008
  7. Nicolas PitreFeb 12, 2008
  8. Martin KoeglerFeb 12, 2008
  9. Nicolas PitreFeb 12, 2008
  10. Martin KoeglerFeb 12, 2008
  11. Nicolas PitreFeb 12, 2008
  12. Shawn O. PearceFeb 13, 2008
  13. Martin KoeglerFeb 13, 2008
  14. Remove object-refs from fsckShawn O. Pearce, Feb 14, 2008
  15. Martin KoeglerFeb 14, 2008
  16. Shawn O. PearceFeb 13, 2008
  17. Martin KoeglerFeb 13, 2008
  18. Johannes SchindelinFeb 13, 2008
  19. Shawn O. PearceFeb 14, 2008
  20. Martin KoeglerFeb 14, 2008
  21. Johannes SchindelinFeb 15, 2008
  22. Martin KoeglerFeb 15, 2008
  23. Martin KoeglerFeb 12, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.