git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] Speedup recursive by flushing index only once for all entries

From
Alex Riesen <fork0@t-online.de>
Date
Jan 11, 2007, 22:10 UTC
Message-ID
<20070111221053.GD6058@steel.home>
In-Reply-To
<7vfyah48j2.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano, Thu, Jan 11, 2007 21:23:45 +0100:
Show 26 quoted lines
> > That said, I think we actually have another problem entirely:
> >
> > Look at "write_cache()", Junio: isn't it leaking memory like mad?
> >
> > Shouldn't we have something like this?
> >
> > It's entirely possible that the _real_ problem with the "flush the index 
> > all the time" was that it just caused this bug: tons and tons of lost 
> > memory, causing git-merge-recursive to grow explosively (~6MB per 
> > cache flush, and a _lot_ of cache flushes), which on a 384MB machine 
> > quickly uses up memory and causes totally unnecessary swapping.
> 
> You are right -- there is absolutely no reason to retain this
> memory.  It is a serialized representation of cache-tree data
> only to be stored in the index, and no other user of this data
> exists.  Thanks for spotting this.
> 
> Writing out 6MB per every path changed in a merge would still be
> an unnecessary overhead over the one in 'next', so there is no
> reason to replace 'next' with this single liner of yours, but I
> am interested in seeing how much of the 20-minute vs 1-minute
> difference is attributable to this leak, just out of curiosity.
> 
> Alex, if you have a chance, could you apply Linus's single-liner
> on top of 'master', without either of the merge-recursive
> patches in 'next', and see what kind of numbers you would get?

With regard to speed: not noticable on the cygwin machine. The 384Mb-laptop liked it: moved into 40-50 sec range (it had real problems (minutes) doing that merge without at least my first patch. Because of the leak, as we now understand).

It must have been large leak, as I really have seen the memory usage dropping down significantly.

Previous: Junio C HamanoNext: Linus Torvalds
Message 22 of 40 in “Speedup recursive by flushing index only once for all entries”
  1. Speedup recursive by flushing index only once for all entriesAlex Riesen, Jan 4, 2007
  2. Johannes SchindelinJan 4, 2007
  3. Alex RiesenJan 4, 2007
  4. Junio C HamanoJan 4, 2007
  5. Alex RiesenJan 5, 2007
  6. Alex RiesenJan 7, 2007
  7. Junio C HamanoJan 10, 2007
  8. Junio C HamanoJan 10, 2007
  9. Junio C HamanoJan 10, 2007
  10. Alex RiesenJan 10, 2007
  11. Linus TorvaldsJan 10, 2007
  12. Johannes SchindelinJan 11, 2007
  13. Alex RiesenJan 11, 2007
  14. Alex RiesenJan 11, 2007
  15. Junio C HamanoJan 11, 2007
  16. Alex RiesenJan 11, 2007
  17. Linus TorvaldsJan 11, 2007
  18. Alex RiesenJan 11, 2007
  19. Linus TorvaldsJan 11, 2007
  20. Alex RiesenJan 11, 2007
  21. Junio C HamanoJan 11, 2007
  22. Alex RiesenJan 11, 2007
  23. Linus TorvaldsJan 11, 2007
  24. Junio C HamanoJan 11, 2007
  25. Alex RiesenJan 12, 2007
  26. Junio C HamanoJan 11, 2007
  27. Johannes SchindelinJan 11, 2007
  28. Sergey VlasovJan 12, 2007
  29. Alex RiesenJan 12, 2007
  30. Sergey VlasovJan 12, 2007
  31. Junio C HamanoJan 12, 2007
  32. merge-recursive: do not report the resulting tree object nameJunio C Hamano, Jan 12, 2007
  33. Johannes SchindelinJan 12, 2007
  34. Junio C HamanoJan 13, 2007
  35. Jakub NarebskiJan 13, 2007
  36. Johannes SchindelinJan 13, 2007
  37. Shawn O. PearceJan 13, 2007
  38. Junio C HamanoJan 13, 2007
  39. Alex RiesenJan 12, 2007
  40. Sergey VlasovJan 12, 2007

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.