git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] git exproll: steps to tackle gc aggression

From
MFMartin Fick <mfick@codeaurora.org>
Date
Aug 8, 2013, 17:34 UTC
Message-ID
<201308081134.35735.mfick@codeaurora.org>
In-Reply-To
<7v61vgazp5.fsf@alter.siamese.dyndns.org>

On Thursday, August 08, 2013 10:56:38 am Junio C Hamano wrote:

Show 8 quoted lines
> I thought the discussion was about making the local gc
> cheaper, and the "Imagine we have a cheap way" was to
> address it by assuming that the daily "pack young
> objects into a single pack" can be sped up if we did not
> have to traverse history.  More permanent packs (the
> older ones in "set of packs staggered by age" Martin
> proposes) in the repository should go through the normal
> history traversal route.

Assuming I understand what you are suggesting, would these "young object" likely still get "deduped" in an efficient way without doing history traversal (it sounds like they would)? In other words, if I understand correctly, it would save time by not pruning unreferenced objects, but it would still be deduping things and delta compressing also, so you would still likely get a great benefit from creating these young object packs? In other words, is there still a good chance that my 317 new pack files which included a 33M pack file will still get consolidated down to something near 8M?

If so, then yeah this might be nice, especially if the history traversal is what would speed this up. Because today, my solution mostly saves IO and not time. I think it still saves time, I believe I have seen up to a 50% savings, but that is nothing compared to massive, several orders of magnitude IO savings. But if what you suggest could also give massive time (orders of magnitude) savings along with the IO improvements I am seeing, then suddenly repacking regularly would become very cheap even on large repos.

The only time consuming piece would be pruning then? Could bitmaps eventually help out there?

-Martin
-- 
The Qualcomm Innovation Center, Inc. is a member of Code 
Aurora Forum, hosted by The Linux Foundation
Previous: Junio C HamanoNext: Junio C Hamano
Message 8 of 33 in “git exproll: steps to tackle gc aggression”
  1. git exproll: steps to tackle gc aggressionRamkumar Ramachandra, Aug 6, 2013
  2. Duy NguyenAug 6, 2013
  3. Junio C HamanoAug 6, 2013
  4. Ramkumar RamachandraAug 7, 2013
  5. Junio C HamanoAug 8, 2013
  6. Ramkumar RamachandraAug 8, 2013
  7. Junio C HamanoAug 8, 2013
  8. Martin FickAug 8, 2013
  9. Junio C HamanoAug 8, 2013
  10. Ramkumar RamachandraAug 8, 2013
  11. Ramkumar RamachandraAug 8, 2013
  12. Junio C HamanoAug 8, 2013
  13. Ramkumar RamachandraAug 8, 2013
  14. Martin LanghoffAug 8, 2013
  15. Jeff KingAug 9, 2013
  16. Ramkumar RamachandraAug 9, 2013
  17. Junio C HamanoAug 9, 2013
  18. Jeff KingAug 9, 2013
  19. Duy NguyenAug 10, 2013
  20. Jeff KingAug 10, 2013
  21. Junio C HamanoAug 10, 2013
  22. Ramkumar RamachandraAug 10, 2013
  23. Duy NguyenAug 10, 2013
  24. Duy NguyenAug 10, 2013
  25. Jeff KingAug 10, 2013
  26. Duy NguyenAug 10, 2013
  27. Ramkumar RamachandraAug 10, 2013
  28. Duy NguyenAug 10, 2013
  29. Jeff KingAug 10, 2013
  30. Martin FickAug 7, 2013
  31. Duy NguyenAug 8, 2013
  32. Martin FickAug 7, 2013
  33. Ramkumar RamachandraAug 7, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.