git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC] Add --create-cache to repack

From
Shawn Pearce <spearce@spearce.org>
Date
Jan 30, 2011, 22:13 UTC
Message-ID
<AANLkTin6dcfMAD1tg+ROujy-_Dvi6KG-+-nfgG44u=Mh@mail.gmail.com>
In-Reply-To
<AANLkTi=U7qRRij=BQXC1Goqa9toDFfaVKT=+-8zYxCcc@mail.gmail.com>
On Fri, Jan 28, 2011 at 17:32, Shawn Pearce <spearce@spearce.org> wrote:
Show 6 quoted lines
>
> I fully implemented the reuse of a cached pack behind a thin pack idea
> I was trying to describe in this thread.  It saved 1m7s off the JGit
> running time, but increased the data transfer by 25 MiB.  I didn't
> expect this much of an increase, I honestly expected the thin pack
> portion to be well, thinner.
JGit's thin pack creation is crap.  For example, this is the same fetch:

$ git fetch ../tmp_linux26 remote: Counting objects: 61521, done. remote: Compressing objects: 100% (12096/12096), done. remote: Total 50275 (delta 42578), reused 45220 (delta 37524) Receiving objects: 100% (50275/50275), 11.13 MiB | 7.29 MiB/s, done. Resolving deltas: 100% (42578/42578), completed with 4968 local objects.

$ git fetch git://localhost/tmp_linux26 remote: Counting objects: 144190, done remote: Finding sources: 100% (50275/50275) remote: Compressing objects: 100% (106568/106568) remote: Compressing objects: 100% (12750/12750) Receiving objects: 100% (50275/50275), 24.66 MiB | 10.93 MiB/s, done. Resolving deltas: 100% (40345/40345), completed with 2218 local objects.

JGit produced an extra 13.53 MiB for this pack, because it missed about 2,233 delta opportunities. It turns out we are too aggressive at pushing objects from the edges into the delta windows. JGit pushes *everything* in the edge commits, rather than only the paths that are actually used by the objects we need to send. This floods the delta search window with garbage, and makes it less likely that an object to be sent will find a relevant delta base in the search window.

-- 
Shawn.
Previous: Shawn PearceNext: Shawn Pearce
Message 24 of 26 in “[RFC] Add --create-cache to repack”
  1. Shawn O. PearceJan 28, 2011
  2. Johannes SixtJan 28, 2011
  3. Shawn PearceJan 28, 2011
  4. Johannes SixtJan 28, 2011
  5. Shawn PearceJan 28, 2011
  6. Jay SoffianJan 28, 2011
  7. Shawn PearceJan 28, 2011
  8. Nicolas PitreJan 28, 2011
  9. Shawn PearceJan 28, 2011
  10. Nicolas PitreJan 28, 2011
  11. Shawn PearceJan 29, 2011
  12. Shawn PearceJan 29, 2011
  13. Junio C HamanoJan 30, 2011
  14. Shawn PearceJan 30, 2011
  15. Junio C HamanoJan 30, 2011
  16. Shawn PearceJan 30, 2011
  17. Nicolas PitreJan 30, 2011
  18. Nicolas PitreJan 29, 2011
  19. Shawn PearceJan 29, 2011
  20. Junio C HamanoJan 30, 2011
  21. Nicolas PitreJan 30, 2011
  22. A Large Angry SCMJan 30, 2011
  23. Shawn PearceJan 30, 2011
  24. Shawn PearceJan 30, 2011
  25. Shawn PearceJan 31, 2011
  26. Nicolas PitreJan 31, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.