git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Weird growth in packfile during initial push

From
Junio C Hamano <gitster@pobox.com>
Date
Apr 29, 2009, 23:57 UTC
Message-ID
<7vy6tj109a.fsf@gitster.siamese.dyndns.org>
In-Reply-To
<alpine.LFD.2.00.0904151443030.6741@xanadu.home>
Nicolas Pitre <nico@cam.org> writes:
Show 36 quoted lines
>> $ git push origin master:master
>> Initialized empty Git repository in /var/gitroot/exp/gentoo-x86.git/
>> Counting objects: 4969800, done.
>> Delta compression using up to 8 threads.
>> Compressing objects: 100% (1217809/1217809), done.
>> Writing objects: 100% (4969800/4969800), 810.56 MiB | 21608 KiB/s, done.
>> Total 4969800 (delta 3735812), reused 4969800 (delta 3735812)
>
> Here we know for sure that all objects were directly reused, so no 
> attempt at recompressing them was done.  The only thing that 
> pack-objects might do in this case in addition to directly streaming the 
> existing pack is to convert delta object headers from OFS_DELTA to 
> REF_DELTA.
>
>> $ ls -la /var/gitroot/exp/gentoo-x86.git/objects/pack
>> total 966876
>> drwxr-xr-x 2 git git      4096 Apr 14 08:43 .
>> drwxr-xr-x 4 git git      4096 Apr 14 08:35 ..
>> -r--r--r-- 1 git git 139155472 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.idx
>> -r--r--r-- 1 git git 849936308 Apr 14 08:43 pack-f805bb448f864becfeac9c7f8a8ac2ef90c26787.pack
>
> Let's see if my theory stands:
>
> 	849936308 - 786336481 = 63599827
> 	63599827 / 3735812 = 17.02
>
> Hence an average difference of 17 bytes per delta.  Given that REF_DELTA 
> objects have a 20-byte SHA1 base reference which is replaced with a 
> variable length encoding of a pack offset in the OFS_DELTA case, we're 
> talking about 2.98 bytes for that offset encoding which feels about 
> right.
>
> [...]
>
> And the code matches this theory as well.  Can you try this patch if you 
> have a chance?
Is there any progress on this?

I think you did a veryclear analysis. 8% size reduction is not only unignorable but use of delta offset should also help runtime efficiency, right?

Previous: Nicolas PitreNext: Nicolas Pitre
Message 3 of 16 in “Weird growth in packfile during initial push”
  1. Robin H. JohnsonApr 15, 2009
  2. Nicolas PitreApr 15, 2009
  3. Junio C HamanoApr 29, 2009
  4. Nicolas PitreApr 30, 2009
  5. Robin H. JohnsonMay 1, 2009
  6. allow OFS_DELTA objects during a pushNicolas Pitre, May 1, 2009
  7. Junio C HamanoMay 1, 2009
  8. Compatibility between git.git and jgitShawn O. Pearce, May 2, 2009
  9. A Large Angry SCMMay 2, 2009
  10. Nicolas PitreMay 2, 2009
  11. Shawn O. PearceMay 2, 2009
  12. Ealdwulf WuffingaMay 2, 2009
  13. Michael WittenMay 2, 2009
  14. Nicolas PitreMay 2, 2009
  15. Shawn O. PearceMay 4, 2009
  16. Shawn O. PearceMay 4, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.