Re: Why repository grows after "git gc"? / Purpose of *.keep files?
- From
Nicolas Pitre <nico@cam.org>
- Date
- May 14, 2008, 01:03 UTC
- Message-ID
- <alpine.LFD.1.10.0805132005550.23581@xanadu.home>
- In-Reply-To
- <20080513001252.GB29038@spearce.org>
On Mon, 12 May 2008, Shawn O. Pearce wrote:
Show 19 quoted lines
> Mike Hommey <mh@glandium.org> wrote: > > On Mon, May 12, 2008 at 11:03:04PM +0200, Mike Hommey wrote: > > > On Mon, May 12, 2008 at 11:24:14PM +0300, Teemu Likonen wrote: > > > > But I have experienced the same earlier with some other post-1.5.5 > > > > version so I believe you can reproduce this yourself. After cloning > > > > Linus's linux-2.6 repo its .git directory weights 209MB. After single > > > > "git pull" and "git gc" it was 298MB in my test. > > > > > > I noticed that a while ago: when repacking multiple packs when one has a > > > .keep file, the resulting additional pack contains too many blobs and > > > trees, contrary to when only packing loose objects: > > (...) > > > > That is, it seems to also contain all the blobs and subtrees for all the > > commits the pack contains, even when they already are in the pack having > > a .keep file. > > I've noticed this too. Like since day 1 when we added .keep. > But uh, nobody else complained and I forgot about it.
Well, now that I've reproduced Teemu Likonen's test case, I can confirm this is actually a problem. Here I get:
|remote: Counting objects: 523, done. |remote: Compressing objects: 100% (57/57), done. |remote: Total 362 (delta 305), reused 362 (delta 305) |Receiving objects: 100% (362/362), 65.37 KiB, done. |Resolving deltas: 100% (305/305), completed with 105 local objects. |From ../test1 | 492c2e4..9404ef0 master -> master
The received pack is 449135 bytes large. This is much larger than the actually received data which is 65.37 KiB, but we're completing a thin pack with 105 undeltified objects accounting for the size increase which is expected. So far so good.
Now, in theory, running 'git gc' should only repack those 362 + 105 objects, since the remaining ones are all found in the .keep flagged pack. But that's not what's happening at all:
|Counting objects: 26559, done. |Compressing objects: 100% (24708/24708), done. |Writing objects: 100% (26559/26559), done. |Total 26559 (delta 3054), reused 14011 (delta 1613)
So... there is something definitively wrong here. The expectation was to get a pack in the same size range as the one received during the pack, or somewhat smaller due to a better delta compression of the added objects. But instead we get a pack containing 26559 objects!!! And in that lot, only 3054 (11%) are deltas. That makes for a pack that started from 449135 bytes and grew to 72395940 bytes.
Show 5 quoted lines
> My theory (totally unproven) is that the new pack has objects we > copied from the .keep pack, because those objects were the best > delta-bases for the loose objects we have deltafied and want to > store in the new pack. Except they aren't yet packed in the new > pack, so we pack them too. Tada, duplicates. :-\
Well, not exactly.
Let's see what happens here even before any packing is attempted
|$ git rev-list --objects 492c2e4..9404ef0 |362 | |$ git rev-list --objects --all \ | --unpacked=pack-6a3438b2702be06697023d80b77e67a73a0b0b5c.pack | | wc -l |26559
So this --unpacked= argument (which undocumented semantics I still have issues with) is certainly not doing what is expected.
Nicolas