git push sends more objects than it needs to

I just pulled from Linus' Linux repository ... and my wrapper script alerted me to the fact that he had just re-packed.

So, I logged into kernel.org and linked the new packfiles across to my repo there, and did a "git prune-packed" to clean away all the unneeded unpacked objects.

Back on my local box I merged all the new stuff that had been pulled into my "release" and "test" branches. (just a fast-forward for the release branch).

Next I used "scp" to copy Linus' new pack files to my local objects/pack ... and ran git prune packed here.

Finally, after making sure that release and test
branches still built with all the new stuff I did
a "git push" to update my kernel repo.  I saw this:
updating 'refs/heads/test'
  from 0711c558ff3cf4d1e498a9686f4d8ce48aa8d79f
  to   2e488d3f8f299eb0ed826055bc30003e2c27d909
updating 'refs/heads/release'
  from 8a212ab6b8a4ccc6f3c3d1beba5f92655c576404
  to   581c1b14394aee60aff46ea67d05483261ed6527
Packing 10785 objects
Unpacking 10785 objects
 100% (10785/10785) done
refs/heads/test: 0711c558ff3cf4d1e498a9686f4d8ce48aa8d79f -> 2e488d3f8f299eb0ed826055bc30003e2c27d909
refs/heads/release: 8a212ab6b8a4ccc6f3c3d1beba5f92655c576404 -> 581c1b14394aee60aff46ea67d05483261ed6527

Now the "unpack" on kernel.org did the right thing and noticed that over 9000 of the objects were already in the packfile. But I wonder if it couldn't have been smarter and not sent them?

Or am I just subverting the whole paradigm by hand-copying packfiles around?

-Tony

On Mon, 31 Oct 2005, Luck, Tony wrote:

...

It should have been smarter, but I suspect you got caught by the fact that kernel.org by default has git-0.99.8f on it, which has the old pre-multi_ack code to figure out what the common commit was.

The pack-file contains over ten times as many objects, so you definitely didn't get all of them - but because there has been a lot of merges lately, and the common commit finder algorithm wasn't all that careful, you ended up getting many more objects than you really needed.

Remember: the pack generation is not "exact" - it will often generate a 
few extra objects for any non-trivial case (for example, it fundamentally 
happens if there has been reverts: it won't realize that you had the older 
version of a file already). It just so happens that the old algorithm had 
some cases where it would decide on totally the wrong common commit, and 
re-send a _lot: more objects than it needs.

I'm not sure multi-ack fixes it entirely either, but I think it makes it a lot less likely (but even with multi-ack, the "file revert" case still happens, so you should always expect that can get a _couple_ of unnecessary objects).

Now, even the old stupid algorithm got the _easy_ cases obviously right, so people might have incorrectly gotten the idea that it was careful and exact, just because quite often it ends up being that in practice.

		Linus

On Mon, 31 Oct 2005, Linus Torvalds wrote:

Oh, actually, I take that back.

This is not the multi-ack code at all, I suspect.

The problem is totally different: you copied the new pack to your master.kernel.org repository, but you never updated any branches there.

So you had the objects, but git had no way of knowing. The pack generation doesn't look at what _objects_ you have, it looks at what _refs_ you have. And your refs were all to the old state.

So git actually did everything right (you can never rely on objects: you may have a partial object list due to some earlier incomplete pull/push).

This is actually easy enough to fix up in one of several ways:

 a) The "don't do that then" approach:

    Don't go behind git's back and add objects on your own, and expect git 
    to realize what you did. ;^p

 b) The "live with it" approach:

    You copied the pack by hand, and that will keep git-unpack-objects 
    from duplicating the objects, but you'll still waste time and network 
    when trying pushing the objects (just once, though).

    Ie this is what happened this time: nothing really lost, and the end 
    result is fine. Now you know why it happened, and you're fine.

 c) The "I'm smarter than git" approach:

    When you copy my objects, copy my reference to the top-of-tree too 
    (and rename it). NOTE! Now you need to be really careful, and you need 
    to make sure you copy _all_ the objects, because if you screw this up, 
    your repo will be missing objects that you claim are there, and it 
    will be all your fault.

    I really don't advocate this approach at all. It's certainly doable, 
    but it's also the only approach where you can really screw up.

 d) Just let git do it for you.

    Copy the pack-files, or add my object directory as an "alternates" for 
    your object directory, do the "git prune-objects", and then _locally_ 
    on master.kernel.org just do something like

	git fetch ..linus-directory.. master:linus

    which will still create the unnecessary pack-file and unpack it into 
    nothingness (since you have the objects in the pack-file you copied by 
    hand), but at least it won't eat any network bandwidth, and it will do 
    the right thing if it turns out that I've pushed something after doing 
    the pack-file, and fetch those individual objects in _addition_ to the 
    pack-file you snarfed by hand. It will also obviously update a ref in
    your tree (the "linus" branch), so now when you send stuff later, it 
    will know all about the objects you already have.

 e) Re-create the tree entirely

    Blow away your tree on master entirely, just re-create it locally with 
    "git clone -l -s" from my tree (which will do all the "alternates" 
    object files for you), and then populate the result with a simple push
    from your home tree.

 f) any number of variations on a theme. IOW, there are endless ways to do 
    this. 

Hmm?

		Linus

On Mon, Oct 31, 2005 at 11:36:09AM -0800, Linus Torvalds wrote:

To be a responsible kernel.org git user I need to use packfiles. The cheapest and easiest way to do that is to steal them from you.

I like this one best. There's a pretty fat pipe between here and kernel.org ... more of the time wasted appeared to be on the unpacking part than on the transfer.

Longer term (when everyone is pulling using the "git" protocol) adding your object directory to my info/alternates looks to be the best thing.

-Tony

"Luck, Tony" <tony.luck@intel.com> writes:

The fix for this problem is in 0.99.9 (credits to Johannes). Unfortunately, 0.99.8f is still in /usr/bin/ on kernel.org.