Re: [PATCH] pack-objects: reuse data from existing pack.

Andreas Ericsson <ae@op5.se> writes:

I wished the pack format were not so dense as we have today. It is very expensive to obtain the uncompressed size of a deltified object. For this reason, a delta newly created (either from a non-delta in an existing pack or from a loose object) by the experimental algorithm is never made against an object that is in deltified form in a pack. Also it incurs nontrivial cost to obtain the size of the in-pack representation of an object (either deltified or not). But the inefficiency in the resulting pack due to these factors may not matter in practice.

I just packed v2.6.16-rc3 object list (184141 objects) using the current and the experimental, just for fun. Tonight's one runs just under 1 minutes on my Duron 750 (with slow disks I should add). This was done in a repository that has about 1500 loose objects and a single mega pack; reuse rate of packed data by the experimental algorithm is about 99%. I am hoping the one from the "master" would come back before I finish writing this message ;-).

There are subtleties.

For example, in a typical project, files tend to grow rather than shrink on average, and older ones tend to be in packs. If you do packing the traditional way, the largest one (which is typically the latest) is kept as non-delta, and all the smaller ones will be incremental delta from that, no matter how your packs and loose objects are organized.

Usually, you have the latest objects as loose objects in your repository to be packed (either you push from it, or somebody else pulls from you). In other words, as you develop after your last repack, you would accumulate loose objects, and they are the ones that typically matter the most.

Let's say you have been changing the same file in every commit (1..N), then you fully packed and then created another commit (revision N+1) that touches the file. The experimental reuse-packer would:

    - notice blobs from revision 1..(N-1) are deltified,
      relative to the rev one greater than each of them.
      these would be reused.
    - notice blob from revision N is in the pack but not
      deltified.
    - notice blob from revision N+1 is loose.

Then emit the bigger one between N or N+1 as non-delta, the other one as delta. 1..(N-1) are output as delta. If it happens to choose N as plain, it does not have to uncompress and recompress so the pack process would go very fast, but you would end up always having to apply a delta to bring rev N to N+1 on top of the non-delta N to get to the latest blob in rev N+1, and you typically would want to access rev N+1 blob more often.

In other words, the experimental reuse-packer would create a suboptimal pack in such a case. Not a big deal, though.

We may want an option to disable the optimization for weekly/monthly repacking. git-daemon (or whatever runs pack-objects via upload-pack) should use the default with the optimization, since this is so obviously faster.

I would say an optimum single mega-pack would work the best, but "repack -a -d" to create the mega-pack _with_ the optimization may have performance impact for users of resulting packs.

Oh, the traditional one finally came back after 11 minutes.