Re: [PATCH] diff-delta: produce optimal pack data

From: Carl Baldwin <cnb@fc.hp.com>
Date: Feb 24, 2006, 20:40 UTC
Message-ID: <20060224204022.GA15962@hpsvcnb.fc.hp.com>
In-Reply-To: <Pine.LNX.4.64.0602241438521.31162@localhost.localdomain>

On Fri, Feb 24, 2006 at 03:02:07PM -0500, Nicolas Pitre wrote:

Show 5 quoted lines

> Well that is probably a bit tight.  Ideally it should be linear with the 
> size of the data set to process.  If you have 10 files 10MB each it 
> should take about the same time to pack than 10000 files of 10KB each.  
> Of course incrementally packing one additional 10MB file might take more 
> than a second although it is only one file.

Well, I might not have been fair here. I tried an experiment where I packed each of the twelve large blob objects explicitly one-by-one using git-pack-objects. Incrementally packing each single object was very fast. Well under a second per object on my machine.

After the twelve large objects were packed into individual packs the rest of the packing went very quickly and git v1.2.3's date reuse worked very well. This was sort of my attempt at simulating how things would be if git avoided deltification of each of these large files. I'm sorry to have been so harsh earlier I just didn't understand that incrementally packing one-by-one was going to help this much.

This gives me hope that if somehow git were to not attempt to deltify these objects then performance would be much better than acceptible.

[snip]

> However, if you could let me play with two samples of your big file I'd 
> be grateful.  If so I'd like to make git work well with your data set 
> too which is not that uncommon after all.

I would be happy to do this. I will probably need to scrub a bit and make sure that the result shows the same characteristics. How would you like me to deliver these files to you? They are about 25 MB deflated.

Carl

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        RADCAD (R&D CAD)
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -