Re: git-clone --how-much-disk-space-will-this-cost-me? [--depth n]

From: Shawn O. Pearce <spearce@spearce.org>
Date: Dec 17, 2008, 15:44 UTC
Message-ID: <20081217154407.GZ32487@spearce.org>
In-Reply-To: <alpine.LFD.2.00.0812160039180.30035@xanadu.home>

Nicolas Pitre <nico@cam.org> wrote:

Show 14 quoted lines

> On Tue, 16 Dec 2008, Jean-Luc Herren wrote:
> > jidanni@jidanni.org wrote:
> > > JH> So maybe what you really want is an ETA display during the cloning
> > > JH> process?  Sounds like a good idea to me.
> 
> And then you'll end up being the unlucky bastard to be the first to 
> clones the new latest revision of a repository, and ETA won't be 
> available, and you'll complain about the fact that sometimes it is there 
> and sometimes it is not.
> 
> The fact is, fundamentally, we don't know how many bytes to push when 
> generating a pack to answer the clone request.  Sometimes we _could_ but 
> not always.  It is therefore better to be consistent and let people know 
> that there is simply no ETA.

Hmm.

What if on an initial clone (no "have" lines received) we sum up the sizes of the *.pack and all of the loose objects and sent that as an initial size estimate. Its going to be the upper bound of the final pack that we send. At worst it over-estimates on the size and download finishes faster.

I'm willing to bet that most of the "big" repositories out there don't have a lot of garbage in them. Linus' kernel repository doesn't rewind, so he has 0 garbage. Anyone cloning from him would get a reasonable estimate. Likewise with a Gentoo/KDE/WebKit/gcc sort of giant tree most of that is in a huge historical pack. That one pack file alone is completely reachable and dominates the transfer size.

On smaller trees where people may have a lot of rebase garbage or everything is loose the estimate will be quite a bit above what we transfer, but how much so that it matters?

Yea, a single stray binary of some *.mpg or *.iso accidentally added and then removed (and now unreachable) will vastly inflate the numbers. In which case the repository owner will be encouraged to prune when people won't clone his estimated 8 GiB download, which is actually only 1 MiB.

-- 
Shawn.