Re: dumb transports not being welcomed..

From: Linus Torvalds <torvalds@osdl.org>
Date: Sep 14, 2005, 15:07 UTC
Message-ID: <Pine.LNX.4.58.0509140759030.26803@g5.osdl.org>
In-Reply-To: <Pine.LNX.4.63.0509141014580.30708@wgmdd8.biozentrum.uni-wuerzburg.de>

On Wed, 14 Sep 2005, Johannes Schindelin wrote:

Show 6 quoted lines

> 
> What I see when fetching all heads (thanks to Junio, this is one call to 
> git-fetch now), where all but origin are up to date, is that it takes a 
> very long time. Swapping kicks in, and top tells me that 26.6% of the 
> memory is occupied by git-rev-list (The server has 128M, with 1G swap, and 
> I am unfortunately not the only user of this machine).

Ok. As mentioned, I've not looked at memory usage. The machines I play with tend to have 2GB or more, simply because bk needed at least 1GB to be nice and cached on the kernel ;)

Git has needed less than bk, so I've not cared ;)

> I fail to see why it should need those amounts of memory. (I tested this 
> over the ssh protocol, which should essentially do the same as git-daemon, 
> right?) After all, the merge point between the branches should be marked 
> uninteresting after one single step from each of my private branches.

One of the issues is that git-rev-list will (for example) keep track of the commit messages too for every commit. That in itself can be a lot of stuff, depending on how active the tree is and how large the messages are.

Now, that should be easy enough to fix (parse_commit() normally saves the buffer it parses into "commit->buffer", so we'd just need to do something like

	if (!verbose_header && commit->buffer) {
		free(commit->buffer);
		commit->buffer = NULL;
	}

for each commit.

But for --objects, the bigger memory pressure is that it needs to track the "struct object" for every single object when it generates the reference tracking. And THAT tends to be expensive. The object lists are also not very space-efficient (ie one small allocation for each list entry).

We could probably make objects/lists more space-efficient.

> I also see other strange things like packing 0 objects, and packing >0 
> objects after just having fetched from that repository. Hopefully I will 
> have time to look into that (and understand the code to begin with).

Well, the "packing 0 objects" should be normal. I'm surprised at the ">0" case after a fetch: the packign is _not_ guaranteed to be exact, but if you have the exact same state as (or a superset of) the other end, you should always see a zero.

		Linus