Re: git-index-pack really does suck..

From: Nicolas Pitre <nico@cam.org>
Date: Apr 3, 2007, 16:40 UTC
Message-ID: <alpine.LFD.0.98.0704031235490.28181@xanadu.home>
In-Reply-To: <Pine.LNX.4.64.0704030913060.6730@woody.linux-foundation.org>

On Tue, 3 Apr 2007, Linus Torvalds wrote:

Show 12 quoted lines

> 
> 
> On Tue, 3 Apr 2007, Linus Torvalds wrote:
> > 
> > and it uses 52s of CPU-time, and on my 4GB machine it actually started 
> > doing IO and swapping, because git-index-pack grew to 4.8GB in size.
> 
> Ahh. False alarm.
> 
> The problem is actually largely a really stupid memory leak in the SHA1 
> collision checking (which wouldn't trigger on a normal pull, but obviously 
> does trigger for every single object when testing!)

Damn!

Please don't report those things when I'm out for lunch so I could have a chance to fix my own stupidities myself in time! :-)

Show 5 quoted lines

> This trivial patch fixes most of it. git-index-pack still uses too much 
> memory, but it does a *lot* better.
> 
> Junio, please get this into 1.5.1 (I *think* the SHA1 checking is new, but 
> if it exists in 1.5.0 too, it obviously needs the same fix).

No, it is new to 1.5.1.

Show 9 quoted lines

> It still grows, but it grew to just 287M in size now for the 170M kernel 
> object:
> 
> 	41.59user 1.39system 0:43.64elapsed
> 	0major+73552minor
> 
> which is quite a lot better.
> 
> Duh.

Indeed.

Acked-by: Nicolas Pitre <nico@cam.org>

Show 20 quoted lines

>  index-pack.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/index-pack.c b/index-pack.c
> index 6284fe3..3c768fb 100644
> --- a/index-pack.c
> +++ b/index-pack.c
> @@ -358,6 +358,7 @@ static void sha1_object(const void *data, unsigned long size,
>  		if (size != has_size || type != has_type ||
>  		    memcmp(data, has_data, size) != 0)
>  			die("SHA1 COLLISION FOUND WITH %s !", sha1_to_hex(sha1));
> +		free(has_data);
>  	}
>  }
>  
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Nicolas