Re: [PU PATCH] Fix git fetch for very large ref counts

From: Junio C Hamano <junkio@cox.net>
Date: Feb 13, 2007, 03:18 UTC
Message-ID: <7vd54e4ufk.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <11713297014015-git-send-email-julian@quantumfyre.co.uk>

"Julian Phillips" <jp3@quantumfyre.co.uk> writes:

Show 12 quoted lines

> The updated git fetch in pu is vastly improved on repositories with very
> large numbers of refs.  The time taken for a no-op fetch over ~9000 refs
> drops from ~48m to ~0.5m.
>
> However, before git fetch will actually run on a repository with ~9000
> refs the calling interface between fetch and fetch--tool needs to be
> changed.  The existing version passes the entire reflist on the command
> line, which means that it is subject to the maxiumum environment size
> passed to a child process by execve.
>
> The following patches add a stdin based interface to fetch--tool allowing
> the ~9000 refs to be passed without exceeding the environment limit.

Thanks.

But the ones in 'pu' were done primarily as demonstration of where the bottlenecks are, and not meant for real-world consumption. I think the final shaving of 0.5m down to a few seconds needs to move the ls_remote_result string currently kept as a shell variable to a list of strings represented in a git-fetch largely rewritten in C, and at that point the interface from outside fetch--tool to throw 9000 refs at it would be an internal function call and the code you fixed along with new function you introduced would probably need to be discarded.