Restraining git pull/fetch to the current branch

While trying out git on a large repository (10000s of commits, 1000s of branches, ~2.5Gb when packed) at work I noticed that doing a pull was taking a long time (longer than I was prepared to wait anyway).

A quick test showed that a small repository (1 commit, 24k .git/objects) with 1000 branches took 1m30 to do "git pull" (local xfs partition). I don't know if this is reasonable or not, but all I actually cared about was updating the current branch, which "git pull origin <branch_name>" did in 0.3s.

So what I would like to know is: is there any way to make a pull/fetch with no options default to only fetching the current branch? (other than scripting "git pull/fetch origin $(git symbolic-ref HEAD)" that is)

TIA

Julian Phillips <julian@quantumfyre.co.uk> wrote:

No, but fortunately bash has a fancy alias tool:

	alias gp='git pull origin $(git symbolic-ref HEAD)'

perhaps your shell can help.  :-)

Life is going to be painful with that repository with current Git (1.5.0 and later) as the new default configuration for a clone is to copy every branch into refs/remotes/origin/*, where * is wildcarded against the current set of branches on the remote repository. If that takes "a long time" you will be processing a lot of refs you don't care about (or need to care about).

Julian Phillips <julian@quantumfyre.co.uk> writes:

Are they all real branches? In other words, does your project have 1000s of active parallel development?

Also, assuming the answer to the above question is yes, will you have 1000s of branches on your end and will work on any one of them?

The default configuration created by git-clone makes you track all branches from the remote side by putting:

	remote.origin.fetch = +refs/heads/*:refs/remotes/origin/*

If you do not care all 1000s branches but only are interested in selected few, you could change that configuration to suit your needs better.

    remote.origin.fetch = +refs/heads/stable:refs/remotes/origin/stable
    remote.origin.fetch = +refs/heads/testing:refs/remotes/origin/testing
    remote.origin.fetch = +refs/heads/unstable:refs/remotes/origin/unstable

I suspect most of the time is being spent in the append-fetch-head loop in fetch_main shell function in git-fetch.sh The true fix would not be to limit the number of branches updated, but to speed that part of the code up.

On Thu, 11 Jan 2007, Junio C Hamano wrote:

(Oops, over enthusiastic with the 0 there, I mean 100s of branches, about 880 atm).

They are mostly topic style branches, with only 20 or so in active use at any one time. The idea of having to cope with 100s of active branches at the same time (given that we currently are using subversion) is quite frankly terrifying.

It would be necessary to have access to all of the currently active branches at least, with the added complication that the set of current active branches changes quite rapidly.

I think the problem here would be keeping track of which branches are currently active. Some scheme could probably be derived, but I was hoping that fetching an unchanged branch would be sufficently fast that it would be necessary. I appear to have been wrong :(

Indeed, each call to append_fetch_head is taking ~1.7s (~1.5s user, ~0.2s sys). So simply looping over all the branches explains the ~27m that a complete fetch takes. (This is for fetch with no updates). Given that a "clone orig new" takes ~8m30 (half of which would seem to be IO), it looks like it may be faster to create a new repository each time instead of updating the old one, which is certainly a viable workaround - but might imply that fetch has some room for improvement?

On Fri, 12 Jan 2007, Julian Phillips wrote:

I have had a chance to spend a little more time looking at this. It would appear that the major culprit is show-ref.

Running "git show-ref --hash <ref>" takes ~1.7s, compared to 0.002s for "cat $GIT_DIR/<ref>". If I add the following to the top of append_fetch_head a null fetch takes 1m28s instead of ~27m.

local_head_=$(cat $GIT_DIR/$local_name_);
if [ "$head_" == "$local_head_" ]; then
 	return;
fi

Looking at the code for show-ref it appears to looks at all the refs to find the one you ask for. This makes fetch O(n^2) in no of branches, which would seem not strictly necessary - but then I am not really familiar with the internal working of git. I noticed that the man page for show-ref says that its use over direct access is encouraged, but as it stands it is far too slow to be used in fetch when you have a large many-branched repository...

(In looking at this I also discovered that if you have too many branches then fetch will die with a too long command line error when calling git-fetch-pack.)