[RFC] Speeding up a null fetch
- From
Julian Phillips <julian@quantumfyre.co.uk>
- Date
- Feb 11, 2007, 23:32 UTC
- Message-ID
- <200702112332.14698.julian@quantumfyre.co.uk>
I was investigating replacing an existing subversion setup with git, and was mostly pleased with the results - until it came to trying to update a clone ... which took very much longer than the original clone.
An artifical test repository that has similar features (~25000 commits, ~8000 tags, ~900 branches and a 2.5Gb packfile) when running locally takes ~20m to clone and ~48m to fetch (with no new commits in the original repository - i.e. the fetch does not update anything) with a current code base (i.e. newer than 1.5.0-rc4). As a side note, performance was actually better with an older version - packed refs makes things quite a bit worse (clone was only ~30m with 1.4 IIRC).
Investigation showed that the main culprit seemed to be show-ref having to build a sorted list of all refs for every ref that was being checked. So I used the patch below to reduce this to a single call to show-ref (unless the ref had been updated). With this patch the fetch timed dropped to just under 1m - obviously quite a lot faster (better than I expected in fact).
However, this seems more band-aid than fix, and I wondered if someone more familiar with the git internals could point me in the right direction for a better fix, e.g. should I look at rewriting fetch in C?
diff --git a/Makefile b/Makefile index 5d31e6d..6baf043 100644 --- a/Makefile +++ b/Makefile @@ -120,7 +120,7 @@ ALL_CFLAGS = $(CFLAGS) ALL_LDFLAGS = $(LDFLAGS) STRIP ?= strip -prefix = $(HOME) +prefix = $(HOME)/git bindir = $(prefix)/bin gitexecdir = $(bindir) template_dir = $(prefix)/share/git-core/templates/ @@ -188,7 +188,7 @@ SCRIPT_PERL = \ SCRIPTS = $(patsubst %.sh,%,$(SCRIPT_SH)) \ $(patsubst %.perl,%,$(SCRIPT_PERL)) \ - git-cherry-pick git-status git-instaweb + git-cherry-pick git-status git-instaweb git-ref-diff.py # ... and all the rest that could be moved out of bindir to gitexecdir PROGRAMS = \ diff --git a/git-fetch.sh b/git-fetch.sh index 357cac2..ce135a5 100755 --- a/git-fetch.sh +++ b/git-fetch.sh @@ -108,11 +108,12 @@ ls_remote_result=$(git ls-remote $exec "$remote") || append_fetch_head () { head_="$1" - remote_="$2" - remote_name_="$3" - remote_nick_="$4" - local_name_="$5" - case "$6" in + local_head_="$2" + remote_="$3" + remote_name_="$4" + remote_nick_="$5" + local_name_="$6" + case "$7" in t) not_for_merge_='not-for-merge' ;; '') not_for_merge_= ;; esac @@ -151,10 +152,15 @@ append_fetch_head () { echo "$head_ not-for-merge $note_" >>"$GIT_DIR/FETCH_HEAD" fi - update_local_ref "$local_name_" "$head_" "$note_" + update_local_ref "$local_name_" "$head_" "$note_" "$local_head_" } update_local_ref () { + if [ "$2" == "$4" ]; then + [ "$verbose" ] && echo >&2 "* $1: same as $3" + return 0 + fi + # If we are storing the head locally make sure that it is # a fast forward (aka "reverse push"). @@ -392,7 +398,7 @@ fetch_main () { ( git-fetch-pack --thin $exec $keep $shallow_depth "$remote" $rref || echo failed "$remote" - ) | + ) | git-ref-diff.py "$reflist" | ( trap ' if test -n "$keepfile" && test -f "$keepfile" @@ -402,7 +408,7 @@ fetch_main () { ' 0 keepfile= - while read sha1 remote_name + while read sha1 remote_name local_sha1 do case "$sha1" in failed) @@ -441,7 +447,7 @@ fetch_main () { esac done local_name=$(expr "z$found" : 'z[^:]*:\(.*\)') - append_fetch_head "$sha1" "$remote" \ + append_fetch_head "$sha1" "$local_sha1" "$remote" \ "$remote_name" "$remote_nick" "$local_name" \ "$not_for_merge" || exit done diff --git a/git-ref-diff.py b/git-ref-diff.py new file mode 100755 index 0000000..2b30e4c --- /dev/null +++ b/git-ref-diff.py @@ -0,0 +1,33 @@ +#!/usr/bin/python + +import os +import re +import sys + +ref_map_re = re.compile("^\.?\+?(?P<remote>.*?):(?P<local>.*)$") + +refs = {} +refsp = os.popen("git-show-ref") +for ref in refsp.readlines(): + (sha, ref) = ref.strip().split(' ') + refs[ref] = sha +refsp.close() + +ref_map = {} +for line in sys.argv[1].split('\n'): + ref_map_m = ref_map_re.search(line) + if ref_map_m: + remote = ref_map_m.group('remote') + local = ref_map_m.group('local') + ref_map[remote] = local + +while True: + try: + (sha, ref) = raw_input().split(' ') + except EOFError: + sys.exit(0) + lref = ref_map.get(ref, None) + if refs.has_key(lref): + print "%s %s %s" % (sha, ref, refs[lref]) + else: + print "%s %s -" % (sha, ref)
-- Julian --- Why bother building any more nuclear warheads until we use the ones we have?