# Why is git-clone --reference so slow? 4 messages from 2007-02-05 to 2007-02-07. Participants: David Woodhouse, Johannes Schindelin, Junio C Hamano. Thread: https://gitlist.dev/t/6692 ## David Woodhouse, 2007-02-05 11:46 Subject: Why is git-clone --reference so slow? Message-ID: <1170676004.29759.767.camel@pmac.infradead.org> URL: https://gitlist.dev/e/1170676004.29759.767.camel%40pmac.infradead.org ``` My DSL line sucks; I know this. But why is git-clone so bad at using it? This is slow and seems to be downloading a lot of stuff it doesn't need... pmac /pmac/git $ git-clone --reference /pmac/git/linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git remote: Generating pack... remote: Done counting 14428 objects. remote: Deltifying 14428 objects. remote: 100% (14428/14428) done Indexing 14428 objects. 25% (3662/14428) done At this point I got bored and hit ^C, then logged in to hera and did it there, then rsync'd the result... hera /home/dwmw2/x $ git-clone -n --bare --reference /pub/scm/linux/kernel/git/torvalds/linux-2.6.git /pub/scm/linux/kernel/git/geoff/ps3-linux.git remote: Generating pack... remote: Done counting 14490 objects. remote: Deltifying 14490 objects. remote: 100% (14490/14490) done Indexing 14490 objects. remote: Total 14490, written 14490 (delta 579), reused 5598 (delta 367) 100% (14490/14490) done Resolving 579 deltas. 100% (579/579) done pmac /pmac/git/ps3-linux $ rsync -avz master.kernel.org:/home/dwmw2/x/ps3-linux/ .git pmac /pmac/git/ps3-linux $ echo /pmac/git/linux-2.6/.git/objects > .git/objects/info/alternates That was a _lot_ faster, and abused my poor DSL line a lot less. Why's git so bad at it? -- dwmw2 ``` ## Johannes Schindelin, 2007-02-05 15:31 Subject: Re: Why is git-clone --reference so slow? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0702051629450.22628%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <1170676004.29759.767.camel@pmac.infradead.org> ``` Hi, On Mon, 5 Feb 2007, David Woodhouse wrote: > My DSL line sucks; I know this. But why is git-clone so bad at using it? > > This is slow and seems to be downloading a lot of stuff it doesn't > need... > > pmac /pmac/git $ git-clone --reference /pmac/git/linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git Is this a Mac? Is it Mac OS X? Which git version do you use? We recently had a case like this, and it was fixed: apparently, mmap() on Mac OS X is less than fast... Hth, Dscho ``` ## David Woodhouse, 2007-02-05 15:35 Subject: Re: Why is git-clone --reference so slow? Message-ID: <1170689730.29759.822.camel@pmac.infradead.org> URL: https://gitlist.dev/e/1170689730.29759.822.camel%40pmac.infradead.org In-Reply-To: ``` On Mon, 2007-02-05 at 16:31 +0100, Johannes Schindelin wrote: > Is this a Mac? Is it Mac OS X? Which git version do you use? > > We recently had a case like this, and it was fixed: apparently, mmap() > on Mac OS X is less than fast... Yes, it's a Mac G5. No, it's not OSX; it's Fedora Core 6. As far as I can tell it's downloading far more than it needs before discarding objects which are already in the local repository used in --reference. -- dwmw2 ``` ## Junio C Hamano, 2007-02-07 10:03 Subject: Re: Why is git-clone --reference so slow? Message-ID: <7vhcty8eui.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vhcty8eui.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <1170676004.29759.767.camel@pmac.infradead.org> ``` David Woodhouse writes: > My DSL line sucks; I know this. But why is git-clone so bad at using it? I did the same and it initially got 45MB pack (14.5k objects). For my case, the reason was because the repository I used as the reference was pruned and packed its refs, and git-clone was not fully using the refs from the reference repository. The following patch made it to be almost instantaneous. The resulting clone has 600kB pack (0.5k objects). diff --git a/git-clone.sh b/git-clone.sh index 1710996..1bd54de 100755 --- a/git-clone.sh +++ b/git-clone.sh @@ -178,46 +178,32 @@ esac && export GIT_DIR && git-init ${template+"$template"} || usage if test -n "$reference" then + ref_git= if test -d "$reference" then if test -d "$reference/.git/objects" then - reference="$reference/.git" + ref_git="$reference/.git" + elif test -d "$reference/objects" + then + ref_git="$reference" fi - reference=$(cd "$reference" && pwd) - echo "$reference/objects" >"$GIT_DIR/objects/info/alternates" - (cd "$reference" && tar cf - refs) | - (cd "$GIT_DIR/refs" && - mkdir reference-tmp && - cd reference-tmp && - tar xf - && - find refs ! -type d -print | - while read ref - do - if test -h "$ref" - then - # Old-style symbolic link ref. Not likely - # to appear under refs/ but we might as well - # deal with them. - : - elif test -f "$ref" - then - point=$(cat "$ref") && - case "$point" in - 'ref: '*) ;; - *) continue ;; - esac - fi - # The above makes true ref to 'continue' and - # we will come here when we are looking at - # symbolic link ref or a textual symref (or - # garbage, like fifo). - # The true ref pointed at by it is enough to - # ensure that we do not fetch objects reachable - # from it. - rm -f "$ref" - done - ) + fi + if test -n "$ref_git" + then + ref_git=$(cd "$ref_git" && pwd) + echo "$ref_git/objects" >"$GIT_DIR/objects/info/alternates" + ( + GIT_DIR="$ref_git" git for-each-ref \ + --format='%(objectname) %(*objectname)' + ) | + while read a b + do + test -z "$a" || + git update-ref "refs/reference-tmp/$a" "$a" + test -z "$b" || + git update-ref "refs/reference-tmp/$b" "$b" + done else die "reference repository '$reference' is not a local directory." fi ```