{"thread":{"id":"6692","subject":"Why is git-clone --reference so slow?","startedAt":"2007-02-05T11:46:44Z","lastAt":"2007-02-07T10:03:01Z","messageCount":4,"participants":["David Woodhouse","Johannes Schindelin","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"33625","messageId":"1170676004.29759.767.camel@pmac.infradead.org","threadId":"6692","inReplyTo":null,"subject":"Why is git-clone --reference so slow?","fromName":"David Woodhouse","fromEmail":"dwmw2@infradead.org","sentAt":"2007-02-05T11:46:44Z","receivedAt":"2007-02-05T11:46:44Z","isPatch":false,"sender":{"key":"dwmw2@infradead.org","avatar":"https://gravatar.com/avatar/7afd4f07e0cf7d7e046ae2d23678296b37777c96488e6f3451e78a5514154ebd?d=mp&s=160"},"body":"My DSL line sucks; I know this. But why is git-clone so bad at using it?\n\nThis is slow and seems to be downloading a lot of stuff it doesn't\nneed...\n\npmac /pmac/git $ git-clone --reference /pmac/git/linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git\nremote: Generating pack...\nremote: Done counting 14428 objects.\nremote: Deltifying 14428 objects.\nremote:  100% (14428/14428) done\nIndexing 14428 objects.\n  25% (3662/14428) done\n\nAt this point I got bored and hit ^C, then logged in to hera and did it\nthere, then rsync'd the result...\n\nhera /home/dwmw2/x $ git-clone -n --bare --reference /pub/scm/linux/kernel/git/torvalds/linux-2.6.git /pub/scm/linux/kernel/git/geoff/ps3-linux.git \nremote: Generating pack...\nremote: Done counting 14490 objects.\nremote: Deltifying 14490 objects.\nremote:  100% (14490/14490) done\nIndexing 14490 objects.\nremote: Total 14490, written 14490 (delta 579), reused 5598 (delta 367)\n 100% (14490/14490) done\nResolving 579 deltas.\n 100% (579/579) done\n\npmac /pmac/git/ps3-linux $  rsync -avz master.kernel.org:/home/dwmw2/x/ps3-linux/ .git\npmac /pmac/git/ps3-linux $ echo /pmac/git/linux-2.6/.git/objects > .git/objects/info/alternates\n\nThat was a _lot_ faster, and abused my poor DSL line a lot less. Why's\ngit so bad at it?\n\n-- \ndwmw2\n"},{"id":"33634","messageId":"Pine.LNX.4.63.0702051629450.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6692","inReplyTo":"1170676004.29759.767.camel@pmac.infradead.org","subject":"Re: Why is git-clone --reference so slow?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-05T15:31:08Z","receivedAt":"2007-02-05T15:31:08Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 5 Feb 2007, David Woodhouse wrote:\n\n> My DSL line sucks; I know this. But why is git-clone so bad at using it?\n> \n> This is slow and seems to be downloading a lot of stuff it doesn't\n> need...\n> \n> pmac /pmac/git $ git-clone --reference /pmac/git/linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git\n\nIs this a Mac? Is it Mac OS X? Which git version do you use?\n\nWe recently had a case like this, and it was fixed: apparently, mmap() on \nMac OS X is less than fast...\n\nHth,\nDscho\n"},{"id":"33635","messageId":"1170689730.29759.822.camel@pmac.infradead.org","threadId":"6692","inReplyTo":"Pine.LNX.4.63.0702051629450.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Why is git-clone --reference so slow?","fromName":"David Woodhouse","fromEmail":"dwmw2@infradead.org","sentAt":"2007-02-05T15:35:30Z","receivedAt":"2007-02-05T15:35:30Z","isPatch":false,"sender":{"key":"dwmw2@infradead.org","avatar":"https://gravatar.com/avatar/7afd4f07e0cf7d7e046ae2d23678296b37777c96488e6f3451e78a5514154ebd?d=mp&s=160"},"body":"On Mon, 2007-02-05 at 16:31 +0100, Johannes Schindelin wrote:\n> Is this a Mac? Is it Mac OS X? Which git version do you use?\n> \n> We recently had a case like this, and it was fixed: apparently, mmap()\n> on Mac OS X is less than fast... \n\nYes, it's a Mac G5. No, it's not OSX; it's Fedora Core 6. As far as I\ncan tell it's downloading far more than it needs before discarding\nobjects which are already in the local repository used in --reference.\n\n\n-- \ndwmw2\n"},{"id":"33787","messageId":"7vhcty8eui.fsf@assigned-by-dhcp.cox.net","threadId":"6692","inReplyTo":"1170676004.29759.767.camel@pmac.infradead.org","subject":"Re: Why is git-clone --reference so slow?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-07T10:03:01Z","receivedAt":"2007-02-07T10:03:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Woodhouse <dwmw2@infradead.org> writes:\n\n> My DSL line sucks; I know this. But why is git-clone so bad at using it?\n\nI did the same and it initially got 45MB pack (14.5k objects).\n\nFor my case, the reason was because the repository I used as the\nreference was pruned and packed its refs, and git-clone was not\nfully using the refs from the reference repository.\n\nThe following patch made it to be almost instantaneous.  The\nresulting clone has 600kB pack (0.5k objects).\n\n\ndiff --git a/git-clone.sh b/git-clone.sh\nindex 1710996..1bd54de 100755\n--- a/git-clone.sh\n+++ b/git-clone.sh\n@@ -178,46 +178,32 @@ esac && export GIT_DIR && git-init ${template+\"$template\"} || usage\n \n if test -n \"$reference\"\n then\n+\tref_git=\n \tif test -d \"$reference\"\n \tthen\n \t\tif test -d \"$reference/.git/objects\"\n \t\tthen\n-\t\t\treference=\"$reference/.git\"\n+\t\t\tref_git=\"$reference/.git\"\n+\t\telif test -d \"$reference/objects\"\n+\t\tthen\n+\t\t\tref_git=\"$reference\"\n \t\tfi\n-\t\treference=$(cd \"$reference\" && pwd)\n-\t\techo \"$reference/objects\" >\"$GIT_DIR/objects/info/alternates\"\n-\t\t(cd \"$reference\" && tar cf - refs) |\n-\t\t(cd \"$GIT_DIR/refs\" &&\n-\t\t mkdir reference-tmp &&\n-\t\t cd reference-tmp &&\n-\t\t tar xf - &&\n-\t\t find refs ! -type d -print |\n-\t\t while read ref\n-\t\t do\n-\t\t\tif test -h \"$ref\"\n-\t\t\tthen\n-\t\t\t\t# Old-style symbolic link ref.  Not likely\n-\t\t\t\t# to appear under refs/ but we might as well\n-\t\t\t\t# deal with them.\n-\t\t\t\t:\n-\t\t\telif test -f \"$ref\"\n-\t\t\tthen\n-\t\t\t\tpoint=$(cat \"$ref\") &&\n-\t\t\t\t\tcase \"$point\" in\n-\t\t\t\t\t'ref: '*) ;;\n-\t\t\t\t\t*) continue ;;\n-\t\t\t\t\tesac\n-\t\t\tfi\n-\t\t\t# The above makes true ref to 'continue' and\n-\t\t\t# we will come here when we are looking at\n-\t\t\t# symbolic link ref or a textual symref (or\n-\t\t\t# garbage, like fifo).\n-\t\t\t# The true ref pointed at by it is enough to\n-\t\t\t# ensure that we do not fetch objects reachable\n-\t\t\t# from it.\n-\t\t\trm -f \"$ref\"\n-\t\t done\n-\t\t)\n+\tfi\n+\tif test -n \"$ref_git\"\n+\tthen\n+\t\tref_git=$(cd \"$ref_git\" && pwd)\n+\t\techo \"$ref_git/objects\" >\"$GIT_DIR/objects/info/alternates\"\n+\t\t(\n+\t\t\tGIT_DIR=\"$ref_git\" git for-each-ref \\\n+\t\t\t\t--format='%(objectname) %(*objectname)'\n+\t\t) |\n+\t\twhile read a b\n+\t\tdo\n+\t\t\ttest -z \"$a\" ||\n+\t\t\tgit update-ref \"refs/reference-tmp/$a\" \"$a\"\n+\t\t\ttest -z \"$b\" ||\n+\t\t\tgit update-ref \"refs/reference-tmp/$b\" \"$b\"\n+\t\tdone\n \telse\n \t\tdie \"reference repository '$reference' is not a local directory.\"\n \tfi\n"}]}