threads / discuss / 1104

expensive local git clone

Subject: expensive local git clone

## tl;dr

14 messages between Jul 4, 2005 and Jul 6, 2005.

replies: 13people: 4as markdown or json

David S. Miller· Jul 4, 2005, 19:57 UTC · lore

I'm still seeing git-local-pull going crazy when attemping local clones of kernel trees, this is from top:

22032 davem     18   0 63392  59m  920 R 76.0  5.9   5:33.15 git-local-pull     
and it's still chugging along.  This is from a simple:
bash$ git-clone-script /home/davem/src/GIT/linux-2.6/.git /home/davem/src/GIT/sparc-2.6
invocation.

I can't be the only person in the world seeing this :-) It usually grows to around 88MB and runs for 8 minutes before it finishes.

It must be walking the whole commit tree to the root or something silly like that.

Linus Torvalds· Jul 4, 2005, 20:39 UTC · re: David S. Miller · lore

Re: expensive local git clone

On Mon, 4 Jul 2005, David S. Miller wrote:
> 
> I can't be the only person in the world seeing this :-)
> It usually grows to around 88MB and runs for 8 minutes before
> it finishes.
I'm making something based on pack-files.

git-local-pull is indeed walking the whole tree and copying one file at a time. We shouldn't use "git-fetch-script" in "git clone" (it's really designed for doing one file at a time and for small updates),

That said, the pack-file thing I'm working on won't be perfect either, and the main advantage is that it should work over ssh. The fastest way to do a clone is really to do a recursive hardlinked tree, so you may well want to just do

	#!/bin/sh
	mkdir "$2" && cp -rl "$1/.git" "$2/.git"
and it should be about a million times faster and equally effective.
Untested, of course.
		Linus
Petr Baudis· Jul 4, 2005, 20:42 UTC · re: Linus Torvalds · lore

Re: expensive local git clone

Dear diary, on Mon, Jul 04, 2005 at 10:39:09PM CEST, I got a letter where Linus Torvalds <torvalds@osdl.org> told me that...

Show 11 quoted lines
> That said, the pack-file thing I'm working on won't be perfect either, and
> the main advantage is that it should work over ssh. The fastest way to do
> a clone is really to do a recursive hardlinked tree, so you may well want
> to just do
> 
> 	#!/bin/sh
> 	mkdir "$2" && cp -rl "$1/.git" "$2/.git"
> 
> and it should be about a million times faster and equally effective.
> 
> Untested, of course.
I wouldn't do that on anything but .git/objects. ;-)
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
David S. Miller· Jul 4, 2005, 21:00 UTC · re: Petr Baudis · lore

Re: expensive local git clone

From: Petr Baudis <pasky@suse.cz>
Date: Mon, 4 Jul 2005 22:42:35 +0200
Show 15 quoted lines
> Dear diary, on Mon, Jul 04, 2005 at 10:39:09PM CEST, I got a letter
> where Linus Torvalds <torvalds@osdl.org> told me that...
> > That said, the pack-file thing I'm working on won't be perfect either, and
> > the main advantage is that it should work over ssh. The fastest way to do
> > a clone is really to do a recursive hardlinked tree, so you may well want
> > to just do
> > 
> > 	#!/bin/sh
> > 	mkdir "$2" && cp -rl "$1/.git" "$2/.git"
> > 
> > and it should be about a million times faster and equally effective.
> > 
> > Untested, of course.
> 
> I wouldn't do that on anything but .git/objects. ;-)

Agreed. When I do it by hand (but I shouldn't have to do this by hand, git-clone-script should do it this way if that's the best and most efficient) I usually link the object directory then copy the rest of the stuff non-linked.

I keep hoping git-clone-script is going to be a good way to clone two local trees. Is my hope misguided? :-)

Junio C Hamano· Jul 4, 2005, 21:40 UTC · re: David S. Miller · lore

Re: expensive local git clone

>>>>> "DSM" == David S Miller <davem@davemloft.net> writes:

DSM> I keep hoping git-clone-script is going to be a good way DSM> to clone two local trees. Is my hope misguided? :-)

Something along these lines? ------------ Short-cut "git fetch" when cloning locally.

Instead of calling local-pull to "do the right thing", just hardlink or copy the object files over when we know we are doing the clone locally.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

cd /opt/packrat/playpen/public/in-place/git/git.junio/ jit-diff # - master: git-format-patch: Prepare patches for e-mail submission. # + (working tree)

diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,7 +1,39 @@
 #!/bin/sh
 repo="$1"
 dir="$2"
-mkdir $dir || exit 1
-cd $dir
-git-init-db
-git fetch "$repo" && ( git-rev-parse FETCH_HEAD > .git/HEAD )
+mkdir "$dir" &&
+D=$(
+    (cd "$dir" && git-init-db && pwd)
+) || exit 1
+
+# See if repo is a local directory.
+if (
+	cd "$repo/objects" 2>/dev/null
+)
+then
+	# See if we can hardlink and drop "l" if not.
+	sample_file=$(cd "$repo" && find objects -type f -print | sed -e 1q)
+
+	# objects directory should not be empty since we are cloning!
+	test -f "$repo/$sample_file" || exit 
+
+	if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	then
+	    l=l
+	else
+	    l=
+	fi &&
+	rm -f "$D/.git/objects/sample" &&
+
+	cd "$repo" &&
+	find objects -type f -print |
+	cpio -puam$l "$D/.git" || exit 1
+
+	# FETCH_HEAD is always HEAD because we do not do the
+	# extra parameter to "git fetch".
+	pwd
+	cat "HEAD" >"$D/.git/FETCH_HEAD"
+	cd "$D"
+else
+	cd "$D" && git fetch "$repo"
+fi && ( git-rev-parse FETCH_HEAD > .git/HEAD )

Compilation finished at Mon Jul  4 14:37:29
David S. Miller· Jul 4, 2005, 22:49 UTC · re: Junio C Hamano · lore

Re: expensive local git clone

From: Junio C Hamano <junkio@cox.net>
Date: Mon, 04 Jul 2005 14:40:11 -0700
Show 6 quoted lines
> >>>>> "DSM" == David S Miller <davem@davemloft.net> writes:
> 
> DSM> I keep hoping git-clone-script is going to be a good way
> DSM> to clone two local trees.  Is my hope misguided?  :-)
> 
> Something along these lines?

Looks interesting. Any particular reason to use "cpio" instead of "cp"?

Junio C Hamano· Jul 4, 2005, 23:07 UTC · re: David S. Miller · lore

Re: expensive local git clone

>>>>> "DSM" == David S Miller <davem@davemloft.net> writes:

DSM> Looks interesting. Any particular reason to use "cpio" DSM> instead of "cp"?

No particular reason other than I am just used to doing things that way.

Linus Torvalds· Jul 4, 2005, 21:44 UTC · re: David S. Miller · lore

Re: expensive local git clone

On Mon, 4 Jul 2005, David S. Miller wrote:
> 
> I keep hoping git-clone-script is going to be a good way
> to clone two local trees.  Is my hope misguided?  :-)
Well, I'm not working on it, but tested patches...
		Linus
Junio C Hamano· Jul 5, 2005, 20:17 UTC · re: Linus Torvalds · lore

[PATCH] Short-cut git-fetch-pack while cloning locally.

>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
LT> Well, I'm not working on it, but tested patches...
Prodded by the hint...

------------ The git-fetch-pack command used internally by git-fetch-script which in turn is used by git-clone-script refuses to run if there is no common head. When cloning locally, just copy/link the objects directory to avoid its use, since we know there will not be any common head (the target is empty). It also should be more efficient.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

*** This is slightly different from the one I sent as a response *** to David's inquiry, in that it uses cp -rl (or just cp -r) *** instead of cpio. Also I added the copyright notices and one *** liner description.

 git-clone-script |   43 +++++++++++++++++++++++++++++++++++++++----
 rev-list.c       |    5 ++++-
 2 files changed, 43 insertions(+), 5 deletions(-)
f33df082e175f7fc717a75e528eab3db1d294576
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,7 +1,42 @@
 #!/bin/sh
+#
+# Copyright (c) 2005, Linus Torvalds
+# Copyright (c) 2005, Junio C Hamano
+# 
+# Clone a repository into a different directory that does not yet exist.
+
 repo="$1"
 dir="$2"
-mkdir $dir || exit 1
-cd $dir
-git-init-db
-git fetch "$repo" && ( git-rev-parse FETCH_HEAD > .git/HEAD )
+mkdir "$dir" &&
+D=$(
+	(cd "$dir" && git-init-db && pwd)
+) || exit 1
+
+# See if repo is a local directory.
+if (
+	cd "$repo/objects" 2>/dev/null
+)
+then
+	# See if we can hardlink and drop "l" if not.
+	sample_file=$(cd "$repo" && find objects -type f -print | sed -e 1q)
+
+	# objects directory should not be empty since we are cloning!
+	test -f "$repo/$sample_file" || exit
+
+	if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	then
+		l=l
+	else
+		l=
+	fi &&
+	rm -f "$D/.git/objects/sample" &&
+
+	cp -r$l "$repo/objects" "$D/.git/" || exit 1
+
+	# FETCH_HEAD is always HEAD because we do not do the
+	# extra parameter to "git fetch".
+	cat "$repo/HEAD" >"$D/.git/FETCH_HEAD"
+	cd "$D"
+else
+	cd "$D" && git fetch "$repo"
+fi && git-rev-parse FETCH_HEAD > .git/HEAD
diff --git a/rev-list.c b/rev-list.c
--- a/rev-list.c
+++ b/rev-list.c
@@ -482,7 +482,10 @@ int main(int argc, char **argv)
 		commit = get_commit_reference(arg, flags);
 		if (!commit)
 			continue;
-		insert_by_date(&list, commit);
+		if (!merge_order)
+			insert_by_date(&list, commit);
+		else
+			commit_list_insert(commit, &list);
 	}
 
 	if (!merge_order) {		
Linus Torvalds· Jul 6, 2005, 00:42 UTC · re: Linus Torvalds · lore

Re: expensive local git clone

On Mon, 4 Jul 2005, Linus Torvalds wrote:
> 
> I'm making something based on pack-files.

Ok, as of a few minutes ago there's now a pack-based "git clone" there. Of course, it hasn't percolated out to the mirrors yet, so you'll not see it for a while unless you have a master.kernel.org account like David, but basically you can do

	git clone /some/absolute/path new-dir

and it will clone the old project into new-dir using pack-files. The need for an absolute path in the source is just because the "git clone" script ends up "cd'ing" to the new directory before the clone, so a relative path would need to be relative to the "new-dir" location, which is just confusing. Oh, well.

Anyway, as mentioned, you can certainly do a local clone a lot faster with "cp -rl" (and yes, I'll apply Junio's patch if he makes it available against the new version, and adds a flag to make it conditional), but using a pack-file means that the new thing not only will be totally independent of the old one (which you may _want_ to do, especially if they are on different filesystems), but it also means that the above works over ssh too, ie

	git clone master.kernel.org:/pub/scm/git/git.git my-git

should now do the right thing (indeed, I've even tested it - although I've not tried it with things like multiple branches etc, which _should_ all work automatically but.. I'm pulling Jeff's tree as I write this, but master.kernel.org is slow, so it will be some time..).

Btw, one small note: when doing a "git clone", the newly cloned repo will not be checked out, and "master" will be the default HEAD regardless of what the other happened to be at (unless the other side was really screwed up and we can't match up any "master" at all). So use "git checkout xxx" to actually set whatever branch you want to use after a clone.

		Linus
Junio C Hamano· Jul 6, 2005, 08:14 UTC · re: Linus Torvalds · lore

[PATCH] Short-circuit git-clone-pack while cloning locally.

>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:

LT> Anyway, as mentioned, you can certainly do a local clone a lot faster with LT> "cp -rl" (and yes, I'll apply Junio's patch if he makes it available LT> against the new version, and adds a flag to make it conditional),...

By invitation.

------------ When we are cloning a repository on a local filesystem, it is faster to just create a hard linkfarm of .git/object hierarchy and copy the .git/refs files. By default, the script uses the clone-pack method, but with -l and -c parameters it can be told to do the hard linkfarm and recursive file copy to replicate .git/object hierarchy.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 git-clone-script |   80 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 79 insertions(+), 1 deletions(-)
1eb81b1791a751ca0fd1c3e7c1d4a4baacdd042a
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,4 +1,81 @@
 #!/bin/sh
+#
+# Copyright (c) 2005, Linus Torvalds
+# Copyright (c) 2005, Junio C Hamano
+# 
+# Clone a repository into a different directory that does not yet exist.
+
+usage() {
+	echo >&2 "* git clone [-l|-c|-p] <repo> <dir>"
+	exit 1
+}
+
+# what to do when running locally
+local_use=default
+while
+	case "$#,$1" in
+	0,*) break ;;
+	*,-l) local_use=link ;;
+	*,-c) local_use=copy ;;
+	*,-p) local_use=pack ;;
+	*,-*) usage ;;
+	*) break ;;
+	esac
+do
+	shift
+done
+
 repo="$1"
 dir="$2"
-mkdir "$dir" && cd "$dir" && git-init-db && git-clone-pack "$repo"
+mkdir "$dir" &&
+D=$(
+	(cd "$dir" && git-init-db && pwd)
+) || exit 1
+
+# See if repo is a local directory.
+is_local=f
+if (
+	cd "$repo/objects" 2>/dev/null
+)
+then
+	is_local=t
+fi
+
+case "$local_use,$is_local" in
+default,f)
+	;;
+*,f)	echo >&2 "-l|-c|-p flag can be used only when cloning a local repo"
+	exit 1
+	;;
+copy,t | link,t)
+	l=
+	case "$local_use" in
+	link)
+	    # See if we can hardlink and drop "l" if not.
+	    sample_file=$(cd "$repo" && \
+	    		  find objects -type f -print | sed -e 1q)
+
+	    # objects directory should not be empty since we are cloning!
+	    test -f "$repo/$sample_file" || exit
+
+	    if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	    then
+		    l=l
+	    fi &&
+	    rm -f "$D/.git/objects/sample"
+	    ;;
+	esac &&
+	cp -r$l "$repo/objects" "$D/.git/" || exit 1
+
+	# Make a duplicate of refs and HEAD pointer
+	HEAD=
+	if test -f "$repo/HEAD"
+	then
+		HEAD=HEAD
+	fi
+	tar Ccf "$repo" - refs $HEAD | tar Cxf "$D/.git" - || exit 1
+	;;
+*)
+	cd "$D" && git clone-pack "$repo"
+	;;
+esac
Linus Torvalds· Jul 6, 2005, 16:13 UTC · re: Junio C Hamano · lore

Re: [PATCH] Short-circuit git-clone-pack while cloning locally.

On Wed, 6 Jul 2005, Junio C Hamano wrote:
Show 5 quoted lines
> 
> By invitation.
> 
> ------------
> When we are cloning a repository on a local filesystem [...]
Hmm.. Did you test the ssh case?
> +case "$local_use,$is_local" in
> +default,f)
> +	;;
It would seem that you don't do anything at all for the non-local case.
		Linus
Junio C Hamano· Jul 6, 2005, 20:04 UTC · re: Linus Torvalds · lore

[PATCH] Short-circuit git-clone-pack while cloning locally (take 2).

When we are cloning a repository on a local filesystem, it is faster to just create a hard linkfarm of .git/object hierarchy and copy the .git/refs files. By default, the script uses the clone-pack method, but it can be told with the -l flag to do the hard linkfarm (falling back on recursive file copy) to replicate the .git/object hierarchy.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 git-clone-script |   68 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 67 insertions(+), 1 deletions(-)
2bff57b39b9fd13b1319d0192e2a887eb82fa189
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,4 +1,70 @@
 #!/bin/sh
+#
+# Copyright (c) 2005, Linus Torvalds
+# Copyright (c) 2005, Junio C Hamano
+# 
+# Clone a repository into a different directory that does not yet exist.
+
+usage() {
+	echo >&2 "* git clone [-l] <repo> <dir>"
+	exit 1
+}
+
+use_local=no
+while
+	case "$#,$1" in
+	0,*) break ;;
+        *,-l|*,--l|*,--lo|*,--loc|*,--loca|*,--local) use_local=yes ;;
+	*,-*) usage ;;
+	*) break ;;
+	esac
+do
+	shift
+done
+
 repo="$1"
 dir="$2"
-mkdir "$dir" && cd "$dir" && git-init-db && git-clone-pack "$repo"
+mkdir "$dir" &&
+D=$(
+	(cd "$dir" && git-init-db && pwd)
+) &&
+test -d "$D" || usage
+
+# We do local magic only when the user tells us to.
+case "$use_local" in
+yes)
+	( cd "$repo/objects" ) || {
+		repo="$repo/.git"
+		( cd "$repo/objects" ) || {
+		    echo >&2 "-l flag seen but $repo is not local."
+		    exit 1
+		}
+	}
+
+	# See if we can hardlink and drop "l" if not.
+	sample_file=$(cd "$repo" && \
+		      find objects -type f -print | sed -e 1q)
+
+	# objects directory should not be empty since we are cloning!
+	test -f "$repo/$sample_file" || exit
+
+	l=
+	if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	then
+		l=l
+	fi &&
+	rm -f "$D/.git/objects/sample" &&
+	cp -r$l "$repo/objects" "$D/.git/" || exit 1
+
+	# Make a duplicate of refs and HEAD pointer
+	HEAD=
+	if test -f "$repo/HEAD"
+	then
+		HEAD=HEAD
+	fi
+	tar Ccf "$repo" - refs $HEAD | tar Cxf "$D/.git" - || exit 1
+	exit 0
+	;;
+esac
+
+cd "$D" && git clone-pack "$repo"

← back to recent threads