threads / rfc / 22606

[RFC] submodule+shallow clone feature request

Subject: [RFC] submodule+shallow clone feature request

## tl;dr

8 messages between Feb 10, 2010 and Feb 11, 2010.

replies: 7people: 5as markdown or json

Schuyler Duveen· Feb 10, 2010, 21:39 UTC · lore

My use case is deploying from a git repository, which would be even more graceful with the following features:

1. When 'git clone' has both --recursive and --depth, then submodules
are also checked out shallow (for speed/bandwidth).
2. Some way to specify an override on .gitmodules sources.  This is
because our .gitmodules includes public, read-only sources (github),
rather than our local repos we would prefer to deploy from (for the
purpose of reliability).

The other use-case for feature #2 is the read-only vs. writable repository sources. Developers that are also working on the submodules should be able to clone from separate repository sources. Though this could be (and perhaps is) satisfied with a pushurl= in .gitmodules, I'd like to keep the push url non-public, and that still would leave us unable to deploy from different urls.

For #2 maybe something like this:
$ git clone --recursive --depth 1 --modules foo.modules\
> git@example.com/foo.git
where foo.modules contains:
<begin>
[submodule "bar"]
	path = bar
	url = git@example.com/bar.git
[submodule "bar/baz"]
	path = bar/baz
	url = git@example.com/baz.git
<end>

Notice how the .gitmodules needs to be able to specify recursive sources as well. Presumably `git submodule init` would take --modules argument as well (and maybe git submodule update, too).

I'm not well-versed in the git source code yet, but poking around suggests that this is doable. Would a patch be well received (or does someone want to do it for me :-)

cheers, sky

Avery Pennarun· Feb 10, 2010, 22:10 UTC · re: Schuyler Duveen · lore

Re: [RFC] submodule+shallow clone feature request

On Wed, Feb 10, 2010 at 4:39 PM, Schuyler Duveen <sky@columbia.edu> wrote:
> 2. Some way to specify an override on .gitmodules sources.  This is
> because our .gitmodules includes public, read-only sources (github),
> rather than our local repos we would prefer to deploy from (for the
> purpose of reliability).

You can already override this by editing .git/config after running 'git submodule init'. Of course, doing so precludes using --recursive in "git clone."

> I'm not well-versed in the git source code yet, but poking around
> suggests that this is doable.  Would a patch be well received (or does
> someone want to do it for me :-)

The git-submodule stuff is rarely maintained. I expect patches that improve it will meet with generally positive response, but it's somewhat unlikely that anyone will write them for you.

Have fun,
Avery
Junio C Hamano· Feb 10, 2010, 22:14 UTC · re: Schuyler Duveen · lore

Re: [RFC] submodule+shallow clone feature request

Schuyler Duveen <sky@columbia.edu> writes:
Show 10 quoted lines
> My use case is deploying from a git repository, which would be even more
> graceful with the following features:
>
> 1. When 'git clone' has both --recursive and --depth, then submodules
> are also checked out shallow (for speed/bandwidth).
>
> 2. Some way to specify an override on .gitmodules sources.  This is
> because our .gitmodules includes public, read-only sources (github),
> rather than our local repos we would prefer to deploy from (for the
> purpose of reliability).

These should be doable if you do not use --recursive, so I don't think they are insurmountable issues. I suspect many people would welcome such enhancements to the "git submodule" potty.

I think the current implementation of "--recursive" is an attempt to help people with a particular view (all submodules are interesting and the user needs an access to them immediately) while not trying to help others with different needs. Contribution from the latter class of users to change the situation would be good.

Johannes Schindelin· Feb 10, 2010, 22:57 UTC · re: Junio C Hamano · lore

Re: [RFC] submodule+shallow clone feature request

Hi,
On Wed, 10 Feb 2010, Junio C Hamano wrote:
Show 16 quoted lines
> Schuyler Duveen <sky@columbia.edu> writes:
> 
> > My use case is deploying from a git repository, which would be even more
> > graceful with the following features:
> >
> > 1. When 'git clone' has both --recursive and --depth, then submodules
> > are also checked out shallow (for speed/bandwidth).
> >
> > 2. Some way to specify an override on .gitmodules sources.  This is
> > because our .gitmodules includes public, read-only sources (github),
> > rather than our local repos we would prefer to deploy from (for the
> > purpose of reliability).
> 
> These should be doable if you do not use --recursive, so I don't think 
> they are insurmountable issues.  I suspect many people would welcome 
> such enhancements to the "git submodule" potty.

Yes. Note, though, that the problems of enhancing git-submodule are not technical, as we can learn from the recent history, including the lack of support for rebasing submodules (there _were_ patches!).

Ciao, Dscho

Junio C Hamano· Feb 10, 2010, 23:09 UTC · re: Johannes Schindelin · lore

Re: [RFC] submodule+shallow clone feature request

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> Yes. Note, though, that the problems of enhancing git-submodule are not 
> technical, as we can learn from the recent history, including the lack of 
> support for rebasing submodules (there _were_ patches!).

Sorry I don't recall. Were they of 'next' quality? How well were they reviewed?

Johannes Schindelin· Feb 10, 2010, 23:59 UTC · re: Junio C Hamano · lore

Re: [RFC] submodule+shallow clone feature request

Hi,
On Wed, 10 Feb 2010, Junio C Hamano wrote:
Show 8 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > Yes. Note, though, that the problems of enhancing git-submodule are 
> > not technical, as we can learn from the recent history, including the 
> > lack of support for rebasing submodules (there _were_ patches!).
> 
> Sorry I don't recall.  Were they of 'next' quality?  How well were they 
> reviewed?
Obviously not, otherwise you would have applied them, no?

OTOH I found the technical details rather trivial, so maybe they were 'next' quality, but there was another reason you did not apply them.

I just know that from my daily workflow, I deeply miss rebasing submodules. But then, I listed the issues I have with submodules, and this list was welcomed with unbelievable enthusiasm.

Ciao, Dscho

Junio C Hamano· Feb 11, 2010, 06:19 UTC · re: Johannes Schindelin · lore

Re* [RFC] submodule+shallow clone feature request

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 15 quoted lines
> On Wed, 10 Feb 2010, Junio C Hamano wrote:
>
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>> 
>> > Yes. Note, though, that the problems of enhancing git-submodule are 
>> > not technical, as we can learn from the recent history, including the 
>> > lack of support for rebasing submodules (there _were_ patches!).
>> 
>> Sorry I don't recall.  Were they of 'next' quality?  How well were they 
>> reviewed?
>
> Obviously not, otherwise you would have applied them, no?
>
> OTOH I found the technical details rather trivial, so maybe they were 
> 'next' quality, but there was another reason you did not apply them.

Well, I spent some time digging the mail archive myself. I think you were talking about this thread:

    http://thread.gmane.org/gmane.comp.version-control.git/116918

If this is not the thread you were talking about, please discard/disregard the remainder of this message, and give me a better pointer instead.

The thread ends with you asking me:
    Junio, how about it? post 1.6.3 or not?  It is a well contained change, 
    with little chance of breaking something, but offers a more sensible 
    workflow here.
and I said:
    I am afraid it is a bit too late for "improvements" after -rc1.
    People survived without the new feature until now, and they can wait a
    bit longer for the next one....

Obviously, after that nothing happened. We have some people who send reminders for good topics after the original thread died without producing a visible result. I'd ask you to do the same (when you can---as everybody else, you don't work full time on git; neither do I).

An honest answer to my question would have been "Yes, I reviewed it and mentored the original author through a few cycles of revisions. After that process the patch was of 'next' quality. This however happened during the pre-release freeze, and unfortunately nobody (including you or I) rememebered to bring up the issue again after the release to move things forward. It fell through cracks."

Instead, what you wrote makes it sounds as if I blocked a technically correct solution to a real problem on some political grounds, doesn't it? What effect did you hope your insinuation to have on readers of this list?

It does not help to give a new person a false impression that it takes more than producing a good solution to a real problem to contribute to the project. Especially, the "falling through the crack" didn't have anything to do with it being submodule changes. Please stop spreading FUD and try to be constructive as others do.

To restart the discussion so that we can have it (if it is a good change) after 1.7.0 ships, here is a pointer to the last revision of the patch.

    http://article.gmane.org/gmane.comp.version-control.git/117394/raw

I didn't check if it needs rebasing to the recent codebase, but I expect people who work with submodules in real life (I don't count) to comment on it, and re-polish it into an applicable shape if necessary, by the time post 1.7.0 cycle opens.

-- >8 --
From: Peter Hutterer <peter.hutterer@who-t.net>
Subject: Re: [PATCH/RESEND] git-submodule: add support for --rebase.
Date: Fri, 24 Apr 2009 09:06:38 +1000

'git submodule update --rebase' rebases your local branch on top of what would have been checked out to a detached HEAD otherwise.

In some cases, detaching the HEAD when updating a submodule complicates the workflow to commit to this submodule (checkout master, rebase, then commit). For submodules that require frequent updates but infrequent (if any) commits, a rebase can be executed directly by the git-submodule command, ensuring that the submodules stay on their respective branches.

git-config key: submodule.$name.rebase (bool)
Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
Show 13 quoted lines
> With this squashed in, I actually would not be too opposed to put this 
> into 1.6.3, as it _is_ an improvement.
> 
> -- snipsnap --
> [PATCH] To be squashed in
> 
> This does three things:
> 
> - add .gitmodules support for rebase,
> - use --bool for the git config calls for type checking, and
> - rename ambiguous t7404 to t7406.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Squashed in, thank you. One typo fixed in last testcase (does-not-mater -> does-not-matter).

 Documentation/git-submodule.txt |   14 ++++-
 Documentation/gitmodules.txt    |    3 +
 git-submodule.sh                |   33 ++++++++-
 t/t7406-submodule-update.sh     |  140 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 184 insertions(+), 6 deletions(-)
 create mode 100755 t/t7406-submodule-update.sh
diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 3b8df44..0286409 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -12,7 +12,7 @@ SYNOPSIS
 'git submodule' [--quiet] add [-b branch] [--] <repository> <path>
 'git submodule' [--quiet] status [--cached] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
-'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--] [<path>...]
+'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase] [--] [<path>...]
 'git submodule' [--quiet] summary [--summary-limit <n>] [commit] [--] [<path>...]
 'git submodule' [--quiet] foreach <command>
 'git submodule' [--quiet] sync [--] [<path>...]
@@ -113,7 +113,8 @@ init::
 update::
 	Update the registered submodules, i.e. clone missing submodules and
 	checkout the commit specified in the index of the containing repository.
-	This will make the submodules HEAD be detached.
+	This will make the submodules HEAD be detached unless '--rebase' is
+	specified or the key `submodule.$name.rebase` is set to `true`.
 +
 If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
@@ -177,6 +178,15 @@ OPTIONS
 	This option is only valid for the update command.
 	Don't fetch new objects from the remote site.
 
+--rebase::
+	This option is only valid for the update command.
+	Rebase the current branch onto the commit recorded in the
+	superproject. If this option is given, the submodule's HEAD will not
+	be detached. If a a merge failure prevents this process, you will have
+	to resolve these failures with linkgit:git-rebase[1].
+	If the key `submodule.$name.rebase` is set to `true`, this option is
+	implicit.
+
 <path>...::
 	Paths to submodule(s). When specified this will restrict the command
 	to only operate on the submodules found at the specified paths.
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index d1a17e2..7c22c40 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -30,6 +30,9 @@ submodule.<name>.path::
 submodule.<name>.url::
 	Defines an url from where the submodule repository can be cloned.
 
+submodule.<name>.rebase::
+	Defines that the submodule should be rebased by default.
+
 
 EXAMPLES
 --------
diff --git a/git-submodule.sh b/git-submodule.sh
index 7c2e060..b7c9bdc 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -17,6 +17,7 @@ branch=
 quiet=
 cached=
 nofetch=
+rebase=
 
 #
 # print stuff on stdout unless -q was specified
@@ -287,6 +288,11 @@ cmd_init()
 		git config submodule."$name".url "$url" ||
 		die "Failed to register url for submodule path '$path'"
 
+		test true != "$(git config -f .gitmodules --bool \
+			submodule."$name".rebase)" ||
+		git config submodule."$name".rebase true ||
+		die "Failed to register submodule path '$path' as rebasing"
+
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
 }
@@ -314,6 +320,10 @@ cmd_update()
 			shift
 			nofetch=1
 			;;
+		-r|--rebase)
+			shift
+			rebase=true
+			;;
 		--)
 			shift
 			break
@@ -332,6 +342,7 @@ cmd_update()
 	do
 		name=$(module_name "$path") || exit
 		url=$(git config submodule."$name".url)
+		rebase_module=$(git config --bool submodule."$name".rebase)
 		if test -z "$url"
 		then
 			# Only mention uninitialized submodules when its
@@ -352,6 +363,11 @@ cmd_update()
 			die "Unable to find current revision in submodule path '$path'"
 		fi
 
+		if test true = "$rebase"
+		then
+			rebase_module=true
+		fi
+
 		if test "$subsha1" != "$sha1"
 		then
 			force=
@@ -367,11 +383,20 @@ cmd_update()
 				die "Unable to fetch in submodule path '$path'"
 			fi
 
-			(unset GIT_DIR; cd "$path" &&
-				  git-checkout $force -q "$sha1") ||
-			die "Unable to checkout '$sha1' in submodule path '$path'"
+			if test true = "$rebase_module"
+			then
+				command="git-rebase"
+				action="rebase"
+				msg="rebased onto"
+			else
+				command="git-checkout $force -q"
+				action="checkout"
+				msg="checked out"
+			fi
 
-			say "Submodule path '$path': checked out '$sha1'"
+			(unset GIT_DIR; cd "$path" && $command "$sha1") ||
+			die "Unable to $action '$sha1' in submodule path '$path'"
+			say "Submodule path '$path': $msg '$sha1'"
 		fi
 	done
 }
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
new file mode 100755
index 0000000..3442c05
--- /dev/null
+++ b/t/t7406-submodule-update.sh
@@ -0,0 +1,140 @@
+#!/bin/sh
+#
+# Copyright (c) 2009 Red Hat, Inc.
+#
+
+test_description='Test updating submodules
+
+This test verifies that "git submodule update" detaches the HEAD of the
+submodule and "git submodule update --rebase" does not detach the HEAD.
+'
+
+. ./test-lib.sh
+
+
+compare_head()
+{
+    sha_master=`git-rev-list --max-count=1 master`
+    sha_head=`git-rev-list --max-count=1 HEAD`
+
+    test "$sha_master" = "$sha_head"
+}
+
+
+test_expect_success 'setup a submodule tree' '
+	echo file > file &&
+	git add file &&
+	test_tick &&
+	git commit -m upstream
+	git clone . super &&
+	git clone super submodule &&
+	(cd super &&
+	 git submodule add ../submodule submodule &&
+	 test_tick &&
+	 git commit -m "submodule" &&
+	 git submodule init submodule
+	) &&
+	(cd submodule &&
+	echo "line2" > file &&
+	git add file &&
+	git commit -m "Commit 2"
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  git pull --rebase origin
+	 ) &&
+	 git add submodule &&
+	 git commit -m "submodule update"
+	)
+'
+
+test_expect_success 'submodule update detaching the HEAD ' '
+	(cd super/submodule &&
+	 git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update submodule &&
+	 cd submodule &&
+	 ! compare_head
+	)
+'
+
+test_expect_success 'submodule update --rebase staying on master' '
+	(cd super/submodule &&
+	  git checkout master
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update --rebase submodule &&
+	 cd submodule &&
+	 compare_head
+	)
+'
+
+test_expect_success 'submodule update - rebase true in .git/config' '
+	(cd super &&
+	 git config submodule.submodule.rebase true
+	) &&
+	(cd super/submodule &&
+	  git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update submodule &&
+	 cd submodule &&
+	 compare_head
+	)
+'
+
+test_expect_success 'submodule update - rebase false in .git/config but --rebase given' '
+	(cd super &&
+	 git config submodule.submodule.rebase false
+	) &&
+	(cd super/submodule &&
+	  git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update --rebase submodule &&
+	 cd submodule &&
+	 compare_head
+	)
+'
+
+test_expect_success 'submodule update - rebase false in .git/config' '
+	(cd super &&
+	 git config submodule.submodule.rebase false
+	) &&
+	(cd super/submodule &&
+	  git reset --hard HEAD^
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update submodule &&
+	 cd submodule &&
+	 ! compare_head
+	)
+'
+
+test_expect_success 'submodule init picks up rebase' '
+	(cd super &&
+	 git config submodule.rebasing.url git://non-existing/git &&
+	 git config submodule.rebasing.path does-not-matter &&
+	 git config submodule.rebasing.rebase true &&
+	 git submodule init rebasing &&
+	 test true = $(git config --bool submodule.rebasing.rebase)
+	)
+'
+
+test_done
-- 
1.6.3.rc1.2.gfa66a
Peter Hutterer· Feb 11, 2010, 06:51 UTC · re: Junio C Hamano · lore

Re: Re* [RFC] submodule+shallow clone feature request

On Wed, Feb 10, 2010 at 10:19:42PM -0800, Junio C Hamano wrote:
Show 42 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > On Wed, 10 Feb 2010, Junio C Hamano wrote:
> >
> >> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> >> 
> >> > Yes. Note, though, that the problems of enhancing git-submodule are 
> >> > not technical, as we can learn from the recent history, including the 
> >> > lack of support for rebasing submodules (there _were_ patches!).
> >> 
> >> Sorry I don't recall.  Were they of 'next' quality?  How well were they 
> >> reviewed?
> >
> > Obviously not, otherwise you would have applied them, no?
> >
> > OTOH I found the technical details rather trivial, so maybe they were 
> > 'next' quality, but there was another reason you did not apply them.
> 
> Well, I spent some time digging the mail archive myself.  I think you were
> talking about this thread:
> 
>     http://thread.gmane.org/gmane.comp.version-control.git/116918
> 
> If this is not the thread you were talking about, please discard/disregard
> the remainder of this message, and give me a better pointer instead.
> 
> The thread ends with you asking me:
> 
>     Junio, how about it? post 1.6.3 or not?  It is a well contained change, 
>     with little chance of breaking something, but offers a more sensible 
>     workflow here.
> 
> and I said:
> 
>     I am afraid it is a bit too late for "improvements" after -rc1.
>     People survived without the new feature until now, and they can wait a
>     bit longer for the next one....
> 
> Obviously, after that nothing happened.  We have some people who send
> reminders for good topics after the original thread died without producing
> a visible result.  I'd ask you to do the same (when you can---as everybody
> else, you don't work full time on git; neither do I).
[...]
> To restart the discussion so that we can have it (if it is a good change)
> after 1.7.0 ships, here is a pointer to the last revision of the patch.
> 
>     http://article.gmane.org/gmane.comp.version-control.git/117394/raw

Thanks for CCing me, I'm not on the list at the moment. FWIW, Johannes pinged me about this patch a few weeks after that, but after revisiting it a few times I found some issues with it. Here's the email I sent to Johannes on April 24, my apologies that this was a private email only and did not reach the list.

"Sorry about the delay again, I've been a bit busy lately. I've thought about it a bit more and tbh. I don't think this patch - even if rebased - should be merged.

The original idea was that a module can be marked for auto-rebasing in .gitmodules. The issue with that is that AFAIK git submodule does not store branch info. So such auto-rebasing would only work provided it would be on the master branch. Anything else would require a fancier script than my patch including specifying which branch should be checked out in the original clone.

Right now, I don't have the time to design such a patch and I'm not even sure how much it is needed. With git submodule foreach it's relatively simple to just do the auto-rebase setting for all modules and I would not be surprised if the majority of use-cases require auto-rebasing on all modules anyway.

Does that make sense? "

So in this particular case the patchset was withdrawn by me for technical reasons (and the lack of time to sort out the details). It should have communicated better - again, my apologies for that.

Cheers,
  Peter

← back to recent threads