threads / discuss / 26857

GSOC proposal

Subject: GSOC proposal

## tl;dr

6 messages between Mar 24, 2011 and Mar 25, 2011.

replies: 5people: 3as markdown or json

Fredrik Gustafsson· Mar 24, 2011, 22:01 UTC · lore

Hi, I'm interested in working as a student for git in the GSOC program this year.

I'm running a lot of web projects with heavy use of git submodules (each project has around 40 submodules) and therefore I'm very interested in enchant the git submodule experience.

I'm asking for your oppinions and idées for my planned todolist for this summer (if I get accepted of course).

== Preventing false "pointers" == It's far too easy to push a gitrepo pointing to a submodule version that is not existing on the server. Prevent this by checking for available submodule versions before acceptingt the push.

== Threat every module alike == When failing fetching a submodule, continue fetching the next one instead of dying. There's no need to prevent fetching a submodule beginning at 'z' just because a failing in fetching a submodule beginning at 'a'. The submodules should not be alphabetically dependant as they are now.

== Make submodule changes globally visible == Give git-log submodule support. A git log of showing a new version of a module should (if choosen by --submodules or alike) also list the changes to that submodule since the last version of the submodule was commited.

== Move submodules into core == This last bit is excellent described in the link below.

So, what do you all think? Is it too much? Too little? Is the quests relevant to git?

See also https://git.wiki.kernel.org/index.php/SoC2011Ideas#Git_Submodules_Enhancements

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Jens Lehmann· Mar 24, 2011, 23:27 UTC · re: Fredrik Gustafsson · lore

Re: GSOC proposal

Am 24.03.2011 23:01, schrieb Fredrik Gustafsson:

Show 7 quoted lines

> Hi,
> I'm interested in working as a student for git in the GSOC program this
> year.
> 
> I'm running a lot of web projects with heavy use of git submodules (each
> project has around 40 submodules) and therefore I'm very interested in
> enchant the git submodule experience.

Great to hear that!

Show 7 quoted lines

> I'm asking for your oppinions and idées for my planned todolist for this
> summer (if I get accepted of course).
> 
> == Preventing false "pointers" ==
> It's far too easy to push a gitrepo pointing to a submodule version that
> is not existing on the server. Prevent this by checking for available
> submodule versions before acceptingt the push.

Yes, requiring to force such a push is an issue raised quite often (I think addressing this issue would be a good starting point for people who want to get involved in enhancing the submodule experience).

Show 6 quoted lines

> == Threat every module alike ==
> When failing fetching a submodule, continue fetching the next one
> instead of dying. There's no need to prevent fetching a submodule
> beginning at 'z' just because a failing in fetching a submodule
> beginning at 'a'. The submodules should not be alphabetically dependant
> as they are now.

I assume you are talking about the implicit fetch done by "git submodule update" here. The recursive submodule fetch that "git fetch" learned recently continues to fetch other submodules even if one or more fetches failed. But you are right that "git submodule update" should attempt to continue updating the remaining submodules too even if one of those updates failed along the way (This should be achieved with even less effort than the push issue mentioned above, so it would be an even easier starting point for people who want to get involved).

Show 5 quoted lines

> == Make submodule changes globally visible ==
> Give git-log submodule support. A git log of showing a new version of a
> module should (if choosen by --submodules or alike) also list the
> changes to that submodule since the last version of the submodule was
> commited.

Yeah, adding an option so the user is able to tell "git log" she wants to see all submodule commits in detail too would also be very nice (Also some other commands could profit from being able to tell them to recurse into submodules).

> == Move submodules into core ==
> This last bit is excellent described in the link below.

Assuming that you are referring to placing the repository for each populated submodule in the superproject's $GIT_DIR/modules, me thinks that that is the core part of this GSoC project. So I'd be very glad to have someone on board who wants to solve this issue.

> So, what do you all think? Is it too much? Too little? Is the quests
> relevant to git?

I like it! I think all these issues are relevant to achieve better submodule support in git and I'm looking forward to see a student (maybe you? ;-) starting to work on this.

(And, as every year, it's a good idea for a prospective student to get involved in the git community before his application is accepted ... sending some patches is a good way to do that, maybe regarding one of the first two issues raised here? ;-)

Junio C Hamano· Mar 24, 2011, 23:47 UTC · re: Jens Lehmann · lore

Re: GSOC proposal

Jens Lehmann <Jens.Lehmann@web.de> writes:

Show 8 quoted lines

>> == Preventing false "pointers" ==
>> It's far too easy to push a gitrepo pointing to a submodule version that
>> is not existing on the server. Prevent this by checking for available
>> submodule versions before acceptingt the push.
>
> Yes, requiring to force such a push is an issue raised quite often (I
> think addressing this issue would be a good starting point for people
> who want to get involved in enhancing the submodule experience).

You need to be careful, though.

That check can only be sanely done at a hosting site that hosts _both_ the superproject and the submodule repositories. It might make more sense to have this check on the side that pushes, which by definition should have both superprojet and the submodule. Fail, or prompt to confirm, a push from the superproject when it is detected that the submodule commits bound to the commits in the superproject have not been pushed out.

> (And, as every year, it's a good idea for a prospective student to get
> involved in the git community before his application is accepted ...
> sending some patches is a good way to do that, maybe regarding one of
> the first two issues raised here? ;-)

While I agree that it is a good idea, I think the advice is about a few weeks too late for this year already.

Fredrik Gustafsson· Mar 25, 2011, 10:06 UTC · re: Junio C Hamano · lore

Re: GSOC proposal

On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:

Show 19 quoted lines

> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
> >> == Preventing false "pointers" ==
> >> It's far too easy to push a gitrepo pointing to a submodule version that
> >> is not existing on the server. Prevent this by checking for available
> >> submodule versions before acceptingt the push.
> >
> > Yes, requiring to force such a push is an issue raised quite often (I
> > think addressing this issue would be a good starting point for people
> > who want to get involved in enhancing the submodule experience).
> 
> You need to be careful, though.
> 
> That check can only be sanely done at a hosting site that hosts _both_ the
> superproject and the submodule repositories.  It might make more sense to
> have this check on the side that pushes, which by definition should have
> both superprojet and the submodule.  Fail, or prompt to confirm, a push
> from the superproject when it is detected that the submodule commits bound
> to the commits in the superproject have not been pushed out.

I can see three different ways of doing this:
1. the reciever (server) checks for available submodules before
accepting a commit

2. the sender checks for available submodules on the server before
sending a commit.

3. each submodule contains a flag that tells if the last command was a
commit or a push. The core-repository wont allow a push if one (or more)
submodules has a commit as "last command".

Although alternative 1 is the only one that is certain of preventing the problem with "false pointers", it has several other drawbacks. I believe that alt. 3 is the proper way of handling this in git. Although we doesn't guarantee a sane server-repo we does prevent the client from doing stupid things by mistake.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Jens Lehmann· Mar 25, 2011, 23:07 UTC · re: Fredrik Gustafsson · lore

Re: GSOC proposal

Am 25.03.2011 11:06, schrieb Fredrik Gustafsson:

Show 37 quoted lines

> On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:
>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>
>>>> == Preventing false "pointers" ==
>>>> It's far too easy to push a gitrepo pointing to a submodule version that
>>>> is not existing on the server. Prevent this by checking for available
>>>> submodule versions before acceptingt the push.
>>>
>>> Yes, requiring to force such a push is an issue raised quite often (I
>>> think addressing this issue would be a good starting point for people
>>> who want to get involved in enhancing the submodule experience).
>>
>> You need to be careful, though.
>>
>> That check can only be sanely done at a hosting site that hosts _both_ the
>> superproject and the submodule repositories.  It might make more sense to
>> have this check on the side that pushes, which by definition should have
>> both superprojet and the submodule.  Fail, or prompt to confirm, a push
>> from the superproject when it is detected that the submodule commits bound
>> to the commits in the superproject have not been pushed out.
> 
> I can see three different ways of doing this:
> 1. the reciever (server) checks for available submodules before
> accepting a commit
> 
> 2. the sender checks for available submodules on the server before
> sending a commit.
> 
> 3. each submodule contains a flag that tells if the last command was a
> commit or a push. The core-repository wont allow a push if one (or more)
> submodules has a commit as "last command".
> 
> Although alternative 1 is the only one that is certain of preventing the
> problem with "false pointers", it has several other drawbacks. I believe
> that alt. 3 is the proper way of handling this in git. Although we
> doesn't guarantee a sane server-repo we does prevent the client from
> doing stupid things by mistake.

I concur with Junio and am in favor of the following solution, as I think we already have all the information needed present on the side that pushes:

Before doing a push in the superproject collect all submodule commits that are recorded in the commits to be pushed out to the superproject's remote. Then check that they all are contained in at least one remote ref of the submodule they are recorded for. If not, error out and tell the user he should push the submodule first or has to use the "-f" flag to force the push if he really knows what he is doing. Commits of those submodules that aren't checked out are ignored.

This approach would achieve the goal you stated in your last sentence, which me thinks is a sane thing to do.

Fredrik Gustafsson· Mar 25, 2011, 10:08 UTC · re: Jens Lehmann · lore

Re: GSOC proposal

On Fri, Mar 25, 2011 at 12:27:20AM +0100, Jens Lehmann wrote:

Show 22 quoted lines

> > == Threat every module alike ==
> > When failing fetching a submodule, continue fetching the next one
> > instead of dying. There's no need to prevent fetching a submodule
> > beginning at 'z' just because a failing in fetching a submodule
> > beginning at 'a'. The submodules should not be alphabetically dependant
> > as they are now.
> 
> I assume you are talking about the implicit fetch done by "git submodule
> update" here. The recursive submodule fetch that "git fetch" learned
> recently continues to fetch other submodules even if one or more fetches
> failed. But you are right that "git submodule update" should attempt to
> continue updating the remaining submodules too even if one of those
> updates failed along the way (This should be achieved with even less
> effort than the push issue mentioned above, so it would be an even
> easier starting point for people who want to get involved).
> 
> /* snip */
> 
> (And, as every year, it's a good idea for a prospective student to get
> involved in the git community before his application is accepted ...
> sending some patches is a good way to do that, maybe regarding one of
> the first two issues raised here? ;-)

I've attached a patch solving the submodule update-problem. A thing I though about is if it's a good idéa to end the output with an error summary. A git submodule update with 40-50 modules would easily let an error go by undetected with this patch.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com


>From 1bc9a5e69f7b3296bad6dd0449054de8501be5bd Mon Sep 17 00:00:00 2001
From: iveqy <iveqy@iveqy.com>
Date: Fri, 25 Mar 2011 10:43:42 +0100
Subject: [PATCH] When one submodule fails, continue to the next

---
 git-submodule.sh |   60 ++++++++++++++++++++++++++++++++++++------------------
 1 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 3a13397..ae3abd1 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -342,8 +342,11 @@ cmd_init()
 		test -z "$url" || continue
 
 		url=$(git config -f .gitmodules submodule."$name".url)
-		test -z "$url" &&
-		die "No url found for submodule path '$path' in .gitmodules"
+		if test -z "$url" 
+		then
+			say "No url found for submodule path '$path' in .gitmodules"
+			continue
+		fi
 
 		# Possibly a url relative to parent
 		case "$url" in
@@ -352,13 +355,17 @@ cmd_init()
 			;;
 		esac
 
-		git config submodule."$name".url "$url" ||
-		die "Failed to register url for submodule path '$path'"
+		if !(git config submodule."$name".url "$url")
+		then
+			say "Failed to register url for submodule path '$path'"
+		fi
 
 		upd="$(git config -f .gitmodules submodule."$name".update)"
-		test -z "$upd" ||
-		git config submodule."$name".update "$upd" ||
-		die "Failed to register update mode for submodule path '$path'"
+		if !(test -z "$upd" || git config submodule."$name".update "$upd")
+		then
+			say "Failed to register update mode for submodule path '$path'"
+			continue
+		fi
 
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
@@ -446,9 +453,11 @@ cmd_update()
 			cloned_modules="$cloned_modules;$name"
 			subsha1=
 		else
-			subsha1=$(clear_local_git_env; cd "$path" &&
-				git rev-parse --verify HEAD) ||
-			die "Unable to find current revision in submodule path '$path'"
+			if !(subsha1=$(clear_local_git_env; cd "$path" && git rev-parse --verify HEAD))
+			then
+				say "Unable to find current revision in submodule path '$path'"
+				continue
+			fi
 		fi
 
 		if ! test -z "$update"
@@ -466,9 +475,11 @@ cmd_update()
 
 			if test -z "$nofetch"
 			then
-				(clear_local_git_env; cd "$path" &&
-					git-fetch) ||
-				die "Unable to fetch in submodule path '$path'"
+				if !(clear_local_git_env; cd "$path" && git-fetch)
+				then
+					say "Unable to fetch in submodule path '$path'"
+					continue
+				fi
 			fi
 
 			# Is this something we just cloned?
@@ -496,15 +507,21 @@ cmd_update()
 				;;
 			esac
 
-			(clear_local_git_env; cd "$path" && $command "$sha1") ||
-			die "Unable to $action '$sha1' in submodule path '$path'"
+			if !(clear_local_git_env; cd "$path" && $command "$sha1")
+			then
+				say "Unable to $action '$sha1' in submodule path '$path'"
+				continue;
+			fi
 			say "Submodule path '$path': $msg '$sha1'"
 		fi
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") ||
-			die "Failed to recurse into submodule path '$path'"
+			if !(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") 
+			then
+				say "Failed to recurse into submodule path '$path'"
+				continue;
+			fi
 		fi
 	done
 }
@@ -790,13 +807,16 @@ cmd_status()
 
 		if test -n "$recursive"
 		then
-			(
+			if ! (
 				prefix="$displaypath/"
 				clear_local_git_env
 				cd "$path" &&
 				eval cmd_status "$orig_args"
-			) ||
-			die "Failed to recurse into submodule path '$path'"
+			) 
+			then
+				say "Failed to recurse into submodule path '$path'"
+				continue
+			fi
 		fi
 	done
 }
-- 
1.7.4.1.433.gcd306.dirty