# GSOC proposal

6 messages from 2011-03-24 to 2011-03-25. Participants: Fredrik Gustafsson, Jens Lehmann, Junio C Hamano.
Thread: https://gitlist.dev/t/26857

## Fredrik Gustafsson, 2011-03-24 22:01

Subject: GSOC proposal
Message-ID: <20110324220104.GA18721@paksenarrion.iveqy.com>
URL: https://gitlist.dev/e/20110324220104.GA18721%40paksenarrion.iveqy.com

```
Hi,
I'm interested in working as a student for git in the GSOC program this
year.

I'm running a lot of web projects with heavy use of git submodules (each
project has around 40 submodules) and therefore I'm very interested in
enchant the git submodule experience.

I'm asking for your oppinions and idées for my planned todolist for this
summer (if I get accepted of course).

== Preventing false "pointers" ==
It's far too easy to push a gitrepo pointing to a submodule version that
is not existing on the server. Prevent this by checking for available
submodule versions before acceptingt the push.

== Threat every module alike ==
When failing fetching a submodule, continue fetching the next one
instead of dying. There's no need to prevent fetching a submodule
beginning at 'z' just because a failing in fetching a submodule
beginning at 'a'. The submodules should not be alphabetically dependant
as they are now.

== Make submodule changes globally visible ==
Give git-log submodule support. A git log of showing a new version of a
module should (if choosen by --submodules or alike) also list the
changes to that submodule since the last version of the submodule was
commited.

== Move submodules into core ==
This last bit is excellent described in the link below.

So, what do you all think? Is it too much? Too little? Is the quests
relevant to git?

See also
https://git.wiki.kernel.org/index.php/SoC2011Ideas#Git_Submodules_Enhancements

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

```

## Jens Lehmann, 2011-03-24 23:27

Subject: Re: GSOC proposal
Message-ID: <4D8BD358.1030603@web.de>
URL: https://gitlist.dev/e/4D8BD358.1030603%40web.de
In-Reply-To: <20110324220104.GA18721@paksenarrion.iveqy.com>

```
Am 24.03.2011 23:01, schrieb Fredrik Gustafsson:
> Hi,
> I'm interested in working as a student for git in the GSOC program this
> year.
> 
> I'm running a lot of web projects with heavy use of git submodules (each
> project has around 40 submodules) and therefore I'm very interested in
> enchant the git submodule experience.

Great to hear that!

> I'm asking for your oppinions and idées for my planned todolist for this
> summer (if I get accepted of course).
> 
> == Preventing false "pointers" ==
> It's far too easy to push a gitrepo pointing to a submodule version that
> is not existing on the server. Prevent this by checking for available
> submodule versions before acceptingt the push.

Yes, requiring to force such a push is an issue raised quite often (I
think addressing this issue would be a good starting point for people
who want to get involved in enhancing the submodule experience).

> == Threat every module alike ==
> When failing fetching a submodule, continue fetching the next one
> instead of dying. There's no need to prevent fetching a submodule
> beginning at 'z' just because a failing in fetching a submodule
> beginning at 'a'. The submodules should not be alphabetically dependant
> as they are now.

I assume you are talking about the implicit fetch done by "git submodule
update" here. The recursive submodule fetch that "git fetch" learned
recently continues to fetch other submodules even if one or more fetches
failed. But you are right that "git submodule update" should attempt to
continue updating the remaining submodules too even if one of those
updates failed along the way (This should be achieved with even less
effort than the push issue mentioned above, so it would be an even
easier starting point for people who want to get involved).

> == Make submodule changes globally visible ==
> Give git-log submodule support. A git log of showing a new version of a
> module should (if choosen by --submodules or alike) also list the
> changes to that submodule since the last version of the submodule was
> commited.

Yeah, adding an option so the user is able to tell "git log" she wants
to see all submodule commits in detail too would also be very nice
(Also some other commands could profit from being able to tell them to
recurse into submodules).

> == Move submodules into core ==
> This last bit is excellent described in the link below.

Assuming that you are referring to placing the repository for each
populated submodule in the superproject's $GIT_DIR/modules, me thinks
that that is the core part of this GSoC project. So I'd be very glad
to have someone on board who wants to solve this issue.

> So, what do you all think? Is it too much? Too little? Is the quests
> relevant to git?

I like it! I think all these issues are relevant to achieve better
submodule support in git and I'm looking forward to see a student
(maybe you? ;-) starting to work on this.

(And, as every year, it's a good idea for a prospective student to get
involved in the git community before his application is accepted ...
sending some patches is a good way to do that, maybe regarding one of
the first two issues raised here? ;-)

```

## Junio C Hamano, 2011-03-24 23:47

Subject: Re: GSOC proposal
Message-ID: <7vtyes6pya.fsf@alter.siamese.dyndns.org>
URL: https://gitlist.dev/e/7vtyes6pya.fsf%40alter.siamese.dyndns.org
In-Reply-To: <4D8BD358.1030603@web.de>

```
Jens Lehmann <Jens.Lehmann@web.de> writes:

>> == Preventing false "pointers" ==
>> It's far too easy to push a gitrepo pointing to a submodule version that
>> is not existing on the server. Prevent this by checking for available
>> submodule versions before acceptingt the push.
>
> Yes, requiring to force such a push is an issue raised quite often (I
> think addressing this issue would be a good starting point for people
> who want to get involved in enhancing the submodule experience).

You need to be careful, though.

That check can only be sanely done at a hosting site that hosts _both_ the
superproject and the submodule repositories.  It might make more sense to
have this check on the side that pushes, which by definition should have
both superprojet and the submodule.  Fail, or prompt to confirm, a push
from the superproject when it is detected that the submodule commits bound
to the commits in the superproject have not been pushed out.

> (And, as every year, it's a good idea for a prospective student to get
> involved in the git community before his application is accepted ...
> sending some patches is a good way to do that, maybe regarding one of
> the first two issues raised here? ;-)

While I agree that it is a good idea, I think the advice is about a few
weeks too late for this year already.

```

## Fredrik Gustafsson, 2011-03-25 10:06

Subject: Re: GSOC proposal
Message-ID: <20110325100600.GA30376@paksenarrion.iveqy.com>
URL: https://gitlist.dev/e/20110325100600.GA30376%40paksenarrion.iveqy.com
In-Reply-To: <7vtyes6pya.fsf@alter.siamese.dyndns.org>

```
On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
> >> == Preventing false "pointers" ==
> >> It's far too easy to push a gitrepo pointing to a submodule version that
> >> is not existing on the server. Prevent this by checking for available
> >> submodule versions before acceptingt the push.
> >
> > Yes, requiring to force such a push is an issue raised quite often (I
> > think addressing this issue would be a good starting point for people
> > who want to get involved in enhancing the submodule experience).
> 
> You need to be careful, though.
> 
> That check can only be sanely done at a hosting site that hosts _both_ the
> superproject and the submodule repositories.  It might make more sense to
> have this check on the side that pushes, which by definition should have
> both superprojet and the submodule.  Fail, or prompt to confirm, a push
> from the superproject when it is detected that the submodule commits bound
> to the commits in the superproject have not been pushed out.

I can see three different ways of doing this:
1. the reciever (server) checks for available submodules before
accepting a commit

2. the sender checks for available submodules on the server before
sending a commit.

3. each submodule contains a flag that tells if the last command was a
commit or a push. The core-repository wont allow a push if one (or more)
submodules has a commit as "last command".

Although alternative 1 is the only one that is certain of preventing the
problem with "false pointers", it has several other drawbacks. I believe
that alt. 3 is the proper way of handling this in git. Although we
doesn't guarantee a sane server-repo we does prevent the client from
doing stupid things by mistake.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

```

## Fredrik Gustafsson, 2011-03-25 10:08

Subject: Re: GSOC proposal
Message-ID: <20110325100835.GB30376@paksenarrion.iveqy.com>
URL: https://gitlist.dev/e/20110325100835.GB30376%40paksenarrion.iveqy.com
In-Reply-To: <4D8BD358.1030603@web.de>

```
On Fri, Mar 25, 2011 at 12:27:20AM +0100, Jens Lehmann wrote:
> > == Threat every module alike ==
> > When failing fetching a submodule, continue fetching the next one
> > instead of dying. There's no need to prevent fetching a submodule
> > beginning at 'z' just because a failing in fetching a submodule
> > beginning at 'a'. The submodules should not be alphabetically dependant
> > as they are now.
> 
> I assume you are talking about the implicit fetch done by "git submodule
> update" here. The recursive submodule fetch that "git fetch" learned
> recently continues to fetch other submodules even if one or more fetches
> failed. But you are right that "git submodule update" should attempt to
> continue updating the remaining submodules too even if one of those
> updates failed along the way (This should be achieved with even less
> effort than the push issue mentioned above, so it would be an even
> easier starting point for people who want to get involved).
> 
> /* snip */
> 
> (And, as every year, it's a good idea for a prospective student to get
> involved in the git community before his application is accepted ...
> sending some patches is a good way to do that, maybe regarding one of
> the first two issues raised here? ;-)

I've attached a patch solving the submodule update-problem. A thing I 
though about is if it's a good idéa to end the output with an error 
summary. A git submodule update with 40-50 modules would easily let 
an error go by undetected with this patch.

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com


>From 1bc9a5e69f7b3296bad6dd0449054de8501be5bd Mon Sep 17 00:00:00 2001
From: iveqy <iveqy@iveqy.com>
Date: Fri, 25 Mar 2011 10:43:42 +0100
Subject: [PATCH] When one submodule fails, continue to the next

---
 git-submodule.sh |   60 ++++++++++++++++++++++++++++++++++++------------------
 1 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 3a13397..ae3abd1 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -342,8 +342,11 @@ cmd_init()
 		test -z "$url" || continue
 
 		url=$(git config -f .gitmodules submodule."$name".url)
-		test -z "$url" &&
-		die "No url found for submodule path '$path' in .gitmodules"
+		if test -z "$url" 
+		then
+			say "No url found for submodule path '$path' in .gitmodules"
+			continue
+		fi
 
 		# Possibly a url relative to parent
 		case "$url" in
@@ -352,13 +355,17 @@ cmd_init()
 			;;
 		esac
 
-		git config submodule."$name".url "$url" ||
-		die "Failed to register url for submodule path '$path'"
+		if !(git config submodule."$name".url "$url")
+		then
+			say "Failed to register url for submodule path '$path'"
+		fi
 
 		upd="$(git config -f .gitmodules submodule."$name".update)"
-		test -z "$upd" ||
-		git config submodule."$name".update "$upd" ||
-		die "Failed to register update mode for submodule path '$path'"
+		if !(test -z "$upd" || git config submodule."$name".update "$upd")
+		then
+			say "Failed to register update mode for submodule path '$path'"
+			continue
+		fi
 
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
@@ -446,9 +453,11 @@ cmd_update()
 			cloned_modules="$cloned_modules;$name"
 			subsha1=
 		else
-			subsha1=$(clear_local_git_env; cd "$path" &&
-				git rev-parse --verify HEAD) ||
-			die "Unable to find current revision in submodule path '$path'"
+			if !(subsha1=$(clear_local_git_env; cd "$path" && git rev-parse --verify HEAD))
+			then
+				say "Unable to find current revision in submodule path '$path'"
+				continue
+			fi
 		fi
 
 		if ! test -z "$update"
@@ -466,9 +475,11 @@ cmd_update()
 
 			if test -z "$nofetch"
 			then
-				(clear_local_git_env; cd "$path" &&
-					git-fetch) ||
-				die "Unable to fetch in submodule path '$path'"
+				if !(clear_local_git_env; cd "$path" && git-fetch)
+				then
+					say "Unable to fetch in submodule path '$path'"
+					continue
+				fi
 			fi
 
 			# Is this something we just cloned?
@@ -496,15 +507,21 @@ cmd_update()
 				;;
 			esac
 
-			(clear_local_git_env; cd "$path" && $command "$sha1") ||
-			die "Unable to $action '$sha1' in submodule path '$path'"
+			if !(clear_local_git_env; cd "$path" && $command "$sha1")
+			then
+				say "Unable to $action '$sha1' in submodule path '$path'"
+				continue;
+			fi
 			say "Submodule path '$path': $msg '$sha1'"
 		fi
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") ||
-			die "Failed to recurse into submodule path '$path'"
+			if !(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") 
+			then
+				say "Failed to recurse into submodule path '$path'"
+				continue;
+			fi
 		fi
 	done
 }
@@ -790,13 +807,16 @@ cmd_status()
 
 		if test -n "$recursive"
 		then
-			(
+			if ! (
 				prefix="$displaypath/"
 				clear_local_git_env
 				cd "$path" &&
 				eval cmd_status "$orig_args"
-			) ||
-			die "Failed to recurse into submodule path '$path'"
+			) 
+			then
+				say "Failed to recurse into submodule path '$path'"
+				continue
+			fi
 		fi
 	done
 }
-- 
1.7.4.1.433.gcd306.dirty


```

## Jens Lehmann, 2011-03-25 23:07

Subject: Re: GSOC proposal
Message-ID: <4D8D2014.3050603@web.de>
URL: https://gitlist.dev/e/4D8D2014.3050603%40web.de
In-Reply-To: <20110325100600.GA30376@paksenarrion.iveqy.com>

```
Am 25.03.2011 11:06, schrieb Fredrik Gustafsson:
> On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:
>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>
>>>> == Preventing false "pointers" ==
>>>> It's far too easy to push a gitrepo pointing to a submodule version that
>>>> is not existing on the server. Prevent this by checking for available
>>>> submodule versions before acceptingt the push.
>>>
>>> Yes, requiring to force such a push is an issue raised quite often (I
>>> think addressing this issue would be a good starting point for people
>>> who want to get involved in enhancing the submodule experience).
>>
>> You need to be careful, though.
>>
>> That check can only be sanely done at a hosting site that hosts _both_ the
>> superproject and the submodule repositories.  It might make more sense to
>> have this check on the side that pushes, which by definition should have
>> both superprojet and the submodule.  Fail, or prompt to confirm, a push
>> from the superproject when it is detected that the submodule commits bound
>> to the commits in the superproject have not been pushed out.
> 
> I can see three different ways of doing this:
> 1. the reciever (server) checks for available submodules before
> accepting a commit
> 
> 2. the sender checks for available submodules on the server before
> sending a commit.
> 
> 3. each submodule contains a flag that tells if the last command was a
> commit or a push. The core-repository wont allow a push if one (or more)
> submodules has a commit as "last command".
> 
> Although alternative 1 is the only one that is certain of preventing the
> problem with "false pointers", it has several other drawbacks. I believe
> that alt. 3 is the proper way of handling this in git. Although we
> doesn't guarantee a sane server-repo we does prevent the client from
> doing stupid things by mistake.

I concur with Junio and am in favor of the following solution, as I
think we already have all the information needed present on the side
that pushes:

Before doing a push in the superproject collect all submodule commits
that are recorded in the commits to be pushed out to the superproject's
remote. Then check that they all are contained in at least one remote
ref of the submodule they are recorded for. If not, error out and tell
the user he should push the submodule first or has to use the "-f" flag
to force the push if he really knows what he is doing. Commits of those
submodules that aren't checked out are ignored.

This approach would achieve the goal you stated in your last sentence,
which me thinks is a sane thing to do.

```