{"thread":{"id":"26857","subject":"GSOC proposal","startedAt":"2011-03-24T22:01:04Z","lastAt":"2011-03-25T23:07:00Z","messageCount":6,"participants":["Fredrik Gustafsson","Jens Lehmann","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"164258","messageId":"20110324220104.GA18721@paksenarrion.iveqy.com","threadId":"26857","inReplyTo":null,"subject":"GSOC proposal","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2011-03-24T22:01:04Z","receivedAt":"2011-03-24T22:01:04Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"Hi,\nI'm interested in working as a student for git in the GSOC program this\nyear.\n\nI'm running a lot of web projects with heavy use of git submodules (each\nproject has around 40 submodules) and therefore I'm very interested in\nenchant the git submodule experience.\n\nI'm asking for your oppinions and idées for my planned todolist for this\nsummer (if I get accepted of course).\n\n== Preventing false \"pointers\" ==\nIt's far too easy to push a gitrepo pointing to a submodule version that\nis not existing on the server. Prevent this by checking for available\nsubmodule versions before acceptingt the push.\n\n== Threat every module alike ==\nWhen failing fetching a submodule, continue fetching the next one\ninstead of dying. There's no need to prevent fetching a submodule\nbeginning at 'z' just because a failing in fetching a submodule\nbeginning at 'a'. The submodules should not be alphabetically dependant\nas they are now.\n\n== Make submodule changes globally visible ==\nGive git-log submodule support. A git log of showing a new version of a\nmodule should (if choosen by --submodules or alike) also list the\nchanges to that submodule since the last version of the submodule was\ncommited.\n\n== Move submodules into core ==\nThis last bit is excellent described in the link below.\n\nSo, what do you all think? Is it too much? Too little? Is the quests\nrelevant to git?\n\nSee also\nhttps://git.wiki.kernel.org/index.php/SoC2011Ideas#Git_Submodules_Enhancements\n\n-- \nMed vänliga hälsningar\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"},{"id":"164264","messageId":"4D8BD358.1030603@web.de","threadId":"26857","inReplyTo":"20110324220104.GA18721@paksenarrion.iveqy.com","subject":"Re: GSOC proposal","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-03-24T23:27:20Z","receivedAt":"2011-03-24T23:27:20Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 24.03.2011 23:01, schrieb Fredrik Gustafsson:\n> Hi,\n> I'm interested in working as a student for git in the GSOC program this\n> year.\n> \n> I'm running a lot of web projects with heavy use of git submodules (each\n> project has around 40 submodules) and therefore I'm very interested in\n> enchant the git submodule experience.\n\nGreat to hear that!\n\n> I'm asking for your oppinions and idées for my planned todolist for this\n> summer (if I get accepted of course).\n> \n> == Preventing false \"pointers\" ==\n> It's far too easy to push a gitrepo pointing to a submodule version that\n> is not existing on the server. Prevent this by checking for available\n> submodule versions before acceptingt the push.\n\nYes, requiring to force such a push is an issue raised quite often (I\nthink addressing this issue would be a good starting point for people\nwho want to get involved in enhancing the submodule experience).\n\n> == Threat every module alike ==\n> When failing fetching a submodule, continue fetching the next one\n> instead of dying. There's no need to prevent fetching a submodule\n> beginning at 'z' just because a failing in fetching a submodule\n> beginning at 'a'. The submodules should not be alphabetically dependant\n> as they are now.\n\nI assume you are talking about the implicit fetch done by \"git submodule\nupdate\" here. The recursive submodule fetch that \"git fetch\" learned\nrecently continues to fetch other submodules even if one or more fetches\nfailed. But you are right that \"git submodule update\" should attempt to\ncontinue updating the remaining submodules too even if one of those\nupdates failed along the way (This should be achieved with even less\neffort than the push issue mentioned above, so it would be an even\neasier starting point for people who want to get involved).\n\n> == Make submodule changes globally visible ==\n> Give git-log submodule support. A git log of showing a new version of a\n> module should (if choosen by --submodules or alike) also list the\n> changes to that submodule since the last version of the submodule was\n> commited.\n\nYeah, adding an option so the user is able to tell \"git log\" she wants\nto see all submodule commits in detail too would also be very nice\n(Also some other commands could profit from being able to tell them to\nrecurse into submodules).\n\n> == Move submodules into core ==\n> This last bit is excellent described in the link below.\n\nAssuming that you are referring to placing the repository for each\npopulated submodule in the superproject's $GIT_DIR/modules, me thinks\nthat that is the core part of this GSoC project. So I'd be very glad\nto have someone on board who wants to solve this issue.\n\n> So, what do you all think? Is it too much? Too little? Is the quests\n> relevant to git?\n\nI like it! I think all these issues are relevant to achieve better\nsubmodule support in git and I'm looking forward to see a student\n(maybe you? ;-) starting to work on this.\n\n(And, as every year, it's a good idea for a prospective student to get\ninvolved in the git community before his application is accepted ...\nsending some patches is a good way to do that, maybe regarding one of\nthe first two issues raised here? ;-)\n"},{"id":"164267","messageId":"7vtyes6pya.fsf@alter.siamese.dyndns.org","threadId":"26857","inReplyTo":"4D8BD358.1030603@web.de","subject":"Re: GSOC proposal","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-03-24T23:47:41Z","receivedAt":"2011-03-24T23:47:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n>> == Preventing false \"pointers\" ==\n>> It's far too easy to push a gitrepo pointing to a submodule version that\n>> is not existing on the server. Prevent this by checking for available\n>> submodule versions before acceptingt the push.\n>\n> Yes, requiring to force such a push is an issue raised quite often (I\n> think addressing this issue would be a good starting point for people\n> who want to get involved in enhancing the submodule experience).\n\nYou need to be careful, though.\n\nThat check can only be sanely done at a hosting site that hosts _both_ the\nsuperproject and the submodule repositories.  It might make more sense to\nhave this check on the side that pushes, which by definition should have\nboth superprojet and the submodule.  Fail, or prompt to confirm, a push\nfrom the superproject when it is detected that the submodule commits bound\nto the commits in the superproject have not been pushed out.\n\n> (And, as every year, it's a good idea for a prospective student to get\n> involved in the git community before his application is accepted ...\n> sending some patches is a good way to do that, maybe regarding one of\n> the first two issues raised here? ;-)\n\nWhile I agree that it is a good idea, I think the advice is about a few\nweeks too late for this year already.\n"},{"id":"164286","messageId":"20110325100600.GA30376@paksenarrion.iveqy.com","threadId":"26857","inReplyTo":"7vtyes6pya.fsf@alter.siamese.dyndns.org","subject":"Re: GSOC proposal","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2011-03-25T10:06:00Z","receivedAt":"2011-03-25T10:06:00Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n> >> == Preventing false \"pointers\" ==\n> >> It's far too easy to push a gitrepo pointing to a submodule version that\n> >> is not existing on the server. Prevent this by checking for available\n> >> submodule versions before acceptingt the push.\n> >\n> > Yes, requiring to force such a push is an issue raised quite often (I\n> > think addressing this issue would be a good starting point for people\n> > who want to get involved in enhancing the submodule experience).\n> \n> You need to be careful, though.\n> \n> That check can only be sanely done at a hosting site that hosts _both_ the\n> superproject and the submodule repositories.  It might make more sense to\n> have this check on the side that pushes, which by definition should have\n> both superprojet and the submodule.  Fail, or prompt to confirm, a push\n> from the superproject when it is detected that the submodule commits bound\n> to the commits in the superproject have not been pushed out.\n\nI can see three different ways of doing this:\n1. the reciever (server) checks for available submodules before\naccepting a commit\n\n2. the sender checks for available submodules on the server before\nsending a commit.\n\n3. each submodule contains a flag that tells if the last command was a\ncommit or a push. The core-repository wont allow a push if one (or more)\nsubmodules has a commit as \"last command\".\n\nAlthough alternative 1 is the only one that is certain of preventing the\nproblem with \"false pointers\", it has several other drawbacks. I believe\nthat alt. 3 is the proper way of handling this in git. Although we\ndoesn't guarantee a sane server-repo we does prevent the client from\ndoing stupid things by mistake.\n\n-- \nMed vänliga hälsningar\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"},{"id":"164287","messageId":"20110325100835.GB30376@paksenarrion.iveqy.com","threadId":"26857","inReplyTo":"4D8BD358.1030603@web.de","subject":"Re: GSOC proposal","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2011-03-25T10:08:35Z","receivedAt":"2011-03-25T10:08:35Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Fri, Mar 25, 2011 at 12:27:20AM +0100, Jens Lehmann wrote:\n> > == Threat every module alike ==\n> > When failing fetching a submodule, continue fetching the next one\n> > instead of dying. There's no need to prevent fetching a submodule\n> > beginning at 'z' just because a failing in fetching a submodule\n> > beginning at 'a'. The submodules should not be alphabetically dependant\n> > as they are now.\n> \n> I assume you are talking about the implicit fetch done by \"git submodule\n> update\" here. The recursive submodule fetch that \"git fetch\" learned\n> recently continues to fetch other submodules even if one or more fetches\n> failed. But you are right that \"git submodule update\" should attempt to\n> continue updating the remaining submodules too even if one of those\n> updates failed along the way (This should be achieved with even less\n> effort than the push issue mentioned above, so it would be an even\n> easier starting point for people who want to get involved).\n> \n> /* snip */\n> \n> (And, as every year, it's a good idea for a prospective student to get\n> involved in the git community before his application is accepted ...\n> sending some patches is a good way to do that, maybe regarding one of\n> the first two issues raised here? ;-)\n\nI've attached a patch solving the submodule update-problem. A thing I \nthough about is if it's a good idéa to end the output with an error \nsummary. A git submodule update with 40-50 modules would easily let \nan error go by undetected with this patch.\n\n-- \nMed vänliga hälsningar\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n\n\n>From 1bc9a5e69f7b3296bad6dd0449054de8501be5bd Mon Sep 17 00:00:00 2001\nFrom: iveqy <iveqy@iveqy.com>\nDate: Fri, 25 Mar 2011 10:43:42 +0100\nSubject: [PATCH] When one submodule fails, continue to the next\n\n---\n git-submodule.sh |   60 ++++++++++++++++++++++++++++++++++++------------------\n 1 files changed, 40 insertions(+), 20 deletions(-)\n\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 3a13397..ae3abd1 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -342,8 +342,11 @@ cmd_init()\n \t\ttest -z \"$url\" || continue\n \n \t\turl=$(git config -f .gitmodules submodule.\"$name\".url)\n-\t\ttest -z \"$url\" &&\n-\t\tdie \"No url found for submodule path '$path' in .gitmodules\"\n+\t\tif test -z \"$url\" \n+\t\tthen\n+\t\t\tsay \"No url found for submodule path '$path' in .gitmodules\"\n+\t\t\tcontinue\n+\t\tfi\n \n \t\t# Possibly a url relative to parent\n \t\tcase \"$url\" in\n@@ -352,13 +355,17 @@ cmd_init()\n \t\t\t;;\n \t\tesac\n \n-\t\tgit config submodule.\"$name\".url \"$url\" ||\n-\t\tdie \"Failed to register url for submodule path '$path'\"\n+\t\tif !(git config submodule.\"$name\".url \"$url\")\n+\t\tthen\n+\t\t\tsay \"Failed to register url for submodule path '$path'\"\n+\t\tfi\n \n \t\tupd=\"$(git config -f .gitmodules submodule.\"$name\".update)\"\n-\t\ttest -z \"$upd\" ||\n-\t\tgit config submodule.\"$name\".update \"$upd\" ||\n-\t\tdie \"Failed to register update mode for submodule path '$path'\"\n+\t\tif !(test -z \"$upd\" || git config submodule.\"$name\".update \"$upd\")\n+\t\tthen\n+\t\t\tsay \"Failed to register update mode for submodule path '$path'\"\n+\t\t\tcontinue\n+\t\tfi\n \n \t\tsay \"Submodule '$name' ($url) registered for path '$path'\"\n \tdone\n@@ -446,9 +453,11 @@ cmd_update()\n \t\t\tcloned_modules=\"$cloned_modules;$name\"\n \t\t\tsubsha1=\n \t\telse\n-\t\t\tsubsha1=$(clear_local_git_env; cd \"$path\" &&\n-\t\t\t\tgit rev-parse --verify HEAD) ||\n-\t\t\tdie \"Unable to find current revision in submodule path '$path'\"\n+\t\t\tif !(subsha1=$(clear_local_git_env; cd \"$path\" && git rev-parse --verify HEAD))\n+\t\t\tthen\n+\t\t\t\tsay \"Unable to find current revision in submodule path '$path'\"\n+\t\t\t\tcontinue\n+\t\t\tfi\n \t\tfi\n \n \t\tif ! test -z \"$update\"\n@@ -466,9 +475,11 @@ cmd_update()\n \n \t\t\tif test -z \"$nofetch\"\n \t\t\tthen\n-\t\t\t\t(clear_local_git_env; cd \"$path\" &&\n-\t\t\t\t\tgit-fetch) ||\n-\t\t\t\tdie \"Unable to fetch in submodule path '$path'\"\n+\t\t\t\tif !(clear_local_git_env; cd \"$path\" && git-fetch)\n+\t\t\t\tthen\n+\t\t\t\t\tsay \"Unable to fetch in submodule path '$path'\"\n+\t\t\t\t\tcontinue\n+\t\t\t\tfi\n \t\t\tfi\n \n \t\t\t# Is this something we just cloned?\n@@ -496,15 +507,21 @@ cmd_update()\n \t\t\t\t;;\n \t\t\tesac\n \n-\t\t\t(clear_local_git_env; cd \"$path\" && $command \"$sha1\") ||\n-\t\t\tdie \"Unable to $action '$sha1' in submodule path '$path'\"\n+\t\t\tif !(clear_local_git_env; cd \"$path\" && $command \"$sha1\")\n+\t\t\tthen\n+\t\t\t\tsay \"Unable to $action '$sha1' in submodule path '$path'\"\n+\t\t\t\tcontinue;\n+\t\t\tfi\n \t\t\tsay \"Submodule path '$path': $msg '$sha1'\"\n \t\tfi\n \n \t\tif test -n \"$recursive\"\n \t\tthen\n-\t\t\t(clear_local_git_env; cd \"$path\" && eval cmd_update \"$orig_flags\") ||\n-\t\t\tdie \"Failed to recurse into submodule path '$path'\"\n+\t\t\tif !(clear_local_git_env; cd \"$path\" && eval cmd_update \"$orig_flags\") \n+\t\t\tthen\n+\t\t\t\tsay \"Failed to recurse into submodule path '$path'\"\n+\t\t\t\tcontinue;\n+\t\t\tfi\n \t\tfi\n \tdone\n }\n@@ -790,13 +807,16 @@ cmd_status()\n \n \t\tif test -n \"$recursive\"\n \t\tthen\n-\t\t\t(\n+\t\t\tif ! (\n \t\t\t\tprefix=\"$displaypath/\"\n \t\t\t\tclear_local_git_env\n \t\t\t\tcd \"$path\" &&\n \t\t\t\teval cmd_status \"$orig_args\"\n-\t\t\t) ||\n-\t\t\tdie \"Failed to recurse into submodule path '$path'\"\n+\t\t\t) \n+\t\t\tthen\n+\t\t\t\tsay \"Failed to recurse into submodule path '$path'\"\n+\t\t\t\tcontinue\n+\t\t\tfi\n \t\tfi\n \tdone\n }\n-- \n1.7.4.1.433.gcd306.dirty\n\n"},{"id":"164340","messageId":"4D8D2014.3050603@web.de","threadId":"26857","inReplyTo":"20110325100600.GA30376@paksenarrion.iveqy.com","subject":"Re: GSOC proposal","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-03-25T23:07:00Z","receivedAt":"2011-03-25T23:07:00Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 25.03.2011 11:06, schrieb Fredrik Gustafsson:\n> On Thu, Mar 24, 2011 at 04:47:41PM -0700, Junio C Hamano wrote:\n>> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>>\n>>>> == Preventing false \"pointers\" ==\n>>>> It's far too easy to push a gitrepo pointing to a submodule version that\n>>>> is not existing on the server. Prevent this by checking for available\n>>>> submodule versions before acceptingt the push.\n>>>\n>>> Yes, requiring to force such a push is an issue raised quite often (I\n>>> think addressing this issue would be a good starting point for people\n>>> who want to get involved in enhancing the submodule experience).\n>>\n>> You need to be careful, though.\n>>\n>> That check can only be sanely done at a hosting site that hosts _both_ the\n>> superproject and the submodule repositories.  It might make more sense to\n>> have this check on the side that pushes, which by definition should have\n>> both superprojet and the submodule.  Fail, or prompt to confirm, a push\n>> from the superproject when it is detected that the submodule commits bound\n>> to the commits in the superproject have not been pushed out.\n> \n> I can see three different ways of doing this:\n> 1. the reciever (server) checks for available submodules before\n> accepting a commit\n> \n> 2. the sender checks for available submodules on the server before\n> sending a commit.\n> \n> 3. each submodule contains a flag that tells if the last command was a\n> commit or a push. The core-repository wont allow a push if one (or more)\n> submodules has a commit as \"last command\".\n> \n> Although alternative 1 is the only one that is certain of preventing the\n> problem with \"false pointers\", it has several other drawbacks. I believe\n> that alt. 3 is the proper way of handling this in git. Although we\n> doesn't guarantee a sane server-repo we does prevent the client from\n> doing stupid things by mistake.\n\nI concur with Junio and am in favor of the following solution, as I\nthink we already have all the information needed present on the side\nthat pushes:\n\nBefore doing a push in the superproject collect all submodule commits\nthat are recorded in the commits to be pushed out to the superproject's\nremote. Then check that they all are contained in at least one remote\nref of the submodule they are recorded for. If not, error out and tell\nthe user he should push the submodule first or has to use the \"-f\" flag\nto force the push if he really knows what he is doing. Commits of those\nsubmodules that aren't checked out are ignored.\n\nThis approach would achieve the goal you stated in your last sentence,\nwhich me thinks is a sane thing to do.\n"}]}