{"thread":{"id":"27714","subject":"[GSoC 11 submodule] Status update","startedAt":"2011-06-27T19:34:44Z","lastAt":"2011-06-30T17:54:19Z","messageCount":6,"participants":["Fredrik Gustafsson","Phil Hord","Heiko Voigt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"170591","messageId":"20110627193444.GA26778@paksenarrion.iveqy.com","threadId":"27714","inReplyTo":null,"subject":"[GSoC 11 submodule] Status update","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2011-06-27T19:34:44Z","receivedAt":"2011-06-27T19:34:44Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"Hi,\ntime for a status update on the git submodule improvements GSoC 11 project.\nThis will be divided in to three section, technical progress, personal\nreflection and how to follow my work.\n\nYou can read my previous status update here [1].\n\nTechnical progress\n------------------\nThree patch-series has been sent with a total of five patches (six if you\ncount patches that I don't have written).\n\nFirst a patch series to make git submodule update continue update other submodules\nwhen one submodule fails to be updated [2]. This patch is now in Junios\npu-branch.\n\nSecond a minor patch to reduce memory consumption. The improvements was\npretty small, but I took it as an exercise to write and send patches (in the\nfirst patch-series I'd too many non-code related error for my taste). This\npatch was however rejected [3].\n\nMy third patch series was about making push submodule aware to prevent the\nuser to forgot to push. This is currently sent to the list as an RFC [4].\nThis was the most challenging patch to write and a good start for my next\ntask.\n\nMy fourth task (and the main task of this summer) will start on June 27 and\nwill be to move a submodules .git-dir into the super-projects .git-dir.\nDesign of this is already done and approved by my mentors.\n\nPersonal reflection\n-------------------\nBefore starting this project I was a frequent git user. I use git every\nday. Apparently, you can be a frequent git user without using the power of\ngit. I've learned a lot of git as a tool the past weeks. I've learned git\nrebase very well, a tool I never used before. It's really useful and\ndangerous. Other commands I started to use (but not had any need for\nbefore) was cherry-pick and branches.\n\nI can clearly see that git can be a huge problem in a workspace with\npower-git-users and unmotivated anti-scm users.\n\nAlthough I was familiar with valgrind and gdb before, I never had any\nreally use for the tools in my development. Actually, they are really good\nand I believe I have a lot to learn on this area.\n\nMy start has been very slow, it was harder than I thought to write a proper\npatch and I spent a lot of time on formalities and test-writing, parts that\nI previously thought where small or non-existing.\n\nFor the first time I've truly used a test driven development cycle. My\nschool experience of this was very bad. It was hard, slow and not needed.\nHowever, now I learned that you actually can benefit from a test-driven\ndevelopment cycle, that it can save time and that tests actually can be\nfairly easy to write.\n\nThe mentor/student communication has also been something completely new.\nHow much help can you ask for? That's a very hard question. My mentors\nfortunately helps me with this as well. And so far this has been a really\ngood support. It's far, far better than the support I've at school.\n\nThe code review cycle is pretty amazing. All code that I submits is viewed\nby at least three persons, probably ever more. Even though a few bugs has\nslipped past to the last guard (Junio) and this proves the real value of\ncode reviewing. Git has by far the most serious code review process I ever\nworked with. This is amazing and gives a good platform for having good code\nwritten. This is also something that we do not practice in school.\n\nHow to follow my work\n---------------------\nYou can follow my work on github [5]. There are as of now five interesting\nbranches:\n\n* gsoc11_submodule_enhancements\n  Contains all patches sent to the list (this does not include RFC\n  patches). This will always be clean and \"stable\".\n* git-submodule-update\n  Contains all commits for the git-submodule update patch series. This is\n  to be considered \"stable\"\n* buggfix\n  Contains the minor patch to reduce memory consumption. This wasn't\n  accepted by Junio. This is to be considered \"stable\".\n* push_limits\n  Contains all commits for the git push limitation patch series. This is\n  subject to unstable updates.\n* move_gitdir\n  Contains all commits for the move git-dir patch-series. This is subject\n  to unstable updates.\n\nLinks\n-----\n[1] My previous update\nhttp://article.gmane.org/gmane.comp.version-control.git/173095/match=iveqy\n\n[2] submodule update continue patch.\nhttp://thread.gmane.org/gmane.comp.version-control.git/175500/focus=175725\n\n[3] Use correct value when hinting strbuf_read()\nhttp://article.gmane.org/gmane.comp.version-control.git/175844/match=iveqy\n\n[4] push checks for unpushed remotes in submodules\nhttp://thread.gmane.org/gmane.comp.version-control.git/176328/focus=176327\n\n[5] Github-page\nhttps://github.com/iveqy/git\n-- \nMed vänliga hälsningar\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"},{"id":"170601","messageId":"4E09205E.2080904@cisco.com","threadId":"27714","inReplyTo":"20110627193444.GA26778@paksenarrion.iveqy.com","subject":"Re: [GSoC 11 submodule] Status update","fromName":"Phil Hord","fromEmail":"hordp@cisco.com","sentAt":"2011-06-28T00:29:18Z","receivedAt":"2011-06-28T00:29:18Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"Hi Fredrik and git-submodule folks,\n\nOn 06/27/2011 03:34 PM, Fredrik Gustafsson wrote:\n> My fourth task (and the main task of this summer) will start on June 27 and\n> will be to move a submodules .git-dir into the super-projects .git-dir.\n> Design of this is already done and approved by my mentors.\n\nThis frightens me a bit, so I read the wiki link about it.  Thanks for\nexplaining where I can find this information.\n\nBut I'm still confused.\n\nIf I understand right, the submodule/.git dirs will be moved into the\ntop-level at .git/submodule/.git.  The benefit is supposed to be that\nthis will free up contention on the non-empty submodule directory when\nthe super-project switches branches.\n\nIn the simple case, git warns \"unable to rmdir sub: Directory not\nempty\".  But I can think of other conflicts as well.\n\nMy question is, how does this proposed change help the situation?\n\nPhil\n"},{"id":"170638","messageId":"20110628184358.GA3700@book.hvoigt.net","threadId":"27714","inReplyTo":"4E09205E.2080904@cisco.com","subject":"Re: [GSoC 11 submodule] Status update","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2011-06-28T18:43:58Z","receivedAt":"2011-06-28T18:43:58Z","isPatch":false,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"Hi,\n\nOn Mon, Jun 27, 2011 at 08:29:18PM -0400, Phil Hord wrote:\n> On 06/27/2011 03:34 PM, Fredrik Gustafsson wrote:\n> > My fourth task (and the main task of this summer) will start on June 27 and\n> > will be to move a submodules .git-dir into the super-projects .git-dir.\n> > Design of this is already done and approved by my mentors.\n> \n> This frightens me a bit, so I read the wiki link about it.  Thanks for\n> explaining where I can find this information.\n\nI do not know what part of this change frightens you?\n\n> But I'm still confused.\n> \n> If I understand right, the submodule/.git dirs will be moved into the\n> top-level at .git/submodule/.git.  The benefit is supposed to be that\n> this will free up contention on the non-empty submodule directory when\n> the super-project switches branches.\n> \n> In the simple case, git warns \"unable to rmdir sub: Directory not\n> empty\".  But I can think of other conflicts as well.\n> \n> My question is, how does this proposed change help the situation?\n\nThe proposed change allows us to implement that a submodules directory\ncan be completely removed if it was deleted or moved. If we would do\nthat currently you would loose all local history of the submodule. I do\nnot know what you mean with \"conflicts\" but this change will help\nsubmodule towards behaving like they were ordinary directories in git.\n\nCheers Heiko\n"},{"id":"170681","messageId":"4E0B98CA.2050207@cisco.com","threadId":"27714","inReplyTo":"20110628184358.GA3700@book.hvoigt.net","subject":"Re: [GSoC 11 submodule] Status update","fromName":"Phil Hord","fromEmail":"hordp@cisco.com","sentAt":"2011-06-29T21:27:38Z","receivedAt":"2011-06-29T21:27:38Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On 06/28/2011 02:43 PM, Heiko Voigt wrote:\n> Hi,\n>\n> On Mon, Jun 27, 2011 at 08:29:18PM -0400, Phil Hord wrote:\n>> On 06/27/2011 03:34 PM, Fredrik Gustafsson wrote:\n>>> My fourth task (and the main task of this summer) will start on June 27 and\n>>> will be to move a submodules .git-dir into the super-projects .git-dir.\n>>> Design of this is already done and approved by my mentors.\n>> This frightens me a bit, so I read the wiki link about it.  Thanks for\n>> explaining where I can find this information.\n> I do not know what part of this change frightens you?\n\nIt frightens me because it seems like a fundamental break from the\ncurrent submodule functionality.  Today a submodule exists as a git\nrepository with no knowledge that it is a submodule or who its\nsuper-repository is. Maybe this is a design mistake in need of\ncorrection.  But this change seems both huge and subtle to me.  I\nsuspect it will affect many tools which expect the traditional git\nlayout, submodules or not.  For example, a third-party tool might seek\nout the \".git\" directory by walking upwards.  Once it finds it, it will\n(safely, today) assume that is the .git directory relating to its\nfiles.  After this change, the tool will be broken.\n\n    # Find my .git directory\n    mygit=${PWD}\n    while test -d \"${mygit}\" && ! test -d \"${mygit}/.git\" ; do\n        mygit=$(dirname \"$mygit\")\n    done\n\n    # Now lets do things with \"our\" repo\n    fetched_sha1=$(cat ${mygit}/FETCH_HEAD)\n        .\n        .\n        .\n\nGranted, script-writers should be using the git plumbing as much as\npossible to avoid this kind of change.  But not everyone can afford to\nbe so conscientious.  \n\n>> But I'm still confused.\n>>\n>> If I understand right, the submodule/.git dirs will be moved into the\n>> top-level at .git/submodule/.git.  The benefit is supposed to be that\n>> this will free up contention on the non-empty submodule directory when\n>> the super-project switches branches.\n>>\n>> In the simple case, git warns \"unable to rmdir sub: Directory not\n>> empty\".  But I can think of other conflicts as well.\n>>\n>> My question is, how does this proposed change help the situation?\n> The proposed change allows us to implement that a submodules directory\n> can be completely removed if it was deleted or moved. If we would do\n> that currently you would loose all local history of the submodule. I do\n> not know what you mean with \"conflicts\" but this change will help\n> submodule towards behaving like they were ordinary directories in git.\n\nI see.  By moving the local history out of the way, the submodule\ndirectory is free to be changed or removed without harming the local\nhistory.  That's clever.  I think Android's 'repo' tool does something\nsimilar.\n\nI think I can answer my other concerns now.  Do these answers sound right?\n\nWhat happens if the submodule working directory is dirty?  \n    Treat it the same as git does for its own working directory.\n\nBut what if the submodule working directory is 'clean' after considering\n.gitignore?  Do untracked/ignored files also get deleted?\n    Treat this the same as git does for its own working directory.\n\nWhat if a 'git checkout' results in the submodule being removed?\n    Remove the entire submodule directory (or just remove tracked files?)\n\nWhat if a 'git checkout' or 'git merge' results in submodule 'foo' being\nadded where there is already a file named 'foo'?\n    This is a working-directory merge conflict.\n\nThanks for explaining.  I feel better about it all now.  I remain\nconcerned about backwards compatibility, but I'm not so worried about\nconflict-resolution anymore.\n\nPhil\n"},{"id":"170685","messageId":"7vpqlws3jb.fsf@alter.siamese.dyndns.org","threadId":"27714","inReplyTo":"4E0B98CA.2050207@cisco.com","subject":"Re: [GSoC 11 submodule] Status update","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-29T21:58:48Z","receivedAt":"2011-06-29T21:58:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phil Hord <hordp@cisco.com> writes:\n\n> It frightens me because it seems like a fundamental break from the\n> current submodule functionality.  Today a submodule exists as a git\n> repository with no knowledge that it is a submodule or who its\n> super-repository is.\n\nThe use of .git that is a text file that records where the real directory\nis not limited to submodules.  Placing that \"real directory\" somewhere in\nthe .git directory of the superproject is merely a convention.\n\nIn other words, it does not change anything fundamental.\n\n> Once it finds it, it will\n> (safely, today) assume that is the .git directory relating to its\n> files.  After this change, the tool will be broken.\n\nThen it is already broken if it does not pay attention to .git that is not\na directory but is a text file that records where the real directory is.\n\n> I think I can answer my other concerns now.  Do these answers sound right?\n>\n> - What happens if the submodule working directory is dirty?  \n> - But what if the submodule working directory is 'clean' after considering\n>   .gitignore?  Do untracked/ignored files also get deleted?\n> - What if a 'git checkout' results in the submodule being removed?\n> - What if a 'git checkout' or 'git merge' results in submodule 'foo' being\n>   added where there is already a file named 'foo'?\n\nThese \"different\" questions are answered exactly the same way, which is:\n\n>   Treat it the same as git does for its own working directory.\n\nWhen switching to another branch, a directory that does not exist in the\nswitched-to branch needs to be removed, but we would refrain from \"rm -fr\"\nthat directory if it has any leftover cruft in it (untracked and unignored\nfiles). A submodule directory should behave in the same way.\n"},{"id":"170717","messageId":"4E0CB84B.3060600@cisco.com","threadId":"27714","inReplyTo":"7vpqlws3jb.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC 11 submodule] Status update","fromName":"Phil Hord","fromEmail":"hordp@cisco.com","sentAt":"2011-06-30T17:54:19Z","receivedAt":"2011-06-30T17:54:19Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On 06/29/2011 05:58 PM, Junio C Hamano wrote:\n> Phil Hord <hordp@cisco.com> writes:\n>> It frightens me because it seems like a fundamental break from the\n>> current submodule functionality.  Today a submodule exists as a git\n>> repository with no knowledge that it is a submodule or who its\n>> super-repository is.\n> The use of .git that is a text file that records where the real directory\n> is not limited to submodules.  Placing that \"real directory\" somewhere in\n> the .git directory of the superproject is merely a convention.\n>\n> In other words, it does not change anything fundamental.\n\nThanks for pointing that out.  I was unaware of that feature until I saw\nit discussed in another thread this week.  Even then, it was not clear\nto me that this is the same feature being employed here.\n\n> When switching to another branch, a directory that does not exist in the\n> switched-to branch needs to be removed, but we would refrain from \"rm -fr\"\n> that directory if it has any leftover cruft in it (untracked and unignored\n> files). A submodule directory should behave in the same way.\n\nThanks.  I am suitably enclued and no longer afraid.\n\nPhil\n"}]}