{"thread":{"id":"46209","subject":"in case you want a use-case with lots of submodules","startedAt":"2017-06-19T16:14:54Z","lastAt":"2017-06-20T05:43:44Z","messageCount":4,"participants":["Yaroslav Halchenko","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"322572","messageId":"20170619155924.7ra2vwvaelr2yj2v@hopa.kiewit.dartmouth.edu","threadId":"46209","inReplyTo":null,"subject":"in case you want a use-case with lots of submodules","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2017-06-19T15:59:24Z","receivedAt":"2017-06-19T16:14:54Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"Hi All,\n\nOn a recent trip I've listened to the git minutes podcast episode and\ngot excited to hear  Stefan Beller (CCed just in case) describing\nongoing work on submodules mechanism.  I got excited, since e.g.\nperformance improvements would be of great benefit to us too.\n\nIn our project, http://datalad.org, git submodules is the basic\nmechanism to bring multiple \"datasets\" (mix of git and git-annex'ed\nrepositories)  under the same roof so we could non-ambiguously\nversion them all at any level.\n\nhttp://datasets.datalad.org ATM provides quite a sizeable (ATM 370\nrepositories, up to 4 levels deep) hierarchy of git/git-annex\nrepositories all tied together via git submodules mechanism.  And as the\ncollection grows, interactions with it become slower, so additional\noptions (such as --ignore-submodules=dirty  to status) become our\nfriends.\n\nSo I thought to share this as a use-case happen you need more\nmotivation or just a real-case test-bed for your work.  And thank\nyou again for making Git even Greater.\n\nP.S. Please CCme in your replies (if any), I am not on the list\n\nWith best regards,\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"},{"id":"322600","messageId":"CAGZ79kZhj31eBYnboyxDLuFp1ceeqk8kj0nrnQaCmpRJCVFU4w@mail.gmail.com","threadId":"46209","inReplyTo":"20170619155924.7ra2vwvaelr2yj2v@hopa.kiewit.dartmouth.edu","subject":"Re: in case you want a use-case with lots of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-19T19:30:28Z","receivedAt":"2017-06-19T19:30:40Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jun 19, 2017 at 8:59 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:\n> Hi All,\n>\n> On a recent trip I've listened to the git minutes podcast episode and\n> got excited to hear  Stefan Beller (CCed just in case) describing\n> ongoing work on submodules mechanism.  I got excited, since e.g.\n> performance improvements would be of great benefit to us too.\n\nIf you're mostly interested in performance improvements of the status\nquo (i.e. \"make git-submodule fast\"), then the work of Prathamesh\nChavan (cc'd) might be more interesting to you than what I do.\nHe is porting git-submodule (which is mostly a shell script nowadays)\nto C, such that we can save a lot of process invocations and can do\nprocessing within one process.\n\n> In our project, http://datalad.org, git submodules is the basic\n> mechanism to bring multiple \"datasets\" (mix of git and git-annex'ed\n> repositories)  under the same roof so we could non-ambiguously\n> version them all at any level.\n\nCool, glad to here submodules being useful. :)\n\n> http://datasets.datalad.org ATM provides quite a sizeable (ATM 370\n> repositories, up to 4 levels deep) hierarchy of git/git-annex\n> repositories all tied together via git submodules mechanism.  And as the\n> collection grows, interactions with it become slower, so additional\n> options (such as --ignore-submodules=dirty  to status) become our\n> friends.\n\nI am not as much concerned about the 370 number than about the\n4 layers of nesting. In my experience the nested submodule case\nis a little bit error prone and the bug reports are not as frequent as\nthere are not as many users of nesting, yet(?)\n\nIn a neighboring thread on the mailing list we have a discussion\non the usefulness of being on branches than in detached HEAD\nin the submodules.\nhttps://public-inbox.org/git/0092CDD27C5F9D418B0F3E9B5D05BE08010287DF@SBS2011.opfingen.plc2.de/\n\nThis would not break non-ambiguously, rather it would add\nease of use.\n\n> So I thought to share this as a use-case happen you need more\n> motivation or just a real-case test-bed for your work.  And thank\n> you again for making Git even Greater.\n\nThanks for the motivation. :)\n\n> P.S. Please CCme in your replies (if any), I am not on the list\n>\n> With best regards,\n\nCheers,\nStefan\n"},{"id":"322609","messageId":"20170619202021.dmomy5ztwoeat3eg@hopa.kiewit.dartmouth.edu","threadId":"46209","inReplyTo":"CAGZ79kZhj31eBYnboyxDLuFp1ceeqk8kj0nrnQaCmpRJCVFU4w@mail.gmail.com","subject":"Re: in case you want a use-case with lots of submodules","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2017-06-19T20:20:21Z","receivedAt":"2017-06-19T20:20:45Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"\nOn Mon, 19 Jun 2017, Stefan Beller wrote:\n\n> On Mon, Jun 19, 2017 at 8:59 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:\n> > Hi All,\n\n> > On a recent trip I've listened to the git minutes podcast episode and\n> > got excited to hear  Stefan Beller (CCed just in case) describing\n> > ongoing work on submodules mechanism.  I got excited, since e.g.\n> > performance improvements would be of great benefit to us too.\n\n> If you're mostly interested in performance improvements of the status\n> quo (i.e. \"make git-submodule fast\"), then the work of Prathamesh\n> Chavan (cc'd) might be more interesting to you than what I do.\n> He is porting git-submodule (which is mostly a shell script nowadays)\n> to C, such that we can save a lot of process invocations and can do\n> processing within one process.\n\nah -- cool.  I would be eager to test it out, thanks!  would be\ninteresting to see if it positively affects our overall performance.\nPointers to that development would be welcome!\n\n> > http://datasets.datalad.org ATM provides quite a sizeable (ATM 370\n> > repositories, up to 4 levels deep) hierarchy of git/git-annex\n> > repositories all tied together via git submodules mechanism.  And as the\n> > collection grows, interactions with it become slower, so additional\n> > options (such as --ignore-submodules=dirty  to status) become our\n> > friends.\n\n> I am not as much concerned about the 370 number than about the\n> 4 layers of nesting. In my experience the nested submodule case\n> is a little bit error prone and the bug reports are not as frequent as\n> there are not as many users of nesting, yet(?)\n\nwell -- part of the story here is that we are forced to use/have full\nblown .git/ directories (for git-annex symlinks to content files to\nwork) within submodules instead of .git file with a reference under\nparent's .git/modules.   So we can 'slice' at any level and I\nguess that is why may be avoiding some possibly issues due to nesting\nand the \"parent has all .git/modules\" approach.\n\n> In a neighboring thread on the mailing list we have a discussion\n> on the usefulness of being on branches than in detached HEAD\n> in the submodules.\n> https://public-inbox.org/git/0092CDD27C5F9D418B0F3E9B5D05BE08010287DF@SBS2011.opfingen.plc2.de/\n\n> This would not break non-ambiguously, rather it would add\n> ease of use.\n\nthat is indeed a common caveat... I am not sure if any heuristic\napproach would provide a 'bullet proof' solution.  I might even prefer a\nhardcoded 'branch-name' to be listed/associated with each submodule\nwithin .gitmodules.  In the datalad case, detached HEAD is common\nwhenever someone installs \"outdated\" (branch of which progressed\nforward) submodule.  In this case we just check if the branch after \"git\nclone\"  (but before git submodule update) includes the pointed by\nSubproject commit, and if so -- we announce that it must be the branch\n(so far it is always \"master\" branch anyways ;) )\n\n> > So I thought to share this as a use-case happen you need more\n> > motivation or just a real-case test-bed for your work.  And thank\n> > you again for making Git even Greater.\n\n> Thanks for the motivation. :)\n\nthe least I could do ;)\n\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"},{"id":"322675","messageId":"CAGZ79kbVAfhgqug=UVXwriwxtckB0=x7mgxetVnQA2JFA0DYPg@mail.gmail.com","threadId":"46209","inReplyTo":"20170619202021.dmomy5ztwoeat3eg@hopa.kiewit.dartmouth.edu","subject":"Re: in case you want a use-case with lots of submodules","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-20T05:43:37Z","receivedAt":"2017-06-20T05:43:44Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jun 19, 2017 at 1:20 PM, Yaroslav Halchenko <yoh@onerussian.com> wrote:\n>\n> On Mon, 19 Jun 2017, Stefan Beller wrote:\n>\n>> On Mon, Jun 19, 2017 at 8:59 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:\n>> > Hi All,\n>\n>> > On a recent trip I've listened to the git minutes podcast episode and\n>> > got excited to hear  Stefan Beller (CCed just in case) describing\n>> > ongoing work on submodules mechanism.  I got excited, since e.g.\n>> > performance improvements would be of great benefit to us too.\n>\n>> If you're mostly interested in performance improvements of the status\n>> quo (i.e. \"make git-submodule fast\"), then the work of Prathamesh\n>> Chavan (cc'd) might be more interesting to you than what I do.\n>> He is porting git-submodule (which is mostly a shell script nowadays)\n>> to C, such that we can save a lot of process invocations and can do\n>> processing within one process.\n>\n> ah -- cool.  I would be eager to test it out, thanks!  would be\n> interesting to see if it positively affects our overall performance.\n> Pointers to that development would be welcome!\n\nThe latest from today:\nhttps://public-inbox.org/git/CAME+mvUQJFneV7b1G7zmAidP-5L=nimvY43V0ug-Gtesr83tzg@mail.gmail.com/\n\n\n>\n>> > http://datasets.datalad.org ATM provides quite a sizeable (ATM 370\n>> > repositories, up to 4 levels deep) hierarchy of git/git-annex\n>> > repositories all tied together via git submodules mechanism.  And as the\n>> > collection grows, interactions with it become slower, so additional\n>> > options (such as --ignore-submodules=dirty  to status) become our\n>> > friends.\n>\n>> I am not as much concerned about the 370 number than about the\n>> 4 layers of nesting. In my experience the nested submodule case\n>> is a little bit error prone and the bug reports are not as frequent as\n>> there are not as many users of nesting, yet(?)\n>\n> well -- part of the story here is that we are forced to use/have full\n> blown .git/ directories (for git-annex symlinks to content files to\n> work) within submodules instead of .git file with a reference under\n> parent's .git/modules.   So we can 'slice' at any level and I\n> guess that is why may be avoiding some possibly issues due to nesting\n> and the \"parent has all .git/modules\" approach.\n\nThat sounds like you either want to configure to have the submodules\ngit dirs in-place or you want to convince git-annex to learn about the\ngitdir pointer files.\n\n>\n>> In a neighboring thread on the mailing list we have a discussion\n>> on the usefulness of being on branches than in detached HEAD\n>> in the submodules.\n>> https://public-inbox.org/git/0092CDD27C5F9D418B0F3E9B5D05BE08010287DF@SBS2011.opfingen.plc2.de/\n>\n>> This would not break non-ambiguously, rather it would add\n>> ease of use.\n>\n> that is indeed a common caveat... I am not sure if any heuristic\n> approach would provide a 'bullet proof' solution.  I might even prefer a\n> hardcoded 'branch-name' to be listed/associated with each submodule\n> within .gitmodules.\n\nhardcoded as submodule.NAME.branch, maybe?\nhttps://git-scm.com/docs/gitmodules\n\n>  In the datalad case, detached HEAD is common\n\nSo you are accustomed to detached HEADs and would not\ngain much from being back on a branch?  That's cool, too.\n\n\n> whenever someone installs \"outdated\" (branch of which progressed\n> forward) submodule.  In this case we just check if the branch after \"git\n> clone\"  (but before git submodule update) includes the pointed by\n> Subproject commit, and if so -- we announce that it must be the branch\n> (so far it is always \"master\" branch anyways ;) )\n\nheh, having just one branch. That is retro-style. :)\n"}]}