{"thread":{"id":"44246","subject":"Uninitialized submodules as symlinks","startedAt":"2016-10-07T18:17:15Z","lastAt":"2016-10-17T15:12:50Z","messageCount":10,"participants":["David Turner","Stefan Beller","Heiko Voigt","Kevin Daudt","Junio C Hamano","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"303597","messageId":"fd5bcf57f92944c0b7f6f2f8342c342c@exmbdft7.ad.twosigma.com","threadId":"44246","inReplyTo":null,"subject":"Uninitialized submodules as symlinks","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2016-10-07T18:17:05Z","receivedAt":"2016-10-07T18:17:15Z","isPatch":false,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"Presently, uninitialized submodules are materialized in the working tree as empty directories. We would like to consider having them be symlinks. Specifically, we'd like them to be symlinks into a FUSE filesystem which retrieves files on demand.\n\nWe've actually already got a FUSE filesystem written, but we use a different (semi-manual) means to connect it to the initialized submodules. We hope to release this FUSE filesystem as free software at some point soon, but we do not yet have a fixed schedule for doing so. Having to run a command to create the symlink-based \"union\" filesystem is not optimal (since we have to re-run it every time we initialize or deinitialize a submodule).\n\nBut if the uninitialized submodules could be symlinks into the FUSE filesystem, we wouldn't have this problem. This solution isn't necessarily FUSE-specific -- perhaps someone would want copies of the same submodule in multiple repos, and would want to save disk space by having all copies point to the same place. So the symlinks would be configured by a per-submodule config variable.\n\nNaturally, this would require some changes to code that examines the working tree -- git status, git diff, etc. They would have to report \"unchanged\" for submodules which were still symlinks to the configured location. I have not yet looked at the implementation details beyond this.\n\nDoes this idea make any sense? If I were to implement it (probably in a few months, but no official timeline yet), would patches be considered?\n"},{"id":"303608","messageId":"CAGZ79ka1rJCks5np9Q__HAHYFDrdGDFLtquhb3fY_iL8q9DEkw@mail.gmail.com","threadId":"44246","inReplyTo":"fd5bcf57f92944c0b7f6f2f8342c342c@exmbdft7.ad.twosigma.com","subject":"Re: Uninitialized submodules as symlinks","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-10-07T18:56:03Z","receivedAt":"2016-10-07T19:01:18Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Oct 7, 2016 at 11:17 AM, David Turner wrote:\n> Presently, uninitialized submodules are materialized in the working tree as empty directories.\n\nRight, there has to be something, to hint at the user that creating a\nfile with that\npath is probably not what they want.\n\n> We would like to consider having them be symlinks. Specifically, we'd like them to be symlinks into a FUSE filesystem which retrieves files on demand.\n>\n> We've actually already got a FUSE filesystem written, but we use a different (semi-manual) means to connect it to the initialized submodules.\n\nSo you currently do a\n\n git submodule init \n custom-submodule make-symlink \n\n?\n\n> We hope to release this FUSE filesystem as free software at some point soon, but we do not yet have a fixed schedule for doing so. Having to run a command to create the symlink-based \"union\" filesystem is not optimal (since we have to re-run it every time we initialize or deinitialize a submodule).\n>\n> But if the uninitialized submodules could be symlinks into the FUSE filesystem, we wouldn't have this problem. This solution isn't necessarily FUSE-specific -- perhaps someone would want copies of the same submodule in multiple repos, and would want to save disk space by having all copies point to the same place. So the symlinks would be configured by a per-submodule config variable.\n\nI'd imagine that you want both a per-submodule config variable as\nwell as a global variable that is a default for all submodules?\n\n git config submodule.trySymlinkDefault /mounted/fuse/\n # any (new) submodule tries to be linked to /mounted/fuse/\n git config submodule..symlinked ~/my/private/symlinked\n # The submodule goes into another path.\n\nAs you propose the FUSE filesystem fetches files on demand, you\nprobably want to disable things that scan the whole submodule,\ne.g. look at submodule..ignore to suppress status looking\nat all files.\n\nWhen looking through the options, you could add the value \"symlink\" to\nsubmodule..update, which then respects the\nsubmodule.trySymlinkDefault if present, such that\n\n git clone --recurse-submodules ...\n\nworks and sets up the FUSE thing correctly.\n\nHow does the FUSE system handle different versions, i.e.\n`git submodule update` to checkout another version of the submodule?\n(btw, I plan on working on integrating submodules to \"git checkout\",\nso \"submodule update\" would not need to be run there, but we'd hook\nit into checkout instead)\n\n>\n> Naturally, this would require some changes to code that examines the working tree -- git status, git diff, etc. They would have to report \"unchanged\" for submodules which were still symlinks to the configured location. I have not yet looked at the implementation details beyond this.\n>\n> Does this idea make any sense? If I were to implement it (probably in a few months, but no official timeline yet), would patches be considered?\n\nI am happy to review patches.\n"},{"id":"303609","messageId":"f0c11de1bd4d414d8a9464e6ccc1d58a@exmbdft7.ad.twosigma.com","threadId":"44246","inReplyTo":"CAGZ79ka1rJCks5np9Q__HAHYFDrdGDFLtquhb3fY_iL8q9DEkw@mail.gmail.com","subject":"RE: Uninitialized submodules as symlinks","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2016-10-07T19:59:13Z","receivedAt":"2016-10-07T19:59:30Z","isPatch":false,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"\n\n> -----Original Message-----\n> From: Stefan Beller [mailto:sbeller@google.com]\n> Sent: Friday, October 07, 2016 2:56 PM\n> To: David Turner\n> Cc: git@vger.kernel.org\n> Subject: Re: Uninitialized submodules as symlinks\n> \n> On Fri, Oct 7, 2016 at 11:17 AM, David Turner \n> wrote:\n> > Presently, uninitialized submodules are materialized in the working tree\n> > as empty directories.\n> \n> Right, there has to be something, to hint at the user that creating a file\n> with that path is probably not what they want.\n> \n> > We would like to consider having them be symlinks. Specifically, we'd\n> > like them to be symlinks into a FUSE filesystem which retrieves files on\n> > demand.\n> >\n> > We've actually already got a FUSE filesystem written, but we use a\n> > different (semi-manual) means to connect it to the initialized submodules.\n> \n> So you currently do a\n> \n> git submodule init \n> custom-submodule make-symlink \n> \n> ?\n\nWe do something like\n\nFor each initialized submodule: symlink it into the right place in .../somedir\nFor each uninitialized submodule: symlink from the FUSE into the right place in .../somedir\n\nSo .../somedir has the structure of the git main repo, but is all symlinks -- some into FUSE, some into the git repo.\n\nThis means that when we initialize (or deinitialize) a submodule, we need to re-run the linking script. \n\n> > We hope to release this FUSE filesystem as free software at some point\n> > soon, but we do not yet have a fixed schedule for doing so. Having to run\n> > a command to create the symlink-based \"union\" filesystem is not optimal\n> > (since we have to re-run it every time we initialize or deinitialize a\n> > submodule).\n> >\n> > But if the uninitialized submodules could be symlinks into the FUSE\n> > filesystem, we wouldn't have this problem. This solution isn't\n> > necessarily FUSE-specific -- perhaps someone would want copies of the same\n> > submodule in multiple repos, and would want to save disk space by having\n> > all copies point to the same place. So the symlinks would be configured\n> > by a per-submodule config variable.\n> \n> I'd imagine that you want both a per-submodule config variable as well as\n> a global variable that is a default for all submodules?\n> \n> git config submodule.trySymlinkDefault /mounted/fuse/\n> # any (new) submodule tries to be linked to /mounted/fuse/\n> git config submodule..symlinked ~/my/private/symlinked\n> # The submodule goes into another path.\n> \n> As you propose the FUSE filesystem fetches files on demand, you probably\n> want to disable things that scan the whole submodule, e.g. look at\n> submodule..ignore to suppress status looking at all files.\n\nI would actually expect that git would detect that the symlink is unmodified from the configured symlink and automatically decide not to look there.\n \n> When looking through the options, you could add the value \"symlink\" to\n> submodule..update, which then respects the\n> submodule.trySymlinkDefault if present, such that\n> \n> git clone --recurse-submodules ...\n> \n> works and sets up the FUSE thing correctly.\n> \n> How does the FUSE system handle different versions, i.e.\n> `git submodule update` to checkout another version of the submodule?\n> (btw, I plan on working on integrating submodules to \"git checkout\", so\n> \"submodule update\" would not need to be run there, but we'd hook it into\n> checkout instead)\n\nThe fuse has a (virtual) directory for each SHA of the main repo, with each submodule mapped to the then-current version of the submodule's code. Actually, it's a bit more complicated because the uninitialized modules point to already-built binaries -- that is, the symlink is to something like $fuse/$SHA/built/$submodule. \n\nIf you check out a new version of the main module, in our current setup, you need to again update all of the submodule symlinks (as described above). \n\nUnder my proposal, I guess this would still need to happen. A post-checkout hook could handle it either way. Despite this flaw, switching a submodule between an initialized and deinitialized state would still be more seamless with the symlinks.\n\n> > Naturally, this would require some changes to code that examines the\n> working tree -- git status, git diff, etc. They would have to report\n> \"unchanged\" for submodules which were still symlinks to the configured\n> location. I have not yet looked at the implementation details beyond\n> this.\n> >\n> > Does this idea make any sense? If I were to implement it (probably in a\n> few months, but no official timeline yet), would patches be considered?\n> \n> I am happy to review patches.\n\nThanks.\n"},{"id":"304083","messageId":"20161013161017.GC29710@book.hvoigt.net","threadId":"44246","inReplyTo":"fd5bcf57f92944c0b7f6f2f8342c342c@exmbdft7.ad.twosigma.com","subject":"Re: Uninitialized submodules as symlinks","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-10-13T16:10:17Z","receivedAt":"2016-10-13T16:12:18Z","isPatch":false,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Fri, Oct 07, 2016 at 06:17:05PM +0000, David Turner wrote:\n> Presently, uninitialized submodules are materialized in the working\n> tree as empty directories. We would like to consider having them be\n> symlinks. Specifically, we'd like them to be symlinks into a FUSE\n> filesystem which retrieves files on demand.\n\nHow about portability? This feature would only work on Unix like\noperating systems. You have to be careful to not break Windows since\nthey do not have symlinks.\n\nCheers Heiko\n"},{"id":"304091","messageId":"20161013193558.GC1698@ikke.info","threadId":"44246","inReplyTo":"20161013161017.GC29710@book.hvoigt.net","subject":"Re: Uninitialized submodules as symlinks","fromName":"Kevin Daudt","fromEmail":"me@ikke.info","sentAt":"2016-10-13T19:35:58Z","receivedAt":"2016-10-13T19:46:36Z","isPatch":false,"sender":{"key":"me@ikke.info","avatar":"https://avatars.githubusercontent.com/u/135698?v=4"},"body":"On Thu, Oct 13, 2016 at 06:10:17PM +0200, Heiko Voigt wrote:\n> On Fri, Oct 07, 2016 at 06:17:05PM +0000, David Turner wrote:\n> > Presently, uninitialized submodules are materialized in the working\n> > tree as empty directories. We would like to consider having them be\n> > symlinks. Specifically, we'd like them to be symlinks into a FUSE\n> > filesystem which retrieves files on demand.\n> \n> How about portability? This feature would only work on Unix like\n> operating systems. You have to be careful to not break Windows since\n> they do not have symlinks.\n> \n\nNTFS does have symlinks, but you need admin right to create them though\n(unless you change the policy).\n"},{"id":"304174","messageId":"xmqqr37izxkf.fsf@gitster.mtv.corp.google.com","threadId":"44246","inReplyTo":"20161013193558.GC1698@ikke.info","subject":"Re: Uninitialized submodules as symlinks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-10-14T16:48:16Z","receivedAt":"2016-10-14T16:48:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kevin Daudt writes:\n\n> On Thu, Oct 13, 2016 at 06:10:17PM +0200, Heiko Voigt wrote:\n>> On Fri, Oct 07, 2016 at 06:17:05PM +0000, David Turner wrote:\n>> > Presently, uninitialized submodules are materialized in the working\n>> > tree as empty directories. We would like to consider having them be\n>> > symlinks. Specifically, we'd like them to be symlinks into a FUSE\n>> > filesystem which retrieves files on demand.\n>> \n>> How about portability? This feature would only work on Unix like\n>> operating systems. You have to be careful to not break Windows since\n>> they do not have symlinks.\n>\n> NTFS does have symlinks, but you need admin right to create them though\n> (unless you change the policy).\n\nThat sounds like saying \"It has, but it practically is not usable by\nGit as a mechanism to achieve this goal\" to me.\n\n\n\n"},{"id":"304265","messageId":"20161017092838.GC43997@book.hvoigt.net","threadId":"44246","inReplyTo":"xmqqr37izxkf.fsf@gitster.mtv.corp.google.com","subject":"Re: Uninitialized submodules as symlinks","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2016-10-17T09:28:38Z","receivedAt":"2016-10-17T09:28:50Z","isPatch":false,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Fri, Oct 14, 2016 at 09:48:16AM -0700, Junio C Hamano wrote:\n> Kevin Daudt writes:\n> \n> > On Thu, Oct 13, 2016 at 06:10:17PM +0200, Heiko Voigt wrote:\n> >> On Fri, Oct 07, 2016 at 06:17:05PM +0000, David Turner wrote:\n> >> > Presently, uninitialized submodules are materialized in the working\n> >> > tree as empty directories. We would like to consider having them be\n> >> > symlinks. Specifically, we'd like them to be symlinks into a FUSE\n> >> > filesystem which retrieves files on demand.\n> >> \n> >> How about portability? This feature would only work on Unix like\n> >> operating systems. You have to be careful to not break Windows since\n> >> they do not have symlinks.\n> >\n> > NTFS does have symlinks, but you need admin right to create them though\n> > (unless you change the policy).\n> \n> That sounds like saying \"It has, but it practically is not usable by\n> Git as a mechanism to achieve this goal\" to me.\n\nYes and that is why Git for Windows does not use them and I simplified\nto: \"Windows does not have symlinks\". For a normal user there is no such\nthing as symlinks on Windows, unfortunately.\n\nCheers Heiko\n"},{"id":"304268","messageId":"CACsJy8BAB4sqnv1GBk=K9yUuqVhW4Y=CHdON7mnM9PY08HwdGw@mail.gmail.com","threadId":"44246","inReplyTo":"f0c11de1bd4d414d8a9464e6ccc1d58a@exmbdft7.ad.twosigma.com","subject":"Re: Uninitialized submodules as symlinks","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-10-17T09:45:43Z","receivedAt":"2016-10-17T09:46:19Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Oct 8, 2016 at 2:59 AM, David Turner wrote:\n>\n>\n>> -----Original Message-----\n>> From: Stefan Beller [mailto:sbeller@google.com]\n>> Sent: Friday, October 07, 2016 2:56 PM\n>> To: David Turner\n>> Cc: git@vger.kernel.org\n>> Subject: Re: Uninitialized submodules as symlinks\n>>\n>> On Fri, Oct 7, 2016 at 11:17 AM, David Turner \n>> wrote:\n>> > Presently, uninitialized submodules are materialized in the working tree\n>> > as empty directories.\n>>\n>> Right, there has to be something, to hint at the user that creating a file\n>> with that path is probably not what they want.\n>>\n>> > We would like to consider having them be symlinks. Specifically, we'd\n>> > like them to be symlinks into a FUSE filesystem which retrieves files on\n>> > demand.\n>> >\n>> > We've actually already got a FUSE filesystem written, but we use a\n>> > different (semi-manual) means to connect it to the initialized submodules.\n>>\n>> So you currently do a\n>>\n>> git submodule init \n>> custom-submodule make-symlink \n>>\n>> ?\n>\n> We do something like\n>\n> For each initialized submodule: symlink it into the right place in .../somedir\n> For each uninitialized submodule: symlink from the FUSE into the right place in .../somedir\n>\n> So .../somedir has the structure of the git main repo, but is all symlinks -- some into FUSE, some into the git repo.\n>\n> This means that when we initialize (or deinitialize) a submodule, we need to re-run the linking script.\n\nDo .git files work? If .git files point to somewhere in fuse, I guess\nyou still have file retrieval on demand. It depends on what files to\nretrieve I guess. If you want worktree files, not object database then\n.git files won't work because worktree remains in the same filesystem\nas the super repo.\n-- \nDuy\n"},{"id":"304277","messageId":"a50ecef48329481b8c2aa7c0c7d78946@exmbdft7.ad.twosigma.com","threadId":"44246","inReplyTo":"20161013161017.GC29710@book.hvoigt.net","subject":"RE: Uninitialized submodules as symlinks","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2016-10-17T15:11:17Z","receivedAt":"2016-10-17T15:11:24Z","isPatch":false,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"\n\n> -----Original Message-----\n> From: Heiko Voigt [mailto:hvoigt@hvoigt.net]\n> Sent: Thursday, October 13, 2016 12:10 PM\n> To: David Turner\n> Cc: git@vger.kernel.org\n> Subject: Re: Uninitialized submodules as symlinks\n> \n> On Fri, Oct 07, 2016 at 06:17:05PM +0000, David Turner wrote:\n> > Presently, uninitialized submodules are materialized in the working\n> > tree as empty directories. We would like to consider having them be\n> > symlinks. Specifically, we'd like them to be symlinks into a FUSE\n> > filesystem which retrieves files on demand.\n> \n> How about portability? This feature would only work on Unix like operating\n> systems. You have to be careful to not break Windows since they do not\n> have symlinks.\n\nWindows doesn't support FUSE either IIRC. Since this would be an alternate mode of operation, Windows would still work fine on the old model.\n"},{"id":"304278","messageId":"a1bad11df5c549a68d6fab1ade5a36b8@exmbdft7.ad.twosigma.com","threadId":"44246","inReplyTo":"CACsJy8BAB4sqnv1GBk=K9yUuqVhW4Y=CHdON7mnM9PY08HwdGw@mail.gmail.com","subject":"RE: Uninitialized submodules as symlinks","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2016-10-17T15:12:43Z","receivedAt":"2016-10-17T15:12:50Z","isPatch":false,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> From: Duy Nguyen [mailto:pclouds@gmail.com]\n> Sent: Monday, October 17, 2016 5:46 AM\n> To: David Turner\n> Cc: Stefan Beller; git@vger.kernel.org\n> Subject: Re: Uninitialized submodules as symlinks\n> \n> On Sat, Oct 8, 2016 at 2:59 AM, David Turner \n> wrote:\n> >\n> >\n> >> -----Original Message-----\n> >> From: Stefan Beller [mailto:sbeller@google.com]\n> >> Sent: Friday, October 07, 2016 2:56 PM\n> >> To: David Turner\n> >> Cc: git@vger.kernel.org\n> >> Subject: Re: Uninitialized submodules as symlinks\n> >>\n> >> On Fri, Oct 7, 2016 at 11:17 AM, David Turner\n> >> \n> >> wrote:\n> >> > Presently, uninitialized submodules are materialized in the working\n> >> > tree as empty directories.\n> >>\n> >> Right, there has to be something, to hint at the user that creating a\n> >> file with that path is probably not what they want.\n> >>\n> >> > We would like to consider having them be symlinks. Specifically,\n> >> > we'd like them to be symlinks into a FUSE filesystem which\n> >> > retrieves files on demand.\n> >> >\n> >> > We've actually already got a FUSE filesystem written, but we use a\n> >> > different (semi-manual) means to connect it to the initialized\n> submodules.\n> >>\n> >> So you currently do a\n> >>\n> >> git submodule init \n> >> custom-submodule make-symlink \n> >>\n> >> ?\n> >\n> > We do something like\n> >\n> > For each initialized submodule: symlink it into the right place in\n> > .../somedir For each uninitialized submodule: symlink from the FUSE\n> > into the right place in .../somedir\n> >\n> > So .../somedir has the structure of the git main repo, but is all\n> symlinks -- some into FUSE, some into the git repo.\n> >\n> > This means that when we initialize (or deinitialize) a submodule, we\n> need to re-run the linking script.\n> \n> Do .git files work? If .git files point to somewhere in fuse, I guess you\n> still have file retrieval on demand. It depends on what files to retrieve\n> I guess. If you want worktree files, not object database then .git files\n> won't work because worktree remains in the same filesystem as the super\n> repo.\n\nYes, we want worktree files (or even worktree files + built artifacts).\n"}]}