threads / discuss / 39545

Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

Subject: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

## tl;dr

7 messages between Jun 6, 2015 and Jun 15, 2015.

replies: 6people: 5as markdown or json

Phil Hord· Jun 6, 2015, 17:49 UTC · lore
On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
Show 20 quoted lines
>>
>> Some devs of my Team complained that with submodules it is
>> difficult to see the “full picture” of the difference
>> between two SHA1 on the root project, as the submodules
>> would just show as different SHA1s. When you Google
>> “subtree submodules” you find other opinions as well:
>>
>> Just to mention a few:
>> -
>> https://codingkilledthecat.wordpress.com/2012/04/28/why-y
>> our-company-shouldnt-use-git-submodules/ -
>> http://blogs.atlassian.com/2013/05/alternatives-to-git-su
>> bmodule-git-subtree/
>>
>> To be honest with you, I am absolutely fine with
>> submodules as I can easily leave with the “extra pain” of
>> diffing by hand recursively on submodules. But it is true
>> that it may happen to either forget to do a git submodule
>> update or otherwise forget you are in a detached branch
>> and start committing “on the air” without a branch.
...
> Ideally, as a "git clone --recursive" already exists, I would like to
> see a "git diff --recursive" that goes through the submodules as well :-)
>
> Something possibly to propose to the Git mailing list?

I've worked on git diff --recursive a bit myself, along with some simpler use cases (git ls-tree --recursive) as POCs. I think some of the needs there begin to have ui implications which could be high-friction. I really want to finish it someday, but I've been too busy lately at $job, and now my experiments are all rather stale.

It would be a good discussion to have over at the git list (copied). Heiko and Jens have laid some new groundwork in this area and it may be a good time to revisit it. Or maybe they've even moved deeper than that; I have been distracted for well over a year now.

Phil
-- 
-- 
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Luca Milanesio· Jun 6, 2015, 19:53 UTC · re: Phil Hord · lore

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

Thank you Phil, you anticipated me :-)
Luca.
Show 44 quoted lines
> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
> 
> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>> 
>>> Some devs of my Team complained that with submodules it is
>>> difficult to see the “full picture” of the difference
>>> between two SHA1 on the root project, as the submodules
>>> would just show as different SHA1s. When you Google
>>> “subtree submodules” you find other opinions as well:
>>> 
>>> Just to mention a few:
>>> -
>>> https://codingkilledthecat.wordpress.com/2012/04/28/why-y
>>> our-company-shouldnt-use-git-submodules/ -
>>> http://blogs.atlassian.com/2013/05/alternatives-to-git-su
>>> bmodule-git-subtree/
>>> 
>>> To be honest with you, I am absolutely fine with
>>> submodules as I can easily leave with the “extra pain” of
>>> diffing by hand recursively on submodules. But it is true
>>> that it may happen to either forget to do a git submodule
>>> update or otherwise forget you are in a detached branch
>>> and start committing “on the air” without a branch.
> 
> ...
> 
>> Ideally, as a "git clone --recursive" already exists, I would like to
>> see a "git diff --recursive" that goes through the submodules as well :-)
>> 
>> Something possibly to propose to the Git mailing list?
> 
> 
> I've worked on git diff --recursive a bit myself, along with some
> simpler use cases (git ls-tree --recursive) as POCs. I think some of
> the needs there begin to have ui implications which could be
> high-friction. I really want to finish it someday, but I've been too
> busy lately at $job, and now my experiments are all rather stale.
> 
> It would be a good discussion to have over at the git list (copied).
> Heiko and Jens have laid some new groundwork in this area and it may
> be a good time to revisit it.  Or maybe they've even moved deeper than
> that; I have been distracted for well over a year now.
> 
> Phil
Stefan Beller· Jun 7, 2015, 06:26 UTC · re: Luca Milanesio · lore

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

On 06.06.2015 12:53, Luca Milanesio wrote:
Show 47 quoted lines
> Thank you Phil, you anticipated me :-)
> 
> Luca.
> 
>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>
>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>>>
>>>> Some devs of my Team complained that with submodules it is
>>>> difficult to see the “full picture” of the difference
>>>> between two SHA1 on the root project, as the submodules
>>>> would just show as different SHA1s. When you Google
>>>> “subtree submodules” you find other opinions as well:
>>>>
>>>> Just to mention a few:
>>>> -
>>>> https://codingkilledthecat.wordpress.com/2012/04/28/why-y
>>>> our-company-shouldnt-use-git-submodules/ -
>>>> http://blogs.atlassian.com/2013/05/alternatives-to-git-su
>>>> bmodule-git-subtree/
>>>>
>>>> To be honest with you, I am absolutely fine with
>>>> submodules as I can easily leave with the “extra pain” of
>>>> diffing by hand recursively on submodules. But it is true
>>>> that it may happen to either forget to do a git submodule
>>>> update or otherwise forget you are in a detached branch
>>>> and start committing “on the air” without a branch.
>>
>> ...
>>
>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>> see a "git diff --recursive" that goes through the submodules as well :-)
>>>
>>> Something possibly to propose to the Git mailing list?
>>
>>
>> I've worked on git diff --recursive a bit myself, along with some
>> simpler use cases (git ls-tree --recursive) as POCs. I think some of
>> the needs there begin to have ui implications which could be
>> high-friction. I really want to finish it someday, but I've been too
>> busy lately at $job, and now my experiments are all rather stale.
>>
>> It would be a good discussion to have over at the git list (copied).
>> Heiko and Jens have laid some new groundwork in this area and it may
>> be a good time to revisit it.  Or maybe they've even moved deeper than
>> that; I have been distracted for well over a year now.
>>

Glad you're working (or planning to) working on submodulues. This is also on my todo list for the next months as well.

I'd review stuff in that area if you're looking for reviewers.
Stefan
Show 7 quoted lines
>> Phil
> 
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Jens Lehmann· Jun 9, 2015, 18:40 UTC · re: Stefan Beller · lore

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

Am 07.06.2015 um 08:26 schrieb Stefan Beller:
Show 7 quoted lines
> On 06.06.2015 12:53, Luca Milanesio wrote:
>>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>>> see a "git diff --recursive" that goes through the submodules as well :-)
>>>>
>>>> Something possibly to propose to the Git mailing list?

Such an option makes lots of sense to me (though "--recurse-submodules" should be its name for consistency reasons). This could be an alias for "--submodule=full", as the "--submodule" option controls the format of submodule diffs.

Show 14 quoted lines
>>> I've worked on git diff --recursive a bit myself, along with some
>>> simpler use cases (git ls-tree --recursive) as POCs. I think some of
>>> the needs there begin to have ui implications which could be
>>> high-friction. I really want to finish it someday, but I've been too
>>> busy lately at $job, and now my experiments are all rather stale.
>>>
>>> It would be a good discussion to have over at the git list (copied).
>>> Heiko and Jens have laid some new groundwork in this area and it may
>>> be a good time to revisit it.  Or maybe they've even moved deeper than
>>> that; I have been distracted for well over a year now.
>>>
>
> Glad you're working (or planning to) working on submodulues. This is
> also on my todo list for the next months as well.
More hands are always welcome!
> I'd review stuff in that area if you're looking for reviewers.
I'll be happy help too.
Phil Hord· Jun 11, 2015, 16:11 UTC · re: Jens Lehmann · lore

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

On Tue, Jun 9, 2015 at 2:40 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
Show 19 quoted lines
> Am 07.06.2015 um 08:26 schrieb Stefan Beller:
>>
>> On 06.06.2015 12:53, Luca Milanesio wrote:
>>>>
>>>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com>
>>>> wrote:
>>>>>
>>>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>>>> see a "git diff --recursive" that goes through the submodules as well
>>>>> :-)
>>>>>
>>>>> Something possibly to propose to the Git mailing list?
>
>
> Such an option makes lots of sense to me (though "--recurse-submodules"
> should be its name for consistency reasons). This could be an alias for
> "--submodule=full", as the "--submodule" option controls the format of
> submodule diffs.

To me, --recurse-submodules means submodules are still not first-class citizens. But let's put that aside for a moment; I don't care about the switch name too much as long as I can configure 'diff.recurse-submodules = true'.

[The following is rather long. I'm sorry for that. Feel free to look away when it gets too vague.]

Let me set up a submodule like so:
  $ git init /tmp/Super && cd /tmp/Super
  Super$ git submodule add https://github.com/gitster/git.git Foo
I wish to be able to grep from Super and find matches in all my submodules.
  Super$ git grep --recurse-submodules base--int
  Foo/.gitignore:/git-rebase--interactive
  Foo/Makefile:SCRIPT_LIB += git-rebase--interactive

But I want this to work naturally across git-module boundaries, so I want this also to work (grepping a super-project from within a submodule):

  Super$ cd Foo
  Foo$ git grep --recurse-submodules base--int ..
  .gitignore:/git-rebase--interactive
  Makefile:SCRIPT_LIB += git-rebase--interactive

I expect some groans from the audience here, because I think if the syntax above worked, then so would this:

  $ cd /tmp
  tmp$ git grep base--int /tmp/Super/Foo
  /tmp/Super/Foo/.gitignore:/git-rebase--interactive
  /tmp/Super/Foo/Makefile:SCRIPT_LIB += git-rebase--interactive

This usage has nothing to do with submodules, really, except that it allows git commands to reach into foreign git directories by virtue of the path supplied as some argument instead of via $GITDIR, and in doing so it helps solve some git submodules use cases of mine.

But if that did not turn your stomach, try this one:
  $ cd /tmp/Super
  Super$ printf "Some submodule data">Foo/data.txt
  Super$ git add Foo/data.txt
  fatal: Pathspec 'Foo/data.txt' is in submodule 'Foo'
  Super$ git add --recurse-submodules Foo/data.txt
Some notes on this usage:
1. --recurse-submodules seems like a reasonable name for this switch,
especially when you consider the 'git add --recurse-submodules .' use
case.
2. This recursive 'git add' seems dangerous to me unless git-status
also shows all the changed/untracked files in submodules as well if
the --recurse-submodules switch is included.  This would support the
expectation that 'git add .' is going to add the files shown by 'git
status .'
3. Configuring --recurse-submodules as the default mode for 'git add'
but not for 'git status' seems reckless enough that I think there
should not be separate options for these two commands.  There are
probably many other "cross-command" scenarios with similar coupling.

Moving on, as we have :/ to mean 'workdir root', I wonder how you would spell "super-project workdir root". Maybe it would be ::/

I realize the kinds of features I'm talking about require extensive code changes in Git. For example, consider the meaning of this:

  Super$ git diff --recurse-submodules origin/next origin/master

Since I created Super just a few minutes ago and it has no remote named 'origin', this command seems meaningless to me. But suppose that origin/next and origin/master did exist in my Super project. Then, I would expect in my wishlist Git, that

A. Super$ git diff --recurse-submodules origin/next origin/master This would include differences in Foo between origin/master:Foo and origin/next:Foo; that is, the commits referenced from those gitlinks in Super.

B. Super$ git diff --recurse-submodules origin/next HEAD This would include differences in Foo between origin/master:Foo and HEAD:Foo; that is, the commits referenced from those gitlinks in Super.

C. Super$ git diff --recurse-submodules origin/next This would include differences in Foo between origin/master:Foo and the current Foo workdir.

D. Super$ cd Foo && git diff origin/next This would include differences in Foo between the Foo submodule's origin/master and the current Foo workdir.

Now, C and D seem confusingly similar to me and technically very different. I could understand the results, but I could easily be led astray, especially if I am writing a script. But I still think it is reasonable and correct.

I think this could have dire consequences for some commands like 'git apply'. But I think it is reasonable for git apply to reject such cross-project diffs, at least in the beginning. :-)

While I am thinking about it, let me also mention these cases: E. Super$ git diff --recurse-submodules origin/next origin/master -- Foo I think 'origin/next' and 'origin/master' here are referring to Super's refs, but I can imagine an implementer choosing to use Foo's instead.

F.  Super$ cd Foo
      Foo$ git diff --recurse-submodules origin/next origin/master -- ..
If this worked, I would think 'origin/next' and 'origin/master' here
must refer to Super's refs even though I began in Foo.  This one is so
ambiguous I think I would have to call this an error.  More
specifically, I think it would have to be rewritten like this next one
(G).
G.  Super$ cd Foo
      Foo$ git -C .. diff --recurse-submodules origin/next origin/master
That is, at least for 'git diff', the <path> parameter at the end is
only used to filter the results; it is not used to find the git-dir.

But look at me speaking in the present tense. How silly. I live too much in my own imagination.

Phil
Jens Lehmann· Jun 11, 2015, 18:56 UTC · re: Phil Hord · lore

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

Am 11.06.2015 um 18:11 schrieb Phil Hord:
Show 25 quoted lines
> On Tue, Jun 9, 2015 at 2:40 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 07.06.2015 um 08:26 schrieb Stefan Beller:
>>>
>>> On 06.06.2015 12:53, Luca Milanesio wrote:
>>>>>
>>>>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>>>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>>>>> see a "git diff --recursive" that goes through the submodules as well
>>>>>> :-)
>>>>>>
>>>>>> Something possibly to propose to the Git mailing list?
>>
>>
>> Such an option makes lots of sense to me (though "--recurse-submodules"
>> should be its name for consistency reasons). This could be an alias for
>> "--submodule=full", as the "--submodule" option controls the format of
>> submodule diffs.
>
> To me, --recurse-submodules means submodules are still not first-class
> citizens.  But let's put that aside for a moment; I don't care about
> the switch name too much as long as I can configure
> 'diff.recurse-submodules = true'.

After somebody implemented the 'full' mode for 'diff --submodule', setting 'diff.submodule' to 'full' would make --recurse-submodules the default for diff (unless recursing into the submodules is overridden by either the global 'diff.ignoreSubmodules' or the per-submodule 'submodule.<name>.ignore' setting of course).

> [The following is rather long.  I'm sorry for that.  Feel free to look
> away when it gets too vague.]
Sorry, that was too long for todays git time budget ;-)
Heiko Voigt· Jun 15, 2015, 09:03 UTC · re: Jens Lehmann · lore

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

On Tue, Jun 09, 2015 at 08:40:14PM +0200, Jens Lehmann wrote:
Show 13 quoted lines
> Am 07.06.2015 um 08:26 schrieb Stefan Beller:
> >On 06.06.2015 12:53, Luca Milanesio wrote:
> >>>On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
> >>>On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
> >>>>Ideally, as a "git clone --recursive" already exists, I would like to
> >>>>see a "git diff --recursive" that goes through the submodules as well :-)
> >>>>
> >>>>Something possibly to propose to the Git mailing list?
> 
> Such an option makes lots of sense to me (though "--recurse-submodules"
> should be its name for consistency reasons). This could be an alias for
> "--submodule=full", as the "--submodule" option controls the format of
> submodule diffs.

BTW, for long running topics (or low hanging fruits) we collect/link everything in the wiki of Jens git fork on github. This is the central page:

https://github.com/jlehmann/git-submod-enhancements/wiki

Maybe everyone that has work in the queue can add his work there (the work that takes more time) so we can avoid doubling any effort. Not everything there is up to date at the moment but I will look into it to remove outdated things.

Show 20 quoted lines
> >>>I've worked on git diff --recursive a bit myself, along with some
> >>>simpler use cases (git ls-tree --recursive) as POCs. I think some of
> >>>the needs there begin to have ui implications which could be
> >>>high-friction. I really want to finish it someday, but I've been too
> >>>busy lately at $job, and now my experiments are all rather stale.
> >>>
> >>>It would be a good discussion to have over at the git list (copied).
> >>>Heiko and Jens have laid some new groundwork in this area and it may
> >>>be a good time to revisit it.  Or maybe they've even moved deeper than
> >>>that; I have been distracted for well over a year now.
> >>>
> >
> >Glad you're working (or planning to) working on submodulues. This is
> >also on my todo list for the next months as well.
> 
> More hands are always welcome!
> 
> >I'd review stuff in that area if you're looking for reviewers.
> 
> I'll be happy help too.
Me too.
Cheers Heiko

← back to recent threads