[wishlist?] make submodule commands robust to having non-submodule Subprojects

NB echos some questions of mine a few days back on IRC about Subprojects and submodules

If e.g. you just 'git add' a subdirectory which is a git repository, git adds it as a subproject but doesn't initiate any entry in .gitmodules since it is the job done by submodule and git core itself is agnostic of those beasts.

But having then this "Subproject"s which aren't registered as submodules (and I haven't found any other use for them besides being a submodule) brakes "git submodule" commands, e.g.

$> git submodule
 cc6a09ac06c13cf06b4f4c8b54cda9a535e4e385 ds000001 (2.0.0+4)
 0a9f3b66e06a2137311a537b7377c336f1fb30ad ds000002 (1.0.0-3-g0a9f3b6)
 9da7e4f4221699915645ac2003298c6aba2db109 ds000003 (1.1.0+4)
 fe16cacb5cb9b4d53c50e498298fab182500e147 ds000005 (2.0.0+3)
 6898d99ff3ba26880183ed3672a458a7fcde1737 ds000006 (2.0.0+2)
 bbd10f634fe87e9d5853df3a891edbdb18cda7f9 ds000007 (2.0.0+3)
 138e6730193c0585a69b8baf5b9d7a4439e83ecc ds000008 (2.0.0+2)
 ddf3a4cf7ce51a01a664e6faff4b8334b8414b1f ds000009 (2.0.1+1)
 7fa73b4df8166dba950c7dc07c3f8cdd50fca313 ds000011 (1.0.0-5-g7fa73b4)
fatal: no submodule mapping found in .gitmodules for path 'ds000017

which then stops without even looking at other submodules.

I think it would be more logical to make it a 'warning:' not a 'fatal:' and proceed.

Thank you for consideration

On Thu, Sep 15, 2016 at 6:02 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:

So maybe we would want to introduce a switch
  `--existing-but-unconfigure-gitlinks=(warn|ignore)`
as well as
`git config submodule.existing-but-unconfigured (warn|ignore)`
for a more permanent solution?

On Thu, 15 Sep 2016, Stefan Beller wrote:

possibly ignorant question: is gitlink === Subproject or a Subproject is a kinda of a gitlink and there are other gitlinks which aren't Subprojects? ;)

On Thu, Sep 15, 2016 at 9:40 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:

gitlink is the internal name (just like a file is called blob, or a directory is called tree; there is no file system equivalent for e.g. commits, or gitlinks)

gitlinks as a basic building block only points at a sha1 to be part of the repository.

Submodules use gitlinks to point at (usually) different projects, e.g. a library.

Subprojects in Git is a loose term, it could mean submodule or part of your repo merged in via the git-subtree command. Or a sub project can be just a loose repository inside your repository.

gitlinks do not necessarily need to track other projects; it's "just a pointer" to a specific version of a repository.

So to come back to your question: yeah In that response I meant submodule==gitlink, though naming the internal data structure to be more explicit. (What if we invent a new way to do submodules not using gitlinks? Then that config option may be less confusing and backwards compatible. That is unlikely though. So I guess we'll settle with a different name)

Stefan Beller <sbeller@google.com> writes:

Making "git submodule" listing to continue from that point may be one thing, but do we have a sensible way in "git submodule add" to allow the user to recover from this condition? That is, "git add" is a right way to tell the core level that there is a gitlink, but as Yaroslav correctly observed in the early part of his message, having that gitlink alone is not good enough for the world view of "git submodule" that sits at higher-layer. And the usual way to tell the submodule layer that there is a submodule is with "git submodule add", but

	$ git init top
        $ cd top
        $ git commit --allow-empty -m 'initial in top'
        $ git init sub
        $ git -C sub commit --allow-empty -m 'initial in sub'

        $ git add sub
	$ git submodule
        fatal: no submodule mapping found in .gitmodules for path 'sub'

	$ git submodule add ./sub sub
        'sub' already exists in the index
	$ git submodule add -f ./sub sub
        'sub' already exists in the index

I highly suspect that the user will then get stuck at this point, after trying to "submodule add" and then even attempting to force it.

I think that is a more pressing thing to address. Once we make it easier for the user to bring a half-initialized submodule properly into the world view of the submodule subsystem, we would have to worry about the reported failure case even less and you do not need to pile on workaround options to let things continue in a state that is half-broken (that is, in a state that is perfectly sane to the core layer, but is not liked by the submodule layer).

On Thu, 15 Sep 2016, Junio C Hamano wrote:

FWIW I could have sworn that I have tried to 'submodule add' it and it worked... but pragmatically I just did edit .gitmodules, added the record for it, committed it, and then iirc git submodule update --init which seemed to make git happy... FTR:

$> git submodule add ./sub ./sub 'sub' already exists in the index

$> git submodule add ./sub/ ./sub/ 'sub' already exists in the index

$> vim .gitmodules

$> git add .gitmodules
cached/staged changes:                                                                                                                                                                                                                                                                                            
 .gitmodules | 4 ++++
 sub         | 1 +

$> git submodule update --init
Submodule 'sub' (/tmp/111/top/sub) registered for path 'sub'
cached/staged changes:                                                                                                                                                                                                                                                                                            
 .gitmodules | 4 ++++
 sub         | 1 +

$> git commit -m 'added finally'
[master aa6d912] added finally
 2 files changed, 5 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 sub

$> git submodule
 6f574b298ef51aebd36daafad450a3e38802ca03 sub (heads/master)

do you foresee any unpleasant side-effects from above manual editing .gitmodules/submodule update --init ?

On Thu, Sep 15, 2016 at 11:12 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:

I think that is fine, but un(der)documented. So you have to figure it out from experience what to do exactly.

Yaroslav Halchenko <yoh@onerussian.com> writes:

I do not think so; you essentially did what a canned command we should have had should have done by hand because of a lack of such a feature.

On Thu, Sep 15, 2016 at 11:02 AM, Junio C Hamano <gitster@pobox.com> wrote:

Heh, I see.

    When searching around the net, some people use half
    initialized submodules intentionally, e.g. I'll store some private keys in
    sub and I publish the superproject asking for collaborators to my new shiny
    webbased thing. The submodule containing the private keys never leaves
    my hard drive, hence no .gitmodules entry is necessary.

Not sure I agree with such a setup, but people use it.

So how about this fictional work flow:

         $ git init top
         $ cd top
         $ git commit --allow-empty -m 'initial in top'
         $ git init sub
         $ git -C sub commit --allow-empty -m 'initial in sub'
         $ git add sub
        You added a gitlink, but no corresponding entry in
        .gitmodules is found. This is fine for gits core functionality, but
        the submodule command gets confused by this unless you add 'sub'
        to your .gitmodules via `git submodule add --already-in-tree \
        --reuse-submodules-origin-as-URL sub`. Alternatively you can make this
        message disappear by configuring advice.gitlinkPitfalls.

         $ git submodule
         ... similar advice goes here...
         fatal: no submodule mapping found in .gitmodules for path 'sub'

         $ git submodule add --fixup-modules-file ./sub sub
         Adding .gitmodule entry only for `sub` to use `git -C remote
show origin` as URL.

         # user is happy now.

Stefan Beller <sbeller@google.com> writes:

In such a top-level project, people would not use "git submodule" command, would they? I do not think anybody in this thread was pushing to forbid such a use, and it may be perfectly fine if "git submodule" does not work for such a gitlink; after all such a subdirectory is not even meant to be a submodule.

I am not sure if I agree with that direction.

If the trend in Git community collectively these days is to make usage of submodules easier and smoother, I'd imagine that you would want to teach "git add" that was given a submodule to "git submodule add" instead by default, with an option "git add --no-gitmodules sub" to disable it, or something like that.

I agree that a feature like this is needed regardless of what happens at "git add" time.

On Thu, Sep 15, 2016 at 11:27:54AM -0700, Junio C Hamano wrote:

How about just

   git submodule add <submodulepath>

? I remember back in the days when I started with submodules thats the way I imagined submodules would work:

1. clone the submodule into a directory
2. git submodule add it
3. git commit everything

Because that how you basically work with files. So instead of adding another option I would rather like to autodetect that:

 * its a relative path inside this repo that is passed to
   'git submodule add'
 * there is no .gitmodules entry
 * and no .git/config
==> create those from a remote in the submodule

Corner cases:

 * If there is more than one remote we could tell the user to use an
   option to specify which one to use.
 * Barf in case there is no remote (not adding the submodule except -f
   is used).
 * If the gitlink is already there but no .gitmodules entry, 'git
   submodule add' will just add the entry as if it was initially added.

Instead of giving an error message that the submodule is already added we could actually be nicer to the user and try to fix things for him instead.

Cheers Heiko

On Fri, Sep 16, 2016 at 7:11 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:

This makes sense to me. Possibly we could warn in this case, so that the user knows that something was "off" but I don't think we should be failing here...

Regards, Jake

Heiko Voigt <hvoigt@hvoigt.net> writes:

When I said "a feature like this is needed", I didn't care about exact syntax. I am not sure how often people need the "fixup", what kind of causes there are that they need the "fixup", and what the distribution of vaious causes would be like. If the _ONLY_ kind of fixup necessary is "I meant to say 'git submodule add ./path path' but I said 'git add path' instead", then I think it makes sense to teach "submodule add" that the form "git submodule add <path>" is a short-pand for "git submodule add ./<path> <path>". I am not sure if we want to _ignore_ a gitlink that is already in the index unconditionally, i.e. if it is a good idea to let the second one override the first one

	git submodule add $URL sub &&
        git submodule add sub

in this sequence, though.

In other words, I agree with the general direction but I'd add another condition to the above three, i.e.

   * and there is no gitlink for that path in the index yet.