Make "git checkout" automatically update submodules?

Git submodules seem to be a great fit for one of our repos. The biggest pain point is that it's too easy to forget to update submodules.

1. I often forget since most repos don't need it.
2. Infrequent users of our repo almost never know to update submodules and
end up coming to us with strange build errors.
3. Existing scripts that work with Git repos are usually not built to handle
submodules.

In the common case of the submodule content having no local edits, it would be nice if "git checkout" automatically updated submodules [1]. If there are local edits, it could error out (maybe override with "--ignore-modified-submodules" or something).

I'm not a Git expert, though. Is there a reason something like this isn't already implemented? Maybe there's an existing write-up or mailing list thread I can read to get some background information?

Thanks!

[1] Our post-checkout procedure is:

    git submodule sync
    git submodule update --init
    git submodule foreach --recursive \
      'git submodule sync ; git submodule update --init'

(Not sure if this is correct. Different articles/blogs suggest a slightly different set of commands.)

Kannan Goundan <kannan@cakoose.com> writes:

I think it is mostly because the area has a lot of corner cases and different workflows. For example ...

... this will clearly not work well for everybody, as you do not want to instantiate _all_ the submodules _unconditionally_. Well, "you" (Kannan) and your project may want to, but not necessarily other large projects (e.g. imagine Android using submodules).

So you can view the current status being "nothing is instantiated by default", which _is_ far from satisfactory than the ideal case where perhaps the project can specify in .gitmodules which submodules are to be instantiated by default, add labels to modules in .gitmodules so that "git clone" of a top-level project with submodules can be told "git clone --init-submodules=<label>" to instantiate the modules that match the given label after the top-level is cloned, etc. etc.

But such a way to allow these more complicated situations to be handled nicely has not been designed by anybody, so for now "nothing instantiated by default" stands as the safest default.

Thanks for the explanation. I didn't realize some projects don't want to initialize all their submodules, but the explicit opt-in idea you described sounds nice.

I've seen cases where people will financially "sponsor" feature development in open source projects. Is there any precedent for this in the Git project? Is it ok to use this mailing list to ask about such things?

Kannan Goundan <kannan@cakoose.com> writes:

We are unfortunately not set up to handle money well. For a background explanation, please go read [*1*], which I wrote my take on "money" some time ago. Note that it is an explanation and not a justification. It explains why we are not set up to handle money well and what the issues around money that are troublesome for the project are. It does not mean to say that it is a good thing that it is hard to buy feature with money from our project [*2*].

I do not see (and back then in the discussion I do not think anybody saw) how we can make "We, a sponsoring company, pay this money to the project to fund effort Y." work well. But that of course does not mean it is impossible to make it work--somebody with a fresh perspective may come up a way to do so, and that would be a very welcome development.

[Footnote/Reference]

*1* http://thread.gmane.org/gmane.comp.version-control.git/264993/focus=265215

*2* Just like my message that you are responding to was an
    explanation of the reason why we do not recurse and and
    initialize all submodules by default.

Junio C Hamano <gitster <at> pobox.com> writes:

I think the way I described it ("sponsoring a feature") doesn't really reflect how I was imagining it. In my head, it looked like this:

1. Figure out whether the Git community and maintainers seem ok with the
overall feature idea.  If not, give up.
2. Come up with a plan for the UI/UX; see if the Git community and
maintainers seem ok with it.  If not, iterate or give up.
3. Implement it, then go through the regular process of getting it merged
upstream.  If it doesn't go well, might have to iterate or give up.

I could try doing that myself, but someone familiar with the Git codebase/community/history would be better at it (and probably be easier for you guys to work with :-)

I guess I'm just wondering if there are people who meet those qualifications and are interested in going through those steps for pay. Or maybe there's a company that does this, like the old Cygnus Solutions?

In particular, I don't expect anything to change about the project's development process.

(This part is not relevant to the Git project, but I understand that it's hard for anyone to guarantee a feature will make it into an open source project. I imagine these kinds of contracts are set up so that you're primarily paying for the effort, not the outcome. If it ends up not working out, you don't get your money back.)

On Fri, Oct 23, 2015 at 5:46 AM, Kannan Goundan <kannan@cakoose.com> wrote:

Well I am starting to do that for Booking.com. Not sure if it will also be possible for me to work for you as I also work on IPFS (http://ipfs.io) for the company that develops it, but we can discuss it privately.

There was David Kastrup (dak at gnu.org) who previously said he could be interested in such jobs. We wrote a very short article about it in the first edition of Git Rev News last March:

http://git.github.io/rev_news/2015/03/25/edition-1/

We also wrote a very short article "Job Offer" article about Booking.com looking for Git developers in Git Rev News in the third edition last May:

http://git.github.io/rev_news/2015/05/13/edition-3/

so if you want we can write a similar "Job Offer" article in the next Git Rev News edition.

You can even propose such an article yourself by editing the draft of the next edition here:

https://github.com/git/git.github.io/blob/master/rev_news/drafts/edition-9.md

and then creating a pull request.

Kannan Goundan <kannan@cakoose.com> writes:

In other words, the sponsoring entity is paying for effort and not for result--money does not buy inclusion.

That may be workable from the project's point of view; I however wonder if that is workable from the sponsor's point of view. Things that may be problematic inside the sponsoring entity (i.e. between those with money and those who interact with the hired person) include:

 - Does the hired person really meet the right qualifications?

 - Did the hired person make a good faith effort?

 - Was it the right time to start the topic, or was the codebase too
   much in flux at the moment to accept work in that area?

But these are problems between the sponsor and the hired person and do not concern us ;-)

On Thu, Oct 15, 2015 at 3:50 PM, Kannan Goundan <kannan@cakoose.com> wrote:

Checkout [1]. There are lots of good patches, but hard to find. (Including, but not limited to a recursive git checkout enhancement!)

That said I've recently started working on submodules, too. I am trying to push my work upstream as fast as possible as that works best for us.

[1] https://github.com/jlehmann/git-submod-enhancements/wiki

Stefan Beller <sbeller@google.com> writes:

Yes, Jens is not just one of the people who have been working on harder, and thinking longer about, submodules than anybody else, but also has demonstrated that he has good taste and balanced view on the design of the subsystem over time, whose technical judgment we can trust.

Not all the changes listed on the page may necessarily be good as-is (e.g. some may help only a subset of users while hurting others, like the "recursively check-out everything unconditionally" that trigerred this thread), but the page has a good collection to remind anybody, who designs a coherent whole, of issues that need to be taken into account.

Thanks for a pointer to an excellent starting page.

Stefan Beller <sbeller <at> google.com> writes:

Oh wow, Christmas came early!  I'll give this code a try.