Future structure of submodules and .git/, .git/modules/* organization

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Apr 14, 2021, 10:32 UTC
Message-ID: <87v98p17im.fsf@evledraar.gmail.com>
In-Reply-To: <20210408233936.533342-1-emilyshaffer@google.com>

On Fri, Apr 09 2021, Emily Shaffer wrote:

Show 12 quoted lines

> I'm hoping to work on some other submodule-centric stuff over the coming
> months, and it might end up being very useful to be able to tell "am I a
> submodule?" and "how do I talk to my superproject?" more generally - so
> I'm really open to figuring out a better way than this, if folks have
> ideas.
>
> Patch 1 is a small refactor that we can take or leave - I found
> "SCOPE_SUBMODULE" to be pretty ambiguous, especially since it seems to
> refer to configs from .gitmodules. Even though I decided that
> "superproject" was a better name than "submodule" I still wasn't super
> happy with the ambiguity. But we can drop it if folks don't want to
> rename.

This is less on your patch, and more on the larger work you're suggesting, but the two are kind of related. Skip to the paragraph starting with "But why" below for the relevance :)

I very much wish that we could eventually make the use of submodules totally transparent, i.e. (taking the example of git.git):

 * You clone, and we just get objects from
   https://github.com/cr-marcstevens/sha1collisiondetection.git too

 * The fact that we have:

   160000 commit 855827c583bc30645ba427885caa40c5b81764d2  sha1collisiondetection

   Would become totally invisible to most users unless they run some
   gutsy ls-tree/files comand.

   We used to have a full git dir at sha1collisiondetection/.git and all
   the UX issues that entailed (e.g. switching to an old commit without
   the submodule).

   Now it's a stub and the actual repo is at
   .git/modules/sha1collisiondetection/, so we're kind of partially
   there.

 * I would think that the next (but big) logical step would be to use
   some combination of delta islands, upcoming sparse indexes etc. to
   actually share the object stores of the parent and submodule.

   Things like "git fsck" which now just punt on COMMIT would need to
   become smarter, but e.g. we could repack (or not, with islands)
   between parent and submodule.

I would think that this end goal makes more sense than the current status quo of teaching every command that needs to e.g. grep the tree to have a "--recurse-submodules". The distinction would be invisible to the likes of "git-grep".

It would mean more complexity in e.g. "git commit", but we can imagine if you wanted a cross-submodule commit it could do those commits recursively, update parent COMMIT entries etc. (and even, optionally, push out the submodule changes). That particular thing being so ad-hoc is a *very* frequent pain point in submodule use.

But why am I talking about this here when all you're suggesting is another config level?

Well, I think (but have not carefully thought about) that this CONFIG_SCOPE_GITMODULES is probably a narrow net improvement now. If you set most options in your .git/config to you that's the same logical project, why shouldn't you get your diff setting or whatever because you cd'd to a submodule "in the same project" (from the view of the user).

But I think that for a wider "improve submodules" effort it's worth someone (and right now, that sounds like it's you) thinking about where we're going with the feature. Maybe with some technical doc identifying the most common pain points, what we propose (or could envision) doing about them.

So e.g. in this case, having per-submodule config could be a step forward, but it could also be one more step of walking in a circle.

I.e. don't think any user asked for or wanted to stitch together multiple .git directories into one linked pseudo-checkout, that's ultimately something we're exposing as an implementation detail. If we no longer expose that implementation detail, would we be stuck supporting what's ultimately a workaround feature?

None of that means we shouldn't have that one step forward that solves real problems today.

But I think we should think about the end goal(s) sooner than later. E.g. in your case, do you *really* want another config level, or is it just the easiest way to get what you actually want, which is for a "git config" in the submodule dir to perhaps consider its .git/config and .git/modules/sha1collisiondetection/config as the same file for the purposes of config parsing? Sans things like the remote URLs etc.