Re: problems serving non-bare repos with submodules over http

From: Stefan Beller <sbeller@google.com>
Date: Apr 20, 2016, 23:05 UTC
Message-ID: <CAGZ79kbyDuu8hUQXOUNmDLyO88GOi3kqSzsc0UfgzwAL9peZHg@mail.gmail.com>
In-Reply-To: <xmqqshygt1gs.fsf@gitster.mtv.corp.google.com>

On Wed, Apr 20, 2016 at 2:27 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 30 quoted lines

> Stefan Beller <sbeller@google.com> writes:
>
>>> I may be missing the subtleties, but if you are serving others from
>>> a non-bare repository with submodules, I do not think you would want
>>> to expose the in-tree version of the submodule in the first place.
>>
>> Well I would imagine that is the exact point.
>> If I was not trying to expose my state, I could ask you to
>> obtain your copy from $(git remote get-url origin) just as I did.
>
> That wasn't what I had in mind, but if the cloner cloned from your
> repository with a working tree, the cloner would discover submodules
> you use from your .gitmodules file, which would record the location
> you cloned them from, so something like that may come into the
> picture.  What I had in mind was more like this one you mentioned
> below:
>
>>     $GIT_DIR_SUPER_PROJECT/modules/$MODULE_NAME
>> ...
>> Right instead of cloning $WORKTREE/sub/.git you rather want
>> $GITDIR/module/sub
>
>> So currently the protocol doesn't allow to even specify the submodules
>> directories.
>
> Depends on what you exactly mean by "the protocol", but the
> networking protocol is about accessing a single repository.  It is
> up to you to decide where to go next after learning what you can
> learn from the result, typically by following what appears in
> the .gitmodules file.

Right. But the .gitmodules file is not sufficient.

If I clone from a bare hosting location, the .gitmodules file is the best we can do and the .gitmodules file works as intended. But in the non bare I case I think we would want to get the submodule from that location as well.

So in git clone (which calls out to git submodule update, which uses submodule--helper update_clone for cloning submodules) we'd want to see

    if remote was bare:
        do as usual (obtain URL from .gitmodules file)
    else
        take URL=$NON_BARE_REMOTE/module/submodule

Show 12 quoted lines

>
> The only special case is when .gitmodules file records the URL in a
> relative form, I would think.  Traditionally (i.e. when it was
> considered sane to clone only from bare repositories) I think people
> expected a layout like this:
>
>         top.git/
>         top.git/refs/{heads,tags,...}/...
>         top.git/objects/...
>         top.git/sub.git/
>         top.git/sub.git/refs/{heads,tags,...}/...
>         top.git/sub.git/objects/...

which could also be referred to as

top

without the .git suffix as someone thought this was an optimization?

Relative paths for submodules I have seen so far (on github, googlesource, eclipse) start with a ../ such that we'd have

Show 6 quoted lines

>         top.git/
>         top.git/refs/{heads,tags,...}/...
>         top.git/objects/...
>         sub.git/
>         sub.git/refs/{heads,tags,...}/...
>         sub.git/objects/...

and the .git suffix omission works as we only need to check for the last for characters and not somewhere in between. The sub.git is a standalone repository, and you cannot tell it is a submodule (except by its contents)

Show 7 quoted lines

>
> and refer to ./sub.git from .gitmodules recorded in top.git.  It
> still would be norm for common distribution sites (i.e. the original
> place Yaroslav likely has cloned things from) to be bare, and with
> or without $GIT_DIR/modules/, the relative path of submodule seen
> by its superproject would (have to) be different between a bare and
> a non-bare repository.

I think on a hosting site they could even coexist when having the layout as above.

         top.git/
         top.git/refs/{heads,tags,...}/...
         top.git/objects/...
         sub.git/
         sub.git/refs/{heads,tags,...}/...
         sub.git/objects/...

         # the following only exist in non bare:
         top.git/modules/sub.git/
         top.git/modules/sub.git/refs/{heads,tags,...}/...
         top.git/modules/sub.git/objects/...

The later files would be more reflective of what you *really* want if you clone from top.git.

Traditionally (when cloning was done from bare repos only), the .gitmodules file provides a very good way to indicate what the intent of the superproject is as the recorded sha1 in the tree doesn't tell you anything and tracking the remote for the submodule out of tree is cumbersome, so an in tree solution makes perfect sense.

If we have a non bare repo, it is safe to assume that the cloner actually meant to get the whole state from the remote (including submodules)?

I am trying to think of reasons why you would not want to get that copy from the remote. One (weak) reason is that the submodule may be a well known library, which you can obtain faster from a well known git hosting site rather than $remote.

Show 14 quoted lines

>
> I'd imagine that people could agree on a common layout like this
> even for a forest of bare repositories:
>
>         top.git/
>         top.git/refs/{heads,tags,...}/...
>         top.git/objects/...
>         top.git/modules/sub.git/
>         top.git/modules/sub.git/refs/{heads,tags,...}/...
>         top.git/modules/sub.git/objects/...
>
> which would probably make the "relative" relationship between the
> supermodule and its submodules the same between bare and non-bare
> repositories, but I didn't think it too deeply.

Forrests as of now are handled as a flat level thing, e.g.

    git clone git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.aggregator.git

will produce a superproject with 25 submodules, all of them are either at ../ or at ../../ such that it would follow

         projects/top.git/
         projects/top.git/refs/{heads,tags,...}/...
         projects/top.git/objects/...
         projects/sub.git/
         projects/sub.git/refs/{heads,tags,...}/...
         projects/sub.git/objects/...
         libs/sub2.git
         libs/sub2.git/refs/{heads,tags,...}/...
         libs/sub2.git/objects/...

Looking at our internal code search there is no .gitmodules file whose url starts with "./", they all start with ../ or are absolute.