threads / discuss / 57042

[Question] Switching the URI from SSH to HTTPS for submodules

Subject: [Question] Switching the URI from SSH to HTTPS for submodules

## tl;dr

5 messages between Dec 6, 2021 and Dec 8, 2021.

replies: 4people: 3as markdown or json

rsbecker@nexbridge.com· Dec 6, 2021, 23:10 UTC · lore

This is a follow-up to the conversation at #git-devel Standup today.

We encountered the following situation. Our normal clone practice is to use SSH URIs. So, the clone is straightforward with the main (8 year old) repo of the style:

git@bitbucket.org:project/repo.git

with submodules referenced as:

git@bitbucket.org:project/module.git

When in SSH mode, clones are simple with --recurse-submodules doing what we want. However, we had to clone on a system where SSH was locked down and we could only use HTTPS. The form of the URIs changed rather radically:

https://user@bitbucket.org/project/repo.git

which caused the submodule clone not to work. Fortunately, the number of submodules was low, so a manual edit of .gitmodules (with update-index --assume-unchanged) and .git/modules and .git/config was not onerous. However, we should be able to redirect the configuration without having to do that. Our directory references are all relative to the main repo, so those did not have to change. We still use SSH for most work with the repo but needed to get source onto a test system for debugging.

I would like to put together a best practice section in Documentation/gitsubmodules.txt on the subject, but really don't think I did this *temporary* change the most effective way. I'm looking for better practices than I used - I'm sure there is at least one. There may be code changes involved, particularly since .gitmodules is tracked and contains the full URI (maybe it should not). One option we also explored was explicitly specifying the URI for each submodule using set-url, which works but it seems like there should be something in clone that redirects the part of the URI relative to the main repo - although that would only work in the special case where the submodules have the same host and project as the main repo.

-Randall

Robert Coup· Dec 6, 2021, 23:53 UTC · re: rsbecker@nexbridge.com · lore

Re: [Question] Switching the URI from SSH to HTTPS for submodules

Hi Randall,

On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:

Show 10 quoted lines

>
> git@bitbucket.org:project/module.git
>
> When in SSH mode, clones are simple with --recurse-submodules doing what we
> want. However, we had to clone on a system where SSH was locked down and we
> could only use HTTPS. The form of the URIs changed rather radically:
>
> https://user@bitbucket.org/project/repo.git
>
> I'm looking for better practices than I used - I'm sure there is at least one.

AFAIK the existing `url.<base>.insteadOf` config option[1] deals with this...

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf

    $ git config --global url."https://user@bitbucket.org/".insteadOf
"git@bitbucket.org:"

If you don't want to set it globally (not a throwaway CI environment?) then you can do it as a one-off:

    $ git -c url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
clone --recurse-submodules git@bitbucket.org:repo/project.git

But it isn't persisted into your repo config then, so subsequent fetches won't work. You'd need to persist it using something like:

    $ git config url."https://user@bitbucket.org/".insteadOf
"git@bitbucket.org:"
    $ git submodule foreach --recursive 'git config
url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '

Maybe there's an opportunity to make that part easier?

Rob :)

rsbecker@nexbridge.com· Dec 7, 2021, 14:50 UTC · re: Robert Coup · lore

RE: [Question] Switching the URI from SSH to HTTPS for submodules

On December 6, 2021 6:54 PM, Robert Coup wrote:

Show 36 quoted lines

> On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
> >
> > git@bitbucket.org:project/module.git
> >
> > When in SSH mode, clones are simple with --recurse-submodules doing
> > what we want. However, we had to clone on a system where SSH was
> > locked down and we could only use HTTPS. The form of the URIs changed
> rather radically:
> >
> > https://user@bitbucket.org/project/repo.git
> >
> > I'm looking for better practices than I used - I'm sure there is at least one.
> 
> AFAIK the existing `url.<base>.insteadOf` config option[1] deals with this...
> 
> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-
> urlltbasegtinsteadOf
> 
>     $ git config --global url."https://user@bitbucket.org/".insteadOf
> "git@bitbucket.org:"
> 
> If you don't want to set it globally (not a throwaway CI environment?) then
> you can do it as a one-off:
> 
>     $ git -c url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
> clone --recurse-submodules git@bitbucket.org:repo/project.git
> 
> But it isn't persisted into your repo config then, so subsequent fetches won't
> work. You'd need to persist it using something like:
> 
>     $ git config url."https://user@bitbucket.org/".insteadOf
> "git@bitbucket.org:"
>     $ git submodule foreach --recursive 'git config
> url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '
> 
> Maybe there's an opportunity to make that part easier?

Well, I gave this a shot. The mapping did not appear to work - I tried a few combinations. I was left with the original URIs. Note that I also had to do

$ git submodule init

which reported the wrong URIs

$ git submodule update

which obviously failed since they were still using SSH.

I think there may be something not working correctly with the insteadOf operator. This is easily reproduceable.

-Randall

Fabian Stelzer· Dec 8, 2021, 12:51 UTC · re: rsbecker@nexbridge.com · lore

Re: [Question] Switching the URI from SSH to HTTPS for submodules

On 07.12.2021 09:50, rsbecker@nexbridge.com wrote:

Show 50 quoted lines

>On December 6, 2021 6:54 PM, Robert Coup wrote:
>> On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
>> >
>> > git@bitbucket.org:project/module.git
>> >
>> > When in SSH mode, clones are simple with --recurse-submodules doing
>> > what we want. However, we had to clone on a system where SSH was
>> > locked down and we could only use HTTPS. The form of the URIs changed
>> rather radically:
>> >
>> > https://user@bitbucket.org/project/repo.git
>> >
>> > I'm looking for better practices than I used - I'm sure there is at least one.
>>
>> AFAIK the existing `url.<base>.insteadOf` config option[1] deals with this...
>>
>> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-
>> urlltbasegtinsteadOf
>>
>>     $ git config --global url."https://user@bitbucket.org/".insteadOf
>> "git@bitbucket.org:"
>>
>> If you don't want to set it globally (not a throwaway CI environment?) then
>> you can do it as a one-off:
>>
>>     $ git -c url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
>> clone --recurse-submodules git@bitbucket.org:repo/project.git
>>
>> But it isn't persisted into your repo config then, so subsequent fetches won't
>> work. You'd need to persist it using something like:
>>
>>     $ git config url."https://user@bitbucket.org/".insteadOf
>> "git@bitbucket.org:"
>>     $ git submodule foreach --recursive 'git config
>> url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '
>>
>> Maybe there's an opportunity to make that part easier?
>
>Well, I gave this a shot. The mapping did not appear to work - I tried a few combinations. I was left with the original URIs. Note that I also had to do
>
>$ git submodule init
>
>which reported the wrong URIs
>
>$ git submodule update
>
>which obviously failed since they were still using SSH.
>
>I think there may be something not working correctly with the insteadOf operator. This is easily reproduceable.
>

I ran into this as well some time ago. For submodules this only works if you put it into your global config since the parent projects config is not relevant to the submodule.

There's some discussion in the archive about this as well: https://lore.kernel.org/git/404d109f-e5a7-85a3-e64c-ab1b21c3045d@durchholz.org/

rsbecker@nexbridge.com· Dec 8, 2021, 13:44 UTC · re: Fabian Stelzer · lore

RE: [Question] Switching the URI from SSH to HTTPS for submodules

On December 8, 2021 7:52 AM, Fabian Stelzer wrote:

Show 66 quoted lines

> On 07.12.2021 09:50, rsbecker@nexbridge.com wrote:
> >On December 6, 2021 6:54 PM, Robert Coup wrote:
> >> On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
> >> >
> >> > git@bitbucket.org:project/module.git
> >> >
> >> > When in SSH mode, clones are simple with --recurse-submodules doing
> >> > what we want. However, we had to clone on a system where SSH was
> >> > locked down and we could only use HTTPS. The form of the URIs
> >> > changed
> >> rather radically:
> >> >
> >> > https://user@bitbucket.org/project/repo.git
> >> >
> >> > I'm looking for better practices than I used - I'm sure there is at least
> one.
> >>
> >> AFAIK the existing `url.<base>.insteadOf` config option[1] deals with
> this...
> >>
> >> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-
> >> urlltbasegtinsteadOf
> >>
> >>     $ git config --global url."https://user@bitbucket.org/".insteadOf
> >> "git@bitbucket.org:"
> >>
> >> If you don't want to set it globally (not a throwaway CI
> >> environment?) then you can do it as a one-off:
> >>
> >>     $ git -c
> url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
> >> clone --recurse-submodules git@bitbucket.org:repo/project.git
> >>
> >> But it isn't persisted into your repo config then, so subsequent
> >> fetches won't work. You'd need to persist it using something like:
> >>
> >>     $ git config url."https://user@bitbucket.org/".insteadOf
> >> "git@bitbucket.org:"
> >>     $ git submodule foreach --recursive 'git config
> >> url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '
> >>
> >> Maybe there's an opportunity to make that part easier?
> >
> >Well, I gave this a shot. The mapping did not appear to work - I tried
> >a few combinations. I was left with the original URIs. Note that I also
> >had to do
> >
> >$ git submodule init
> >
> >which reported the wrong URIs
> >
> >$ git submodule update
> >
> >which obviously failed since they were still using SSH.
> >
> >I think there may be something not working correctly with the insteadOf
> operator. This is easily reproduceable.
> >
> 
> I ran into this as well some time ago. For submodules this only works if you
> put it into your global config since the parent projects config is not relevant to
> the submodule.
> 
> There's some discussion in the archive about this as well:
> https://lore.kernel.org/git/404d109f-e5a7-85a3-e64c-
> ab1b21c3045d@durchholz.org/

The minimal procedure that I got to work is:

$ git config --global url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" $ git clone --recurse-submodules https://user@bitbucket.org/project/repo.git

This leaves .gitmodules untouched and contains the original URIs, so the status is clean and there is no real risk of someone pushing an unwanted URI change back to the upstream repo. I don't see a major downside to leaving the redirect in global as this would be useful for subsequent clones. Any scripting that might depend on .gitmodules being correct would break - not in my case though - to that should probably be noted.

I think I have enough to document this consideration in Documentation/gitsubmodules.txt. Thanks everyone. -Randall