threads / discuss / 42248

wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

Subject: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

## tl;dr

8 messages between May 6, 2016 and May 11, 2016.

replies: 7people: 3as markdown or json

Yaroslav Halchenko· May 6, 2016, 13:18 UTC · lore
Dear Git Folks,

Originally this issue was mentioned in previous thread [1], and I have decided to bring it into a separate thread. ATM there is a dichotomy in git behavior between cloning non-bare repos: if I clone over ssh or just locally by providing url without trailing /.git, git senses for /.git and works just fine with ssh or local repositories, but fails for "dummy" http ones, the demo script is here [2] which produces output listed below. From which you can see that cloning using http URL to the repository without /.git fails (git version 2.8.1, Debian). As it was noted in [1], concern could have been to not traverse website since could lead to dangerous places. But .git is under originating url directory, as well as info/ or HEAD or any other object accessed by git, so IMHO this concern is not a concern.

So do you think that cloning from http urls could be adjusted so git senses for presence of .git/ subdirectory if originating url is missing necessary /info/refs?.... ? That would make behavior uniform and help us and users in many use-cases IMHO (in particular with relative paths to submodules as in [1])

Thank you in advance, and please maintain CC

$> /tmp/democlone Initiating repo cloning via ssh: localhost:/tmp/repo1 Cloning into 'repo1-ssh'... remote: Counting objects: 3, done. remote: Total 3 (delta 0), reused 0 (delta 0) Receiving objects: 100% (3/3), done. Checking connectivity... done. SUCCESS cloning locally: /tmp/repo1 Cloning into 'repo1-local'... done. SUCCESS starting local http server Serving HTTP on 0.0.0.0 port 8080 ... cloning via http: http://localhost:8080/repo1 Cloning into 'repo1-http-failed'... 127.0.0.1 - - [06/May/2016 09:13:41] code 404, message File not found 127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/info/refs?service=git-upload-pack HTTP/1.1" 404 - fatal: repository 'http://localhost:8080/repo1/' not found FAILED doing with /.git: http://localhost:8080/repo1/.git Cloning into 'repo1-http'... 127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 - 127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/HEAD HTTP/1.1" 200 - 127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/objects/36/31fa81eb6422349035c915a5a11b177688f491 HTTP/1.1" 200 - 127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/objects/12/ddeb87cc045cd67063e95125fbeb014bd1d9b1 HTTP/1.1" 200 - 127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 HTTP/1.1" 200 - Checking connectivity... done. SUCCESS

[1] problems serving non-bare repos with submodules over http http://thread.gmane.org/gmane.comp.version-control.git/292032 [2] http://www.onerussian.com/tmp/democlone

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Yaroslav Halchenko· May 10, 2016, 19:06 UTC · re: Yaroslav Halchenko · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

On Fri, 06 May 2016, Yaroslav Halchenko wrote:
> Dear Git Folks,
Show 12 quoted lines
> Originally this issue was mentioned in previous thread [1], and I have decided
> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
> between cloning non-bare repos:  if I clone over ssh or just locally by
> providing url without trailing /.git, git senses for /.git and works just fine
> with ssh or local repositories, but fails for "dummy" http ones, the demo
> script is here [2] which produces output listed below.  From which you can see
> that  cloning using http URL to the repository without /.git fails (git version
> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
> traverse website since could lead to dangerous places.  But .git is under
> originating url directory, as well as info/ or HEAD or any other object
> accessed by git, so IMHO this concern is not a concern.
> ...

If there is a better venue (bug tracker?) to spark the interest and discussion, please let me know ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Junio C Hamano· May 10, 2016, 20:11 UTC · re: Yaroslav Halchenko · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

Yaroslav Halchenko <yoh@onerussian.com> writes:
Show 15 quoted lines
> On Fri, 06 May 2016, Yaroslav Halchenko wrote:
>
>> Dear Git Folks,
>
>> Originally this issue was mentioned in previous thread [1], and I have decided
>> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
>> between cloning non-bare repos:  if I clone over ssh or just locally by
>> providing url without trailing /.git, git senses for /.git and works just fine
>> with ssh or local repositories, but fails for "dummy" http ones, the demo
>> script is here [2] which produces output listed below.  From which you can see
>> that  cloning using http URL to the repository without /.git fails (git version
>> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
>> traverse website since could lead to dangerous places.  But .git is under
>> originating url directory, as well as info/ or HEAD or any other object
>> accessed by git, so IMHO this concern is not a concern.

I am afraid that the reason why you saw no response is primarily because nobody is interested in extending dumb commit-walker HTTP transport after the world has largely moved on and abandoned it.

The necessary update to the client might as simple as using $GIVEN_URL/.git/ and attempting the request again after seeing the probe for $GIVEN_URL/info/refs fails.

Jacob Keller· May 10, 2016, 20:33 UTC · re: Junio C Hamano · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

On Tue, May 10, 2016 at 1:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 26 quoted lines
> Yaroslav Halchenko <yoh@onerussian.com> writes:
>
>> On Fri, 06 May 2016, Yaroslav Halchenko wrote:
>>
>>> Dear Git Folks,
>>
>>> Originally this issue was mentioned in previous thread [1], and I have decided
>>> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
>>> between cloning non-bare repos:  if I clone over ssh or just locally by
>>> providing url without trailing /.git, git senses for /.git and works just fine
>>> with ssh or local repositories, but fails for "dummy" http ones, the demo
>>> script is here [2] which produces output listed below.  From which you can see
>>> that  cloning using http URL to the repository without /.git fails (git version
>>> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
>>> traverse website since could lead to dangerous places.  But .git is under
>>> originating url directory, as well as info/ or HEAD or any other object
>>> accessed by git, so IMHO this concern is not a concern.
>
> I am afraid that the reason why you saw no response is primarily
> because nobody is interested in extending dumb commit-walker HTTP
> transport after the world has largely moved on and abandoned it.
>
> The necessary update to the client might as simple as using
> $GIVEN_URL/.git/ and attempting the request again after seeing the
> probe for $GIVEN_URL/info/refs fails.
> --

I know at least Jenkin's Git plugin has a workaround to solve this issue that is quite similar.

Thanks, Jake

Yaroslav Halchenko· May 10, 2016, 22:10 UTC · re: Junio C Hamano · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

On Tue, 10 May 2016, Jacob Keller wrote:
Show 5 quoted lines
> > The necessary update to the client might as simple as using
> > $GIVEN_URL/.git/ and attempting the request again after seeing the
> > probe for $GIVEN_URL/info/refs fails.
> I know at least Jenkin's Git plugin has a workaround to solve this
> issue that is quite similar.
On Tue, 10 May 2016, Junio C Hamano wrote:
> >> traverse website since could lead to dangerous places.  But .git is under
> >> originating url directory, as well as info/ or HEAD or any other object
> >> accessed by git, so IMHO this concern is not a concern.
> I am afraid that the reason why you saw no response is primarily
> because nobody is interested in extending dumb commit-walker HTTP
> transport after the world has largely moved on and abandoned it.
> The necessary update to the client might as simple as using
> $GIVEN_URL/.git/ and attempting the request again after seeing the
> probe for $GIVEN_URL/info/refs fails.

Sure -- workarounds are possible, and we are at the state that many dependent projects seems are doing that already (as above noted Jenkin's Git plugin, smth along the lines probably done by github for https as well). In my case I even have managed to erect a lovely apache rewrite rule which seems to work, so I can just 'git clone --recursive' a collection of 30 submodules without a hiccup (we are also interested in .git/annex part here ;) ). Citing here if it comes handy for anyone

    # To overcome http://thread.gmane.org/gmane.comp.version-control.git/293777
    # we need to rewrite urls so that there is no need for explicit .git/
    RewriteEngine On
    RewriteCond "!.*/\.git/.*"
    RewriteRule "(.*?/)((?<!\.git/)(HEAD|config|annex/objects/.*|objects/(info/[^/]+|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}.(pack|idx))|git-(upload|receive)-pack)|info/refs)$" "$1.git/$2" [PT]

and sure thing in our project now we can add such sensing for .git/ and/or reattempts to request again. But that is the point -- it seems to be a relatively common hiccup which could be fixed in the "root". So I (and possibly others) would just very much appreciate if it was (thus it was a [wishlist] ;) )

Cheers,
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Junio C Hamano· May 10, 2016, 22:13 UTC · re: Yaroslav Halchenko · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

Yaroslav Halchenko <yoh@onerussian.com> writes:
Show 5 quoted lines
>> The necessary update to the client might be as simple as using
>> $GIVEN_URL/.git/ and attempting the request again after seeing the
>> probe for $GIVEN_URL/info/refs fails.
>
> Sure -- workarounds are possible,...

Just so that there is no misunderstanding, the above was not a workaround but is an outline of an implementation of updated http client shipped with Git.

Yaroslav Halchenko· May 11, 2016, 00:49 UTC · re: Junio C Hamano · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

On Tue, 10 May 2016, Junio C Hamano wrote:
> >> The necessary update to the client might be as simple as using
> >> $GIVEN_URL/.git/ and attempting the request again after seeing the
> >> probe for $GIVEN_URL/info/refs fails.
> > Sure -- workarounds are possible,...
> Just so that there is no misunderstanding, the above was not a
> workaround but is an outline of an implementation of updated http
> client shipped with Git.

ah, sorry, I have indeed might have misread it. So we are on the same page -- that is I saw also as the tentative implementation ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
Junio C Hamano· May 11, 2016, 21:27 UTC · re: Yaroslav Halchenko · lore

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

Yaroslav Halchenko <yoh@onerussian.com> writes:
Show 13 quoted lines
> On Tue, 10 May 2016, Junio C Hamano wrote:
>> >> The necessary update to the client might be as simple as using
>> >> $GIVEN_URL/.git/ and attempting the request again after seeing the
>> >> probe for $GIVEN_URL/info/refs fails.
>
>> > Sure -- workarounds are possible,...
>
>> Just so that there is no misunderstanding, the above was not a
>> workaround but is an outline of an implementation of updated http
>> client shipped with Git.
>
> ah, sorry, I have indeed might have misread it.  So we are on the same
> page -- that is I saw also as the tentative implementation ;)

Good. Now somebody who is interested in seeing that happen (when I said "shipped with Git" above, I meant "shipped with possible future Git") needs to find (or be) somebody to code that change ;-)

Thanks.

← back to recent threads