threads / rfc / 2206

[RFC] GIT paths

Subject: [RFC] GIT paths

## tl;dr

5 messages between Oct 24, 2005 and Oct 26, 2005.

replies: 4people: 3as markdown or json

Junio C Hamano· Oct 24, 2005, 08:50 UTC · lore

Our networking commands can take either URL or non URL to specify remote repository. This note first attempts to clarify what <path> means in the current implementation, and then discusses two possible enhancements.

A non URL always refers to the pack protocol going over an SSH connection:

	<host> ':' <path>
	. path that starts with a slash '/' is absolute path on
	  the remote site.
	. path that does not start with a slash '/' is relative
	  to the home directory of the incoming user.

However, note that the administrator could futz with the login shell of the user to give restricted access (chroot to change the former). The latter can be made different from the home directory, if git-shell is changed to chdir() to somewhere else first. I am not suggesting this as a best practice -- just mentioning the possibility for completeness.

A URL form is:
	<proto> ':' <host> ( ':' <port> ) '/' <rest-of-path> 

and <proto> is either 'git', or 'ssh' (also spelled as 'ssh+git' or 'git+ssh'). In addition, you can use 'http' or 'rsync', but these transports are not discussed further here. They already have established semantics for <path> = '/' + <rest-of-path>.

For connections over plain TCP talking with git-daemon, or over SSH in this form, path is always relative to the root directory on the remote site, because '/' that terminate either <host> or <port> starts the <path> = '/' + <rest-of-path>.

There are two things I would like to discuss here.
 - It might make sense to have SERVER_ROOT (similar to
   DOCUMENT_ROOT in Apache) for git-daemon, so <path> does not
   have to be relative to the true filesystem root.  Note that
   this is not a security measure, but meant for administration
   convenience [*1*].
 - Over a git-daemon connection, supporting ~user expansion
   makes sense.  E.g git://host.xz/~junio/ refers to my home
   directory on that machine.  It would make it impossible to
   have a directory literally named '~junio' directly underneath
   the root directory, but that is a good limitation anyway.

The above enhancements, especially SERVER_ROOT, however make paths inconsistent between non URL form and URL form. This probably is OK -- people are used to using different paths when uploading to HTTP server and testing a download from it. That leaves one issue. Do we want to support ~user expansion, and if so how, on non git-daemon connections?

I would propose that
	git fetch host.xz:~junio/repo
	git fetch ssh://host.xz/~junio/repo

mean the same thing (i.e. both understand ~user expansion). Also these are equivalent (i.e. no ~user expansion; both mean absolute filesystem path without SERVER_ROOT prefixing):

	git fetch host.xz:/frotz/repo
	git fetch ssh://host.xz/frotz/repo

While these two might not mean the same thing (the former is prefixed with SERVER_ROOT, but not the latter):

	git fetch git://host.xz/frotz/repo
	git fetch ssh://host.xz/frotz/repo
There are small technical issues.
 - connect.c should not be affected at all, since it does not
   know how the remote site arranges SERVER_ROOT (if we support
   it) or user home directories.
 - ssh://host.xz/path and host.xz:path connections spawn
   upload-pack or receive-pack directly, without being mediated
   by git-daemon.  This means that ~user expansion, if we want
   to support it, needs to be done by these programs themselves.
 - git-daemon needs to validate the incoming requested path and
   in order to avoid aliasing issues, we should resolve ~user
   expansion and SERVER_ROOT prefixing first, then validate the
   resulting path against white/black list, before calling
   upload-pack or receive-pack.  However, after git-daemon
   decides to run these programs, they could find out some
   problems with the specified repository and may need to report
   them.  Arguably, this reporting should not reveal the real
   path used to address the repository [*2*].

Although we _could_ forget about the "error reporting exposing real path" issue for now, I think we should at least have a plan to make things consistent and well defined. Here is a strawman:

 - Have a common library code that takes user supplied path and
   does SERVER_ROOT prefixing and ~user expansion.
 - Have git-daemon use it to canonicalize the requested path
   before validating.  Make it invoke the programs with the path
   received from the other end (before SERVER_ROOT prefixing, or
   ~user expansion).
 - Give --server-root=/path/to/root flag to programs that can be
   called by git-daemon, and have git-daemon run them with this
   flag.  Have them use the same library to canonicalize the
   requested path to the real path.  When these programs are run
   via direct SSH connection (i.e. ssh://host/path and
   host:path), this flag is not given so they see filesystem
   path as-is, but make the ~user expansion still available.
[Footnote]

*1* You do not want to advertise your repo is at /mnt/disk1/repo and find out that you need to move the disks around next day. Of course you could plan ahead and have a symlink hanging below the root directory (e.g. '/pub -> /mnt/disk1/git'), but it is so much more convenient if you can just tell git-daemon that the root level used to be /mnt/disk1/git but it is now somewhere else.

*2* This is theoretical right now, since packed transfer protocols cannot report errors back, but Andreas' patch addresses this issue by dying carefully in srvside_chdir(). It falls into security-by-obscurity category, so we may choose not to worry about it, though.

Andreas Ericsson· Oct 25, 2005, 12:31 UTC · re: Junio C Hamano · lore

Re: [RFC] GIT paths

Junio C Hamano wrote:
Show 24 quoted lines
> Our networking commands can take either URL or non URL to
> specify remote repository.  This note first attempts to clarify
> what <path> means in the current implementation, and then
> discusses two possible enhancements.
> 
> For connections over plain TCP talking with git-daemon, or over
> SSH in this form, path is always relative to the root directory
> on the remote site, because '/' that terminate either <host> or
> <port> starts the <path> = '/' + <rest-of-path>.
> 
> There are two things I would like to discuss here.
> 
>  - It might make sense to have SERVER_ROOT (similar to
>    DOCUMENT_ROOT in Apache) for git-daemon, so <path> does not
>    have to be relative to the true filesystem root.  Note that
>    this is not a security measure, but meant for administration
>    convenience [*1*].
> 
>  - Over a git-daemon connection, supporting ~user expansion
>    makes sense.  E.g git://host.xz/~junio/ refers to my home
>    directory on that machine.  It would make it impossible to
>    have a directory literally named '~junio' directly underneath
>    the root directory, but that is a good limitation anyway.
> 

I like this idea, although I'd extend it with a Userdir-like config option in git-daemon (like ~/public_html for apache). This makes it a bit easier to see what's published and what isn't.

About the literally named /~junio directory, it would be possible with this syntax;

	git fetch host.xz:/~junio

The userdir is (with my previous patch) only expanded if the path starts with a tilde.

Show 31 quoted lines
> The above enhancements, especially SERVER_ROOT, however make
> paths inconsistent between non URL form and URL form.  This
> probably is OK -- people are used to using different paths when
> uploading to HTTP server and testing a download from it.  That
> leaves one issue.  Do we want to support ~user expansion, and if
> so how, on non git-daemon connections?
> 
> I would propose that
> 
> 	git fetch host.xz:~junio/repo
> 	git fetch ssh://host.xz/~junio/repo
> 
> mean the same thing (i.e. both understand ~user expansion).
> Also these are equivalent (i.e. no ~user expansion; both mean
> absolute filesystem path without SERVER_ROOT prefixing):
> 
> 	git fetch host.xz:/frotz/repo
> 	git fetch ssh://host.xz/frotz/repo
> 
> While these two might not mean the same thing (the former is
> prefixed with SERVER_ROOT, but not the latter):
> 
> 	git fetch git://host.xz/frotz/repo
> 	git fetch ssh://host.xz/frotz/repo
> 
> There are small technical issues.
> 
>  - connect.c should not be affected at all, since it does not
>    know how the remote site arranges SERVER_ROOT (if we support
>    it) or user home directories.
> 
It must remove the leading slash for this syntax:
	ssh://host.xz/~junio/repo

Otherwise it would be passed as /~junio/repo to the remote end and no ~user interpolation would be done.

Show 11 quoted lines
> 
>  - git-daemon needs to validate the incoming requested path and
>    in order to avoid aliasing issues, we should resolve ~user
>    expansion and SERVER_ROOT prefixing first, then validate the
>    resulting path against white/black list, before calling
>    upload-pack or receive-pack.  However, after git-daemon
>    decides to run these programs, they could find out some
>    problems with the specified repository and may need to report
>    them.  Arguably, this reporting should not reveal the real
>    path used to address the repository [*2*].
> 
This could be done by writing the relative path in the error message;
	.git/foo/bar: failed to do something nifty

The user or the admin should know where that path is and will know what to do. Messages logged on the server-side should ofcourse hold the full path.

Show 12 quoted lines
> Although we _could_ forget about the "error reporting exposing
> real path" issue for now, I think we should at least have a plan
> to make things consistent and well defined.  Here is a strawman:
> 
>  - Have a common library code that takes user supplied path and
>    does SERVER_ROOT prefixing and ~user expansion.
> 
>  - Have git-daemon use it to canonicalize the requested path
>    before validating.  Make it invoke the programs with the path
>    received from the other end (before SERVER_ROOT prefixing, or
>    ~user expansion).
> 

I'd say make it invoke the programs with the canonicalized path. As you say, git-daemon has to verify that it's a proper git repo and in the whitelist anyway so I think it would be silly to add extra complexity to upload-pack and receive-pack.

git-daemon could ofcourse present some uniform error message if git-upload-pack or git-receive-pack fails but this wouldn't really be necessary if they use relative paths as mentioned above (someone who makes one of those two fail while working will already know the path).

Show 8 quoted lines
>  - Give --server-root=/path/to/root flag to programs that can be
>    called by git-daemon, and have git-daemon run them with this
>    flag.  Have them use the same library to canonicalize the
>    requested path to the real path.  When these programs are run
>    via direct SSH connection (i.e. ssh://host/path and
>    host:path), this flag is not given so they see filesystem
>    path as-is, but make the ~user expansion still available.
> 
If we stick with canonicalized paths I suppose this can be dropped.
Show 9 quoted lines
> 
> [Footnote]
> 
> *2* This is theoretical right now, since packed transfer
> protocols cannot report errors back, but Andreas' patch
> addresses this issue by dying carefully in srvside_chdir().  It
> falls into security-by-obscurity category, so we may choose not
> to worry about it, though.
> 
Keeping valid usernames hidden is normally considered best practice.
-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231
H. Peter Anvin· Oct 25, 2005, 16:53 UTC · re: Andreas Ericsson · lore

Re: [RFC] GIT paths

Andreas Ericsson wrote:
Show 12 quoted lines
>>
>>  - Over a git-daemon connection, supporting ~user expansion
>>    makes sense.  E.g git://host.xz/~junio/ refers to my home
>>    directory on that machine.  It would make it impossible to
>>    have a directory literally named '~junio' directly underneath
>>    the root directory, but that is a good limitation anyway.
>>
> 
> I like this idea, although I'd extend it with a Userdir-like config 
> option in git-daemon (like ~/public_html for apache). This makes it a 
> bit easier to see what's published and what isn't.
> 

I've found that whenever one does a network daemon which exports paths, sooner or later one wants namespace management. In Linux, of course, there are a lot more tricks one can play to actually create the namespace one wants in the filesystem (although it's complicated by needing to have an exec-worthy environment.)

It might be worth to consider creating a library to do this.
	-hpa
Junio C Hamano· Oct 26, 2005, 06:12 UTC · re: Andreas Ericsson · lore

Re: [RFC] GIT paths

Andreas Ericsson <ae@op5.se> writes:
Show 7 quoted lines
> About the literally named /~junio directory, it would be possible with 
> this syntax;
>
> 	git fetch host.xz:/~junio
>
> The userdir is (with my previous patch) only expanded if the path starts 
> with a tilde.
I do not necessarily consider that a feature; see next item.
Show 6 quoted lines
> It must remove the leading slash for this syntax:
>
> 	ssh://host.xz/~junio/repo
>
> Otherwise it would be passed as /~junio/repo to the remote end and no 
> ~user interpolation would be done.

Not necessarily. Having the remote end interpret "/~user" and "~user" the same way might make things more consistent; in other words, "http://host/~user" is not spelled "http://host~user".

> I'd say make it invoke the programs with the canonicalized path. As you 
> say, git-daemon has to verify that it's a proper git repo and in the 
> whitelist anyway so I think it would be silly to add extra complexity to 
> upload-pack and receive-pack.
Yeah, I tend to agree here.
>>  - Give --server-root=/path/to/root flag to programs...
>
> If we stick with canonicalized paths I suppose this can be dropped.
Sounds good.
Andreas Ericsson· Oct 26, 2005, 08:56 UTC · re: Junio C Hamano · lore

Re: [RFC] GIT paths

Junio C Hamano wrote:
Show 20 quoted lines
> Andreas Ericsson <ae@op5.se> writes:
> 
>>
>>The userdir is (with my previous patch) only expanded if the path starts 
>>with a tilde.
> 
> 
> I do not necessarily consider that a feature; see next item.
> 
> 
>>It must remove the leading slash for this syntax:
>>
>>	ssh://host.xz/~junio/repo
>>
>>Otherwise it would be passed as /~junio/repo to the remote end and no 
>>~user interpolation would be done.
> 
> 
> Not necessarily.  Having the remote end interpret "/~user" and
> "~user" the same way might make things more consistent;

Except that the shell interprets ~ and /~ differently, so "consistent" would depend on what we're consistent with.

There's also the fact that shell-scripts won't work on the remote end if 
  git_connect() maintains the leading slash. This doesn't matter at 
present, but I think it'd be better to keep all doors open. Having the 
trivial addendum on the client side also helps keeping the server-side 
nice and simple.
> in other
> words, "http://host/~user" is not spelled "http://host~user".
> 

True. I meant for this to be invisible to the users ofcourse, with git_connect() having some snippet such as this.

if(use_ssh || use_git && (*path == '/' && *(path + 1) == '~')
	*path++ = '\0';
else
	copy_path();
Show 18 quoted lines
> 
>>I'd say make it invoke the programs with the canonicalized path. As you 
>>say, git-daemon has to verify that it's a proper git repo and in the 
>>whitelist anyway so I think it would be silly to add extra complexity to 
>>upload-pack and receive-pack.
> 
> 
> Yeah, I tend to agree here.
> 
> 
>>> - Give --server-root=/path/to/root flag to programs...
>>
>>If we stick with canonicalized paths I suppose this can be dropped.
> 
> 
> Sounds good.
> 
> 
I'll get busy then.
-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

← back to recent threads