HOWTO set up a repository which can be pushed into over HTTP

Hi,

I set up such a repository twice now, and it is not exactly straight-forward. So, maybe some people find this useful:

This text assumes that you

- have an Apache webserver,
- can edit the configuration of it,
- can restart it, and
- have permissions to chown a directory

Step 1: setup a bare repository -------------------------------

Create the directory (this assumes you have your Apache installed in /usr/local/apache2):

	$ cd /usr/local/apache/htdocs
	$ mkdir my-new-repo.git

Initialize a bare repository

	$ cd my-new-repo.git
	$ git --bare init-db

Change the ownership to your webserver's credentials

	$ chown -R www.www .

If you do not know which user Apache runs as, you can alternatively do a "chmod -R a+w .", inspect the files which are created later on, and set the permissions appropriately.

Step 2: enable DAV on this repository -------------------------------------

In your httpd.conf, make sure that this line exists:

	DAVLockDB "/usr/local/apache2/temp/DAV.lock"

Of course, it can point somewhere else, but the string is actually just a prefix in some Apache configurations, and therefore the _directory_ has to be writable by the user Apache runs as.

Then, add something like this to your httpd.conf

	<Location /my-new-repo.git>
		DAV on
		AuthType Basic
		AuthName "Git"
		AuthUserFile /usr/local/apache2/conf/passwd.git
		Require valid-user
	</Location>

The password file can be somewhere else, but it has to be readable by Apache.

You can create this file by

	$ htpasswd -c /usr/local/apache2/conf/passwd.git <user>

you will be asked a password, and the file is created. Subsequent calls to htpasswd should omit the '-c' option, since you want to append to the existing file.

You need to restart Apache; 'apachectl --graceful' is sufficient.

Step 3: setup the client ------------------------

Make sure that you have HTTP support, i.e. your git was built with curl. The easiest way to check is to look for the executable 'git-http-push'.

Then, add the following to your .netrc (you can do without, but will be asked to input your password a _lot_ of times):

	machine <servername>
	login <username>
	password <password>

If you want to access the webserver by its IP, you have to type that in, instead of the server name.

Now, add the remote in your existing repository which contains the project you want to export:

	$ git-repo-config remote.upload.url \
		http://<username>@<servername>/my-new-repo.git/

It is important to put the last '/'; Without it, the server will send a redirect which git-http-push does not (yet) understand, and git-http-push will repeat the request inifinitely.

Step 4: make the initial push -----------------------------

	$ git-http-push upload master

(This assumes that the branch you want to export is called 'master' in your client setup...)

Troubleshooting: ----------------

If git-http-push says

	Error: no DAV locking support on remote repo http://...

then it means the webserver did not accept your authentication. Make sure that the user name and password matches in httpd.conf, .netrc and the URL you are uploading to.

If git-http-push shows you an error (22/502) when trying to MOVE a blob, it means that your webserver somehow does not recognize its name in the request; This can happen when you start Apache, but then disable the network interface. A simple restart of Apache helps.

In other cases, reading /usr/local/apache2/logs/error_log is often helpful.

Ciao, Dscho

Nice addition to Documentation/howto/ I would presume...

Hi,

On Fri, 4 Aug 2006, Junio C Hamano wrote:

I actually planned to submit a patch with a file in that directory first... But then I read the files already in it, and they all look like they were emails first, which got -- presumably by several people finding it useful -- into a patch.

Do people find it useful? Or would you like to wait until I implement an option in git-http-push to actually init an HTTP repo remotely?

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

I do not have need for push over http-dav at the moment myself, but I would imagine I would certainly look for it when I would need to later.

Do people find it useful? More importantly, has somebody else independently tried to follow the documentation and found the description accurate and helpful?

On Mon, Aug 07, 2006 at 12:55:43AM -0700, Junio C Hamano wrote:

Today I did try to follow it, and it made an otherwise impossible job for me doable, although still taking too much hours and a patch. The 'impossible job' for me had to do with my zero-apache experience and (probably) a bug in WebDAV.

Find attached updated HOWTO with specific information for Debian users, and a patch to git to git-http-push since initially WebDAV gave me MOVE 22/404 errors.

-- Rutger Nijlunsing ---------------------------------- eludias ed dse.nl never attribute to a conspiracy which can be explained by incompetence ----------------------------------------------------------------------

HOWTO: Setting up a git repository which can be pushed into and pulled
from over HTTP.

What's needed:

- Have an Apache webserver

  On Debian:
    $ apt-get install apache2
    To get apache2 by default started,
    edit /etc/default/apache2 and set NO_START=0

- can edit the configuration of it

  On Debian: this means being able to edit files under /etc/apache2

- can restart it.
  'apachectl --graceful' might do.

  On Debian:
    $ /etc/init.d/apache2 restart

- have permissions to chown a directory

In effect, this probably means you're going to be root.

Step 1: setup a bare GIT repository -----------------------------------

Create the directory (this assumes you have your Apache installed in /usr/local/apache2):

    $ cd /usr/local/apache/htdocs
    $ mkdir my-new-repo.git

  On Debian:

    $ cd /var/www
    $ mkdir my-new-repo.git

Initialize a bare repository
 
    $ cd my-new-repo.git
    $ git --bare init-db

Change the ownership to your webserver's credentials
 
    $ chown -R www.www .

  On Debian:

    $ chown -R www-data.www-data .

If you do not know which user Apache runs as, you can alternatively do a "chmod -R a+w .", inspect the files which are created later on, and set the permissions appropriately.

Restart apache2, and check whether http://server/my-new-repo.git gives a directory listing. If not, check whether apache started up succesfully.

Step 2: enable DAV on this repository -------------------------------------

In your httpd.conf, make sure that this line exists which is the file used for locking DAV operations:

  DAVLockDB "/usr/local/apache2/temp/DAV.lock"

  On Debian:

    Enable the dav and dav_fs modules of apache:
    $ a2enmod dav_fs
    $ a2enmod dav
    The DAV lock is located in /etc/apache2/mods-available/dav_fs.conf:
      DAVLockDB /var/lock/apache2/DAVLock

Of course, it can point somewhere else, but the string is actually just a prefix in some Apache configurations, and therefore the _directory_ has to be writable by the user Apache runs as.

Then, add something like this to your httpd.conf

  <Location /my-new-repo.git>
     DAV on
     AuthType Basic
     AuthName "Git"
     AuthUserFile /usr/local/apache2/conf/passwd.git
     Require valid-user
  </Location>

  On Debian:
    Create (or add to) /etc/apache2/conf.d/git.conf :

    <Location /my-new-repo.git>
       DAV on
       AuthType Basic
       AuthName "Git"
       AuthUserFile /etc/apache2/passwd.git
       Require valid-user
    </Location>

The password file can be somewhere else, but it has to be readable by Apache and preferably not readable by the world.

Create this file by
    $ htpasswd -c /usr/local/apache2/conf/passwd.git <user>

    On Debian:
      $ htpasswd -c /etc/apache2/passwd.git <user>

You will be asked a password, and the file is created. Subsequent calls to htpasswd should omit the '-c' option, since you want to append to the existing file.

You need to restart Apache.

Now go to http://<username>@<servername>/my-new-repo.git in your browser to check whether it asks for a password and accepts the right password.

On Debian:

   To test the WebDAV part, do:

   $ apt-get install litmus
   $ litmus http://<servername>/my-new-repo.git <username> <password>

   Mosts tests should pass.

Step 3: setup the client
------------------------
 
Make sure that you have HTTP support, i.e. your git was built with curl.
The easiest way to check is to look for the executable 'git-http-push'.

Then, add the following to your ~/.netrc (you can do without, but will be asked to input your password a _lot_ of times):

    machine <servername>
    login <username>
    password <password>

...and set permissions:
     chmod 600 ~/.netrc

    On Windows, use %HOMEDRIVE%%HOMEPATH%.netrc instead.

If you want to access the webserver by its IP, you have to type that in, instead of the server name.

To check whether all is OK, do:

   curl --netrc --location -v http://<username>@<servername>/my-new-repo.git/

...this should give a directory listing in HTML of /var/www/my-new-repo.git .

Now, add the remote in your existing repository which contains the project
you want to export:
 
   $ git-repo-config remote.upload.url \
       http://<username>@<servername>/my-new-repo.git/

It is important to put the last '/'; Without it, the server will send a redirect which git-http-push does not (yet) understand, and git-http-push will repeat the request inifinitely.

Step 4: make the initial push -----------------------------

   $ git push upload master

This pushes branch 'master' (which is assumed to be the branch you want to export) to repository called 'upload', which we previously defined with git-repo-config.

Troubleshooting: ----------------

If git-http-push says

   Error: no DAV locking support on remote repo http://...

then it means the webserver did not accept your authentication. Make sure that the user name and password matches in httpd.conf, .netrc and the URL you are uploading to.

If git-http-push shows you an error (22/502) when trying to MOVE a blob, it means that your webserver somehow does not recognize its name in the request; This can happen when you start Apache, but then disable the network interface. A simple restart of Apache helps.

Errors like (22/502) are of format (curl error code/http error code). So (22/404) means something like 'not found' at the server.

Reading /usr/local/apache2/logs/error_log is often helpful. On Debian: Read /var/log/apache2/error.log instead.

commit bd1ba734176277034c6e4b33eb68fde170286538
Author: Rutger Nijlunsing <git@tux.tmfweb.nl>
Date:   Tue Aug 8 14:17:38 2006 +0200

    http-push: Use '_token' instead of '.token' as temporary file before renaming.
    
    WebDAV on Debian unstable cannot handle renames on WebDAV from
    file.ext to newfile (without ext) when newfile* already
    exists. Normally, git creates a file like 'objects/xx/sha1.token',
    which is renamed to 'objects/xx/sha1' when transferred completely.
    
    Just use '_' instead of '.' so WebDAV doesn't see it as an extension
    change.

Hi,

On Tue, 8 Aug 2006, Rutger Nijlunsing wrote:

Thank you for doing that valuable work! I think the patch is fine (take that as an Acked-by:); next time you might want to send it as a proper patch mail ;-)

On some other setups, it is /etc/httpd/. Since it is not specific to Debian, we might want to give some common options, but refer the user to the Apache manual if she cannot find it.

It is Ã'apachectl graceful' here (without'--').

This should work everywhere, except it might be called apache or httpd instead. However, please warn users that this interrupts (however briefly) a running webserver, and if you do that on a production machine, users just downloading an iso file might be rightfully upset.

Right.

Again, it might be cooler to be a bit more portable: if the user found httpd.config, he could just "grep DocumentRoot /where/ever/httpd.conf".

Again, a better way would be to direct the user to "grep ^User httpd.conf" and "grep ^Group httpd.conf".

A useful addition.

I did not find a2enmod on my only Debian setup, so a more portable way would be to tell the user to insert the following lines in httpd.conf:

	LoadModule dav_module libexec/httpd/libdav.so
	AddModule mod_dav.c

(I do not know if there are setups where you have to load dav_fs explicitely...)

Is this picked up automatically?

We should not violate the DRY principle ("Don´'Repeat Yourself").

So, it was my fault to begin with: the '--' is wrong.

Helpful.

There has to be a more portable way to test WebDAV, like an URL to type into a webbrowser. DAV wizards, anyone?

Okay, I thought it was common knowledge where this resides. Although I would say $HOME/.netrc instead.

Yeah, right. I always forget about this.

Huh? You want to access it with something else than cygwin?

Does everybody who has libcurl installed, have curl installed also?

Your wording is much clearer.

I think the error log is more useful than the http error codes.

Ciao, Dscho

Ok, I send it out. Don't know if I did it proper...

Ok, but I find it always very convenient when someone describes the situation for _me_. So if other distro's would join in, I would be delighted. And indeed a general fallback to the Apache documentation would be good for all distro's (+ homegrown) not covered.

However, Debian already differs quite some from the defaults. I wrote this addendum to the documentation _because_ it costed me quite some time since Debian tries to be different.

Debian seperates as much as possible into different files and directories to be able to easily install and deinstall modules with the package manager. And it has scripts to manage those directories.

If you're on a production machine with a webserver already in production, you're probably not using this tutorial (I hope).

If you're downloading a large iso, recent browsers have 'resume' buttons. The internet has never been very reliable.

If you're on Debian, '/etc/init.d/apache2 restart' is the same as '/etc/init.d/apache2 force-reload', so no interruptions.

...but warning is always good :)

Nope. On Debian, httpd.conf is a placeholder which does not contain anything expect for:

# This is here for backwards compatability reasons and to support # installing 3rd party modules directly via apxs2, rather than # through the /etc/apache2/mods-{available,enabled} mechanism. # #LoadModule mod_placeholder /usr/lib/apache2/modules/mod_placeholder.so

And the grep gives nothing. Doing a recursive grep gives:

/etc/apache2$ grep -r DocumentRoot * sites-available/default: DocumentRoot /var/www sites-enabled/000-default: DocumentRoot /var/www

...where sites-enabled is a link to sites-available.

Both come up empty on Debian. The uid + gid appear to be in apache2.conf :

/etc/apache2$ grep ^User * apache2.conf:User www-data

No, a2enmod is the recommended way to enable modules. It adds symlinks from the /etc/apache2/mods-enabled to /etc/apache2/mods-available.

I got those from http://www.debian-administration.org/articles/285

According to http://www.debian-administration.org/articles/285, I had to. Didn't check it myself, though.

Yup. From /etc/apache2/README:

... conf.d/

        Files in this directory are included by this line in
        apache2.conf:

        # Include generic snippets of statements
        Include /etc/apache2/conf.d

        This is a good place to add additional configuration
        directives.

httpd.conf

        Empty file.
...

Ok, a merge could be done, but AuthUserFile is different.
Something like
    +       AuthUserFile <passwd-file>
then.

The renaming bug I found by using Windows Internet Explorer -> Open Location -> http://<servername>/my-new-repo.git [x] Open as webfolder -> login, and copy a file to it, rename it, ...

Another interactive tool is cadaver (see http://www.debian-administration.org/articles/285 again).

With FUSE + fusedav you can mount a WebDAV like a normal filesystem (which is cool ;)

Ok.

You're right, Cygwin should have $HOME set.

Don't know, but I think most people who have apache2 installed have better things to worry about :) Might want to add the dependancy explicitly in the document, yes.

It took me a while to figure out, I (incorrectly) assumed 'upload' was the name of the remote branch. And git-http-push didn't work at once, so I got it working with some extra options before I realised it should be 'git push' instead.

You deleted those two lines (look at the version I sent), so jut put them back :)

Do you want to finish it, or should I do a reedit?

Regards, Rutger.

Hi,

On Wed, 9 Aug 2006, Rutger Nijlunsing wrote:

If everybody did this, you would not _find_ your distro-specific things ;-)

But I agree, Debian is a bureaucrat's dream. I positively _hate_ it. So much so, that I left Knoppix behind.

Thanks for clearing these up for me!

I am a little short on time, and I want to finish the merge-recur stuff, so I would be really grateful if you did it.

Ciao, Dscho