GIT_OBJECT_DIRECTORY

Hi!

I recently noticed GIT_OBJECT_DIRECTORY in the git manpage and wanted to play with it. But it looks as if it doesn't work, the documentation is wrong/insufficient or I can't properly read the documentation. So let me figure out, which one it is.

$ set | grep GIT_OBJECT_DIRECTORY GIT_OBJECT_DIRECTORY=/home/joern/.git

$ ls -l /home/joern/.git total 288 drwxrwxr-x 2 joern joern 4096 Apr 16 01:22 0f [...]

$ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git foo [ stored >200M of data under foo/.git/objects ]

The above looks as if new objects are not stored under /home/joern/.git, as specified by GIT_OBJECT_DIRECTORY. The manpage tells me:

       GIT_OBJECT_DIRECTORY
              If  the  object storage directory is specified via this environ-
              ment variable then the sha1 directories are created underneath -
              otherwise the default $GIT_DIR/objects directory is used.

And I would interpret this as "store all new objects under /home/joern/.git". So far, things don't seem to imply me being too stupid. What went wrong?

Jörn

On Tue, 18 Apr 2006, Jörn Engel wrote:

The "rsync" protocol really doesn't honor git rules. It's basically just a big recursive copy, and it will copy things from the place they were before.

I suspect that if you had used a real git-aware protocol instead, you'd have been fine, ie

	git clone git://git.kernel.org/... foo/

would probably work. (I say "probably", because very few people likely use GIT_OBJECT_DIRECTORY, and it makes a lot less sense with pack-files than it did originally, so it's not getting any testing).

		Linus

On Tue, 18 April 2006 08:25:47 -0700, Linus Torvalds wrote:

Is it possible for non-owners of a kernel.org account to do this?

Well, .git/objects for your kernel still consumes 121M. It's not gigabytes but I still wouldn't want too many copies of that lying around. Right now, I already feel slightly motivated to move the whole content-addressable idea into the kernel. It has disadvantages, but the effect on disk- and pagecache-footprint for people like me would come in handy.

Jörn

On Tue, Apr 18, 2006 at 07:58:53PM +0200, J?rn Engel wrote:
 > 

Yes - everyone can do so. I never use rsync myself anymore.

	Sam

On Tue, 18 Apr 2006, Jörn Engel wrote:

Yes, kernel.org runs the git daemon.

If a repo isn't packed enough, the git protocol can be pretty CPU intensive, but I'm hoping that everybody keeps their repos mostly packed, at which point the git protocol should actually be a lot faster than rsync.

Right. However, these days we have better approaches than GIT_OBJECT_DIRECTORY for that.

In particular, if you create local clones, use "git clone -l -s", which shares its base objects with the thing you clone from. It makes the clone incredibly fast too (the only real cost is the check-out, which can obviously be pretty expensive), and you can then use

	git repack -a -d -l

on all the to repack just the _local_ objects to avoid having packs duplicate objects unnecessarily.

		Linus

On Tue, 18 April 2006 11:08:40 -0700, Linus Torvalds wrote:

Excellent! I have a faint memory of hpa recently saying that the git daemon would be too resource-hungry. One of the cases where being wrong is a Good Thing.

This still isn't good enough for me. Before git, all my trees were hard-linked (cowlinked, actually) and another copy barely consumed any space. "git clone -l -s" creates a copy of the currently 311M of kernel source, quite a bit more expensive.

But it appears as if I could "cp -lr" the git tree and work with that. The nice thing of having cowlinks is that I don't have to rely on git breaking the hard links - which it probably won't. But since the estimated user base of cowlinks is 1, that won't help too many people.

Jörn

On Tue, 18 Apr 2006, Jörn Engel wrote:

That should work. I just personally fear cowlinks, because some things will edit the files in place, and then you're screwed.

I _think_ it should be ok for the .git subdirectory, but quite frankly, I'm not going to guarantee it. Also, you will break the cow-linking when you ever re-pack either the source or the destination, so you'd actually be _better_ off with something that does

	# clone the git directories by hand, no checkout (-n).
	git clone -l -s -n src dst

	# cow-link the checked-out state
	(cd src ; git ls-files | cpio -pudml dst)

	# make sure to refresh the index
	git update-index --refresh

or something like that.

TOTALLY UNTESTED!! (And you need to have made "dst" be an absolute path, of course, since we want it to work even after we've done the "cd src" thing).

		Linus

On Tue, 18 April 2006 11:47:53 -0700, Linus Torvalds wrote:

s/cowlinks/hardlinks/ ?

The reason for me to write the cowlink patches was exactly the fear you are talking about. With those patches, links are broken whenever such a thing happens.

In that case, cowlinks should still turn a blatant bug into some wasted space - which is a hell of a lot better.

And this happens to be a problem. Creating the links when the copy is created is simple. Detecting identical files and linking them after the fact is racy, complicated, racy and, well, racy. I wouldn't want to touch it with a ten foot pole. Not without kernel support.

Jörn

Jörn Engel wrote:

Well, we ended up making some tweaks to the git daemon, and it hasn't been a problem since.

	-hpa

"H. Peter Anvin" <hpa@zytor.com> writes:

Ah, I am glad the daemon expert was listening... Do you have comments on recent patch from Serge E. Hallyn? It looks OK to me, but that standalone daemon part is not something I run myself, so...

-- >8 -- [PATCH] socksetup: don't return on set_reuse_addr() error

The set_reuse_addr() error case was the only error case in socklist() where we returned rather than continued. Not sure why. Either we must free the socklist, or continue. This patch continues on error.

Signed-off-by: Serge E. Hallyn <serue@us.ibm.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>

---

 daemon.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

0032d548db56eac9ea09b4ba05843365f6325b85

Jörn Engel <joern@wohnheim.fh-wedel.de> writes:

That is what "git clone -l -s" is for.  

The alternates pointer mechanism used with the above largely makes GIT_OBJECT_DIRECTORY unnecessary for end users these days. It is a fine mechanism as an implementation detail of the lowlevel and Porcelains, and that is the reason the documentation still mentions the environment.

On Tue, 18 April 2006 11:20:58 -0700, Junio C Hamano wrote:

See my response to Linus. .git/objects is currently the smaller problem. The larger problem is 311M of raw kernel source - without any SCM overhead of any flavour. Like many others, I solved the larger problem with hardlink trees. "git clone -l -s" is imo nearly unethical, as it solved the smaller problem and leaves the larger one unaffected. It reeks of hypocricy.

Hardlink trees still aren't perfect. If I take one tree, "cp -lr" it twice and apply the same patches to both copies, the changed files exist twice for both copies. That sucks, but it is a fairly small problem and there is no simple solution to it.

If git was able to deal with hardlink trees and properly break the links when working in one copy, "cp -lr" would be a lot smarter than "git clone -l -s". It just happens that I have written some kernel patches that automatically break hardlinks, even if applications don't know how to do it. So for my personal use, git has this ability.

Now, going one step further, GIT_OBJECT_DIRECTORY could solve another problem. Just like source files, git object can be pulled twice into two copies of a tree. But for git objects, there appeared to be an easy solution: the central object storage. So we're back where this thread started.

Except that I get the idea of GIT_OBJECT_DIRECTORY not being a simple solution to a small problem, so maybe I should just forget about it.

Jörn