threads / discuss / 1773

Re: What's up with the GIT archive on www.kernel.org?

Subject: Re: What's up with the GIT archive on www.kernel.org?

## tl;dr

18 messages between Sep 11, 2005 and Sep 13, 2005.

replies: 17people: 9as markdown or json

Linus Torvalds· Sep 11, 2005, 19:06 UTC · lore
On Sun, 11 Sep 2005, Sam Ravnborg wrote:
Show 9 quoted lines
> > 
> > Absolutely. The mirroring has been slow again lately. I've packed my 
> > archive, but I suspect others should much more aggressively now be using 
> > the "objects/info/alternates" information to point to my tree, so that 
> > they don't even need to have their objects at all (no packing 
> > even necessary - just running "git prune-packed" on peoples archives 
> > would get rid of any duplicate objects when I pack mine).
> 
> Can you post a small description how to utilize this method?
Just do
	echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects > objects/info/alternates

in your tree, and that will tell git that your tree can use my object directory as an "alternate" source of objects. At that point, you can remove all objects that I have.

However, that only works with a local directory - you can't say that the alternate object directory is over the network (unless you use NFS or similar, of course ;).

Another potential problem is that while the above makes git understand to pick the objects from my directory, it can in theory cause problems for mirrors etc - since they mirror things to a different location and/or may not mirror all of it anyway.

Anyway, modulo those caveats, you can then just do
	git prune-packed

and it will remove all unpacked objects in your git archive that can be reached through a pack-file - including any packfiles in _my_ directory.

Then you never need to pack your own objects any more. Just leave everything unpacked, and rely on me packing every once in a while, and just do "git prune-packed" when I do.

That allows a site like kernel.org to effectively share 99% of all objects, and do it efficiently.

		Linus
Sam Ravnborg· Sep 11, 2005, 19:46 UTC · re: Linus Torvalds · lore
Show 10 quoted lines
> > 
> > Can you post a small description how to utilize this method?
> 
> Just do
> 
> 	echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects > objects/info/alternates
> 
> in your tree, and that will tell git that your tree can use my object 
> directory as an "alternate" source of objects. At that point, you can 
> remove all objects that I have.
OK - what I did:

cd /pub/scm/linux/kernel/git/sam rm -rf kbuild.git git clone /pub/scm/linux/kernel/git/torvalds/linux-2.6.git kbuild.git rename to .git to kbuild.git

I had to specify both GIT_DIR and GIT_OBJECT_DIRECTORY to make git-prune-packed behave as expected. I assume this is normal when I rename the .git directory like in this case.

I will se if any pullers complins (mostly/only Andrew I think).
	Sam
Linus Torvalds· Sep 11, 2005, 19:56 UTC · re: Sam Ravnborg · lore
On Sun, 11 Sep 2005, Sam Ravnborg wrote:
> 
> I had to specify both GIT_DIR and GIT_OBJECT_DIRECTORY to make
> git-prune-packed behave as expected. I assume this is normal when I
> rename the .git directory like in this case.

You should only need to specify GIT_DIR - it should figure out that the object directory follows GIT_DIR on its own.

Also, I forget what version of git is installed on kernel.org. The "alternates" support has been around for a while, and looking at the date of "/usr/bin/git" it _seems_ recent (Sep 7), but I haven't seen any announcement of updating since the last one (which was git-0.99.4, which is too old).

You can try removing all the packs in your .git/objects/packs directory. Everything _should_ still work fine.

Famous last words.
		Linus
Roland Dreier· Sep 11, 2005, 21:09 UTC · re: Linus Torvalds · lore
    Linus> You can try removing all the packs in your
    Linus> .git/objects/packs directory. Everything _should_ still
    Linus> work fine.
Does "everything" include someone doing
    git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/whatever.git
How about http:// instead of rsync://?

In other words, is the git network transport smart enough to handle the alternates path?

Or is the idea that everyone will clone your tree and then pull extra stuff from other trees?

 - R.
Linus Torvalds· Sep 11, 2005, 21:24 UTC · re: Roland Dreier · lore
On Sun, 11 Sep 2005, Roland Dreier wrote:
> 
> Does "everything" include someone doing
> 
>     git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/whatever.git
Nope. Only server-side smart protocols will handle this.

There is such an anonymous server, btw: "git-daemon" implements anonymous access much more efficient than rsync/http. Sadly, kernel.org still doesn't offer it (but it's now used in the wild, ie I've done a couple of merges with people running the git daemon).

> In other words, is the git network transport smart enough to handle
> the alternates path?
The _git_ network transport is. rsync and http aren't.
		Linus
Linus Torvalds· Sep 11, 2005, 21:33 UTC · re: Linus Torvalds · lore
On Sun, 11 Sep 2005, Linus Torvalds wrote:
> 
> The _git_ network transport is. rsync and http aren't.

Btw, there's no reason why a client-side thing couldn't just parse the "alternates" thing, and if it doesn't find the objects in the main object directory, go and fetch them from the alternates itself.

IOW, this is not a fundamental problem with alternates, it's just that since there is no server-side smarts to handle it (ie just raw file access with rsync/http), it needs to be handled at the client side instead.

		Linus
Junio C Hamano· Sep 12, 2005, 01:39 UTC · re: Linus Torvalds · lore
Linus Torvalds <torvalds@osdl.org> writes:
> Btw, there's no reason why a client-side thing couldn't just parse the 
> "alternates" thing, and if it doesn't find the objects in the main object 
> directory, go and fetch them from the alternates itself.
There is.

For kernel.org, you could say '/pub/scm/blah' in your alternates and expect it to work, only because http://kernel.org/pub hierarchy happens to match the absolute path /pub on the filesystem, but for most people's default HTTP server installation, they would need to say /var/www/scm/blah to have alternate work locally, but somebody has to know that the named directory is served as http://machine.xz/pub/scm/blah somewhere.

Client side smarts need some help from the user here to know that '/var/www/scm/blah' read off of objects/info/alternates match that URL.

Dmitry Torokhov· Sep 12, 2005, 02:45 UTC · re: Junio C Hamano · lore
On Sunday 11 September 2005 20:39, Junio C Hamano wrote:
Show 16 quoted lines
> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > Btw, there's no reason why a client-side thing couldn't just parse the 
> > "alternates" thing, and if it doesn't find the objects in the main object 
> > directory, go and fetch them from the alternates itself.
> 
> There is.
> 
> For kernel.org, you could say '/pub/scm/blah' in your alternates
> and expect it to work, only because http://kernel.org/pub
> hierarchy happens to match the absolute path /pub on the
> filesystem, but for most people's default HTTP server
> installation, they would need to say /var/www/scm/blah to have
> alternate work locally, but somebody has to know that the named
> directory is served as http://machine.xz/pub/scm/blah somewhere.
> 

Call me brain-dead but all of this just makes me rsync my tree to kernel.org and then manually do "ln -f" for all the packs that Linus has. This way I am sure tht the tree is what I have plus and it is "pullable".

-- 
Dmitry
Ryan Anderson· Sep 12, 2005, 18:42 UTC · re: Dmitry Torokhov · lore
On Sun, Sep 11, 2005 at 09:45:33PM -0500, Dmitry Torokhov wrote:
Show 5 quoted lines
> 
> Call me brain-dead but all of this just makes me rsync my tree to
> kernel.org and then manually do "ln -f" for all the packs that Linus
> has. This way I am sure tht the tree is what I have plus and it is
> "pullable".

If you have access to make hardlinks, you should be able to use git-relink to do the hard work for you.

>From memory:
	git relink my_dir1 my_dir2 ... master_dir
or:
	git relink my-kernel-tree /pub/scm/.../torvalds/linux.git/

(I think that will work - via a bug in my initial attempt to write git-relink, I look to make sure the path ends in ".git/" not "/.git/". So the above should work. I think.)

-- 
Ryan Anderson
  sometimes Pug Majere
Linus Torvalds· Sep 12, 2005, 03:39 UTC · re: Junio C Hamano · lore
On Sun, 11 Sep 2005, Junio C Hamano wrote:
Show 8 quoted lines
> 
> For kernel.org, you could say '/pub/scm/blah' in your alternates
> and expect it to work, only because http://kernel.org/pub
> hierarchy happens to match the absolute path /pub on the
> filesystem, but for most people's default HTTP server
> installation, they would need to say /var/www/scm/blah to have
> alternate work locally, but somebody has to know that the named
> directory is served as http://machine.xz/pub/scm/blah somewhere.

Yes. We should probably have some well-defined meaning for relative paths in there regardless (eg just define that they are always relative to the main GIT_OBJECT_DIRECTORY or something).

That would also allow mirrors to mirror the git archives in different places, without upsetting the result (as long as they are mirrored together).

		Linus
Junio C Hamano· Sep 13, 2005, 07:05 UTC · re: Linus Torvalds · lore

[PATCH] Define relative .git/objects/info/alternates semantics.

Linus Torvalds <torvalds@osdl.org> writes:
Show 9 quoted lines
> Yes. We should probably have some well-defined meaning for relative paths
> in there regardless (eg just define that they are always relative to the
> main GIT_OBJECT_DIRECTORY or something).
>
> That would also allow mirrors to mirror the git archives in different 
> places, without upsetting the result (as long as they are mirrored 
> together).
>
> 		Linus

This patch is request-for-comments. I have experimented it and have a feeling that it may be more intuitive to make it relative to $project.git/ directory, instead of $project.git/objects as you originally suggested, in which case a maintainer tree would have "../../torvalds/linux-2.6.git/objects" instead (one less dotdot), and if nobody objects that is probably what I'll end up doing.

------------ An entry in the alternates file can name a directory relative to the object store it describes. A typical linux-2.6 maintainer repository would have "../../../torvalds/linux-2.6.git/objects" there, because the subsystem maintainer object store would live in

    /pub/scm/linux/kernel/git/$u/$system.git/objects/
and the object store of Linus tree is in
    /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/

This unfortunately is different from GIT_ALTERNATE_OBJECT_DIRECTORIES which is relative to the cwd of the running process, but there is no way to make it consistent with the behaviour of the environment variable. The process typically is run in $system.git/ directory for a naked repository, or one level up for a repository with a working tree, so we just define it to be relative to the objects/ directory to be different from either ;-).

Later, the dumb transport could be updated to read from info/alternates and make requests for the repository the repository borrows from.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 sha1_file.c |   28 ++++++++++++++++++++++------
 1 files changed, 22 insertions(+), 6 deletions(-)
e2e8a0ba5fc80368bf46c615276e406dd373729c
diff --git a/sha1_file.c b/sha1_file.c
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -240,10 +240,12 @@ static struct alternate_object_database 
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void link_alt_odb_entries(const char *alt, const char *ep, int sep)
+static void link_alt_odb_entries(const char *alt, const char *ep, int sep,
+				 const char *relative_base)
 {
 	const char *cp, *last;
 	struct alternate_object_database *ent;
+	int base_len = -1;
 
 	last = alt;
 	while (last < ep) {
@@ -261,12 +263,25 @@ static void link_alt_odb_entries(const c
 			int pfxlen = cp - last;
 			int entlen = pfxlen + 43;
 
+			if (*last != '/' && relative_base) {
+				/* Relative alt-odb */
+				if (base_len < 0)
+					base_len = strlen(relative_base) + 1;
+				entlen += base_len;
+				pfxlen += base_len;
+			}
 			ent = xmalloc(sizeof(*ent) + entlen);
 			*alt_odb_tail = ent;
 			alt_odb_tail = &(ent->next);
 			ent->next = NULL;
-
-			memcpy(ent->base, last, pfxlen);
+			if (*last != '/' && relative_base) {
+				memcpy(ent->base, relative_base, base_len - 1);
+				ent->base[base_len - 1] = '/';
+				memcpy(ent->base + base_len,
+				       last, cp - last);
+			}
+			else
+				memcpy(ent->base, last, pfxlen);
 			ent->name = ent->base + pfxlen + 1;
 			ent->base[pfxlen] = ent->base[pfxlen + 3] = '/';
 			ent->base[entlen-1] = 0;
@@ -288,12 +303,12 @@ void prepare_alt_odb(void)
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 	if (!alt) alt = "";
 
-	sprintf(path, "%s/info/alternates", get_object_directory());
 	if (alt_odb_tail)
 		return;
 	alt_odb_tail = &alt_odb_list;
-	link_alt_odb_entries(alt, alt + strlen(alt), ':');
+	link_alt_odb_entries(alt, alt + strlen(alt), ':', NULL);
 
+	sprintf(path, "%s/info/alternates", get_object_directory());
 	fd = open(path, O_RDONLY);
 	if (fd < 0)
 		return;
@@ -306,7 +321,8 @@ void prepare_alt_odb(void)
 	if (map == MAP_FAILED)
 		return;
 
-	link_alt_odb_entries(map, map + st.st_size, '\n');
+	link_alt_odb_entries(map, map + st.st_size, '\n',
+			     get_object_directory());
 	munmap(map, st.st_size);
 }
 
Linus Torvalds· Sep 13, 2005, 16:22 UTC · re: Junio C Hamano · lore

Re: [PATCH] Define relative .git/objects/info/alternates semantics.

On Tue, 13 Sep 2005, Junio C Hamano wrote:
Show 5 quoted lines
>
> This patch is request-for-comments.  I have experimented it and
> have a feeling that it may be more intuitive to make it relative
> to $project.git/ directory, instead of $project.git/objects as
> you originally suggested
I don't think you can do that.

I always felt that "alternates" should be per-project, but you're the one who argued that "alternates" is a per-object-directory thing.

Which means that the _same_ "alternates" file can be shared with many different project.git/ directories, and thus it's not well-defined to make it relative to GIT_DIR.

You can make it relative to "GIT_OBJECTS_DIR/.." of course, which in most cases is the same thing as "GIT_DIR".

		Linus
Junio C Hamano· Sep 13, 2005, 17:30 UTC · re: Linus Torvalds · lore

Re: [PATCH] Define relative .git/objects/info/alternates semantics.

Linus Torvalds <torvalds@osdl.org> writes:
> Which means that the _same_ "alternates" file can be shared with many
> different project.git/ directories, and thus it's not well-defined to make
> it relative to GIT_DIR.
Good point.
> You can make it relative to "GIT_OBJECTS_DIR/.." of course, which in most
> cases is the same thing as "GIT_DIR".
Again, good point.
Daniel Barkalow· Sep 13, 2005, 16:31 UTC · re: Junio C Hamano · lore

Re: [PATCH] Define relative .git/objects/info/alternates semantics.

On Tue, 13 Sep 2005, Junio C Hamano wrote:
Show 19 quoted lines
> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > Yes. We should probably have some well-defined meaning for relative paths
> > in there regardless (eg just define that they are always relative to the
> > main GIT_OBJECT_DIRECTORY or something).
> >
> > That would also allow mirrors to mirror the git archives in different 
> > places, without upsetting the result (as long as they are mirrored 
> > together).
> >
> > 		Linus
> 
> This patch is request-for-comments.  I have experimented it and
> have a feeling that it may be more intuitive to make it relative
> to $project.git/ directory, instead of $project.git/objects as
> you originally suggested, in which case a maintainer tree would
> have "../../torvalds/linux-2.6.git/objects" instead (one less
> dotdot), and if nobody objects that is probably what I'll end up
> doing.

It seems odd to have the "objects" at the end and not be starting from "objects". I suspect that the most intuitive thing would be for something under $project.git/info to be .git to .git, while something under .git/objects/info should be objects to objects.

	-Daniel
*This .sig left intentionally blank*
H. Peter Anvin· Sep 12, 2005, 17:10 UTC · re: Junio C Hamano · lore
Junio C Hamano wrote:
Show 5 quoted lines
> 
> For kernel.org, you could say '/pub/scm/blah' in your alternates
> and expect it to work, only because http://kernel.org/pub
> hierarchy happens to match the absolute path /pub on the
> filesystem...
 >

Actually it doesn't. /pub in the root directory on kernel.org is just a convenience symlink.

	-hpa
Tony Luck· Sep 12, 2005, 18:22 UTC · re: Linus Torvalds · lore
On 9/11/05, Linus Torvalds <torvalds@osdl.org> wrote:
> There is such an anonymous server, btw: "git-daemon" implements anonymous
> access much more efficient than rsync/http. Sadly, kernel.org still
> doesn't offer it (but it's now used in the wild, ie I've done a couple of
> merges with people running the git daemon).

Should the git daemon take a look at objects/info/alternates to check that if it exists, it points to a repository that also has a "git-daemon-export-ok" file? I don't see that this could be used for anything nasty, but it does provide a loophole where the daemon may open files outside the initial repository ... so a sanity check seems in order.

-Tony
Linus Torvalds· Sep 12, 2005, 18:37 UTC · re: Tony Luck · lore
On Mon, 12 Sep 2005, Tony Luck wrote:
> 
> Should the git daemon take a look at objects/info/alternates to check
> that if it exists, it points to a repository that also has a
> "git-daemon-export-ok" file?

I considered it, but decided against the complexity. I just don't see the point. The "git-daemon-export-ok" is not so much about security as about _accidental_ exposure.

Remember: the security is in the writing. If you allow "bad people" enough
capabilities that they can create their own git archive and can read the
target archive, those "bad people" could just export the target archive
some other way in the first place (ie they could have just copied the
files over to their own area).

And there are actually real downsides to requiring "git-daemon-export-ok" from a security standpoint. In particular, imagine that a company has a "master archive", and wants to export just a particular "public branch" from that master archive. The way you can do that right now is to create a dummy git archive, that is empty except for having one head (symlink to the public branch head in the master) and an "alternates" pointer to the master.

See? You don't actually want to expose the master archive itself: so requiring that one to also have "git-daemon-export-ok" would actually _defeat_ the security in the system.

So the git approach to security is that you secure the writing side. That's where you use ssh. And even if you happen to run git-daemon, it will never export anything that you didn't explicitly mark for export, so it defaults to a "nothing exported" mode. But once you mark a project for public export, the branches exposed there really are public.

(And the branches _not_ exposed there are private. Sure, if you can guess the SHA1 ID's, you can make git-daemon export them, but the point is that git-daemon will never expose any SHA1's from other projects unless they have the "git-daemon-export-ok" flag set. And the thing is, if you know the SHA1's, you already know the contents and you had a leak some other way, so..).

			Linus
Junio C Hamano· Sep 11, 2005, 20:08 UTC · re: Linus Torvalds · lore
Linus Torvalds <torvalds@osdl.org> writes:
Show 27 quoted lines
> On Sun, 11 Sep 2005, Sam Ravnborg wrote:
>> > 
>> > Absolutely. The mirroring has been slow again lately. I've packed my 
>> > archive, but I suspect others should much more aggressively now be using 
>> > the "objects/info/alternates" information to point to my tree, so that 
>> > they don't even need to have their objects at all (no packing 
>> > even necessary - just running "git prune-packed" on peoples archives 
>> > would get rid of any duplicate objects when I pack mine).
>> 
>> Can you post a small description how to utilize this method?
>
> Just do
>
> 	echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects > objects/info/alternates
>
> in your tree, and that will tell git that your tree can use my object 
> directory as an "alternate" source of objects. At that point, you can 
> remove all objects that I have.
>
> However, that only works with a local directory - you can't say that the
> alternate object directory is over the network (unless you use NFS or
> similar, of course ;).
>
> Another potential problem is that while the above makes git understand to
> pick the objects from my directory, it can in theory cause problems for
> mirrors etc - since they mirror things to a different location and/or may
> not mirror all of it anyway.
And probably it would break fetching over dumb transports.

← back to recent threads