# Re: What's up with the GIT archive on www.kernel.org? 18 messages from 2005-09-11 to 2005-09-13. Participants: Linus Torvalds, Sam Ravnborg, Junio C Hamano, Roland Dreier, Dmitry Torokhov, H. Peter Anvin, Tony Luck, Ryan Anderson, Daniel Barkalow. Thread: https://gitlist.dev/t/1773 ## Linus Torvalds, 2005-09-11 19:06 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509111157360.3242%40g5.osdl.org In-Reply-To: <20050911185711.GA22556@mars.ravnborg.org> ``` On Sun, 11 Sep 2005, Sam Ravnborg wrote: > > > > Absolutely. The mirroring has been slow again lately. I've packed my > > archive, but I suspect others should much more aggressively now be using > > the "objects/info/alternates" information to point to my tree, so that > > they don't even need to have their objects at all (no packing > > even necessary - just running "git prune-packed" on peoples archives > > would get rid of any duplicate objects when I pack mine). > > Can you post a small description how to utilize this method? Just do echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects > objects/info/alternates in your tree, and that will tell git that your tree can use my object directory as an "alternate" source of objects. At that point, you can remove all objects that I have. However, that only works with a local directory - you can't say that the alternate object directory is over the network (unless you use NFS or similar, of course ;). Another potential problem is that while the above makes git understand to pick the objects from my directory, it can in theory cause problems for mirrors etc - since they mirror things to a different location and/or may not mirror all of it anyway. Anyway, modulo those caveats, you can then just do git prune-packed and it will remove all unpacked objects in your git archive that can be reached through a pack-file - including any packfiles in _my_ directory. Then you never need to pack your own objects any more. Just leave everything unpacked, and rely on me packing every once in a while, and just do "git prune-packed" when I do. That allows a site like kernel.org to effectively share 99% of all objects, and do it efficiently. Linus ``` ## Sam Ravnborg, 2005-09-11 19:46 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <20050911194630.GB22951@mars.ravnborg.org> URL: https://gitlist.dev/e/20050911194630.GB22951%40mars.ravnborg.org In-Reply-To: ``` > > > > Can you post a small description how to utilize this method? > > Just do > > echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects > objects/info/alternates > > in your tree, and that will tell git that your tree can use my object > directory as an "alternate" source of objects. At that point, you can > remove all objects that I have. OK - what I did: cd /pub/scm/linux/kernel/git/sam rm -rf kbuild.git git clone /pub/scm/linux/kernel/git/torvalds/linux-2.6.git kbuild.git rename to .git to kbuild.git I had to specify both GIT_DIR and GIT_OBJECT_DIRECTORY to make git-prune-packed behave as expected. I assume this is normal when I rename the .git directory like in this case. I will se if any pullers complins (mostly/only Andrew I think). Sam ``` ## Linus Torvalds, 2005-09-11 19:56 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509111251150.3242%40g5.osdl.org In-Reply-To: <20050911194630.GB22951@mars.ravnborg.org> ``` On Sun, 11 Sep 2005, Sam Ravnborg wrote: > > I had to specify both GIT_DIR and GIT_OBJECT_DIRECTORY to make > git-prune-packed behave as expected. I assume this is normal when I > rename the .git directory like in this case. You should only need to specify GIT_DIR - it should figure out that the object directory follows GIT_DIR on its own. Also, I forget what version of git is installed on kernel.org. The "alternates" support has been around for a while, and looking at the date of "/usr/bin/git" it _seems_ recent (Sep 7), but I haven't seen any announcement of updating since the last one (which was git-0.99.4, which is too old). You can try removing all the packs in your .git/objects/packs directory. Everything _should_ still work fine. Famous last words. Linus ``` ## Junio C Hamano, 2005-09-11 20:08 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <7v64t7s6zc.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v64t7s6zc.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > On Sun, 11 Sep 2005, Sam Ravnborg wrote: >> > >> > Absolutely. The mirroring has been slow again lately. I've packed my >> > archive, but I suspect others should much more aggressively now be using >> > the "objects/info/alternates" information to point to my tree, so that >> > they don't even need to have their objects at all (no packing >> > even necessary - just running "git prune-packed" on peoples archives >> > would get rid of any duplicate objects when I pack mine). >> >> Can you post a small description how to utilize this method? > > Just do > > echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects > objects/info/alternates > > in your tree, and that will tell git that your tree can use my object > directory as an "alternate" source of objects. At that point, you can > remove all objects that I have. > > However, that only works with a local directory - you can't say that the > alternate object directory is over the network (unless you use NFS or > similar, of course ;). > > Another potential problem is that while the above makes git understand to > pick the objects from my directory, it can in theory cause problems for > mirrors etc - since they mirror things to a different location and/or may > not mirror all of it anyway. And probably it would break fetching over dumb transports. ``` ## Roland Dreier, 2005-09-11 21:09 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <52irx7cnw5.fsf@cisco.com> URL: https://gitlist.dev/e/52irx7cnw5.fsf%40cisco.com In-Reply-To: ``` Linus> You can try removing all the packs in your Linus> .git/objects/packs directory. Everything _should_ still Linus> work fine. Does "everything" include someone doing git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/whatever.git How about http:// instead of rsync://? In other words, is the git network transport smart enough to handle the alternates path? Or is the idea that everyone will clone your tree and then pull extra stuff from other trees? - R. ``` ## Linus Torvalds, 2005-09-11 21:24 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509111422510.3242%40g5.osdl.org In-Reply-To: <52irx7cnw5.fsf@cisco.com> ``` On Sun, 11 Sep 2005, Roland Dreier wrote: > > Does "everything" include someone doing > > git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/whatever.git Nope. Only server-side smart protocols will handle this. There is such an anonymous server, btw: "git-daemon" implements anonymous access much more efficient than rsync/http. Sadly, kernel.org still doesn't offer it (but it's now used in the wild, ie I've done a couple of merges with people running the git daemon). > In other words, is the git network transport smart enough to handle > the alternates path? The _git_ network transport is. rsync and http aren't. Linus ``` ## Linus Torvalds, 2005-09-11 21:33 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509111431400.3242%40g5.osdl.org In-Reply-To: ``` On Sun, 11 Sep 2005, Linus Torvalds wrote: > > The _git_ network transport is. rsync and http aren't. Btw, there's no reason why a client-side thing couldn't just parse the "alternates" thing, and if it doesn't find the objects in the main object directory, go and fetch them from the alternates itself. IOW, this is not a fundamental problem with alternates, it's just that since there is no server-side smarts to handle it (ie just raw file access with rsync/http), it needs to be handled at the client side instead. Linus ``` ## Junio C Hamano, 2005-09-12 01:39 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <7virx7njxa.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7virx7njxa.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > Btw, there's no reason why a client-side thing couldn't just parse the > "alternates" thing, and if it doesn't find the objects in the main object > directory, go and fetch them from the alternates itself. There is. For kernel.org, you could say '/pub/scm/blah' in your alternates and expect it to work, only because http://kernel.org/pub hierarchy happens to match the absolute path /pub on the filesystem, but for most people's default HTTP server installation, they would need to say /var/www/scm/blah to have alternate work locally, but somebody has to know that the named directory is served as http://machine.xz/pub/scm/blah somewhere. Client side smarts need some help from the user here to know that '/var/www/scm/blah' read off of objects/info/alternates match that URL. ``` ## Dmitry Torokhov, 2005-09-12 02:45 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <200509112145.33994.dtor_core@ameritech.net> URL: https://gitlist.dev/e/200509112145.33994.dtor_core%40ameritech.net In-Reply-To: <7virx7njxa.fsf@assigned-by-dhcp.cox.net> ``` On Sunday 11 September 2005 20:39, Junio C Hamano wrote: > Linus Torvalds writes: > > > Btw, there's no reason why a client-side thing couldn't just parse the > > "alternates" thing, and if it doesn't find the objects in the main object > > directory, go and fetch them from the alternates itself. > > There is. > > For kernel.org, you could say '/pub/scm/blah' in your alternates > and expect it to work, only because http://kernel.org/pub > hierarchy happens to match the absolute path /pub on the > filesystem, but for most people's default HTTP server > installation, they would need to say /var/www/scm/blah to have > alternate work locally, but somebody has to know that the named > directory is served as http://machine.xz/pub/scm/blah somewhere. > Call me brain-dead but all of this just makes me rsync my tree to kernel.org and then manually do "ln -f" for all the packs that Linus has. This way I am sure tht the tree is what I have plus and it is "pullable". -- Dmitry ``` ## Linus Torvalds, 2005-09-12 03:39 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509112038020.3242%40g5.osdl.org In-Reply-To: <7virx7njxa.fsf@assigned-by-dhcp.cox.net> ``` On Sun, 11 Sep 2005, Junio C Hamano wrote: > > For kernel.org, you could say '/pub/scm/blah' in your alternates > and expect it to work, only because http://kernel.org/pub > hierarchy happens to match the absolute path /pub on the > filesystem, but for most people's default HTTP server > installation, they would need to say /var/www/scm/blah to have > alternate work locally, but somebody has to know that the named > directory is served as http://machine.xz/pub/scm/blah somewhere. Yes. We should probably have some well-defined meaning for relative paths in there regardless (eg just define that they are always relative to the main GIT_OBJECT_DIRECTORY or something). That would also allow mirrors to mirror the git archives in different places, without upsetting the result (as long as they are mirrored together). Linus ``` ## H. Peter Anvin, 2005-09-12 17:10 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <4325B698.1050805@zytor.com> URL: https://gitlist.dev/e/4325B698.1050805%40zytor.com In-Reply-To: <7virx7njxa.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano wrote: > > For kernel.org, you could say '/pub/scm/blah' in your alternates > and expect it to work, only because http://kernel.org/pub > hierarchy happens to match the absolute path /pub on the > filesystem... > Actually it doesn't. /pub in the root directory on kernel.org is just a convenience symlink. -hpa ``` ## Tony Luck, 2005-09-12 18:22 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <12c511ca050912112266470d8b@mail.gmail.com> URL: https://gitlist.dev/e/12c511ca050912112266470d8b%40mail.gmail.com In-Reply-To: ``` On 9/11/05, Linus Torvalds wrote: > There is such an anonymous server, btw: "git-daemon" implements anonymous > access much more efficient than rsync/http. Sadly, kernel.org still > doesn't offer it (but it's now used in the wild, ie I've done a couple of > merges with people running the git daemon). Should the git daemon take a look at objects/info/alternates to check that if it exists, it points to a repository that also has a "git-daemon-export-ok" file? I don't see that this could be used for anything nasty, but it does provide a loophole where the daemon may open files outside the initial repository ... so a sanity check seems in order. -Tony ``` ## Linus Torvalds, 2005-09-12 18:37 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509121128170.3242%40g5.osdl.org In-Reply-To: <12c511ca050912112266470d8b@mail.gmail.com> ``` On Mon, 12 Sep 2005, Tony Luck wrote: > > Should the git daemon take a look at objects/info/alternates to check > that if it exists, it points to a repository that also has a > "git-daemon-export-ok" file? I considered it, but decided against the complexity. I just don't see the point. The "git-daemon-export-ok" is not so much about security as about _accidental_ exposure. Remember: the security is in the writing. If you allow "bad people" enough capabilities that they can create their own git archive and can read the target archive, those "bad people" could just export the target archive some other way in the first place (ie they could have just copied the files over to their own area). And there are actually real downsides to requiring "git-daemon-export-ok" from a security standpoint. In particular, imagine that a company has a "master archive", and wants to export just a particular "public branch" from that master archive. The way you can do that right now is to create a dummy git archive, that is empty except for having one head (symlink to the public branch head in the master) and an "alternates" pointer to the master. See? You don't actually want to expose the master archive itself: so requiring that one to also have "git-daemon-export-ok" would actually _defeat_ the security in the system. So the git approach to security is that you secure the writing side. That's where you use ssh. And even if you happen to run git-daemon, it will never export anything that you didn't explicitly mark for export, so it defaults to a "nothing exported" mode. But once you mark a project for public export, the branches exposed there really are public. (And the branches _not_ exposed there are private. Sure, if you can guess the SHA1 ID's, you can make git-daemon export them, but the point is that git-daemon will never expose any SHA1's from other projects unless they have the "git-daemon-export-ok" flag set. And the thing is, if you know the SHA1's, you already know the contents and you had a leak some other way, so..). Linus ``` ## Ryan Anderson, 2005-09-12 18:42 Subject: Re: What's up with the GIT archive on www.kernel.org? Message-ID: <20050912184214.GB5276@mythryan2.michonline.com> URL: https://gitlist.dev/e/20050912184214.GB5276%40mythryan2.michonline.com In-Reply-To: <200509112145.33994.dtor_core@ameritech.net> ``` On Sun, Sep 11, 2005 at 09:45:33PM -0500, Dmitry Torokhov wrote: > > Call me brain-dead but all of this just makes me rsync my tree to > kernel.org and then manually do "ln -f" for all the packs that Linus > has. This way I am sure tht the tree is what I have plus and it is > "pullable". If you have access to make hardlinks, you should be able to use git-relink to do the hard work for you. >From memory: git relink my_dir1 my_dir2 ... master_dir or: git relink my-kernel-tree /pub/scm/.../torvalds/linux.git/ (I think that will work - via a bug in my initial attempt to write git-relink, I look to make sure the path ends in ".git/" not "/.git/". So the above should work. I think.) -- Ryan Anderson sometimes Pug Majere ``` ## Junio C Hamano, 2005-09-13 07:05 Subject: [PATCH] Define relative .git/objects/info/alternates semantics. Message-ID: <7vk6hl4fd9.fsf_-_@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vk6hl4fd9.fsf_-_%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > Yes. We should probably have some well-defined meaning for relative paths > in there regardless (eg just define that they are always relative to the > main GIT_OBJECT_DIRECTORY or something). > > That would also allow mirrors to mirror the git archives in different > places, without upsetting the result (as long as they are mirrored > together). > > Linus This patch is request-for-comments. I have experimented it and have a feeling that it may be more intuitive to make it relative to $project.git/ directory, instead of $project.git/objects as you originally suggested, in which case a maintainer tree would have "../../torvalds/linux-2.6.git/objects" instead (one less dotdot), and if nobody objects that is probably what I'll end up doing. ------------ An entry in the alternates file can name a directory relative to the object store it describes. A typical linux-2.6 maintainer repository would have "../../../torvalds/linux-2.6.git/objects" there, because the subsystem maintainer object store would live in /pub/scm/linux/kernel/git/$u/$system.git/objects/ and the object store of Linus tree is in /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/ This unfortunately is different from GIT_ALTERNATE_OBJECT_DIRECTORIES which is relative to the cwd of the running process, but there is no way to make it consistent with the behaviour of the environment variable. The process typically is run in $system.git/ directory for a naked repository, or one level up for a repository with a working tree, so we just define it to be relative to the objects/ directory to be different from either ;-). Later, the dumb transport could be updated to read from info/alternates and make requests for the repository the repository borrows from. Signed-off-by: Junio C Hamano --- sha1_file.c | 28 ++++++++++++++++++++++------ 1 files changed, 22 insertions(+), 6 deletions(-) e2e8a0ba5fc80368bf46c615276e406dd373729c diff --git a/sha1_file.c b/sha1_file.c --- a/sha1_file.c +++ b/sha1_file.c @@ -240,10 +240,12 @@ static struct alternate_object_database * SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -static void link_alt_odb_entries(const char *alt, const char *ep, int sep) +static void link_alt_odb_entries(const char *alt, const char *ep, int sep, + const char *relative_base) { const char *cp, *last; struct alternate_object_database *ent; + int base_len = -1; last = alt; while (last < ep) { @@ -261,12 +263,25 @@ static void link_alt_odb_entries(const c int pfxlen = cp - last; int entlen = pfxlen + 43; + if (*last != '/' && relative_base) { + /* Relative alt-odb */ + if (base_len < 0) + base_len = strlen(relative_base) + 1; + entlen += base_len; + pfxlen += base_len; + } ent = xmalloc(sizeof(*ent) + entlen); *alt_odb_tail = ent; alt_odb_tail = &(ent->next); ent->next = NULL; - - memcpy(ent->base, last, pfxlen); + if (*last != '/' && relative_base) { + memcpy(ent->base, relative_base, base_len - 1); + ent->base[base_len - 1] = '/'; + memcpy(ent->base + base_len, + last, cp - last); + } + else + memcpy(ent->base, last, pfxlen); ent->name = ent->base + pfxlen + 1; ent->base[pfxlen] = ent->base[pfxlen + 3] = '/'; ent->base[entlen-1] = 0; @@ -288,12 +303,12 @@ void prepare_alt_odb(void) alt = getenv(ALTERNATE_DB_ENVIRONMENT); if (!alt) alt = ""; - sprintf(path, "%s/info/alternates", get_object_directory()); if (alt_odb_tail) return; alt_odb_tail = &alt_odb_list; - link_alt_odb_entries(alt, alt + strlen(alt), ':'); + link_alt_odb_entries(alt, alt + strlen(alt), ':', NULL); + sprintf(path, "%s/info/alternates", get_object_directory()); fd = open(path, O_RDONLY); if (fd < 0) return; @@ -306,7 +321,8 @@ void prepare_alt_odb(void) if (map == MAP_FAILED) return; - link_alt_odb_entries(map, map + st.st_size, '\n'); + link_alt_odb_entries(map, map + st.st_size, '\n', + get_object_directory()); munmap(map, st.st_size); } ``` ## Linus Torvalds, 2005-09-13 16:22 Subject: Re: [PATCH] Define relative .git/objects/info/alternates semantics. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0509130918170.3351%40g5.osdl.org In-Reply-To: <7vk6hl4fd9.fsf_-_@assigned-by-dhcp.cox.net> ``` On Tue, 13 Sep 2005, Junio C Hamano wrote: > > This patch is request-for-comments. I have experimented it and > have a feeling that it may be more intuitive to make it relative > to $project.git/ directory, instead of $project.git/objects as > you originally suggested I don't think you can do that. I always felt that "alternates" should be per-project, but you're the one who argued that "alternates" is a per-object-directory thing. Which means that the _same_ "alternates" file can be shared with many different project.git/ directories, and thus it's not well-defined to make it relative to GIT_DIR. You can make it relative to "GIT_OBJECTS_DIR/.." of course, which in most cases is the same thing as "GIT_DIR". Linus ``` ## Daniel Barkalow, 2005-09-13 16:31 Subject: Re: [PATCH] Define relative .git/objects/info/alternates semantics. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0509131224290.23242%40iabervon.org In-Reply-To: <7vk6hl4fd9.fsf_-_@assigned-by-dhcp.cox.net> ``` On Tue, 13 Sep 2005, Junio C Hamano wrote: > Linus Torvalds writes: > > > Yes. We should probably have some well-defined meaning for relative paths > > in there regardless (eg just define that they are always relative to the > > main GIT_OBJECT_DIRECTORY or something). > > > > That would also allow mirrors to mirror the git archives in different > > places, without upsetting the result (as long as they are mirrored > > together). > > > > Linus > > This patch is request-for-comments. I have experimented it and > have a feeling that it may be more intuitive to make it relative > to $project.git/ directory, instead of $project.git/objects as > you originally suggested, in which case a maintainer tree would > have "../../torvalds/linux-2.6.git/objects" instead (one less > dotdot), and if nobody objects that is probably what I'll end up > doing. It seems odd to have the "objects" at the end and not be starting from "objects". I suspect that the most intuitive thing would be for something under $project.git/info to be .git to .git, while something under .git/objects/info should be objects to objects. -Daniel *This .sig left intentionally blank* ``` ## Junio C Hamano, 2005-09-13 17:30 Subject: Re: [PATCH] Define relative .git/objects/info/alternates semantics. Message-ID: <7vvf143mfu.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vvf143mfu.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > Which means that the _same_ "alternates" file can be shared with many > different project.git/ directories, and thus it's not well-defined to make > it relative to GIT_DIR. Good point. > You can make it relative to "GIT_OBJECTS_DIR/.." of course, which in most > cases is the same thing as "GIT_DIR". Again, good point. ```