{"thread":{"id":"3906","subject":"GIT_OBJECT_DIRECTORY","startedAt":"2006-04-18T13:38:47Z","lastAt":"2006-04-19T05:00:59Z","messageCount":12,"participants":["Jörn Engel","Linus Torvalds","Sam Ravnborg","Junio C Hamano","H. Peter Anvin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"18803","messageId":"20060418133847.GC4720@wohnheim.fh-wedel.de","threadId":"3906","inReplyTo":null,"subject":"GIT_OBJECT_DIRECTORY","fromName":"Jörn Engel","fromEmail":"joern@wohnheim.fh-wedel.de","sentAt":"2006-04-18T13:38:47Z","receivedAt":"2006-04-18T13:38:47Z","isPatch":false,"sender":{"key":"joern@wohnheim.fh-wedel.de","avatar":null},"body":"Hi!\n\nI recently noticed GIT_OBJECT_DIRECTORY in the git manpage and wanted\nto play with it. But it looks as if it doesn't work, the\ndocumentation is wrong/insufficient or I can't properly read the\ndocumentation. So let me figure out, which one it is.\n\n$ set | grep GIT_OBJECT_DIRECTORY\nGIT_OBJECT_DIRECTORY=/home/joern/.git\n\n$ ls -l /home/joern/.git\ntotal 288\ndrwxrwxr-x 2 joern joern 4096 Apr 16 01:22 0f\n[...]\n\n$ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git foo\n[ stored >200M of data under foo/.git/objects ]\n\n\nThe above looks as if new objects are not stored under\n/home/joern/.git, as specified by GIT_OBJECT_DIRECTORY. The manpage\ntells me:\n\n GIT_OBJECT_DIRECTORY\n If the object storage directory is specified via this environ-\n ment variable then the sha1 directories are created underneath -\n otherwise the default $GIT_DIR/objects directory is used.\n\nAnd I would interpret this as \"store all new objects under\n/home/joern/.git\". So far, things don't seem to imply me being too\nstupid. What went wrong?\n\nJörn\n\n-- \nWhy do musicians compose symphonies and poets write poems?\nThey do it because life wouldn't have any meaning for them if they didn't.\nThat's why I draw cartoons. It's my life.\n-- Charles Shultz\n"},{"id":"18807","messageId":"Pine.LNX.4.64.0604180822470.3701@g5.osdl.org","threadId":"3906","inReplyTo":"20060418133847.GC4720@wohnheim.fh-wedel.de","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-04-18T15:25:47Z","receivedAt":"2006-04-18T15:25:47Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Apr 2006, Jörn Engel wrote:\n> \n> $ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git foo\n> [ stored >200M of data under foo/.git/objects ]\n> \n> The above looks as if new objects are not stored under\n> /home/joern/.git, as specified by GIT_OBJECT_DIRECTORY.\n\nThe \"rsync\" protocol really doesn't honor git rules. It's basically just a \nbig recursive copy, and it will copy things from the place they were \nbefore.\n\nI suspect that if you had used a real git-aware protocol instead, you'd \nhave been fine, ie\n\n\tgit clone git://git.kernel.org/... foo/\n\nwould probably work. (I say \"probably\", because very few people likely use \nGIT_OBJECT_DIRECTORY, and it makes a lot less sense with pack-files than \nit did originally, so it's not getting any testing).\n\n\t\tLinus"},{"id":"18811","messageId":"20060418175853.GA25688@wohnheim.fh-wedel.de","threadId":"3906","inReplyTo":"Pine.LNX.4.64.0604180822470.3701@g5.osdl.org","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Jörn Engel","fromEmail":"joern@wohnheim.fh-wedel.de","sentAt":"2006-04-18T17:58:53Z","receivedAt":"2006-04-18T17:58:53Z","isPatch":false,"sender":{"key":"joern@wohnheim.fh-wedel.de","avatar":null},"body":"On Tue, 18 April 2006 08:25:47 -0700, Linus Torvalds wrote:\n> On Tue, 18 Apr 2006, Jörn Engel wrote:\n> > \n> > $ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git foo\n> > [ stored >200M of data under foo/.git/objects ]\n> > \n> > The above looks as if new objects are not stored under\n> > /home/joern/.git, as specified by GIT_OBJECT_DIRECTORY.\n> \n> The \"rsync\" protocol really doesn't honor git rules. It's basically just a \n> big recursive copy, and it will copy things from the place they were \n> before.\n> \n> I suspect that if you had used a real git-aware protocol instead, you'd \n> have been fine, ie\n> \n> \tgit clone git://git.kernel.org/... foo/\n\nIs it possible for non-owners of a kernel.org account to do this?\n\n> would probably work. (I say \"probably\", because very few people likely use \n> GIT_OBJECT_DIRECTORY, and it makes a lot less sense with pack-files than \n> it did originally, so it's not getting any testing).\n\nWell, .git/objects for your kernel still consumes 121M. It's not\ngigabytes but I still wouldn't want too many copies of that lying\naround. Right now, I already feel slightly motivated to move the\nwhole content-addressable idea into the kernel. It has disadvantages,\nbut the effect on disk- and pagecache-footprint for people like me\nwould come in handy.\n\nJörn\n\n-- \nThe cheapest, fastest and most reliable components of a computer\nsystem are those that aren't there.\n-- Gordon Bell, DEC labratories\n"},{"id":"18813","messageId":"20060418180707.GA737@mars.ravnborg.org","threadId":"3906","inReplyTo":"20060418175853.GA25688@wohnheim.fh-wedel.de","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Sam Ravnborg","fromEmail":"sam@ravnborg.org","sentAt":"2006-04-18T18:07:07Z","receivedAt":"2006-04-18T18:07:07Z","isPatch":false,"sender":{"key":"sam@ravnborg.org","avatar":"https://gravatar.com/avatar/168a912606ed0742d840bb365e3cc21db390c36531a58341dc7a069cc1f15f62?d=mp&s=160"},"body":"On Tue, Apr 18, 2006 at 07:58:53PM +0200, J?rn Engel wrote:\n > \n> > \tgit clone git://git.kernel.org/... foo/\n> \n> Is it possible for non-owners of a kernel.org account to do this?\nYes - everyone can do so.\nI never use rsync myself anymore.\n\n\tSam\n"},{"id":"18814","messageId":"Pine.LNX.4.64.0604181104420.3701@g5.osdl.org","threadId":"3906","inReplyTo":"20060418175853.GA25688@wohnheim.fh-wedel.de","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-04-18T18:08:40Z","receivedAt":"2006-04-18T18:08:40Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Apr 2006, Jörn Engel wrote:\n> > \n> > \tgit clone git://git.kernel.org/... foo/\n> \n> Is it possible for non-owners of a kernel.org account to do this?\n\nYes, kernel.org runs the git daemon.\n\nIf a repo isn't packed enough, the git protocol can be pretty CPU \nintensive, but I'm hoping that everybody keeps their repos mostly packed, \nat which point the git protocol should actually be a lot faster than \nrsync.\n\n> > GIT_OBJECT_DIRECTORY, and it makes a lot less sense with pack-files than \n> > it did originally, so it's not getting any testing).\n> \n> Well, .git/objects for your kernel still consumes 121M. It's not\n> gigabytes but I still wouldn't want too many copies of that lying\n> around.\n\nRight. However, these days we have better approaches than \nGIT_OBJECT_DIRECTORY for that.\n\nIn particular, if you create local clones, use \"git clone -l -s\", which \nshares its base objects with the thing you clone from. It makes the clone \nincredibly fast too (the only real cost is the check-out, which can \nobviously be pretty expensive), and you can then use\n\n\tgit repack -a -d -l\n\non all the to repack just the _local_ objects to avoid having packs \nduplicate objects unnecessarily.\n\n\t\tLinus"},{"id":"18816","messageId":"7vfykag2yd.fsf@assigned-by-dhcp.cox.net","threadId":"3906","inReplyTo":"20060418175853.GA25688@wohnheim.fh-wedel.de","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-04-18T18:20:58Z","receivedAt":"2006-04-18T18:20:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jörn Engel writes:\n\n> Well, .git/objects for your kernel still consumes 121M. It's not\n> gigabytes but I still wouldn't want too many copies of that lying\n> around.\n\nThat is what \"git clone -l -s\" is for. \n\nThe alternates pointer mechanism used with the above largely\nmakes GIT_OBJECT_DIRECTORY unnecessary for end users these days.\nIt is a fine mechanism as an implementation detail of the\nlowlevel and Porcelains, and that is the reason the\ndocumentation still mentions the environment.\n"},{"id":"18817","messageId":"20060418182650.GB25688@wohnheim.fh-wedel.de","threadId":"3906","inReplyTo":"Pine.LNX.4.64.0604181104420.3701@g5.osdl.org","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Jörn Engel","fromEmail":"joern@wohnheim.fh-wedel.de","sentAt":"2006-04-18T18:26:50Z","receivedAt":"2006-04-18T18:26:50Z","isPatch":false,"sender":{"key":"joern@wohnheim.fh-wedel.de","avatar":null},"body":"On Tue, 18 April 2006 11:08:40 -0700, Linus Torvalds wrote:\n> On Tue, 18 Apr 2006, Jörn Engel wrote:\n> > > \n> > > \tgit clone git://git.kernel.org/... foo/\n> > \n> > Is it possible for non-owners of a kernel.org account to do this?\n> \n> Yes, kernel.org runs the git daemon.\n\nExcellent! I have a faint memory of hpa recently saying that the git\ndaemon would be too resource-hungry. One of the cases where being\nwrong is a Good Thing.\n\n> > \n> > Well, .git/objects for your kernel still consumes 121M. It's not\n> > gigabytes but I still wouldn't want too many copies of that lying\n> > around.\n> \n> Right. However, these days we have better approaches than \n> GIT_OBJECT_DIRECTORY for that.\n> \n> In particular, if you create local clones, use \"git clone -l -s\", which \n> shares its base objects with the thing you clone from. It makes the clone \n> incredibly fast too (the only real cost is the check-out, which can \n> obviously be pretty expensive), and you can then use\n> \n> \tgit repack -a -d -l\n> \n> on all the to repack just the _local_ objects to avoid having packs \n> duplicate objects unnecessarily.\n\nThis still isn't good enough for me. Before git, all my trees were\nhard-linked (cowlinked, actually) and another copy barely consumed any\nspace. \"git clone -l -s\" creates a copy of the currently 311M of\nkernel source, quite a bit more expensive.\n\nBut it appears as if I could \"cp -lr\" the git tree and work with that.\nThe nice thing of having cowlinks is that I don't have to rely on git\nbreaking the hard links - which it probably won't. But since the\nestimated user base of cowlinks is 1, that won't help too many people.\n\nJörn\n\n-- \nGood warriors cause others to come to them and do not go to others.\n-- Sun Tzu\n"},{"id":"18820","messageId":"20060418184508.GC25688@wohnheim.fh-wedel.de","threadId":"3906","inReplyTo":"7vfykag2yd.fsf@assigned-by-dhcp.cox.net","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Jörn Engel","fromEmail":"joern@wohnheim.fh-wedel.de","sentAt":"2006-04-18T18:45:08Z","receivedAt":"2006-04-18T18:45:08Z","isPatch":false,"sender":{"key":"joern@wohnheim.fh-wedel.de","avatar":null},"body":"On Tue, 18 April 2006 11:20:58 -0700, Junio C Hamano wrote:\n> Jörn Engel writes:\n> \n> > Well, .git/objects for your kernel still consumes 121M. It's not\n> > gigabytes but I still wouldn't want too many copies of that lying\n> > around.\n> \n> That is what \"git clone -l -s\" is for. \n\nSee my response to Linus. .git/objects is currently the smaller\nproblem. The larger problem is 311M of raw kernel source - without\nany SCM overhead of any flavour. Like many others, I solved the\nlarger problem with hardlink trees. \"git clone -l -s\" is imo nearly\nunethical, as it solved the smaller problem and leaves the larger one\nunaffected. It reeks of hypocricy.\n\nHardlink trees still aren't perfect. If I take one tree, \"cp -lr\" it\ntwice and apply the same patches to both copies, the changed files\nexist twice for both copies. That sucks, but it is a fairly small\nproblem and there is no simple solution to it.\n\nIf git was able to deal with hardlink trees and properly break the\nlinks when working in one copy, \"cp -lr\" would be a lot smarter than\n\"git clone -l -s\". It just happens that I have written some kernel\npatches that automatically break hardlinks, even if applications don't\nknow how to do it. So for my personal use, git has this ability.\n\nNow, going one step further, GIT_OBJECT_DIRECTORY could solve another\nproblem. Just like source files, git object can be pulled twice into\ntwo copies of a tree. But for git objects, there appeared to be an\neasy solution: the central object storage. So we're back where this\nthread started.\n\nExcept that I get the idea of GIT_OBJECT_DIRECTORY not being a simple\nsolution to a small problem, so maybe I should just forget about it.\n\nJörn\n\n-- \nTo my face you have the audacity to advise me to become a thief - the worst\nkind of thief that is conceivable, a thief of spiritual things, a thief of\nideas! It is insufferable, intolerable!\n-- M. Binet in Scarabouche\n"},{"id":"18821","messageId":"Pine.LNX.4.64.0604181139150.3701@g5.osdl.org","threadId":"3906","inReplyTo":"20060418182650.GB25688@wohnheim.fh-wedel.de","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-04-18T18:47:53Z","receivedAt":"2006-04-18T18:47:53Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Apr 2006, Jörn Engel wrote:\n> \n> But it appears as if I could \"cp -lr\" the git tree and work with that.\n\nThat should work. I just personally fear cowlinks, because some things \nwill edit the files in place, and then you're screwed.\n\nI _think_ it should be ok for the .git subdirectory, but quite frankly, \nI'm not going to guarantee it. Also, you will break the cow-linking when \nyou ever re-pack either the source or the destination, so you'd actually \nbe _better_ off with something that does\n\n\t# clone the git directories by hand, no checkout (-n).\n\tgit clone -l -s -n src dst\n\n\t# cow-link the checked-out state\n\t(cd src ; git ls-files | cpio -pudml dst)\n\n\t# make sure to refresh the index\n\tgit update-index --refresh\n\nor something like that.\n\nTOTALLY UNTESTED!! (And you need to have made \"dst\" be an absolute path, \nof course, since we want it to work even after we've done the \"cd src\" \nthing).\n\n\t\tLinus"},{"id":"18822","messageId":"20060418185859.GA7504@wohnheim.fh-wedel.de","threadId":"3906","inReplyTo":"Pine.LNX.4.64.0604181139150.3701@g5.osdl.org","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Jörn Engel","fromEmail":"joern@wohnheim.fh-wedel.de","sentAt":"2006-04-18T18:58:59Z","receivedAt":"2006-04-18T18:58:59Z","isPatch":false,"sender":{"key":"joern@wohnheim.fh-wedel.de","avatar":null},"body":"On Tue, 18 April 2006 11:47:53 -0700, Linus Torvalds wrote:\n> On Tue, 18 Apr 2006, Jörn Engel wrote:\n> > \n> > But it appears as if I could \"cp -lr\" the git tree and work with that.\n> \n> That should work. I just personally fear cowlinks, because some things \n> will edit the files in place, and then you're screwed.\n\ns/cowlinks/hardlinks/ ?\n\nThe reason for me to write the cowlink patches was exactly the fear\nyou are talking about. With those patches, links are broken whenever\nsuch a thing happens.\n\n> I _think_ it should be ok for the .git subdirectory, but quite frankly, \n> I'm not going to guarantee it. Also, you will break the cow-linking when \n> you ever re-pack either the source or the destination, so you'd actually \n\nIn that case, cowlinks should still turn a blatant bug into some\nwasted space - which is a hell of a lot better.\n\n> \t# cow-link the checked-out state\n\nAnd this happens to be a problem. Creating the links when the copy is\ncreated is simple. Detecting identical files and linking them after\nthe fact is racy, complicated, racy and, well, racy. I wouldn't want\nto touch it with a ten foot pole. Not without kernel support.\n\nJörn\n\n-- \nLinux is more the core point of a concept that surrounds \"open source\"\nwhich, in turn, is based on a false concept. This concept is that\npeople actually want to look at source code.\n-- Rob Enderle\n"},{"id":"18858","messageId":"4445C1D6.3070504@zytor.com","threadId":"3906","inReplyTo":"20060418182650.GB25688@wohnheim.fh-wedel.de","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2006-04-19T04:51:34Z","receivedAt":"2006-04-19T04:51:34Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Jörn Engel wrote:\n> On Tue, 18 April 2006 11:08:40 -0700, Linus Torvalds wrote:\n>> On Tue, 18 Apr 2006, Jörn Engel wrote:\n>>>> \tgit clone git://git.kernel.org/... foo/\n>>> Is it possible for non-owners of a kernel.org account to do this?\n>> Yes, kernel.org runs the git daemon.\n> \n> Excellent! I have a faint memory of hpa recently saying that the git\n> daemon would be too resource-hungry. One of the cases where being\n> wrong is a Good Thing.\n> \n\nWell, we ended up making some tweaks to the git daemon, and it hasn't \nbeen a problem since.\n\n\t-hpa\n"},{"id":"18859","messageId":"7vodyyb1mc.fsf@assigned-by-dhcp.cox.net","threadId":"3906","inReplyTo":"4445C1D6.3070504@zytor.com","subject":"Re: GIT_OBJECT_DIRECTORY","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-04-19T05:00:59Z","receivedAt":"2006-04-19T05:00:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"H. Peter Anvin\" writes:\n\n> Jörn Engel wrote:\n>>\n>> Excellent! I have a faint memory of hpa recently saying that the git\n>> daemon would be too resource-hungry. One of the cases where being\n>> wrong is a Good Thing.\n>\n> Well, we ended up making some tweaks to the git daemon, and it hasn't\n> been a problem since.\n\nAh, I am glad the daemon expert was listening... Do you have\ncomments on recent patch from Serge E. Hallyn? It looks OK to\nme, but that standalone daemon part is not something I run\nmyself, so...\n\n-- >8 --\n[PATCH] socksetup: don't return on set_reuse_addr() error\n\nThe set_reuse_addr() error case was the only error case in\nsocklist() where we returned rather than continued. Not sure\nwhy. Either we must free the socklist, or continue. This patch\ncontinues on error.\n\nSigned-off-by: Serge E. Hallyn \nSigned-off-by: Junio C Hamano \n\n---\n\n daemon.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\n0032d548db56eac9ea09b4ba05843365f6325b85\ndiff --git a/daemon.c b/daemon.c\nindex a1ccda3..776749e 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -535,7 +535,7 @@ #endif\n \n \t\tif (set_reuse_addr(sockfd)) {\n \t\t\tclose(sockfd);\n-\t\t\treturn 0;\t/* not fatal */\n+\t\t\tcontinue;\n \t\t}\n \n \t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n-- \n1.3.0.rc4.g5247-dirty\n"}]}