{"thread":{"id":"1511","subject":"symlinked directories in refs are now unreachable","startedAt":"2005-08-15T02:41:17Z","lastAt":"2005-08-18T21:56:36Z","messageCount":9,"participants":["Matt Draisey","Linus Torvalds","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"7257","messageId":"1124073677.27393.15.camel@della.draisey.ca","threadId":"1511","inReplyTo":null,"subject":"symlinked directories in refs are now unreachable","fromName":"Matt Draisey","fromEmail":"mattdraisey@sympatico.ca","sentAt":"2005-08-15T02:41:17Z","receivedAt":"2005-08-15T02:41:17Z","isPatch":false,"sender":{"key":"matt@draisey.ca","avatar":null},"body":"The behaviour of the symlinked in ref directories has changed from\nearlier versions of git.  They used to be taken into account in\ngit-fsck-cache --unreachable.  The code in question would simply stat\nthe contents of .git/refs and recursively expand any S_ISDIR.  Now the\ncode does an lstat and effectively throws symlinks away.\n\nrefs.c lines 48,52 @ do_for_each_ref\n\n...\n\tmemcpy(path + baselen, de->d_name, namelen+1);\n\tif (lstat(git_path(\"%s\", path), &st) < 0)\n\t\tcontinue;\n\tif (S_ISDIR(st.st_mode)) {\n\t\tretval = do_for_each_ref(path, fn);\n...\n\nCan the previous behaviour be reinstated?\n"},{"id":"7259","messageId":"Pine.LNX.4.58.0508142007440.3553@g5.osdl.org","threadId":"1511","inReplyTo":"1124073677.27393.15.camel@della.draisey.ca","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-08-15T03:12:22Z","receivedAt":"2005-08-15T03:12:22Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 15 Aug 2005, Matt Draisey wrote:\n>\n> The behaviour of the symlinked in ref directories has changed from\n> earlier versions of git.\n\nHmm.. There used to be a mix of lstat() (in receive-pack) and stat() (in \nfsck-cache.c, and it got standardized in one function which used lstat.\n\nThe reason for the lstat is really to try to avoid having especially the \nremote protocols follow symlinks, but I guess it's not a very good reason, \nso I don't think it would be wrong to just standardize refs.c to use \n\"stat()\" instead.\n\nYou might sent a patch to Junio..\n\nHOWEVER: symlinks for references really are pretty dangerous. We do things \nlike \"echo new-id > .git/HEAD\" and links (symlinks _or_ hardlinks) thus \nreally aren't safe. You're much better off copying those small files.\n\n\t\tLinus\n"},{"id":"7263","messageId":"7v64u7941c.fsf@assigned-by-dhcp.cox.net","threadId":"1511","inReplyTo":"1124073677.27393.15.camel@della.draisey.ca","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-15T05:12:47Z","receivedAt":"2005-08-15T05:12:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Draisey <mattdraisey@sympatico.ca> writes:\n\n> The behaviour of the symlinked in ref directories has changed from\n> earlier versions of git.  They used to be taken into account in\n> git-fsck-cache --unreachable.\n>\n> Can the previous behaviour be reinstated?\n\nI would not have much problem accepting a patch for that; it\nwould make things safer when a symlink points to a real file\nthat is outside .git/refs/ that holds a pointer to a valid\nobject.\n\nHaving said that, I would first like to know why you have a\nsymlink there, and the real file pointed by it outside .git/refs\nhierarchy.  The core GIT tools do not create such symlinks, so\neither you are creating one by hand, or your Porcelain is\ncreating one for you for whatever reason.  I would like to know\na use case or two to illustrate why there are symlinks pointing\nat real files outside .git/refs/ hierarchy, and how that\narrangement is useful.\n"},{"id":"7276","messageId":"1124089116.749.39.camel@della.draisey.ca","threadId":"1511","inReplyTo":"7v64u7941c.fsf@assigned-by-dhcp.cox.net","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Matt Draisey","fromEmail":"mattdraisey@sympatico.ca","sentAt":"2005-08-15T08:07:29Z","receivedAt":"2005-08-15T08:07:29Z","isPatch":false,"sender":{"key":"matt@draisey.ca","avatar":null},"body":"On Sun, 2005-08-14 at 22:12 -0700, Junio C Hamano wrote:\n> Matt Draisey <mattdraisey@sympatico.ca> writes:\n> \n> > The behaviour of the symlinked in ref directories has changed from\n> > earlier versions of git.  They used to be taken into account in\n> > git-fsck-cache --unreachable.\n> >\n> > Can the previous behaviour be reinstated?\n> \n> I would not have much problem accepting a patch for that; it\n> would make things safer when a symlink points to a real file\n> that is outside .git/refs/ that holds a pointer to a valid\n> object.\n> \n> Having said that, I would first like to know why you have a\n> symlink there, and the real file pointed by it outside .git/refs\n> hierarchy.  The core GIT tools do not create such symlinks, so\n> either you are creating one by hand, or your Porcelain is\n> creating one for you for whatever reason.\n\nIt is my own home-grown Porcelain that creates the symlinks.  I've\nthrown together a python programme to track a nested collection of\nprojects.  My own programming efforts rarely exceed two or three files\nper project, and don't justify there own .git/objects repository.\nStill, a few projects do benefit from having their own commit history,\nwhile the rest are tracked as one big outermost superproject of\nunrelated stuff.\n\n> I would like to know\n> a use case or two to illustrate why there are symlinks pointing\n> at real files outside .git/refs/ hierarchy, and how that\n> arrangement is useful.\n\nWhether or not its useful??  Hmmm.  Debatable.\n\nI've only written a commit tool.  All the other git and cogito tools I\ninvoke from the outermost directory like so \n\n$git-cat-file commit per/Minesweeper/master\n\nSymlinking still works here as expected.  The per directory is just\nthere so I don't stomp on the outermost namespace, the Minesweeper is a\nsymlink to the nested project's refs directory.  Symlinking seems the\nnatural way to do this as they only need updating when I move\nsubdirectories around.\n\nP.S. $echo new-id > .git/per/Minesweeper/master is safe here --- this is\nthe actual behaviour I want.\n"},{"id":"7286","messageId":"1124096475.749.64.camel@della.draisey.ca","threadId":"1511","inReplyTo":"7v1x4v4pfx.fsf@assigned-by-dhcp.cox.net","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Matt Draisey","fromEmail":"mattdraisey@sympatico.ca","sentAt":"2005-08-15T09:01:15Z","receivedAt":"2005-08-15T09:01:15Z","isPatch":false,"sender":{"key":"matt@draisey.ca","avatar":null},"body":"On Mon, 2005-08-15 at 00:41 -0700, Junio C Hamano wrote:\n> Matt Draisey <mattdraisey@sympatico.ca> writes:\n> \n> > ...  My own programming efforts rarely exceed two or three files\n> > per project, and don't justify there own .git/objects repository.\n> > Still, a few projects do benefit from having their own commit history,\n> \n> I am afraid I am not quite getting it.\n> \n> You are interested in many projects that have outside upstream,\n> and you typically modify only small portion of each of them,\n> which is quite a typical behaviour for individual developers.\n> For some reason you want to keep those repository \"clean\"\n> without your own commit objects or changed objects only\n> reachable from your commits.  Is it what is happening here?\n\nNo, all the projects are my own.  I am not a developer at all, merely a\nhobbyist.  Upstream projects don't fit into this scheme.\n\n> > I've only written a commit tool.  All the other git and cogito tools I\n> > invoke from the outermost directory like so \n> >\n> > $git-cat-file commit per/Minesweeper/master\n> >\n> > Symlinking still works here as expected.  The per directory is just\n> > there so I don't stomp on the outermost namespace, the Minesweeper is a\n> > symlink to the nested project's refs directory.\n> \n> Hmm.  So you have two GIT managed trees, $D/matt and $D/Minesweeper,\n> and a symlink between them like this.  Is that what is happening here?\n> \n>   $D/matt/.git/refs/heads/per/Minesweeper -> $D/Minesweeper/.git/refs/heads\n> \n\nNo, they are nested\n\n    $D/.git/refs/heads/per/Minesweeper -> $D/Minesweeper/.git/refs/heads\n\nThe outermost repository merely aggregates a bunch of small unrelated\nprojects that are not yet ready for an independent existence.  The idea\nis to put everything under revision control in the hope that eventually\nsomething useful falls out.\n\nMy commit tool walks up the chain towards root until it finds the\nobjects directory and does the appropriate thing.\n\n> Of course 'git-cat-file commit per/Minesweeper/master' would\n> work in \"$D/matt\" directory.  How do the set of paths recorded\n> in the index file used in these repositories relate to each\n> other?  Is $D/matt/ tracking the same set of files as the other\n> repository tracks?  Is it meant to be a superset?  Subset?  More\n> or less independent \"private additions\"?\n> \n> There must be some advantage to this arrangement than the more\n> typical arrangement I've seen people do, which is to have two\n> branches in Minesweeper (that is the upstream, right?)\n> repository, one \"origin\" and the other \"master\".  Upstream\n> changes you fetch and pull into \"origin\" branch while you commit\n> your changes to \"master\" branch.  I just do not yet see what\n> that advantage is, and I strongly suspect because I misread your\n> description and misunderstood the two repository arrangement you\n> have and how they are used.\n> \n> By the way, did you want to take this discussion private or was\n> it by accident you did not CC: the list?\n> \n\nNo, I didn't want to take it private.  I just don't know how my email\nprogramme works.  I also just discovered that Evolution's Forward As >\nRedirect is really a bounce and not a forward at all (it doesn't change\nthe to: address)\n"},{"id":"7350","messageId":"7vbr3yl7oz.fsf@assigned-by-dhcp.cox.net","threadId":"1511","inReplyTo":"1124171194.762.74.camel@della.draisey.ca","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-16T06:25:16Z","receivedAt":"2005-08-16T06:25:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Draisey <mattdraisey@sympatico.ca> writes:\n\n> On Sun, 2005-08-14 at 22:12 -0700, Junio C Hamano wrote:\n>> I would like to know\n>> a use case or two to illustrate why there are symlinks pointing\n>> at real files outside .git/refs/ hierarchy, and how that\n>> arrangement is useful.\n>...\n> This email is a bit long-winded so I didn't CC it to the mailing list.\n\nThanks for a clear explanation.  Your arrangement indeed is an\nintriguing one, in that there are very similar issues in the\nfsck/prune area even with arrangements quite different from\nyours.  I personally think your reasoning about this issue\ndeserves to be shared with the list.  I'll CC _this_ message to\nthe list and leave it up to you to forward your words there as\nwell.\n\nPeople are known to do something similar to what you are doing\nwithout having any special commit tool.  They just do this:\n\n    $ mkdir A B\n    $ cd A && git init-db\n    $ cd ../B && git init-db\n    $ rm -fr .git/objects && ln -s ../../A/.git/objects .git/objects\n\nThe repositories A and B share the same object database,\nand they have independent sets of refs.  For the exact same\nreason as your arrangement, you cannot \"git prune\" in either\nrepository, because they do not know about objects reachable\nonly from the other side.\n\nFurther, one repository can borrow objects from another\nrepository via the .git/objects/info/alternates mechanism.  This\nis useful when a repository is a local clone of another.  You\nwould do this:\n\n    $ git clone -l -s linux-2.6/.git/ my-linux\n    $ cd my-linux && cat .git/objects/info/alternates\n    /path/to/linux-2.6/.git/\n\nThe new repository my-linux has the .git/objects with 256\nfan-out subdirectories, but starts out without any object files\nin it.  It literally borrows the existing objects from the\nneighbouring repository, and its own .git/objects hierarchy is\nonly used to hold newly created objects in it.  For the same\nreason as your arrangement, you should not \"git prune\" the\nlinux-2.6 repository, either.  However, my-linux repository can\nbe pruned as long as somebody else does not \"borrow\" from it.\n\nSo while I find your \"do follow symlink\" patch an improvement in\nthat it makes things a little bit safer, I think there should be\na more generalized way to say \"this object database holds things\nthat are refered by these refs/ directories outside.  fsck/prune\nhad better hold onto objects referenced by them, not just by the\nrefs directory that happens to be next to th objects directory\".\n\nThat would be the inverse of .git/objects/info/alternates.\n\n-jc\n"},{"id":"7409","messageId":"1124171194.762.74.camel@della.draisey.ca","threadId":"1511","inReplyTo":"1124089116.749.39.camel@della.draisey.ca","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Matt Draisey","fromEmail":"mattdraisey@sympatico.ca","sentAt":"2005-08-17T00:53:05Z","receivedAt":"2005-08-17T00:53:05Z","isPatch":false,"sender":{"key":"matt@draisey.ca","avatar":null},"body":"On Sun, 2005-08-14 at 22:12 -0700, Junio C Hamano wrote:\n> I would like to know\n> a use case or two to illustrate why there are symlinks pointing\n> at real files outside .git/refs/ hierarchy, and how that\n> arrangement is useful.\n\nI've clearly laid out my case very badly.\n\nHere is the patch via sed\n\n$ sed -i -e '49s/lstat/stat/' refs.c\n\nIt changes the do_for_each_ref function to follow symlinks blindly in\nboth the pulling and fscking code.\n\nThe usage I have is simple.  I want to use git to give me a personal\nversioned filesystem of all my current working data.  My problems are\ntwofold.  First, it is not obvious where the divisions between projects\nshould be --- for the most part, this is not distributable software, but\na haphazard collection of one-time code, and various other material that\nstill benefits from version control.  An all inclusive, time-based\ncommit of the entire directory structure turns out to be quite useful.\nObviously this use will never be anything but a personal store of data.\n\nSecondly, I have ambitions for some of the software I put together.\nThese invariably are small projects that may or may not ever become\ndistributable.  Clearly they require their own commit history, yet given\ntheir immature state it seems hard to justify their own objects\ndirectory with its 256 directories for a few kilobytes of code.  Given\nthe existence of the GIT_OBJECT_DIRECTORY environment variable, it is a\nnatural step to sharing a single object store amongst the small\ntightly-focused projects and the all-encompassing but unmanaged\noutermost directory which already is tracking the contents of the\ncontained projects but knows nothing of their commit history.\n\nImplementing a commit tool to do this is actually very easy.  All you\nneed do is walk up the chain of parent directories from your working\ndirectory, noting the .git directories, until you reach one that has\nan .git/objects directory then set up the environment appropriately.\n\nThe only catch is the necessity of maintaining the common object\ndirectory.  The outermost .git directory needs refs to the contained\nsubprojects commit histories that it honours in fsck-cache if it is not\nto silently delete them the next time you prune the repository.\nSymlinking in the refs/heads directories of all subprojects is the most\nstraightforward way to achieve this as it automatically does the right\nthing with very little maintainance.  It also keeps the real refs/heads\nof the subprojects' commit histories properly localized in the\nsubprojects where they belong.  It seems to me that having created the\nGIT_OBJECT_DIRECTORY hook, it only makes sense to follow symlinks in the\nrefs directories.\n\nMatt --- http://free.draisey.ca\n\nP.S.\nThis email is a bit long-winded so I didn't CC it to the mailing list.\n"},{"id":"7472","messageId":"7vslx8thx9.fsf@assigned-by-dhcp.cox.net","threadId":"1511","inReplyTo":"1124246082.776.54.camel@della.draisey.ca","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-17T20:41:38Z","receivedAt":"2005-08-17T20:41:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Draisey <mattdraisey@sympatico.ca> writes:\n\n> Having thus been forced to read the mailing list, I see a slight problem\n> in .git/objects/info/alternates mechanism.  Using the original\n> ALTERNATE_DB_ENVIRONMENT variable you assert to the git programmes that\n> you know all the repositories to search for objects.  In\n> the .git/objects/info/alternates mechanism you implicitly defer to other\n> repositories, which might also implicitly defer to yet another\n> repository.  To ensure an object is truly available you need to compute\n> a transitive closure on all .git/objects/info/alternates --- you can't\n> really rely on .git/objects/info/alternates being transitively closed\n> already.\n\nNo, \"git clone -l -s\" not copying the objects/info/alternates of\nthe repository being cloned was simply a bug; by doing so the\ntransitive closure can be set up \"initially\".\n\nBoth the environment variable and objects/info/alternates share\nthe same problem if the cloned/borrowed from repository suddenly\nstarts to borrow from another repository, losing objects it used\nto have from itself.  You just shouldn't do it.\n\nWith objects/info/alternates, you _could_ do the transitive\nclosure at runtime and do not have to worry about this issue\n(but you now need to worry about cycles), which you cannot do\nwith the environment variable approach.\n"},{"id":"7521","messageId":"1124402196.1557.41.camel@della.draisey.ca","threadId":"1511","inReplyTo":"7vslx8thx9.fsf@assigned-by-dhcp.cox.net","subject":"Re: symlinked directories in refs are now unreachable","fromName":"Matt Draisey","fromEmail":"mattdraisey@sympatico.ca","sentAt":"2005-08-18T21:56:36Z","receivedAt":"2005-08-18T21:56:36Z","isPatch":false,"sender":{"key":"matt@draisey.ca","avatar":null},"body":"Thanks for committing my one-character patch.  In the commit message you\nsaid \n\n> Come to think of\n> it, maybe we should disallow symlink inside .git/refs hierarchy;\n> we update the files there by creat/rename pair, so having\n> symlinks would not work anyway when you do anything that would\n> update them.]\n\nI agree, linking at the file level makes no sense ---  create/rename\npairs will clobber symlinked files (as they would for hardlinked files).\nIf you accept their use at all, symlinked directories are the only way\nto go.\n\nThe alternatives to symlinked directories are:\n\n(1) adding a command line option to fsck that supplies a path to an\nexternal refs directory\n\n(1a) adding a subtool to create a commandline list of sha from a\nsupplied path to an external refs directory\n\n(2) adding an environment variable to do the same\n\n(3) adding a .git configuration file which contains a path --- this is\njust a userspace symlink\n\n(4) create a monster (refcounting objects?) (true cross-references?)\n(conservative garbage collectors that scan your entire hard disk for\npotential references)\n\n(I believe the same arguments hold for the pulling code as for fsck)\n\nCase (1) is easy to implement. (I whipped up a working patch yesterday)\nThe hardest part is thinking up a good name for the command line\nargument.  Case (1a) is too ugly.  Both of these cases place a\nconsiderable burden on the user, and require some Porcelain work.  Case\n(2) makes sense but is too intrusive for me to contemplate.  I defer to\na core developer.  Case (3) offers no advantages over a symlinked\ndirectory under refs.  Case (4) is probably patented by Microsoft.\n\nI like symlinks for their simplicity, and that they work now.  Otherwise\nI am a really only submitting a feature request.\n\nMatt\n"}]}