{"thread":{"id":"39186","subject":"RFC: git cat-file --follow-symlinks?","startedAt":"2015-04-29T20:57:12Z","lastAt":"2015-05-01T21:09:35Z","messageCount":43,"participants":["David Turner","Jonathan Nieder","Junio C Hamano","Jeff King","Michael Haggerty","Andreas Schwab"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"260202","messageId":"1430341032.14907.9.camel@ubuntu","threadId":"39186","inReplyTo":null,"subject":"RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-29T20:57:12Z","receivedAt":"2015-04-29T20:57:12Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"I recently had a situation where I was using git cat-file (--batch) to\nread files and directories out of the repository -- basically, todo the\nequivalent of open, opendir, etc, on an arbitrary revision.\nUnfortunately, I had to do a lot of gymnastics to handle symlinks in the\nrepository.  Instead of just doing echo $SHA:foo/bar/baz | git cat-file\n--batch, I would have to first check if foo was a symlink, and if so,\nfollow it, and then check bar, and so on.\n\nInstead, it would be cool if cat-file had a mode in which it would\nfollow symlinks.\n\nThe major wrinkle is that symlinks can point outside the repository --\neither because they are absolute paths, or because they are relative\npaths with enough ../ in them.  For this case, I propose that\n--follow-symlinks should output [sha] \"symlink\" [target] instead of the\nusual [sha] \"blob\" [bytes].  Since --follow-symlinks is new, this format\nchange will not break any existing code.\n\n(I also propose that we use Linux's limit of 40 symlinks by default, but\n--follow-symlinks could also have =max_links_to_follow to adjust this if\nanyone cares)\n\nDo people think this is reasonable?  If so, I'll see if I can get some\ntime to work on it this month.\n"},{"id":"260205","messageId":"20150429211615.GR5467@google.com","threadId":"39186","inReplyTo":"1430341032.14907.9.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2015-04-29T21:16:15Z","receivedAt":"2015-04-29T21:16:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDavid Turner wrote:\n\n> Instead, it would be cool if cat-file had a mode in which it would\n> follow symlinks.\n\nMakes sense.\n\n> The major wrinkle is that symlinks can point outside the repository --\n> either because they are absolute paths, or because they are relative\n> paths with enough ../ in them.  For this case, I propose that\n> --follow-symlinks should output [sha] \"symlink\" [target] instead of the\n> usual [sha] \"blob\" [bytes].\n\nWhat happens when the symlink payload contains a newline?\n\nThanks,\nJonathan\n"},{"id":"260207","messageId":"xmqqlhha4otr.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"1430341032.14907.9.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-29T21:17:04Z","receivedAt":"2015-04-29T21:17:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner <dturner@twopensource.com> writes:\n\n> Do people think this is reasonable?\n\nI personally don't, exactly because we track the contents of the\nsymlink itself, not the referent.  Your \"major wrinkle\" that they\ncan point outside the repository is a mere manifestation of that.\n\nThe format specifiers the --batch option takes do not exactly give\nyou what the in-tree type of the thing is, to allow the receiving\nend that parses the tagline (which it needs to do anyway in order to\nfind out where the current record ends) act on it.  %(objecttype)\nwould just say \"blob\" and you cannot tell if it is a plain file,\nexecutable or a symbolic link.\n\nPerhaps an ideal interface might be something like this:\n\n    $ echo HEAD:RelNotes |\n      git cat-file --batch='%(objecttype) %(intreemode) %(objectsize)'\n    blob 160000 32\n    Documentation/RelNotes/2.4.0.txt\n\nI suspect it would be just the matter of teaching \"cat-file --batch\"\nto read from get_sha1_with_context() in batch_one_object(), instead\nof reading from get_sha1() which it currently does.\n\nAnd that inteferface I think I can live with.\n"},{"id":"260208","messageId":"1430342670.14907.11.camel@ubuntu","threadId":"39186","inReplyTo":"20150429211615.GR5467@google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-29T21:24:30Z","receivedAt":"2015-04-29T21:24:30Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 14:16 -0700, Jonathan Nieder wrote:\n> Hi,\n> \n> David Turner wrote:\n> \n> > Instead, it would be cool if cat-file had a mode in which it would\n> > follow symlinks.\n> \n> Makes sense.\n> \n> > The major wrinkle is that symlinks can point outside the repository --\n> > either because they are absolute paths, or because they are relative\n> > paths with enough ../ in them.  For this case, I propose that\n> > --follow-symlinks should output [sha] \"symlink\" [target] instead of the\n> > usual [sha] \"blob\" [bytes].\n> \n> What happens when the symlink payload contains a newline?\n\nOh, right.\nSo, how about [sha] \"symlink\" [bytes] \"\\n\" [target] instead?\n"},{"id":"260227","messageId":"1430343059.14907.18.camel@ubuntu","threadId":"39186","inReplyTo":"xmqqlhha4otr.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-29T21:30:59Z","receivedAt":"2015-04-29T21:30:59Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 14:17 -0700, Junio C Hamano wrote:\n> David Turner <dturner@twopensource.com> writes:\n> \n> > Do people think this is reasonable?\n> \n> I personally don't, exactly because we track the contents of the\n> symlink itself, not the referent.  Your \"major wrinkle\" that they\n> can point outside the repository is a mere manifestation of that.\n\nI'm not sure I understand why tracking the contents of the symlink is a\nproblem for this approach.  It seems reasonable to ask what would have\nhappened had I checked out the repo at a certain SHA and said \"cat\nfoo/bar/baz\".\n\n> The format specifiers the --batch option takes do not exactly give\n> you what the in-tree type of the thing is, to allow the receiving\n> end that parses the tagline (which it needs to do anyway in order to\n> find out where the current record ends) act on it.  %(objecttype)\n> would just say \"blob\" and you cannot tell if it is a plain file,\n> executable or a symbolic link.\n> \n> Perhaps an ideal interface might be something like this:\n> \n>     $ echo HEAD:RelNotes |\n>       git cat-file --batch='%(objecttype) %(intreemode) %(objectsize)'\n>     blob 160000 32\n>     Documentation/RelNotes/2.4.0.txt\n> \n> I suspect it would be just the matter of teaching \"cat-file --batch\"\n> to read from get_sha1_with_context() in batch_one_object(), instead\n> of reading from get_sha1() which it currently does.\n> \n> And that inteferface I think I can live with.\n\nEven if I had %(intreemode), I would still have to do a recursive search\nto figure out whether Documentation or RelNotes was a symlink.  This is\nwhy I want a follow-symlinks mode.  And since I am already reading\nRelNotes, I can (and presently do) parse the mode out of that data.\n$(intreedmode) would save me some parsing, but it would not save me any\nreading, nor would it make my code any less complex.  But\n--follow-symlinks would simplify my code.\n"},{"id":"260228","messageId":"20150429214817.GA2725@peff.net","threadId":"39186","inReplyTo":"1430343059.14907.18.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-29T21:48:17Z","receivedAt":"2015-04-29T21:48:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 02:30:59PM -0700, David Turner wrote:\n\n> > I personally don't, exactly because we track the contents of the\n> > symlink itself, not the referent.  Your \"major wrinkle\" that they\n> > can point outside the repository is a mere manifestation of that.\n> \n> I'm not sure I understand why tracking the contents of the symlink is a\n> problem for this approach.  It seems reasonable to ask what would have\n> happened had I checked out the repo at a certain SHA and said \"cat\n> foo/bar/baz\".\n\nBut git can't answer that question, can it? That is something that you\nare asking the filesystem and the OS, and it may involve leaving the git\nrepository altogether (and depend on things like your cwd). Certainly\ngit can ask the filesystem for you, but it's more flexible if you do it\nyourself (at the expense of more code on your end; see below).\n\n> > Perhaps an ideal interface might be something like this:\n> > \n> >     $ echo HEAD:RelNotes |\n> >       git cat-file --batch='%(objecttype) %(intreemode) %(objectsize)'\n> >     blob 160000 32\n> >     Documentation/RelNotes/2.4.0.txt\n> > \n> > I suspect it would be just the matter of teaching \"cat-file --batch\"\n> > to read from get_sha1_with_context() in batch_one_object(), instead\n> > of reading from get_sha1() which it currently does.\n> > \n> > And that inteferface I think I can live with.\n> \n> Even if I had %(intreemode), I would still have to do a recursive search\n> to figure out whether Documentation or RelNotes was a symlink.  This is\n> why I want a follow-symlinks mode.  And since I am already reading\n> RelNotes, I can (and presently do) parse the mode out of that data.\n> $(intreedmode) would save me some parsing, but it would not save me any\n> reading, nor would it make my code any less complex.  But\n> --follow-symlinks would simplify my code.\n\nWouldn't git have to do the same recursive search? That is, with the\ninterface above, you would see \"ah, %(intreemode) says we are a symlink;\nlet me ask again using the filename from the symlink contents\". And\nrepeat until you get a non-symlink. But with a --follow-symlinks option,\ngit is just doing the same thing internally. It cannot ask the\nfilesystem because these are not real files.\n\nSo the advantages of --follow-symlinks are:\n\n  1. It's more efficient. Instead of round-tripping across the pipe, git\n     follows the link internally.\n\n  2. It's easier for callers. Git only has to implement it once, and\n     callers get it for free. Also, callers do not have to have a\n     bidirectional conversation with cat-file (which is doubly awkward\n     if they are trying to send the output of cat-file elsewhere, since\n     they end up having to forward along the non-symlinked output).\n\nThe disadvantages are:\n\n  1. Git has to make a decision about what to do in corner cases. What\n     is our cwd for relative links? The project root? Can we be in a\n     subdir of the repo? What do we do about symlinks that point to\n     non-existent files?  Ones that point outside the repository? If we\n     cat the actual filesystem, what is the cwd of the pretend\n     working-tree that we start from? Do we need to be able to show the\n     contents _and_ the fact that we followed one or more links (and\n     their intermediate names?).\n\nCan you think of arguments (either on pro or con side) that I am\nmissing? Or did I misunderstand what you meant by \"recursive search\"?\n\n-Peff\n"},{"id":"260229","messageId":"xmqqh9ry4nbx.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"1430343059.14907.18.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-29T21:49:22Z","receivedAt":"2015-04-29T21:49:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner <dturner@twopensource.com> writes:\n\n>> Perhaps an ideal interface might be something like this:\n>> \n>>     $ echo HEAD:RelNotes |\n>>       git cat-file --batch='%(objecttype) %(intreemode)\n>>     %(objectsize)' blob 160000 32\n>>     Documentation/RelNotes/2.4.0.txt\n>> \n>> I suspect it would be just the matter of teaching \"cat-file\n>> --batch\" to read from get_sha1_with_context() in\n>> batch_one_object(), instead of reading from get_sha1() which it\n>> currently does.\n>> \n>> And that inteferface I think I can live with.\n>\n> Even if I had %(intreemode), I would still have to do a recursive\n> search to figure out whether Documentation or RelNotes was a\n> symlink.\n\nYes, and why is that a problem?  Think of \"cat-file --batch\" an\n\"object server\" you query interactively.  You start the process, ask\nit about HEAD:RelNotes, and learn that the blob is a link that\npoints at Documentation/RelNotes/2.4.0.txt.  Then you ask it about\n\"HEAD:Documentation/RelNotes/2.4.0.txt\", which _may_ answer \"no such\nfile\", at which point you can start worrying about referring to\nplaces outside the tree (i.e. untracked).\n\n\"cat-file\" does not know about your project, and especially its\nexternal dependencies, if a symbolic link ever steps outside the\ntree objects, better than you do.  Because it is a low-level\nplumbing command, allowing it to make policy decisions (e.g. \"if\noutside repository, always look at the filesystem that the program\nhappens to be running\" [*1*]) is something I would reject as much as\npossible.  It will paint us into a corner we cannot later escape out\nof.\n\n> This is\n> why I want a follow-symlinks mode.  And since I am already reading\n> RelNotes, I can (and presently do) parse the mode out of that\n> data.\n\nmode?  How?  If all you have is an blob object name and no context\naround it (i.e. the top-level tree object has that blob with 160000\nmode bits), you cannot tell a symlink from a regular file.\n\n\n[Footnote]\n\n*1* For example, you may have two projects's working trees A and B\n    sitting next to each other, and A/sibling may be a symbolic link\n    that points at ../B/some/thing.  A Porcelain that uses cat-file\n    --batch as \"the object server\" may notice v1.0:sibling in A's\n    history points at ../B/some/thing and would want to grab\n    some/thing from the contemporary version of B's commit, instead\n    of just blindly going to the filesystem.\n"},{"id":"260231","messageId":"20150429221907.GS5467@google.com","threadId":"39186","inReplyTo":"20150429214817.GA2725@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2015-04-29T22:19:07Z","receivedAt":"2015-04-29T22:19:07Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n\n>   1. Git has to make a decision about what to do in corner cases. What\n>      is our cwd for relative links? The project root?\n\nI don't follow.  Isn't symlink resolution always relative to the\nsymlink, regardless of cwd?\n\nThanks,\nJonathan\n"},{"id":"260232","messageId":"1430346576.14907.40.camel@ubuntu","threadId":"39186","inReplyTo":"20150429214817.GA2725@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-29T22:29:36Z","receivedAt":"2015-04-29T22:29:36Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 17:48 -0400, Jeff King wrote:\n> On Wed, Apr 29, 2015 at 02:30:59PM -0700, David Turner wrote:\n> \n> > > I personally don't, exactly because we track the contents of the\n> > > symlink itself, not the referent.  Your \"major wrinkle\" that they\n> > > can point outside the repository is a mere manifestation of that.\n> > \n> > I'm not sure I understand why tracking the contents of the symlink is a\n> > problem for this approach.  It seems reasonable to ask what would have\n> > happened had I checked out the repo at a certain SHA and said \"cat\n> > foo/bar/baz\".\n> \n> But git can't answer that question, can it? That is something that you\n> are asking the filesystem and the OS, and it may involve leaving the git\n> repository altogether (and depend on things like your cwd). Certainly\n> git can ask the filesystem for you, but it's more flexible if you do it\n> yourself (at the expense of more code on your end; see below).\n\nAs far as I know, symlink resolution doesn't depend on cwd.  And paths\npassed to cat-file are interpreted relative to the repo root regardless\nof the cwd anyway.\n\nIt's true that full symlink resolution might depend on leaving the git\nrepo.  I didn't want git to have to deal with that, so my proposal told\nthe caller about out-of-repo symlinks, allowing the caller to deal. The\ncaller can then use the standard library, which already knows how to\nresolve symlinks.  I guess a sequence of links could leave and then\nreenter the repository; that is indeed a corner case that I am happy to\nleave to the callers; any caller who cares will be no worse off than\nthey are now.\n\n> > > Perhaps an ideal interface might be something like this:\n> > > \n> > >     $ echo HEAD:RelNotes |\n> > >       git cat-file --batch='%(objecttype) %(intreemode) %(objectsize)'\n> > >     blob 160000 32\n> > >     Documentation/RelNotes/2.4.0.txt\n> > > \n> > > I suspect it would be just the matter of teaching \"cat-file --batch\"\n> > > to read from get_sha1_with_context() in batch_one_object(), instead\n> > > of reading from get_sha1() which it currently does.\n> > > \n> > > And that inteferface I think I can live with.\n> > \n> > Even if I had %(intreemode), I would still have to do a recursive search\n> > to figure out whether Documentation or RelNotes was a symlink.  This is\n> > why I want a follow-symlinks mode.  And since I am already reading\n> > RelNotes, I can (and presently do) parse the mode out of that data.\n> > $(intreedmode) would save me some parsing, but it would not save me any\n> > reading, nor would it make my code any less complex.  But\n> > --follow-symlinks would simplify my code.\n> \n> Wouldn't git have to do the same recursive search? That is, with the\n> interface above, you would see \"ah, %(intreemode) says we are a symlink;\n> let me ask again using the filename from the symlink contents\". And\n> repeat until you get a non-symlink. But with a --follow-symlinks option,\n> git is just doing the same thing internally. It cannot ask the\n> filesystem because these are not real files.\n> \n> So the advantages of --follow-symlinks are:\n> \n>   1. It's more efficient. Instead of round-tripping across the pipe, git\n>      follows the link internally.\n> \n>   2. It's easier for callers. Git only has to implement it once, and\n>      callers get it for free. Also, callers do not have to have a\n>      bidirectional conversation with cat-file (which is doubly awkward\n>      if they are trying to send the output of cat-file elsewhere, since\n>      they end up having to forward along the non-symlinked output).\n> \n> The disadvantages are:\n> \n>   1. Git has to make a decision about what to do in corner cases. What\n>      is our cwd for relative links? The project root? Can we be in a\n>      subdir of the repo? What do we do about symlinks that point to\n>      non-existent files?  Ones that point outside the repository? If we\n>      cat the actual filesystem, what is the cwd of the pretend\n>      working-tree that we start from? Do we need to be able to show the\n>      contents _and_ the fact that we followed one or more links (and\n>      their intermediate names?).\n> \n> Can you think of arguments (either on pro or con side) that I am\n> missing? Or did I misunderstand what you meant by \"recursive search\"?\n\nOverall, I agree.  I think the disadvantages are somewhat overstated.\n\nAs I said above, I don't think the cwd is a problem.  The output for\nsymlinks which point outside the repo should be absolute (in the case of\nabsolute symlinks), or relative to the repo root (for relative\nsymlinks).  In other words, if my repo contains:\nfoo/bar -> ../../baz\nthen the output[1] would be \nsymlink 6\n../baz\n\nI can't think of any other output that would be reasonable here, but\nmaybe there's something I don't understand.\n\nNote that I'm not proposing that cat-file should actually read any files\nor directories from outside of the repo.  It should just make it\npossible for the caller to do so.\n\n\n[1] (I realize, now, that having a sha would be useless, so I've omitted\nit).\n"},{"id":"260234","messageId":"1430347672.14907.48.camel@ubuntu","threadId":"39186","inReplyTo":"xmqqh9ry4nbx.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-29T22:47:52Z","receivedAt":"2015-04-29T22:47:52Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 14:49 -0700, Junio C Hamano wrote:\n> David Turner <dturner@twopensource.com> writes:\n> \n> >> Perhaps an ideal interface might be something like this:\n> >> \n> >>     $ echo HEAD:RelNotes |\n> >>       git cat-file --batch='%(objecttype) %(intreemode)\n> >>     %(objectsize)' blob 160000 32\n> >>     Documentation/RelNotes/2.4.0.txt\n> >> \n> >> I suspect it would be just the matter of teaching \"cat-file\n> >> --batch\" to read from get_sha1_with_context() in\n> >> batch_one_object(), instead of reading from get_sha1() which it\n> >> currently does.\n> >> \n> >> And that inteferface I think I can live with.\n> >\n> > Even if I had %(intreemode), I would still have to do a recursive\n> > search to figure out whether Documentation or RelNotes was a\n> > symlink.\n\nI apologize.  I have misread your example.  All of my text was assuming\nthat Documentation/Relnotes/2.4.0.txt was a symlink, instead\nof /RelNotes being a symlink to Documentation/Relnotes/2.4.0.txt.  So my\nprevious message was very difficult to interpret.  I think Jeff King's\nreply is a better starting point for discussion, since it lays out the\nadvantages and disadvantages of the proposal.\n\n> allowing it to make policy decisions (e.g. \"if\n> outside repository, always look at the filesystem that the program\n> happens to be running\" [*1*]) \n\nDespite my confusion, I don't think I ever proposed doing this.  I\nproposed that in the case that a symlink points outside the repo,\ncat-file would tell the caller that it has done so, so that the caller\ncan decide what to do.\n"},{"id":"260236","messageId":"20150429230543.GA3887@peff.net","threadId":"39186","inReplyTo":"20150429221907.GS5467@google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-29T23:05:43Z","receivedAt":"2015-04-29T23:05:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 03:19:07PM -0700, Jonathan Nieder wrote:\n\n> Jeff King wrote:\n> \n> >   1. Git has to make a decision about what to do in corner cases. What\n> >      is our cwd for relative links? The project root?\n> \n> I don't follow.  Isn't symlink resolution always relative to the\n> symlink, regardless of cwd?\n\nYeah, I was being silly to think this was a concern for intra-repository\nlinks. It's well-defined there.  E.g., if \"foo/bar/baz\" points to\n\"../moof\", that is just \"foo/moof\".\n\nBut if we leave the git tree the location of our root matters. E.g., if\n\"foo/bar/baz\" points to \"../../../../moof\", where is that anchored in\nthe filesystem? So perhaps \"cwd\" is not the right term. It is really\n\"where are we pretending our working tree is\".\n\n-Peff\n"},{"id":"260237","messageId":"20150429231150.GB3887@peff.net","threadId":"39186","inReplyTo":"1430346576.14907.40.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-29T23:11:50Z","receivedAt":"2015-04-29T23:11:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 03:29:36PM -0700, David Turner wrote:\n\n> Overall, I agree.  I think the disadvantages are somewhat overstated.\n> \n> As I said above, I don't think the cwd is a problem.  The output for\n> symlinks which point outside the repo should be absolute (in the case of\n> absolute symlinks), or relative to the repo root (for relative\n> symlinks).  In other words, if my repo contains:\n> foo/bar -> ../../baz\n> then the output[1] would be \n> symlink 6\n> ../baz\n> \n> I can't think of any other output that would be reasonable here, but\n> maybe there's something I don't understand.\n\nYeah, I agree if you let git punt on leaving the filesystem, most of the\ncomplicated problems go away. It still feels a bit more magical than I\nexpect out of cat-file, and there are still corner cases (e.g., do we do\ncycle detection? Or just have a limit to the recursion depth?)\n\nAnd if you are punting on some cases, I think you'd still want to be\nable to report on the symlinks you couldn't resolve (e.g., because they\nwent out of tree, pointed to non-existent files, or caused cycles). So\nit seems like %(intreemode) is a good first step, because it lets you\nexpress that (and more).\n\nAnd then you could implement --follow-symlinks on top of that; it can't\ncatch all cases, but you've left callers with an escape hatch to do\ntheir own resolution if they want, without having to implement a new\nsyntax for it.\n\n-Peff\n"},{"id":"260240","messageId":"20150430003750.GA4258@peff.net","threadId":"39186","inReplyTo":"20150429231150.GB3887@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T00:37:51Z","receivedAt":"2015-04-30T00:37:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 07:11:50PM -0400, Jeff King wrote:\n\n> Yeah, I agree if you let git punt on leaving the filesystem, most of the\n> complicated problems go away. It still feels a bit more magical than I\n> expect out of cat-file, and there are still corner cases (e.g., do we do\n> cycle detection? Or just have a limit to the recursion depth?)\n\nI was pondering the \"magical\" above. I think what bugs me is that it\nseems like a feature that is implemented as part of one random bit of\nplumbing, but not available elsewhere.\n\nConceptually, this is like peeling object names. You may give a tag\nname, but if you ask for a tree commit we will peel the tag to a commit,\nand the commit to a tree. This is sort of the same thing; you give a\npath within a tree, and we will peel until we hit a \"real\" non-symlink\nobject.\n\nI don't know what the syntax would look like. To match \"foo^{tree}\" it\nwould be something like:\n\n  HEAD:foo/bar^{resolve}\n\nor something like that. Except that it is a bad idea to allow \"^{}\"\nsyntax on the right-hand side of a colon, as it is ambiguous with\nfilenames that contain \"^{resolve}\". So it would have to look something\nlike:\n\n  HEAD^{resolve}:foo/bar\n\nwhich is a _little_ weird, but actually kind of makes sense. The\n\"resolve\" operation inherently is not just about the filename, but about\nuses HEAD^{tree} as the root context.\n\nSo I dunno. This pushes the resolving logic even _lower_ in the stack\nthan it would be in cat-file. So why do I like it more? Cognitive\ndissonance? I guess I the appeal to me is that it:\n\n  1. Makes the concept available more generally (you can \"rev-parse\" it,\n     you can \"git show\" it, etc). It also lets you _name_ the object in\n     question, so you can ask for other things besides it contents (like\n     its name, its type, etc).\n\n  2. Positions it alongside other peeling name-resolution functions.\n\nThoughts?\n\n-Peff\n"},{"id":"260242","messageId":"1430355983.14907.55.camel@ubuntu","threadId":"39186","inReplyTo":"20150430003750.GA4258@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T01:06:23Z","receivedAt":"2015-04-30T01:06:23Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 20:37 -0400, Jeff King wrote:\n> On Wed, Apr 29, 2015 at 07:11:50PM -0400, Jeff King wrote:\n> \n> > Yeah, I agree if you let git punt on leaving the filesystem, most of the\n> > complicated problems go away. It still feels a bit more magical than I\n> > expect out of cat-file, and there are still corner cases (e.g., do we do\n> > cycle detection? Or just have a limit to the recursion depth?)\n> \n> I was pondering the \"magical\" above. I think what bugs me is that it\n> seems like a feature that is implemented as part of one random bit of\n> plumbing, but not available elsewhere.\n> \n> Conceptually, this is like peeling object names. You may give a tag\n> name, but if you ask for a tree commit we will peel the tag to a commit,\n> and the commit to a tree. This is sort of the same thing; you give a\n> path within a tree, and we will peel until we hit a \"real\" non-symlink\n> object.\n> \n> I don't know what the syntax would look like. To match \"foo^{tree}\" it\n> would be something like:\n> \n>   HEAD:foo/bar^{resolve}\n> \n> or something like that. Except that it is a bad idea to allow \"^{}\"\n> syntax on the right-hand side of a colon, as it is ambiguous with\n> filenames that contain \"^{resolve}\". So it would have to look something\n> like:\n> \n>   HEAD^{resolve}:foo/bar\n> \n> which is a _little_ weird, but actually kind of makes sense. The\n> \"resolve\" operation inherently is not just about the filename, but about\n> uses HEAD^{tree} as the root context.\n> \n> So I dunno. This pushes the resolving logic even _lower_ in the stack\n> than it would be in cat-file. So why do I like it more? Cognitive\n> dissonance? I guess I the appeal to me is that it:\n> \n>   1. Makes the concept available more generally (you can \"rev-parse\" it,\n>      you can \"git show\" it, etc). It also lets you _name_ the object in\n>      question, so you can ask for other things besides it contents (like\n>      its name, its type, etc).\n> \n>   2. Positions it alongside other peeling name-resolution functions.\n\nJust to clarify: if you do git rev-parse, and the result is an\nout-of-tree symlink, you see /foo or ../foo instead of a sha?  And if\nyou \"git show\" it it says \"symlink HEAD:../foo\"?\n\nThis seems totally reasonable to me, and solves my problem.\n"},{"id":"260244","messageId":"20150430011612.GA7530@peff.net","threadId":"39186","inReplyTo":"1430355983.14907.55.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T01:16:12Z","receivedAt":"2015-04-30T01:16:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 06:06:23PM -0700, David Turner wrote:\n\n> >   HEAD^{resolve}:foo/bar\n> [...]\n>\n> Just to clarify: if you do git rev-parse, and the result is an\n> out-of-tree symlink, you see /foo or ../foo instead of a sha?  And if\n> you \"git show\" it it says \"symlink HEAD:../foo\"?\n\nI had imagined we would stop resolution and you would just get the last\nobject peeled object. Combined with teaching cat-file to show more\nobject context, doing:\n\n  echo content >dest ;# actual blob\n  ln -s dest link    ;# link to blob\n  ln -s broken foo   ;# broken link\n  ln -s out ../foo   ;# out-of-tree link\n  git add . && git commit -m foo\n  for i in link broken out; do\n\techo HEAD^{resolve}:$i\n  done |\n  git cat-file --batch=\"%(intreemode) %(size)\"\n\nwould yield:\n\n (1)   100644 8\n       content\n (2)   040000 3\n       foo\n (3)   040000 6\n       ../foo\n\nwhere the left-margin numbers are for reference:\n\n  1. We dereference a real symlink, and pretend like we actually asked\n     for its referent.\n\n  2. For a broken link, we can't dereference, so we return the link\n     itself. You can tell by the mode, and the content tells you what\n     would have been dereferenced.\n\n  3. Ditto for out-of-tree. Note that this would be the _raw_ symlink\n     contents, not any kind of simplification (so if you asked for\n     \"foo/bar/baz\" and it was \"../../../../out\", you would the full path\n     with all those dots, not a simplified \"../out\", which I think is\n     what you were trying to show in earlier examples).\n\n-Peff\n"},{"id":"260245","messageId":"xmqq7fsu4d1x.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"20150430011612.GA7530@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-30T01:31:22Z","receivedAt":"2015-04-30T01:31:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> I had imagined we would stop resolution and you would just get the last\n> object peeled object. Combined with teaching cat-file to show more\n> object context, doing:\n>\n>   echo content >dest ;# actual blob\n>   ln -s dest link    ;# link to blob\n>   ln -s broken foo   ;# broken link\n>   ln -s out ../foo   ;# out-of-tree link\n>   git add . && git commit -m foo\n>   for i in link broken out; do\n> \techo HEAD^{resolve}:$i\n>   done |\n>   git cat-file --batch=\"%(intreemode) %(size)\"\n>\n> would yield:\n>\n>  (1)   100644 8\n>        content\n>  (2)   040000 3\n>        foo\n>  (3)   040000 6\n>        ../foo\n>\n> where the left-margin numbers are for reference:\n>\n>   1. We dereference a real symlink, and pretend like we actually asked\n>      for its referent.\n>\n>   2. For a broken link, we can't dereference, so we return the link\n>      itself. You can tell by the mode, and the content tells you what\n>      would have been dereferenced.\n>\n>   3. Ditto for out-of-tree. Note that this would be the _raw_ symlink\n>      contents, not any kind of simplification (so if you asked for\n>      \"foo/bar/baz\" and it was \"../../../../out\", you would the full path\n>      with all those dots, not a simplified \"../out\", which I think is\n>      what you were trying to show in earlier examples).\n\ns/040000/160000/ I would think (if you really meant to expose a\ntree, write it as 40000 instead, so that people will not get a wrong\nimpression and reimplement a broken tree object encoding some popular\nGit hosting site broke their customer projects with ;-).\n\nI am not sure $treeish^{resolve} is a great syntax, but I like the\nconcept and agree that it is a lot more sensible to handle this at\nthe level of sha1_name.c layer than an ad-hoc solution in the\ncat-file layer.\n"},{"id":"260251","messageId":"1430358345.14907.62.camel@ubuntu","threadId":"39186","inReplyTo":"20150430011612.GA7530@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T01:45:45Z","receivedAt":"2015-04-30T01:45:45Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 21:16 -0400, Jeff King wrote:\n> On Wed, Apr 29, 2015 at 06:06:23PM -0700, David Turner wrote:\n>   3. Ditto for out-of-tree. Note that this would be the _raw_ symlink\n>      contents, not any kind of simplification (so if you asked for\n>      \"foo/bar/baz\" and it was \"../../../../out\", you would the full path\n>      with all those dots, not a simplified \"../out\", which I think is\n>      what you were trying to show in earlier examples).\n\nUnfortunately, we need the simplified version, because we otherwise\ndon't know what the ..s are relative to in the case of a link to a link:\n\n  echo content >dest ;# actual blob\n  mkdir -p foo/bar\n  ln -s foo/bar/baz fleem             # in-tree link-to-link \n  ln -s ../../../external foo/bar/baz # out-of-tree link\n\nIf echo HEAD^{resolve}:fleem were to return ../../../external (after\nfollowing the first symlink to the second), we would have lost\ninformation.\n"},{"id":"260253","messageId":"20150430031841.GA12361@peff.net","threadId":"39186","inReplyTo":"xmqq7fsu4d1x.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T03:18:41Z","receivedAt":"2015-04-30T03:18:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 06:31:22PM -0700, Junio C Hamano wrote:\n\n> > would yield:\n> >\n> >  (1)   100644 8\n> >        content\n> >  (2)   040000 3\n> >        foo\n> >  (3)   040000 6\n> >        ../foo\n> [...]\n> \n> s/040000/160000/ I would think (if you really meant to expose a\n> tree, write it as 40000 instead, so that people will not get a wrong\n> impression and reimplement a broken tree object encoding some popular\n> Git hosting site broke their customer projects with ;-).\n\nWhooops. Yes, definitely 160000. That's what I get for not\ndouble-checking. :)\n\n-Peff\n"},{"id":"260254","messageId":"20150430033725.GB12361@peff.net","threadId":"39186","inReplyTo":"1430358345.14907.62.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T03:37:25Z","receivedAt":"2015-04-30T03:37:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2015 at 06:45:45PM -0700, David Turner wrote:\n\n> On Wed, 2015-04-29 at 21:16 -0400, Jeff King wrote:\n> > On Wed, Apr 29, 2015 at 06:06:23PM -0700, David Turner wrote:\n> >   3. Ditto for out-of-tree. Note that this would be the _raw_ symlink\n> >      contents, not any kind of simplification (so if you asked for\n> >      \"foo/bar/baz\" and it was \"../../../../out\", you would the full path\n> >      with all those dots, not a simplified \"../out\", which I think is\n> >      what you were trying to show in earlier examples).\n> \n> Unfortunately, we need the simplified version, because we otherwise\n> don't know what the ..s are relative to in the case of a link to a link:\n> \n>   echo content >dest ;# actual blob\n>   mkdir -p foo/bar\n>   ln -s foo/bar/baz fleem             # in-tree link-to-link \n>   ln -s ../../../external foo/bar/baz # out-of-tree link\n> \n> If echo HEAD^{resolve}:fleem were to return ../../../external (after\n> following the first symlink to the second), we would have lost\n> information.\n\nUrgh, yeah, thanks for the counter-example.\n\nHere are some possible alternatives:\n\n  1. If we can't resolve fully, don't resolve anything. I.e., return the\n     \"fleem\" object here, and the caller can recurse if they want. This is\n     simple and correct, but not as helpful to somebody who wants to follow\n     the out-of-tree link (they have to re-traverse the fleem->foo/bar/baz\n     link themselves).\n\n  2. Consider it can error if resolution fails. If you ask for\n     \"HEAD^{tree}^{commit}\", that does not resolve to anything (because\n     we can't peel the tree to a commit). Like (1), this is simple and\n     correct, but probably not all that helpful. The caller has to\n     start from scratch and resolve themselves, rather than getting an\n     intermediate result.\n\n  3. Return an object with the symlink relative to the original\n     filename (so \"../external\" in this case). This is kind of weird,\n     though, because we're not just returning a string from the name\n     resolution. It's an actual object.  So we'd be generating a fake\n     object that doesn't actually exist in the object db and\n     returning that. Feeding that sha1 to another program would fail.\n\n  4. Return the last object we could resolve, as I described. So\n     foo/bar/baz (with \"../../../external\" as its content) in this case.\n     When you resolve a name, you can ask for the context we discovered\n     along the way by traversing the tree. The mode is one example we've\n     already discussed, but the path name is another. So something like:\n\n       echo \"HEAD^{resolve}:fleem\" |\n       git cat-file --batch=\"%(objectname) %(size) %(intreepath)\"\n\n     would show:\n\n       1234abcd 17 foo/bar/baz\n       ../../../external\n\n     And then the caller knows that the path is not relative to the\n     original \"fleem\", but rather to \"foo/bar/baz\".\n\n     The problem is that although this context lookup is already part of\n     get_sha1_with_context, that is not exposed through every interface.\n     E.g., \"git rev-parse HEAD^{resolve}:fleem\" will give you an object,\n     but you have no way of knowing the context.\n\nI can't say that I'm excited about any of them. Perhaps you or somebody\nelse can think of a more clever solution.\n\nNote that the complication with (3) does come from my trying to push\nthis down into the name-resolution code.\n\n-Peff\n"},{"id":"260258","messageId":"xmqqfv7i2n81.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"20150430033725.GB12361@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-30T05:34:38Z","receivedAt":"2015-04-30T05:34:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Here are some possible alternatives:\n>\n>   1. If we can't resolve fully, don't resolve anything. I.e., return the\n>      \"fleem\" object here, and the caller can recurse if they want. This is\n>      simple and correct, but not as helpful to somebody who wants to follow\n>      the out-of-tree link (they have to re-traverse the fleem->foo/bar/baz\n>      link themselves).\n>\n>   2. Consider it can error if resolution fails. If you ask for\n>      \"HEAD^{tree}^{commit}\", that does not resolve to anything (because\n>      we can't peel the tree to a commit). Like (1), this is simple and\n>      correct, but probably not all that helpful. The caller has to\n>      start from scratch and resolve themselves, rather than getting an\n>      intermediate result.\n>\n>   3. Return an object with the symlink relative to the original\n>      filename (so \"../external\" in this case). This is kind of weird,\n>      though, because we're not just returning a string from the name\n>      resolution. It's an actual object.  So we'd be generating a fake\n>      object that doesn't actually exist in the object db and\n>      returning that. Feeding that sha1 to another program would fail.\n>\n>   4. Return the last object we could resolve, as I described. So\n>      foo/bar/baz (with \"../../../external\" as its content) in this case.\n>      When you resolve a name, you can ask for the context we discovered\n>      along the way by traversing the tree. The mode is one example we've\n>      already discussed, but the path name is another. So something like:\n>\n>        echo \"HEAD^{resolve}:fleem\" |\n>        git cat-file --batch=\"%(objectname) %(size) %(intreepath)\"\n>\n>      would show:\n>\n>        1234abcd 17 foo/bar/baz\n>        ../../../external\n>\n>      And then the caller knows that the path is not relative to the\n>      original \"fleem\", but rather to \"foo/bar/baz\".\n>\n>      The problem is that although this context lookup is already part of\n>      get_sha1_with_context, that is not exposed through every interface.\n>      E.g., \"git rev-parse HEAD^{resolve}:fleem\" will give you an object,\n>      but you have no way of knowing the context.\n>\n> I can't say that I'm excited about any of them. Perhaps you or somebody\n> else can think of a more clever solution.\n\nMe neither, but if I really had to pick one, it would be the last\none, except that %(intreepath) would have to be somehow quoted,\nperhaps like how the output from \"git diff\" quotes funny pathnames.\n\nWhen we want to use this as an extended SHA-1 syntax (i.e. outside\nof \"cat-file --batch\"), it most likely should just error out if the\nlink does not resolve to a path that would eventually result in an\nin-tree object, with the same error message you would get when you\nask for the object name for \"HEAD:no-such-path\".\n\nBut stepping back a bit.\n\nWe have been talking about HEAD^{resolve}:fleem but how did we learn\nthat there is a path \"fleem\" in the tree of HEAD in the first place?\nI would presume that the answer eventually boils down to \"somebody\nfed HEAD to 'ls-tree -r'\", and then that somebody is an idiot if it\ndid not grab the mode bits to learn what kind of blob fleem is, or\nif it did not tell the guy that wants to drive \"cat-file --batch\".\n\nSo this whole thing somehow starts to smell like a solution that is\nlooking for a problem that has arisen only because the use case\nstory behind it is screwy.  Again, why is it such a huge problem to\nmake it relative and ask a newly formlated question for the guy who\nis driving \"cat-file --batch\" in the first place?\n"},{"id":"260263","messageId":"5541E36D.1010606@alum.mit.edu","threadId":"39186","inReplyTo":"1430341032.14907.9.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-04-30T08:10:21Z","receivedAt":"2015-04-30T08:10:21Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 04/29/2015 10:57 PM, David Turner wrote:\n> I recently had a situation where I was using git cat-file (--batch) to\n> read files and directories out of the repository -- basically, todo the\n> equivalent of open, opendir, etc, on an arbitrary revision.\n> Unfortunately, I had to do a lot of gymnastics to handle symlinks in the\n> repository.  Instead of just doing echo $SHA:foo/bar/baz | git cat-file\n> --batch, I would have to first check if foo was a symlink, and if so,\n> follow it, and then check bar, and so on.\n> \n> Instead, it would be cool if cat-file had a mode in which it would\n> follow symlinks.\n\nI guess it's obvious, but I haven't seen it discussed in this thread, so\nI wanted to point out that this feature has some limitations related to\nhow its arguments are constructed.\n\nIn the examples discussed,\n\n    git cat-file --follow-symlinks $committish:foo/bar/baz\n\n, we know the root of a tree and we know the relative path where the\nsymlink was located, so all is well (modulo a policy for handling\nsymlinks that point outside of the repo). But the following, which would\nnaively seem to be identical, cannot work:\n\n    oid=$(git rev-parse $committish:foo/bar/baz)\n    git cat-file --follow-symlinks $sha1\n\nThe problem is that `$oid` is the name of a blob, and `cat-file` can't\nknow whether the blob represents the contents of a symlink or the\ncontents of a file. (And even if it knew, it would have no idea what\ntree the symlink paths should be interpreted relative to.) What if we\npass `cat-file` a tree and a relative path instead?:\n\n    tree=$(git rev-parse $committish:foo/bar)\n    git cat-file --follow-symlinks $tree:baz\n\nNow it can work, but only if the symlink chain never rises above the\nlevel of `$tree`. So for example, if `foo/bar/baz` points at `../xyzzy`,\nthen the very first example would succeed, whereas the last one would\nhave to fail. Please note that there is no possible way to avoid failure\nby reading files from the filesystem outside of the repository, because\nin this case `cat-file` can have no idea where to look.\n\n> The major wrinkle is that symlinks can point outside the repository --\n> either because they are absolute paths, or because they are relative\n> paths with enough ../ in them.  For this case, I propose that\n> --follow-symlinks should output [sha] \"symlink\" [target] instead of the\n> usual [sha] \"blob\" [bytes].  Since --follow-symlinks is new, this format\n> change will not break any existing code.\n> [...]\n\nI don't think this is doable in the general case, because it is not only\nthe last component of the path that can point outside of the repository.\nSuppose we have\n\n    foo -> ../plugh\n\nand I run\n\n    git cat-file --follow-symlinks HEAD:foo/bar/baz\n\nThe lookup of `foo` already falls outside of the repository, and\n`bar/baz` is relative to *it*, so in this case it would have to return\n\n    ???? \"symlink\" ../plugh/bar/baz\n\nThe question is, what SHA-1 should be output in place of the question\nmarks? The only SHA-1 we have handy is the SHA-1 of `foo`, but that\ndoesn't seem especially useful.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"260264","messageId":"5541E3D4.7040207@alum.mit.edu","threadId":"39186","inReplyTo":"xmqqfv7i2n81.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-04-30T08:12:04Z","receivedAt":"2015-04-30T08:12:04Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 04/30/2015 07:34 AM, Junio C Hamano wrote:\n> [...]\n> But stepping back a bit.\n> \n> We have been talking about HEAD^{resolve}:fleem but how did we learn\n> that there is a path \"fleem\" in the tree of HEAD in the first place?\n> I would presume that the answer eventually boils down to \"somebody\n> fed HEAD to 'ls-tree -r'\", and then that somebody is an idiot if it\n> did not grab the mode bits to learn what kind of blob fleem is, or\n> if it did not tell the guy that wants to drive \"cat-file --batch\".\n\nI think a plausible use case for this feature is to read\n`$tag^{resolve}:RelNotes`, in which case the reason we know it's there\nis \"because the maintainer told us it is there\".\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"260268","messageId":"87383ihqzp.fsf@igel.home","threadId":"39186","inReplyTo":"20150430033725.GB12361@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2015-04-30T10:04:10Z","receivedAt":"2015-04-30T10:04:10Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>   4. Return the last object we could resolve, as I described. So\n>      foo/bar/baz (with \"../../../external\" as its content) in this case.\n>      When you resolve a name, you can ask for the context we discovered\n>      along the way by traversing the tree. The mode is one example we've\n>      already discussed, but the path name is another. So something like:\n>\n>        echo \"HEAD^{resolve}:fleem\" |\n>        git cat-file --batch=\"%(objectname) %(size) %(intreepath)\"\n>\n>      would show:\n>\n>        1234abcd 17 foo/bar/baz\n>        ../../../external\n>\n>      And then the caller knows that the path is not relative to the\n>      original \"fleem\", but rather to \"foo/bar/baz\".\n\nNote that \"..\" will always follow the *physical* structure, so if\nfoo/bar/baz is walking over symbolic links, \"../../..\" may lead you to\nsomewhere else entirely.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"260282","messageId":"1430417023.22711.8.camel@ubuntu","threadId":"39186","inReplyTo":"5541E3D4.7040207@alum.mit.edu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T18:03:43Z","receivedAt":"2015-04-30T18:03:43Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-04-30 at 10:12 +0200, Michael Haggerty wrote:\n> On 04/30/2015 07:34 AM, Junio C Hamano wrote:\n> > [...]\n> > But stepping back a bit.\n> > \n> > We have been talking about HEAD^{resolve}:fleem but how did we learn\n> > that there is a path \"fleem\" in the tree of HEAD in the first place?\n> > I would presume that the answer eventually boils down to \"somebody\n> > fed HEAD to 'ls-tree -r'\", and then that somebody is an idiot if it\n> > did not grab the mode bits to learn what kind of blob fleem is, or\n> > if it did not tell the guy that wants to drive \"cat-file --batch\".\n> \n> I think a plausible use case for this feature is to read\n> `$tag^{resolve}:RelNotes`, in which case the reason we know it's there\n> is \"because the maintainer told us it is there\".\n\nYes, that is approximately my use case.  Read on for details:\nWith a colleague, I'm building a mode for the free and open source Pants\nbuild system that will support build-aware sparse checkouts.  Pants is a\nbuild tool for monorepos (inspired by Google's Blaze and similar to\nFacebook's Buck).  Most individual users will only be using a tiny\nsubset of the full repository, so it would be convenient if they only\nhad to check out what the plan to use.  Assume that they want to check\nout only a certain target (a path, approximately) plus its transitive\ndependencies, on a certain revision.  So pants first checks that\ndirectory (at that rev) for a BUILD file.  That BUILD file might point\nto other BUILD files as dependencies, so again, we must examine those,\nrecursively.  \n\nIn no case did we do a ls-files command, since we want to examine as\nlittle of the repo as possible.  And even if we had done an ls-files, we\nwould still need to resolve all of the symlinks ourselves.\n\nSo that's the motivation here.\n"},{"id":"260284","messageId":"xmqqzj5pzdfo.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"1430417023.22711.8.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-30T18:19:39Z","receivedAt":"2015-04-30T18:19:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner <dturner@twopensource.com> writes:\n\n> In no case did we do a ls-files command,...\n\n\"ls-tree -r\" is what I would have imagined you would be using, as\nsomebody needs to have the full repository in order to resolve the\nsymbolic links _anyway_, and that somebody does not need to have a\ncheckout in order to do so.\n"},{"id":"260285","messageId":"20150430182659.GB23431@peff.net","threadId":"39186","inReplyTo":"87383ihqzp.fsf@igel.home","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T18:27:00Z","receivedAt":"2015-04-30T18:27:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2015 at 12:04:10PM +0200, Andreas Schwab wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> >   4. Return the last object we could resolve, as I described. So\n> >      foo/bar/baz (with \"../../../external\" as its content) in this case.\n> >      When you resolve a name, you can ask for the context we discovered\n> >      along the way by traversing the tree. The mode is one example we've\n> >      already discussed, but the path name is another. So something like:\n> >\n> >        echo \"HEAD^{resolve}:fleem\" |\n> >        git cat-file --batch=\"%(objectname) %(size) %(intreepath)\"\n> >\n> >      would show:\n> >\n> >        1234abcd 17 foo/bar/baz\n> >        ../../../external\n> >\n> >      And then the caller knows that the path is not relative to the\n> >      original \"fleem\", but rather to \"foo/bar/baz\".\n> \n> Note that \"..\" will always follow the *physical* structure, so if\n> foo/bar/baz is walking over symbolic links, \"../../..\" may lead you to\n> somewhere else entirely.\n\nTrue. I had not considered that, as git does not walk over such symlinks\nat all currently. But presumably one would want it to to implement this\nkind of \"follow symlink\" logic. IOW, we cannot just look up\n\"foo/bar/baz\" in the first place, as that may not even exist in the\ntree; we may need to realize that \"foo\" is a symlink and resolve that\nfirst, then find \"bar/baz\" in the destination.\n\nWhich means that I think this has to be implemented as part of the name\nresolution (i.e., the \"^{resolve}\") proposal. cat-file could not say:\n\n  get_sha1_with_context(\"HEAD:foo/bar/baz\", sha1, &ctx);\n  if (S_ISLNK(ctx.mode))\n     ... resolve ...\n\nThe initial get_sha1 would fail if \"foo\" is a symlink. Likewise, one\ncannot implement this by querying cat-file repeatedly without asking for\neach leading prefix (so ask for \"HEAD:foo\", see if it's a link, then\n\"HEAD:foo/bar\", etc).\n\nOf course it does not _have_ to be part of the normal get_sha1 name\nresolution. But if not, it would have to reimplement the tree-walking\npart of that name resolution.\n\nThanks for giving another interesting case to consider.\n\n-Peff\n"},{"id":"260286","messageId":"1430418522.22711.22.camel@ubuntu","threadId":"39186","inReplyTo":"xmqqzj5pzdfo.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T18:28:42Z","receivedAt":"2015-04-30T18:28:42Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-04-30 at 11:19 -0700, Junio C Hamano wrote:\n> David Turner <dturner@twopensource.com> writes:\n> \n> > In no case did we do a ls-files command,...\n> \n> \"ls-tree -r\" is what I would have imagined you would be using, as\n> somebody needs to have the full repository in order to resolve the\n> symbolic links _anyway_, and that somebody does not need to have a\n> checkout in order to do so.\n\nYes, they have the full repo, but we are only exploring a small fraction\nof it. ls-tree -r would require parsing the entire thing.\n"},{"id":"260287","messageId":"20150430183226.GA2855@peff.net","threadId":"39186","inReplyTo":"1430418522.22711.22.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T18:32:26Z","receivedAt":"2015-04-30T18:32:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2015 at 11:28:42AM -0700, David Turner wrote:\n\n> On Thu, 2015-04-30 at 11:19 -0700, Junio C Hamano wrote:\n> > David Turner <dturner@twopensource.com> writes:\n> > \n> > > In no case did we do a ls-files command,...\n> > \n> > \"ls-tree -r\" is what I would have imagined you would be using, as\n> > somebody needs to have the full repository in order to resolve the\n> > symbolic links _anyway_, and that somebody does not need to have a\n> > checkout in order to do so.\n> \n> Yes, they have the full repo, but we are only exploring a small fraction\n> of it. ls-tree -r would require parsing the entire thing.\n\ngit ls-tree HEAD -- BUILD ?\n\n-Peff\n"},{"id":"260288","messageId":"1430419490.22711.30.camel@ubuntu","threadId":"39186","inReplyTo":"20150430183226.GA2855@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T18:44:50Z","receivedAt":"2015-04-30T18:44:50Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-04-30 at 14:32 -0400, Jeff King wrote:\n> On Thu, Apr 30, 2015 at 11:28:42AM -0700, David Turner wrote:\n> \n> > On Thu, 2015-04-30 at 11:19 -0700, Junio C Hamano wrote:\n> > > David Turner <dturner@twopensource.com> writes:\n> > > \n> > > > In no case did we do a ls-files command,...\n> > > \n> > > \"ls-tree -r\" is what I would have imagined you would be using, as\n> > > somebody needs to have the full repository in order to resolve the\n> > > symbolic links _anyway_, and that somebody does not need to have a\n> > > checkout in order to do so.\n> > \n> > Yes, they have the full repo, but we are only exploring a small fraction\n> > of it. ls-tree -r would require parsing the entire thing.\n> \n> git ls-tree HEAD -- BUILD ?\n\n\nThis does not actually seem to work (even with -r); it only recurses\ninto directories that are named BUILD, rather than being equivalent to\ngit ls-tree -r HEAD |grep /BUILD$.\n\nAlso, BUILD files are scattered throughout the tree, so the entire tree\nwould still need to be traversed.  At present, our monorepo is not quite\nlarge enough for this to matter (a full ls-tree only takes me 0.6s), but\nit is growing.\n"},{"id":"260289","messageId":"20150430184944.GA3661@peff.net","threadId":"39186","inReplyTo":"1430419490.22711.30.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T18:49:44Z","receivedAt":"2015-04-30T18:49:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2015 at 11:44:50AM -0700, David Turner wrote:\n\n> > git ls-tree HEAD -- BUILD ?\n> \n> This does not actually seem to work (even with -r); it only recurses\n> into directories that are named BUILD, rather than being equivalent to\n> git ls-tree -r HEAD |grep /BUILD$.\n\nAh, I thought that was what you wanted (to find specific files, not a\npattern). I think `ls-tree` doesn't understand our normal pathspecs, for\nhistorical reasons.\n\n> Also, BUILD files are scattered throughout the tree, so the entire tree\n> would still need to be traversed.  At present, our monorepo is not quite\n> large enough for this to matter (a full ls-tree only takes me 0.6s), but\n> it is growing.\n\nBut aren't you asking git to do that internally? I.e., it can limit the\ntraversal for a prefix-match, but it cannot do so for an arbitrary\nfilename. It has to open every tree. So the extra expense is really just\nthe I/O over the pipe. That's not optimal, but it is a constant factor\nslowdown from what git would do internally.\n\n-Peff\n"},{"id":"260290","messageId":"1430420422.22711.37.camel@ubuntu","threadId":"39186","inReplyTo":"20150430184944.GA3661@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T19:00:22Z","receivedAt":"2015-04-30T19:00:22Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-04-30 at 14:49 -0400, Jeff King wrote:\n> On Thu, Apr 30, 2015 at 11:44:50AM -0700, David Turner wrote:\n> \n> > > git ls-tree HEAD -- BUILD ?\n> > \n> > This does not actually seem to work (even with -r); it only recurses\n> > into directories that are named BUILD, rather than being equivalent to\n> > git ls-tree -r HEAD |grep /BUILD$.\n> \n> Ah, I thought that was what you wanted (to find specific files, not a\n> pattern). I think `ls-tree` doesn't understand our normal pathspecs, for\n> historical reasons.\n> \n> > Also, BUILD files are scattered throughout the tree, so the entire tree\n> > would still need to be traversed.  At present, our monorepo is not quite\n> > large enough for this to matter (a full ls-tree only takes me 0.6s), but\n> > it is growing.\n> \n> But aren't you asking git to do that internally? I.e., it can limit the\n> traversal for a prefix-match, but it cannot do so for an arbitrary\n> filename. It has to open every tree. So the extra expense is really just\n> the I/O over the pipe. That's not optimal, but it is a constant factor\n> slowdown from what git would do internally.\n\nNo, I'm not trying to find all BUILD files -- only ones that are in the\ntransitive dependency tree of the target I'm trying to sparsely check\nout. So if the target foo/bar/baz depends on morx/fleem, and morx/fleem\ndepends on plugh/xyzzy, then I have to examine those three places only.\nI don't have to examine anything in the gibbberish/ subtree, for\ninstance.  \n"},{"id":"260291","messageId":"20150430191043.GA4461@peff.net","threadId":"39186","inReplyTo":"1430420422.22711.37.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T19:10:43Z","receivedAt":"2015-04-30T19:10:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2015 at 12:00:22PM -0700, David Turner wrote:\n\n> > > Also, BUILD files are scattered throughout the tree, so the entire tree\n> > > would still need to be traversed.  At present, our monorepo is not quite\n> > > large enough for this to matter (a full ls-tree only takes me 0.6s), but\n> > > it is growing.\n> > \n> > But aren't you asking git to do that internally? I.e., it can limit the\n> > traversal for a prefix-match, but it cannot do so for an arbitrary\n> > filename. It has to open every tree. So the extra expense is really just\n> > the I/O over the pipe. That's not optimal, but it is a constant factor\n> > slowdown from what git would do internally.\n> \n> No, I'm not trying to find all BUILD files -- only ones that are in the\n> transitive dependency tree of the target I'm trying to sparsely check\n> out. So if the target foo/bar/baz depends on morx/fleem, and morx/fleem\n> depends on plugh/xyzzy, then I have to examine those three places only.\n> I don't have to examine anything in the gibbberish/ subtree, for\n> instance.  \n\nOK, let me see if I understand your use case by parroting it back.\n\nYou _don't_ want to feed git a \"find all BUILD\" pattern, which is good\n(because it doesn't work ;) ). You do want to feed it a set of raw\npaths to find, because you're going to discover the paths yourself at\neach step as you recurse through the dependency-chain of build files. \nYou don't actually care about feeding those paths to \"ls-tree\" at all.\nYou care only about the _content_ at each path (and will parse that\ncontent to see if you need to take a further recursive step).\n\nSo I think git out-of-the-box supports that pretty well (via cat-file).\nAnd your sticking point is that some of the paths may involve symlinks\nin the tree, so you want cat-file to answer \"if I had checked this out\nand typed cat /some/path/to/BUILD, what content would I get\". Which\nbrings us back to the original symlink question.\n\nIs that all accurate?\n\nI'm not sure that helps with the \"how to handle symlinks\" discussion,\nbut at least your goals make sense to me at this point.\n\n-Peff\n"},{"id":"260292","messageId":"1430421479.22711.50.camel@ubuntu","threadId":"39186","inReplyTo":"20150430191043.GA4461@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T19:17:59Z","receivedAt":"2015-04-30T19:17:59Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-04-30 at 15:10 -0400, Jeff King wrote:\n> On Thu, Apr 30, 2015 at 12:00:22PM -0700, David Turner wrote:\n> \n> > > > Also, BUILD files are scattered throughout the tree, so the entire tree\n> > > > would still need to be traversed.  At present, our monorepo is not quite\n> > > > large enough for this to matter (a full ls-tree only takes me 0.6s), but\n> > > > it is growing.\n> > > \n> > > But aren't you asking git to do that internally? I.e., it can limit the\n> > > traversal for a prefix-match, but it cannot do so for an arbitrary\n> > > filename. It has to open every tree. So the extra expense is really just\n> > > the I/O over the pipe. That's not optimal, but it is a constant factor\n> > > slowdown from what git would do internally.\n> > \n> > No, I'm not trying to find all BUILD files -- only ones that are in the\n> > transitive dependency tree of the target I'm trying to sparsely check\n> > out. So if the target foo/bar/baz depends on morx/fleem, and morx/fleem\n> > depends on plugh/xyzzy, then I have to examine those three places only.\n> > I don't have to examine anything in the gibbberish/ subtree, for\n> > instance.  \n> \n> OK, let me see if I understand your use case by parroting it back.\n> \n> You _don't_ want to feed git a \"find all BUILD\" pattern, which is good\n> (because it doesn't work ;) ). You do want to feed it a set of raw\n> paths to find, because you're going to discover the paths yourself at\n> each step as you recurse through the dependency-chain of build files. \n> You don't actually care about feeding those paths to \"ls-tree\" at all.\n> You care only about the _content_ at each path (and will parse that\n> content to see if you need to take a further recursive step).\n> \n> So I think git out-of-the-box supports that pretty well (via cat-file).\n> And your sticking point is that some of the paths may involve symlinks\n> in the tree, so you want cat-file to answer \"if I had checked this out\n> and typed cat /some/path/to/BUILD, what content would I get\". Which\n> brings us back to the original symlink question.\n> \n> Is that all accurate?\n\nYes.  That is a very good summary.\n"},{"id":"260293","messageId":"xmqqvbgdzaqc.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"20150430182659.GB23431@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-30T19:18:03Z","receivedAt":"2015-04-30T19:18:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Which means that I think this has to be implemented as part of the name\n> resolution (i.e., the \"^{resolve}\") proposal. cat-file could not say:\n>\n>   get_sha1_with_context(\"HEAD:foo/bar/baz\", sha1, &ctx);\n>   if (S_ISLNK(ctx.mode))\n>      ... resolve ...\n>\n> The initial get_sha1 would fail if \"foo\" is a symlink. Likewise, one\n> cannot implement this by querying cat-file repeatedly without asking for\n> each leading prefix (so ask for \"HEAD:foo\", see if it's a link, then\n> \"HEAD:foo/bar\", etc).\n>\n> Of course it does not _have_ to be part of the normal get_sha1 name\n> resolution. But if not, it would have to reimplement the tree-walking\n> part of that name resolution.\n>\n> Thanks for giving another interesting case to consider.\n\nYup, everything above makes sense, and I think it is an argument for\nmaking this new feature as part of the sha1-name infrastructure, if\nonly that it has to do some sort of tree-walking already anyway.\n\nThanks.\n"},{"id":"260294","messageId":"1430421924.22711.57.camel@ubuntu","threadId":"39186","inReplyTo":"20150430033725.GB12361@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-04-30T19:25:24Z","receivedAt":"2015-04-30T19:25:24Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 23:37 -0400, Jeff King wrote:\n>   3. Return an object with the symlink relative to the original\n>      filename (so \"../external\" in this case). This is kind of weird,\n>      though, because we're not just returning a string from the name\n>      resolution. It's an actual object.  So we'd be generating a fake\n>      object that doesn't actually exist in the object db and\n>      returning that. Feeding that sha1 to another program would fail.\n\n> I can't say that I'm excited about any of them. Perhaps you or somebody\n> else can think of a more clever solution.\n> \n> Note that the complication with (3) does come from my trying to push\n> this down into the name-resolution code.\n\nAll else being equal, I would prefer the more general solution.  But\nhere, the generality comes with a price that seems somewhat high. \n\nWhen I think about the commands that might use this, cat-file and\nls-tree are at the top of the list (although as noted, I am only likely\nto use cat-file, and it's not clear what ls-tree should do in the event\nof an out-of-repo link). \n\nI could imagine someone caring about grep and diff.  Someone who cares\nabout grep would likely want it to be willing to go out-of-worktree (as\nopposed to silently missing things).  I think we all agree that having\ngit go out-of-worktree is a mistake, so I'm not sure this use-case is\none that is supportable.\n\nThe weirdest case is log.  If I say git log HEAD^{resolve} --\nfoo/bar/baz, does it mean \"commits that have touched what is now pointed\nto by foo/bar/baz\"?  Or does it mean \"commits that have touched a thing\nthat was at that time pointed to by foo/bar/baz\"? [1]  The second one is\nmore useful, since it could not otherwise be achieved.  But I think this\nwould require additional code in log on top of whatever additional code\nis in sha1_name.  In other words, we would not get it for free just by\nadjusting sha1_name.\n\nAre there other relevant commands that I'm missing?  \n\nIf not, I think we should reconsider the original thought of just\nsupporting cat-file.  The nice thing about just supporting cat-file is\nthat for out-of-repo links we can add a special form to the output, that\ndoes not contain a sha (since there is no corresponding sha in the\nrepo).  In other words, something like your solution 3, quoted above.\n\n\n[1] See page 11 of http://inform7.com/learn/documents/WhitePaper.pdf\n(\"Has the president ever been ill?\") for a similar case.\n"},{"id":"260295","messageId":"xmqqegn1z9fp.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"1430421924.22711.57.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-30T19:46:02Z","receivedAt":"2015-04-30T19:46:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner <dturner@twopensource.com> writes:\n\n> The weirdest case is log.  If I say git log HEAD^{resolve} --\n> foo/bar/baz,...\n\nThat invocation does not make any sense to me, at least within the\ncontext of what has been discussed for ^{resolve}, which is an\ninstruction to the \"name to object name\" mapping layer to notice\nsymbolic links while it traverses the tree containment relationships\nstarting from the root of the tree to arrive at a single object\nname.\n\n    git rev-parse HEAD^{resolve}:path/that/might/involve/symlink/some/where\n    git cat-file HEAD^{resolve}:path/that/might/involve/symlink/some/where\n    git grep -e pattern HEAD^{resolve}:path/that/might/involve/symlink/some/where\n\nwould, though.  In other words, ^{resolve} that is not followed by a\ncolon and path is something entirely different from what we have\nbeen discussing.\n"},{"id":"260297","messageId":"20150430195135.GA6742@peff.net","threadId":"39186","inReplyTo":"xmqqegn1z9fp.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-04-30T19:51:35Z","receivedAt":"2015-04-30T19:51:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2015 at 12:46:02PM -0700, Junio C Hamano wrote:\n\n> David Turner <dturner@twopensource.com> writes:\n> \n> > The weirdest case is log.  If I say git log HEAD^{resolve} --\n> > foo/bar/baz,...\n> \n> That invocation does not make any sense to me, at least within the\n> context of what has been discussed for ^{resolve}, which is an\n> instruction to the \"name to object name\" mapping layer to notice\n> symbolic links while it traverses the tree containment relationships\n> starting from the root of the tree to arrive at a single object\n> name.\n> \n>     git rev-parse HEAD^{resolve}:path/that/might/involve/symlink/some/where\n>     git cat-file HEAD^{resolve}:path/that/might/involve/symlink/some/where\n>     git grep -e pattern HEAD^{resolve}:path/that/might/involve/symlink/some/where\n> \n> would, though.  In other words, ^{resolve} that is not followed by a\n> colon and path is something entirely different from what we have\n> been discussing.\n\nYeah, I agree that HEAD^{resolve} without a colon does not make any\nsense. In fact, I wanted to originally suggest a syntax that replaced\nthe colon with something else, to make it clear that it the modifier is\nreally about the colon. But I could not think of a character that was\nreadable and would not have backward-compatibility issues.\n\nI guess you could spell it:\n\n  HEAD^{resolve:foo/bar/baz}\n\nbut that opens up parsing questions for the filename. Would we allow \"}\"\nin the filename? Or require that the \"}\" balance, which means\nspeculative parsing if there is more content after the trailing \"}\"\n(e.g., you could in theory resolve to a tree and then stick a \":\" with\nmore path after that. Yech).\n\n-Peff\n"},{"id":"260301","messageId":"xmqqoam5xtyj.fsf@gitster.dls.corp.google.com","threadId":"39186","inReplyTo":"xmqqegn1z9fp.fsf@gitster.dls.corp.google.com","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-30T20:05:40Z","receivedAt":"2015-04-30T20:05:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> ... In other words, ^{resolve} that is not followed by a\n> colon and path is something entirely different from what we have\n> been discussing.\n\nHaving said that, I am not saying that such an alternative \"follow\nsymbolic links in many other places\" is a worthless suggestion.\n\nIt just does not work as an extended SHA-1 syntax.\n\n    git rev-parse HEAD^{resolve}:RelNotes HEAD:Documentation\n\nwould make sense; RelNotes, if it were a symbolic link, is resolved,\nwhile Documentation will never be.  On the other hand\n\n    git log next^{resolve} master -- Documentation\n\nwill not make any sense, as it is a totally conflicting request.\nDoes it resolve symlinks only when encountering a commit that the\ntraversal that started from 'next' happened to have reached before\nthe traversal from 'master' got there?  What should happen for\ncommits that are reachable from both?\n\nSo even if (and this is a big if) such an \"aggressive symlink\nfollowing\" mode were useful for some commands, I think the switch\nbelongs to the command, not the per-object syntax.\n"},{"id":"260307","messageId":"1430450954.22711.69.camel@ubuntu","threadId":"39186","inReplyTo":"20150430033725.GB12361@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-05-01T03:29:14Z","receivedAt":"2015-05-01T03:29:14Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2015-04-29 at 23:37 -0400, Jeff King wrote:\n> On Wed, Apr 29, 2015 at 06:45:45PM -0700, David Turner wrote:\n> \n> > On Wed, 2015-04-29 at 21:16 -0400, Jeff King wrote:\n> > > On Wed, Apr 29, 2015 at 06:06:23PM -0700, David Turner wrote:\n> > >   3. Ditto for out-of-tree. Note that this would be the _raw_ symlink\n> > >      contents, not any kind of simplification (so if you asked for\n> > >      \"foo/bar/baz\" and it was \"../../../../out\", you would the full path\n> > >      with all those dots, not a simplified \"../out\", which I think is\n> > >      what you were trying to show in earlier examples).\n> > \n> > Unfortunately, we need the simplified version, because we otherwise\n> > don't know what the ..s are relative to in the case of a link to a link:\n> > \n> >   echo content >dest ;# actual blob\n> >   mkdir -p foo/bar\n> >   ln -s foo/bar/baz fleem             # in-tree link-to-link \n> >   ln -s ../../../external foo/bar/baz # out-of-tree link\n> > \n> > If echo HEAD^{resolve}:fleem were to return ../../../external (after\n> > following the first symlink to the second), we would have lost\n> > information.\n> \n> Urgh, yeah, thanks for the counter-example.\n> \n> Here are some possible alternatives:\n> \n>   1. If we can't resolve fully, don't resolve anything. I.e., return the\n>      \"fleem\" object here, and the caller can recurse if they want. This is\n>      simple and correct, but not as helpful to somebody who wants to follow\n>      the out-of-tree link (they have to re-traverse the fleem->foo/bar/baz\n>      link themselves).\n> \n>   2. Consider it can error if resolution fails. If you ask for\n>      \"HEAD^{tree}^{commit}\", that does not resolve to anything (because\n>      we can't peel the tree to a commit). Like (1), this is simple and\n>      correct, but probably not all that helpful. The caller has to\n>      start from scratch and resolve themselves, rather than getting an\n>      intermediate result.\n> \n>   3. Return an object with the symlink relative to the original\n>      filename (so \"../external\" in this case). This is kind of weird,\n>      though, because we're not just returning a string from the name\n>      resolution. It's an actual object.  So we'd be generating a fake\n>      object that doesn't actually exist in the object db and\n>      returning that. Feeding that sha1 to another program would fail.\n> \n>   4. Return the last object we could resolve, as I described. So\n>      foo/bar/baz (with \"../../../external\" as its content) in this case.\n>      When you resolve a name, you can ask for the context we discovered\n>      along the way by traversing the tree. The mode is one example we've\n>      already discussed, but the path name is another. So something like:\n> \n>        echo \"HEAD^{resolve}:fleem\" |\n>        git cat-file --batch=\"%(objectname) %(size) %(intreepath)\"\n> \n>      would show:\n> \n>        1234abcd 17 foo/bar/baz\n>        ../../../external\n> \n>      And then the caller knows that the path is not relative to the\n>      original \"fleem\", but rather to \"foo/bar/baz\".\n> \n>      The problem is that although this context lookup is already part of\n>      get_sha1_with_context, that is not exposed through every interface.\n>      E.g., \"git rev-parse HEAD^{resolve}:fleem\" will give you an object,\n>      but you have no way of knowing the context.\n> \n> I can't say that I'm excited about any of them. Perhaps you or somebody\n> else can think of a more clever solution.\n> \n> Note that the complication with (3) does come from my trying to push\n> this down into the name-resolution code.\n\nActually, I think 4 has an insurmountable problem.  Here's the case I'm\nthinking of:\n\nln -s ..  morx\n\nImagine that we go to look up 'morx/fleem'.  Now morx is the \"last\nobject we could resolve\", but we don't know how much of our input has\nbeen consumed at this point.  So consumers don't know that after they\nexit the repo, they still need to find fleem next to it.\n"},{"id":"260308","messageId":"20150501053559.GA13393@peff.net","threadId":"39186","inReplyTo":"1430450954.22711.69.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-05-01T05:36:00Z","receivedAt":"2015-05-01T05:36:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2015 at 08:29:14PM -0700, David Turner wrote:\n\n> >   4. Return the last object we could resolve, as I described. So\n> [...]\n> \n> Actually, I think 4 has an insurmountable problem.  Here's the case I'm\n> thinking of:\n> \n> ln -s ..  morx\n> \n> Imagine that we go to look up 'morx/fleem'.  Now morx is the \"last\n> object we could resolve\", but we don't know how much of our input has\n> been consumed at this point.  So consumers don't know that after they\n> exit the repo, they still need to find fleem next to it.\n\nYes, agreed (my list was written before Andreas brought up the idea of\nsymlinks in the intermediate paths). I think to let the caller pick up\nwhere you left off, you would have to create a new string that has the\n\"remainder\" concatenated to it.\n\n-Peff\n"},{"id":"260360","messageId":"1430501355.2806.2.camel@ubuntu","threadId":"39186","inReplyTo":"20150501053559.GA13393@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-05-01T17:29:15Z","receivedAt":"2015-05-01T17:29:15Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-05-01 at 01:36 -0400, Jeff King wrote:\n> On Thu, Apr 30, 2015 at 08:29:14PM -0700, David Turner wrote:\n> \n> > >   4. Return the last object we could resolve, as I described. So\n> > [...]\n> > \n> > Actually, I think 4 has an insurmountable problem.  Here's the case I'm\n> > thinking of:\n> > \n> > ln -s ..  morx\n> > \n> > Imagine that we go to look up 'morx/fleem'.  Now morx is the \"last\n> > object we could resolve\", but we don't know how much of our input has\n> > been consumed at this point.  So consumers don't know that after they\n> > exit the repo, they still need to find fleem next to it.\n> \n> Yes, agreed (my list was written before Andreas brought up the idea of\n> symlinks in the intermediate paths). I think to let the caller pick up\n> where you left off, you would have to create a new string that has the\n> \"remainder\" concatenated to it.\n\nSince that new string does not exist in the object db, isn't that pretty\nmuch proposal 3?  We could, in this case, provide a fake sha as well\n(\"0\"*40), to make it clear that the object does not exist.\n"},{"id":"260377","messageId":"20150501201136.GA3152@peff.net","threadId":"39186","inReplyTo":"1430501355.2806.2.camel@ubuntu","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-05-01T20:11:36Z","receivedAt":"2015-05-01T20:11:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 01, 2015 at 10:29:15AM -0700, David Turner wrote:\n\n> > > Actually, I think 4 has an insurmountable problem.  Here's the case I'm\n> > > thinking of:\n> > > \n> > > ln -s ..  morx\n> > > \n> > > Imagine that we go to look up 'morx/fleem'.  Now morx is the \"last\n> > > object we could resolve\", but we don't know how much of our input has\n> > > been consumed at this point.  So consumers don't know that after they\n> > > exit the repo, they still need to find fleem next to it.\n> > \n> > Yes, agreed (my list was written before Andreas brought up the idea of\n> > symlinks in the intermediate paths). I think to let the caller pick up\n> > where you left off, you would have to create a new string that has the\n> > \"remainder\" concatenated to it.\n> \n> Since that new string does not exist in the object db, isn't that pretty\n> much proposal 3?  We could, in this case, provide a fake sha as well\n> (\"0\"*40), to make it clear that the object does not exist.\n\nYes, I think it is the same as proposal 3. Complete with all of the\nfake-object awkwardness. I'm not sure I like the fake-sha1 idea. The\ngeneral pattern for accessing an object is:\n\n  1. Turn some user-provided name into an object (get_sha1).\n\n  2. Retrieve that object content (read_sha1_file).\n\nBy pushing the symlink resolution into step 1, it \"just works\"\neverywhere. But if we hand back a fake sha1, now every call-site has to\nbe aware of it.\n\nI think the solutions range from:\n\n  a. Put resolution in get_sha1. Return an error when we can't\n     resolve. Callers are on their own to do anything else.\n\n  b. Put resolution in get_sha1. If we can't resolve, return an error.\n     If the _with_context variant is called, leave our partial result\n     string there. Some callers may choose to expose that information\n     (e.g., cat-file might), at which point the user can \"pick up\" where\n     git leaves off for out-of-tree links.\n\n  c. Forget about get_sha1. This gets implemented elsewhere (e.g., as a\n     cat-file feature as you originally proposed).\n\nCertainly (a) is tempting and simple, but my understanding of your use\ncase is that you would like to follow out-of-tree links.\n\nIt seems like (b) is the most flexible, in the sense that it would\nsolve your case, and allows \"git rev-parse HEAD^{resolve}:foo\" when the\nresult is well-formed inside the repository. But I wonder if it's\nactually worth the complexity. Without exposing the information for the\nuser to continue the traversal, it seems like only half a solution for\nthose parts of the code. And we still have to design some kind of custom\noutput for cat-file to expose the context.\n\nSo maybe (c) really is the simplest way forward. I dunno. I know that's\ncoming full circle to your original proposal. Hopefully that isn't too\ninfuriating for you. ;)\n\n-Peff\n"},{"id":"260378","messageId":"1430514575.2806.7.camel@ubuntu","threadId":"39186","inReplyTo":"20150501201136.GA3152@peff.net","subject":"Re: RFC: git cat-file --follow-symlinks?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-05-01T21:09:35Z","receivedAt":"2015-05-01T21:09:35Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-05-01 at 16:11 -0400, Jeff King wrote:\n\n> So maybe (c) really is the simplest way forward. I dunno. I know that's\n> coming full circle to your original proposal. Hopefully that isn't too\n> infuriating for you. ;)\n\nIt's exactly the opposite of infuriating.  Now I understand all the\nissues much better, and it will probably be easier for me to implement.\n"}]}