{"thread":{"id":"21999","subject":"git's fascination with absolute paths","startedAt":"2009-12-21T18:42:02Z","lastAt":"2009-12-22T17:21:27Z","messageCount":5,"participants":["J Chapman Flack","Avery Pennarun","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"130222","messageId":"4B2FC17A.3010705@math.purdue.edu","threadId":"21999","inReplyTo":null,"subject":"git's fascination with absolute paths","fromName":"J Chapman Flack","fromEmail":"jflack@math.purdue.edu","sentAt":"2009-12-21T18:42:02Z","receivedAt":"2009-12-21T18:42:02Z","isPatch":false,"sender":{"key":"jflack@math.purdue.edu","avatar":null},"body":"\nHi list,\n\nI have a requirement involving the reasonably common Unix-y design\npattern where a directory owned by a particular user lives in a\ndirectory that user can't access, with a setuid gate that chdirs to\nthe right place and then drops to the real user's id to do certain\nallowed things.\n\nI wanted to use git for some of those allowed things and I can't,\nbecause the code seems to call make_absolute_path on approximately\neverything, and this is one of the situations that illustrates why it\nisn't safe to assume you can get an absolute path that's even\nusable (let alone race-free) corresponding to a relative path\nin general.\n\ngit init tries to do this on the db path (even if\nit's specified explicitly in GIT_DIR), and even if I do the git init\nas root in advance, all the other git subcommands I've tried also\ntry to do it and fail when they chdir up out of the\nworking directory and try to chdir (not fchdir) back in.\n\nIn general it seems best for a program to stay free of assumptions\nabout absolute paths except when there is a specific functional\nrequirement that needs them.  I assume there is something git does\nthat requires it to have this limitation, but it's not intuitive\nto me if I just think about what I expect an scm system to do.\nI've searched on 'absolute' in the list archive to see if there\nwas a past discussion like \"we've decided we need absolute paths\neverywhere because X\" but I didn't find any.  Can someone\ndescribe what the reasoning was?  A security concern perhaps?\n(And one more serious than the race condition built into\nmake_absolute_path?)\n\nOr, perhaps I should be asking, what is there in git that will\nbreak if I recompile it with make_absolute_path(p){return p;}?\nDoes it store absolute paths in the db?  Would a recompiled\nversion produce a db other gits couldn't read?\n\n(Or less drastic perhaps, what if there were a version of m_a_p\nthat still returned an absolute path when possible and safe, and\njust returned p otherwise so the program could still be usable?)\n\nThanks,\nChapman Flack\nmathematics, purdue university\n"},{"id":"130225","messageId":"32541b130912211409j540928c0g8e944fcc05c44f82@mail.gmail.com","threadId":"21999","inReplyTo":"4B2FC17A.3010705@math.purdue.edu","subject":"Re: git's fascination with absolute paths","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-12-21T22:09:29Z","receivedAt":"2009-12-21T22:09:29Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Dec 21, 2009 at 1:42 PM, J Chapman Flack <jflack@math.purdue.edu> wrote:\n> In general it seems best for a program to stay free of assumptions\n> about absolute paths except when there is a specific functional\n> requirement that needs them.  I assume there is something git does\n> that requires it to have this limitation, but it's not intuitive\n> to me if I just think about what I expect an scm system to do.\n> I've searched on 'absolute' in the list archive to see if there\n> was a past discussion like \"we've decided we need absolute paths\n> everywhere because X\" but I didn't find any.  Can someone\n> describe what the reasoning was?  A security concern perhaps?\n> (And one more serious than the race condition built into\n> make_absolute_path?)\n\nI think it's probably just because it's easier to deal with absolute\npaths than relative ones.  Those \"..\" things can be annoying,\nparticularly inside scripts, etc, and git uses a lot of scripts.  Much\nmore straightforward to just normalize all the paths once and be sure\nthere are no weird dots in them.\n\nNot to say that it can't be done... just that it seems nobody has been\ninspired to do so.  I'm guessing most of the existing developers still\nwon't be inspired to do it based on your rather unusual use case;\nhowever, they might accept patches.\n\n> Or, perhaps I should be asking, what is there in git that will\n> break if I recompile it with make_absolute_path(p){return p;}?\n> Does it store absolute paths in the db?  Would a recompiled\n> version produce a db other gits couldn't read?\n\nYou might try this and then see what happens in 'make test'.  I\nimagine a set of clean patches that removed a lot of assumptions about\nabsolute paths, without breaking any unit tests, would be something\nworth considering for integration into git.\n\nHave fun,\n\nAvery\n"},{"id":"130228","messageId":"7v637z6ehg.fsf@alter.siamese.dyndns.org","threadId":"21999","inReplyTo":"32541b130912211409j540928c0g8e944fcc05c44f82@mail.gmail.com","subject":"Re: git's fascination with absolute paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-12-22T00:26:19Z","receivedAt":"2009-12-22T00:26:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Avery Pennarun <apenwarr@gmail.com> writes:\n\n> On Mon, Dec 21, 2009 at 1:42 PM, J Chapman Flack <jflack@math.purdue.edu> wrote:\n>> In general it seems best for a program to stay free of assumptions\n>> about absolute paths except when there is a specific functional\n>> requirement that needs them.  I assume there is something git does\n>> that requires it to have this limitation, but it's not intuitive\n>> to me if I just think about what I expect an scm system to do.\n>> I've searched on 'absolute' in the list archive to see if there\n>> was a past discussion like \"we've decided we need absolute paths\n>> everywhere because X\" but I didn't find any.  Can someone\n>> describe what the reasoning was?  A security concern perhaps?\n>> (And one more serious than the race condition built into\n>> make_absolute_path?)\n>\n> I think it's probably just because it's easier to deal with absolute\n> paths than relative ones.  Those \"..\" things can be annoying,\n> particularly inside scripts, etc, and git uses a lot of scripts.  Much\n> more straightforward to just normalize all the paths once and be sure\n> there are no weird dots in them.\n\nNot really.  The scripts can work with \"..\" just fine, as long as they\nknow how to use \"cd_to_topdir\" and \"rev-parse --show-prefix\" correctly.\n\nWhile I do not necessarily agree with the original claim that hiding\nhigher level of hierarchies are \"standard\" practice in UNIX (it instead\nfalls into \"an unusual set-up that is permitted but you have to be\ncareful\" category), I don't think it is fundamental that we need read\naccess all the way up to the root level.  It is only that getcwd(3) does.\n\nAt the basic work tree and index operations operate relative to the root\nof the work tree.  Originally, almost no privision was made to run from a\nsubdirectory of a work tree (you were expected to run from the top-level,\nhaving ./.git as the meta information sture), and we didn't have to run\nany getcwd(3).  Later we added \"look at parent directories until we find\nthe one that has .git subdirectory, while remembering how many levels we\nwent up\", in order to support operations from a subdirectory of a work\ntree.  The commands chdir up to the root of the work tree and would use\nthe path they climbed as a pathspec to limit the scope of their operation.\n\nWhile \"counting how many levels we went up\" can be expressed by a sequence\nof \"../\", turning it to the directory prefix means at some point you would\nneed to do what getcwd(3) does.  It wants to be able to read \"..\" to give\nyou an absolute path.\n\nBy rewriting that part of the \"root-level-discovery\" code to do something\nlike\n\n - while test -d .git is not true:\n   - stat(\".\") to get the inum;\n   - chdir(\"..\"); and\n   - opendir(\".\") and readdir() to find where we were;\n\nwhile going up every level, you should be able to construct the prefix\nwithout being to able to read all the way up to the filesystem root.  You\nonly need to be able to read your work tree.\n\nAdmittedly the code complexity got worse later when we added support for\nGIT_WORK_TREE and also GIT_CEILING_DIRECTORIES, as they fundamentally need\nto know where you are relative to the root of the filesystem tree and need\na working getcwd(3) support, which J Chapman's set-up refuses to give.\n\nAlso I wouldn't be surprised if the support for these two features cheat\nin order to reduce code complexity by always using absolute paths even in\nplaces where a path relative to the root of the work tree might have\nsufficed.\n"},{"id":"130230","messageId":"7vy6kv4j2u.fsf@alter.siamese.dyndns.org","threadId":"21999","inReplyTo":"7v637z6ehg.fsf@alter.siamese.dyndns.org","subject":"Re: git's fascination with absolute paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-12-22T06:30:01Z","receivedAt":"2009-12-22T06:30:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Not really.  The scripts can work with \"..\" just fine, as long as they\n> know how to use \"cd_to_topdir\" and \"rev-parse --show-prefix\" correctly.\n>\n> While I do not necessarily agree with the original claim that hiding\n> higher level of hierarchies are \"standard\" practice in UNIX (it instead\n> falls into \"an unusual set-up that is permitted but you have to be\n> careful\" category), I don't think it is fundamental that we need read\n> access all the way up to the root level.  It is only that getcwd(3) does.\n>\n> At the basic level, work tree and index operations operate relative to the root\n> of the work tree.  Originally, almost no privision was made to run from a\n> subdirectory of a work tree (you were expected to run from the top-level,\n> having ./.git as the meta information sture), and we didn't have to run\n> any getcwd(3).  Later we added \"look at parent directories until we find\n> the one that has .git subdirectory, while remembering how many levels we\n> went up\", in order to support operations from a subdirectory of a work\n> tree.  The commands chdir up to the root of the work tree and would use\n> the path they climbed as a pathspec to limit the scope of their operation.\n>\n> While \"counting how many levels we went up\" can be expressed by a sequence\n> of \"../\", turning it to the directory prefix means at some point you would\n> need to do what getcwd(3) does.  It wants to be able to read \"..\" to give\n> you an absolute path.\n>\n> By rewriting that part of the \"root-level-discovery\" code to do something\n> like\n>\n>  - while test -d .git is not true:\n>    - stat(\".\") to get the inum;\n>    - chdir(\"..\"); and\n>    - opendir(\".\") and readdir() to find where we were;\n>\n> while going up every level, you should be able to construct the prefix\n> without being to able to read all the way up to the filesystem root.  You\n> only need to be able to read your work tree.\n>\n> Admittedly the code complexity got worse later when we added support for\n> GIT_WORK_TREE and also GIT_CEILING_DIRECTORIES, as they fundamentally need\n> to know where you are relative to the root of the filesystem tree and need\n> a working getcwd(3) support, which J Chapman's set-up refuses to give.\n>\n> Also I wouldn't be surprised if the support for these two features cheat\n> in order to reduce code complexity by always using absolute paths even in\n> places where a path relative to the root of the work tree might have\n> sufficed.\n\nClarificaiton.\n\nThe above, like many other messages from me, was not meant as a\njustification, but a mere explanation of the historical fact.  IOW, don't\nget me wrong by interpreting that I am not interested in seeing a solution\nthat does not use absolute paths.\n\nWhile I think the original \"higher levels in the filesystems may not be\naccessible\" is a rather unusual set-up, making paths absolute and relying\non being able to always do so have another drawback in a not-so-unusual\nsetup.  A work tree that is shallow (say, has only one t/ subdirectory and\nshort filenames) may not be usable if it is so deep in the filesystem\nhierarchy that the result of getcwd(3) exceeds PATH_MAX.  The \"hand roll\nwhat getcwd(3) did in traditional UNIX while looking for the root level of\nthe work tree\" approach I outlined in the previous message will be a way\nto fix such a use case; as long as the deepest level of your work tree\nrelative to the root of the work tree does not exceed PATH_MAX, you'll be\nOk.\n\nWe have a few known issues in the GIT_WORK_TREE (IIRC, it has a funny\ninteraction with alias expansion).  When we reexamine the codepath that\nthe introduction of the feature needed to touch, I would love to see us at\nleast try to see if it is feasible to redo this without calling getcwd(3)\nwhen no GIT_WORK_TREE (or core.worktree) is set.\n"},{"id":"130243","messageId":"4B310017.8030009@math.purdue.edu","threadId":"21999","inReplyTo":"7vy6kv4j2u.fsf@alter.siamese.dyndns.org","subject":"Re: git's fascination with absolute paths","fromName":"J Chapman Flack","fromEmail":"jflack@math.purdue.edu","sentAt":"2009-12-22T17:21:27Z","receivedAt":"2009-12-22T17:21:27Z","isPatch":false,"sender":{"key":"jflack@math.purdue.edu","avatar":null},"body":"\nJunio C Hamano wrote:\n > Clarificaiton.\n >\n > The above, like many other messages from me, was not meant as a\n > justification, but a mere explanation of the historical fact.  IOW,\n > don't get me wrong by interpreting that I am not interested in seeing\n > a solution that does not use absolute paths.\n\nNo worries - a sense of the history is exactly the kind of response\nI was hoping for. :)\n\n > While I think the original \"higher levels in the filesystems may not\n > be accessible\" is a rather unusual set-up, making paths absolute and\n > relying on being able to always do so have another drawback in a\n > not-so-unusual setup.  A work tree that is shallow (say, has only one\n > t/ subdirectory and short filenames) may not be usable if it is so\n > deep in the filesystem hierarchy that the result of getcwd(3) exceeds\n > PATH_MAX.  The \"hand roll\n\nThis is funny; I think in my own career I've seen applications that keep\nsensitive data in subtrees restricted at the top somewhat routinely (which\nmight not mean \"really often in absolute terms\" but something more like\n\"whenever it made sense for the app\") going back at least as far as uucp\n(IIRC) ... Solaris Zones are set up that way too (when viewed from the\nglobal zone).  By the same token, while I'd never be surprised to hear\nof someone with deep hierarchies that exceed PATH_MAX, I'm not sure I've\never actually seen it happen myself.\n\nI don't intend that as the start of a \"which case is more unusual\"\ncomparison, I think it just illustrates the difficulty in making such\njudgments of usualness, as different people's career trajectories expose\nthem to very different things. I'd rather spend the mental effort trying\nto extract whatever general principle can be used to code robustly so the\nfewest judgments of usualness need to be made. Here the general principle\n(which I think you've already kind of stated yourself, so I'm not trying\nto preach but just to finish the thought) is that, for various reasons,\ntrying to transform a relative into an absolute path is not always well\ndefined, can't even always be done, can have implementation-dependent\nside effects and race conditions, and ought to be an operation that gets\npulled out of the arsenal only with deliberation and only for specific\npaths that must be made absolute if that's the only way to satisfy some\nknown functional requirement of the app.\n\nIt might be tied in to the principle of least astonishment, just by\nassuming that whatever path the caller, user, or admin provides is\nprobably the path s/he wants you to use, for any number of possible\nreasons that you don't need to be able to foresee.\n\n >> By rewriting that part of the \"root-level-discovery\" code to do\n >> something like\n >>\n >>  - while test -d .git is not true:\n >>    - stat(\".\") to get the inum;\n >>    - chdir(\"..\"); and\n >>    - opendir(\".\") and readdir() to find where we were;\n >>\n >> while going up every level, you should be able to construct the prefix\n >> without being to able to read all the way up to the filesystem root. \n  You\n >> only need to be able to read your work tree.\n\nYes, that's exactly what I would have suggested for the root level\ndiscovery.  As long as you can find .git before reaching any inaccessible\nancestor, life is good.  (And if you can't it's a perfectly good reason\nto give up without astonishing the user.)\n\nIt would still be better to open \".\" once at the beginning and\nfchdir back to it, rather than trying to chdir back to the\nconstructed path string (an inherent race condition), just as the\nNotes section in linux getcwd(3) says.\n\nAn interesting point, the chdir ../opendir/readdir algorithm you\ngive above is no longer necessarily what getcwd does, though it\ntraditionally was. These days there's often a kernel name cache\nthat getcwd gets at through a syscall or /proc.  You can tell,\nbecause when I tried to use 'git init' in my situation, the error\nwas not from getcwd but from the later access() call done on the\nfull path that getcwd successfully returned.\n\n   [there's a side issue: access(2) isn't for what a lot of people\n   think it's for. It's a rather esoteric call for testing access\n   by the real ids instead of the effective ones, and it's needed\n   in code that (a) runs set{u,g}id AND (b) wants to confirm that\n   a user-specified file is really something the user has rights to.\n   Using access() routinely just to ask \"can I open this\" has\n   a couple of problems: (1) it has a race condition and gives you\n   less information than just trying the open and testing errno;\n   (2) if anybody down the road does try to use your program or\n   code under set{u,g}id circumstances, astonishing failures can\n   result. Even for its intended purpose access() suffers from a\n   race; current OSes make it fairly easy to just drop to the real\n   user's IDs, try the open, and test errno, so access() is a bit\n   of a dinosaur.]\n\nAnyway, it would also be possible to make use of the modern\noptimized getcwd in the root-level-discovery algorithm. If getcwd\ngives a result you can just follow it backwards until you find .git,\nconfirming in each step that you can stat the name of the child and\nthe inums match (an O(1) test instead of O(directory-size) for readdir).\nJust don't follow it back past the directory containing .git. The\nslower fallback code is only needed for less modern systems in case\ngetcwd returns EACCES.  But now we're in the realm of optimization;\nthe traditional loop does the trick.\n\n(well, one nice feature of the modern approach, beyond speed, is that\nit only needs x on the directories up to .git and not r.  but since\nI'm doubtful anything else git does would actually work without r,\nthat's probably moot.)\n\n >> Admittedly the code complexity got worse later when we added support\n >> for GIT_WORK_TREE and also GIT_CEILING_DIRECTORIES, as they\n >> fundamentally need to know where you are relative to the root of the\n >> filesystem tree and need a working getcwd(3) support\n\nThat's the kind of thing I wondered about, are there particular\nfeatures that genuinely require an absolute path?  I'm a git newbie\nand I don't know what these features do, so I can't comment. (This\nproject was going to be my excuse for learning git, but rcs actually\nsuffices and I need to get it done.)\n\nDo these features actually need to traverse the full path, or just\nto know what it is?  On a system with modern getcwd that can return\nthe path even if it isn't traversable, could they make use of that?\n\n-Chap\n"}]}