{"thread":{"id":"55955","subject":"Using .gitignore symbolic links?","startedAt":"2021-06-18T02:34:43Z","lastAt":"2021-07-03T17:38:12Z","messageCount":5,"participants":["Tessa L. H. Lovelace","Robert Karszniewicz","Ævar Arnfjörð Bjarmason","Jeff King","Tessa L."],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"427805","messageId":"1623983680.3494.0@smtp.dreamhost.com","threadId":"55955","inReplyTo":null,"subject":"Using .gitignore symbolic links?","fromName":"Tessa L. H. Lovelace","fromEmail":"tessa@assorted.tech","sentAt":"2021-06-18T02:34:40Z","receivedAt":"2021-06-18T02:34:43Z","isPatch":false,"sender":{"key":"tessa@assorted.tech","avatar":null},"body":"The recent release candidate of Git (v2.32.0) hit my OS this week, and \nit included a line () on symbolic links for several specific files are \nnow ignored.\n\nThank you for putting the changelogs in an accessible location, knowing \nthat this was a known breaking change was useful in debugging why my \nworkflows stopped working.\n\nI have two concerns.\n\nFirst, the error thrown is\n\n > \"warning: unable to access '.gitignore': Too many levels of symbolic \nlinks\",\n\n,,,which does not accurately represent what is happening.\n\nI spent a bit of time convinced that I'd broken something with the \nsymbolic links during setup, and an error such as \"symbolic linking no \nlonger allowed for 'filename'.\" would make more sense, given the change \nunder discussion eliminates *any* use of symbolic links.\n\n\nSecondly, and more personally important to me, a system administrator:\nMy repositories use symbolic links to allow a single .gitignore file to \ndefine my folder structure, allowing me to avoid hardcoding the \nrepo-specific folder paths into my configs.\n\nIs there a flag to disable this new behavior?\n\nIf not, this change means I need to update dozens of files, duplicates \nall, or completely rewrite my .gitignore files to have shyteloads of \narbitrary file paths in them, which I'd rather not do.\n\nAlso, is there a justification for forcing this as the on-update \ndefault new behavior, when a user-querying behavior (such as with 'git \npull' defaults as they've changed recently) exists?\n\n---\n\nref \nhttps://github.com/git/git/commit/142430338477d9d1bb25be66267225fb58498d92#diff-eae5facd145e2748250f7b275e45cb001c0b8e2c47c529a4e28bbfa208e5fb59R7\n\n\n===\n\n\nThoughts?\n\n-- \nTessa L. H. Lovelace\n----\noffice:\t\t503.893.9709\nconsulting:\tassorted.tech\n\n"},{"id":"427812","messageId":"YMxAteKYn0qf/YNj@BDZ","threadId":"55955","inReplyTo":"1623983680.3494.0@smtp.dreamhost.com","subject":"Re: Using .gitignore symbolic links?","fromName":"Robert Karszniewicz","fromEmail":"avoidr@posteo.de","sentAt":"2021-06-18T06:44:05Z","receivedAt":"2021-06-18T06:44:11Z","isPatch":false,"sender":{"key":"avoidr@posteo.de","avatar":null},"body":"On Thu, Jun 17, 2021 at 07:34:40PM -0700, Tessa L. H. Lovelace wrote:\n> Secondly, and more personally important to me, a system administrator:\n> My repositories use symbolic links to allow a single .gitignore file to \n> define my folder structure, allowing me to avoid hardcoding the \n> repo-specific folder paths into my configs.\n> \n> Is there a flag to disable this new behavior?\n> \n> If not, this change means I need to update dozens of files, duplicates \n> all, or completely rewrite my .gitignore files to have shyteloads of \n> arbitrary file paths in them, which I'd rather not do.\n\nHmm, it sounds like `core.excludesFile` described in git-config(1) could\ndo what you need:\n\n  core.excludesFile\n      Specifies the pathname to the file that contains patterns to\n      describe paths that are not meant to be tracked, in addition to\n      .gitignore (per-directory) and .git/info/exclude. Defaults to\n      $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set\n      or empty, $HOME/.config/git/ignore is used instead. See\n      gitignore(5).\n\nRegards,\nRobert Karszniewicz\n"},{"id":"427817","messageId":"87o8c34dq6.fsf@evledraar.gmail.com","threadId":"55955","inReplyTo":"1623983680.3494.0@smtp.dreamhost.com","subject":"Re: Using .gitignore symbolic links?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-18T11:15:46Z","receivedAt":"2021-06-18T11:24:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 17 2021, Tessa L. H. Lovelace wrote:\n\n> The recent release candidate of Git (v2.32.0) hit my OS this week, and\n> it included a line () on symbolic links for several specific files are \n> now ignored.\n>\n> Thank you for putting the changelogs in an accessible location,\n> knowing that this was a known breaking change was useful in debugging\n> why my workflows stopped working.\n>\n> I have two concerns.\n>\n> First, the error thrown is\n>\n>> \"warning: unable to access '.gitignore': Too many levels of symbolic\n>   links\",\n>\n> ,,,which does not accurately represent what is happening.\n>\n> I spent a bit of time convinced that I'd broken something with the\n> symbolic links during setup, and an error such as \"symbolic linking no \n> longer allowed for 'filename'.\" would make more sense, given the\n> change under discussion eliminates *any* use of symbolic links.\n>\n>\n> Secondly, and more personally important to me, a system administrator:\n> My repositories use symbolic links to allow a single .gitignore file\n> to define my folder structure, allowing me to avoid hardcoding the \n> repo-specific folder paths into my configs.\n>\n> Is there a flag to disable this new behavior?\n>\n> If not, this change means I need to update dozens of files, duplicates\n> all, or completely rewrite my .gitignore files to have shyteloads of \n> arbitrary file paths in them, which I'd rather not do.\n>\n> Also, is there a justification for forcing this as the on-update\n> default new behavior, when a user-querying behavior (such as with 'git \n> pull' defaults as they've changed recently) exists?\n\n[CC-ing Jeff]\n\nBreaking this was intentional, see https://github.com/git/git/commit/2ef579e261\n\nThat doesn't mean we can't take it back.\n\nAs discussed by Robert's reply and in that commit there's the workaround\nof .git/info/exclude and the core.excludesFile.\n\nHowever, we realize that sucks for many users. Let's say you have a\nscript to clone a \"tree\" of repositories similar to but not using\ngit-submodule (or they live side-by-side), such a thing won't Just Work\nanymore.\n\nAt the end of the day there's an inherent conflict here between security\nand convenience. We really want a repository to be safe to just \"git\nclone\", i.e. we don't set up any hooks, execute code etc.; these\ngitattributes and gitignore issues were on edges of that.\n\nWe can make it work as before, but it gets hard to distinguish the\ngitignore you mean, from a gitignore that's pointing to /dev/urandom\n(annoying), or to some crafted out-of-tree thing that'll cause an\noverflow in the parser and an RCE.\n\nAny way out of that that's configurable is going to be be the same\nopt-in problem as core.excludesFile is now.\n\nSo I'd think our options are basically:\n\n 1) Do nothing, it sucks for some people (like you) but we think it's worth it\n\n 2) Some DWYM middle ground, e.g. we could discover if the link points\n    to another git repo, and only trust it then, or if it's in the\n    user's $HOME or whatever.\n\n 3) Bring back the old behavior, it was more of a \"while we're at it for\n    gitattributes...\" fix than something specifically a problem with\n    gitignore, the RCE threat is a hypothetical, and we can more easily\n    audit/be confident in the gitignore parser, probably...\n"},{"id":"427818","messageId":"YMyX2dZ1Ut40hb1L@coredump.intra.peff.net","threadId":"55955","inReplyTo":"87o8c34dq6.fsf@evledraar.gmail.com","subject":"Re: Using .gitignore symbolic links?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-18T12:55:53Z","receivedAt":"2021-06-18T12:55:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 18, 2021 at 01:15:46PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Breaking this was intentional, see https://github.com/git/git/commit/2ef579e261\n> \n> That doesn't mean we can't take it back.\n> \n> As discussed by Robert's reply and in that commit there's the workaround\n> of .git/info/exclude and the core.excludesFile.\n\nI'd prefer not to undo it because of the security implications that led\nus there in the first place. And for the most part, there should be a\nbetter way to accomplish the same things:\n\n  - these symlinked files were already subtly broken; Git was not\n    following the links when it read them from the index or a tree.\n\n  - repetitive links within a tree can be refactored to use patterns in\n    the top-level .gitignore, .gitattributes, etc. It may be that our\n    pattern language is not sufficient for some cases, but improving\n    that seems like a better path forward.\n\n  - links that span repos can use core.excludesFile or similar (and\n    conditional config can help enable them only when you want to). It\n    may also be that this could be extended to cover more cases (e.g.,\n    you can only have one configured excludesFile, but you may want\n    several). Or just a symlink from .git/info/exclude if there's one\n    per repo.\n\nI'd be curious to hear if any of those solutions don't help in this\ncase.\n\n> At the end of the day there's an inherent conflict here between security\n> and convenience. We really want a repository to be safe to just \"git\n> clone\", i.e. we don't set up any hooks, execute code etc.; these\n> gitattributes and gitignore issues were on edges of that.\n> \n> We can make it work as before, but it gets hard to distinguish the\n> gitignore you mean, from a gitignore that's pointing to /dev/urandom\n> (annoying), or to some crafted out-of-tree thing that'll cause an\n> overflow in the parser and an RCE.\n\nI agree with all of this, but I would soften the \"RCE\" part a bit. An\nuntrusted repository can already feed whatever it wants into the parser.\nThe danger of symlinks is that accessing out-of-tree paths may cause\nunexpected results (information disclosure in some situations, but also\nweirdness when opening files in /dev, /proc, etc).\n\n> Any way out of that that's configurable is going to be be the same\n> opt-in problem as core.excludesFile is now.\n> \n> So I'd think our options are basically:\n> \n>  1) Do nothing, it sucks for some people (like you) but we think it's worth it\n\nI hope the \"it sucks\" is \"the transition sucks\", but they're still able\nto configure Git differently to achieve the same goals in a roughly\nsimilar way. Again, I'd be curious to hear about cases where this isn't\ntrue.\n\nI'm not completely opposed to having a config switch for \"allow\ngitignore symlinks\" as an escape hatch, as long as the default is still\n\"off\". One of the things I don't like about it is that the config option\nneeds to come with a warning explaining how the result is still subtly\nbroken.\n\n>  2) Some DWYM middle ground, e.g. we could discover if the link points\n>     to another git repo, and only trust it then, or if it's in the\n>     user's $HOME or whatever.\n\nWe've talked before about identifying out-of-tree symlinks. It's not\nclear to me in this case if the symlinks are to other paths within the\nrepository, or if they go out-of-tree.\n\nIn-tree symlinks are OK. It's just complicated and error-prone to detect\nthem (because of course interior paths may themselves be symlinks).\n\nI think we'd always want to forbid out-of-tree symlinks, no matter what\nthey're pointing to (because we don't have any idea what's \"safe\" in the\nuser's filesystem). It's easier both us and the user to just have a\nswitch for \"look at these symlinks anyway\".\n\n>  3) Bring back the old behavior, it was more of a \"while we're at it for\n>     gitattributes...\" fix than something specifically a problem with\n>     gitignore, the RCE threat is a hypothetical, and we can more easily\n>     audit/be confident in the gitignore parser, probably...\n\nHopefully it's obvious at this point that I'd prefer not to go that\nroute. :)\n\nTessa mentioned one other thing, which is somewhat orthogonal to the\noptions you listed. The error message is just:\n\n  warning: unable to access '.gitignore': Too many levels of symbolic links\n\nThis comes from a generic open-or-warn function. The kernel is giving us\nELOOP, which we feed to strerror(). And it's _technically_ true, in that\nwe allow 0 levels of symbolic links. But we could perhaps intercept\nELOOP in the gitignore and gitattributes code to produce a more coherent\nwarning.\n\nTBH, I didn't give too much thought to user experience in the original\npatches because my digging showed that using symlinks for these files\nwas exceedingly rare (at least on the corpus of GitHub repos I scanned,\nbut of course all the world is not hosted on GitHub, and there will\nalways be edge cases anyway).\n\n-Peff\n"},{"id":"429130","messageId":"81d63ed42d5c7b442cff616dcc2766bebc265381.camel@assorted.tech","threadId":"55955","inReplyTo":"87o8c34dq6.fsf@evledraar.gmail.com","subject":"Re: Using .gitignore symbolic links?","fromName":"Tessa L.","fromEmail":"tessa@assorted.tech","sentAt":"2021-07-03T17:29:33Z","receivedAt":"2021-07-03T17:38:12Z","isPatch":false,"sender":{"key":"tessa@assorted.tech","avatar":null},"body":"Robert, Peff, and Ævar,\n\nThanks for the responses on this.\n\nWill attempt to better-articulate both my use case and my concerns.\n\n\n--- \n\nFirst, an attempt to provide clarity regarding my use case.\n\nThis is a hypothetical representational of my project's structure\n(symlinks designated by '->'):\n```\n..\n.\n./.gitignore  -> ./doc/.gitignore\n./doc\n./doc/readme.md\n./doc/.gitignore\n./main\n./main/.gitignore -> ../doc/.gitignore./main/00_textFile.src\n./main/00_textfile.src~\n./main/00_textFile.src~\n./main/01_differenTextFile.src\n./static\n./static/.gitignore -> ../doc/.gitignore\n./static/executeable.sh\n./static/executeable.sh~\n```\nThe contents of these listed files (aside from the .gitignore, as\narticulated below) don't actually matter, though it's possibly relevant\nthat the tilde-differentiated files are the default \"temp/backup\" file\nfor GNU nano.\n\nBy creating symbolic links between the single file in ./doc/.gitignore,\nand other directories (repository root, ./main, ./static, etc...) means\nrepo-wide ignoring of nano's backup files can be added with the single\nline:\n```\n*~\n```\n...in my ./doc/.gitignore. And just like that, I've eliminated the\nentire visible clutter of those backup files from my filetree.\n\n---\n\n\nHaving said that, it's worth noting that I was, also, trying to solve\nthe \"...but adding the metaphorical '*~' to my gitignore on every new\nproject is a pain, and then if there's another global-ish file-pattern\nto squash, updating each and every repo is a pain, too...\" problem.\n\nThank you, Robert, for calling my attention to core.excludesFile,\nthat's going to be horrifically misused in my homelab sometime soon-\nish.\n\n\n---\n\nHaving said *that*, I did not have visibility on commit 2ef579e261, and\nam especially in favour of the mentality of \"Let's do X instead of Y so\nthat (the) two cases behave consistently.\"\n\nThe subtle difference between my actual case of (ab)using symlinks to\npoint at a single 'ignore' file within the repo (vs linking from\noutside the repo) is definitely not within the supported use of git as\na version control system, and I'm not expecting to halt progress\nbecause of some edge case use like this.\n\nI'd rather come up with a better path than \"take it back\", if that\nmakes sense.\n\n---\n\n\nThat being said, I have concerns about the implemented approach here.\n\nAs a former information security professional, I'm comfortable saying\nthat the presented dichotomy of the \"inherent conflict between security\nand convenience\" ignores how effective communication and user-consent\nfixes the problem.\n\nArbitrarily breaking system-wide expectations for a low-level utility\n(symlinking) in the name of security (it's still a hypothetical RCE,\nright? There's not currently a working proof of this exploit?)\nseems...problematic...in that it prioritizes something that *may*\nhappen over pushing breaking changes down the pipe and assuming that\nthe user will just figure it out...when often even highly technical\nusers completely miss that something has changed, much less on what\nlevel.\n\nObviously, security has to happen at all levels...and, a bash-\ncompatible system has literally countably infinite amount of ways to\ncause harm, if you're running someone else's code without attempting\nsome form of analysis beforehand.\n\n\nI, and others who use bash or similar shells, are going to expect\nsymlinks to work consistently both inside git repos and out of it, and\n'what if a malicious repo symlinks to Something Bad' seems...outside\nthe scope of doing version control. \n\nTrying to detect exploits at this level (the version control system)\nseems like a lot of complexity to add for something that is\nfundamentally at odds with the philosophy of 'do one thing well'. And,\nchoosing to pursue and define what is or is not an acceptable use of\nsymlinks, or even changing behavior based on internal/external linking,\nsounds a lot like scope creep to me.\n\n\nI'm not saying that to downplay the seriousness of the security\nconcern...but hopefully to contextualize my deeper concerns about a\nchoice to not just start breaking the (working, system-level-\nconsistent) defaults, but to do so without somehow informing the user\nand giving them a form of choice regarding the change.\n\n\n---\n\nI acknowledge that 'too many levels of symlinks' is technically valid\nfor 'zero levels allowed', but that's not what is functionally\ncommunicated to the end user. I debated about calling out this\ndistinction, but decided to orient on how a less-technical user would\nperceive the error.\n\n\nIn that mindset (of keeping an eye on what a less-technical user will\nhave perceive), I've watched with happy interest the process of\nadjusting the defaults for various commands (git pull and git init,\nespecially).\n\nThe excellent use of user-communicating blocks of text in those cases\nwhile preserving the in-use legacy defaults, while allowing an informed\nchoice (eg, presenting the user with a suggestion to run specific\ncommand(s) to change the fast-forward behavior or to rename the default\nbranch on init, respectively) seems like a much better path to me.\n\n\nIf nothing else, I'd like to see that model of user-\nquerying/informing/consenting behaviour (from the fast-forward and init\nexamples) happen with this case, for consistency at the very least.\n\n\n---\n\nBack to my specific use case.\n\nAll three of the potential solutions subtly miss my need, so my\nsuggestion for a 'fourth option' would look something like a flag to\nprompt a 'flat' (non-tree) interpretation of the file(s) inside the\nrepo when filtering the displayed file-list via gitignore/excludesFile.\n\nSo, instead of checking if each folder has a rule matching against it,\nany rule in the .gitignore (or wherever the core.excludesFile points)\napplies to the base-level of any directory inside the repo, resulting\nin essentially the same behaviour.\n\nThis would eliminate my specific use of symlinks entirely, though it\ndoesn't touch on my concern about symlinks behaving differently inside\nversion control than pretty much everywhere else inside a symlink-\ncapable filesystem.\n\n\nI don't have visibility on the complexities of adjusting/adding this,\nso please correct my assumptions where they conflict with the realities\nof developing the next release candidate.\n\n\nOnce again, I appreciate your time and communication on this.\n\n--\nTessa L.\n\noffice: 503.893.9709\nweb: \thttps://assorted.tech\n\nOn Fri, 2021-06-18 at 13:15 +0200, Ævar Arnfjörð Bjarmason wrote:\n> On Thu, Jun 17 2021, Tessa L. H. Lovelace wrote:\n> \n> > The recent release candidate of Git (v2.32.0) hit my OS this week,\n> > and\n> > it included a line () on symbolic links for several specific files\n> > are \n> > now ignored.\n> > \n> > Thank you for putting the changelogs in an accessible location,\n> > knowing that this was a known breaking change was useful in\n> > debugging\n> > why my workflows stopped working.\n> > \n> > I have two concerns.\n> > \n> > First, the error thrown is\n> > \n> > > \"warning: unable to access '.gitignore': Too many levels of\n> > > symbolic\n> > \n> >   links\",\n> > \n> > ,,,which does not accurately represent what is happening.\n> > \n> > I spent a bit of time convinced that I'd broken something with the\n> > symbolic links during setup, and an error such as \"symbolic linking\n> > no \n> > longer allowed for 'filename'.\" would make more sense, given the\n> > change under discussion eliminates *any* use of symbolic links.\n> > \n> > \n> > Secondly, and more personally important to me, a system\n> > administrator:\n> > My repositories use symbolic links to allow a single .gitignore\n> > file\n> > to define my folder structure, allowing me to avoid hardcoding the \n> > repo-specific folder paths into my configs.\n> > \n> > Is there a flag to disable this new behavior?\n> > \n> > If not, this change means I need to update dozens of files,\n> > duplicates\n> > all, or completely rewrite my .gitignore files to have shyteloads\n> > of \n> > arbitrary file paths in them, which I'd rather not do.\n> > \n> > Also, is there a justification for forcing this as the on-update\n> > default new behavior, when a user-querying behavior (such as with\n> > 'git \n> > pull' defaults as they've changed recently) exists?\n> \n> [CC-ing Jeff]\n> \n> Breaking this was intentional, see \n> https://github.com/git/git/commit/2ef579e261\n> \n> That doesn't mean we can't take it back.\n> \n> As discussed by Robert's reply and in that commit there's the\n> workaround\n> of .git/info/exclude and the core.excludesFile.\n> \n> However, we realize that sucks for many users. Let's say you have a\n> script to clone a \"tree\" of repositories similar to but not using\n> git-submodule (or they live side-by-side), such a thing won't Just\n> Work\n> anymore.\n> \n> At the end of the day there's an inherent conflict here between\n> security\n> and convenience. We really want a repository to be safe to just \"git\n> clone\", i.e. we don't set up any hooks, execute code etc.; these\n> gitattributes and gitignore issues were on edges of that.\n> \n> We can make it work as before, but it gets hard to distinguish the\n> gitignore you mean, from a gitignore that's pointing to /dev/urandom\n> (annoying), or to some crafted out-of-tree thing that'll cause an\n> overflow in the parser and an RCE.\n> \n> Any way out of that that's configurable is going to be be the same\n> opt-in problem as core.excludesFile is now.\n> \n> So I'd think our options are basically:\n> \n>  1) Do nothing, it sucks for some people (like you) but we think it's\n> worth it\n> \n>  2) Some DWYM middle ground, e.g. we could discover if the link\n> points\n>     to another git repo, and only trust it then, or if it's in the\n>     user's $HOME or whatever.\n> \n>  3) Bring back the old behavior, it was more of a \"while we're at it\n> for\n>     gitattributes...\" fix than something specifically a problem with\n>     gitignore, the RCE threat is a hypothetical, and we can more\n> easily\n>     audit/be confident in the gitignore parser, probably...\n> \n\n"}]}