{"thread":{"id":"55738","subject":"Standardized escaping to store a .git in git?","startedAt":"2021-05-19T21:01:01Z","lastAt":"2021-05-20T03:34:18Z","messageCount":5,"participants":["Josh Triplett","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"424989","messageId":"YKV8hEAxIzolnROX@localhost","threadId":"55738","inReplyTo":null,"subject":"Standardized escaping to store a .git in git?","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2021-05-19T21:00:52Z","receivedAt":"2021-05-19T21:01:01Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On rare occasions, a project may need to store and version a .git\ndirectory in a git repository. For instance, a project that interacts\nwith git repositories may need test cases. Or, a project using git to\nstore backups may also want to back up git repositories. `.git` is the\nonly filename that git can't transparently store and version.\n\nI've seen projects take different approaches to work around this. For\ninstance, the libgit2 project renames the `.git` directory to `.gitted`,\nand then their test framework copies that to a temporary directory as\n`.git`.\n\nWould it make sense to have a standardized escaping mechanism for this,\nthat git could then standardize the handling of in a safe way (taking\nboth project configuration and local configuration into account)? Such a\nmechanism would not, by default, result in git checking out a `.git`\ndirectory verbatim, as that wouldn't be safe (due to hook scripts and\ndue to searches for .git directories), but a user could configure their\nown system to do so for a specific project, tools like `git archive`\ncould have a way to un-escape the directory in a generated archive, and\nreferences to objects within a treeish could use such paths.\nStandardizing this would allow tools to interoperate rather than each\ninventing their own convention.\n\n(Note that today, git *can* successfully check in, version, update, and\ncheck out a bare repo.git directory, just not a non-bare .git\ndirectory.)\n\nAs one possible escaping (absolutely subject to bikeshedding):\n\n- Reserve names starting with a specified character (e.g. \\x01); call\n  that escape character E.\n- Encode filenames that actually start with E to start with EE\n- Encode .git as E.git\n- Require an opt-in to interpret this escaping; tools that don't\n  interpret this escaping will still be able to operate on the files, in\n  much the same way that it's possible to operate on a symlink as if it\n  were a file containing the target path.\n\nThere are tradeoffs here: using a more type-able escape character would\nbe convenient if a user ever had to deal with the raw name, but on the\nother hand, using a more type-able escape character would make the need\nto escape the escape character come up more often.\n\nRegardless of the specific approach to escaping `.git`, does the general\nidea of standardizing such escaping across tools seem like something git\ncould potentially do, to allow transparently storing *any* file or\ndirectory in a git repository?\n\n- Josh Triplett\n"},{"id":"424990","messageId":"YKWDlF59jWoyE+xJ@google.com","threadId":"55738","inReplyTo":"YKV8hEAxIzolnROX@localhost","subject":"Re: Standardized escaping to store a .git in git?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2021-05-19T21:31:00Z","receivedAt":"2021-05-19T21:31:07Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Josh,\n\nJosh Triplett wrote:\n\n> On rare occasions, a project may need to store and version a .git\n> directory in a git repository. For instance, a project that interacts\n> with git repositories may need test cases. Or, a project using git to\n> store backups may also want to back up git repositories. `.git` is the\n> only filename that git can't transparently store and version.\n\nMy take on this might be a bit surprising, but it's probably worth\nspelling out anyway: Git is first and foremost a source code\nmanagement tool, and \".git\" directories are not a good interchange\nformat, so while I have sympathy for this use case, I do _not_ think\nthat Git should make changes that hurt other use cases in order to\nsupport it.\n\nInstead, I recommend doing one of the following, in order from most to\nleast preferred:\n\n 1. Make the test case run git commands to create a Git repository.\n    This makes it obvious what the test is trying to do, without\n    having to deal with unrelated details also recorded in \".git\".\n    This is what Git's test suite does, for example.\n\n 2. Check in a fast-import file and use \"git fast-import\" to make a\n    Git repository out of it.\n\n 3. Check in a \"git bundle\" file and use \"git clone\" to make a Git\n    repository out of it.\n\n 4. Check in an archive file (e.g., tar) containing a .git directory.\n    (I consider this preferable over checking in a .git directory\n    directly because it prevents a user from accidentally \"cd\"-ing\n    into it and running git commands within the checked-in repository\n    that they intended to run in the top-level repository.  That seems\n    especially worth preventing because the checked-in repository can\n    contain git aliases and other settings such as core.pager that\n    cause automatic code execution, as you mentioned.)\n\nThanks and hope that helps,\nJonathan\n"},{"id":"424995","messageId":"YKWMbh/j1ZiMZiGs@localhost","threadId":"55738","inReplyTo":"YKWDlF59jWoyE+xJ@google.com","subject":"Re: Standardized escaping to store a .git in git?","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2021-05-19T22:08:46Z","receivedAt":"2021-05-19T22:08:53Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Wed, May 19, 2021 at 02:31:00PM -0700, Jonathan Nieder wrote:\n> Josh Triplett wrote:\n> > On rare occasions, a project may need to store and version a .git\n> > directory in a git repository. For instance, a project that interacts\n> > with git repositories may need test cases. Or, a project using git to\n> > store backups may also want to back up git repositories. `.git` is the\n> > only filename that git can't transparently store and version.\n>\n> My take on this might be a bit surprising, but it's probably worth\n> spelling out anyway: Git is first and foremost a source code\n> management tool, and \".git\" directories are not a good interchange\n> format, so while I have sympathy for this use case, I do _not_ think\n> that Git should make changes that hurt other use cases in order to\n> support it.\n\nI absolutely agree that such changes would be entirely inappropriate if\nthey hurt other use cases. That's part of why I'm suggesting that I\ndon't think any *defaults* in git should change. My hope is more to have\nsome kind of guidance along the lines of \"if you need to do escaping, do\nit this way\", to lead towards having one canonical way to do such\nescaping rather than multiple incompatible ways.\n\nPart of my motivation, here, is that I'm looking to implement one such\nescaping mechanism (in a tool built atop libgit2 that needs to handle\nand version arbitrary files), and rather than inventing something\nbespoke I'd love to interoperate. And since I've seen various approaches\nused in the wild, I didn't want to add Yet Another distinct approach\nbefore starting a design conversation about it.\n\n> Instead, I recommend doing one of the following, in order from most to\n> least preferred:\n> \n>  1. Make the test case run git commands to create a Git repository.\n>     This makes it obvious what the test is trying to do, without\n>     having to deal with unrelated details also recorded in \".git\".\n>     This is what Git's test suite does, for example.\n> \n>  2. Check in a fast-import file and use \"git fast-import\" to make a\n>     Git repository out of it.\n> \n>  3. Check in a \"git bundle\" file and use \"git clone\" to make a Git\n>     repository out of it.\n\nFor the test-case approach, these are potentially workable, though they\nonly work if you just need a git repo with a given set of semantics,\nrather than a binary-identical test case.\n\nFor the storing-arbitrary-files case, these wouldn't apply.\n\n>  4. Check in an archive file (e.g., tar) containing a .git directory.\n>     (I consider this preferable over checking in a .git directory\n>     directly because it prevents a user from accidentally \"cd\"-ing\n>     into it and running git commands within the checked-in repository\n>     that they intended to run in the top-level repository.  That seems\n>     especially worth preventing because the checked-in repository can\n>     contain git aliases and other settings such as core.pager that\n>     cause automatic code execution, as you mentioned.)\n\nStoring as an archive is an option, but that would then require tools\nthat want to track arbitrary files to distinguish between \"tar file that\nshould be unpacked\" and \"tar file that was originally a tar file\". It's\nalso a harder format to interoperate with.\n\nTo clarify, I don't think the default behavior of git should be to\nun-escape this escaping mechanism. Rather, I think the default behavior\nshould be to treat the filenames as literal, and the user could opt in\nto un-escaping on checkout and escaping on check-in.\n"},{"id":"424997","messageId":"YKWTGMw3nShH9VKt@google.com","threadId":"55738","inReplyTo":"YKWMbh/j1ZiMZiGs@localhost","subject":"Re: Standardized escaping to store a .git in git?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2021-05-19T22:37:12Z","receivedAt":"2021-05-19T22:37:16Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJosh Triplett wrote:\n\n> Part of my motivation, here, is that I'm looking to implement one such\n> escaping mechanism (in a tool built atop libgit2 that needs to handle\n> and version arbitrary files), and rather than inventing something\n> bespoke I'd love to interoperate. And since I've seen various approaches\n> used in the wild, I didn't want to add Yet Another distinct approach\n> before starting a design conversation about it.\n\n*nod* To be clear, I'm glad you brought it up, among other reasons\nbecause it means this discussion becomes available in the list archive\nfor when people are wondering about the same thing in the future.\n\n> On Wed, May 19, 2021 at 02:31:00PM -0700, Jonathan Nieder wrote:\n\n>> Instead, I recommend doing one of the following, in order from most to\n>> least preferred:\n[...]\n> For the test-case approach, these are potentially workable, though they\n> only work if you just need a git repo with a given set of semantics,\n> rather than a binary-identical test case.\n\nFor cases wanting something binary-indentical, it still seems\npreferable to check in the individual relevant binary file (e.g., an\nindex file or a packfile) instead of a full repository.  In addition\nto the safety improvement involved, this makes the test case easier to\nunderstand.\n\n> For the storing-arbitrary-files case, these wouldn't apply.\n\nCan you say a little more about the storing-arbitrary-files case?\n\nFor example, 'bup' is a tool built on top of Git formats that stores\narbitrary files without using Git tree objects for it.  'etckeeper' is\nanother tool that stores additional information that Git does not (such\nas detailed filesystem permissions).\n\nIf you have a use case in common with other tools, then finding a way\nto interoperate sounds great. :)  The best way to do that is likely to\ndepend on the details of what the family of tools want to do.\n\nThere are some other filenames that \"git fsck\" also forbids, so this\ncomes down to more than figuring out how to handle \".git\".\n\nThanks,\nJonathan\n"},{"id":"425027","messageId":"YKXW/Elvf8l780Yp@localhost","threadId":"55738","inReplyTo":"YKWTGMw3nShH9VKt@google.com","subject":"Re: Standardized escaping to store a .git in git?","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2021-05-20T03:26:52Z","receivedAt":"2021-05-20T03:34:18Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Wed, May 19, 2021 at 03:37:12PM -0700, Jonathan Nieder wrote:\n> Josh Triplett wrote:\n> > Part of my motivation, here, is that I'm looking to implement one such\n> > escaping mechanism (in a tool built atop libgit2 that needs to handle\n> > and version arbitrary files), and rather than inventing something\n> > bespoke I'd love to interoperate. And since I've seen various approaches\n> > used in the wild, I didn't want to add Yet Another distinct approach\n> > before starting a design conversation about it.\n>\n> *nod* To be clear, I'm glad you brought it up, among other reasons\n> because it means this discussion becomes available in the list archive\n> for when people are wondering about the same thing in the future.\n>\n> > For the storing-arbitrary-files case, these wouldn't apply.\n>\n> Can you say a little more about the storing-arbitrary-files case?\n\nSure. I'm using git to record before-and-after states of running\ncommands in an isolated environment, to see the differences caused by\nthose commands. The \"before\" state includes everything the command\nneeds, and the delta from \"before\" to \"after\" is exactly what the\ncommand changed. Some commands create git repositories; for instance,\nsome software build scripts `git clone` their dependencies or other\ndata. So when I go to record the \"after\" state, it might include a .git\ndirectory. And I need to record that as transparently as possible.\n\nI'd like to use git repositories so that people *can* push and pull\ndata using git, inspect the repository with things like \"git show\", use\n\"git diff\", and similar.\n\n> There are some other filenames that \"git fsck\" also forbids, so this\n> comes down to more than figuring out how to handle \".git\".\n\nAre you talking about the case-insensitive check for paths that can be\nconfused with .git on some platforms, or something more than that?\n"}]}