Standardized escaping to store a .git in git?

On rare occasions, a project may need to store and version a .git directory in a git repository. For instance, a project that interacts with git repositories may need test cases. Or, a project using git to store backups may also want to back up git repositories. `.git` is the only filename that git can't transparently store and version.

I've seen projects take different approaches to work around this. For instance, the libgit2 project renames the `.git` directory to `.gitted`, and then their test framework copies that to a temporary directory as `.git`.

Would it make sense to have a standardized escaping mechanism for this, that git could then standardize the handling of in a safe way (taking both project configuration and local configuration into account)? Such a mechanism would not, by default, result in git checking out a `.git` directory verbatim, as that wouldn't be safe (due to hook scripts and due to searches for .git directories), but a user could configure their own system to do so for a specific project, tools like `git archive` could have a way to un-escape the directory in a generated archive, and references to objects within a treeish could use such paths. Standardizing this would allow tools to interoperate rather than each inventing their own convention.

(Note that today, git *can* successfully check in, version, update, and check out a bare repo.git directory, just not a non-bare .git directory.)

As one possible escaping (absolutely subject to bikeshedding):

- Reserve names starting with a specified character (e.g. \x01); call
  that escape character E.
- Encode filenames that actually start with E to start with EE
- Encode .git as E.git
- Require an opt-in to interpret this escaping; tools that don't
  interpret this escaping will still be able to operate on the files, in
  much the same way that it's possible to operate on a symlink as if it
  were a file containing the target path.

There are tradeoffs here: using a more type-able escape character would be convenient if a user ever had to deal with the raw name, but on the other hand, using a more type-able escape character would make the need to escape the escape character come up more often.

Regardless of the specific approach to escaping `.git`, does the general idea of standardizing such escaping across tools seem like something git could potentially do, to allow transparently storing *any* file or directory in a git repository?

- Josh Triplett

Hi Josh,

Josh Triplett wrote:

My take on this might be a bit surprising, but it's probably worth spelling out anyway: Git is first and foremost a source code management tool, and ".git" directories are not a good interchange format, so while I have sympathy for this use case, I do _not_ think that Git should make changes that hurt other use cases in order to support it.

Instead, I recommend doing one of the following, in order from most to least preferred:

 1. Make the test case run git commands to create a Git repository.
    This makes it obvious what the test is trying to do, without
    having to deal with unrelated details also recorded in ".git".
    This is what Git's test suite does, for example.

 2. Check in a fast-import file and use "git fast-import" to make a
    Git repository out of it.

 3. Check in a "git bundle" file and use "git clone" to make a Git
    repository out of it.

 4. Check in an archive file (e.g., tar) containing a .git directory.
    (I consider this preferable over checking in a .git directory
    directly because it prevents a user from accidentally "cd"-ing
    into it and running git commands within the checked-in repository
    that they intended to run in the top-level repository.  That seems
    especially worth preventing because the checked-in repository can
    contain git aliases and other settings such as core.pager that
    cause automatic code execution, as you mentioned.)

Thanks and hope that helps, Jonathan

On Wed, May 19, 2021 at 02:31:00PM -0700, Jonathan Nieder wrote:

I absolutely agree that such changes would be entirely inappropriate if they hurt other use cases. That's part of why I'm suggesting that I don't think any *defaults* in git should change. My hope is more to have some kind of guidance along the lines of "if you need to do escaping, do it this way", to lead towards having one canonical way to do such escaping rather than multiple incompatible ways.

Part of my motivation, here, is that I'm looking to implement one such escaping mechanism (in a tool built atop libgit2 that needs to handle and version arbitrary files), and rather than inventing something bespoke I'd love to interoperate. And since I've seen various approaches used in the wild, I didn't want to add Yet Another distinct approach before starting a design conversation about it.

For the test-case approach, these are potentially workable, though they only work if you just need a git repo with a given set of semantics, rather than a binary-identical test case.

For the storing-arbitrary-files case, these wouldn't apply.

Storing as an archive is an option, but that would then require tools that want to track arbitrary files to distinguish between "tar file that should be unpacked" and "tar file that was originally a tar file". It's also a harder format to interoperate with.

To clarify, I don't think the default behavior of git should be to un-escape this escaping mechanism. Rather, I think the default behavior should be to treat the filenames as literal, and the user could opt in to un-escaping on checkout and escaping on check-in.

Hi,

Josh Triplett wrote:

*nod* To be clear, I'm glad you brought it up, among other reasons because it means this discussion becomes available in the list archive for when people are wondering about the same thing in the future.

[...]

For cases wanting something binary-indentical, it still seems preferable to check in the individual relevant binary file (e.g., an index file or a packfile) instead of a full repository. In addition to the safety improvement involved, this makes the test case easier to understand.

Can you say a little more about the storing-arbitrary-files case?

For example, 'bup' is a tool built on top of Git formats that stores arbitrary files without using Git tree objects for it. 'etckeeper' is another tool that stores additional information that Git does not (such as detailed filesystem permissions).

If you have a use case in common with other tools, then finding a way to interoperate sounds great. :) The best way to do that is likely to depend on the details of what the family of tools want to do.

There are some other filenames that "git fsck" also forbids, so this comes down to more than figuring out how to handle ".git".

Thanks, Jonathan

On Wed, May 19, 2021 at 03:37:12PM -0700, Jonathan Nieder wrote:

Sure. I'm using git to record before-and-after states of running commands in an isolated environment, to see the differences caused by those commands. The "before" state includes everything the command needs, and the delta from "before" to "after" is exactly what the command changed. Some commands create git repositories; for instance, some software build scripts `git clone` their dependencies or other data. So when I go to record the "after" state, it might include a .git directory. And I need to record that as transparently as possible.

I'd like to use git repositories so that people *can* push and pull data using git, inspect the repository with things like "git show", use "git diff", and similar.

Are you talking about the case-insensitive check for paths that can be confused with .git on some platforms, or something more than that?