Re: [PATCH 1/4] docs: add a question on syncing repositories to the FAQ

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Mar 15, 2021, 20:40 UTC
Message-ID: <YE/GNoqt1Ja2xZU3@camp.crustytoothpaste.net>
In-Reply-To: <87eeh0l52n.fsf@evledraar.gmail.com>

On 2021-02-28 at 13:01:04, Ævar Arnfjörð Bjarmason wrote:

Show 8 quoted lines

> make this more understandable by going to the heart of the matter, i.e.:
> 
>  * We prefer push/pull/bundle to copy/replicate .git content
> 
>  * Regardless, a .git directory can be copied across systems just fine
>    if you recursively guarantee snapshot integrity, e.g. it doesn't
>    depend on the endian-ness of the OS, or has anything like symlinks in
>    there made by git itself.

I'll revise to make this clearer.

Show 10 quoted lines

>  * Anything which copies .git data on a concurrently updated repo can
>    lead to corruption, whether that's cp -R, rsync with any combination
>    of flags, some cloud syncing service that expects to present that
>    tree to two computers without guaranteeing POSIX fs semantics between
>    the two etc.
> 
>  * A common pitfall with such copying of a .git directory is that file
>    deletions are also critical, e.g. rsync without --delete is almost
>    guaranteed to produce a corrupt .git if repeated enough times
>    (e.g. git might prefer stale loose refs over now-packed ones).

I'll include that as well.

>  * It's OK to copy .git between system that differ in their support of
>    symbolic links, but the work tree may be in an inconsistent state and
>    need some manner of "git reset" to repair it.

And this.

Show 9 quoted lines

> And, not sure if this is correct:
> 
>  * It may be OK to edit a .git directory on a non-POSIX conforming fs
>    (but perhaps validate the result with "git fsck"). But it's not OK to
>    have two writing git processes work on such a repository at the same
>    time. Keep in mind that certain operations and default settings (such
>    as background gc, see `gc.autoDetach` in linkgit:git-config[1]) might
>    result in two processes working on the directory even if you're
>    changing it only in one terminal window at a time.

I'll reflect the fact that only one process may modify the repository at a time.

Show 6 quoted lines

> I.e. to go a bit beyond the docs you have of basically saying "there be
> dragons in non-POSIX" and describe the particular scenarios where it can
> go wrong. Something like the above still leaves the door open to users
> using cloud syncing services, which they can then judge for themselves
> as being OK or not. I'm sure there's some that are far from POSIX
> compliance that are OK in practice if the above warnings are observed.

Unfortunately, it's a bit hard to make a concise entry that explains all the different ways that using a non-POSIX compliant file system can break things. We've seen NFS where open(2) is broken, both with permissions and O_EXCL; cloud syncing services that restore files that have been deleted; DAV mounts that don't support the necessary semantics; systems where O_APPEND doesn't have POSIX semantics; and a whole host of other sadness. I don't therefore think that we want to tell people that we think that using a file system that doesn't support POSIX semantics is okay, because in most cases, they are not.

Users frequently try to judge for themselves what works and then they try one of the above things which clearly does not, so saying, "Try it and see if it breaks," just tends to result in users complaining about repository corruption later on. I'm really tired of answering these same questions again and again and telling users that their repositories are hosed and that they've lost data, so I want to be definitive that we don't recommend or support these various broken environments and that users should not use them. It may be in rare cases that users have extensive knowledge about the behavior of their file systems and Git's requirements and can make a sound judgment to use a non-POSIX file system, but the people on the planet who can do that effectively are almost all Git developers. I will state that "File systems that don't provide POSIX semantics, such as DAV mounts, shouldn't be used without fully understanding the situation and requirements," which I think is the most generous recommendation we can safely give.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US