git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] refs: work around network caching on Windows

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jul 15, 2022, 08:30 UTC
Message-ID
<220715.86a69a22oy.gmgdl@evledraar.gmail.com>
In-Reply-To
<pull.1291.git.1657872416216.gitgitgadget@gmail.com>
On Fri, Jul 15 2022, Johannes Schindelin via GitGitGadget wrote:
Show 33 quoted lines
> From: Pierre Garnier <pgarnier@mega.com>
>
> Network shares sometimes use aggressive caching, in which case a
> just-created directory might not be immediately visible to Git.
>
> One symptom of this scenario is the following error:
>
> 	$ git tag -a -m "automatic tag creation"  test_dir/test_tag
> 	fatal: cannot lock ref 'refs/tags/test_dir/test_tag': unable to resolve reference 'refs/tags/test_dir/test_tag': Not a directory
>
> Note: This does not necessarily happen in all Windows setups. One setup
> where it _did_ happen is a Windows Server 2019 VM, and as hinted in
>
> 	http://woshub.com/slow-network-shared-folder-refresh-windows-server/
>
> the following commands worked around it:
>
> 	Set-SmbClientConfiguration -DirectoryCacheLifetime 0
> 	Set-SmbClientConfiguration -FileInfoCacheLifetime 0
> 	Set-SmbClientConfiguration -FileNotFoundCacheLifetime 0
>
> This would impact performance negatively, though, as it essentially
> turns off all caching, therefore we do not want to require users to do
> that just to be able to use Git on Windows.
>
> The underlying culprit is that `GetFileAttributesExW()` that is called from
> `mingw_lstat()` can raise an error `ERROR_PATH_NOT_FOUND`, which is
> translated to `ENOTDIR`, as opposed to `ENOENT` as expected on Linux.
>
> Therefore, when trying to read a ref, let's allow `ENOTDIR` in addition
> to `ENOENT` to indicate that said ref is missing.
>
> This fixes https://github.com/git-for-windows/git/issues/3727

This really has much wider implications, as we hard depend on POSIX semantics in various other places. E.g. we'll the SHA-1 collision detection sanity check (not sha1dc, the "does it exist?") would be racy on such a system, wouldn't it?

Show 32 quoted lines
>  refs/files-backend.c  | 2 +-
>  refs/packed-backend.c | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/refs/files-backend.c b/refs/files-backend.c
> index 8db7882aacb..b2a880f62f0 100644
> --- a/refs/files-backend.c
> +++ b/refs/files-backend.c
> @@ -381,7 +381,7 @@ stat_ref:
>  	if (lstat(path, &st) < 0) {
>  		int ignore_errno;
>  		myerr = errno;
> -		if (myerr != ENOENT || skip_packed_refs)
> +		if ((myerr != ENOENT && myerr != ENOTDIR) || skip_packed_refs)
>  			goto out;
>  		if (refs_read_raw_ref(refs->packed_ref_store, refname, oid,
>  				      referent, type, &ignore_errno)) {
> diff --git a/refs/packed-backend.c b/refs/packed-backend.c
> index 97b68377673..23d478627a7 100644
> --- a/refs/packed-backend.c
> +++ b/refs/packed-backend.c
> @@ -480,7 +480,7 @@ static int load_contents(struct snapshot *snapshot)
>  
>  	fd = open(snapshot->refs->path, O_RDONLY);
>  	if (fd < 0) {
> -		if (errno == ENOENT) {
> +		if (errno == ENOENT || errno == ENOTDIR) {
>  			/*
>  			 * This is OK; it just means that no
>  			 * "packed-refs" file has been written yet,
>
> base-commit: bbea4dcf42b28eb7ce64a6306cdde875ae5d09ca

So I'm skeptical that this can work at all, but in any case wrapping this non-POSIX hack in an #ifdef for the relevant platform is somtething I really think we should have here, or "#ifdef NON_POSIX_FS_HACK" or something.

You don't want to be carefully reviewing this code thinking wtf, only to discover later that it's impossible on a well-behaved FS.

Also, NFS has similar options (which I've seen hard break git repos & corrupt them in the past)< how do its various dangerous caching options behave in these scenarios?

IOW if we're supporting non-POSIX behavior on platform A, are we inadvertently making the non-POSIX behavior on platform B even worse? Even more of a reason to wrap it in ifdefs...

But I really think the answer to this is similar to brian's FAQ patches for git repos on "cloud mounts", I.e. document carefully that it's likely to corrupt your repo in unexpected ways.

So I'd be much more comfortable with a workaround that stole what we do for the *.lock spinning here, i.e. we'd detect this errno, say "wtf, non-POSIX?" then spin for N ms, and hope to get "past the race".

That would be guaranteed not to suffer from odd corruption issues (as the behavior wouldn't change, we'd just wait and hope to "catch up")>

Wouldn't that be narrower & better here?
Previous: Junio C HamanoNext: Johannes Schindelin via GitGitGadget
Message 5 of 9 in “refs: work around network caching on Windows”
  1. refs: work around network caching on WindowsJohannes Schindelin via GitGitGadget, Jul 15, 2022
  2. Eric SunshineJul 15, 2022
  3. Johannes SchindelinJul 15, 2022
  4. Junio C HamanoJul 15, 2022
  5. Ævar Arnfjörð BjarmasonJul 15, 2022
  6. lstat(mingw): correctly detect ENOTDIR scenariosJohannes Schindelin via GitGitGadget, Jul 28, 2022
  7. Junio C HamanoJul 28, 2022
  8. Eric SunshineJul 28, 2022
  9. lstat(mingw): correctly detect ENOTDIR scenariosJohannes Schindelin via GitGitGadget, Jul 29, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.