threads / discuss / 63842

git clone fsync error on FSx Ontap

Subject: git clone fsync error on FSx Ontap

## tl;dr

3 messages between Jul 24, 2025 and Jul 25, 2025.

replies: 2people: 2as markdown or json

Tim Cederquist· Jul 24, 2025, 21:49 UTC · lore
What did you do before the bug happened? (Steps to reproduce your issue)
  Mounted NFS from FSx Ontap, typical home folder posix permission of
700 with inheritance. This is nfs4.2, hard mount. users home folder
has full complement of ACLs A:FD:OWNER@:rwaDxTtNnCcoy and the owner is
my account. I can write into the folders just fine, I can append,
chmod, create folders, delete, append files, etc. I can dd with fsync,
i can run install file with permissions set to 444 as indicated below
so basic IO operations similar to what git clone is doing should be
represented and are working fine.
What did you expect to happen? (Expected behavior)
  git clone https://github.com/githubtraining/hellogitworld.git
  Command should have cloned a public repository the NFS hosted home folder
What happened instead? (Actual behavior)

Cloning into 'hellogitworld'... remote: Enumerating objects: 306, done. remote: Counting objects: 100% (32/32), done. remote: Compressing objects: 100% (13/13), done. remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1) Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done. Resolving deltas: 100% (70/70), done. fatal: fsync error on '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY': Permission denied fatal: fetch-pack: invalid index-pack output

What's different between what you expected and what actually happened?
  fsync error - permission denied
Anything else you want to add:
  From Ontap storage host sectrace events command, it indicates the
failure is due to the user not having 'Append' permissions to the file
and generated the fsync error.
  Running an strace on the git clone command shows the process runs
"openat" with 444 file permission but with O_RDWR flag.
  Ontap creates the file with 444 (read only) posix permission
  git continues on to write() into the read only file - ontap rejects
it and fails the command due to read only status of the file
  I've tested with linux "instruction" command to set permission and
copy a file in a similar fashion:
    strace -f -o trace_install install -m 444 src.txt test/a/test4.txt
    This command opens the file handle 600, writes into it, and then
chmods to 444 << not using 444 to start the file as git clone is
trying to do.
  Additionally, I've added an inherited non-intrinsic permission to
the user of A:FD:tcederquist@domain:wa << this appends the required
write + append attribute and the FD means it is inherited by all files
and not overridden by posix permissions. This is not a solution but a
testable method. With this permission added, the git clone works as
expected.
  Suggestion is to use 600 on the openat/fopen for the pack index file
instead of 444. This is how 'install' sets up the file. However, I
don't know if this was an attempt at a cross platform mutex? I cannot
imagine any other reason why 444 would have been used for a file that
would have content written after opening with read only permissions.

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

[System Info]
git version 2.47.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
libcurl: 8.3.0
OpenSSL: OpenSSL 1.0.2k-fips 26 Jan 2017
zlib: 1.2.7
uname: Linux 5.10.236-228.935.amzn2.x86_64 #1 SMP Tue May 6 04:10:50
UTC 2025 x86_64
compiler info: gnuc: 7.3
libc info: glibc: 2.26
$SHELL (typically, interactive shell): /bin/bash

[Enabled Hooks] not run from a git repository - no hooks to show

brian m. carlson· Jul 24, 2025, 23:14 UTC · re: Tim Cederquist · lore

Re: git clone fsync error on FSx Ontap

On 2025-07-24 at 21:49:41, Tim Cederquist wrote:
Show 16 quoted lines
> What did you expect to happen? (Expected behavior)
>   git clone https://github.com/githubtraining/hellogitworld.git
>   Command should have cloned a public repository the NFS hosted home folder
> 
> What happened instead? (Actual behavior)
> 
> Cloning into 'hellogitworld'...
> remote: Enumerating objects: 306, done.
> remote: Counting objects: 100% (32/32), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1)
> Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done.
> Resolving deltas: 100% (70/70), done.
> fatal: fsync error on
> '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY':
> Permission denied

This is definitely a bug in your NFS server. EACCES is only a code you should see before you have a file descriptor. Once you have a file descriptor (e.g., open(2) succeeded), the proper status code if it is not suitable for your purpose is EBADF. (For instance, if you attempt to write(2) to a file open only for reading.)

However, fsync(2) should never return EBADF or EACCES on a file open for writing.

Show 14 quoted lines
> fatal: fetch-pack: invalid index-pack output
> 
> What's different between what you expected and what actually happened?
>   fsync error - permission denied
> 
> Anything else you want to add:
>   From Ontap storage host sectrace events command, it indicates the
> failure is due to the user not having 'Append' permissions to the file
> and generated the fsync error.
>   Running an strace on the git clone command shows the process runs
> "openat" with 444 file permission but with O_RDWR flag.
>   Ontap creates the file with 444 (read only) posix permission
>   git continues on to write() into the read only file - ontap rejects
> it and fails the command due to read only status of the file
Yup, this is explicitly allowed by POSIX.
Show 17 quoted lines
>   I've tested with linux "instruction" command to set permission and
> copy a file in a similar fashion:
>     strace -f -o trace_install install -m 444 src.txt test/a/test4.txt
>     This command opens the file handle 600, writes into it, and then
> chmods to 444 << not using 444 to start the file as git clone is
> trying to do.
>   Additionally, I've added an inherited non-intrinsic permission to
> the user of A:FD:tcederquist@domain:wa << this appends the required
> write + append attribute and the FD means it is inherited by all files
> and not overridden by posix permissions. This is not a solution but a
> testable method. With this permission added, the git clone works as
> expected.
>   Suggestion is to use 600 on the openat/fopen for the pack index file
> instead of 444. This is how 'install' sets up the file. However, I
> don't know if this was an attempt at a cross platform mutex? I cannot
> imagine any other reason why 444 would have been used for a file that
> would have content written after opening with read only permissions.

The goal is to create a file which has permissions honoured by the umask but is not at all writable. There is no reason to write to a pack file or loose object once it's written: the file is immutable until it's no longer useful, at which point it's removed (which does not require write permission on the file). POSIX requires that the restrictions set by the file mode be ignored when determining whether the file is open for writing, so your NFS server is not following the POSIX spec correctly.

Note this is possible to do correctly over NFS, and many servers do so, but there are also unfortunately a large number of servers which do not honour the POSIX standard correctly. In addition, this problem not only affects Git, but a wide variety of other software as well, including zsh and Emacs, as well as every other Git implementation I'm aware of, so us trying to work around it would still leave you with a server that didn't work properly with lots of software.

-- 
brian m. carlson (they/them)
Toronto, Ontario, CA
Tim Cederquist· Jul 25, 2025, 17:56 UTC · re: brian m. carlson · lore

Re: git clone fsync error on FSx Ontap

Hi brian, Thank you for your quick feedback! I was not thinking of the immutable file aspect, that's new for me and makes sense. Your feedback enabled us to press the storage team and we are discovering we had a single nic ontap that works and a dual nic ontap storage that doesn't, so it appears your assessment of it being a storage failure is right on the money.

Thank you again for your detailed response. It was very helpful! Tim

On Thu, Jul 24, 2025 at 7:14 PM brian m. carlson <sandals@crustytoothpaste.net> wrote:

Show 81 quoted lines
>
> On 2025-07-24 at 21:49:41, Tim Cederquist wrote:
> > What did you expect to happen? (Expected behavior)
> >   git clone https://github.com/githubtraining/hellogitworld.git
> >   Command should have cloned a public repository the NFS hosted home folder
> >
> > What happened instead? (Actual behavior)
> >
> > Cloning into 'hellogitworld'...
> > remote: Enumerating objects: 306, done.
> > remote: Counting objects: 100% (32/32), done.
> > remote: Compressing objects: 100% (13/13), done.
> > remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1)
> > Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done.
> > Resolving deltas: 100% (70/70), done.
> > fatal: fsync error on
> > '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY':
> > Permission denied
>
> This is definitely a bug in your NFS server.  EACCES is only a code you
> should see before you have a file descriptor.  Once you have a file
> descriptor (e.g., open(2) succeeded), the proper status code if it is
> not suitable for your purpose is EBADF.  (For instance, if you attempt
> to write(2) to a file open only for reading.)
>
> However, fsync(2) should never return EBADF or EACCES on a file open for
> writing.
>
> > fatal: fetch-pack: invalid index-pack output
> >
> > What's different between what you expected and what actually happened?
> >   fsync error - permission denied
> >
> > Anything else you want to add:
> >   From Ontap storage host sectrace events command, it indicates the
> > failure is due to the user not having 'Append' permissions to the file
> > and generated the fsync error.
> >   Running an strace on the git clone command shows the process runs
> > "openat" with 444 file permission but with O_RDWR flag.
> >   Ontap creates the file with 444 (read only) posix permission
> >   git continues on to write() into the read only file - ontap rejects
> > it and fails the command due to read only status of the file
>
> Yup, this is explicitly allowed by POSIX.
>
> >   I've tested with linux "instruction" command to set permission and
> > copy a file in a similar fashion:
> >     strace -f -o trace_install install -m 444 src.txt test/a/test4.txt
> >     This command opens the file handle 600, writes into it, and then
> > chmods to 444 << not using 444 to start the file as git clone is
> > trying to do.
> >   Additionally, I've added an inherited non-intrinsic permission to
> > the user of A:FD:tcederquist@domain:wa << this appends the required
> > write + append attribute and the FD means it is inherited by all files
> > and not overridden by posix permissions. This is not a solution but a
> > testable method. With this permission added, the git clone works as
> > expected.
> >   Suggestion is to use 600 on the openat/fopen for the pack index file
> > instead of 444. This is how 'install' sets up the file. However, I
> > don't know if this was an attempt at a cross platform mutex? I cannot
> > imagine any other reason why 444 would have been used for a file that
> > would have content written after opening with read only permissions.
>
> The goal is to create a file which has permissions honoured by the umask
> but is not at all writable.  There is no reason to write to a pack file
> or loose object once it's written: the file is immutable until it's no
> longer useful, at which point it's removed (which does not require write
> permission on the file).  POSIX requires that the restrictions set by
> the file mode be ignored when determining whether the file is open for
> writing, so your NFS server is not following the POSIX spec correctly.
>
> Note this is possible to do correctly over NFS, and many servers do so,
> but there are also unfortunately a large number of servers which do not
> honour the POSIX standard correctly.  In addition, this problem not only
> affects Git, but a wide variety of other software as well, including zsh
> and Emacs, as well as every other Git implementation I'm aware of, so us
> trying to work around it would still leave you with a server that didn't
> work properly with lots of software.
> --
> brian m. carlson (they/them)
> Toronto, Ontario, CA

← back to recent threads