# git clone fsync error on FSx Ontap 3 messages from 2025-07-24 to 2025-07-25. Participants: Tim Cederquist, brian m. carlson. Thread: https://gitlist.dev/t/63842 ## Tim Cederquist, 2025-07-24 21:49 Subject: git clone fsync error on FSx Ontap Message-ID: URL: https://gitlist.dev/e/CAFb3DeA9TwRnieajpa5S88ioc%3D65bUWh55mvDYz3CBp-xUcB3A%40mail.gmail.com ``` What did you do before the bug happened? (Steps to reproduce your issue) Mounted NFS from FSx Ontap, typical home folder posix permission of 700 with inheritance. This is nfs4.2, hard mount. users home folder has full complement of ACLs A:FD:OWNER@:rwaDxTtNnCcoy and the owner is my account. I can write into the folders just fine, I can append, chmod, create folders, delete, append files, etc. I can dd with fsync, i can run install file with permissions set to 444 as indicated below so basic IO operations similar to what git clone is doing should be represented and are working fine. What did you expect to happen? (Expected behavior) git clone https://github.com/githubtraining/hellogitworld.git Command should have cloned a public repository the NFS hosted home folder What happened instead? (Actual behavior) Cloning into 'hellogitworld'... remote: Enumerating objects: 306, done. remote: Counting objects: 100% (32/32), done. remote: Compressing objects: 100% (13/13), done. remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1) Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done. Resolving deltas: 100% (70/70), done. fatal: fsync error on '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY': Permission denied fatal: fetch-pack: invalid index-pack output What's different between what you expected and what actually happened? fsync error - permission denied Anything else you want to add: From Ontap storage host sectrace events command, it indicates the failure is due to the user not having 'Append' permissions to the file and generated the fsync error. Running an strace on the git clone command shows the process runs "openat" with 444 file permission but with O_RDWR flag. Ontap creates the file with 444 (read only) posix permission git continues on to write() into the read only file - ontap rejects it and fails the command due to read only status of the file I've tested with linux "instruction" command to set permission and copy a file in a similar fashion: strace -f -o trace_install install -m 444 src.txt test/a/test4.txt This command opens the file handle 600, writes into it, and then chmods to 444 << not using 444 to start the file as git clone is trying to do. Additionally, I've added an inherited non-intrinsic permission to the user of A:FD:tcederquist@domain:wa << this appends the required write + append attribute and the FD means it is inherited by all files and not overridden by posix permissions. This is not a solution but a testable method. With this permission added, the git clone works as expected. Suggestion is to use 600 on the openat/fopen for the pack index file instead of 444. This is how 'install' sets up the file. However, I don't know if this was an attempt at a cross platform mutex? I cannot imagine any other reason why 444 would have been used for a file that would have content written after opening with read only permissions. Please review the rest of the bug report below. You can delete any lines you don't wish to share. [System Info] git version 2.47.1 cpu: x86_64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh libcurl: 8.3.0 OpenSSL: OpenSSL 1.0.2k-fips 26 Jan 2017 zlib: 1.2.7 uname: Linux 5.10.236-228.935.amzn2.x86_64 #1 SMP Tue May 6 04:10:50 UTC 2025 x86_64 compiler info: gnuc: 7.3 libc info: glibc: 2.26 $SHELL (typically, interactive shell): /bin/bash [Enabled Hooks] not run from a git repository - no hooks to show ``` ## brian m. carlson, 2025-07-24 23:14 Subject: Re: git clone fsync error on FSx Ontap Message-ID: URL: https://gitlist.dev/e/aIK-YVZWWAjHiIxE%40fruit.crustytoothpaste.net In-Reply-To: ``` On 2025-07-24 at 21:49:41, Tim Cederquist wrote: > What did you expect to happen? (Expected behavior) > git clone https://github.com/githubtraining/hellogitworld.git > Command should have cloned a public repository the NFS hosted home folder > > What happened instead? (Actual behavior) > > Cloning into 'hellogitworld'... > remote: Enumerating objects: 306, done. > remote: Counting objects: 100% (32/32), done. > remote: Compressing objects: 100% (13/13), done. > remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1) > Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done. > Resolving deltas: 100% (70/70), done. > fatal: fsync error on > '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY': > Permission denied This is definitely a bug in your NFS server. EACCES is only a code you should see before you have a file descriptor. Once you have a file descriptor (e.g., open(2) succeeded), the proper status code if it is not suitable for your purpose is EBADF. (For instance, if you attempt to write(2) to a file open only for reading.) However, fsync(2) should never return EBADF or EACCES on a file open for writing. > fatal: fetch-pack: invalid index-pack output > > What's different between what you expected and what actually happened? > fsync error - permission denied > > Anything else you want to add: > From Ontap storage host sectrace events command, it indicates the > failure is due to the user not having 'Append' permissions to the file > and generated the fsync error. > Running an strace on the git clone command shows the process runs > "openat" with 444 file permission but with O_RDWR flag. > Ontap creates the file with 444 (read only) posix permission > git continues on to write() into the read only file - ontap rejects > it and fails the command due to read only status of the file Yup, this is explicitly allowed by POSIX. > I've tested with linux "instruction" command to set permission and > copy a file in a similar fashion: > strace -f -o trace_install install -m 444 src.txt test/a/test4.txt > This command opens the file handle 600, writes into it, and then > chmods to 444 << not using 444 to start the file as git clone is > trying to do. > Additionally, I've added an inherited non-intrinsic permission to > the user of A:FD:tcederquist@domain:wa << this appends the required > write + append attribute and the FD means it is inherited by all files > and not overridden by posix permissions. This is not a solution but a > testable method. With this permission added, the git clone works as > expected. > Suggestion is to use 600 on the openat/fopen for the pack index file > instead of 444. This is how 'install' sets up the file. However, I > don't know if this was an attempt at a cross platform mutex? I cannot > imagine any other reason why 444 would have been used for a file that > would have content written after opening with read only permissions. The goal is to create a file which has permissions honoured by the umask but is not at all writable. There is no reason to write to a pack file or loose object once it's written: the file is immutable until it's no longer useful, at which point it's removed (which does not require write permission on the file). POSIX requires that the restrictions set by the file mode be ignored when determining whether the file is open for writing, so your NFS server is not following the POSIX spec correctly. Note this is possible to do correctly over NFS, and many servers do so, but there are also unfortunately a large number of servers which do not honour the POSIX standard correctly. In addition, this problem not only affects Git, but a wide variety of other software as well, including zsh and Emacs, as well as every other Git implementation I'm aware of, so us trying to work around it would still leave you with a server that didn't work properly with lots of software. -- brian m. carlson (they/them) Toronto, Ontario, CA ``` ## Tim Cederquist, 2025-07-25 17:56 Subject: Re: git clone fsync error on FSx Ontap Message-ID: URL: https://gitlist.dev/e/CAFb3DeD34EO3sYjinCzHVb8odO5%2Bk13HNQ6m7Ru5KOCFP-nSGg%40mail.gmail.com In-Reply-To: ``` Hi brian, Thank you for your quick feedback! I was not thinking of the immutable file aspect, that's new for me and makes sense. Your feedback enabled us to press the storage team and we are discovering we had a single nic ontap that works and a dual nic ontap storage that doesn't, so it appears your assessment of it being a storage failure is right on the money. Thank you again for your detailed response. It was very helpful! Tim On Thu, Jul 24, 2025 at 7:14 PM brian m. carlson wrote: > > On 2025-07-24 at 21:49:41, Tim Cederquist wrote: > > What did you expect to happen? (Expected behavior) > > git clone https://github.com/githubtraining/hellogitworld.git > > Command should have cloned a public repository the NFS hosted home folder > > > > What happened instead? (Actual behavior) > > > > Cloning into 'hellogitworld'... > > remote: Enumerating objects: 306, done. > > remote: Counting objects: 100% (32/32), done. > > remote: Compressing objects: 100% (13/13), done. > > remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1) > > Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done. > > Resolving deltas: 100% (70/70), done. > > fatal: fsync error on > > '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY': > > Permission denied > > This is definitely a bug in your NFS server. EACCES is only a code you > should see before you have a file descriptor. Once you have a file > descriptor (e.g., open(2) succeeded), the proper status code if it is > not suitable for your purpose is EBADF. (For instance, if you attempt > to write(2) to a file open only for reading.) > > However, fsync(2) should never return EBADF or EACCES on a file open for > writing. > > > fatal: fetch-pack: invalid index-pack output > > > > What's different between what you expected and what actually happened? > > fsync error - permission denied > > > > Anything else you want to add: > > From Ontap storage host sectrace events command, it indicates the > > failure is due to the user not having 'Append' permissions to the file > > and generated the fsync error. > > Running an strace on the git clone command shows the process runs > > "openat" with 444 file permission but with O_RDWR flag. > > Ontap creates the file with 444 (read only) posix permission > > git continues on to write() into the read only file - ontap rejects > > it and fails the command due to read only status of the file > > Yup, this is explicitly allowed by POSIX. > > > I've tested with linux "instruction" command to set permission and > > copy a file in a similar fashion: > > strace -f -o trace_install install -m 444 src.txt test/a/test4.txt > > This command opens the file handle 600, writes into it, and then > > chmods to 444 << not using 444 to start the file as git clone is > > trying to do. > > Additionally, I've added an inherited non-intrinsic permission to > > the user of A:FD:tcederquist@domain:wa << this appends the required > > write + append attribute and the FD means it is inherited by all files > > and not overridden by posix permissions. This is not a solution but a > > testable method. With this permission added, the git clone works as > > expected. > > Suggestion is to use 600 on the openat/fopen for the pack index file > > instead of 444. This is how 'install' sets up the file. However, I > > don't know if this was an attempt at a cross platform mutex? I cannot > > imagine any other reason why 444 would have been used for a file that > > would have content written after opening with read only permissions. > > The goal is to create a file which has permissions honoured by the umask > but is not at all writable. There is no reason to write to a pack file > or loose object once it's written: the file is immutable until it's no > longer useful, at which point it's removed (which does not require write > permission on the file). POSIX requires that the restrictions set by > the file mode be ignored when determining whether the file is open for > writing, so your NFS server is not following the POSIX spec correctly. > > Note this is possible to do correctly over NFS, and many servers do so, > but there are also unfortunately a large number of servers which do not > honour the POSIX standard correctly. In addition, this problem not only > affects Git, but a wide variety of other software as well, including zsh > and Emacs, as well as every other Git implementation I'm aware of, so us > trying to work around it would still leave you with a server that didn't > work properly with lots of software. > -- > brian m. carlson (they/them) > Toronto, Ontario, CA ```