# [Bug] Git ReadOnly Temp Packfile Causes "Bad file descriptor" And -13 Access Error With NFSv4 4 messages from 2025-02-10 to 2025-02-11. Participants: Maloney, Bryan, brian m. carlson, Jeff King. Thread: https://gitlist.dev/t/62931 ## Maloney, Bryan, 2025-02-10 15:56 Subject: [Bug] Git ReadOnly Temp Packfile Causes "Bad file descriptor" And -13 Access Error With NFSv4 Message-ID: <17DE42DF-D2A3-4E25-9198-5AF3CA516E35@amazon.com> URL: https://gitlist.dev/e/17DE42DF-D2A3-4E25-9198-5AF3CA516E35%40amazon.com In-Reply-To: <18589D54-5E7F-434D-A890-B39102ECB293@amazon.com> ``` ### Error Kernel logs: ˋˋˋ NFSv4: state recovery failed for open file pack/tmp_pack_aR0Mu3, error = -13 ˋˋˋ Git clone output: ˋˋˋ fatal: write error: Bad file descriptor, 137.31 MiB | 45.77 MiB/s fatal: fetch-pack: invalid index-pack output ˋˋˋ ### Context The following error is seen when running git clone over NFSv4 and a failover, or server restart, occurs: ˋˋˋ NFSv4: state recovery failed for open file pack/tmp_pack_aR0Mu3, error = -13 ˋˋˋ This error is an access denied error that happens when you try to open a file with insufficient permissions. In this case the file being opened is a read only file and it is attempted to be opened with write access. Git opens/creates this file with the O_RDWR flag but then applies read only permissions to it, 0444. Since the permissions are changed after the file is opened, the file handle works fine. However if the file was attempted to be re-opened with that same file handle we would see a -13 error. This is what we see following a failover in NFSv4. When clients reclaim their open files, the NFS server re-evaluates the file access. ˋˋˋ     int mode = 0444;     git_path_buf(temp_filename, "objects/%s", pattern);     fd = git_mkstemp_mode(temp_filename->buf, mode); ˋˋˋ - https://github.com/git/git/blob/master/object-file.c#L478 ˋˋˋ         fd = open(pattern, O_CREAT | O_EXCL | O_RDWR, mode); ˋˋˋ - https://github.com/git/git/blob/master/wrapper.c#L491 This is an issue for active/passive HA file servers. Since NFSv4 evaluates file permissions at the time of opening a file, this FD will always get an access denied error if a failover occurs during git clone. ### Reproducer On the client, mount the NFS share with NFSv4 and run git clone using the NFS mount as the target directory. ˋˋˋ root@ip-10-0-85-241:/home/ubuntu# mount -t nfs -o nfsvers=4.2,nconnect=16 10.0.66.58:/share /nfsmnt root@ip-10-0-85-241:/home/ubuntu# mkdir -p /nfsmnt/react root@ip-10-0-85-241:/home/ubuntu# git clone -v https://github.com/facebook/react.git /nfsmnt/react Cloning into '/nfsmnt/react'... remote: Enumerating objects: 356868, done. remote: Counting objects: 100% (543/543), done. remote: Compressing objects: 100% (256/256), done. fatal: write error: Bad file descriptor, 137.31 MiB | 45.77 MiB/s fatal: fetch-pack: invalid index-pack output root@ip-10-0-85-241:/home/ubuntu# grep NFS /var/log/kern.log 2025-02-07T21:54:42.562725+00:00 ip-10-0-85-241 kernel: NFSv4: state recovery failed for open file pack/tmp_pack_LkbrKa, error = -13 ˋˋˋ On the server, restart NFS server client is on the “Receiving objects” step of git clone. ˋˋˋ # systemctl restart nfs-server.service  ˋˋˋ ``` ## brian m. carlson, 2025-02-10 22:32 Subject: Re: [Bug] Git ReadOnly Temp Packfile Causes "Bad file descriptor" And -13 Access Error With NFSv4 Message-ID: URL: https://gitlist.dev/e/Z6p-eIxzrFepC3py%40tapette.crustytoothpaste.net In-Reply-To: <17DE42DF-D2A3-4E25-9198-5AF3CA516E35@amazon.com> ``` On 2025-02-10 at 15:56:59, Maloney, Bryan wrote: > ### Error > Kernel logs: > ˋˋˋ > NFSv4: state recovery failed for open file pack/tmp_pack_aR0Mu3, error = -13 > ˋˋˋ > Git clone output: > ˋˋˋ > fatal: write error: Bad file descriptor, 137.31 MiB | 45.77 MiB/s > fatal: fetch-pack: invalid index-pack output > ˋˋˋ > > > ### Context > > The following error is seen when running git clone over NFSv4 and a failover, or server restart, occurs: > ˋˋˋ > NFSv4: state recovery failed for open file pack/tmp_pack_aR0Mu3, error = -13 > ˋˋˋ > This error is an access denied error that happens when you try to open a file with insufficient permissions. In this case the file being opened is a read only file and it is attempted to be opened with write access. > > Git opens/creates this file with the O_RDWR flag but then applies read only permissions to it, 0444. Since the permissions are changed after the file is opened, the file handle works fine. However if the file was attempted to be re-opened with that same file handle we would see a -13 error. This is what we see following a failover in NFSv4. When clients reclaim their open files, the NFS server re-evaluates the file access. Your description of the problem is spot on. We intentionally set the permissions to 0444 because we never want anyone to change loose object files or packs, since doing so would corrupt the repository. This behaviour is specifically allowed by POSIX[0]: The argument following the oflag argument does not affect whether the file is open for reading, writing, or for both. POSIX does not allow the re-evaluation of file system access once the file is open, so it sounds like your file system is not POSIX compliant, and Git generally requires lots of POSIX-compliant functionality from the file system. For instance, we also require the POSIX consistency guarantees[1], among myriad others: If a read() of file data can be proven (by any means) to occur after a write() of the data, it must reflect that write(), even if the calls are made by different threads. A similar requirement applies to multiple write operations to the same file position. The implicit violation of that particular requirement is why cloud syncing services often corrupt the repository. Could you adjust your NFSv4 server such that is synchronizes state among the primary and replicas in case of a required failover? I know we have people successfully using Git with NFS without problems, although this particular issue does often hit non-POSIX-compliant NFS implementations in a variety of ways. (This particular variant is new to me, though.) > This is an issue for active/passive HA file servers. Since NFSv4 evaluates file permissions at the time of opening a file, this FD will always get an access denied error if a failover occurs during git clone. I'm not sure there's even a good way to solve this problem on the Git side, since I suspect that if we opened the file as 0644 and then immediately did an fchmod to 0444, if you'd still fail here if the file is reopened. Is that correct? I'll also point out that there's a variety of other software that does the same thing as Git does, including zsh and Emacs, so fixing this in Git doesn't really fix the entire problem that your NFS server has, since all of that other software will also be broken in at least some cases and require similar workarounds. (I discovered this with a simple, 30-second search on GitHub some time back.) As far as I'm aware, all other Git implementations also do the same thing as Git does, so you'd also need to patch go-git, libgit2, and every other implementation as well. [0] https://pubs.opengroup.org/onlinepubs/9799919799/functions/open.html [1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/write.html -- brian m. carlson (they/them or he/him) Toronto, Ontario, CA ``` ## Jeff King, 2025-02-11 21:39 Subject: Re: [Bug] Git ReadOnly Temp Packfile Causes "Bad file descriptor" And -13 Access Error With NFSv4 Message-ID: <20250211213940.GB3113114@coredump.intra.peff.net> URL: https://gitlist.dev/e/20250211213940.GB3113114%40coredump.intra.peff.net In-Reply-To: ``` On Mon, Feb 10, 2025 at 10:32:24PM +0000, brian m. carlson wrote: > > This is an issue for active/passive HA file servers. Since NFSv4 evaluates file permissions at the time of opening a file, this FD will always get an access denied error if a failover occurs during git clone. > > I'm not sure there's even a good way to solve this problem on the Git > side, since I suspect that if we opened the file as 0644 and then > immediately did an fchmod to 0444, if you'd still fail here if the file > is reopened. Is that correct? We could probably create and write the tempfile as 0644, and then switch it to 0444 before renaming it into place. We already do something similar with adjust_shared_perm(), I think. I don't know if it's worth doing it that way or not. It should just work even on systems that are happy with the current code, so I think the only downside would be a few extra lines of code and an extra chmod() syscall. -Peff ``` ## brian m. carlson, 2025-02-11 22:46 Subject: Re: [Bug] Git ReadOnly Temp Packfile Causes "Bad file descriptor" And -13 Access Error With NFSv4 Message-ID: URL: https://gitlist.dev/e/Z6vTNW9Dz4jiJvXO%40tapette.crustytoothpaste.net In-Reply-To: <20250211213940.GB3113114@coredump.intra.peff.net> ``` On 2025-02-11 at 21:39:40, Jeff King wrote: > We could probably create and write the tempfile as 0644, and then switch > it to 0444 before renaming it into place. We already do something > similar with adjust_shared_perm(), I think. > > I don't know if it's worth doing it that way or not. It should just work > even on systems that are happy with the current code, so I think the > only downside would be a few extra lines of code and an extra chmod() > syscall. That may work here for this instance. My concern is that we'll find some other weird, hard-to-reproduce behaviour because files are silently getting reopened and permissions are getting re-checked. This bug is much worse than the standard NFS/CIFS implementation bug where open with O_RDWR and 0444 just fails up front, because it can happen in the middle of an operation we thought was going to be successful. I'm not completely opposed to a small workaround here (although, as I mentioned, a decent amount of other software would also need patching for such a system, which argues against it), but I don't want us to have lots of workarounds scattered all over the code to patch one flaky set of NFS servers, so we'd want to limit it to this one spot, especially since this won't be easy to test. I will also mention that this NFS server will also break lots of proprietary software. One of my former employers had software that would do lots of open/fchmod in quick succession (for good and valuable reasons that are too long to fit in this parenthetical), and I'm pretty sure that won't work properly on this server based on the described behaviour. -- brian m. carlson (they/them or he/him) Toronto, Ontario, CA ```