{"thread":{"id":"63842","subject":"git clone fsync error on FSx Ontap","startedAt":"2025-07-24T21:49:57Z","lastAt":"2025-07-25T17:56:42Z","messageCount":3,"participants":["Tim Cederquist","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"522704","messageId":"CAFb3DeA9TwRnieajpa5S88ioc=65bUWh55mvDYz3CBp-xUcB3A@mail.gmail.com","threadId":"63842","inReplyTo":null,"subject":"git clone fsync error on FSx Ontap","fromName":"Tim Cederquist","fromEmail":"timcederquist@gmail.com","sentAt":"2025-07-24T21:49:41Z","receivedAt":"2025-07-24T21:49:57Z","isPatch":false,"sender":{"key":"timcederquist@gmail.com","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n  Mounted NFS from FSx Ontap, typical home folder posix permission of\n700 with inheritance. This is nfs4.2, hard mount. users home folder\nhas full complement of ACLs A:FD:OWNER@:rwaDxTtNnCcoy and the owner is\nmy account. I can write into the folders just fine, I can append,\nchmod, create folders, delete, append files, etc. I can dd with fsync,\ni can run install file with permissions set to 444 as indicated below\nso basic IO operations similar to what git clone is doing should be\nrepresented and are working fine.\n\nWhat did you expect to happen? (Expected behavior)\n  git clone https://github.com/githubtraining/hellogitworld.git\n  Command should have cloned a public repository the NFS hosted home folder\n\nWhat happened instead? (Actual behavior)\n\nCloning into 'hellogitworld'...\nremote: Enumerating objects: 306, done.\nremote: Counting objects: 100% (32/32), done.\nremote: Compressing objects: 100% (13/13), done.\nremote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1)\nReceiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done.\nResolving deltas: 100% (70/70), done.\nfatal: fsync error on\n'/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY':\nPermission denied\nfatal: fetch-pack: invalid index-pack output\n\nWhat's different between what you expected and what actually happened?\n  fsync error - permission denied\n\nAnything else you want to add:\n  From Ontap storage host sectrace events command, it indicates the\nfailure is due to the user not having 'Append' permissions to the file\nand generated the fsync error.\n  Running an strace on the git clone command shows the process runs\n\"openat\" with 444 file permission but with O_RDWR flag.\n  Ontap creates the file with 444 (read only) posix permission\n  git continues on to write() into the read only file - ontap rejects\nit and fails the command due to read only status of the file\n  I've tested with linux \"instruction\" command to set permission and\ncopy a file in a similar fashion:\n    strace -f -o trace_install install -m 444 src.txt test/a/test4.txt\n    This command opens the file handle 600, writes into it, and then\nchmods to 444 << not using 444 to start the file as git clone is\ntrying to do.\n  Additionally, I've added an inherited non-intrinsic permission to\nthe user of A:FD:tcederquist@domain:wa << this appends the required\nwrite + append attribute and the FD means it is inherited by all files\nand not overridden by posix permissions. This is not a solution but a\ntestable method. With this permission added, the git clone works as\nexpected.\n  Suggestion is to use 600 on the openat/fopen for the pack index file\ninstead of 444. This is how 'install' sets up the file. However, I\ndon't know if this was an attempt at a cross platform mutex? I cannot\nimagine any other reason why 444 would have been used for a file that\nwould have content written after opening with read only permissions.\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n[System Info]\ngit version 2.47.1\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nlibcurl: 8.3.0\nOpenSSL: OpenSSL 1.0.2k-fips 26 Jan 2017\nzlib: 1.2.7\nuname: Linux 5.10.236-228.935.amzn2.x86_64 #1 SMP Tue May 6 04:10:50\nUTC 2025 x86_64\ncompiler info: gnuc: 7.3\nlibc info: glibc: 2.26\n$SHELL (typically, interactive shell): /bin/bash\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n"},{"id":"522712","messageId":"aIK-YVZWWAjHiIxE@fruit.crustytoothpaste.net","threadId":"63842","inReplyTo":"CAFb3DeA9TwRnieajpa5S88ioc=65bUWh55mvDYz3CBp-xUcB3A@mail.gmail.com","subject":"Re: git clone fsync error on FSx Ontap","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-07-24T23:14:41Z","receivedAt":"2025-07-24T23:14:51Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-07-24 at 21:49:41, Tim Cederquist wrote:\n> What did you expect to happen? (Expected behavior)\n>   git clone https://github.com/githubtraining/hellogitworld.git\n>   Command should have cloned a public repository the NFS hosted home folder\n> \n> What happened instead? (Actual behavior)\n> \n> Cloning into 'hellogitworld'...\n> remote: Enumerating objects: 306, done.\n> remote: Counting objects: 100% (32/32), done.\n> remote: Compressing objects: 100% (13/13), done.\n> remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1)\n> Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done.\n> Resolving deltas: 100% (70/70), done.\n> fatal: fsync error on\n> '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY':\n> Permission denied\n\nThis is definitely a bug in your NFS server.  EACCES is only a code you\nshould see before you have a file descriptor.  Once you have a file\ndescriptor (e.g., open(2) succeeded), the proper status code if it is\nnot suitable for your purpose is EBADF.  (For instance, if you attempt\nto write(2) to a file open only for reading.)\n\nHowever, fsync(2) should never return EBADF or EACCES on a file open for\nwriting.\n\n> fatal: fetch-pack: invalid index-pack output\n> \n> What's different between what you expected and what actually happened?\n>   fsync error - permission denied\n> \n> Anything else you want to add:\n>   From Ontap storage host sectrace events command, it indicates the\n> failure is due to the user not having 'Append' permissions to the file\n> and generated the fsync error.\n>   Running an strace on the git clone command shows the process runs\n> \"openat\" with 444 file permission but with O_RDWR flag.\n>   Ontap creates the file with 444 (read only) posix permission\n>   git continues on to write() into the read only file - ontap rejects\n> it and fails the command due to read only status of the file\n\nYup, this is explicitly allowed by POSIX.\n\n>   I've tested with linux \"instruction\" command to set permission and\n> copy a file in a similar fashion:\n>     strace -f -o trace_install install -m 444 src.txt test/a/test4.txt\n>     This command opens the file handle 600, writes into it, and then\n> chmods to 444 << not using 444 to start the file as git clone is\n> trying to do.\n>   Additionally, I've added an inherited non-intrinsic permission to\n> the user of A:FD:tcederquist@domain:wa << this appends the required\n> write + append attribute and the FD means it is inherited by all files\n> and not overridden by posix permissions. This is not a solution but a\n> testable method. With this permission added, the git clone works as\n> expected.\n>   Suggestion is to use 600 on the openat/fopen for the pack index file\n> instead of 444. This is how 'install' sets up the file. However, I\n> don't know if this was an attempt at a cross platform mutex? I cannot\n> imagine any other reason why 444 would have been used for a file that\n> would have content written after opening with read only permissions.\n\nThe goal is to create a file which has permissions honoured by the umask\nbut is not at all writable.  There is no reason to write to a pack file\nor loose object once it's written: the file is immutable until it's no\nlonger useful, at which point it's removed (which does not require write\npermission on the file).  POSIX requires that the restrictions set by\nthe file mode be ignored when determining whether the file is open for\nwriting, so your NFS server is not following the POSIX spec correctly.\n\nNote this is possible to do correctly over NFS, and many servers do so,\nbut there are also unfortunately a large number of servers which do not\nhonour the POSIX standard correctly.  In addition, this problem not only\naffects Git, but a wide variety of other software as well, including zsh\nand Emacs, as well as every other Git implementation I'm aware of, so us\ntrying to work around it would still leave you with a server that didn't\nwork properly with lots of software.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"522761","messageId":"CAFb3DeD34EO3sYjinCzHVb8odO5+k13HNQ6m7Ru5KOCFP-nSGg@mail.gmail.com","threadId":"63842","inReplyTo":"aIK-YVZWWAjHiIxE@fruit.crustytoothpaste.net","subject":"Re: git clone fsync error on FSx Ontap","fromName":"Tim Cederquist","fromEmail":"timcederquist@gmail.com","sentAt":"2025-07-25T17:56:29Z","receivedAt":"2025-07-25T17:56:42Z","isPatch":false,"sender":{"key":"timcederquist@gmail.com","avatar":null},"body":"Hi brian,\nThank you for your quick feedback! I was not thinking of the immutable\nfile aspect, that's new for me and makes sense. Your feedback enabled\nus to press the storage team and we are discovering we had a single\nnic ontap that works and a dual nic ontap storage that doesn't, so it\nappears your assessment of it being a storage failure is right on the\nmoney.\n\nThank you again for your detailed response. It was very helpful!\nTim\n\nOn Thu, Jul 24, 2025 at 7:14 PM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n>\n> On 2025-07-24 at 21:49:41, Tim Cederquist wrote:\n> > What did you expect to happen? (Expected behavior)\n> >   git clone https://github.com/githubtraining/hellogitworld.git\n> >   Command should have cloned a public repository the NFS hosted home folder\n> >\n> > What happened instead? (Actual behavior)\n> >\n> > Cloning into 'hellogitworld'...\n> > remote: Enumerating objects: 306, done.\n> > remote: Counting objects: 100% (32/32), done.\n> > remote: Compressing objects: 100% (13/13), done.\n> > remote: Total 306 (delta 22), reused 19 (delta 19), pack-reused 274 (from 1)\n> > Receiving objects: 100% (306/306), 95.63 KiB | 1.84 MiB/s, done.\n> > Resolving deltas: 100% (70/70), done.\n> > fatal: fsync error on\n> > '/mnt/home/tcederquist/hellogitworld/.git/objects/pack/tmp_idx_gpahXY':\n> > Permission denied\n>\n> This is definitely a bug in your NFS server.  EACCES is only a code you\n> should see before you have a file descriptor.  Once you have a file\n> descriptor (e.g., open(2) succeeded), the proper status code if it is\n> not suitable for your purpose is EBADF.  (For instance, if you attempt\n> to write(2) to a file open only for reading.)\n>\n> However, fsync(2) should never return EBADF or EACCES on a file open for\n> writing.\n>\n> > fatal: fetch-pack: invalid index-pack output\n> >\n> > What's different between what you expected and what actually happened?\n> >   fsync error - permission denied\n> >\n> > Anything else you want to add:\n> >   From Ontap storage host sectrace events command, it indicates the\n> > failure is due to the user not having 'Append' permissions to the file\n> > and generated the fsync error.\n> >   Running an strace on the git clone command shows the process runs\n> > \"openat\" with 444 file permission but with O_RDWR flag.\n> >   Ontap creates the file with 444 (read only) posix permission\n> >   git continues on to write() into the read only file - ontap rejects\n> > it and fails the command due to read only status of the file\n>\n> Yup, this is explicitly allowed by POSIX.\n>\n> >   I've tested with linux \"instruction\" command to set permission and\n> > copy a file in a similar fashion:\n> >     strace -f -o trace_install install -m 444 src.txt test/a/test4.txt\n> >     This command opens the file handle 600, writes into it, and then\n> > chmods to 444 << not using 444 to start the file as git clone is\n> > trying to do.\n> >   Additionally, I've added an inherited non-intrinsic permission to\n> > the user of A:FD:tcederquist@domain:wa << this appends the required\n> > write + append attribute and the FD means it is inherited by all files\n> > and not overridden by posix permissions. This is not a solution but a\n> > testable method. With this permission added, the git clone works as\n> > expected.\n> >   Suggestion is to use 600 on the openat/fopen for the pack index file\n> > instead of 444. This is how 'install' sets up the file. However, I\n> > don't know if this was an attempt at a cross platform mutex? I cannot\n> > imagine any other reason why 444 would have been used for a file that\n> > would have content written after opening with read only permissions.\n>\n> The goal is to create a file which has permissions honoured by the umask\n> but is not at all writable.  There is no reason to write to a pack file\n> or loose object once it's written: the file is immutable until it's no\n> longer useful, at which point it's removed (which does not require write\n> permission on the file).  POSIX requires that the restrictions set by\n> the file mode be ignored when determining whether the file is open for\n> writing, so your NFS server is not following the POSIX spec correctly.\n>\n> Note this is possible to do correctly over NFS, and many servers do so,\n> but there are also unfortunately a large number of servers which do not\n> honour the POSIX standard correctly.  In addition, this problem not only\n> affects Git, but a wide variety of other software as well, including zsh\n> and Emacs, as well as every other Git implementation I'm aware of, so us\n> trying to work around it would still leave you with a server that didn't\n> work properly with lots of software.\n> --\n> brian m. carlson (they/them)\n> Toronto, Ontario, CA\n"}]}