{"thread":{"id":"66817","subject":"APFS, mmap, and git diff reporting no changes when there are changes?","startedAt":"2026-11-09T22:05:55Z","lastAt":"2026-11-11T07:20:29Z","messageCount":3,"participants":["Harold Hunt","brian m. carlson","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"558003","messageId":"CA+-E0BVhYRsktwaUDxtobAXV98dymKg2n4ghod292imymE5C_A@mail.gmail.com","threadId":"66817","inReplyTo":null,"subject":"APFS, mmap, and git diff reporting no changes when there are changes?","fromName":"Harold Hunt","fromEmail":"harold@pwrdrvr.com","sentAt":"2026-11-09T22:05:55Z","receivedAt":"2026-11-09T22:05:55Z","isPatch":false,"sender":{"key":"harold@pwrdrvr.com","avatar":null},"body":"Hi All - I used to maintain Cygwin/X (the X Server for Cygwin on\nWindows) 20 years ago... it's been a while since I've used a mailing\nlist.\n\nI'm working on my own implementation of copy on write cloning for git\non a couple Linux fs options and APFS on Mac.  While working on this I\ncreated tests around mmap'd files and how to handle sharing them when\ncreating worktrees with clones.\n\nI discovered that mmap'd files on Mac can cause existing git to\nmisreport no changes on various commands, while tools that read the\nbytes of the files will see the changed bytes.  I could not find any\ndiscussion of this being an intentional tradeoff of speed over\ncorrectness for git on APFS.  I don't believe this is a security\nproblem.  However, any tool that mmaps a file, changes it, does not\ncall msync(), and holds that mmap open will cause this misreport to\ncontinue until munmap() is called, at which point the ctime of the\nfile changes (mtime does not change on this path).\n\nOne could argue that not calling msync() (which updates both ctime and\nmtime) is bad form and that git not reporting the changes in this case\nis an acceptable tradeoff for performance reasons.  However, on Linux,\ngit correctly detects the changes via the mmap even without calling\nmsync().\n\nOne could also argue that this is Apple's bug or deficiency.  But that\ncould take months / years to resolve?\n\nI'm writing to ask a few questions before either extracting and\nsubmitting just the tests or carefully vetting the fixes and preparing\nmy own PRs:\n1. Is this already known or is this nothing?\n2. How far do we want to go for correctness?\n3. Does anyone else have experience with this on APFS and have some advice?\n\nI have an initial LLM-written PR for this on my fork (I am NOT asking\nfor review on this... if you want to look, look, but if you don't want\nto: please don't): https://github.com/huntharo/git/pull/14\n\nThe impact outside of APFS file systems is zero.  The trade off on\nAPFS depends on how many files are mmap'd and how large they are. If\nno files are ever mmap'd by any tools, ever, then the cost can be\nzero.  The technique I'm using looks at generation number being 0 as\nthe indication that we have to examine the file contents as the\ngeneration number drops to 0 while mapped.\n\nThere is a related follow-on problem: changing a file via mmap/munmap\n(or any other mechanism that updates ctime but not mtime) in the same\nsecond as the last commit will cause git to continue to report no\nchanges even with the file unmapped.\n\nThanks for any input you can provide here.  If these aren't really\nconsidered to be a problem then I'll leave them unsolved in my CoW\nclone patches when I'm ready to submit.\n\nHarold\n\n"},{"id":"558006","messageId":"avJLR6EdZ4OYmTpj@fruit.crustytoothpaste.net","threadId":"66817","inReplyTo":"CA+-E0BVhYRsktwaUDxtobAXV98dymKg2n4ghod292imymE5C_A@mail.gmail.com","subject":"Re: APFS, mmap, and git diff reporting no changes when there are changes?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-11-09T22:38:00Z","receivedAt":"2026-11-09T22:38:00Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2026-11-09 at 22:05:55, Harold Hunt wrote:\n> Hi All - I used to maintain Cygwin/X (the X Server for Cygwin on\n> Windows) 20 years ago... it's been a while since I've used a mailing\n> list.\n> \n> I'm working on my own implementation of copy on write cloning for git\n> on a couple Linux fs options and APFS on Mac.  While working on this I\n> created tests around mmap'd files and how to handle sharing them when\n> creating worktrees with clones.\n> \n> I discovered that mmap'd files on Mac can cause existing git to\n> misreport no changes on various commands, while tools that read the\n> bytes of the files will see the changed bytes.  I could not find any\n> discussion of this being an intentional tradeoff of speed over\n> correctness for git on APFS.  I don't believe this is a security\n> problem.  However, any tool that mmaps a file, changes it, does not\n> call msync(), and holds that mmap open will cause this misreport to\n> continue until munmap() is called, at which point the ctime of the\n> file changes (mtime does not change on this path).\n> \n> One could argue that not calling msync() (which updates both ctime and\n> mtime) is bad form and that git not reporting the changes in this case\n> is an acceptable tradeoff for performance reasons.  However, on Linux,\n> git correctly detects the changes via the mmap even without calling\n> msync().\n\nGit doesn't use mmap on files that change in place, only on\ncontent-addressed files like packs, indexes, and (for interoperability\ncode) object maps, all of which have hashes in their names, or files\nwhich are replaced by rename(2), like the index.\n\nIn general, it's not safe to use mmap on files that can change in place\nbecause if you access beyond the end of the file, you can get SIGBUS,\nbut you don't know if the file may have been modified at any point, so\nyou effectively have to deal with crashing anywhere in your program.\n\n> One could also argue that this is Apple's bug or deficiency.  But that\n> could take months / years to resolve?\n\nThis behaviour is allowed by POSIX:\n\n    The last data modification and last file status change timestamps of\n    a file that is mapped with MAP_SHARED and PROT_WRITE shall be marked\n    for update at some point in the interval between a write reference\n    to the mapped region and the next call to msync() with MS_ASYNC or\n    MS_SYNC for that portion of the file by any process.\n\nHowever, Git relies on the operating system to reflect an accurate\nmodification time.  If you write files to disk with mmap, then you're\nresponsible for calling msync() in a timely manner.  However, as I said,\nthat's extremely improvident for things like files in the working tree,\nwhich is why text editors don't use mmap.\n\n> I'm writing to ask a few questions before either extracting and\n> submitting just the tests or carefully vetting the fixes and preparing\n> my own PRs:\n> 1. Is this already known or is this nothing?\n> 2. How far do we want to go for correctness?\n> 3. Does anyone else have experience with this on APFS and have some advice?\n> \n> I have an initial LLM-written PR for this on my fork (I am NOT asking\n> for review on this... if you want to look, look, but if you don't want\n> to: please don't): https://github.com/huntharo/git/pull/14\n\nPlease read the entirety of SubmittingPatches, including\nhttps://git-scm.com/docs/SubmittingPatches#ai, before sending that in.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\nSpeaking for myself only\n"},{"id":"558159","messageId":"8b881ed0-f30f-41af-9e59-f4e93006545a@kdbg.org","threadId":"66817","inReplyTo":"avJLR6EdZ4OYmTpj@fruit.crustytoothpaste.net","subject":"Re: APFS, mmap, and git diff reporting no changes when there are changes?","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2026-11-11T07:20:29Z","receivedAt":"2026-11-11T07:20:29Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 09.11.26 um 23:38 schrieb brian m. carlson:\n> On 2026-11-09 at 22:05:55, Harold Hunt wrote:\n>> One could also argue that this is Apple's bug or deficiency.  But that\n>> could take months / years to resolve?\n> \n> This behaviour is allowed by POSIX:\n> \n>     The last data modification and last file status change timestamps of\n>     a file that is mapped with MAP_SHARED and PROT_WRITE shall be marked\n>     for update at some point in the interval between a write reference\n>     to the mapped region and the next call to msync() with MS_ASYNC or\n>     MS_SYNC for that portion of the file by any process.\n> \n> However, Git relies on the operating system to reflect an accurate\n> modification time.  If you write files to disk with mmap, then you're\n> responsible for calling msync() in a timely manner.  However, as I said,\n> that's extremely improvident for things like files in the working tree,\n> which is why text editors don't use mmap.\n\nNot to mention that our mmap emulation for Windows cannot reproduce the\nLinux behavior (timely update of timestamps) and is more like the macOS\nbehavior (delayed update of timestamps). We cannot have an accurate\n(atomic) msync emulation, either, because not even FlushViewOfFile[1]\nupdates the timestamp.\n\n[1]\nhttps://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-flushviewoffile\n\n-- Hannes\n\n\n"}]}