{"thread":{"id":"66172","subject":"[RFC] git worktree: use filesystem cloning where supported","startedAt":"2026-08-14T10:40:47Z","lastAt":"2026-08-15T18:09:12Z","messageCount":8,"participants":["Peter Morris","Kristoffer Haugsbakk","Junio C Hamano","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"550602","messageId":"CAOqWQbKn88m=OBDF7W8bBPjeOxtRsvNmhsqNy9AryMKrOKtLUA@mail.gmail.com","threadId":"66172","inReplyTo":null,"subject":"[RFC] git worktree: use filesystem cloning where supported","fromName":"Peter Morris","fromEmail":"mrpmorris@gmail.com","sentAt":"2026-08-14T10:40:30Z","receivedAt":"2026-08-14T10:40:47Z","isPatch":false,"body":"Hi,\n\nI'd like to suggest a change to how git worktree creates files.\n\n# Problem\ngit worktree add creates a new working tree containing copies of the\nfiles from the existing working tree. This is normally fine, but it\ncan result in a lot of unnecessary data being written to disk.\n\nThis seems increasingly relevant with AI coding harnesses. These often\nuse Git worktrees to let multiple agents work on the same repository\nconcurrently. If several agents are working on a large repository,\neach worktree can result in another copy of a large number of files\nbeing written to the SSD.\n\nSSD storage is expensive, and SSDs also have a limited write lifetime.\nIt seems wasteful to physically write the same data to disk several\ntimes when the filesystem may be able to avoid doing so.\n\n# Proposed solution\nWhere the filesystem supports copy-on-write or block cloning, could\ngit worktree use it when creating the working tree?\n\nFor example, Windows Dev Drives (which I use) support ReFS block\ncloning. A file can be cloned without physically copying all of its\ndata, with the filesystem sharing the underlying blocks until one of\nthe files is modified.\n\nIf Git knows that the destination file will initially contain exactly\nthe same contents as the source file currently in the folder, it seems\nlike a good opportunity to use this facility.\n\nThe normal behaviour could remain unchanged on filesystems that don't\nsupport this or when the existing file is modified or a different\nversion from the one that will be checked out.\n\n# Why\nThis would potentially:\n\n* reduce SSD writes when creating worktrees, extending my SSD lifespan\n* reduce physical disk space used by multiple worktrees\n* make creating worktrees faster for large repositories\n* be particularly useful when AI agents are creating multiple\nworktrees concurrently\n\nI'm not suggesting that Git should become dependent on ReFS or any\nparticular filesystem. I'm wondering whether there is a suitable\nabstraction for filesystem-level cloning, with\nplatform/filesystem-specific implementations where available.\n\nI'd be interested to know whether this is something that would fit\nwith the future plans for Git, and whether there are technical reasons\nwhy this couldn't work for worktrees?\n\nPete\n"},{"id":"550603","messageId":"7d0e9933-1a5f-4755-8bc5-fa4fea42f61c@app.fastmail.com","threadId":"66172","inReplyTo":"CAOqWQbKn88m=OBDF7W8bBPjeOxtRsvNmhsqNy9AryMKrOKtLUA@mail.gmail.com","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2026-08-14T10:54:02Z","receivedAt":"2026-08-14T10:54:36Z","isPatch":false,"body":"On Fri, Aug 14, 2026, at 12:40, Peter Morris wrote:\n> I'd like to suggest a change to how git worktree creates files.\n>\n> # Problem\n>[snip]\n\nhttps://lore.kernel.org/git/pull.2317.git.git.1780685368.gitgitgadget@gmail.com/\n"},{"id":"550607","messageId":"b00dbcb8-8f83-46b9-9271-bb789994b262@app.fastmail.com","threadId":"66172","inReplyTo":"CAOqWQbLEakpQEPsfw-GB70fdwbxW8EcdZ8EWzaN6ZAaHUU+jGQ@mail.gmail.com","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2026-08-14T11:34:03Z","receivedAt":"2026-08-14T11:34:43Z","isPatch":false,"body":"On Fri, Aug 14, 2026, at 13:12, Peter Morris wrote:\n> The PR says it only supports Linux and macOS.\n>\n> ReFS is available on Windows too.\n>\n> I believe that if the source and destination are on the same ReFS\n> volume, the Windows CopyFile API will use block cloning automatically,\n> so if CopyFile were used we would get this for free on Windows Dev\n> Drives.\n>\n>\n>> [snip]\n>> https://lore.kernel.org/git/pull.2317.git.git.1780685368.gitgitgadget@gmail.com/\n\nYou must remember to Reply-All here. Thanks. :)\n"},{"id":"550612","messageId":"CAOqWQb+XY_u2OUNnBJ9GBGBz8B73ocHWp+V1tDBS-4a5-OviYA@mail.gmail.com","threadId":"66172","inReplyTo":"CAOqWQb+YzvVeqS85qYjQKK8jrUqDwV01eKqC8i1jgT886ixCwA@mail.gmail.com","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"Peter Morris","fromEmail":"mrpmorris@gmail.com","sentAt":"2026-08-14T13:29:26Z","receivedAt":"2026-08-14T13:29:40Z","isPatch":false,"body":"The PR says it only supports Linux and macOS, but ReFS\nis available on Windows too.\n\nI believe that if the source and destination are on the same ReFS\nvolume the Windows CopyFile API will use block cloning automatically,\nso if CopyFile were used we would get this for free on Windows Dev\nDrives.\n\n> [snip]\n> Kristoffer Haugsbakk said:\n> https://lore.kernel.org/git/pull.2317.git.git.1780685368.gitgitgadget@gmail.com/\n"},{"id":"550620","messageId":"xmqq4igwpswr.fsf@gitster.g","threadId":"66172","inReplyTo":"7d0e9933-1a5f-4755-8bc5-fa4fea42f61c@app.fastmail.com","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-14T16:29:24Z","receivedAt":"2026-08-14T16:29:27Z","isPatch":false,"body":"\"Kristoffer Haugsbakk\" <kristofferhaugsbakk@fastmail.com> writes:\n\n> On Fri, Aug 14, 2026, at 12:40, Peter Morris wrote:\n>> I'd like to suggest a change to how git worktree creates files.\n>>\n>> # Problem\n>>[snip]\n>\n> https://lore.kernel.org/git/pull.2317.git.git.1780685368.gitgitgadget@gmail.com/\n\nIn that thread, Brian makes a good point that you cannot \"copy\"\ndirty working tree files from an existing worktree, and also that\nyou cannot have the same branch checked out in multiple worktrees at\nthe same time, to avoid making other worktrees out of sync when a\ncommit is made in one of the worktrees to advance the branch tip.\n\nBut these issues only mean that you cannot call it done by just\ncreating an identical CoW clone of the whole directory.  As long as\nyou are willing to accept that, instead of CoW-copying all existing\nworking tree files, you may have to give the new worktree its own\ncopy of the path by writing unique contents yourself, the above two\nare surmountable.\n\nHowever, when creating a new worktree, what happens is we check out\nthe working tree files for the new worktree from the index.  The\ncode paths doing the work to materialize these files on the\nfilesystem do not have any visibility into what _other_ worktrees,\nincluding the original, have checked out in their working tree.\n\nIf anybody wants to work on this, first you'd need to stop thinking\nabout \"there are many unchanged files already checked out in this\nworktree so why not CoW copy them?\"  Instead, you'd need to think at\nthe level of the checkout_entry() helper function and devise a way\nto teach it not to do its thing, and instead do your CoW thing.\n\nRoughly speaking, checkout_entry() takes an index entry that\nrecords filetype (regular, executable, or symlink) and blob object\nname, and the path to store the blob contents in.  It takes the\ncontents of the blob from the object database and writes them out\nto the working tree.  Your enhancement to the system may go like\nthis:\n\n - First, iterate over the linked (non-bare) worktrees, examine the\n   index of each of them, and make a mapping from each blob object\n   to files in the working trees that have clean checkouts (there\n   may be more than one such file that has a clean checkout of the\n   same blob object).  Make sure you do not include any files with\n   local modifications.\n\n - Hook into checkout_entry() to look at the mapping you created\n   above.  When you notice that checkout_entry() is trying to check\n   out a blob object known to your mapping, instead of letting it\n   write the blob contents out by calling write_entry(), intercept\n   the request and do your favorite CoW thing.  Make sure that you\n   do not lose the race where somebody else may have updated the\n   working tree file you CoW from since you made the above mapping\n   while excluding locally modified files.\n\nThe TOCTOU issue may turn out to be nasty.  The cleanest way I can\nthink of to solve it is to hash the resulting file after making the\nCoW copy to verify that what you CoW'ed was a good copy against your\nindex.  I personally am not interested in making such a change to\nthe system myself, mostly because of this.\n\nIf you hook into checkout_entry(), it will be used not only by\n\"git worktree add\".  Anything that goes through checkout_entry(),\nwhich is practically everything in Git that updates files in the\nworking tree with what is in the object database, will learn to\nCoW-borrow from an existing checkout elsewhere in sibling worktrees.\n\nHTH.\n"},{"id":"550664","messageId":"aoB4pOTtJ65PjwPA@fruit.crustytoothpaste.net","threadId":"66172","inReplyTo":"CAOqWQb+XY_u2OUNnBJ9GBGBz8B73ocHWp+V1tDBS-4a5-OviYA@mail.gmail.com","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-08-15T14:33:09Z","receivedAt":"2026-08-15T14:33:18Z","isPatch":false,"body":"On 2026-08-14 at 13:29:26, Peter Morris wrote:\n> The PR says it only supports Linux and macOS, but ReFS\n> is available on Windows too.\n> \n> I believe that if the source and destination are on the same ReFS\n> volume the Windows CopyFile API will use block cloning automatically,\n> so if CopyFile were used we would get this for free on Windows Dev\n> Drives.\n\nReFS has serious defects in its implementation.  If the data has been\nwritten but not flushed to disk, the resulting copy will be corrupt.  I\njust saw this come in with Git LFS[0] and I described it to a colleague\nas \"horribly broken\".  I definitely don't recommend adding support for\nthis to Git until only fixed versions of Windows are publicly available.\n\n[0] https://github.com/git-lfs/git-lfs/issues/6312\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"550666","messageId":"CAOqWQbKuD_u5d8XbZ=6x9qc61EXNj9hDQjRfe=XE1FkWCp45bg@mail.gmail.com","threadId":"66172","inReplyTo":"xmqq4igwpswr.fsf@gitster.g","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"Peter Morris","fromEmail":"mrpmorris@gmail.com","sentAt":"2026-08-15T18:03:12Z","receivedAt":"2026-08-15T18:03:24Z","isPatch":false,"body":"> On Fri, 14 Aug 2026 at 17:29, Junio C Hamano <gitster@pobox.com> wrote:\n> In that thread, Brian makes a good point that you cannot \"copy\"\n> dirty working tree files from an existing worktree, and also that\n> you cannot have the same branch checked out in multiple worktrees at\n> the same time, to avoid making other worktrees out of sync when a\n> commit is made in one of the worktrees to advance the branch tip.\n> [snip]\n\nHi Junio,\n\nThanks for the detailed reply. That makes sense, and it's useful to\nknow that the approach is technically viable.\n\nI'm afraid I'm not sufficiently familiar with Git's internals to take\nthis on myself, but hopefully someone with more experience in that\narea might be interested in implementing it.\n\nI've seen lots of complaints about Solid State Drives dying due to\nworktree use. I personally have a repo containing over 80GB of images\nthat never change, so currently worktrees just aren't an option for\nme. I hope this will change in the near future!\n\nPete\n"},{"id":"550667","messageId":"CAOqWQbL24ZsLfDnc8pzCAdwaumWuoNaJOGz01PNSPxSkw6ZCqw@mail.gmail.com","threadId":"66172","inReplyTo":"aoB4pOTtJ65PjwPA@fruit.crustytoothpaste.net","subject":"Re: [RFC] git worktree: use filesystem cloning where supported","fromName":"Peter Morris","fromEmail":"mrpmorris@gmail.com","sentAt":"2026-08-15T18:09:00Z","receivedAt":"2026-08-15T18:09:12Z","isPatch":false,"body":"> ReFS has serious defects in its implementation.  If the data has been\n> written but not flushed to disk, the resulting copy will be corrupt.  I\n> just saw this come in with Git LFS[0] and I described it to a colleague\n> as \"horribly broken\".  I definitely don't recommend adding support for\n> this to Git until only fixed versions of Windows are publicly available.\n> [snip]\n\nWindows CopyFile API opens the source file shared-read deny-write, so you\ncannot write to the file whilst it is being copied or you cannot copy\nthe file because\nsomeone already has a write handle open.\n\nSo this wouldn't happen on Windows.\n\nOn Linux and Mac it seems they don't do the same. Linux FICLONE is supposed\nto be atomic so no problems if that is used, and for MacOS I can't see anything\nthat would enable this to work.\n\nSo it would be Windows and Linux only.\n"}]}