From: Jason Newton via GitGitGadget Date: Tue, 20 Oct 2026 16:04:07 GMT Subject: [PATCH v2 0/2] worktree: copy-on-write creation and ephemeral branches Message-ID: In-Reply-To: When many worktrees share one repository -- for example a fleet of agents each needing an isolated checkout -- "git worktree add" is costly at scale. Objects are shared via the common directory, but the working tree is not: each add rewrites every tracked file, so N worktrees cost N full checkouts of disk and I/O. A branch created only to back one of those worktrees is also left behind when the worktree is removed. Patch 1 adds "git worktree add --reflink". On a copy-on-write filesystem it populates the new worktree by reflinking the current worktree's files and index, then "git reset --hard" rewrites only the paths that differ from . reflink_file() in copy.c uses FICLONE (Linux), clonefile() (macOS), and ReFS block cloning (FSCTL_DUPLICATE_EXTENTS_TO_FILE, as Git LFS does). Elsewhere it is probed up front and falls back to a normal checkout. worktree.reflink (true/false/auto) sets the default; --no-reflink overrides. Uncommitted edits to tracked files in the source do not leak into the new worktree. The copied index is refreshed before reset, and because every reflinked file is a new inode the refresh re-validates by content. Only untracked files are carried over. The option stays opt-in for that reason: it copies the source's untracked and ignored files, which a plain checkout does not. For many worktrees at one commit, the series does not share a branch. git-worktree(1) shows a single update-ref transaction that creates a ref per worktree (or a detached HEAD). A shared branch still lets a stale index in one worktree silently revert a commit made in another. Patch 2 treats a branch that "git worktree add " created itself as owned by that worktree. "git worktree remove" deletes it. Branches named with -b/-B, or that already existed, are left alone. --keep-branch keeps an owned branch. Changes since v1: * Dropped the patch that allowed one branch to be checked out in several worktrees. Documented the per-ref / detached-HEAD pattern instead (brian m. carlson, Junio). * Explained, in the commit message and the manual, why a dirty source worktree does not leak tracked content (brian m. carlson). * Added the ReFS block-cloning arm (brian m. carlson). * Noted overlayfs as a supported setup when the lower filesystem supports reflinks (brian m. carlson). * Kept --reflink opt-in. A whole-tree reflink is not invisible: it carries untracked files, and cloning every file can be slower than checkout when the files are small (Junio; qub1n on the ReFS work). * Rebased onto 47fdf2208c ("The 7th batch"). reflink_file() now takes a repository argument, matching copy_file(). * Added ephemeral-branch reaping for the auto-named branch that "worktree add " creates. Jason Newton (2): worktree: add --reflink for copy-on-write worktree creation worktree: reap ephemeral branches created by "worktree add" Documentation/config/worktree.adoc | 10 + Documentation/git-worktree.adoc | 71 ++++++- builtin/branch.c | 28 +++ builtin/worktree.c | 301 ++++++++++++++++++++++++++++- copy.c | 190 ++++++++++++++++++ copy.h | 14 ++ t/t2400-worktree-add.sh | 143 ++++++++++++++ t/t3200-branch.sh | 2 +- t/t3400-rebase.sh | 2 +- t/t5516-fetch-push.sh | 2 +- 10 files changed, 754 insertions(+), 9 deletions(-) base-commit: 47fdf2208c5b94c8cc01c106b8f7c85efb91c926 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2317%2Fnevion%2Fworktree-reflink-cow-v2 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2317/nevion/worktree-reflink-cow-v2 Pull-Request: https://github.com/git/git/pull/2317 Range-diff vs v1: 1: 7dbd9ca94f ! 1: 83841020f4 worktree: add --reflink for copy-on-write worktree creation @@ Commit message via the common directory, but the working tree is not: N worktrees mean N full checkouts on disk and N times the file I/O. - Add a "--reflink" option that, on copy-on-write filesystems, populates + Add a "--reflink" option that, on a copy-on-write filesystem, populates the new worktree by reflinking the current worktree's files and index instead. The subsequent "git reset --hard" then only rewrites the paths that actually differ between the current worktree and ; everything else (including untracked files such as build - outputs) keeps sharing storage with the source until modified. Because - the cloned index still carries the source files' stat data, it is - refreshed against the reflinked files first so that reset recognizes - the unchanged paths as up to date and leaves them sharing extents - rather than rewriting them. + outputs) keeps sharing storage with the source until modified. - The clones are made by a new reflink_file() helper in copy.c, which - uses the FICLONE ioctl on Linux and clonefile() on macOS and reports - an error otherwise so callers fall back to a normal copy. Support is - probed up front; when unavailable -- including on filesystems without - copy-on-write and on platforms such as Windows that lack a reflink - primitive -- "--reflink" transparently falls back to an ordinary - checkout, so the worst case is no slower than today rather than a - byte-for-byte copy of the source tree. The directory walk skips the - new worktree itself when it lives inside the source one, and preserves - symlinks and modes. + A new reflink_file() helper in copy.c clones every regular file. There + is no size cutoff. + + On Linux it uses the FICLONE ioctl. That is the primitive btrfs, XFS + and bcachefs implement, and the one overlayfs forwards when the lower + filesystem can clone. This path has been tested on btrfs. + + On macOS it uses clonefile(), the APFS primitive. That path is + implemented and has not been run here. + + On Windows it uses ReFS block cloning, + FSCTL_DUPLICATE_EXTENTS_TO_FILE, as Git LFS does. The destination is + sized to the source, extents are cloned in chunks of at most 1GiB, and + the tail is rounded up to a cluster (64K, then 4K) and cloned; Windows + ignores the range past EOF. The tail is not byte-copied, and small + files are not skipped. That path is implemented and has not been run + on ReFS. A volume that rejects the control code, including NTFS, fails + the probe. + + Anywhere else there is no reflink primitive. Support is probed up front + by cloning a one-byte file, and where that fails "--reflink" falls back + to an ordinary checkout instead of byte-copying the source tree. The + directory walk skips the new worktree itself when it lives inside the + source one, and preserves symlinks and modes. + + Uncommitted changes in the source do not leak into the new worktree. + After reflinking the source's files we copy its index and run + "update-index --refresh" before "reset --hard": because every reflinked + file is a fresh inode, the cached stat never matches, so the refresh is + forced to re-validate by content rather than by stat. A tracked path is + therefore only left sharing extents when its content equals the indexed + object; any path whose content differs from -- which + includes every uncommitted modification in the source -- is rewritten + by the reset. The new worktree is thus always a faithful checkout of + for tracked paths, and only untracked files are carried + over. The behavior can be made the default with the worktree.reflink configuration ("true", "false" or "auto", the last suppressing the unsupported-filesystem warning), and turned off per-invocation with --no-reflink. A configured default degrades quietly in modes that cannot reflink (--orphan, --no-checkout) instead of erroring, so - enabling it never breaks those commands. The checkout step continues - to honor checkout.workers, so parallel checkout composes with - --reflink for the paths that do need rewriting. + enabling it never breaks those commands. The checkout step continues to + honor checkout.workers, so parallel checkout composes with --reflink for + the paths that do need rewriting. + + To check out many worktrees at the same commit safely, give each its + own ref (or a detached HEAD) rather than sharing one branch; a + git-worktree(1) example shows the one-shot "update-ref" transaction + that pairs with --reflink. Signed-off-by: Jason Newton @@ Documentation/git-worktree.adoc: To remove a locked worktree, specify `--force` + worktrees from the same base. `--no-reflink` forces an ordinary + checkout, overriding the `worktree.reflink` configuration. ++ -+Reflinks require a copy-on-write filesystem (for example btrfs, XFS, -+bcachefs or ZFS on Linux, or APFS on macOS). On filesystems or platforms -+that do not support reflinks, `--reflink` transparently falls back to an -+ordinary checkout. Because the source worktree's untracked and ignored -+files are cloned as well, only use `--reflink` when that is acceptable. ++Reflinks require a copy-on-write filesystem: for example btrfs, XFS, ++bcachefs or ZFS on Linux (and overlayfs on top of one of these, as used by ++containers), APFS on macOS, or ReFS on Windows. On a filesystem that does ++not support reflinks, `--reflink` transparently falls back to an ordinary ++checkout. Because the source worktree's untracked and ignored files are ++cloned as well, only use `--reflink` when that is acceptable. ++ +This option cannot be combined with `--no-checkout` or `--orphan`. It can +be enabled by default with the `worktree.reflink` configuration; see @@ Documentation/git-worktree.adoc: To remove a locked worktree, specify `--force` `--guess-remote`:: `--no-guess-remote`:: With `worktree add `, without __, instead +@@ Documentation/git-worktree.adoc: $ popd + $ git worktree remove ../temp + ------------ + ++To run several independent worktrees from the same starting commit -- for ++example a pool of automated agents -- give each one its own branch rather than ++sharing a single branch between them. A shared branch is unsafe here: if one ++worktree commits and another then commits from its now-stale index, the second ++commit silently drops the first one's changes. Create the branches in a single ++ref transaction and check each out in its own (optionally reflinked) worktree: ++ ++------------ ++$ oid=$(git rev-parse HEAD) ++$ printf 'create refs/heads/agent-%s %s\n' 1 "$oid" 2 "$oid" 3 "$oid" | ++ git update-ref --stdin ++$ for n in 1 2 3; do git worktree add --reflink ../agent-$n agent-$n; done ++------------ ++ ++The single linkgit:git-update-ref[1] transaction is cheap and is especially ++efficient with the `reftable` backend. If a worktree does not need to advance a ++branch at all, use a detached `HEAD` (`git worktree add --detach`) instead. ++ + CONFIGURATION + ------------- + ## builtin/worktree.c ## @@ @@ builtin/worktree.c: worktree_copy_cleanup: +/* + * Probe whether the filesystem backing "dir" supports reflinks. We do this -+ * once up front so that, on filesystems without copy-on-write support (or on -+ * platforms such as Windows that lack a reflink primitive entirely), we can ++ * once up front so that, on filesystems without copy-on-write support (and on ++ * platforms that lack a reflink primitive entirely), we can + * fall back to a normal checkout instead of byte-copying the whole source + * working tree -- which would include untracked files and be slower than the + * checkout we are trying to avoid. @@ builtin/worktree.c: worktree_copy_cleanup: + if (fd >= 0) { + write_in_full(fd, "x", 1); + close(fd); -+ if (!reflink_file(dst.buf, src.buf, 0600)) ++ if (!reflink_file(the_repository, dst.buf, src.buf, 0600)) + ok = 1; + unlink(dst.buf); + } @@ builtin/worktree.c: worktree_copy_cleanup: + */ +static int reflink_or_copy(const char *dst, const char *src, int mode) +{ -+ if (!reflink_file(dst, src, mode)) ++ if (!reflink_file(the_repository, dst, src, mode)) + return 0; -+ return copy_file(dst, src, mode); ++ return copy_file(the_repository, dst, src, mode); +} + +/* @@ copy.c +#elif defined(__APPLE__) +#include +#include ++#elif defined(GIT_WINDOWS_NATIVE) ++#include ++/* ++ * Block cloning on ReFS uses FSCTL_DUPLICATE_EXTENTS_TO_FILE. Older Windows ++ * SDK headers may lack the control code and its payload struct, so define ++ * them ourselves when missing (the values are part of the stable Win32 API). ++ */ ++#ifndef FSCTL_DUPLICATE_EXTENTS_TO_FILE ++#define FSCTL_DUPLICATE_EXTENTS_TO_FILE \ ++ CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 209, METHOD_BUFFERED, FILE_WRITE_DATA) ++typedef struct _DUPLICATE_EXTENTS_DATA { ++ HANDLE FileHandle; ++ LARGE_INTEGER SourceFileOffset; ++ LARGE_INTEGER TargetFileOffset; ++ LARGE_INTEGER ByteCount; ++} DUPLICATE_EXTENTS_DATA, *PDUPLICATE_EXTENTS_DATA; ++#endif +#endif + int copy_fd(int ifd, int ofd) { while (1) { -@@ copy.c: int copy_file_with_time(const char *dst, const char *src, int mode) +@@ copy.c: int copy_file_with_time(struct repository *repo, return copy_times(dst, src); return status; } + -+int reflink_file(const char *dst, const char *src, int mode) ++#ifdef GIT_WINDOWS_NATIVE ++/* ++ * Clone [offset, offset+count) of "src" onto "dst" via ReFS block cloning. ++ * "count" is rounded up to a cluster boundary by the caller; a range that ++ * runs past the source's end of file is harmless and ignored by Windows. ++ */ ++static int clone_extent_windows(HANDLE dst, HANDLE src, ++ uint64_t offset, uint64_t count) ++{ ++ DUPLICATE_EXTENTS_DATA dup; ++ DWORD bytes; ++ ++ dup.FileHandle = src; ++ dup.SourceFileOffset.QuadPart = offset; ++ dup.TargetFileOffset.QuadPart = offset; ++ dup.ByteCount.QuadPart = count; ++ if (!DeviceIoControl(dst, FSCTL_DUPLICATE_EXTENTS_TO_FILE, ++ &dup, sizeof(dup), NULL, 0, &bytes, NULL)) ++ return -1; ++ return 0; ++} ++#endif ++ ++int reflink_file(struct repository *repo, ++ const char *dst, const char *src, int mode) +{ +#if defined(__APPLE__) + /* @@ copy.c: int copy_file_with_time(const char *dst, const char *src, int mode) + (void)mode; + if (clonefile(src, dst, 0) < 0) + return -1; -+ if (adjust_shared_perm(the_repository, dst)) ++ if (adjust_shared_perm(repo, dst)) + return -1; + return 0; +#elif defined(__linux__) @@ copy.c: int copy_file_with_time(const char *dst, const char *src, int mode) + } + if (close(fdo) != 0) + return -1; -+ if (adjust_shared_perm(the_repository, dst)) ++ if (adjust_shared_perm(repo, dst)) ++ return -1; ++ return 0; ++#elif defined(GIT_WINDOWS_NATIVE) ++ /* ++ * ReFS supports copy-on-write block cloning via ++ * FSCTL_DUPLICATE_EXTENTS_TO_FILE, the same mechanism Git LFS uses. ++ * Cloned ranges must start and end on a cluster boundary (ReFS uses ++ * 64K or 4K clusters) and span less than 4GiB, so clone in <=1GiB ++ * chunks and round the final chunk up to a cluster; the destination is ++ * sized to the source's exact length and an over-long final range past ++ * the source's EOF is ignored. ++ */ ++ const uint64_t GiB = (uint64_t)1 << 30; ++ const uint64_t clusters[] = { 64 * 1024, 4 * 1024 }; ++ int fdi, fdo, ret = -1; ++ HANDLE hi, ho; ++ LARGE_INTEGER size; ++ uint64_t off; ++ size_t i; ++ ++ mode = (mode & 0111) ? 0777 : 0666; ++ if ((fdi = open(src, O_RDONLY | O_BINARY)) < 0) ++ return -1; ++ if ((fdo = open(dst, O_RDWR | O_CREAT | O_EXCL | O_BINARY, mode)) < 0) { ++ int saved = errno; ++ close(fdi); ++ errno = saved; ++ return -1; ++ } ++ hi = (HANDLE)_get_osfhandle(fdi); ++ ho = (HANDLE)_get_osfhandle(fdo); ++ if (hi == INVALID_HANDLE_VALUE || ho == INVALID_HANDLE_VALUE || ++ !GetFileSizeEx(hi, &size)) { ++ errno = EINVAL; ++ goto cleanup; ++ } ++ ++ /* the destination must be the source's exact size before cloning */ ++ if (!SetFilePointerEx(ho, size, NULL, FILE_BEGIN) || !SetEndOfFile(ho)) { ++ errno = EIO; ++ goto cleanup; ++ } ++ ++ ret = 0; ++ /* clone whole 1GiB regions (already cluster aligned) */ ++ for (off = 0; (uint64_t)size.QuadPart > off && ++ (uint64_t)size.QuadPart - off > GiB; off += GiB) { ++ if (clone_extent_windows(ho, hi, off, GiB)) { ++ errno = ENOSYS; ++ ret = -1; ++ goto cleanup; ++ } ++ } ++ /* clone the tail, rounding its length up to a cluster (64K then 4K) */ ++ if (!ret && (uint64_t)size.QuadPart > off) { ++ uint64_t rem = (uint64_t)size.QuadPart - off; ++ ++ ret = -1; ++ for (i = 0; i < ARRAY_SIZE(clusters); i++) { ++ uint64_t count = (rem + clusters[i] - 1) & ++ ~(clusters[i] - 1); ++ if (!clone_extent_windows(ho, hi, off, count)) { ++ ret = 0; ++ break; ++ } ++ } ++ if (ret) ++ errno = ENOSYS; /* not a CoW volume */ ++ } ++ ++cleanup: ++ close(fdi); ++ if (close(fdo) && !ret) { ++ errno = EIO; ++ ret = -1; ++ } ++ if (ret) { ++ int saved = errno; ++ unlink(dst); ++ errno = saved; ++ return -1; ++ } ++ if (adjust_shared_perm(repo, dst)) + return -1; + return 0; +#else -+ /* No reflink support on this platform (e.g. Windows). */ ++ /* No reflink primitive on this platform. */ ++ (void)repo; + (void)dst; + (void)src; + (void)mode; @@ copy.c: int copy_file_with_time(const char *dst, const char *src, int mode) +} ## copy.h ## -@@ copy.h: int copy_fd(int ifd, int ofd); - int copy_file(const char *dst, const char *src, int mode); - int copy_file_with_time(const char *dst, const char *src, int mode); +@@ copy.h: int copy_file(struct repository *repo, + int copy_file_with_time(struct repository *repo, + const char *dst, const char *src, int mode); +/* + * Create "dst" as a copy-on-write (reflink) clone of the regular file @@ copy.h: int copy_fd(int ifd, int ofd); + * ENOSYS (or the underlying error). Callers are expected to fall back to + * copy_file() in that case. Returns 0 on success. + */ -+int reflink_file(const char *dst, const char *src, int mode); ++int reflink_file(struct repository *repo, ++ const char *dst, const char *src, int mode); + #endif /* COPY_H */ @@ t/t2400-worktree-add.sh: test_expect_success 'relative worktree sets extension c + echo artifact >expect && + test_cmp expect wt/untracked.bin && + git -C wt status --porcelain >status && -+ grep "?? untracked.bin" status ++ test_grep "?? untracked.bin" status + ) +' + 2: 97f0fdad9b < -: ---------- worktree: allow sharing a checked-out branch across worktrees -: ---------- > 2: 6e2acdc5e2 worktree: reap ephemeral branches created by "worktree add" -- gitgitgadget