{"thread":{"id":"66672","subject":"[Question] Shared object store for many clones and submodules","startedAt":"2026-10-26T21:42:56Z","lastAt":"2026-10-26T21:42:56Z","messageCount":1,"participants":["Massimiliano Cialdi"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"556410","messageId":"20261026214256.87554-1-cialdi@gmail.com","threadId":"66672","inReplyTo":null,"subject":"[Question] Shared object store for many clones and submodules","fromName":"Massimiliano Cialdi","fromEmail":"cialdi@gmail.com","sentAt":"2026-10-26T21:42:56Z","receivedAt":"2026-10-26T21:42:56Z","isPatch":false,"sender":{"key":"cialdi@gmail.com","avatar":null},"body":"Hi all,\n\nI would like advice on a use case before we build anything. We are\nopen to any approach, including a change to Git itself if that is\nthe right place. Below I describe the problem, what we verified in\nthe documentation, the source and with tests, the options we see\n(in no particular order), and our questions.\n\nIn short, we would like to know:\n- how to keep GC of a shared alternate from deleting objects that\n  borrowers still reach (question 1);\n- how to keep in a borrower local copies of selected objects, which\n  \"git gc\" (repack -l) drops because the alternate has them (3);\n- whether a remote helper with the \"fetch\" capability that adds the\n  alternate is a sound way to make all this transparent (2);\n- how to apply versioned sparse rules to submodules at clone time\n  (5);\n- whether changes to Git in these areas would be welcome (7).\n\nThe problem\n-----------\nWe are a small embedded team. Every project is a superproject with\nmany submodules: vendor HALs and SDKs, CMSIS, FreeRTOS, lwIP, mbedTLS,\nand Zephyr with its modules through west. Each developer keeps many\npersistent workspaces at the same time (one per product, per branch,\nper review). So the same upstream history is stored many times on\nthe same machine.\n\nMeasured on one developer machine (Git 2.53.0): 250 object stores\n(repositories and submodule git dirs, nested up to 4 levels), 17.9 GiB\nof .git in total, of which 15.1 GiB are Git objects (the rest is\nmostly git-lfs, which alternates do not share). Examples: 4 clones of\nzephyr (3.4 GiB of objects, 0.9 GiB if stored once), 9 copies of\nCMSIS_5 (2.2 GiB, 0.25 GiB once), 9 copies of an NXP SDK (1.3 GiB,\n0.2 GiB once). Estimates of the Git objects, from the pack sizes:\n- today:                     15.1 GiB;\n- one store per remote URL:  about 5.1 GiB;\n- one object database:       about 2.7 GiB.\nThe single database saves more because forks have different URLs\n(about 1.3 GiB) and because unrelated histories contain identical\nfiles (about 1.1 GiB).\n\nThe core of the problem, in short: with alternates, neither side's\nGC knows about the other. GC in the shared store deletes objects that\nthe borrowers still need (F1 below). GC in a borrower deletes the\nlocal copies that it would need to survive the loss of the store\n(F2 below). We want both to be safe, and the borrowers to stay small.\n\nWhat we need, as precisely as we can say it:\n\nR1. Each published object is stored once per machine, in a shared\n    store, and workspaces borrow from it.\n\nR2. The .git of a workspace stays small over time, not only right\n    after clone. A workspace borrows only history that upstream does\n    not rewrite (for us: protected branches such as the default\n    branch; whether tags count is still open, because our server\n    does not protect them). It keeps locally everything that\n    upstream may rewrite or delete (feature branches, forced updates)\n    and all local work (unpushed commits, stash, reflog, index, other\n    worktrees). For all 250 stores we estimated about 0.6 GiB of such\n    local objects (with tags and main/master/develop/release* counted\n    as stable, reflogs not counted).\n\nR3. Maintenance of the shared store must never delete an object that\n    a workspace still reaches.\n\nR4. The shared store must always be rebuildable. If it is deleted or\n    corrupted, we must be able to rebuild it from upstream and from\n    the workspaces, and afterwards every workspace must be consistent\n    (fsck clean, local work intact). Only remote-tracking refs and\n    reflog entries that point to history that upstream no longer has\n    may be dropped. R2 exists to make R4 possible.\n\nR5. \"Push succeeded\" keeps its meaning: a push goes only to upstream.\n    The shared store is a derived mirror, filled only by fetching\n    from upstream. Pushes are never written to it.\n\nR6. Clone and fetch work offline from the shared store when it is\n    recent enough. Push offline fails at once, as today.\n\nR7. Transparency. A user enables the store in their own config, per\n    URL prefix, and then keeps using plain \"git clone\", \"git fetch\",\n    \"git submodule update\" and \"west update\". With the config\n    removed, Git behaves exactly as today. Occasional administrative\n    commands (enable, migrate existing workspaces, clean up, make a\n    workspace independent again, status) may be acceptable; this is\n    still open for us.\n\nR8. It covers submodules, nested submodules, and repositories that\n    west creates (west runs \"git init\", \"git remote add\" and\n    \"git fetch <url>\"; it uses refs/heads/manifest-rev, not\n    refs/remotes/*).\n\nR9. Sparse checkout rules are versioned in the superproject, for the\n    superproject and for its submodules (a project needs only one MCU\n    family of a vendor SDK). The clone should come out sparse in one\n    step, without a full checkout first.\n\nR10. CI runners (GitLab Runner, shell and docker executors) share the\n    same kind of store for ephemeral jobs, fetch by commit id, with\n    concurrent jobs.\n\nR11. It works with the Git of common distributions. The oldest in our\n    team is Ubuntu 24.04 (Git 2.43.0); the GitLab runner helper image\n    (ubuntu flavour) is also 2.43.0.\n\nRuled out, and why:\n- git worktree: does not cover submodules and independent projects;\n- hardlinked local clones: the store may be on another filesystem;\n- partial or shallow clones: conflict with offline use and full\n  history;\n- writing pushes also to the store (write-through or write-back):\n  not atomic, and offline the user may believe a push was published.\n\nWhat we verified (Git 2.53.0 man pages and source)\n--------------------------------------------------\nF1. GC of the store breaks borrowers. The script at the end shows it:\n    with \"gc --prune=now\", and with an explicit \"repack -a -d\" even\n    with gc.pruneExpire=never, the workspace loses commit B, the base\n    of its local branch. Plain \"git gc\" with gc.pruneExpire=never\n    kept B in the cruft pack. Same result on two machines.\n\nF2. GC of the workspace removes local copies. \"git gc\" always runs\n    \"repack -d -l\": pack-objects --local leaves out every object that\n    an alternate has, even if a local copy exists, and prune-packed\n    removes local loose objects that an alternate has in a pack. The\n    same happens with \"gc --auto\" after fetch, commit, merge, am and\n    rebase, once its thresholds are exceeded. The second script at\n    the end shows the effect on R2/R4: the user publishes commit C,\n    the store fetches it, a plain \"git gc\" in the workspace drops\n    the local copy of C, upstream force-pushes C away, the store\n    follows and is GC'd, and the workspace loses the user's own\n    commit. (If C is still loose in the store, the local copy stays.)\n\nF3. extensions.preciousObjects makes \"prune\" and \"repack -d\" refuse\n    and makes \"gc\" skip them in the store. It also stops pack\n    consolidation, so packs accumulate.\n\nF4. Nothing tells a repository that others use it as an alternate.\n    core.alternateRefsCommand and core.alternateRefsPrefixes work in\n    the other direction.\n\nF5. Remote helpers: \"connect\" and \"fetch\" are documented capabilities;\n    \"stateless-connect\" is \"Experimental; for internal use only\".\n    After a helper's \"fetch\", Git calls odb_reprepare(), which\n    re-reads objects/info/alternates. The call is explicit since\n    2.44, with fba732c462 (transport-helper: re-examine object dir\n    after fetching, 2024-01-24); the lazy re-read of alternates is\n    there since 2.41, with e2d003dbed (object-file: reprepare\n    alternates when necessary, 2023-03-08). So, reading the source,\n    a helper should be able to add the store as alternate during the\n    first clone. We have not tested this yet.\n\nF6. URL rewriting: the longest insteadOf wins; \"git clone\" stores the\n    URL as typed in remote.origin.url; without a pushurl, the push\n    URL is computed from the original URL with pushInsteadOf, so an\n    identity pushInsteadOf rule keeps pushes away from the helper.\n    A \"<scheme>::\" helper is blocked inside \"git submodule\" unless\n    protocol.<scheme>.allow=always.\n\nF7. git sparse-checkout is still marked experimental. There is no\n    in-tree file or .gitmodules key for sparse rules, and\n    clone.filterSubmodules passes only --filter. Submodules are\n    cloned with --no-checkout and then checked out separately.\n\nF8. The reference-transaction hook fires on fetch and push ref\n    updates. Before 2.54 (hooks defined in config) it can be enabled\n    for all repositories only with core.hooksPath in the user's\n    config, which replaces the hooks directory of every repository.\n\nF9. With protocol v2, upload-pack accepts a want for any object it\n    has; uploadpack.allow*SHA1InWant apply only to v0/v1.\n\nOptions we see (in no particular order)\n---------------------------------------\nWithout changes to Git:\n- Store layout: one bare mirror per remote URL (O1), one object\n  database for all remotes (O2), or per URL with forks grouped by\n  common root commits (O3).\n- Transparency (R7): a remote helper \"git-remote-<x>\" selected by\n  url.<x>::<prefix>.insteadOf, with an identity pushInsteadOf (O4).\n  It updates the mirror under a lock, adds it as alternate of\n  $GIT_DIR, and then either (a) uses the \"fetch\" capability and\n  writes locally only the objects that R2 wants local, or (b) uses\n  \"connect\" and runs upload-pack on the mirror. The alternative is\n  plain alternates set up at creation time, without a helper (O5).\n- R2 against F2: the alternate holds only stable refs (O6); objects\n  that must stay local go into local .keep packs (O7); or two\n  mirrors per URL, a stable one used as alternate and a full one\n  from which non-stable objects are copied into the workspace (O8).\n- R3: an append-only store (preciousObjects, gc.auto=0, maintenance\n  off, multi-pack-index for lookups) (O9), plus a rare cleanup that\n  keeps the union of the roots computed in every registered\n  workspace, under a lock (O10).\n- R9: write the versioned rules between the submodule clone\n  (--no-checkout) and its checkout (O11), or use a dedicated command\n  to create the workspace (O12).\n\nChanges to Git itself (also an option for us):\nO13. A repository used as alternate knows its borrowers (a registry),\n    and gc, repack and maintenance keep the borrowers' roots.\nO14. A way for \"repack -l\" / prune-packed to keep local copies of\n    objects reachable from selected refs, even if an alternate has\n    them.\nO15. Sparse rules versioned in the superproject and applied to\n    submodules during clone and submodule update.\n\nQuestions\n---------\n1. GC of the store. Is there a supported way to make a repository\n   that serves as an alternate respect the reachability of its\n   borrowers? If not, in a repository with preciousObjects, can any\n   command or maintenance task still delete objects? We found only\n   \"multi-pack-index expire\", which does not check the extension.\n   Is a root set of \"--all --reflog --indexed-objects\" in each\n   borrower, plus its other worktrees, complete?\n\n2. Remote helper. Is a helper with the \"fetch\" capability that only\n   adds an alternate and writes a chosen subset of objects a correct\n   use of the protocol? Can we rely on the re-read of alternates after\n   the helper's fetch (F5), and what should we expect on 2.43? Is\n   \"connect\" a better choice, also with protocol v2? Known pitfalls\n   with submodules, credentials, or the helper's own fetch from\n   upstream? (One idea against recursion is a longer identity\n   insteadOf rule passed through GIT_CONFIG_COUNT.)\n\n3. Small borrowers, selectively (F2). Is there a supported way to keep\n   in a borrower the local copies of objects reachable from some refs,\n   although the alternate has them? Or to copy objects back from the\n   alternate for a given set of refs, other than pack-objects with our\n   own object list?\n\n4. Rebuild (R4). What is the recommended way to rebuild a deleted or\n   corrupted alternate from upstream plus the borrowers' local\n   objects, and to find out which objects each borrower is missing?\n   fsck in a borrower checks the alternate too and does not tell\n   local from borrowed objects; a normal fetch does not re-send\n   missing objects, and \"fetch --refetch\" copies everything locally.\n\n5. Sparse (R9). Is git sparse-checkout expected to leave the\n   experimental state? Is there a native way, or ongoing work, to\n   apply versioned sparse rules at clone time, also to submodules?\n\n6. CI (R10). Any concern with serving fetches by commit id from one\n   store over protocol v2 (F9), for example when one store holds\n   objects of many projects?\n\n7. Changes to Git. Would any of O13, O14 or O15 be welcome upstream,\n   and in which shape? Or is one of them already possible and we\n   missed it?\n\n8. Prior art. Pointers to similar work are welcome (for example\n   the shared object cache of Scalar in microsoft/git, or the caches\n   of repo or depot_tools).\n\nReproduction of F1\n------------------\nUsage: sh repro-gc-alternates.sh [prune-now|gc-never|repack-ad]\nExpected on 2.53.0: BROKEN, intact, BROKEN.\n\n---8<---\n#!/bin/sh\n# A repo used as alternate is GC'd and breaks a borrower.\n# Usage: sh repro-gc-alternates.sh [prune-now|gc-never|repack-ad]\nset -e\nmode=${1:-prune-now}\nexport GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1\nexport GIT_AUTHOR_NAME=A GIT_AUTHOR_EMAIL=a@example.com\nexport GIT_COMMITTER_NAME=A GIT_COMMITTER_EMAIL=a@example.com\ncd \"$(mktemp -d)\"\ngit --version\n\n# \"upstream\" plus a colleague's clone\ngit init -q --bare -b main upstream.git\ngit clone -q upstream.git dev 2>/dev/null\ngit -C dev commit -q --allow-empty -m base\ngit -C dev push -q origin HEAD:main\ngit -C dev switch -q -c feature-x\ngit -C dev commit -q --allow-empty -m B\ngit -C dev push -q origin feature-x\nB=$(git -C dev rev-parse feature-x)\n\n# shared cache (mirror) and a workspace borrowing from it\ngit clone -q --mirror upstream.git cache.git\ngit -C cache.git repack -q -a -d   # long-lived cache: all packed\n# file:// avoids the local-clone shortcut that copies all objects\ngit clone -q --reference cache.git \"file://$PWD/upstream.git\" ws\ngit -C ws switch -q -c mine origin/feature-x\ngit -C ws commit -q --allow-empty -m C\n\n# precondition: B is NOT in the workspace's own object store\ngit -C ws count-objects -v | grep -E '^(count|in-pack):'\nmv ws/.git/objects/info/alternates alt.save\nif git -C ws cat-file -e \"$B\" 2>/dev/null; then\n  echo \"setup error: B is local\"; exit 2\nfi\nmv alt.save ws/.git/objects/info/alternates\n\n# upstream drops feature-x, the cache follows and is GC'd\ngit -C dev push -q origin --delete feature-x\ngit -C cache.git fetch -q --prune\ncase $mode in\n  prune-now) git -C cache.git gc -q --prune=now ;;\n  gc-never)  git -C cache.git config gc.pruneExpire never\n             git -C cache.git gc -q ;;\n  repack-ad) git -C cache.git config gc.pruneExpire never\n             git -C cache.git repack -q -a -d ;;\nesac\n\necho \"--- fsck in the workspace:\"\nif git -C ws fsck --connectivity-only; then\n  echo \"RESULT: workspace intact\"\nelse\n  echo \"RESULT: workspace BROKEN (B=$B lost)\"\nfi\n---8<---\n\nReproduction of F2\n------------------\nUsage: sh repro-gc-borrower.sh\nExpected on 2.53.0: \"C is no longer local\", then BROKEN.\n\n---8<---\n#!/bin/sh\n# gc in the borrower drops the local copy of the user's own commit;\n# a later forced update plus GC of the alternate breaks the borrower.\nset -e\nexport GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1\nexport GIT_AUTHOR_NAME=A GIT_AUTHOR_EMAIL=a@example.com\nexport GIT_COMMITTER_NAME=A GIT_COMMITTER_EMAIL=a@example.com\ncd \"$(mktemp -d)\"\ngit --version\n\ngit init -q --bare -b main upstream.git\ngit clone -q upstream.git dev 2>/dev/null\ngit -C dev commit -q --allow-empty -m base\ngit -C dev push -q origin HEAD:main\ngit clone -q --mirror upstream.git cache.git\ngit clone -q --reference cache.git \"file://$PWD/upstream.git\" ws\n\n# the user publishes C; the cache fetches it and packs it\ngit -C ws switch -q -c topic\necho c >ws/c.txt && git -C ws add c.txt && git -C ws commit -q -m C\nC=$(git -C ws rev-parse HEAD)\ngit -C ws push -q origin topic\ngit -C cache.git fetch -q\ngit -C cache.git repack -q -a -d\n\n# plain gc in the workspace (repack -d -l) drops the local copy of C\ngit -C ws gc -q\nmv ws/.git/objects/info/alternates alt.save\nif git -C ws cat-file -e \"$C\" 2>/dev/null; then\n  echo \"C is still local\"\nelse\n  echo \"C is no longer local\"\nfi\nmv alt.save ws/.git/objects/info/alternates\n\n# upstream rewrites topic; the cache follows and is GC'd\ngit clone -q upstream.git dev2 2>/dev/null\ngit -C dev2 switch -q -c topic main\ngit -C dev2 commit -q --allow-empty -m C2\ngit -C dev2 push -q -f origin topic\ngit -C cache.git fetch -q --prune\ngit -C cache.git gc -q --prune=now\n\nif git -C ws fsck --connectivity-only >/dev/null 2>&1; then\n  echo \"RESULT: workspace intact\"\nelse\n  echo \"RESULT: workspace BROKEN (own commit C=$C lost)\"\nfi\n---8<---\n\nThanks for any feedback,\nMassimiliano Cialdi\n\n"}]}