# [Question] Shared object store for many clones and submodules

1 messages from 2026-10-26 to 2026-10-26. Participants: Massimiliano Cialdi.
Thread: https://gitlist.dev/t/66672

## Massimiliano Cialdi, 2026-10-26 21:42

Subject: [Question] Shared object store for many clones and submodules
Message-ID: <20261026214256.87554-1-cialdi@gmail.com>
URL: https://gitlist.dev/e/20261026214256.87554-1-cialdi%40gmail.com

```
Hi all,

I would like advice on a use case before we build anything. We are
open to any approach, including a change to Git itself if that is
the right place. Below I describe the problem, what we verified in
the documentation, the source and with tests, the options we see
(in no particular order), and our questions.

In short, we would like to know:
- how to keep GC of a shared alternate from deleting objects that
  borrowers still reach (question 1);
- how to keep in a borrower local copies of selected objects, which
  "git gc" (repack -l) drops because the alternate has them (3);
- whether a remote helper with the "fetch" capability that adds the
  alternate is a sound way to make all this transparent (2);
- how to apply versioned sparse rules to submodules at clone time
  (5);
- whether changes to Git in these areas would be welcome (7).

The problem
-----------
We are a small embedded team. Every project is a superproject with
many submodules: vendor HALs and SDKs, CMSIS, FreeRTOS, lwIP, mbedTLS,
and Zephyr with its modules through west. Each developer keeps many
persistent workspaces at the same time (one per product, per branch,
per review). So the same upstream history is stored many times on
the same machine.

Measured on one developer machine (Git 2.53.0): 250 object stores
(repositories and submodule git dirs, nested up to 4 levels), 17.9 GiB
of .git in total, of which 15.1 GiB are Git objects (the rest is
mostly git-lfs, which alternates do not share). Examples: 4 clones of
zephyr (3.4 GiB of objects, 0.9 GiB if stored once), 9 copies of
CMSIS_5 (2.2 GiB, 0.25 GiB once), 9 copies of an NXP SDK (1.3 GiB,
0.2 GiB once). Estimates of the Git objects, from the pack sizes:
- today:                     15.1 GiB;
- one store per remote URL:  about 5.1 GiB;
- one object database:       about 2.7 GiB.
The single database saves more because forks have different URLs
(about 1.3 GiB) and because unrelated histories contain identical
files (about 1.1 GiB).

The core of the problem, in short: with alternates, neither side's
GC knows about the other. GC in the shared store deletes objects that
the borrowers still need (F1 below). GC in a borrower deletes the
local copies that it would need to survive the loss of the store
(F2 below). We want both to be safe, and the borrowers to stay small.

What we need, as precisely as we can say it:

R1. Each published object is stored once per machine, in a shared
    store, and workspaces borrow from it.

R2. The .git of a workspace stays small over time, not only right
    after clone. A workspace borrows only history that upstream does
    not rewrite (for us: protected branches such as the default
    branch; whether tags count is still open, because our server
    does not protect them). It keeps locally everything that
    upstream may rewrite or delete (feature branches, forced updates)
    and all local work (unpushed commits, stash, reflog, index, other
    worktrees). For all 250 stores we estimated about 0.6 GiB of such
    local objects (with tags and main/master/develop/release* counted
    as stable, reflogs not counted).

R3. Maintenance of the shared store must never delete an object that
    a workspace still reaches.

R4. The shared store must always be rebuildable. If it is deleted or
    corrupted, we must be able to rebuild it from upstream and from
    the workspaces, and afterwards every workspace must be consistent
    (fsck clean, local work intact). Only remote-tracking refs and
    reflog entries that point to history that upstream no longer has
    may be dropped. R2 exists to make R4 possible.

R5. "Push succeeded" keeps its meaning: a push goes only to upstream.
    The shared store is a derived mirror, filled only by fetching
    from upstream. Pushes are never written to it.

R6. Clone and fetch work offline from the shared store when it is
    recent enough. Push offline fails at once, as today.

R7. Transparency. A user enables the store in their own config, per
    URL prefix, and then keeps using plain "git clone", "git fetch",
    "git submodule update" and "west update". With the config
    removed, Git behaves exactly as today. Occasional administrative
    commands (enable, migrate existing workspaces, clean up, make a
    workspace independent again, status) may be acceptable; this is
    still open for us.

R8. It covers submodules, nested submodules, and repositories that
    west creates (west runs "git init", "git remote add" and
    "git fetch <url>"; it uses refs/heads/manifest-rev, not
    refs/remotes/*).

R9. Sparse checkout rules are versioned in the superproject, for the
    superproject and for its submodules (a project needs only one MCU
    family of a vendor SDK). The clone should come out sparse in one
    step, without a full checkout first.

R10. CI runners (GitLab Runner, shell and docker executors) share the
    same kind of store for ephemeral jobs, fetch by commit id, with
    concurrent jobs.

R11. It works with the Git of common distributions. The oldest in our
    team is Ubuntu 24.04 (Git 2.43.0); the GitLab runner helper image
    (ubuntu flavour) is also 2.43.0.

Ruled out, and why:
- git worktree: does not cover submodules and independent projects;
- hardlinked local clones: the store may be on another filesystem;
- partial or shallow clones: conflict with offline use and full
  history;
- writing pushes also to the store (write-through or write-back):
  not atomic, and offline the user may believe a push was published.

What we verified (Git 2.53.0 man pages and source)
--------------------------------------------------
F1. GC of the store breaks borrowers. The script at the end shows it:
    with "gc --prune=now", and with an explicit "repack -a -d" even
    with gc.pruneExpire=never, the workspace loses commit B, the base
    of its local branch. Plain "git gc" with gc.pruneExpire=never
    kept B in the cruft pack. Same result on two machines.

F2. GC of the workspace removes local copies. "git gc" always runs
    "repack -d -l": pack-objects --local leaves out every object that
    an alternate has, even if a local copy exists, and prune-packed
    removes local loose objects that an alternate has in a pack. The
    same happens with "gc --auto" after fetch, commit, merge, am and
    rebase, once its thresholds are exceeded. The second script at
    the end shows the effect on R2/R4: the user publishes commit C,
    the store fetches it, a plain "git gc" in the workspace drops
    the local copy of C, upstream force-pushes C away, the store
    follows and is GC'd, and the workspace loses the user's own
    commit. (If C is still loose in the store, the local copy stays.)

F3. extensions.preciousObjects makes "prune" and "repack -d" refuse
    and makes "gc" skip them in the store. It also stops pack
    consolidation, so packs accumulate.

F4. Nothing tells a repository that others use it as an alternate.
    core.alternateRefsCommand and core.alternateRefsPrefixes work in
    the other direction.

F5. Remote helpers: "connect" and "fetch" are documented capabilities;
    "stateless-connect" is "Experimental; for internal use only".
    After a helper's "fetch", Git calls odb_reprepare(), which
    re-reads objects/info/alternates. The call is explicit since
    2.44, with fba732c462 (transport-helper: re-examine object dir
    after fetching, 2024-01-24); the lazy re-read of alternates is
    there since 2.41, with e2d003dbed (object-file: reprepare
    alternates when necessary, 2023-03-08). So, reading the source,
    a helper should be able to add the store as alternate during the
    first clone. We have not tested this yet.

F6. URL rewriting: the longest insteadOf wins; "git clone" stores the
    URL as typed in remote.origin.url; without a pushurl, the push
    URL is computed from the original URL with pushInsteadOf, so an
    identity pushInsteadOf rule keeps pushes away from the helper.
    A "<scheme>::" helper is blocked inside "git submodule" unless
    protocol.<scheme>.allow=always.

F7. git sparse-checkout is still marked experimental. There is no
    in-tree file or .gitmodules key for sparse rules, and
    clone.filterSubmodules passes only --filter. Submodules are
    cloned with --no-checkout and then checked out separately.

F8. The reference-transaction hook fires on fetch and push ref
    updates. Before 2.54 (hooks defined in config) it can be enabled
    for all repositories only with core.hooksPath in the user's
    config, which replaces the hooks directory of every repository.

F9. With protocol v2, upload-pack accepts a want for any object it
    has; uploadpack.allow*SHA1InWant apply only to v0/v1.

Options we see (in no particular order)
---------------------------------------
Without changes to Git:
- Store layout: one bare mirror per remote URL (O1), one object
  database for all remotes (O2), or per URL with forks grouped by
  common root commits (O3).
- Transparency (R7): a remote helper "git-remote-<x>" selected by
  url.<x>::<prefix>.insteadOf, with an identity pushInsteadOf (O4).
  It updates the mirror under a lock, adds it as alternate of
  $GIT_DIR, and then either (a) uses the "fetch" capability and
  writes locally only the objects that R2 wants local, or (b) uses
  "connect" and runs upload-pack on the mirror. The alternative is
  plain alternates set up at creation time, without a helper (O5).
- R2 against F2: the alternate holds only stable refs (O6); objects
  that must stay local go into local .keep packs (O7); or two
  mirrors per URL, a stable one used as alternate and a full one
  from which non-stable objects are copied into the workspace (O8).
- R3: an append-only store (preciousObjects, gc.auto=0, maintenance
  off, multi-pack-index for lookups) (O9), plus a rare cleanup that
  keeps the union of the roots computed in every registered
  workspace, under a lock (O10).
- R9: write the versioned rules between the submodule clone
  (--no-checkout) and its checkout (O11), or use a dedicated command
  to create the workspace (O12).

Changes to Git itself (also an option for us):
O13. A repository used as alternate knows its borrowers (a registry),
    and gc, repack and maintenance keep the borrowers' roots.
O14. A way for "repack -l" / prune-packed to keep local copies of
    objects reachable from selected refs, even if an alternate has
    them.
O15. Sparse rules versioned in the superproject and applied to
    submodules during clone and submodule update.

Questions
---------
1. GC of the store. Is there a supported way to make a repository
   that serves as an alternate respect the reachability of its
   borrowers? If not, in a repository with preciousObjects, can any
   command or maintenance task still delete objects? We found only
   "multi-pack-index expire", which does not check the extension.
   Is a root set of "--all --reflog --indexed-objects" in each
   borrower, plus its other worktrees, complete?

2. Remote helper. Is a helper with the "fetch" capability that only
   adds an alternate and writes a chosen subset of objects a correct
   use of the protocol? Can we rely on the re-read of alternates after
   the helper's fetch (F5), and what should we expect on 2.43? Is
   "connect" a better choice, also with protocol v2? Known pitfalls
   with submodules, credentials, or the helper's own fetch from
   upstream? (One idea against recursion is a longer identity
   insteadOf rule passed through GIT_CONFIG_COUNT.)

3. Small borrowers, selectively (F2). Is there a supported way to keep
   in a borrower the local copies of objects reachable from some refs,
   although the alternate has them? Or to copy objects back from the
   alternate for a given set of refs, other than pack-objects with our
   own object list?

4. Rebuild (R4). What is the recommended way to rebuild a deleted or
   corrupted alternate from upstream plus the borrowers' local
   objects, and to find out which objects each borrower is missing?
   fsck in a borrower checks the alternate too and does not tell
   local from borrowed objects; a normal fetch does not re-send
   missing objects, and "fetch --refetch" copies everything locally.

5. Sparse (R9). Is git sparse-checkout expected to leave the
   experimental state? Is there a native way, or ongoing work, to
   apply versioned sparse rules at clone time, also to submodules?

6. CI (R10). Any concern with serving fetches by commit id from one
   store over protocol v2 (F9), for example when one store holds
   objects of many projects?

7. Changes to Git. Would any of O13, O14 or O15 be welcome upstream,
   and in which shape? Or is one of them already possible and we
   missed it?

8. Prior art. Pointers to similar work are welcome (for example
   the shared object cache of Scalar in microsoft/git, or the caches
   of repo or depot_tools).

Reproduction of F1
------------------
Usage: sh repro-gc-alternates.sh [prune-now|gc-never|repack-ad]
Expected on 2.53.0: BROKEN, intact, BROKEN.

---8<---
#!/bin/sh
# A repo used as alternate is GC'd and breaks a borrower.
# Usage: sh repro-gc-alternates.sh [prune-now|gc-never|repack-ad]
set -e
mode=${1:-prune-now}
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
export GIT_AUTHOR_NAME=A GIT_AUTHOR_EMAIL=a@example.com
export GIT_COMMITTER_NAME=A GIT_COMMITTER_EMAIL=a@example.com
cd "$(mktemp -d)"
git --version

# "upstream" plus a colleague's clone
git init -q --bare -b main upstream.git
git clone -q upstream.git dev 2>/dev/null
git -C dev commit -q --allow-empty -m base
git -C dev push -q origin HEAD:main
git -C dev switch -q -c feature-x
git -C dev commit -q --allow-empty -m B
git -C dev push -q origin feature-x
B=$(git -C dev rev-parse feature-x)

# shared cache (mirror) and a workspace borrowing from it
git clone -q --mirror upstream.git cache.git
git -C cache.git repack -q -a -d   # long-lived cache: all packed
# file:// avoids the local-clone shortcut that copies all objects
git clone -q --reference cache.git "file://$PWD/upstream.git" ws
git -C ws switch -q -c mine origin/feature-x
git -C ws commit -q --allow-empty -m C

# precondition: B is NOT in the workspace's own object store
git -C ws count-objects -v | grep -E '^(count|in-pack):'
mv ws/.git/objects/info/alternates alt.save
if git -C ws cat-file -e "$B" 2>/dev/null; then
  echo "setup error: B is local"; exit 2
fi
mv alt.save ws/.git/objects/info/alternates

# upstream drops feature-x, the cache follows and is GC'd
git -C dev push -q origin --delete feature-x
git -C cache.git fetch -q --prune
case $mode in
  prune-now) git -C cache.git gc -q --prune=now ;;
  gc-never)  git -C cache.git config gc.pruneExpire never
             git -C cache.git gc -q ;;
  repack-ad) git -C cache.git config gc.pruneExpire never
             git -C cache.git repack -q -a -d ;;
esac

echo "--- fsck in the workspace:"
if git -C ws fsck --connectivity-only; then
  echo "RESULT: workspace intact"
else
  echo "RESULT: workspace BROKEN (B=$B lost)"
fi
---8<---

Reproduction of F2
------------------
Usage: sh repro-gc-borrower.sh
Expected on 2.53.0: "C is no longer local", then BROKEN.

---8<---
#!/bin/sh
# gc in the borrower drops the local copy of the user's own commit;
# a later forced update plus GC of the alternate breaks the borrower.
set -e
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
export GIT_AUTHOR_NAME=A GIT_AUTHOR_EMAIL=a@example.com
export GIT_COMMITTER_NAME=A GIT_COMMITTER_EMAIL=a@example.com
cd "$(mktemp -d)"
git --version

git init -q --bare -b main upstream.git
git clone -q upstream.git dev 2>/dev/null
git -C dev commit -q --allow-empty -m base
git -C dev push -q origin HEAD:main
git clone -q --mirror upstream.git cache.git
git clone -q --reference cache.git "file://$PWD/upstream.git" ws

# the user publishes C; the cache fetches it and packs it
git -C ws switch -q -c topic
echo c >ws/c.txt && git -C ws add c.txt && git -C ws commit -q -m C
C=$(git -C ws rev-parse HEAD)
git -C ws push -q origin topic
git -C cache.git fetch -q
git -C cache.git repack -q -a -d

# plain gc in the workspace (repack -d -l) drops the local copy of C
git -C ws gc -q
mv ws/.git/objects/info/alternates alt.save
if git -C ws cat-file -e "$C" 2>/dev/null; then
  echo "C is still local"
else
  echo "C is no longer local"
fi
mv alt.save ws/.git/objects/info/alternates

# upstream rewrites topic; the cache follows and is GC'd
git clone -q upstream.git dev2 2>/dev/null
git -C dev2 switch -q -c topic main
git -C dev2 commit -q --allow-empty -m C2
git -C dev2 push -q -f origin topic
git -C cache.git fetch -q --prune
git -C cache.git gc -q --prune=now

if git -C ws fsck --connectivity-only >/dev/null 2>&1; then
  echo "RESULT: workspace intact"
else
  echo "RESULT: workspace BROKEN (own commit C=$C lost)"
fi
---8<---

Thanks for any feedback,
Massimiliano Cialdi


```
