Re: "git fetch" fails for a --reference clone after an outer forced push

On Wed, Dec 25, 2024 at 05:23:13PM +0000, Mirochnik, Oleg V wrote:

This is the expected behavior, and what the warning in "git help clone" is talking about:

  NOTE: this is a possibly dangerous operation; do not use it unless you
  understand what it does. If you clone your repository using this
  option and then delete branches (or use any other Git command that
  makes any existing commit unreferenced) in the source repository, some
  objects may become unreferenced (or dangling). These objects may be
  removed by normal Git operations (such as git commit) which
  automatically call git maintenance run --auto. (See
  git-maintenance(1).) If these objects are removed and were referenced
  by the cloned repository, then the cloned repository will become
  corrupt.

Your "mirror" repository has no idea that other repositories are depending on it. To safely do a "git gc" there, it would need to know all of the objects that are referenced by the dependent repositories, to count them as reachable.

One way to do that is something like:

  1. Enable the "preciousObjects" flag in the mirror repo, to prevent
     accidental destruction (e.g., from auto-gc):

       git -C mirror config core.repositoryFormatVersion 1
       git -C mirror config extensions.preciousObjects true

  2. When you do want to run gc on the mirror repo, collect all of the
     references from child repos first:

       # collect references from all child repos; the destination
       # doesn't really matter here, and you could even delete
       # refs/child/* after the gc if you want
       for $repo in local*; do
         git -C mirror fetch --prune ../$repo refs/*:refs/child/$repo/*
       done

       # now gc, disabling preciousObjects temporarily
       git -c extensions.preciousObjects=false gc --prune=now

This is (roughly) what a site like GitHub is doing on the backend with repository forks. But Git doesn't ship any scripts to help with it, and I don't offhand know of any public ones. I assume GitLab does something similar, and their system may be open source.

Some gotchas:

  - this is obviously racy with simultaneous updates to the local repos

  - you'd probably want to fetch HEAD as well, to cover detached HEADs

  - it won't cover blobs/trees referenced by the index of each child
    repo (but those are probably going to be local to those repos
    anyway).

  - it won't cover reflogs in the local repos either (but it's not the
    end of the world if a reflog entry goes stale)

Another, perhaps simpler approach, is to just never expire objects from the mirror repo (with the obvious downside being that you might carry objects forever that nobody cares about). You can set gc.pruneExpire to something high, and then look into gc.cruftPacks to store the old objects in a more efficient form.

Hope that helps.

-Peff