"git fetch" fails for a --reference clone after an outer forced push

Hi all,

To reduce traffic, we use local mirrors via "git clone --reference mirror ..." commands. And we regularly get problems with update of few weeks old clones and must run something like below to get rid of outdated branches:

git for-each-ref --format="%(refname)" | while read ref; do
    git show-ref --quiet --verify $ref 2>/dev/null || git update-ref -d $ref
done

This can be quite long operation as number of refs is significant.

Details are below.

Thanks, Oleg.

Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

$ cat ./doit #!/bin/sh set -xe rm -rf tst mkdir tst cd tst mkdir master git -C master init --bare git clone master local touch local/foo git -C local add . git -C local commit -m init-commit git -C local push echo foo > local/foo git -C local commit -a -m dummy-commit git -C local push origin HEAD:refs/heads/dummy git clone --mirror file://`pwd`/master mirror git clone --reference `pwd`/mirror file://`pwd`/master local1 git -C local1 log --oneline origin/dummy git -C local commit --amend -m new-dummy-commit git -C local push -f origin HEAD:dummy git -C mirror fetch git -C mirror gc --prune=now git -C local1 fetch git -C local1 log --oneline origin/dummy

What did you expect to happen? (Expected behavior)

No errors; last few lines like:

+ git -C local1 fetch
From file:///tmp/tst/master
 + 552a293...248c7fa dummy      -> origin/dummy  (forced update)
+ git -C local1 log --oneline origin/dummy
248c7fa (origin/dummy) new-dummy-commit
0636429 (HEAD -> master, origin/master, origin/HEAD) init-commit

What happened instead? (Actual behavior)

+ git -C local1 fetch
fatal: bad object refs/remotes/origin/dummy
error: file:///tmp/tst/master did not send all necessary objects

What's different between what you expected and what actually happened?

Obvious - should work and does not work

Anything else you want to add:

It works if "git -C mirror gc --prune=now" step is skipped or if local path is used for the clone steps instead of the "file://" The same issue is with https://github.com/... The same issue if use "git -C local1 remote update --prune" instead of the "git -C local1 fetch"

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

podman ub24 container is used to get a newer git version. Exactly the same behavior is for the git v2.39.1

[System Info] git version: git version 2.43.0 cpu: x86_64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh uname: Linux 5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr 14 12:42:38 EDT 2022 x86_64 compiler info: gnuc: 13.2 libc info: glibc: 2.39 $SHELL (typically, interactive shell): <unset>

[Enabled Hooks] not run from a git repository - no hooks to show

On Wed, Dec 25, 2024 at 05:23:13PM +0000, Mirochnik, Oleg V wrote:

This is the expected behavior, and what the warning in "git help clone" is talking about:

  NOTE: this is a possibly dangerous operation; do not use it unless you
  understand what it does. If you clone your repository using this
  option and then delete branches (or use any other Git command that
  makes any existing commit unreferenced) in the source repository, some
  objects may become unreferenced (or dangling). These objects may be
  removed by normal Git operations (such as git commit) which
  automatically call git maintenance run --auto. (See
  git-maintenance(1).) If these objects are removed and were referenced
  by the cloned repository, then the cloned repository will become
  corrupt.

Your "mirror" repository has no idea that other repositories are depending on it. To safely do a "git gc" there, it would need to know all of the objects that are referenced by the dependent repositories, to count them as reachable.

One way to do that is something like:

  1. Enable the "preciousObjects" flag in the mirror repo, to prevent
     accidental destruction (e.g., from auto-gc):

       git -C mirror config core.repositoryFormatVersion 1
       git -C mirror config extensions.preciousObjects true

  2. When you do want to run gc on the mirror repo, collect all of the
     references from child repos first:

       # collect references from all child repos; the destination
       # doesn't really matter here, and you could even delete
       # refs/child/* after the gc if you want
       for $repo in local*; do
         git -C mirror fetch --prune ../$repo refs/*:refs/child/$repo/*
       done

       # now gc, disabling preciousObjects temporarily
       git -c extensions.preciousObjects=false gc --prune=now

This is (roughly) what a site like GitHub is doing on the backend with repository forks. But Git doesn't ship any scripts to help with it, and I don't offhand know of any public ones. I assume GitLab does something similar, and their system may be open source.

Some gotchas:

  - this is obviously racy with simultaneous updates to the local repos

  - you'd probably want to fetch HEAD as well, to cover detached HEADs

  - it won't cover blobs/trees referenced by the index of each child
    repo (but those are probably going to be local to those repos
    anyway).

  - it won't cover reflogs in the local repos either (but it's not the
    end of the world if a reflog entry goes stale)

Another, perhaps simpler approach, is to just never expire objects from the mirror repo (with the obvious downside being that you might carry objects forever that nobody cares about). You can set gc.pruneExpire to something high, and then look into gc.cruftPacks to store the old objects in a more efficient form.

Hope that helps.

-Peff

Hi Jeff,

Thank you for the detailed explanation. It explains perfectly why skipping the git-gc for the mirror in the reproducer fixes the problem. But one thing is still confusing and inconsistent IMO - why using local path for clone still works (even with the git-gc in both master and mirror which removes the obsoleted commit object) while it should not (based on your explanation and the "git help clone" snippet you provided) Could you please clarify?

Thanks, Oleg.

Steps: $ cat doit #!/bin/sh set -xe rm -rf tst mkdir tst cd tst mkdir master git -C master init --bare git clone master local touch local/foo git -C local add . git -C local commit -m init-commit git -C local push echo foo > local/foo git -C local commit -a -m dummy-commit rev=`git -C local log -1 --format=%h` git -C local push origin HEAD:refs/heads/dummy git clone --mirror file://`pwd`/master mirror git clone --reference `pwd`/mirror `pwd`/master local1 git -C local1 log --oneline origin/dummy git -C local commit --amend -m new-dummy-commit git -C local push -f origin HEAD:dummy git -C local log --oneline origin/dummy git -C mirror fetch git -C mirror gc --prune=now git -C mirror log -1 --oneline $rev -- || echo FAIL git -C master gc --prune=now git -C master log -1 --oneline $rev -- || echo FAIL git -C local1 fetch #git -C local1 remote update --prune git -C local1 log --oneline origin/dummy

Tail of the script execution output:

+ git -C mirror fetch
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 230 bytes | 230.00 KiB/s, done.
From file:///tmp/tst/master
 + 6e72f6a...553da2c dummy      -> dummy  (forced update)
+ git -C mirror gc --prune=now
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 96 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (6/6), done.
Building bitmaps: 100% (2/2), done.
Total 6 (delta 1), reused 4 (delta 0), pack-reused 0
+ git -C mirror log -1 --oneline 6e72f6a --
fatal: bad revision '6e72f6a'
+ echo FAIL
FAIL
+ git -C master gc --prune=now
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 96 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (6/6), done.
Building bitmaps: 100% (2/2), done.
Total 6 (delta 1), reused 0 (delta 0), pack-reused 0
+ git -C master log -1 --oneline 6e72f6a --
fatal: bad revision '6e72f6a'
+ echo FAIL
FAIL
+ git -C local1 fetch
From /tmp/tst/master
 + 6e72f6a...553da2c dummy      -> origin/dummy  (forced update)
+ git -C local1 log --oneline origin/dummy
553da2c (origin/dummy) new-dummy-commit
f5eab7f (HEAD -> master, origin/master, origin/HEAD) init-commit

-----Original Message-----
From: Jeff King <peff@peff.net> 
Sent: Thursday, December 26, 2024 7:53 AM
To: Mirochnik, Oleg V <oleg.v.mirochnik@intel.com>
Cc: git@vger.kernel.org
Subject: Re: "git fetch" fails for a --reference clone after an outer forced push

On Wed, Dec 25, 2024 at 05:23:13PM +0000, Mirochnik, Oleg V wrote:

This is the expected behavior, and what the warning in "git help clone" is talking about:

  NOTE: this is a possibly dangerous operation; do not use it unless you
  understand what it does. If you clone your repository using this
  option and then delete branches (or use any other Git command that
  makes any existing commit unreferenced) in the source repository, some
  objects may become unreferenced (or dangling). These objects may be
  removed by normal Git operations (such as git commit) which
  automatically call git maintenance run --auto. (See
  git-maintenance(1).) If these objects are removed and were referenced
  by the cloned repository, then the cloned repository will become
  corrupt.

Your "mirror" repository has no idea that other repositories are depending on it. To safely do a "git gc" there, it would need to know all of the objects that are referenced by the dependent repositories, to count them as reachable.

One way to do that is something like:

  1. Enable the "preciousObjects" flag in the mirror repo, to prevent
     accidental destruction (e.g., from auto-gc):

       git -C mirror config core.repositoryFormatVersion 1
       git -C mirror config extensions.preciousObjects true

  2. When you do want to run gc on the mirror repo, collect all of the
     references from child repos first:

       # collect references from all child repos; the destination
       # doesn't really matter here, and you could even delete
       # refs/child/* after the gc if you want
       for $repo in local*; do
         git -C mirror fetch --prune ../$repo refs/*:refs/child/$repo/*
       done

       # now gc, disabling preciousObjects temporarily
       git -c extensions.preciousObjects=false gc --prune=now

This is (roughly) what a site like GitHub is doing on the backend with repository forks. But Git doesn't ship any scripts to help with it, and I don't offhand know of any public ones. I assume GitLab does something similar, and their system may be open source.

Some gotchas:

  - this is obviously racy with simultaneous updates to the local repos

  - you'd probably want to fetch HEAD as well, to cover detached HEADs

  - it won't cover blobs/trees referenced by the index of each child
    repo (but those are probably going to be local to those repos
    anyway).

  - it won't cover reflogs in the local repos either (but it's not the
    end of the world if a reflog entry goes stale)

Another, perhaps simpler approach, is to just never expire objects from the mirror repo (with the obvious downside being that you might carry objects forever that nobody cares about). You can set gc.pruneExpire to something high, and then look into gc.cruftPacks to store the old objects in a more efficient form.

Hope that helps.

-Peff

On Thu, Dec 26, 2024 at 05:36:34PM +0000, Mirochnik, Oleg V wrote:

When clone is given a regular path (and not a file:// url), the "--local" mode optimizations are used by default. And in that mode, all of the objects from the source repo are actually hard-linked into the cloned repo. We still set up a pointer to the --reference repository, but none of its objects will be used immediately (since we should have everything we need by hard-linking the source repo).

So here:

...there will be hard-links from local1 to the packs in master. So even if "mirror" gets rid of those objects, there will still be copies in local1.

I think if you were to repack local1, it would drop those duplicate objects (because it passes "-l" to "git-repack" under the hood). And then you'd still be subject to the same problems if "mirror" also gets rid of its copies.

-Peff

Thanks!

-----Original Message-----
From: Jeff King <peff@peff.net> 
Sent: Thursday, December 26, 2024 8:06 PM
To: Mirochnik, Oleg V <oleg.v.mirochnik@intel.com>
Cc: git@vger.kernel.org
Subject: Re: "git fetch" fails for a --reference clone after an outer forced push

On Thu, Dec 26, 2024 at 05:36:34PM +0000, Mirochnik, Oleg V wrote:

When clone is given a regular path (and not a file:// url), the "--local" mode optimizations are used by default. And in that mode, all of the objects from the source repo are actually hard-linked into the cloned repo. We still set up a pointer to the --reference repository, but none of its objects will be used immediately (since we should have everything we need by hard-linking the source repo).

So here:

...there will be hard-links from local1 to the packs in master. So even if "mirror" gets rid of those objects, there will still be copies in local1.

I think if you were to repack local1, it would drop those duplicate objects (because it passes "-l" to "git-repack" under the hood). And then you'd still be subject to the same problems if "mirror" also gets rid of its copies.

-Peff