git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present

Han Jiang· Aug 5, 2024, 21:37 UTC · lore

Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

(repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup ".git/objects/info/alternates" && copy/get_pack_of objects;) (repo7 and repo8 shouldn't: forget to copy/get_pack_of objects.) (The problem only surfaces after "./client/mirror.git/" is removed: repo7 and repo8 should have the original objects (so git log shouldn't give error) but they don't have.)

cd '/'; cd '/'; rm --force --recursive -- './test_git'; mkdir "$_"; cd "$_"; mkdir --parents -- './server' './client';

git -C './server' init --bare './repo.git' git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty -m "$((++number))" find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared '../server/repo.git' './repo1' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --local '../server/repo.git' './repo2' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --no-local '../server/repo.git' './repo3' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared 'file://'"$(realpath './server/repo.git')" './repo4' find './server/repo.git/objects' -printf '%p %n\n'

cp --recursive --no-target-directory -- './server/repo.git' './client/mirror.git/'

git -C './client' clone --reference='./mirror.git' '../server/repo.git' './repo5' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --local '../server/repo.git' './repo6' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --no-local '../server/repo.git' './repo7' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' 'file://'"$(realpath './server/repo.git')" './repo8' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate '../server/repo.git' './repo9' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate --local '../server/repo.git' './repo10' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate --no-local '../server/repo.git' './repo11' find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate 'file://'"$(realpath './server/repo.git')" './repo12' find './server/repo.git/objects' -printf '%p %n\n'

tree './client/repo1/.git/objects' tree './client/repo2/.git/objects' tree './client/repo3/.git/objects' tree './client/repo4/.git/objects' tree './client/repo5/.git/objects' tree './client/repo6/.git/objects' tree './client/repo7/.git/objects' tree './client/repo8/.git/objects' tree './client/repo9/.git/objects' tree './client/repo10/.git/objects' tree './client/repo11/.git/objects' tree './client/repo12/.git/objects'

git -C './client/repo1' log git -C './client/repo2' log git -C './client/repo3' log git -C './client/repo4' log git -C './client/repo5' log git -C './client/repo6' log git -C './client/repo7' log git -C './client/repo8' log git -C './client/repo9' log git -C './client/repo10' log git -C './client/repo11' log git -C './client/repo12' log

git -C './server/repo.git' cat-file --batch-check --batch-all-objects git --git-dir='./server/repo.git' --work-tree='.' commit --amend --allow-empty -m "$((++number))" git -C './server/repo.git' cat-file --batch-check --batch-all-objects git -C './server/repo.git' -c 'gc.pruneExpire=now' -c 'gc.reflogExpireUnreachable=now' gc git -C './server/repo.git' cat-file --batch-check --batch-all-objects git -C './client/repo1' log git -C './client/repo2' log git -C './client/repo3' log git -C './client/repo4' log git -C './client/repo5' log git -C './client/repo6' log git -C './client/repo7' log git -C './client/repo8' log git -C './client/repo9' log git -C './client/repo10' log git -C './client/repo11' log git -C './client/repo12' log

rm --force --recursive -- './client/mirror.git/' cp --recursive --no-target-directory -- './server/repo.git' './client/mirror.git/' git -C './client/repo1' log git -C './client/repo2' log git -C './client/repo3' log git -C './client/repo4' log git -C './client/repo5' log git -C './client/repo6' log git -C './client/repo7' log git -C './client/repo8' log git -C './client/repo9' log git -C './client/repo10' log git -C './client/repo11' log git -C './client/repo12' log

rm --force --recursive -- './client/mirror.git/' git -C './client/repo1' log git -C './client/repo2' log git -C './client/repo3' log git -C './client/repo4' log git -C './client/repo5' log git -C './client/repo6' log git -C './client/repo7' log git -C './client/repo8' log git -C './client/repo9' log git -C './client/repo10' log git -C './client/repo11' log git -C './client/repo12' log

What did you expect to happen? (Expected behavior)

repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup ".git/objects/info/alternates" && copy/get_pack_of objects; repo7 and repo8 shouldn't: forget to copy/get_pack_of objects. The problem only surfaces after "./client/mirror.git/" is removed: repo7 and repo8 should have the original objects (so git log shouldn't give error) but they don't have. BTW, --shared (setup ".git/objects/info/alternates") / --local (hard_link) / --no-hardlinks (copy) / --no-local (use the regular Git transport, gives a pack) should be mutually exclusive.

What happened instead? (Actual behavior)

What's different between what you expected and what actually happened?

Anything else you want to add:

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

[System Info]
git version:
git version 2.46.0.windows.1
cpu: x86_64
built from commit: 2e6a859ffc0471f60f79c1256f766042b0d5d17d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.9.0
OpenSSL: OpenSSL 3.2.2 4 Jun 2024
zlib: 1.3.1
uname: Windows 10.0 22631
compiler info: gnuc: 14.1
libc info: no libc information available
$SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe

[Enabled Hooks] not run from a git repository - no hooks to show

Jeff King· Aug 13, 2024, 13:16 UTC · re: Han Jiang · lore

Re: git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present

On Tue, Aug 06, 2024 at 09:37:33AM +1200, Han Jiang wrote:

Show 8 quoted lines

> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> (repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup
> ".git/objects/info/alternates" && copy/get_pack_of objects;)
> (repo7 and repo8 shouldn't: forget to copy/get_pack_of objects.)
> (The problem only surfaces after "./client/mirror.git/" is removed:
> repo7 and repo8 should have the original objects (so git log shouldn't
> give error) but they don't have.)

I don't think this expectation is correct. Using "--reference" without "-dissociate" creates a dependency on the mirror repository. It's not safe to delete it. This is discussed in the "--shared" description of git-clone(1), and the "--reference" description refers to it.

So in your example:

> git -C './client' clone --reference='./mirror.git' --no-local '../server/repo.git' './repo7'

I would not expect repo7 to have any objects. Its needs were satisfied by the mirror, and when it told the upload-pack process running in server/repo.git what it had, that process knew that it did not need to send any objects.

It's actually your repo5 and repo6 that are the oddballs:

> git -C './client' clone --reference='./mirror.git' '../server/repo.git' './repo5'

Here since we are cloning a local-file repo, we take a shortcut and do not do any object negotiation at all! Instead, we just take all of the objects from server/repo.git as hardlinks, since doing so is less work than figuring out what we might need.

It makes us robust to mirror.git being removed (since we ended up with duplicates of all of its objects anyway). But it's still not safe in the long run. If the mirror gets new objects, we'd rely on them for subsequent fetches (rather than getting new copies), and the same dependency is created.

-Peff

Han Jiang· Aug 14, 2024, 04:56 UTC · re: Jeff King · lore

Re: git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present

Thank you for pointing out the error! My commands' use of git is incorrect, so the results don't make sense. I will post the corrected commands here in case anyone needs them.

1. git clone --reference=<repository>, then the referenced repository
is updated (e.g. amend/rebase), how to keep up to date: