# git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present

3 messages from 2024-08-05 to 2024-08-14. Participants: Han Jiang, Jeff King.
Thread: https://gitlist.dev/t/61904

## Han Jiang, 2024-08-05 21:37

Subject: git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present
Message-ID: <CANrWfmR2sB8abN3tjuavu-sDwuENMh3DPdFKS9fXan_1kYURPA@mail.gmail.com>
URL: https://gitlist.dev/e/CANrWfmR2sB8abN3tjuavu-sDwuENMh3DPdFKS9fXan_1kYURPA%40mail.gmail.com

```
Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

(repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup
".git/objects/info/alternates" && copy/get_pack_of objects;)
(repo7 and repo8 shouldn't: forget to copy/get_pack_of objects.)
(The problem only surfaces after "./client/mirror.git/" is removed:
repo7 and repo8 should have the original objects (so git log shouldn't
give error) but they don't have.)

cd '/'; cd '/'; rm --force --recursive -- './test_git'; mkdir "$_"; cd "$_";
mkdir --parents -- './server' './client';

git -C './server' init --bare './repo.git'
git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared '../server/repo.git' './repo1'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --local '../server/repo.git' './repo2'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --no-local '../server/repo.git' './repo3'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared 'file://'"$(realpath
'./server/repo.git')" './repo4'
find './server/repo.git/objects' -printf '%p %n\n'

cp --recursive --no-target-directory -- './server/repo.git'
'./client/mirror.git/'

git -C './client' clone --reference='./mirror.git'
'../server/repo.git' './repo5'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --local
'../server/repo.git' './repo6'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --no-local
'../server/repo.git' './repo7'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git'
'file://'"$(realpath './server/repo.git')" './repo8'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
'../server/repo.git' './repo9'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
--local '../server/repo.git' './repo10'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
--no-local '../server/repo.git' './repo11'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
'file://'"$(realpath './server/repo.git')" './repo12'
find './server/repo.git/objects' -printf '%p %n\n'

tree './client/repo1/.git/objects'
tree './client/repo2/.git/objects'
tree './client/repo3/.git/objects'
tree './client/repo4/.git/objects'
tree './client/repo5/.git/objects'
tree './client/repo6/.git/objects'
tree './client/repo7/.git/objects'
tree './client/repo8/.git/objects'
tree './client/repo9/.git/objects'
tree './client/repo10/.git/objects'
tree './client/repo11/.git/objects'
tree './client/repo12/.git/objects'

git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

git -C './server/repo.git' cat-file --batch-check --batch-all-objects
git --git-dir='./server/repo.git' --work-tree='.' commit --amend
--allow-empty -m "$((++number))"
git -C './server/repo.git' cat-file --batch-check --batch-all-objects
git -C './server/repo.git' -c 'gc.pruneExpire=now' -c
'gc.reflogExpireUnreachable=now' gc
git -C './server/repo.git' cat-file --batch-check --batch-all-objects
git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

rm --force --recursive -- './client/mirror.git/'
cp --recursive --no-target-directory -- './server/repo.git'
'./client/mirror.git/'
git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

rm --force --recursive -- './client/mirror.git/'
git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log


What did you expect to happen? (Expected behavior)

repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup
".git/objects/info/alternates" && copy/get_pack_of objects;
repo7 and repo8 shouldn't: forget to copy/get_pack_of objects.
The problem only surfaces after "./client/mirror.git/" is removed:
repo7 and repo8 should have the original objects (so git log shouldn't
give error) but they don't have.
BTW, --shared (setup ".git/objects/info/alternates") / --local
(hard_link) / --no-hardlinks (copy) / --no-local (use the regular Git
transport, gives a pack) should be mutually exclusive.

What happened instead? (Actual behavior)

What's different between what you expected and what actually happened?

Anything else you want to add:

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.46.0.windows.1
cpu: x86_64
built from commit: 2e6a859ffc0471f60f79c1256f766042b0d5d17d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.9.0
OpenSSL: OpenSSL 3.2.2 4 Jun 2024
zlib: 1.3.1
uname: Windows 10.0 22631
compiler info: gnuc: 14.1
libc info: no libc information available
$SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe


[Enabled Hooks]
not run from a git repository - no hooks to show

```

## Jeff King, 2024-08-13 13:16

Subject: Re: git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present
Message-ID: <20240813131614.GF968816@coredump.intra.peff.net>
URL: https://gitlist.dev/e/20240813131614.GF968816%40coredump.intra.peff.net
In-Reply-To: <CANrWfmR2sB8abN3tjuavu-sDwuENMh3DPdFKS9fXan_1kYURPA@mail.gmail.com>

```
On Tue, Aug 06, 2024 at 09:37:33AM +1200, Han Jiang wrote:

> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> (repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup
> ".git/objects/info/alternates" && copy/get_pack_of objects;)
> (repo7 and repo8 shouldn't: forget to copy/get_pack_of objects.)
> (The problem only surfaces after "./client/mirror.git/" is removed:
> repo7 and repo8 should have the original objects (so git log shouldn't
> give error) but they don't have.)

I don't think this expectation is correct. Using "--reference" without
"-dissociate" creates a dependency on the mirror repository. It's not
safe to delete it. This is discussed in the "--shared" description of
git-clone(1), and the "--reference" description refers to it.

So in your example:

> git -C './client' clone --reference='./mirror.git' --no-local '../server/repo.git' './repo7'

I would not expect repo7 to have any objects. Its needs were satisfied
by the mirror, and when it told the upload-pack process running in
server/repo.git what it had, that process knew that it did not need to
send any objects.

It's actually your repo5 and repo6 that are the oddballs:

> git -C './client' clone --reference='./mirror.git' '../server/repo.git' './repo5'

Here since we are cloning a local-file repo, we take a shortcut and do
not do any object negotiation at all! Instead, we just take all of the
objects from server/repo.git as hardlinks, since doing so is less work
than figuring out what we might need.

It makes us robust to mirror.git being removed (since we ended up with
duplicates of all of its objects anyway). But it's still not safe in the
long run. If the mirror gets new objects, we'd rely on them for
subsequent fetches (rather than getting new copies), and the same
dependency is created.

-Peff

```

## Han Jiang, 2024-08-14 04:56

Subject: Re: git clone --reference=<repository> (without --dissociate), forgets to copy/get_pack_of objects from <repository> when "--no-local"/"file://" is present
Message-ID: <CANrWfmQw4pG6Dz3kCPqc-P4wLKm_VqyHmb+4MwKgfCUUaN5bzw@mail.gmail.com>
URL: https://gitlist.dev/e/CANrWfmQw4pG6Dz3kCPqc-P4wLKm_VqyHmb%2B4MwKgfCUUaN5bzw%40mail.gmail.com
In-Reply-To: <20240813131614.GF968816@coredump.intra.peff.net>

```
Thank you for pointing out the error! My commands' use of git is
incorrect, so the results don't make sense.
I will post the corrected commands here in case anyone needs them.

1. git clone --reference=<repository>, then the referenced repository
is updated (e.g. amend/rebase), how to keep up to date:

cd '/'; cd '/'; rm --force --recursive -- './test_git'; mkdir "$_"; cd "$_";
mkdir --parents -- './server' './client';

git -C './server' init --bare './repo.git'
git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared '../server/repo.git' './repo1'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --local '../server/repo.git' './repo2'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --no-local '../server/repo.git' './repo3'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared 'file://'"$(realpath
'./server/repo.git')" './repo4'
find './server/repo.git/objects' -printf '%p %n\n'

cp --recursive --no-target-directory -- './server/repo.git'
'./client/mirror.git/'

git -C './client' clone --reference='./mirror.git'
'../server/repo.git' './repo5'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --local
'../server/repo.git' './repo6'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --no-local
'../server/repo.git' './repo7'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git'
'file://'"$(realpath './server/repo.git')" './repo8'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
'../server/repo.git' './repo9'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
--local '../server/repo.git' './repo10'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
--no-local '../server/repo.git' './repo11'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
'file://'"$(realpath './server/repo.git')" './repo12'
find './server/repo.git/objects' -printf '%p %n\n'

tree './client/repo1/.git/objects'
tree './client/repo2/.git/objects'
tree './client/repo3/.git/objects'
tree './client/repo4/.git/objects'
tree './client/repo5/.git/objects'
tree './client/repo6/.git/objects'
tree './client/repo7/.git/objects'
tree './client/repo8/.git/objects'
tree './client/repo9/.git/objects'
tree './client/repo10/.git/objects'
tree './client/repo11/.git/objects'
tree './client/repo12/.git/objects'

git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

git -C './server/repo.git' cat-file --batch-check --batch-all-objects
git --git-dir='./server/repo.git' --work-tree='.' commit --amend
--allow-empty -m "$((++number))"
git -C './server/repo.git' cat-file --batch-check --batch-all-objects
git -C './server/repo.git' -c 'gc.pruneExpire=now' -c
'gc.reflogExpireUnreachable=now' gc
git -C './server/repo.git' cat-file --batch-check --batch-all-objects
git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

rm --force --recursive -- './client/mirror.git/'
cp --recursive --no-target-directory -- './server/repo.git'
'./client/mirror.git/'
git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

git -C './client/repo7' branch --all --verbose --verbose
git -C './client/repo7' branch --all --verbose
git -C './client/repo7' branch --all
git -C './client/repo7' update-ref -d 'refs/heads/main'
git -C './client/repo7' update-ref -d 'refs/remotes/origin/main'
git -C './client/repo7' pull
git -C './client/repo7' branch --all --verbose --verbose
tree './client/repo7/.git/objects'

git -C './client/repo8' branch --all --verbose --verbose
git -C './client/repo8' branch --all --verbose
git -C './client/repo8' branch --all
git -C './client/repo8' update-ref -d 'refs/heads/main'
git -C './client/repo8' update-ref -d 'refs/remotes/origin/main'
git -C './client/repo8' pull
git -C './client/repo8' branch --all --verbose --verbose
tree './client/repo8/.git/objects'

2. git clone --reference=<repository>, then the referenced repository
is removed (e.g. rm), how to switch back to
not-referencing-the-repository mode:

cd '/'; cd '/'; rm --force --recursive -- './test_git'; mkdir "$_"; cd "$_";
mkdir --parents -- './server' './client';

git -C './server' init --bare './repo.git'
git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared '../server/repo.git' './repo1'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --local '../server/repo.git' './repo2'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared --no-local '../server/repo.git' './repo3'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --shared 'file://'"$(realpath
'./server/repo.git')" './repo4'
find './server/repo.git/objects' -printf '%p %n\n'

cp --recursive --no-target-directory -- './server/repo.git'
'./client/mirror.git/'

git -C './client' clone --reference='./mirror.git'
'../server/repo.git' './repo5'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --local
'../server/repo.git' './repo6'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --no-local
'../server/repo.git' './repo7'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git'
'file://'"$(realpath './server/repo.git')" './repo8'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
'../server/repo.git' './repo9'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
--local '../server/repo.git' './repo10'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
--no-local '../server/repo.git' './repo11'
find './server/repo.git/objects' -printf '%p %n\n'

git -C './client' clone --reference='./mirror.git' --dissociate
'file://'"$(realpath './server/repo.git')" './repo12'
find './server/repo.git/objects' -printf '%p %n\n'

tree './client/repo1/.git/objects'
tree './client/repo2/.git/objects'
tree './client/repo3/.git/objects'
tree './client/repo4/.git/objects'
tree './client/repo5/.git/objects'
tree './client/repo6/.git/objects'
tree './client/repo7/.git/objects'
tree './client/repo8/.git/objects'
tree './client/repo9/.git/objects'
tree './client/repo10/.git/objects'
tree './client/repo11/.git/objects'
tree './client/repo12/.git/objects'

git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

rm --force --recursive -- './client/mirror.git/'
git -C './client/repo1' log
git -C './client/repo2' log
git -C './client/repo3' log
git -C './client/repo4' log
git -C './client/repo5' log
git -C './client/repo6' log
git -C './client/repo7' log
git -C './client/repo8' log
git -C './client/repo9' log
git -C './client/repo10' log
git -C './client/repo11' log
git -C './client/repo12' log

git -C './client/repo7' branch --all --verbose --verbose
git -C './client/repo7' branch --all --verbose
git -C './client/repo7' branch --all
rm './client/repo7/.git/objects/info/alternates'
git -C './client/repo7' branch --all --verbose --verbose
git -C './client/repo7' branch --all --verbose
git -C './client/repo7' branch --all
git -C './client/repo7' update-ref -d 'refs/heads/main'
git -C './client/repo7' pull
git -C './client/repo7' branch --all --verbose --verbose
tree './client/repo7/.git/objects'

git -C './client/repo8' branch --all --verbose --verbose
git -C './client/repo8' branch --all --verbose
git -C './client/repo8' branch --all
rm './client/repo8/.git/objects/info/alternates'
git -C './client/repo8' branch --all --verbose --verbose
git -C './client/repo8' branch --all --verbose
git -C './client/repo8' branch --all
git -C './client/repo8' update-ref -d 'refs/heads/main'
git -C './client/repo8' pull
git -C './client/repo8' branch --all --verbose --verbose
tree './client/repo8/.git/objects'

On Wed, Aug 14, 2024 at 1:16 AM Jeff King <peff@peff.net> wrote:
>
> On Tue, Aug 06, 2024 at 09:37:33AM +1200, Han Jiang wrote:
>
> > What did you do before the bug happened? (Steps to reproduce your issue)
> >
> > (repo7 and repo8 should behave similar to repo5 and repo6, i.e. setup
> > ".git/objects/info/alternates" && copy/get_pack_of objects;)
> > (repo7 and repo8 shouldn't: forget to copy/get_pack_of objects.)
> > (The problem only surfaces after "./client/mirror.git/" is removed:
> > repo7 and repo8 should have the original objects (so git log shouldn't
> > give error) but they don't have.)
>
> I don't think this expectation is correct. Using "--reference" without
> "-dissociate" creates a dependency on the mirror repository. It's not
> safe to delete it. This is discussed in the "--shared" description of
> git-clone(1), and the "--reference" description refers to it.
>
> So in your example:
>
> > git -C './client' clone --reference='./mirror.git' --no-local '../server/repo.git' './repo7'
>
> I would not expect repo7 to have any objects. Its needs were satisfied
> by the mirror, and when it told the upload-pack process running in
> server/repo.git what it had, that process knew that it did not need to
> send any objects.
>
> It's actually your repo5 and repo6 that are the oddballs:
>
> > git -C './client' clone --reference='./mirror.git' '../server/repo.git' './repo5'
>
> Here since we are cloning a local-file repo, we take a shortcut and do
> not do any object negotiation at all! Instead, we just take all of the
> objects from server/repo.git as hardlinks, since doing so is less work
> than figuring out what we might need.
>
> It makes us robust to mirror.git being removed (since we ended up with
> duplicates of all of its objects anyway). But it's still not safe in the
> long run. If the mirror gets new objects, we'd rely on them for
> subsequent fetches (rather than getting new copies), and the same
> dependency is created.
>
> -Peff

```