threads / discuss / 60747

Strange behaviour when pushing a commit object to remote's refs/HEAD

Subject: Strange behaviour when pushing a commit object to remote's refs/HEAD

## tl;dr

5 messages between Jan 15, 2024 and Jan 16, 2024.

replies: 4people: 3as markdown or json

Pratyush Yadav· Jan 15, 2024, 19:08 UTC · lore

Hi,

I ran into a strange Magit bug, where when I ran magit-show-refs on a particular repo it threw an error. The details of the Magit bug are not very interesting, but when attempting to reproduce it, I also saw git misbehaving for such repos.

The strange behaviour happens when you push a commit object to remote's refs/HEAD instead of pushing a symbolic ref. Such a repository can be found at https://github.com/prati0100/magit-reproducer. I roughly used the below steps to create such a repo:

    $ git init
    $ echo 1 > foo && git add foo && git commit
    $ echo 2 > bar && git add bar && git commit
    $ git push
    $ git checkout 79264c3
    $ echo 2.1 > bar && git add bar && git commit
    $ git push origin 707a3d5:refs/heads/HEAD

Now with such a repo, if you do `git log --all --oneline` it would look something like:

    707a3d5 (origin/HEAD) 2.1
    86e1c97 (HEAD -> main, origin/main) 2
    79264c3 1

And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:

    ,origin,refs/remotes/origin/HEAD,2.1
    ,origin/main,refs/remotes/origin/main,2

All well and good so far. Now delete the repo and attempt to clone it. This time `git log --all --oneline` gives:

    86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2
    79264c3 1

And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:

    origin/main,origin,refs/remotes/origin/HEAD,2
    ,origin/main,refs/remotes/origin/main,2

So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the commit (707a3d5, "2.1") is nowhere to be found. It neither shows up in `git rev-list --all` nor in `git log --all`. The files and trees associated with it also do not show up in `git rev-list --all --object`. Yet if you do `git show 707a3d5` it shows up. So it does exist and did get cloned, but git cannot properly see it.

Interestingly enough, even the GitHub UI is confused and it won't show you the repo correctly. It will show the commit (86e1c97, "2") for both "branches" main and HEAD. cgit's UI [0] seems to work fine with this, though cloning from cgit still suffers from this bug.

There _is_ a way to clone the repo correctly. If you do:

    $ git init magit-reproducer
    $ git remote add origin https://github.com/prati0100/magit-reproducer.git
    $ git remote update

Now if you do git log --all or git for-each-ref, you see the correct result.

I don't really know how to fix this but it certainly is a bug in git since it can't clone the repo correctly. And at least one major Git host can't display such a repo properly (I haven't tried others).

I used Git v2.40.1 to do most of this but I did compile the latest master d4dbce1db5 ("The seventh batch") and attempted to clone using it and I see the same problem.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/pratyush/magit-reproducer.git/

-- 
Regards,
Pratyush Yadav

Karthik Nayak· Jan 16, 2024, 09:54 UTC · re: Pratyush Yadav · lore

Re: Strange behaviour when pushing a commit object to remote's refs/HEAD

Pratyush Yadav <me@yadavpratyush.com> writes:

> Hi,
>

Hello,

Show 18 quoted lines

> I ran into a strange Magit bug, where when I ran magit-show-refs on a
> particular repo it threw an error. The details of the Magit bug are not
> very interesting, but when attempting to reproduce it, I also saw git
> misbehaving for such repos.
>
> The strange behaviour happens when you push a commit object to remote's
> refs/HEAD instead of pushing a symbolic ref. Such a repository can be
> found at https://github.com/prati0100/magit-reproducer. I roughly used
> the below steps to create such a repo:
>
>     $ git init
>     $ echo 1 > foo && git add foo && git commit
>     $ echo 2 > bar && git add bar && git commit
>     $ git push
>     $ git checkout 79264c3
>     $ echo 2.1 > bar && git add bar && git commit
>     $ git push origin 707a3d5:refs/heads/HEAD
>

Just to note here that pushing to "refs/heads/HEAD" is not actually updating the remote repositories $GIT_DIR/HEAD file, rather it creates a new reference $GIT_DIR/refs/heads/HEAD.

With this understanding you'll see that this is not a bug, because the remote HEAD was never updated, but only a new branch called HEAD was created [0].

Show 18 quoted lines

> Now with such a repo, if you do `git log --all --oneline` it would look
> something like:
>
>     707a3d5 (origin/HEAD) 2.1
>     86e1c97 (HEAD -> main, origin/main) 2
>     79264c3 1
>
> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:
>
>     ,origin,refs/remotes/origin/HEAD,2.1
>     ,origin/main,refs/remotes/origin/main,2
>
> All well and good so far. Now delete the repo and attempt to clone it.
> This time `git log --all --oneline` gives:
>
>     86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2
>     79264c3 1
>

This is expected since you cloned the repository and you got the default branch 'main'.

Show 9 quoted lines

> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:
>
>     origin/main,origin,refs/remotes/origin/HEAD,2
>     ,origin/main,refs/remotes/origin/main,2
>
> So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the
> commit (707a3d5, "2.1") is nowhere to be found. It neither shows up in
> `git rev-list --all` nor in `git log --all`. The files and trees
> associated with it also do not show up in `git rev-list --all --object`.

Because rev-list's `--all`, iterates over all refs. Since you only cloned, the HEAD branch is not pulled.

Everything else is a consequence of the subtle but important difference between updating $GIT_DIR/HEAD vs creating $GIT_DIR/refs/heads/HEAD.

[0]: https://github.com/prati0100/magit-reproducer/branches/all

Thanks, Karthik

Pratyush Yadav· Jan 16, 2024, 11:33 UTC · re: Karthik Nayak · lore

Re: Strange behaviour when pushing a commit object to remote's refs/HEAD

On Tue, Jan 16 2024, Karthik Nayak wrote:

Show 29 quoted lines

> Pratyush Yadav <me@yadavpratyush.com> writes:
>
>> Hi,
>>
>
> Hello,
>
>> I ran into a strange Magit bug, where when I ran magit-show-refs on a
>> particular repo it threw an error. The details of the Magit bug are not
>> very interesting, but when attempting to reproduce it, I also saw git
>> misbehaving for such repos.
>>
>> The strange behaviour happens when you push a commit object to remote's
>> refs/HEAD instead of pushing a symbolic ref. Such a repository can be
>> found at https://github.com/prati0100/magit-reproducer. I roughly used
>> the below steps to create such a repo:
>>
>>     $ git init
>>     $ echo 1 > foo && git add foo && git commit
>>     $ echo 2 > bar && git add bar && git commit
>>     $ git push
>>     $ git checkout 79264c3
>>     $ echo 2.1 > bar && git add bar && git commit
>>     $ git push origin 707a3d5:refs/heads/HEAD
>>
>
> Just to note here that pushing to "refs/heads/HEAD" is not actually
> updating the remote repositories $GIT_DIR/HEAD file, rather it creates a
> new reference $GIT_DIR/refs/heads/HEAD.

Yes, that is what I would also expect. I checked one of the Git servers we have and this is exactly what happens. $GIT_DIR/HEAD is a symref pointing to refs/heads/main and $GIT_DIR/refs/heads/HEAD points to the commit. But behaviour from client side is not consistent.

>
> With this understanding you'll see that this is not a bug, because the
> remote HEAD was never updated, but only a new branch called HEAD was
> created [0].

GitHub thinks so but try opening the branch. It won't show you the commit (707a3d5, "2.1") but instead shows you 86e1c97 ("2"). So something is wrong _at least_ with Github.

Show 22 quoted lines

>
>> Now with such a repo, if you do `git log --all --oneline` it would look
>> something like:
>>
>>     707a3d5 (origin/HEAD) 2.1
>>     86e1c97 (HEAD -> main, origin/main) 2
>>     79264c3 1
>>
>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:
>>
>>     ,origin,refs/remotes/origin/HEAD,2.1
>>     ,origin/main,refs/remotes/origin/main,2
>>
>> All well and good so far. Now delete the repo and attempt to clone it.
>> This time `git log --all --oneline` gives:
>>
>>     86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2
>>     79264c3 1
>>
>
> This is expected since you cloned the repository and you got the default
> branch 'main'.

No.

First, if I clone a repo with multiple branches (say https://github.com/prati0100/git-gui) I get _all_ the remote branches. Yet here I clearly don't get the so called "HEAD" branch. This is not expected behaviour.

Second, git really does misunderstand refs/remotes/origin/HEAD. For example, when running git for-each-ref command with the clone method, I get:

    origin/main,origin,refs/remotes/origin/HEAD,2

So it clearly thinks refs/remotes/origin/HEAD is at 86e1c97 ("2"). Or, to be more specific, it thinks the ref points to origin/main which is at 86e1c97 ("2"). But we set it at (707a3d5, "2.1"). So it tells me the wrong thing. Now if I do the git remote add && git remote update method, git for-each-ref says:

    ,origin,refs/remotes/origin/HEAD,2.1

So now it thinks refs/remotes/origin/HEAD points at (707a3d5, "2.1"). I do not see it as expected behaviour.

We can also see this when inspecting the contents of .git/refs/remotes/origin/HEAD. With clone it says:

    ref: refs/remotes/origin/main

With git remote add && git remote update it says:

    707a3d587c61c089710e3924eb63a51763b5a4c8

The same ref points to different places based on how you pull the repo.

Looking deeper, if you clone a repo that does not have a branch called "HEAD" (like git-gui), git creates a file in .git/refs/remotes/origin/HEAD that says:

    ref: refs/remotes/origin/master

So it certainly seems to use refs/remotes/origin/HEAD to point to the remote's HEAD, and not as a regular branch.

I find this to be inconsistent behaviour on git's part and do not think it is (or should be) expected behaviour.

Show 14 quoted lines

>
>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:
>>
>>     origin/main,origin,refs/remotes/origin/HEAD,2
>>     ,origin/main,refs/remotes/origin/main,2
>>
>> So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the
>> commit (707a3d5, "2.1") is nowhere to be found. It neither shows up in
>> `git rev-list --all` nor in `git log --all`. The files and trees
>> associated with it also do not show up in `git rev-list --all --object`.
>
>
> Because rev-list's `--all`, iterates over all refs. Since you only
> cloned, the HEAD branch is not pulled.

Why not? When you clone all branches should get pulled.

Show 5 quoted lines

>
> Everything else is a consequence of the subtle but important difference
> between updating $GIT_DIR/HEAD vs creating $GIT_DIR/refs/heads/HEAD.
>
> [0]: https://github.com/prati0100/magit-reproducer/branches/all

-- 
Regards,
Pratyush Yadav

Karthik Nayak· Jan 16, 2024, 13:24 UTC · re: Pratyush Yadav · lore

Re: Strange behaviour when pushing a commit object to remote's refs/HEAD

Pratyush Yadav <me@yadavpratyush.com> writes:

Show 9 quoted lines

>> Just to note here that pushing to "refs/heads/HEAD" is not actually
>> updating the remote repositories $GIT_DIR/HEAD file, rather it creates a
>> new reference $GIT_DIR/refs/heads/HEAD.
>
> Yes, that is what I would also expect. I checked one of the Git servers
> we have and this is exactly what happens. $GIT_DIR/HEAD is a symref
> pointing to refs/heads/main and $GIT_DIR/refs/heads/HEAD points to the
> commit. But behaviour from client side is not consistent.
>

What is the non _consistent_ part?

Show 9 quoted lines

>>
>> With this understanding you'll see that this is not a bug, because the
>> remote HEAD was never updated, but only a new branch called HEAD was
>> created [0].
>
> GitHub thinks so but try opening the branch. It won't show you the
> commit (707a3d5, "2.1") but instead shows you 86e1c97 ("2"). So
> something is wrong _at least_ with Github.
>

I don't know how GitHub operates, but I'm guessing because there is ambiguity between a branch called HEAD and the actual HEAD. So this is probably the reason.

Show 30 quoted lines

>>
>>> Now with such a repo, if you do `git log --all --oneline` it would look
>>> something like:
>>>
>>>     707a3d5 (origin/HEAD) 2.1
>>>     86e1c97 (HEAD -> main, origin/main) 2
>>>     79264c3 1
>>>
>>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:
>>>
>>>     ,origin,refs/remotes/origin/HEAD,2.1
>>>     ,origin/main,refs/remotes/origin/main,2
>>>
>>> All well and good so far. Now delete the repo and attempt to clone it.
>>> This time `git log --all --oneline` gives:
>>>
>>>     86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2
>>>     79264c3 1
>>>
>>
>> This is expected since you cloned the repository and you got the default
>> branch 'main'.
>
> No.
>
> First, if I clone a repo with multiple branches (say
> https://github.com/prati0100/git-gui) I get _all_ the remote branches.
> Yet here I clearly don't get the so called "HEAD" branch. This is not
> expected behaviour.
>

You're right, I meant to say that the remote branches don't have the corresponding local branches. But that does not matter here.

I'm not saying that there is a path for git to work properly when creating a branch called "HEAD". It's just that "HEAD" is more of a reserved word for git and creating a branch with the same name has unintended effects.

Show 14 quoted lines

> Second, git really does misunderstand refs/remotes/origin/HEAD. For
> example, when running git for-each-ref command with the clone method, I
> get:
>
>     origin/main,origin,refs/remotes/origin/HEAD,2
>
> So it clearly thinks refs/remotes/origin/HEAD is at 86e1c97 ("2"). Or,
> to be more specific, it thinks the ref points to origin/main which is at
> 86e1c97 ("2"). But we set it at (707a3d5, "2.1"). So it tells me the
> wrong thing. Now if I do the git remote add && git remote update method,
> git for-each-ref says:
>
>     ,origin,refs/remotes/origin/HEAD,2.1
>

This is one of those ambiguities, we store HEAD for remotes as
     $GIT_DIR/refs/remotes/<remote>/HEAD
and remote branches as
     $GIT_DIR/refs/remotes/<remote>/<branch>

So what happens if there is a branch named HEAD? This is the problem you're facing...

Show 26 quoted lines

> So now it thinks refs/remotes/origin/HEAD points at (707a3d5, "2.1"). I
> do not see it as expected behaviour.
>
> We can also see this when inspecting the contents of
> .git/refs/remotes/origin/HEAD. With clone it says:
>
>     ref: refs/remotes/origin/main
>
> With git remote add && git remote update it says:
>
>     707a3d587c61c089710e3924eb63a51763b5a4c8
>
> The same ref points to different places based on how you pull the repo.
>
> Looking deeper, if you clone a repo that does not have a branch called
> "HEAD" (like git-gui), git creates a file in
> .git/refs/remotes/origin/HEAD that says:
>
>     ref: refs/remotes/origin/master
>
> So it certainly seems to use refs/remotes/origin/HEAD to point to the
> remote's HEAD, and not as a regular branch.
>
> I find this to be inconsistent behaviour on git's part and do not think
> it is (or should be) expected behaviour.
>

Maybe we should explicitly mention that using HEAD as the branch name has unintended effects and should be avoided.

Show 17 quoted lines

>>
>>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:
>>>
>>>     origin/main,origin,refs/remotes/origin/HEAD,2
>>>     ,origin/main,refs/remotes/origin/main,2
>>>
>>> So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the
>>> commit (707a3d5, "2.1") is nowhere to be found. It neither shows up in
>>> `git rev-list --all` nor in `git log --all`. The files and trees
>>> associated with it also do not show up in `git rev-list --all --object`.
>>
>>
>> Because rev-list's `--all`, iterates over all refs. Since you only
>> cloned, the HEAD branch is not pulled.
>
> Why not? When you clone all branches should get pulled.
>

I think I jumped too quick here, it is because the branch HEAD is never realized locally as I explained above.

Jeff King· Jan 16, 2024, 15:00 UTC · re: Karthik Nayak · lore

Re: Strange behaviour when pushing a commit object to remote's refs/HEAD

On Tue, Jan 16, 2024 at 08:24:04AM -0500, Karthik Nayak wrote:

Show 7 quoted lines

> This is one of those ambiguities, we store HEAD for remotes as
>      $GIT_DIR/refs/remotes/<remote>/HEAD
> and remote branches as
>      $GIT_DIR/refs/remotes/<remote>/<branch>
> 
> So what happens if there is a branch named HEAD? This is the problem
> you're facing...

Yeah, this is a long-standing issue. The reason we have not fixed it is that it would require a new refs/remotes layout, which implies new lookup rules (e.g., dwim_ref() will convert the name "foo" to "refs/remotes/foo/HEAD", but would need to be taught about the new layout). Likewise, a new layout should probably store per-remote tags (rather than splatting them into the main refs/tags/) along with new dwim_ref() rules to make lookup work more or less as it does now.

So it's not impossible, but some care has to be given the design and to handling compatibility. If anybody is interested, there are probably some nuggets of wisdom to mine from this old thread:

  https://lore.kernel.org/git/AANLkTi=yFwOAQMHhvLsB1_xmYOE9HHP2YB4H4TQzwwc8@mail.gmail.com/

In the meantime, I think the current wisdom is "don't name a branch HEAD". ;) We even added logic to "git branch" to forbid this, but tools like "git push" are a bit more flexible.

-Peff