{"thread":{"id":"60747","subject":"Strange behaviour when pushing a commit object to remote's refs/HEAD","startedAt":"2024-01-15T19:08:16Z","lastAt":"2024-01-16T15:00:29Z","messageCount":5,"participants":["Pratyush Yadav","Karthik Nayak","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"486815","messageId":"mafs0fryypg82.fsf@yadavpratyush.com","threadId":"60747","inReplyTo":null,"subject":"Strange behaviour when pushing a commit object to remote's refs/HEAD","fromName":"Pratyush Yadav","fromEmail":"me@yadavpratyush.com","sentAt":"2024-01-15T19:08:13Z","receivedAt":"2024-01-15T19:08:16Z","isPatch":false,"sender":{"key":"me@yadavpratyush.com","avatar":"https://avatars.githubusercontent.com/u/8817931?v=4"},"body":"Hi,\n\nI ran into a strange Magit bug, where when I ran magit-show-refs on a\nparticular repo it threw an error. The details of the Magit bug are not\nvery interesting, but when attempting to reproduce it, I also saw git\nmisbehaving for such repos.\n\nThe strange behaviour happens when you push a commit object to remote's\nrefs/HEAD instead of pushing a symbolic ref. Such a repository can be\nfound at https://github.com/prati0100/magit-reproducer. I roughly used\nthe below steps to create such a repo:\n\n    $ git init\n    $ echo 1 > foo && git add foo && git commit\n    $ echo 2 > bar && git add bar && git commit\n    $ git push\n    $ git checkout 79264c3\n    $ echo 2.1 > bar && git add bar && git commit\n    $ git push origin 707a3d5:refs/heads/HEAD\n\nNow with such a repo, if you do `git log --all --oneline` it would look\nsomething like:\n\n    707a3d5 (origin/HEAD) 2.1\n    86e1c97 (HEAD -> main, origin/main) 2\n    79264c3 1\n\nAnd running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n\n    ,origin,refs/remotes/origin/HEAD,2.1\n    ,origin/main,refs/remotes/origin/main,2\n\nAll well and good so far. Now delete the repo and attempt to clone it.\nThis time `git log --all --oneline` gives:\n\n    86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2\n    79264c3 1\n\nAnd running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n\n    origin/main,origin,refs/remotes/origin/HEAD,2\n    ,origin/main,refs/remotes/origin/main,2\n\nSo suddenly the remote's HEAD becomes origin/main (symbolic ref) and the\ncommit (707a3d5, \"2.1\") is nowhere to be found. It neither shows up in\n`git rev-list --all` nor in `git log --all`. The files and trees\nassociated with it also do not show up in `git rev-list --all --object`.\nYet if you do `git show 707a3d5` it shows up. So it does exist and did\nget cloned, but git cannot properly see it.\n\nInterestingly enough, even the GitHub UI is confused and it won't show\nyou the repo correctly. It will show the commit (86e1c97, \"2\") for both\n\"branches\" main and HEAD. cgit's UI [0] seems to work fine with this,\nthough cloning from cgit still suffers from this bug.\n\nThere _is_ a way to clone the repo correctly. If you do:\n\n    $ git init magit-reproducer\n    $ git remote add origin https://github.com/prati0100/magit-reproducer.git\n    $ git remote update\n\nNow if you do git log --all or git for-each-ref, you see the correct\nresult.\n\nI don't really know how to fix this but it certainly is a bug in git\nsince it can't clone the repo correctly. And at least one major Git host\ncan't display such a repo properly (I haven't tried others).\n\nI used Git v2.40.1 to do most of this but I did compile the latest\nmaster d4dbce1db5 (\"The seventh batch\") and attempted to clone using it\nand I see the same problem.\n\n[0] https://git.kernel.org/pub/scm/linux/kernel/git/pratyush/magit-reproducer.git/\n\n-- \nRegards,\nPratyush Yadav\n"},{"id":"486827","messageId":"CAOLa=ZS8YBhzaYx=9016KxErsMsazsF09rcuPs=-WpEGjV+ruw@mail.gmail.com","threadId":"60747","inReplyTo":"mafs0fryypg82.fsf@yadavpratyush.com","subject":"Re: Strange behaviour when pushing a commit object to remote's refs/HEAD","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-16T09:54:35Z","receivedAt":"2024-01-16T09:54:37Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Pratyush Yadav <me@yadavpratyush.com> writes:\n\n> Hi,\n>\n\nHello,\n\n> I ran into a strange Magit bug, where when I ran magit-show-refs on a\n> particular repo it threw an error. The details of the Magit bug are not\n> very interesting, but when attempting to reproduce it, I also saw git\n> misbehaving for such repos.\n>\n> The strange behaviour happens when you push a commit object to remote's\n> refs/HEAD instead of pushing a symbolic ref. Such a repository can be\n> found at https://github.com/prati0100/magit-reproducer. I roughly used\n> the below steps to create such a repo:\n>\n>     $ git init\n>     $ echo 1 > foo && git add foo && git commit\n>     $ echo 2 > bar && git add bar && git commit\n>     $ git push\n>     $ git checkout 79264c3\n>     $ echo 2.1 > bar && git add bar && git commit\n>     $ git push origin 707a3d5:refs/heads/HEAD\n>\n\nJust to note here that pushing to \"refs/heads/HEAD\" is not actually\nupdating the remote repositories $GIT_DIR/HEAD file, rather it creates a\nnew reference $GIT_DIR/refs/heads/HEAD.\n\nWith this understanding you'll see that this is not a bug, because the\nremote HEAD was never updated, but only a new branch called HEAD was\ncreated [0].\n\n> Now with such a repo, if you do `git log --all --oneline` it would look\n> something like:\n>\n>     707a3d5 (origin/HEAD) 2.1\n>     86e1c97 (HEAD -> main, origin/main) 2\n>     79264c3 1\n>\n> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n>\n>     ,origin,refs/remotes/origin/HEAD,2.1\n>     ,origin/main,refs/remotes/origin/main,2\n>\n> All well and good so far. Now delete the repo and attempt to clone it.\n> This time `git log --all --oneline` gives:\n>\n>     86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2\n>     79264c3 1\n>\n\nThis is expected since you cloned the repository and you got the default\nbranch 'main'.\n\n> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n>\n>     origin/main,origin,refs/remotes/origin/HEAD,2\n>     ,origin/main,refs/remotes/origin/main,2\n>\n> So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the\n> commit (707a3d5, \"2.1\") is nowhere to be found. It neither shows up in\n> `git rev-list --all` nor in `git log --all`. The files and trees\n> associated with it also do not show up in `git rev-list --all --object`.\n\n\nBecause rev-list's `--all`, iterates over all refs. Since you only\ncloned, the HEAD branch is not pulled.\n\nEverything else is a consequence of the subtle but important difference\nbetween updating $GIT_DIR/HEAD vs creating $GIT_DIR/refs/heads/HEAD.\n\n[0]: https://github.com/prati0100/magit-reproducer/branches/all\n\nThanks,\nKarthik\n"},{"id":"486829","messageId":"mafs0a5p5pl6y.fsf@yadavpratyush.com","threadId":"60747","inReplyTo":"CAOLa=ZS8YBhzaYx=9016KxErsMsazsF09rcuPs=-WpEGjV+ruw@mail.gmail.com","subject":"Re: Strange behaviour when pushing a commit object to remote's refs/HEAD","fromName":"Pratyush Yadav","fromEmail":"me@yadavpratyush.com","sentAt":"2024-01-16T11:33:09Z","receivedAt":"2024-01-16T11:33:12Z","isPatch":false,"sender":{"key":"me@yadavpratyush.com","avatar":"https://avatars.githubusercontent.com/u/8817931?v=4"},"body":"On Tue, Jan 16 2024, Karthik Nayak wrote:\n\n> Pratyush Yadav <me@yadavpratyush.com> writes:\n>\n>> Hi,\n>>\n>\n> Hello,\n>\n>> I ran into a strange Magit bug, where when I ran magit-show-refs on a\n>> particular repo it threw an error. The details of the Magit bug are not\n>> very interesting, but when attempting to reproduce it, I also saw git\n>> misbehaving for such repos.\n>>\n>> The strange behaviour happens when you push a commit object to remote's\n>> refs/HEAD instead of pushing a symbolic ref. Such a repository can be\n>> found at https://github.com/prati0100/magit-reproducer. I roughly used\n>> the below steps to create such a repo:\n>>\n>>     $ git init\n>>     $ echo 1 > foo && git add foo && git commit\n>>     $ echo 2 > bar && git add bar && git commit\n>>     $ git push\n>>     $ git checkout 79264c3\n>>     $ echo 2.1 > bar && git add bar && git commit\n>>     $ git push origin 707a3d5:refs/heads/HEAD\n>>\n>\n> Just to note here that pushing to \"refs/heads/HEAD\" is not actually\n> updating the remote repositories $GIT_DIR/HEAD file, rather it creates a\n> new reference $GIT_DIR/refs/heads/HEAD.\n\nYes, that is what I would also expect. I checked one of the Git servers\nwe have and this is exactly what happens. $GIT_DIR/HEAD is a symref\npointing to refs/heads/main and $GIT_DIR/refs/heads/HEAD points to the\ncommit. But behaviour from client side is not consistent.\n\n>\n> With this understanding you'll see that this is not a bug, because the\n> remote HEAD was never updated, but only a new branch called HEAD was\n> created [0].\n\nGitHub thinks so but try opening the branch. It won't show you the\ncommit (707a3d5, \"2.1\") but instead shows you 86e1c97 (\"2\"). So\nsomething is wrong _at least_ with Github.\n\n>\n>> Now with such a repo, if you do `git log --all --oneline` it would look\n>> something like:\n>>\n>>     707a3d5 (origin/HEAD) 2.1\n>>     86e1c97 (HEAD -> main, origin/main) 2\n>>     79264c3 1\n>>\n>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n>>\n>>     ,origin,refs/remotes/origin/HEAD,2.1\n>>     ,origin/main,refs/remotes/origin/main,2\n>>\n>> All well and good so far. Now delete the repo and attempt to clone it.\n>> This time `git log --all --oneline` gives:\n>>\n>>     86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2\n>>     79264c3 1\n>>\n>\n> This is expected since you cloned the repository and you got the default\n> branch 'main'.\n\nNo.\n\nFirst, if I clone a repo with multiple branches (say\nhttps://github.com/prati0100/git-gui) I get _all_ the remote branches.\nYet here I clearly don't get the so called \"HEAD\" branch. This is not\nexpected behaviour.\n\nSecond, git really does misunderstand refs/remotes/origin/HEAD. For\nexample, when running git for-each-ref command with the clone method, I\nget:\n\n    origin/main,origin,refs/remotes/origin/HEAD,2\n\nSo it clearly thinks refs/remotes/origin/HEAD is at 86e1c97 (\"2\"). Or,\nto be more specific, it thinks the ref points to origin/main which is at\n86e1c97 (\"2\"). But we set it at (707a3d5, \"2.1\"). So it tells me the\nwrong thing. Now if I do the git remote add && git remote update method,\ngit for-each-ref says:\n\n    ,origin,refs/remotes/origin/HEAD,2.1\n\nSo now it thinks refs/remotes/origin/HEAD points at (707a3d5, \"2.1\"). I\ndo not see it as expected behaviour.\n\nWe can also see this when inspecting the contents of\n.git/refs/remotes/origin/HEAD. With clone it says:\n\n    ref: refs/remotes/origin/main\n\nWith git remote add && git remote update it says:\n\n    707a3d587c61c089710e3924eb63a51763b5a4c8\n\nThe same ref points to different places based on how you pull the repo.\n\nLooking deeper, if you clone a repo that does not have a branch called\n\"HEAD\" (like git-gui), git creates a file in\n.git/refs/remotes/origin/HEAD that says:\n\n    ref: refs/remotes/origin/master\n\nSo it certainly seems to use refs/remotes/origin/HEAD to point to the\nremote's HEAD, and not as a regular branch.\n\nI find this to be inconsistent behaviour on git's part and do not think\nit is (or should be) expected behaviour.\n\n>\n>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n>>\n>>     origin/main,origin,refs/remotes/origin/HEAD,2\n>>     ,origin/main,refs/remotes/origin/main,2\n>>\n>> So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the\n>> commit (707a3d5, \"2.1\") is nowhere to be found. It neither shows up in\n>> `git rev-list --all` nor in `git log --all`. The files and trees\n>> associated with it also do not show up in `git rev-list --all --object`.\n>\n>\n> Because rev-list's `--all`, iterates over all refs. Since you only\n> cloned, the HEAD branch is not pulled.\n\nWhy not? When you clone all branches should get pulled.\n\n>\n> Everything else is a consequence of the subtle but important difference\n> between updating $GIT_DIR/HEAD vs creating $GIT_DIR/refs/heads/HEAD.\n>\n> [0]: https://github.com/prati0100/magit-reproducer/branches/all\n\n-- \nRegards,\nPratyush Yadav\n"},{"id":"486830","messageId":"CAOLa=ZRfr+oEKCo8AfFSAFtS8pbDgmG_EeBSwm7GwukzVcqSrg@mail.gmail.com","threadId":"60747","inReplyTo":"mafs0a5p5pl6y.fsf@yadavpratyush.com","subject":"Re: Strange behaviour when pushing a commit object to remote's refs/HEAD","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-01-16T13:24:04Z","receivedAt":"2024-01-16T13:24:08Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Pratyush Yadav <me@yadavpratyush.com> writes:\n\n\n>> Just to note here that pushing to \"refs/heads/HEAD\" is not actually\n>> updating the remote repositories $GIT_DIR/HEAD file, rather it creates a\n>> new reference $GIT_DIR/refs/heads/HEAD.\n>\n> Yes, that is what I would also expect. I checked one of the Git servers\n> we have and this is exactly what happens. $GIT_DIR/HEAD is a symref\n> pointing to refs/heads/main and $GIT_DIR/refs/heads/HEAD points to the\n> commit. But behaviour from client side is not consistent.\n>\n\nWhat is the non _consistent_ part?\n\n>>\n>> With this understanding you'll see that this is not a bug, because the\n>> remote HEAD was never updated, but only a new branch called HEAD was\n>> created [0].\n>\n> GitHub thinks so but try opening the branch. It won't show you the\n> commit (707a3d5, \"2.1\") but instead shows you 86e1c97 (\"2\"). So\n> something is wrong _at least_ with Github.\n>\n\nI don't know how GitHub operates, but I'm guessing because there is\nambiguity between a branch called HEAD and the actual HEAD. So this is\nprobably the reason.\n\n>>\n>>> Now with such a repo, if you do `git log --all --oneline` it would look\n>>> something like:\n>>>\n>>>     707a3d5 (origin/HEAD) 2.1\n>>>     86e1c97 (HEAD -> main, origin/main) 2\n>>>     79264c3 1\n>>>\n>>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n>>>\n>>>     ,origin,refs/remotes/origin/HEAD,2.1\n>>>     ,origin/main,refs/remotes/origin/main,2\n>>>\n>>> All well and good so far. Now delete the repo and attempt to clone it.\n>>> This time `git log --all --oneline` gives:\n>>>\n>>>     86e1c97 (HEAD -> main, origin/main, origin/HEAD) 2\n>>>     79264c3 1\n>>>\n>>\n>> This is expected since you cloned the repository and you got the default\n>> branch 'main'.\n>\n> No.\n>\n> First, if I clone a repo with multiple branches (say\n> https://github.com/prati0100/git-gui) I get _all_ the remote branches.\n> Yet here I clearly don't get the so called \"HEAD\" branch. This is not\n> expected behaviour.\n>\n\nYou're right, I meant to say that the remote branches don't have the\ncorresponding local branches. But that does not matter here.\n\nI'm not saying that there is a path for git to work properly when\ncreating a branch called \"HEAD\". It's just that \"HEAD\" is more of a\nreserved word for git and creating a branch with the same name has\nunintended effects.\n\n> Second, git really does misunderstand refs/remotes/origin/HEAD. For\n> example, when running git for-each-ref command with the clone method, I\n> get:\n>\n>     origin/main,origin,refs/remotes/origin/HEAD,2\n>\n> So it clearly thinks refs/remotes/origin/HEAD is at 86e1c97 (\"2\"). Or,\n> to be more specific, it thinks the ref points to origin/main which is at\n> 86e1c97 (\"2\"). But we set it at (707a3d5, \"2.1\"). So it tells me the\n> wrong thing. Now if I do the git remote add && git remote update method,\n> git for-each-ref says:\n>\n>     ,origin,refs/remotes/origin/HEAD,2.1\n>\n\nThis is one of those ambiguities, we store HEAD for remotes as\n     $GIT_DIR/refs/remotes/<remote>/HEAD\nand remote branches as\n     $GIT_DIR/refs/remotes/<remote>/<branch>\n\nSo what happens if there is a branch named HEAD? This is the problem\nyou're facing...\n\n> So now it thinks refs/remotes/origin/HEAD points at (707a3d5, \"2.1\"). I\n> do not see it as expected behaviour.\n>\n> We can also see this when inspecting the contents of\n> .git/refs/remotes/origin/HEAD. With clone it says:\n>\n>     ref: refs/remotes/origin/main\n>\n> With git remote add && git remote update it says:\n>\n>     707a3d587c61c089710e3924eb63a51763b5a4c8\n>\n> The same ref points to different places based on how you pull the repo.\n>\n> Looking deeper, if you clone a repo that does not have a branch called\n> \"HEAD\" (like git-gui), git creates a file in\n> .git/refs/remotes/origin/HEAD that says:\n>\n>     ref: refs/remotes/origin/master\n>\n> So it certainly seems to use refs/remotes/origin/HEAD to point to the\n> remote's HEAD, and not as a regular branch.\n>\n> I find this to be inconsistent behaviour on git's part and do not think\n> it is (or should be) expected behaviour.\n>\n\nMaybe we should explicitly mention that using HEAD as the branch name\nhas unintended effects and should be avoided.\n\n>>\n>>> And running `git for-each-ref --format='%(symref:short),%(refname:short),%(refname),%(subject)' refs/remotes/origin` gives:\n>>>\n>>>     origin/main,origin,refs/remotes/origin/HEAD,2\n>>>     ,origin/main,refs/remotes/origin/main,2\n>>>\n>>> So suddenly the remote's HEAD becomes origin/main (symbolic ref) and the\n>>> commit (707a3d5, \"2.1\") is nowhere to be found. It neither shows up in\n>>> `git rev-list --all` nor in `git log --all`. The files and trees\n>>> associated with it also do not show up in `git rev-list --all --object`.\n>>\n>>\n>> Because rev-list's `--all`, iterates over all refs. Since you only\n>> cloned, the HEAD branch is not pulled.\n>\n> Why not? When you clone all branches should get pulled.\n>\n\nI think I jumped too quick here, it is because the branch HEAD is never\nrealized locally as I explained above.\n"},{"id":"486841","messageId":"20240116150027.GB2119690@coredump.intra.peff.net","threadId":"60747","inReplyTo":"CAOLa=ZRfr+oEKCo8AfFSAFtS8pbDgmG_EeBSwm7GwukzVcqSrg@mail.gmail.com","subject":"Re: Strange behaviour when pushing a commit object to remote's refs/HEAD","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-01-16T15:00:27Z","receivedAt":"2024-01-16T15:00:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 16, 2024 at 08:24:04AM -0500, Karthik Nayak wrote:\n\n> This is one of those ambiguities, we store HEAD for remotes as\n>      $GIT_DIR/refs/remotes/<remote>/HEAD\n> and remote branches as\n>      $GIT_DIR/refs/remotes/<remote>/<branch>\n> \n> So what happens if there is a branch named HEAD? This is the problem\n> you're facing...\n\nYeah, this is a long-standing issue. The reason we have not fixed it is\nthat it would require a new refs/remotes layout, which implies new\nlookup rules (e.g., dwim_ref() will convert the name \"foo\" to\n\"refs/remotes/foo/HEAD\", but would need to be taught about the new\nlayout). Likewise, a new layout should probably store per-remote tags\n(rather than splatting them into the main refs/tags/) along with new\ndwim_ref() rules to make lookup work more or less as it does now.\n\nSo it's not impossible, but some care has to be given the design and\nto handling compatibility. If anybody is interested, there are probably\nsome nuggets of wisdom to mine from this old thread:\n\n  https://lore.kernel.org/git/AANLkTi=yFwOAQMHhvLsB1_xmYOE9HHP2YB4H4TQzwwc8@mail.gmail.com/\n\nIn the meantime, I think the current wisdom is \"don't name a branch\nHEAD\". ;) We even added logic to \"git branch\" to forbid this, but tools\nlike \"git push\" are a bit more flexible.\n\n-Peff\n"}]}