{"thread":{"id":"56901","subject":"[BUG] git push --mirror ignores refs outside head & tags","startedAt":"2021-11-15T06:54:12Z","lastAt":"2021-11-20T06:05:48Z","messageCount":5,"participants":["Robin H. Johnson","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"441114","messageId":"robbat2-20211115T063838-612792475Z@orbis-terrarum.net","threadId":"56901","inReplyTo":null,"subject":"[BUG] git push --mirror ignores refs outside head & tags","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2021-11-15T06:53:21Z","receivedAt":"2021-11-15T06:54:12Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"Hi,\n\nTL;DR: \"git push --mirror\" does not in fact push all refs, despite\ndocumentation stating it does. It ONLY pushes refs in refs/heads/ and\nrefs/tags/.\n\nI ran into this while preparing to migrate Gentoo's primary Git server\n(which uses Gitolite).\n\nGitolite ships a post-receive hook, \"save-push-signatures\" [1] that save\nPush certificates into refs/push-certs.\n\nGitolite mirroring invokes 'git push --mirror' to mirror from the\nprimary location to replicas.\n\nI was surprised to find, on the new server presently configured as\nreplica, but intended to take over as the primary, that the\nrefs/push-certs was missing on every single repo.\n\n```\n# git push -h |grep -e mirror -e all\n --all push all refs\n\t--mirror mirror all refs\n```\ngit-push(1):\n```\n--mirror\n Instead of naming each ref to push, specifies that all refs under refs/ (which includes\n but is not limited to refs/heads/, refs/remotes/, and refs/tags/) be mirrored to the\n remote repository. ...\n```\n\ngit --version: 2.32.0\n\nAttached I also include the trace ref & trace packet output as well that shows\nit on a sample repo.\n```\nGIT_TRACE_REFS=/tmp/trace-refs \\\nGIT_TRACE_PACKET=/tmp/trace-packet \\\ngit push --mirror $REPLICA:$REPO\n```\n\nThe repo's gitconfig does not contain any remotes, because that's the way gitolite is built.\n\n[1] https://github.com/sitaramc/gitolite/blob/master/contrib/hooks/repo-specific/save-push-signatures\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Dev, Infra Lead, Foundation Treasurer\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85\nGnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136\n\n\n23:07:49.011947 pkt-line.c:80 packet: push< bd7bd3a513c8ddc554e211316c990b5f98110982 refs/heads/bug450326\\0report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta push-cert=1636931268-40bfd0dc11027620d3ea77024b5aa5aa90ed2342 object-format=sha1 agent=git/2.32.0\n23:07:49.011987 pkt-line.c:80 packet: push< 65c923b603247af68f1b4405363d92c1c19009eb refs/heads/bug504458\n23:07:49.011993 pkt-line.c:80 packet: push< 6e81ff9f85736180516941e630b1db78bd9bf2a2 refs/heads/iw\n23:07:49.011998 pkt-line.c:80 packet: push< e860f40ffd07a41e47a6b49d9400ad20e6a8b7ca refs/heads/master\n23:07:49.012002 pkt-line.c:80 packet: push< 56dd91ee7737dd3d4ebc57185632f407a2cdf3c1 refs/heads/remove-bash-arrays\n23:07:49.012007 pkt-line.c:80 packet: push< 09c6d951fe3c4419317fcd9488df907bd11ac02c refs/tags/0.1\n23:07:49.012012 pkt-line.c:80 packet: push< cb257cea980b74aa7dff4b7bfcfb66b82565bb01 refs/tags/0.2\n23:07:49.012016 pkt-line.c:80 packet: push< 22b3841561b0b686919be466f1caeae6833e30ba refs/tags/0.2.1\n23:07:49.012021 pkt-line.c:80 packet: push< 47d23f5abdac4a2bd95bcbbaadacf994773cbf51 refs/tags/0.2.2\n23:07:49.012025 pkt-line.c:80 packet: push< e2e76bb9c2a7bee77e29faf8f9afb2a89fe19e11 refs/tags/0.2.3\n23:07:49.012030 pkt-line.c:80 packet: push< 986029121ae49984c98a76efa83d4e48d5c9f650 refs/tags/0.3.0\n23:07:49.012035 pkt-line.c:80 packet: push< 904d1b04cb0a147fe39f9952a281563faeb19d48 refs/tags/0.3.1\n23:07:49.012039 pkt-line.c:80 packet: push< a143c1b3a6896cdfe62dd24a0e95ee442c5469e2 refs/tags/0.4.0\n23:07:49.012044 pkt-line.c:80 packet: push< edd52106d7c868c86c396e0c2f486e58a6132d3c refs/tags/0.5.0\n23:07:49.012048 pkt-line.c:80 packet: push< c1b5722fe2dd9aee540cc08b73e7ac59c2f22c4d refs/tags/0.5.1\n23:07:49.012055 pkt-line.c:80 packet: push< 8ec196c7c708340e888573f66e34f9c5171efabf refs/tags/0.6.0\n23:07:49.012060 pkt-line.c:80 packet: push< 7ba81c3198da28101290eed259aef6645d258ebc refs/tags/0.6.1\n23:07:49.012064 pkt-line.c:80 packet: push< 8cdfc30e5cca5da9ea12b7e275f044d452e6894d refs/tags/0.7.0\n23:07:49.012069 pkt-line.c:80 packet: push< 646886f098efddd62162bebb655e413fd99c214e refs/tags/0.7.1\n23:07:49.012073 pkt-line.c:80 packet: push< 93a79ef4b89ee383cdfb5e4be5f07c2ce4e61c90 refs/tags/0.7.2\n23:07:49.012078 pkt-line.c:80 packet: push< 3067e568abbf803e8f47a6e16bc39412a4539b1c refs/tags/0.7.3\n23:07:49.012082 pkt-line.c:80 packet: push< 0000\n23:07:49.012126 pkt-line.c:80 packet: push> 0000\n\n\n23:07:48.501620 refs/debug.c:27 ref_store for .\n23:07:48.501670 refs/debug.c:252 read_raw_ref: HEAD: 0000000000000000000000000000000000000000 (=> refs/heads/master) type 1: 0\n23:07:48.501687 refs/debug.c:252 read_raw_ref: refs/heads/master: e860f40ffd07a41e47a6b49d9400ad20e6a8b7ca (=> refs/heads/master) type 2: 0\n23:07:48.501738 refs/debug.c:235 ref_iterator_begin: (0x0)\n23:07:48.501790 refs/debug.c:191 iterator_advance: refs/heads/bug450326 (0)\n23:07:48.501797 refs/debug.c:191 iterator_advance: refs/heads/bug504458 (0)\n23:07:48.501801 refs/debug.c:191 iterator_advance: refs/heads/iw (0)\n23:07:48.501805 refs/debug.c:191 iterator_advance: refs/heads/master (0)\n23:07:48.501809 refs/debug.c:191 iterator_advance: refs/heads/remove-bash-arrays (0)\n23:07:48.501812 refs/debug.c:191 iterator_advance: refs/push-certs (0)\n23:07:48.501816 refs/debug.c:191 iterator_advance: refs/tags/0.1 (0)\n23:07:48.501820 refs/debug.c:191 iterator_advance: refs/tags/0.2 (0)\n23:07:48.501823 refs/debug.c:191 iterator_advance: refs/tags/0.2.1 (0)\n23:07:48.501827 refs/debug.c:191 iterator_advance: refs/tags/0.2.2 (0)\n23:07:48.501831 refs/debug.c:191 iterator_advance: refs/tags/0.2.3 (0)\n23:07:48.501834 refs/debug.c:191 iterator_advance: refs/tags/0.3.0 (0)\n23:07:48.501838 refs/debug.c:191 iterator_advance: refs/tags/0.3.1 (0)\n23:07:48.501841 refs/debug.c:191 iterator_advance: refs/tags/0.4.0 (0)\n23:07:48.501845 refs/debug.c:191 iterator_advance: refs/tags/0.5.0 (0)\n23:07:48.501848 refs/debug.c:191 iterator_advance: refs/tags/0.5.1 (0)\n23:07:48.501852 refs/debug.c:191 iterator_advance: refs/tags/0.6.0 (0)\n23:07:48.501855 refs/debug.c:191 iterator_advance: refs/tags/0.6.1 (0)\n23:07:48.501861 refs/debug.c:191 iterator_advance: refs/tags/0.7.0 (0)\n23:07:48.501865 refs/debug.c:191 iterator_advance: refs/tags/0.7.1 (0)\n23:07:48.501868 refs/debug.c:191 iterator_advance: refs/tags/0.7.2 (0)\n23:07:48.501872 refs/debug.c:191 iterator_advance: refs/tags/0.7.3 (0)\n23:07:48.501875 refs/debug.c:189 iterator_advance: (-1)\n23:07:49.042542 refs/debug.c:252 read_raw_ref: HEAD: 0000000000000000000000000000000000000000 (=> refs/heads/master) type 1: 0\n23:07:49.042563 refs/debug.c:252 read_raw_ref: refs/heads/master: e860f40ffd07a41e47a6b49d9400ad20e6a8b7ca (=> refs/heads/master) type 2: 0\n"},{"id":"441161","messageId":"YZLLUtPVQt2XtgQX@coredump.intra.peff.net","threadId":"56901","inReplyTo":"robbat2-20211115T063838-612792475Z@orbis-terrarum.net","subject":"Re: [BUG] git push --mirror ignores refs outside head & tags","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-11-15T21:04:18Z","receivedAt":"2021-11-15T21:27:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 15, 2021 at 06:53:21AM +0000, Robin H. Johnson wrote:\n\n> TL;DR: \"git push --mirror\" does not in fact push all refs, despite\n> documentation stating it does. It ONLY pushes refs in refs/heads/ and\n> refs/tags/.\n\nPerhaps there's a bug, but it is meant to and does push all refs in a\nsimple case:\n\n git init\n git commit --allow-empty -m foo\n git update-ref refs/heads/other-branch HEAD\n git update-ref refs/foo/bar HEAD\n\n git init --bare dst.git\n git push --mirror dst.git\n\nproduces:\n\n To dst.git\n * [new reference] refs/foo/bar -> refs/foo/bar\n * [new branch] main -> main\n * [new branch] other-branch -> other-branch\n\n> I was surprised to find, on the new server presently configured as\n> replica, but intended to take over as the primary, that the\n> refs/push-certs was missing on every single repo.\n\nAh. You are not likely to have success pushing a single-level ref like\nthat, as receive-pack rejects it:\n\n $ git update-ref refs/push-certs HEAD\n $ git push dst.git refs/push-certs\n Total 0 (delta 0), reused 0 (delta 0), pack-reused 0\n remote: error: refusing to create funny ref 'refs/push-certs' remotely\n To dst.git\n ! [remote rejected] refs/push-certs -> refs/push-certs (funny refname)\n error: failed to push some refs to 'dst.git'\n\nThere I used an explicit refspec naming it. But if I used \"refs/*\"\n(which is what --mirror is doing under the hood), then it doesn't even\ntry sending it, as the client filters it out from the wildcard\n(otherwise, everyone would get server-side errors from refs/stash).\n\nSo you probably want to choose a different refname to store your data.\n\nI do think the status of these single-level refs is not well documented.\nThe rules in git-check-ref-format(1) imply that they are not valid:\n\n They must contain at least one /. This enforces the presence of a\n category like heads/, tags/ etc. but the actual names are not\n restricted.\n\nbut that rule is not enforced internally, as \"refs/\" is sufficient for\nthe internal check_refname_format() to allow it. But receive-pack has,\nsince 1a7141ff28 (Ignore funny refname sent from remote, 2005-10-13),\nimplemented the format check after stripping refs/. And then the client\nside followed that lead in 30affa1e9a (send-pack: do not send out\nsingle-level refs such as refs/stash, 2008-10-29).\n\n-Peff\n"},{"id":"441257","messageId":"robbat2-20211116T061554-223495439Z@orbis-terrarum.net","threadId":"56901","inReplyTo":"YZLLUtPVQt2XtgQX@coredump.intra.peff.net","subject":"Re: [BUG] git push --mirror ignores refs outside head & tags","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2021-11-16T06:28:28Z","receivedAt":"2021-11-16T06:32:14Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"On Mon, Nov 15, 2021 at 04:04:18PM -0500, Jeff King wrote:\n> On Mon, Nov 15, 2021 at 06:53:21AM +0000, Robin H. Johnson wrote:\n> \n> > TL;DR: \"git push --mirror\" does not in fact push all refs, despite\n> > documentation stating it does. It ONLY pushes refs in refs/heads/ and\n> > refs/tags/.\n> \n> Perhaps there's a bug, but it is meant to and does push all refs in a\n> simple case:\n...\n> There I used an explicit refspec naming it. But if I used \"refs/*\"\n> (which is what --mirror is doing under the hood), then it doesn't even\n> try sending it, as the client filters it out from the wildcard\n> (otherwise, everyone would get server-side errors from refs/stash).\n> \n> So you probably want to choose a different refname to store your data.\n> \n> I do think the status of these single-level refs is not well documented.\n> The rules in git-check-ref-format(1) imply that they are not valid:\nOk, that's great, but this single-level ref have been in existence for\nmany years at this point. Gitolite added the save-push-signatures hook\nin 2014.\n\n- What is a meaningful name for the push-certs ref that follows the\n 2-level system?\n- What's a safe migration path for Gitolite?\n - How can it atomically fix each existing repo with the problem?\n- What's a safe migration path for other consumers?\n\n> \n> They must contain at least one /. This enforces the presence of a\n> category like heads/, tags/ etc. but the actual names are not\n> restricted.\n> \n> but that rule is not enforced internally, as \"refs/\" is sufficient for\n> the internal check_refname_format() to allow it. But receive-pack has,\n> since 1a7141ff28 (Ignore funny refname sent from remote, 2005-10-13),\n> implemented the format check after stripping refs/. And then the client\n> side followed that lead in 30affa1e9a (send-pack: do not send out\n> single-level refs such as refs/stash, 2008-10-29).\n\nAnd the format of the ref wasn't noticed in the hook was previously\nposted back in 2014 :-(.\nhttps://www.spinics.net/lists/git/msg244053.html\nhttps://groups.google.com/forum/#!topic/gitolite/7cSrU6JorEY\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Dev, Infra Lead, Foundation Treasurer\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85\nGnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136\n"},{"id":"441823","messageId":"robbat2-20211119T234158-903098822Z@orbis-terrarum.net","threadId":"56901","inReplyTo":"robbat2-20211116T061554-223495439Z@orbis-terrarum.net","subject":"[RFC] single-level refs vs push --all/--mirror","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2021-11-20T00:03:36Z","receivedAt":"2021-11-20T00:04:03Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"On Tue, Nov 16, 2021 at 06:28:28AM +0000, Robin H. Johnson wrote:\n> > Perhaps there's a bug, but it is meant to and does push all refs in a\n> > simple case:\n> ...\n> > There I used an explicit refspec naming it. But if I used \"refs/*\"\n> > (which is what --mirror is doing under the hood), then it doesn't even\n> > try sending it, as the client filters it out from the wildcard\n> > (otherwise, everyone would get server-side errors from refs/stash).\n> > \n> > So you probably want to choose a different refname to store your data.\n> > \n> > I do think the status of these single-level refs is not well documented.\n> > The rules in git-check-ref-format(1) imply that they are not valid:\n..\n> > They must contain at least one /. This enforces the presence of a\n> > category like heads/, tags/ etc. but the actual names are not\n> > restricted.\n> > \n> > but that rule is not enforced internally, as \"refs/\" is sufficient for\n> > the internal check_refname_format() to allow it. But receive-pack has,\n> > since 1a7141ff28 (Ignore funny refname sent from remote, 2005-10-13),\n> > implemented the format check after stripping refs/. And then the client\n> > side followed that lead in 30affa1e9a (send-pack: do not send out\n> > single-level refs such as refs/stash, 2008-10-29).\n> \n> And the format of the ref wasn't noticed in the hook was previously\n> posted back in 2014 :-(.\n> https://www.spinics.net/lists/git/msg244053.html\n> https://groups.google.com/forum/#!topic/gitolite/7cSrU6JorEY\n\nHi,\n\nI didn't see response from Jeff King, so I'm wondering good next steps\nhere.\n\nI'm esp. surprised that git-stash ends up using single-level refs when\ngit-check-ref-format(1) says they aren't valid.\n\nI think Git should change git-stash and start issuing warnings to users\nfor single-level refs.\n\nTherein leads a question:\nWhat should be done with refs/stash?\n\nBeing able to work on stashes between multiple systems MIGHT be useful\nto some users, so maybe they do want it pushed with --mirror/--all.\n\nChoices (not mutually exclusive):\n1. Push the refs anyway\n2. Make a configurable way to exclude refs from being pushed w/ --mirror/--all\n e.g. move refs/stash to the exclude list, then push everything not\n that is not excluded.\n3. Change the rules to accept single-level refs?\n4. Cleanup phases of warnings before errors and good migration\n mechanisms?\n5. Something else?\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Dev, Infra Lead, Foundation Treasurer\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85\nGnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136\n"},{"id":"441838","messageId":"xmqqlf1jnyyz.fsf@gitster.g","threadId":"56901","inReplyTo":"robbat2-20211119T234158-903098822Z@orbis-terrarum.net","subject":"Re: [RFC] single-level refs vs push --all/--mirror","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-20T06:05:40Z","receivedAt":"2021-11-20T06:05:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Robin H. Johnson\" writes:\n\n> I didn't see response from Jeff King, so I'm wondering good next steps\n> here.\n>\n> I'm esp. surprised that git-stash ends up using single-level refs when\n> git-check-ref-format(1) says they aren't valid.\n\nThey aren't valid name to be used as regular branches, tags, etc.,\nbut as an implementation detail of stash, \"refs/stash\" is perfectly\ngood. There is nothing to \"fix\". You cannot and you do not want to\npretend \"stash\" is yet another ref in the first place.\n\nYou can \"transfer\" refs/stash by pushing the commit to an arbitrary\nref with a regular \"three level\" name, like so:\n\n $ git push there refs/stash:refs/my/stash\n\nbut it wouldn't be of much use to begin with.\n\nThe thing is, the \"list of random quick changes stashed away\" is not\nsomething stored in the \"stash\" ref. These list entries are stored\nas reflog entries for the stash ref, which is *never* transferred\nover the network. So, if the higher level issue you want to address\nis to allow \"stash\" to be shared across repositories, none of these\n5 choices you listed at the end of your message helps all that much.\n\nThe single-level refs are the least of your problems.\n\nInstead, you'd probably want to reimplement \"stash\" as a set of\nnormal refs, whose current value only matters, e.g. refs/stash/0\nmay be the oldest stash, refs/stash/1 is the next one, refs/stash/2\nis yet another new one, etc., and have UIs like \"git stash list\" \nlist them in a new way that is different from the current reflog\nbased implementation, and \"git stash pop/apply\" take them, e.g.\n\n $ git stash list\n stash/0: WIP on main: cd3e606211 Git 2.34\n stash/4: WIP on maint: Merge branch 'vd/pthread-setspecific-g11-fix' into maint\n $ git stash apply stash/0\n\nAnd at that point you'd have refs/stash/* as an intermediate\nhierarchy, with another level of real refs hanging there, so you can\ntransfer them just like refs/tags/* all you want.\n\n> I think Git should change git-stash and start issuing warnings to users\n> for single-level refs.\n\nNo, single-level refs is perfectly fine, as long as you are using\nthe current Git and using these refs locally. The problem arises\nonly when you start wanting to share stash across repositories, and\nit is not from the levels of the refname hierarchy but from the fact\nthat stash is implemented in terms of reflog mechanism that is not\nshared across repositories.\n\n\n"}]}