{"thread":{"id":"62746","subject":"Testing for existence of a remote branch from a script","startedAt":"2025-01-06T04:40:47Z","lastAt":"2025-01-06T20:51:06Z","messageCount":7,"participants":["Chris Packham","Torsten Bögershausen","Theodore Ts'o","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"509947","messageId":"CAFOYHZDQs-mftqLQn5HiFgBWcFN6Z-WDscJt=zVLRyGTo36=HQ@mail.gmail.com","threadId":"62746","inReplyTo":null,"subject":"Testing for existence of a remote branch from a script","fromName":"Chris Packham","fromEmail":"judge.packham@gmail.com","sentAt":"2025-01-06T04:40:36Z","receivedAt":"2025-01-06T04:40:47Z","isPatch":false,"sender":{"key":"judge.packham@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155667?v=4"},"body":"Hi,\n\nI look after some scripts we use at $dayjob for pushing changes though\nour review system.\n\nFor some of our repositories we operate a triangular workflow where\nchanges are fetched from one branch (e.g. 'foo') but are pushed to a\ndifferent one ('foo_incoming'). Our CI system runs to test the changes\nand when they pass 'foo_incoming' is merged (fast-forward most of the\ntime) into 'foo'.\n\nThe problem I have is not all our projects use this workflow so I've\ntried to automate the detection of this. My script does something like\n\n br=$(git rev-parse --symbolic-full-name\nrefs/remotes/origin/foo_incoming -- 2>/dev/null || echo\nrefs/remotes/origin/foo)\n\nThe '--' is necessary because if foo_incoming doesn't exist then there\nis extra output on stdout that puts off users. But when foo_incoming\ndoes exist then br gets set to `refs/remotes/origin/foo_incoming\\n--`.\n\nIs there a better way of checking for the existence of a remote branch?\n\nThanks,\nChris\n"},{"id":"509949","messageId":"20250106065121.GA8844@tb-raspi4","threadId":"62746","inReplyTo":"CAFOYHZDQs-mftqLQn5HiFgBWcFN6Z-WDscJt=zVLRyGTo36=HQ@mail.gmail.com","subject":"Re: Testing for existence of a remote branch from a script","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2025-01-06T06:51:21Z","receivedAt":"2025-01-06T06:51:30Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, Jan 06, 2025 at 05:40:36PM +1300, Chris Packham wrote:\n> Hi,\n>\n> I look after some scripts we use at $dayjob for pushing changes though\n> our review system.\n>\n> For some of our repositories we operate a triangular workflow where\n> changes are fetched from one branch (e.g. 'foo') but are pushed to a\n> different one ('foo_incoming'). Our CI system runs to test the changes\n> and when they pass 'foo_incoming' is merged (fast-forward most of the\n> time) into 'foo'.\n>\n> The problem I have is not all our projects use this workflow so I've\n> tried to automate the detection of this. My script does something like\n>\n> br=$(git rev-parse --symbolic-full-name\n> refs/remotes/origin/foo_incoming -- 2>/dev/null || echo\n> refs/remotes/origin/foo)\n>\n> The '--' is necessary because if foo_incoming doesn't exist then there\n> is extra output on stdout that puts off users. But when foo_incoming\n> does exist then br gets set to `refs/remotes/origin/foo_incoming\\n--`.\n>\n> Is there a better way of checking for the existence of a remote branch?\n\nI may have missed something, would that work:\ngit fetch -p\ngit branch -r\n"},{"id":"509994","messageId":"20250106150549.GE1284777@mit.edu","threadId":"62746","inReplyTo":"CAFOYHZDQs-mftqLQn5HiFgBWcFN6Z-WDscJt=zVLRyGTo36=HQ@mail.gmail.com","subject":"Re: Testing for existence of a remote branch from a script","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2025-01-06T15:05:49Z","receivedAt":"2025-01-06T15:05:54Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Jan 06, 2025 at 05:40:36PM +1300, Chris Packham wrote:\n> I look after some scripts we use at $dayjob for pushing changes though\n> our review system.\n> ..\n> Is there a better way of checking for the existence of a remote branch?\n\nIn gce-xfstests[1] I do this via \"git ls-remote\":\n\nvalidate_branch_name()\n{\n if test -z \"$GIT_REPO\"\n then\n echo \"GIT_REPO is neither found in the config file nor provided with --repo\"\n exit 1\n fi\n if [[ \"$GIT_REPO\" != *\".git\" ]]; then\n GIT_REPO=\"$GIT_REPO.git\"\n fi\n if ! git ls-remote \"$GIT_REPO\" > /dev/null; then\n\techo -e \"Repo not found: $GIT_REPO\\n\"\n\texit 1\n elif ! git ls-remote --heads --exit-code \"$GIT_REPO\" $1 > /dev/null; then\n\techo -e \"$1 is not a valid branch of $GIT_REPO\"\n\texit 1\n fi\n}\n\nSee run-fstests/util/parse_cli[2] for this function, and a related\nfunction, validate_commit_name if you also want to accept git tags.\n(The validate_commit_name function isn't perfect, since I don't want\nto fetch the full git repo to validate a SHA hash specifier, but it's\ngood enough to validate typo'ed tag or branch names.)\n\n[1] https://thunk.org/gce-xfstests\n[2] https://github.com/tytso/xfstests-bld/blob/master/run-fstests/util/parse_cli\n\nCheers,\n\n\t\t\t\t\t\t- Ted\n"},{"id":"509995","messageId":"xmqqsepw0xk7.fsf@gitster.g","threadId":"62746","inReplyTo":"20250106065121.GA8844@tb-raspi4","subject":"Re: Testing for existence of a remote branch from a script","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-06T15:30:32Z","receivedAt":"2025-01-06T15:30:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> changes are fetched from one branch (e.g. 'foo') but are pushed to a\n>> different one ('foo_incoming'). Our CI system runs to test the changes\n>> and when they pass 'foo_incoming' is merged (fast-forward most of the\n>> time) into 'foo'.\n>> ...\n>> Is there a better way of checking for the existence of a remote branch?\n>\n> I may have missed something, would that work:\n> git fetch -p\n> git branch -r\n\nOr \"git ls-remote origin refs/heads/foo-incoming\" and see if it\nyields anything?\n\nThe \"workflow\" makes me wonder how it would be bootstrapped, though.\n\nWhen you add a new branch, \"bar\", to be pre-reviewed before getting\nmerged at the server side, you want people to push to\n\"bar-incoming\". Before the very initial update to \"bar-incoming\"\nthat pushes into \"bar-incoming\" and creates it, there wouldn't be\n\"bar-incoming\" on the remote side, would there? So \"see if\nfoo-incoming exists and change the behaviour on the client end\naccordingly\" may not be a strategy to pursue.\n\nIf we wanted to support the \"workflow\" natively, the way we would do\nso would be to introduce a protocol capability that allows the\nserver side to advertise:\n\n If you want to update branch B, you are not allowed to do so\n directly. Instead, you are expected to push your changes to\n update 'refs/for/B'\n\nand then \"git push\" on the client end would notice the capability\nand turns \"git push origin B\" (or, more likely, the user is on local\nbranch B that is to build on the remote branch B from 'origin', and\n\"git push\" with no arguments would do the right thing) into such an\nupdate.\n\nI am not suggesting that we jump to the above immediately. But the\nreason why I am bringing it up is because The \"how would I see if\nthey have foo-incoming?\" smells like seeking a way to implement such\na custom capability advertisement outside Git.\n\nA few random thoughts.\n\n - Would it be useful if we introduced the ability to advertise\n \"custom capabilities\" from the receiving end of the connection,\n that does not affect how the rest of Git behaves at all? It\n would be sort of the reverse of --server-option, which is a\n mechanism to let the client to tell the other side out of band\n information that the rest of Git is oblivious.\n\n The other side of course needs a way to inspect what capabilities\n are advertised. For \"--server-option\", I do not think our server\n end does anything special, but other implementations can act on\n them. This new thing can start the same way.\n\n - If this is a poor-man's custom capability advertisement, perhaps\n the server end can create a ref \"refs/capabilities/incoming\"\n (whose value does not really matter) and your client side can see\n if there is such a ref with \"ls-remote\"? That may be a more\n robust thing to do instead, perhaps, as you do not need to worry\n about \"What about a new branch 'bar'?\" bootstrapping issues.\n\n"},{"id":"510002","messageId":"20250106163636.GH1284777@mit.edu","threadId":"62746","inReplyTo":"xmqqsepw0xk7.fsf@gitster.g","subject":"Re: Testing for existence of a remote branch from a script","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2025-01-06T16:36:36Z","receivedAt":"2025-01-06T16:36:42Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Jan 06, 2025 at 07:30:32AM -0800, Junio C Hamano wrote:\n> - Would it be useful if we introduced the ability to advertise\n> \"custom capabilities\" from the receiving end of the connection,\n> that does not affect how the rest of Git behaves at all?\n\nSo if enhancing the git server's functionality (either via git\nls-remotes or some other operation) is on the table, one of the things\nthat I would really love is some way of asking the question is \"git\ncommit \" in the remote repository reachable via some branch\nor git tag?\", and optionally, \"which git branch/tag should be fetched\nif the testing infrastructure wants to be able to test that specific\ngit commit ID?\"\n\nI could solve that problem by doing a \"git fetch\", but in my case, the\nspecific repository in question might not even be on the local client\nwhere the user is executing \"gce-xfstests -c ext4/4k -g auto --repo\nstable-rc.git --commit 47c2f92131c4\". (This command launches a VM\nwhich actually builds the kernel, and I'd like to be able to give\nimmediate feedback about whether the command will fail due to the user\ntypo'ing the git commit id without either waiting for the Google Cloud\nVM to launch and then fail, or doing a \"git fetch\" and determining\nwhether the git commit is there. I might be executing this command\nwhile on a very slow networking link, such as when I'm in an airplane\nor on a cruise ship, and so the cost of doing the \"git fetch\" might be\nprohibitive.)\n\nThanks,\n\n\t\t\t\t\t\t- Ted\n"},{"id":"510004","messageId":"xmqqy0znye76.fsf@gitster.g","threadId":"62746","inReplyTo":"20250106163636.GH1284777@mit.edu","subject":"Re: Testing for existence of a remote branch from a script","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-06T18:44:45Z","receivedAt":"2025-01-06T18:44:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Theodore Ts'o\" writes:\n\n> So if enhancing the git server's functionality (either via git\n> ls-remotes or some other operation) is on the table, ...\n\nEnhancements that do not require breaking backward compatibility is\nalways on the table ;-).\n\n> one of the things\n> that I would really love is some way of asking the question is \"git\n> commit \" in the remote repository reachable via some branch\n> or git tag?\", and optionally, \"which git branch/tag should be fetched\n> if the testing infrastructure wants to be able to test that specific\n> git commit ID?\"\n\nBoth sounds like a useful thing to do, but I wonder how generic\nthese should be and at the same time how common a narrowed-down\nfeature would suffice. If we try to make it generally very useful,\nat some point, we'd cross the line where we'd be better off doing\n\"run ssh and execute these Git commands\" over the wire X-<.\n\nThere are server-side-minded folks who are extending \"cat-file --batch\"\nto allow you to ask about objects you do not have but the other end\nhas, if I am not mistaken, by the \"remote-object-info\" feature?\n\nI wonder if these more advanced \"info\" about objects you mentioned\nfit into the picture well as part of it.\n\nThanks.\n"},{"id":"510018","messageId":"CAFOYHZCWhwmHDUsB0jyz+kDLwMOtOX_W+8PXisi2NBP=HYARVA@mail.gmail.com","threadId":"62746","inReplyTo":"xmqqsepw0xk7.fsf@gitster.g","subject":"Re: Testing for existence of a remote branch from a script","fromName":"Chris Packham","fromEmail":"judge.packham@gmail.com","sentAt":"2025-01-06T20:50:54Z","receivedAt":"2025-01-06T20:51:06Z","isPatch":false,"sender":{"key":"judge.packham@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155667?v=4"},"body":"On Tue, Jan 7, 2025 at 4:30 AM Junio C Hamano wrote:\n>\n> Torsten Bögershausen writes:\n>\n> >> changes are fetched from one branch (e.g. 'foo') but are pushed to a\n> >> different one ('foo_incoming'). Our CI system runs to test the changes\n> >> and when they pass 'foo_incoming' is merged (fast-forward most of the\n> >> time) into 'foo'.\n> >> ...\n> >> Is there a better way of checking for the existence of a remote branch?\n> >\n> > I may have missed something, would that work:\n> > git fetch -p\n> > git branch -r\n>\n> Or \"git ls-remote origin refs/heads/foo-incoming\" and see if it\n> yields anything?\n\ngit ls-remote --exit-code origin refs/heads/foo_incoming would work\nfor me. And I think it would save me a fetch to get past one\nbootstrapping issue.\n\n>\n> The \"workflow\" makes me wonder how it would be bootstrapped, though.\n>\n\nCurrently we require the first person to know what they're doing and\ndo the initial push as `HEAD:refs/heads/foo_incoming` the CI tooling\nwe have set up knows to create `foo` if it doesn't already exist.\nThat's probably not too bad because for an unborn branch you'd need to\nsay `HEAD:refs/heads/foo` anyway before subsequent pushes would work\nwithout the refspec.\n\nOur workflow does require some discipline as that first push should\njust be either an existing commit (when branching in an existing repo)\nor a largely empty \"Initial commit\" for a new repository. That all\nrelies on a \"gentlepersons's agreement\" that the first push to one of\nthese branches doesn't require review because there's nothing really\nto enforce that policy. Pushing to the non-incoming branch is\nrestricted to the CI user via a hook on the server for plain git or\nwith access permissions for gerrit.\n\n> When you add a new branch, \"bar\", to be pre-reviewed before getting\n> merged at the server side, you want people to push to\n> \"bar-incoming\". Before the very initial update to \"bar-incoming\"\n> that pushes into \"bar-incoming\" and creates it, there wouldn't be\n> \"bar-incoming\" on the remote side, would there? So \"see if\n> foo-incoming exists and change the behaviour on the client end\n> accordingly\" may not be a strategy to pursue.\n\nI know git has some tooling for a triangular workflow\n(branch..pushRemote) I'm not sure if that can be setup to push\nto a differently named branch. Some kind of branch..pushRefSpec\nmight help if it could support *:*_incoming.\n\nThe main thing I'm scripting on top of is Gerrit's magic refs/for/* so\neven if git supported our version of a triangular workflow workflow\nI'd still need something to decide when to use refs/for/foo or\nrefs/for/foo_incoming when dealing with Gerrit.\n\n> If we wanted to support the \"workflow\" natively, the way we would do\n> so would be to introduce a protocol capability that allows the\n> server side to advertise:\n>\n> If you want to update branch B, you are not allowed to do so\n> directly. Instead, you are expected to push your changes to\n> update 'refs/for/B'\n>\n> and then \"git push\" on the client end would notice the capability\n> and turns \"git push origin B\" (or, more likely, the user is on local\n> branch B that is to build on the remote branch B from 'origin', and\n> \"git push\" with no arguments would do the right thing) into such an\n> update.\n\nFor the most part we're happy with using a pre-receive hook on the\nserver to block updates when required. We're able to put out an error\nmessage that tells the user to push to foo_incoming.\n\nHaving some kind of config that made `git push` do the right thing\nwould be great as it would save the user some typing. I don't know how\nmuch advertising from the git server this would really require (we\ncould easily update our reject hook to advertise some `git config`\nsettings).\n\n>\n> I am not suggesting that we jump to the above immediately. But the\n> reason why I am bringing it up is because The \"how would I see if\n> they have foo-incoming?\" smells like seeking a way to implement such\n> a custom capability advertisement outside Git.\n>\n> A few random thoughts.\n>\n> - Would it be useful if we introduced the ability to advertise\n> \"custom capabilities\" from the receiving end of the connection,\n> that does not affect how the rest of Git behaves at all? It\n> would be sort of the reverse of --server-option, which is a\n> mechanism to let the client to tell the other side out of band\n> information that the rest of Git is oblivious.\n>\n> The other side of course needs a way to inspect what capabilities\n> are advertised. For \"--server-option\", I do not think our server\n> end does anything special, but other implementations can act on\n> them. This new thing can start the same way.\n>\n> - If this is a poor-man's custom capability advertisement, perhaps\n> the server end can create a ref \"refs/capabilities/incoming\"\n> (whose value does not really matter) and your client side can see\n> if there is such a ref with \"ls-remote\"? That may be a more\n> robust thing to do instead, perhaps, as you do not need to worry\n> about \"What about a new branch 'bar'?\" bootstrapping issues.\n>\n"}]}