{"thread":{"id":"60284","subject":"Possible to update-ref remote repository?","startedAt":"2023-09-30T16:35:01Z","lastAt":"2023-10-04T01:13:44Z","messageCount":5,"participants":["Jesse Hopkins","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482469","messageId":"CAL3By--HowOL1ffKBPfmwnfUdJd4KXcnkpS2BgkbO=9E2WnHKw@mail.gmail.com","threadId":"60284","inReplyTo":null,"subject":"Possible to update-ref remote repository?","fromName":"Jesse Hopkins","fromEmail":"jesse.hops@gmail.com","sentAt":"2023-09-30T16:34:41Z","receivedAt":"2023-09-30T16:35:01Z","isPatch":false,"sender":{"key":"jesse.hops@gmail.com","avatar":null},"body":"Hello -\n\nWondering if it's possible to do the equivalent of the update-ref\ncommand remotely.  Or I guess another way of putting it would be to\ngit-push to a remote repository without needing a local clone of the\nrepo.\n\nTrying do something like:\n\ngit push <remote-repo-url>  <sha1>:refs/heads/mybranchtoupdate\n\nwhere I know that <sha1> already exists on the remote.  I'd like to\navoid the need to clone a local copy of the repo.  Wondering if there\nmight be some plumbing command(s) that could accomplish this?\n\nRegards,\nJesse\n\nP.S. I think think a minimal way to do this using git push would be:\n\ngit init\ngit remote add origin <remote-repo-url>\ngit fetch origin <sha1>:refs/heads/mybranchtoupdate\ngit push origin refs/heads/mybranchtoupdate:refs/heads/mybranchtoupdate\n\nSeems that if it's known that <sha1> already exists on the remote, the\nfetch is unnecessary network overhead?\n"},{"id":"482470","messageId":"xmqqo7hjfumj.fsf@gitster.g","threadId":"60284","inReplyTo":"CAL3By--HowOL1ffKBPfmwnfUdJd4KXcnkpS2BgkbO=9E2WnHKw@mail.gmail.com","subject":"Re: Possible to update-ref remote repository?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-30T17:17:40Z","receivedAt":"2023-09-30T17:17:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jesse Hopkins <jesse.hops@gmail.com> writes:\n\n> Wondering if it's possible to do the equivalent of the update-ref\n> command remotely.  Or I guess another way of putting it would be to\n> git-push to a remote repository without needing a local clone of the\n> repo.\n>\n> Trying do something like:\n>\n> git push <remote-repo-url>  <sha1>:refs/heads/mybranchtoupdate\n>\n> where I know that <sha1> already exists on the remote.  I'd like to\n> avoid the need to clone a local copy of the repo.  Wondering if there\n> might be some plumbing command(s) that could accomplish this?\n\nThere is no such command shipped with Git.  I do not think anybody\never proposed to add such a feature, as it won't be generally useful\nin contexts other than one-shot surgery.\n\nBut that is different from that it is impossible to do.  At the\nprotocol level, as long as the receiving end is convinced that such\na push, which does not transfer any object but proposes to update a\nref to a new value, does not corrupt the resulting repository state\nafter acceptingit, the receiving end does not care if the sending\nend has what object, or even any repository---the receiving end\ncannot even know what the sending end actually has.\n\nThe devil however is in the details of ensuring the \"as long as the\nreceiving end is convinced\" part.  If you know that an object whose\nobject name is X is sitting at the tip of branch A at the remote,\nand you try to update the tip of branch B to the same object, it is\nlikely [*] that the remote would notice that such an update after\nreceiving no new objects is safe.  But if the object X is not\nsitting at the tip of any branch or ref (it may be a few commits\nbehind an existing ref, or it may be dangling ahead of all refs), it\nwould depend on how the receiving end determines that it has object\nchains necessary to complete the new DAG that has X.  The official\nversions of Git client historically have done a thorough job in\ncheck_connected() to even discover fully connected object chains\nthat allows us to resurrect such a dangling tip of a DAG, but we\ncannot complain if third-party Git implementations misses less\nobvious cases.\n\nHaving said all that, writing a specialized \"push lookalike\" that\nsends the required protocol message that would have been spewed by a\nreal \"git push\" client operating in the right environment (i.e., it\nhas a repository that is a good clone of the receiving repository at\nthe <remote-repo-url>, the receiving repository has <sha1> and all\nthe objects that are reachable from it, refs/heads/mybranchtoupdate\npoints at a commit that is an ancestor of <sha1> right now) should\nnot be brain surgery, and with such a program you can\n\n $ git-push-empty <remote-repo-url> <sha1>:refs/heads/mybranchtoupdate\n\nand fool the receiving end to do what you want it to do, as it\ncannot tell if it is talking to a real \"git push\" client or to your\n\"push-empty\" program from what is coming over the wire.\n\n\n\n"},{"id":"482477","messageId":"CAL3By-_jrYDenCc8HSrtnR3cZD19z7bvwVOOoO4XG+6aNFxyeQ@mail.gmail.com","threadId":"60284","inReplyTo":"xmqqo7hjfumj.fsf@gitster.g","subject":"Re: Possible to update-ref remote repository?","fromName":"Jesse Hopkins","fromEmail":"jesse.hops@gmail.com","sentAt":"2023-10-01T22:03:29Z","receivedAt":"2023-10-01T22:03:53Z","isPatch":false,"sender":{"key":"jesse.hops@gmail.com","avatar":null},"body":"Thanks for the reply.  Would you be able to point me to some\nbreadcrumbs for the \"required protocol messages\"?  I might try to\ntinker in some spare time.\n\nFWIW, I had put something like this together a while back using\nGitolite hosting (a dedicated SSH command similar to Gitolite's\nsymbolic-ref command:\nhttps://github.com/sitaramc/gitolite/blob/master/src/commands/symbolic-ref).\n\nOur org is using Gitlab now, and I have been able to put some\nfunctionality together using the Gitlab API's, but it's quite ugly,\nand was hoping that maybe there was an intrinsic git protocol\nsolution, which seems there could be, but somewhere past trivial and\nbefore impossible.\n\n-Jesse\n\nOn Sat, Sep 30, 2023 at 11:17 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Jesse Hopkins <jesse.hops@gmail.com> writes:\n>\n> > Wondering if it's possible to do the equivalent of the update-ref\n> > command remotely.  Or I guess another way of putting it would be to\n> > git-push to a remote repository without needing a local clone of the\n> > repo.\n> >\n> > Trying do something like:\n> >\n> > git push <remote-repo-url>  <sha1>:refs/heads/mybranchtoupdate\n> >\n> > where I know that <sha1> already exists on the remote.  I'd like to\n> > avoid the need to clone a local copy of the repo.  Wondering if there\n> > might be some plumbing command(s) that could accomplish this?\n>\n> There is no such command shipped with Git.  I do not think anybody\n> ever proposed to add such a feature, as it won't be generally useful\n> in contexts other than one-shot surgery.\n>\n> But that is different from that it is impossible to do.  At the\n> protocol level, as long as the receiving end is convinced that such\n> a push, which does not transfer any object but proposes to update a\n> ref to a new value, does not corrupt the resulting repository state\n> after acceptingit, the receiving end does not care if the sending\n> end has what object, or even any repository---the receiving end\n> cannot even know what the sending end actually has.\n>\n> The devil however is in the details of ensuring the \"as long as the\n> receiving end is convinced\" part.  If you know that an object whose\n> object name is X is sitting at the tip of branch A at the remote,\n> and you try to update the tip of branch B to the same object, it is\n> likely [*] that the remote would notice that such an update after\n> receiving no new objects is safe.  But if the object X is not\n> sitting at the tip of any branch or ref (it may be a few commits\n> behind an existing ref, or it may be dangling ahead of all refs), it\n> would depend on how the receiving end determines that it has object\n> chains necessary to complete the new DAG that has X.  The official\n> versions of Git client historically have done a thorough job in\n> check_connected() to even discover fully connected object chains\n> that allows us to resurrect such a dangling tip of a DAG, but we\n> cannot complain if third-party Git implementations misses less\n> obvious cases.\n>\n> Having said all that, writing a specialized \"push lookalike\" that\n> sends the required protocol message that would have been spewed by a\n> real \"git push\" client operating in the right environment (i.e., it\n> has a repository that is a good clone of the receiving repository at\n> the <remote-repo-url>, the receiving repository has <sha1> and all\n> the objects that are reachable from it, refs/heads/mybranchtoupdate\n> points at a commit that is an ancestor of <sha1> right now) should\n> not be brain surgery, and with such a program you can\n>\n>  $ git-push-empty <remote-repo-url> <sha1>:refs/heads/mybranchtoupdate\n>\n> and fool the receiving end to do what you want it to do, as it\n> cannot tell if it is talking to a real \"git push\" client or to your\n> \"push-empty\" program from what is coming over the wire.\n>\n>\n>\n"},{"id":"482595","messageId":"20231003200018.GB1562@coredump.intra.peff.net","threadId":"60284","inReplyTo":"CAL3By-_jrYDenCc8HSrtnR3cZD19z7bvwVOOoO4XG+6aNFxyeQ@mail.gmail.com","subject":"Re: Possible to update-ref remote repository?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-10-03T20:00:18Z","receivedAt":"2023-10-03T20:00:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Oct 01, 2023 at 04:03:29PM -0600, Jesse Hopkins wrote:\n\n> Thanks for the reply.  Would you be able to point me to some\n> breadcrumbs for the \"required protocol messages\"?  I might try to\n> tinker in some spare time.\n\nTry \"git help protocol-pack\" in a recent version of Git (this used to be\nin Documentation/technical/ of the repository, but much of that content\nwas moved into manpages around the v2.38 timeframe).\n\nFor a local or ssh connection, I think it is as simple as:\n\n  # you somehow happen to know this commit exists on the server,\n  # and what the current value of the ref is. If you don't know the\n  # current value, you can pull it from receive-pack's ref\n  # advertisement (I'll leave that as an exercise for the reader).\n  old=1234abcd...\n  new=5678cdef...\n  ref=refs/heads/main\n\n  # we'll use a local repository here, but you can replace receive-pack\n  # invocation below with with \"ssh $host git receive-pack $repo\"\n  repo=/path/to/repo.git\n\n  {\n    # git's pkt-line format is a 4-byte header with the ascii hex size of\n    # the packet, followed by N-4 bytes of data. Each ref update is\n    # in its own pkt, but we have just one.\n    cmd=\"$old $new $ref\"\n    printf \"%04x%s\" $((${#cmd} + 4)) \"$cmd\"\n\n    # An all-zero flush packet indicates the end of the list of updates.\n    printf \"0000\"\n\n    # the server insists that we send a valid packfile, even if it is\n    # empty. This is from \"git help format-pack\" (the section on .pack\n    # files), though you could also generate it with \"git pack-objects\n    # --stdout </dev/null\".\n    printf 'PACK' ;# packfile\n    printf '\\0\\0\\0\\2' ;# version 2\n    printf '\\0\\0\\0\\0' ;# zero objects\n    # checksum, which is the sha1 of the rest of the pack\n    printf \"\\2\\235\\10\\202\\73\\330\\250\\352\\265\\20\\255\\152\\307\\134\\202\\74\\375\\76\\323\\36\"\n  } |\n  git receive-pack \"$repo\"\n\nYou can get fancier by specifying capabilities (you might want\n\"report-status\", for example).\n\nThat will work for local or ssh repos. For http, it gets a little more\ncomplicated. See the section \"smart service git-receive-pack\" of \"git\nhelp protocol-http\".\n\nAll that said, I do think it might be reasonable for git-push to support\nthis directly. It is basically:\n\n  1. Let the command run in a non-repo, skipping anything that requires\n     it. This _might_ be a maintenance headache, but as a fallback you\n     could always run from an empty local repository.\n\n  2. Tell it to always generate an empty pack (basically, a \"trust me,\n     the other side will be OK with it\" option).\n\nThe second part looks like something like this:\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 89aca9d829..c54463c181 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -58,6 +58,9 @@ static void feed_object(const struct object_id *oid, FILE *fh, int negative)\n \tputc('\\n', fh);\n }\n \n+/* obviously this should be passed down somehow in a real patch */\n+#define SPECIAL_EMPTY_PACK_OPTION 1\n+\n /*\n  * Make a pack stream and spit it out into file descriptor fd\n  */\n@@ -103,17 +106,19 @@ static int pack_objects(int fd, struct ref *refs, struct oid_array *advertised,\n \t * parameters by writing to the pipe.\n \t */\n \tpo_in = xfdopen(po.in, \"w\");\n-\tfor (i = 0; i < advertised->nr; i++)\n-\t\tfeed_object(&advertised->oid[i], po_in, 1);\n-\tfor (i = 0; i < negotiated->nr; i++)\n-\t\tfeed_object(&negotiated->oid[i], po_in, 1);\n-\n-\twhile (refs) {\n-\t\tif (!is_null_oid(&refs->old_oid))\n-\t\t\tfeed_object(&refs->old_oid, po_in, 1);\n-\t\tif (!is_null_oid(&refs->new_oid))\n-\t\t\tfeed_object(&refs->new_oid, po_in, 0);\n-\t\trefs = refs->next;\n+\tif (!SPECIAL_EMPTY_PACK_OPTION) {\n+\t\tfor (i = 0; i < advertised->nr; i++)\n+\t\t\tfeed_object(&advertised->oid[i], po_in, 1);\n+\t\tfor (i = 0; i < negotiated->nr; i++)\n+\t\t\tfeed_object(&negotiated->oid[i], po_in, 1);\n+\n+\t\twhile (refs) {\n+\t\t\tif (!is_null_oid(&refs->old_oid))\n+\t\t\t\tfeed_object(&refs->old_oid, po_in, 1);\n+\t\t\tif (!is_null_oid(&refs->new_oid))\n+\t\t\t\tfeed_object(&refs->new_oid, po_in, 0);\n+\t\t\trefs = refs->next;\n+\t\t}\n \t}\n \n \tfflush(po_in);\n\nCome to think of it, you could probably fake it by wrapping\ngit-pack-objects with a script that throws away its input. Maybe hard to\ndo because its a builtin (and we run it as \"git pack-objects\", which\nexecutes it directly in-process).\n\n-Peff\n"},{"id":"482631","messageId":"xmqqil7ntcjj.fsf@gitster.g","threadId":"60284","inReplyTo":"20231003200018.GB1562@coredump.intra.peff.net","subject":"Re: Possible to update-ref remote repository?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-04T01:13:36Z","receivedAt":"2023-10-04T01:13:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> All that said, I do think it might be reasonable for git-push to support\n> this directly.\n\nYup.  It certainly is simpler if you can leverage existing helpers.\n\nIt will become even simpler in a reasonably modularlized world that\nhopefully may materialize before we all retire ;-).  I am hoping\nthat some of the folks who are interested in and talking about\nlibification can be fooled into doing the necessary work to\nintroduce proper abstraction, in addition to whatever they are\ndoing.\n\nWouldn't it be great if you can have an in-core repository object,\nthat knows what its object store is, has an index_state object that\nis tied to that object store, has a reference database whose values\npoint into the object store, and if you can choose and mix these\nrepository components' implementations?  If done right, parts of the\nabove set of components can be replaced with mock implementations\nthat are in-core only.\n\nTo run \"git push --repoint-only there 01beef23:master\", you should\nbe able to start your process totally outside an repository, yet\ncreate an in-core-only repository instance with an in-core-only\nobject store instance, and because you took the object name to push\non the command line, your in-core object store can \"lie\" to a call\n\"create an in-core object for this SHA-1\" by returning a fake\nin-core commit object, and your in-core-only ref database has that\ncommit pointed at by some ref.  Then because higher level \"client\"\ncode to walk revisions, enumerate refs, etc., would all implement\nwhat they need to do by calling vtable of these in-core objects, you\ncan do the \"repoint-only\" push without being in any repository, as\nsuch an implementation would not touch any filesystem (you can then\nplug in different implementation of object store etc., and even make\nthem perform reasonably well if you manage to do the abstraction\nright).\n\nBut that would probably be at least 6 months away, even if we had a\nhandful of competent developers totally dedicated to the effort\nwithout any distraction, which I do not know how likely it is to\nhappen.\n\n\n"}]}