{"thread":{"id":"63260","subject":"Way to \"impersonate\" remote or sync remotes without fetching everything?","startedAt":"2025-04-05T14:01:43Z","lastAt":"2025-04-14T19:40:55Z","messageCount":6,"participants":["Klaus Frank","D. Ben Knoble"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"515698","messageId":"2afd91ca-13d8-4b40-aa8b-c1917f0882df@frank.fyi","threadId":"63260","inReplyTo":null,"subject":"Way to \"impersonate\" remote or sync remotes without fetching everything?","fromName":"Klaus Frank","fromEmail":"vger.kernel.org@frank.fyi","sentAt":"2025-04-05T14:01:41Z","receivedAt":"2025-04-05T14:01:43Z","isPatch":false,"sender":{"key":"vger.kernel.org@frank.fyi","avatar":null},"body":"Hi,\n\nwhat is the best way to sync multiple remotes with each other without \nhaving to pull everything into a local checkout first?\n\nNow as many projects have moved off of github onto their own gitlab \ninstances it is kinda hard to keep track of all of the contributions as \nyou basically have to create a fork on each of these instances to open \nPRs and because I'd like to have my contributions also on my gitlab \nserver too I've now the issue of having to keep multiple remote \nrepositories in sync. And then sometimes there are additional remotes \nfrom others that you'd like to interact with as well, so now these need \nto be mirrored as well. Then there is also the issue of archiving, e.g. \nfreeing oneself from a dependency getting deleted upstream that I see at \nmore and more companies I work with. They want to have an internal fork \nof everything (someties also wanting to review all of the commits before \npulling them into their internal git server; but for most it was enough \nto just fail on rewritten/force-pushed history so far). And just for my \npersonal projects I'd like to be able to have them on github, gitlab, \nand my own gitlab server, while being able to make edits (and accept \nPRs) at any of these and keep the repos in sync (or at least \nautomatically sync them as long as nothing conflicts).\n\nIs there a way to use some of the more advanced features of git to \naccomplis this? Like e.g. the alternative object database mechanism in a \nlocal temp project and pushing to the other remotes or something?\n\nIf any of you ran into similar issues in the past, how did you solve them?\n\n\nWhat I tried so far in order:\n\nMy initial thought was to look for a way to do:\n\n1. \"impersonate\" a remote (something like \"git clone --bare\" but\n    without actually cloning any of the objects and querying the remote\n    when needed)\n2. Any git command within that repo will now be treated as if it was\n    ran within/by the impersonated remote. E.g. It'll consider all of\n    the objects the remote has as its own.\n3. Add all of the other remotes as usual (the impersonated one wouldn't\n    be reported as remote but as the local one the others were added to)\n4. Doing a \"git push --all\" to any of these remotes will cause git to\n    download thouse objects from the impersonated one that it needs to\n    push towards that remote (and hasn't cached locally already).\n\nAs I wasn't able to find such a feature I tried to workaround this. At \nfirst I tried to use a shallow clone but that didn't really work as I \ndidn't know where I'd have to start in order to be able to give all of \nthe remotes a common commit to be detected as belonging together (esp. \nif one of the remotes didn't exist and had to be created by pushing to \nit...).\n\nThen I managed to find these two suboptimal ones so far:\n\n1. pull-push-style:\n     1. \"Pull --mirror --bare\" the first one\n     2. \"Push --all\" to all of the other(s)\n2. Without always pulling the same server for the entire repo,\n    regardless of it having changes or not:\n     1. Create new and empty repo locally\n     2. Add all of the remotes\n     3. Fetch from the nearest server,\n     4. \"lfs fetch --all\" from the nearest server\n     5. fetch from all others\n     6. \"lfs fetch --all\" from all others\n     7. Hackishly update all of the local refs to basically be the ones\n        of the remote that should be used as source (aka \"rm -rf\n        .git/refs/heads/*; cp -r .git/refs/remotes/origin/*\n        .git/refs/heads/\")\n     8. \"Push --all\" to all of the remotes\n\nThe 1st one is the most simple, it almost always works (only fails in \nsome very rare cases, like when the remote contains \"zero-padded file \nmodes\" and such) but it'll cause unecessary load on the origin server as \nit has to always first make an entire clone of the repo. Even if the \nremotes are already in sync. Having local state would partially would \nsolve that but then I'd need a lot of disk space and I wouldn't be able \nto just run this within a CI job. => Therefore undesirable.\n\nThe 2nd one is more sophisticated, it may look kinda righit at first, \nbut it keeps breaking all the time for countless different reasons. Even \nthough it allows to only pull new/changed commits from the origin server \n(aka it doesn't overload a single server unecessarily), it'll still \nclone all of the repos from the local git server onto the CI worker, \nesp. considering larger repos using git-lfs (or trying to do it in \nparallel with multiple/all of my repos) it causes the CI worker to run \nout of disk space as well as a lot of unecessary network and disk IO => \nTherefore also undesirable.\n\nAlso not to mention that none of my current approaches can do a real \nsync, they are all relying upon having one of the remotes designated as \nthe authoritative source and if any of the others changed instead it'll \njust fail.\n\nSincerely,\nKlaus Frank\n\n"},{"id":"516006","messageId":"CALnO6CCUHE6-RcZkTBYT-CeXHnYPBPe1eGSogWd7=5tK5NyuxA@mail.gmail.com","threadId":"63260","inReplyTo":"2afd91ca-13d8-4b40-aa8b-c1917f0882df@frank.fyi","subject":"Re: Way to \"impersonate\" remote or sync remotes without fetching everything?","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-04-11T18:43:24Z","receivedAt":"2025-04-11T18:43:37Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Sat, Apr 5, 2025 at 10:01 AM Klaus Frank <vger.kernel.org@frank.fyi> wrote:\n> Also not to mention that none of my current approaches can do a real\n> sync, they are all relying upon having one of the remotes designated as\n> the authoritative source and if any of the others changed instead it'll\n> just fail.\n\nMaybe I haven't totally understood your use-case, but what if the\nauthoritative source is your local repository, and then you push to\nall your remote mirrors to publish your trees? That is, I don't think\nit's a good idea to have a remote worker automatically pushing changes\nacross all the mirrors; rather, you get to be in control of when you\npush to those mirrors.\n\n(I thought there was a push-equivalent of remotes.<group>, which I was\ngoing to suggest as being helpful for this kind of mode where some\nremotes are your mirrors and others are collaborators, but I can't\nfind it.) Configuring a new remote with pushurls that point to the\nother remotes [1] seems to be the way to make pushing to multiple\nremotes easy.\n\nOf course, this doesn't help CI, but then it can just pick any mirror\nit wants to fetch a commit to build?\n\n[1]: https://stackoverflow.com/a/14290145/4400820\n\n-- \nD. Ben Knoble\n"},{"id":"516009","messageId":"669a6a0f-bcfb-4596-a0c8-73c722a6ec10@frank.fyi","threadId":"63260","inReplyTo":"CALnO6CCUHE6-RcZkTBYT-CeXHnYPBPe1eGSogWd7=5tK5NyuxA@mail.gmail.com","subject":"Re: Way to \"impersonate\" remote or sync remotes without fetching everything?","fromName":"Klaus Frank","fromEmail":"vger.kernel.org@frank.fyi","sentAt":"2025-04-11T19:02:47Z","receivedAt":"2025-04-11T19:02:54Z","isPatch":false,"sender":{"key":"vger.kernel.org@frank.fyi","avatar":null},"body":"On 2025-04-11 20:43:24, D. Ben Knoble wrote:\n> Maybe I haven't totally understood your use-case, but what if the\n> authoritative source is your local repository, and then you push to\n\nThere is no local repository, that's kinda the source of all of this.\nThe sync script runs in a CI/CD. I'm kinda abusing CI/CD here to run\na kind of cron job, in a separate repository that does the sync, maybe it\nis easier to just call it scheduled pipeline/action or just stateless \ncron job?\n\nLets make a more quick example:\n\ngdm is being developed here: https://gitlab.gnome.org/GNOME/gdm\nso in order to make a PR I'll have to create a fork in that GitLab\ninstance so now we're at 2 repositories. Then I want to have my own\nindependent archive mirror in my own gitlab instance. Then I also\nwant to mirror it onto gitlab.com and github.com just for the sake of\nthis example. Now we're at 5 remotes.\n\nNow I'd like to have a script in CI/CD (that runs server side) to sync\nall of them. In example the gnome.org one could probably mostly be the\nautoritative source (except for the branches that contain my changes).\n\n"},{"id":"516061","messageId":"CALnO6CALCFLWSs9HBWX13V_BUZkoeZuVjHVnF4_e0OMy3LfEjw@mail.gmail.com","threadId":"63260","inReplyTo":"669a6a0f-bcfb-4596-a0c8-73c722a6ec10@frank.fyi","subject":"Re: Way to \"impersonate\" remote or sync remotes without fetching everything?","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-04-13T21:52:14Z","receivedAt":"2025-04-13T21:52:26Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Fri, Apr 11, 2025 at 3:02 PM Klaus Frank <vger.kernel.org@frank.fyi> wrote:\n>\n> On 2025-04-11 20:43:24, D. Ben Knoble wrote:\n> > Maybe I haven't totally understood your use-case, but what if the\n> > authoritative source is your local repository, and then you push to\n>\n> There is no local repository, that's kinda the source of all of this.\n> The sync script runs in a CI/CD. I'm kinda abusing CI/CD here to run\n> a kind of cron job, in a separate repository that does the sync, maybe it\n> is easier to just call it scheduled pipeline/action or just stateless\n> cron job?\n>\n> Lets make a more quick example:\n>\n> gdm is being developed here: https://gitlab.gnome.org/GNOME/gdm\n> so in order to make a PR I'll have to create a fork in that GitLab\n> instance so now we're at 2 repositories. Then I want to have my own\n> independent archive mirror in my own gitlab instance. Then I also\n> want to mirror it onto gitlab.com and github.com just for the sake of\n> this example. Now we're at 5 remotes.\n>\n> Now I'd like to have a script in CI/CD (that runs server side) to sync\n> all of them. In example the gnome.org one could probably mostly be the\n> autoritative source (except for the branches that contain my changes).\n\nThat all makes sense, except: why need the sync (cron) job? Treat a\nlocal copy as authoritative for you and push to all your remotes. This\nputs you in control at the cost of not happening automatically. (You\ncould conceivably have a local cron job that did this.)\n\n-- \nD. Ben Knoble\n"},{"id":"516066","messageId":"351a6036-3bac-43f4-88b0-cd52240312cd@frank.fyi","threadId":"63260","inReplyTo":"CALnO6CALCFLWSs9HBWX13V_BUZkoeZuVjHVnF4_e0OMy3LfEjw@mail.gmail.com","subject":"Re: Way to \"impersonate\" remote or sync remotes without fetching everything?","fromName":"Klaus Frank","fromEmail":"vger.kernel.org@frank.fyi","sentAt":"2025-04-14T00:34:41Z","receivedAt":"2025-04-14T00:34:43Z","isPatch":false,"sender":{"key":"vger.kernel.org@frank.fyi","avatar":null},"body":"On 2025-04-13 23:52:14, D. Ben Knoble wrote:\n> That all makes sense, except: why need the sync (cron) job? Treat a\n> local copy as authoritative for you and push to all your remotes. This\n> puts you in control at the cost of not happening automatically. (You\n> could conceivably have a local cron job that did this.)\n\nCause I don't want to have it locally on e.g. a notebook that can break \nmainly :D\n(I know myself I won't be making enough backups if that is their primary \nlocation)\n\nBut I have been thinking about something similar earlier today. Maybe I \nshould just\ntrash my software forge (gitlab) and just use git from the cli via ssh.\nThen writing a cron job to do the syncing would also be easier as I'd have\na local copy to work with and maybe add some Stagit sparkles to replace the\nweb-ui (aka some static page generator for git repos)\nhttps://codemadness.org/stagit.html\n\n(Still not quite satisfies with Stagit nor cgit, maybe I'll find \nsomething that needs,\nless files generated on the server side and some \"git in javascript\" to \ndisplay\nthe more exotic views instead of doing it server side like cgit does)\n\n--\nKlaus Frank\n\n"},{"id":"516124","messageId":"CALnO6CCsyF0w6jP+Xk+qQZCW-r_004b5_80XzsPLH2n7fnogsg@mail.gmail.com","threadId":"63260","inReplyTo":"351a6036-3bac-43f4-88b0-cd52240312cd@frank.fyi","subject":"Re: Way to \"impersonate\" remote or sync remotes without fetching everything?","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-04-14T19:40:41Z","receivedAt":"2025-04-14T19:40:55Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Mon, Apr 14, 2025 at 11:44 AM Klaus Frank <vger.kernel.org@frank.fyi> wrote:\n>\n> On 2025-04-13 23:52:14, D. Ben Knoble wrote:\n> > That all makes sense, except: why need the sync (cron) job? Treat a\n> > local copy as authoritative for you and push to all your remotes. This\n> > puts you in control at the cost of not happening automatically. (You\n> > could conceivably have a local cron job that did this.)\n>\n> Cause I don't want to have it locally on e.g. a notebook that can break\n> mainly :D\n> (I know myself I won't be making enough backups if that is their primary\n> location)\n\nIf you push to other remotes (or cron, as below), then you'll have\n\"backups for free\"?\n\n>\n> But I have been thinking about something similar earlier today. Maybe I\n> should just\n> trash my software forge (gitlab) and just use git from the cli via ssh.\n> Then writing a cron job to do the syncing would also be easier as I'd have\n> a local copy to work with\n[…]\n\n-- \nD. Ben Knoble\n"}]}