{"thread":{"id":"46397","subject":"Keeping a non-1:1 mirror in sync and keeping private branches","startedAt":"2017-07-16T00:59:32Z","lastAt":"2017-07-16T14:22:04Z","messageCount":4,"participants":["Jan Danielsson","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"324595","messageId":"699d0274-285f-3d30-654d-d9ca59fe4dce@gmail.com","threadId":"46397","inReplyTo":null,"subject":"Keeping a non-1:1 mirror in sync and keeping private branches","fromName":"Jan Danielsson","fromEmail":"jan.m.danielsson@gmail.com","sentAt":"2017-07-16T00:59:23Z","receivedAt":"2017-07-16T00:59:32Z","isPatch":false,"sender":{"key":"jan.m.danielsson@gmail.com","avatar":null},"body":"Hello,\n\n   Let's say there'a s hosting service hosting a repository which I want\nto self-host.  I don't just want to do a mirror, but I want to keep it\nin sync (using a cronjob).  In addition, I want to have private branches\non the self-hosted repository.  (In this particular case, the \"hosting\nservice\" is github, and \"self-hosting\" is bitbucket -- but I'm looking\nfor a solution which is agnostic with regards to hosting service(s)).\n\n   Searching, reading and asking around led me to the following (these\nare scripts which are run on a separate system which acts as a git bridge):\n\n   init:\n\n   git clone --mirror $UPSTREAMURL $DSTDIR\n   cd $DSTDIR\n   git remote rename origin upstream\n   git config remotes.default 'upstream'\n   git remote add --mirror=push origin $DOWNSTREAMURL\n\n   And then to keep self-hosted repository in sync with upstream:\n\n   cd $DSTDIR\n   git remote update --prune\n   git push\n\n   This seems to accomplish everything I want except that the the \"git\npush\" deletes any branches I have created on my self-hosted repository.\n\n   Am I doing it completely wrong?  If not, how do I make my branches\nsurvive the push?\n\n-- \nKind regards,\nJan Danielsson\n\n"},{"id":"324603","messageId":"20170716101045.aslfct7g5vqfqnhi@sigill.intra.peff.net","threadId":"46397","inReplyTo":"699d0274-285f-3d30-654d-d9ca59fe4dce@gmail.com","subject":"Re: Keeping a non-1:1 mirror in sync and keeping private branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-16T10:10:45Z","receivedAt":"2017-07-16T10:10:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jul 16, 2017 at 02:59:23AM +0200, Jan Danielsson wrote:\n\n>    This seems to accomplish everything I want except that the the \"git\n> push\" deletes any branches I have created on my self-hosted\n> repository.\n\nA mirrored push is basically:\n\n  - push all refs, i.e., a \"+refs/*:refs/*\" refspec\n\n  - enable --prune, to delete any branches that don't exist on the local\n    side\n\nBut you can do those two things separately if you like.  So your options\nare either:\n\n  1. Drop the pruning (in which case deleted branches from the sync may\n     accumulate, but depending on the patterns that may or may not be a\n     problem).\n\n  2. Use two different namespaces for the synced branches and the\n     private ones (e.g., refs/mirror/* in addition to your branches in\n     refs/heads/*). The obvious downside is that anybody cloning your\n     downstream mirror doesn't pick up refs/mirror unless they configure\n     that refspec explicitly.\n\n-Peff\n"},{"id":"324622","messageId":"eb5e02da-662e-5561-7548-e50efba13dca@gmail.com","threadId":"46397","inReplyTo":"20170716101045.aslfct7g5vqfqnhi@sigill.intra.peff.net","subject":"Re: Keeping a non-1:1 mirror in sync and keeping private branches","fromName":"Jan Danielsson","fromEmail":"jan.m.danielsson@gmail.com","sentAt":"2017-07-16T13:42:09Z","receivedAt":"2017-07-16T13:42:18Z","isPatch":false,"sender":{"key":"jan.m.danielsson@gmail.com","avatar":null},"body":"On 07/16/17 12:10, Jeff King wrote:\n>>    This seems to accomplish everything I want except that the the \"git\n>> push\" deletes any branches I have created on my self-hosted\n>> repository.\n> \n> A mirrored push is basically:\n> \n>   - push all refs, i.e., a \"+refs/*:refs/*\" refspec\n> \n>   - enable --prune, to delete any branches that don't exist on the local\n>     side\n> \n> But you can do those two things separately if you like.  So your options\n> are either:\n> \n>   1. Drop the pruning (in which case deleted branches from the sync may\n>      accumulate, but depending on the patterns that may or may not be a\n>      problem).\n\n   I don't think that's a problem.  Or rather, I'd be willing to try it\nand see if it becomes a problem.\n\n   That said; does \"drop the pruning\" mean simply removing \"--prune\"\nfrom the remove update?  I did that, but it still deletes my test-branch\non push.  Is there an implicit pruning happening due to some\nconfiguration option or the specific commands I'm using?\n\n>   2. Use two different namespaces for the synced branches and the\n>      private ones (e.g., refs/mirror/* in addition to your branches in\n>      refs/heads/*). The obvious downside is that anybody cloning your\n>      downstream mirror doesn't pick up refs/mirror unless they configure\n>      that refspec explicitly.\n   This sounds very useful.  How would one go about setting up this\nconfiguration?\n\n-- \nKind regards,\nJan Danielsson\n\n"},{"id":"324625","messageId":"20170716141637.bsdkj4i5rncd7h5m@sigill.intra.peff.net","threadId":"46397","inReplyTo":"eb5e02da-662e-5561-7548-e50efba13dca@gmail.com","subject":"Re: Keeping a non-1:1 mirror in sync and keeping private branches","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-16T14:16:37Z","receivedAt":"2017-07-16T14:22:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jul 16, 2017 at 03:42:09PM +0200, Jan Danielsson wrote:\n\n> >   1. Drop the pruning (in which case deleted branches from the sync may\n> >      accumulate, but depending on the patterns that may or may not be a\n> >      problem).\n> \n>    I don't think that's a problem.  Or rather, I'd be willing to try it\n> and see if it becomes a problem.\n> \n>    That said; does \"drop the pruning\" mean simply removing \"--prune\"\n> from the remove update?  I did that, but it still deletes my test-branch\n> on push.  Is there an implicit pruning happening due to some\n> configuration option or the specific commands I'm using?\n\nNo, that prune is fetching from your upstream into the bridge repo. So\nit is dropping refs from the bridge that went away upstream. My\nassumption is that your \"private\" branches are not in the bridge, but\njust in the $DOWNSTREAMURL.\n\nThe issue is in the \"git push\" after that.  Your \"git remote add\n--mirror=push\" set up config like this:\n\n  [remote \"origin\"]\n  url = ...\n  mirror = true\n\nand the \"mirror\" there implies the prune. Instead, you'd want to do:\n\n  git config remote.origin.url $DOWNSTREAMURL\n  git config remote.origin.push \"+refs/*:refs/*\"\n\nNote that the \"+\" means a force-push. If you do have personal branches\nin the downstream, their contents may be overwritten by the sync from\nupstream. You can drop the \"+\", but then if upstream ever rewinds a\nbranch, your sync would fail.\n\n> >   2. Use two different namespaces for the synced branches and the\n> >      private ones (e.g., refs/mirror/* in addition to your branches in\n> >      refs/heads/*). The obvious downside is that anybody cloning your\n> >      downstream mirror doesn't pick up refs/mirror unless they configure\n> >      that refspec explicitly.\n>    This sounds very useful.  How would one go about setting up this\n> configuration?\n\nI'm not 100% clear on where your private branches are. Are they in the\nbridge repo, too, or only in the eventual downstream? I'll assume\nthey're only in the downstream, and that the bridge is purely a tool for\nsyncing.\n\nIn that case, you might do:\n\n  git clone --mirror -o upstream $UPSTREAM bridge.git\n  cd bridge.git\n  git config remote.downstream.url $DOWNSTREAM\n  git config remote.downstream.push \"+refs/*:refs/mirror/*\"\n  git config remote.downstream.prune true\n\nThough I'm not 100% sure that the \"prune\" config is respected for\npushes. You might have to skip that last config and just issue:\n\n  git push --prune downstream\n\nIf you're scripting it, I'd actually consider doing the whole thing\nwithout config at all:\n\n  # do this once\n  git init bridge.git\n  cd bridge.git\n\n  # and put this in your sync script\n  git fetch --prune $UPSTREAM +refs/*:refs/*\n  git push --prune $DOWNSTREAM +refs/*:refs/mirror/*\n\n-Peff\n"}]}