{"thread":{"id":"10052","subject":"backup or mirror a repository","startedAt":"2007-09-27T19:35:42Z","lastAt":"2007-10-30T00:33:05Z","messageCount":9,"participants":["Dan Farina","Johannes Schindelin","Junio C Hamano","Dmitry Potapov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"54190","messageId":"1190921742.2263.17.camel@Tenacity","threadId":"10052","inReplyTo":null,"subject":"backup or mirror a repository","fromName":"Dan Farina","fromEmail":"drfarina@gmail.com","sentAt":"2007-09-27T19:35:42Z","receivedAt":"2007-09-27T19:35:42Z","isPatch":false,"sender":{"key":"drfarina@gmail.com","avatar":"https://gravatar.com/avatar/49f33838093961a07741035efe38c3dec91ded76cad83492ed47df405a2348ce?d=mp&s=160"},"body":"Hello list,\n\nI have been poking around on IRC and gmane, but haven't yet seen a\nsolution to my problem:\n\nWhat I really want is something semantically identical to a recursive\ncopy of my entire git repository on one machine sent to another machine.\nRight now I can't simply use \"rsync -ar --delete ...\" because the pack\nchanges names, thus fooling rsync and sending a bunch of bits over the\nnetwork.\n\nI saw git-mirror on gmane and asked around in IRC, but the former seems\nto have died sometime last November and the latter told me I'd probably\nhave to write some glue to process git show-ref and git ls-remote. This\nseems viable, but suboptimal. Another user chimed in that they also had\nthe same use case as myself.\n\n\nfdr\n\n(Justifications: I am super paranoid, so I want to push my local\nbranches to another disk to be safe, but then they proliferate over time\nsince deleted local branches are not also deleted on the target\nrepository. I also want to work on multiple machines seamlessly, and as\nsuch doing a bunch of \"git checkout -r ...\" when I need to change\nbranches is also suboptimal. As I said, I can get what I want by simply\nusing rsync, so this is entirely about efficiency by exploiting git's\nstructure)\n"},{"id":"54210","messageId":"Pine.LNX.4.64.0709272255360.28395@racer.site","threadId":"10052","inReplyTo":"1190921742.2263.17.camel@Tenacity","subject":"Re: backup or mirror a repository","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-09-27T21:56:55Z","receivedAt":"2007-09-27T21:56:55Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 27 Sep 2007, Dan Farina wrote:\n\n> I have been poking around on IRC and gmane, but haven't yet seen a \n> solution to my problem:\n\nI guess you're looking for \"git remote add --mirror origin <url>\".  AFAIK \nthis is not in any released version yet, though.\n\nCiao,\nDscho\n"},{"id":"54219","messageId":"1190940704.2263.32.camel@Tenacity","threadId":"10052","inReplyTo":"Pine.LNX.4.64.0709272255360.28395@racer.site","subject":"Re: backup or mirror a repository","fromName":"Dan Farina","fromEmail":"drfarina@gmail.com","sentAt":"2007-09-28T00:51:44Z","receivedAt":"2007-09-28T00:51:44Z","isPatch":false,"sender":{"key":"drfarina@gmail.com","avatar":"https://gravatar.com/avatar/49f33838093961a07741035efe38c3dec91ded76cad83492ed47df405a2348ce?d=mp&s=160"},"body":"On Thu, 2007-09-27 at 22:56 +0100, Johannes Schindelin wrote:\n> I guess you're looking for \"git remote add --mirror origin <url>\".  AFAIK \n> this is not in any released version yet, though.\n\nI just tried a bleeding edge git with --mirror.\n\nIt's close, except (unless there is some other provision to take care of\nthis) remote branch deletes. (Which, after more gmane trolling, seemed\nto be \"git remote rm\", which doesn't seem present just yet).\n\nOf course, it is completely possible I'm using it incorrectly. In brief,\nI did \"git init\" \"git remote add --mirror backup <url>\" \"git\npull\" (perhaps I should have used \"git remote update\"?) And the behavior\nof this is more or less what I wanted, as far as sending branches.\nConvenient removal of branches and propagating that removal still stumps\nme. Is there a preferred way to do this yet?\n\nThanks,\nfdr\n"},{"id":"54220","messageId":"Pine.LNX.4.64.0709280253150.28395@racer.site","threadId":"10052","inReplyTo":"1190940704.2263.32.camel@Tenacity","subject":"Re: backup or mirror a repository","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-09-28T01:56:49Z","receivedAt":"2007-09-28T01:56:49Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 27 Sep 2007, Dan Farina wrote:\n\n> On Thu, 2007-09-27 at 22:56 +0100, Johannes Schindelin wrote:\n> > I guess you're looking for \"git remote add --mirror origin <url>\".  \n> > AFAIK this is not in any released version yet, though.\n> \n> I just tried a bleeding edge git with --mirror.\n> \n> It's close, except (unless there is some other provision to take care of \n> this) remote branch deletes. (Which, after more gmane trolling, seemed \n> to be \"git remote rm\", which doesn't seem present just yet).\n\ngit remote rm is about removing the remote nick, not a remote branch.\n\n> Of course, it is completely possible I'm using it incorrectly. In brief, \n> I did \"git init\" \"git remote add --mirror backup <url>\" \"git pull\"\n\n\"git pull\" without further parameters accesses the remote \"origin\", not \n\"backup\", and it merges _one_ branch into the current branch.\n\nWhat you should have done is \"git fetch backup\".\n\nOr use \"origin\" instead of \"backup\" right from the start, and then use \n\"git fetch\".\n\nBut no pull.\n\n> (perhaps I should have used \"git remote update\"?) And the behavior of \n> this is more or less what I wanted, as far as sending branches. \n> Convenient removal of branches and propagating that removal still stumps \n> me. Is there a preferred way to do this yet?\n\nYou want to remove branches from the local repo which are no longer there \non the remote side?\n\nThen \"git remote prune\" is for you.  For details, please see the man page.\n\nCiao,\nDscho\n"},{"id":"54222","messageId":"1190947063.2263.46.camel@Tenacity","threadId":"10052","inReplyTo":"Pine.LNX.4.64.0709280253150.28395@racer.site","subject":"Re: backup or mirror a repository","fromName":"Dan Farina","fromEmail":"drfarina@gmail.com","sentAt":"2007-09-28T02:37:43Z","receivedAt":"2007-09-28T02:37:43Z","isPatch":false,"sender":{"key":"drfarina@gmail.com","avatar":"https://gravatar.com/avatar/49f33838093961a07741035efe38c3dec91ded76cad83492ed47df405a2348ce?d=mp&s=160"},"body":"On Fri, 2007-09-28 at 02:56 +0100, Johannes Schindelin wrote:\n> Hi,\n> \n> On Thu, 27 Sep 2007, Dan Farina wrote:\n\n> git remote rm is about removing the remote nick, not a remote branch.\n\nAh, I thought for some reason there'd be a \"remove\" (since the other\noperations have long names) and, much like \"ls-remote\" there'd be some\n\"rm-remote.\" An insane presupposition of mine from having too little\ninformation.\n\n\n> What you should have done is \"git fetch backup\".\n> \n> Or use \"origin\" instead of \"backup\" right from the start, and then use \n> \"git fetch\".\n> \n> But no pull.\n\nI did use \"pull backup\" (my error for being sloppy and omitting it from\nmy mail) and it does work provided I do the dirty thing of doing a\nsubsequent reset --hard (I thought perhaps things were not yet\nfully-there). Fetch does it perfectly, as you say, thanks!\n\n> You want to remove branches from the local repo which are no longer there \n> on the remote side?\n> \n> Then \"git remote prune\" is for you.  For details, please see the man page.\n> \n\nI did look at prune and update, but my problem is the opposite: I want\nsomething that will remove branches from the remote repo when they no\nlonger exist locally. As-is over time I will proliferate little local\nbranches unless I occasionally sit down and delete branches by operating\ndirectly on the bare backup repository. (and then use prune on the\nremote nodes)\n\nThanks,\nfdr\n"},{"id":"54240","messageId":"7vsl4zqp8l.fsf@gitster.siamese.dyndns.org","threadId":"10052","inReplyTo":"1190947063.2263.46.camel@Tenacity","subject":"Re: backup or mirror a repository","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-09-28T06:27:06Z","receivedAt":"2007-09-28T06:27:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dan Farina <drfarina@gmail.com> writes:\n\n> I did look at prune and update, but my problem is the opposite: I want\n> something that will remove branches from the remote repo when they no\n> longer exist locally. As-is over time I will proliferate little local\n> branches unless I occasionally sit down and delete branches by operating\n> directly on the bare backup repository. (and then use prune on the\n> remote nodes)\n\nThe \"git remote add --mirror\" setup is about setting up the\nlocal repository _AS_ the backup of the remote.  In other words,\nthe contents come from the remote by fetching from it and safely\nkept away from disaster on the local side.  And for that,\n\"remote prune\" is a perfect thing to do.\n\nI think what you are asking for is an opposite, a backup remote\nsite you would push into.  That is not what \"remote add --mirror\"\nis about.\n\nYou can almost do it with\n\n\tgit push --all $remote\n\nexcept there is no way to automagically remove the branch you\nremoved from the local repository.  For that, we would need a\nnew --mirror option to \"git-push\".\n\nI think it is trivial to do for native transports, as we first\nget the list of all refs from the remote side before starting\nthe transfer.  You need to change the last parameter called\n'all' to remote.c::match_refs() into an enum ('push_all' being\none of choices), introduce another enum 'push_mirror', and teach\nit to \"match\" the remote (i.e. dst) ref that does not have\ncorresponding entry on our side (i.e. src) with an empty object\nname to mark it removed.  Then the part marked as \"Finally, tell\nthe other end!\"  in send-pack.c::send_pack() will take care of\nthe actual removal.\n"},{"id":"54245","messageId":"1190964003.31911.13.camel@Tenacity","threadId":"10052","inReplyTo":"7vsl4zqp8l.fsf@gitster.siamese.dyndns.org","subject":"Re: backup or mirror a repository","fromName":"Dan Farina","fromEmail":"drfarina@gmail.com","sentAt":"2007-09-28T07:20:03Z","receivedAt":"2007-09-28T07:20:03Z","isPatch":false,"sender":{"key":"drfarina@gmail.com","avatar":"https://gravatar.com/avatar/49f33838093961a07741035efe38c3dec91ded76cad83492ed47df405a2348ce?d=mp&s=160"},"body":"On Thu, 2007-09-27 at 23:27 -0700, Junio C Hamano wrote:\n> You can almost do it with\n> \n> \tgit push --all $remote\n> \n> except there is no way to automagically remove the branch you\n> removed from the local repository.  For that, we would need a\n> new --mirror option to \"git-push\".\n\nYup! I actually was looking for such a thing in the git-push man page,\nbut was unsuccessful. I was living with git-push --all for a little\nwhile before I thought I'd ask. Unfortunately, the preferred setup\n(having the backup machine actively perform fetch and prune) is not very\nnice for me due to firewalls.\n\n> \n> I think it is trivial to do for native transports, as we first\n> get the list of all refs from the remote side before starting\n> the transfer.  You need to change the last parameter called\n> 'all' to remote.c::match_refs() into an enum ('push_all' being\n> one of choices), introduce another enum 'push_mirror', and teach\n> it to \"match\" the remote (i.e. dst) ref that does not have\n> corresponding entry on our side (i.e. src) with an empty object\n> name to mark it removed.  Then the part marked as \"Finally, tell\n> the other end!\"  in send-pack.c::send_pack() will take care of\n> the actual removal.\n\nI should have a look, but if someone else wants to work on this they\nshouldn't block on me: work already made me get hit by a bus on one\nproject (luckily I wasn't important), and the storm is not over.\n\nThanks,\nfdr\n"},{"id":"56349","messageId":"20071018053226.GA20588@dpotapov.dyndns.org","threadId":"10052","inReplyTo":"7vsl4zqp8l.fsf@gitster.siamese.dyndns.org","subject":"Re: backup or mirror a repository","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2007-10-18T05:32:26Z","receivedAt":"2007-10-18T05:32:26Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Thu, Sep 27, 2007 at 11:27:06PM -0700, Junio C Hamano wrote:\n> The \"git remote add --mirror\" setup is about setting up the\n> local repository _AS_ the backup of the remote.  In other words,\n> the contents come from the remote by fetching from it and safely\n> kept away from disaster on the local side.  And for that,\n> \"remote prune\" is a perfect thing to do.\n\nI have tried to do that but I am getting a warning:\n$ git remote prune origin\nWarning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*\nand no branch is removed.\n\nI suspect that the change that introduced --mirror option for the 'add'\ncommand did not adjust the prune procedure to handle the new situation\nproperly. Or is just me doing something wrong?\n\nAnyway, because the official released git still does not have --mirror\nand it could be difficult to convince the admin to use the bleading edge\nversion of Git for secure backup, I wrote a simple script instead, which\nprovides the same functionality but using the released version of Git.\nMaybe, someone else finds it useful.\n\n$ cat git-mirror\n#!/bin/sh\n\nusage() {\n\techo >&2 \"Usage: $0 URL\"\n\texit 1\n}\n\nset -e\n\ntest $# -eq 1 || usage\nURL=\"$1\"\n\n# Initialize Git bare directory\ngit --bare init\n\n# Initialize the mirror\ngit remote add origin \"$URL\"\ngit fetch\nrmdir refs/heads\nln -s remotes/origin refs/heads\n\n# Adding creating a script that will be run by cron\ncat <<EOF > git-mirror-sync\ncd \"$PWD\"\ngit fetch\ngit remote prune origin\nEOF\nchmod +x git-mirror-sync\n\n# Adding git-mirror-sync to cron\necho \"Please, add $PWD/git-mirror-sync to cron\"\n\n#########################################################\n\nDmitry\n"},{"id":"57500","messageId":"Pine.LNX.4.64.0710300030530.4362@racer.site","threadId":"10052","inReplyTo":"20071018053226.GA20588@dpotapov.dyndns.org","subject":"Re: backup or mirror a repository","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-10-30T00:33:05Z","receivedAt":"2007-10-30T00:33:05Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 18 Oct 2007, Dmitry Potapov wrote:\n\n> On Thu, Sep 27, 2007 at 11:27:06PM -0700, Junio C Hamano wrote:\n> > The \"git remote add --mirror\" setup is about setting up the\n> > local repository _AS_ the backup of the remote.  In other words,\n> > the contents come from the remote by fetching from it and safely\n> > kept away from disaster on the local side.  And for that,\n> > \"remote prune\" is a perfect thing to do.\n> \n> I have tried to do that but I am getting a warning:\n> $ git remote prune origin\n> Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*\n> and no branch is removed.\n> \n> I suspect that the change that introduced --mirror option for the 'add' \n> command did not adjust the prune procedure to handle the new situation \n> properly. Or is just me doing something wrong?\n\nNo, you're right.  I did not anticipate git-remote to be written the way \nit is.  After fiddling with it for several hours, I am giving up for now.  \nIMHO the script is too married to the idea that the remote branches live \nin refs/remotes/<remote>/*.  Probably it would be very easy by now to \nimplement it as a builtin, using remote.[ch].\n\nCiao,\nDscho\n"}]}