{"thread":{"id":"26696","subject":"rsync busy non-bare git repo 'source' to quiet","startedAt":"2011-03-08T21:25:00Z","lastAt":"2011-03-08T23:00:59Z","messageCount":5,"participants":["Neal Kreitzinger","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"163025","messageId":"il66rd$46u$1@dough.gmane.org","threadId":"26696","inReplyTo":null,"subject":"rsync busy non-bare git repo 'source' to quiet","fromName":"Neal Kreitzinger","fromEmail":"neal@rsss.com","sentAt":"2011-03-08T21:25:00Z","receivedAt":"2011-03-08T21:25:00Z","isPatch":false,"sender":{"key":"neal@rsss.com","avatar":null},"body":"Does anyone have an example of an rsync bash script that will make a good \ncopy of a non-bare git repo (including the working tree) while the \"source\" \ngit repo is busy and the \"destination\" git repo is quiet?\n\nWe do not use symlinks in our working-tree or in our .git directory (unless \ngit is using symlinks on its own behind-the-scenes that I am not aware of).\n\nThis would make it very easy for us to refresh our \"beta\" livebox to emulate \nthe current \"gold\" livebox using a single rsync instead of a combination of \nrsync and git-clone/pull due to the pieces that git does not replicate (ie, \nhooks) and the non-git components of our git based change control menu \nsystem (which is written in bash scripts on linux).\n\nThanks!\n\nv/r,\nNeal \n"},{"id":"163029","messageId":"20110308213959.GB5786@sigill.intra.peff.net","threadId":"26696","inReplyTo":"il66rd$46u$1@dough.gmane.org","subject":"Re: rsync busy non-bare git repo 'source' to quiet","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-08T21:39:59Z","receivedAt":"2011-03-08T21:39:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 08, 2011 at 03:25:00PM -0600, Neal Kreitzinger wrote:\n\n> Does anyone have an example of an rsync bash script that will make a good \n> copy of a non-bare git repo (including the working tree) while the \"source\" \n> git repo is busy and the \"destination\" git repo is quiet?\n\nDon't copy the working tree. It's redundant with the repo data (assuming\nyour working tree is clean), so you are probably faster to sync .git and\nthen \"reset --hard\". And then you don't have to worry about fetching an\ninconsistent working tree state.\n\nFor syncing the repo, I think you would need to do:\n\n  1. Copy the refs to a local temp space.\n\n  2. Copy the object db.\n\n  3. Install your temp refs into place.\n\nThat way, for any updates in progress you will either not copy them\n(because the refs weren't in place in step 1, though you may have some\nof their objects), or if you do copy them, you are guaranteed to have\nall of the necessary objects (because git will not update the ref until\nall objects are in place).\n\nBut I really have to wonder why you don't simply use git to do the\nfetch? It already does the right thing with respect to updates, and it\nwill be way more efficient than rsync in the face of repacking (or if\nyou really do want to use rsync, there is even an rsync transport for\ngit already). It sounds like:\n\n> This would make it very easy for us to refresh our \"beta\" livebox to emulate \n> the current \"gold\" livebox using a single rsync instead of a combination of \n> rsync and git-clone/pull due to the pieces that git does not replicate (ie, \n> hooks) and the non-git components of our git based change control menu \n> system (which is written in bash scripts on linux).\n\nYou just don't want to do a pull in addition to an rsync. But I don't\nthink this solution will be any less complex.\n\n-Peff\n"},{"id":"163034","messageId":"4D76ABB1.9080001@gmail.com","threadId":"26696","inReplyTo":"20110308213959.GB5786@sigill.intra.peff.net","subject":"Re: rsync busy non-bare git repo 'source' to quiet","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-03-08T22:20:33Z","receivedAt":"2011-03-08T22:20:33Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 3/8/2011 3:39 PM, Jeff King wrote:\n> On Tue, Mar 08, 2011 at 03:25:00PM -0600, Neal Kreitzinger wrote:\n>\n>> Does anyone have an example of an rsync bash script that will make\n>> a good copy of a non-bare git repo (including the working tree)\n>> while the \"source\" git repo is busy and the \"destination\" git repo\n>> is quiet?\n>\n> Don't copy the working tree. It's redundant with the repo data\n> (assuming your working tree is clean), so you are probably faster to\n> sync .git and then \"reset --hard\". And then you don't have to worry\n> about fetching an inconsistent working tree state.\n>\n> For syncing the repo, I think you would need to do:\n>\n> 1. Copy the refs to a local temp space.\n>\n> 2. Copy the object db.\n>\n> 3. Install your temp refs into place.\n>\n> That way, for any updates in progress you will either not copy them\n> (because the refs weren't in place in step 1, though you may have\n> some of their objects), or if you do copy them, you are guaranteed to\n> have all of the necessary objects (because git will not update the\n> ref until all objects are in place).\n>\n> But I really have to wonder why you don't simply use git to do the\n> fetch? It already does the right thing with respect to updates, and\n> it will be way more efficient than rsync in the face of repacking (or\n> if you really do want to use rsync, there is even an rsync transport\n> for git already). It sounds like:\n>\n>> This would make it very easy for us to refresh our \"beta\" livebox\n>> to emulate the current \"gold\" livebox using a single rsync instead\n>> of a combination of rsync and git-clone/pull due to the pieces that\n>> git does not replicate (ie, hooks) and the non-git components of\n>> our git based change control menu system (which is written in bash\n>> scripts on linux).\n>\n> You just don't want to do a pull in addition to an rsync. But I\n> don't think this solution will be any less complex.\n>\nOne reason is that we only have one development box.  We would have to\nopen up the canonical repo and development repos to git:// protocol\naccess.  If I use git I have to do the initial clones for creation and \nthen the pulls for refreshes.  rsync will do the creation and refreshes \nvia the same script.  If I use git I would have to rsync the hooks, \nconfig, and anything else git doesn't bring over.  On the goldbox I have \na bare repo that mirrors the canonical repo and then has additional \nbranches and is in turn mirrored by another bare mirror which has all \nthe additional branches.  I would have to recreate the original remote \nbranch setup and then still maintain the remotes to the goldbox.  Then I \nwouldn't really have a simulation of the goldbox anymore because I have \nextra remotes (maybe that wouldn't really hurt anything).  Rsync seems \nlike a simpler solution and more accurate solution for creating a copy \nof an ecosystem of interrelated git repos colocated on the same box.\n\nA previous post in the newsgroup states:\n\"> If you want your rsync backup to be fine, you need to follow some\n> ordering.  You need to copy the refs first (.git/packed-refs and\n> .git/refs/), then the loose objects (.git/objects/??/*), and then all\n> the rest.  If files are copied in a different order while some write\n> operations are performed on the source repository then you may end up\n> with an incoherent repository.\"\n\nWould that work?\n\nv/r,\nneal\n"},{"id":"163039","messageId":"20110308223841.GA6648@sigill.intra.peff.net","threadId":"26696","inReplyTo":"4D76ABB1.9080001@gmail.com","subject":"Re: rsync busy non-bare git repo 'source' to quiet","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-08T22:38:41Z","receivedAt":"2011-03-08T22:38:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 08, 2011 at 04:20:33PM -0600, Neal Kreitzinger wrote:\n\n> Rsync seems like a simpler solution and more accurate solution for\n> creating a copy of an ecosystem of interrelated git repos colocated on\n> the same box.\n\nSure. It is simpler, but not atomic unless you do a multi-stage rsync.\n\n> A previous post in the newsgroup states:\n> > If you want your rsync backup to be fine, you need to follow some\n> >ordering.  You need to copy the refs first (.git/packed-refs and\n> >.git/refs/), then the loose objects (.git/objects/??/*), and then all\n> >the rest.  If files are copied in a different order while some write\n> >operations are performed on the source repository then you may end up\n> >with an incoherent repository.\"\n> \n> Would that work?\n\nIf you do it in that order, the end result will be a consistent repo.\nBut during the copy, the refs at the destination will point to objects\nyou don't have. I don't know if that matters for your case.\n\n-Peff\n"},{"id":"163041","messageId":"4D76B52B.8070307@gmail.com","threadId":"26696","inReplyTo":"20110308223841.GA6648@sigill.intra.peff.net","subject":"Re: rsync busy non-bare git repo 'source' to quiet","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-03-08T23:00:59Z","receivedAt":"2011-03-08T23:00:59Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 3/8/2011 4:38 PM, Jeff King wrote:\n> On Tue, Mar 08, 2011 at 04:20:33PM -0600, Neal Kreitzinger wrote:\n>\n>> Rsync seems like a simpler solution and more accurate solution for\n>> creating a copy of an ecosystem of interrelated git repos colocated on\n>> the same box.\n>\n> Sure. It is simpler, but not atomic unless you do a multi-stage rsync.\n>\n>> A previous post in the newsgroup states:\n>>> If you want your rsync backup to be fine, you need to follow some\n>>> ordering.  You need to copy the refs first (.git/packed-refs and\n>>> .git/refs/), then the loose objects (.git/objects/??/*), and then all\n>>> the rest.  If files are copied in a different order while some write\n>>> operations are performed on the source repository then you may end up\n>>> with an incoherent repository.\"\n>>\n>> Would that work?\n>\n> If you do it in that order, the end result will be a consistent repo.\n> But during the copy, the refs at the destination will point to objects\n> you don't have. I don't know if that matters for your case.\n>\nWe won't be trying to used the \"destination\" repos during the rsync. \nThe workflow will be:\n(1) I need to test a change to the goldbox change control menu system.\n(2) I determine that my testbox \"copy\" of the change control menu system \nhas repos that are too out-of-date for a good test, so I run rsync to \nmake a fresh copy of the goldbox.\n(3) After the rsync is finished, I pull over just the branch that \ncontains my untested change menu scripts.  (this is the only git pull i \ndo from goldbox to testbox.)\n(4) I test the changes and see they are good.\n(5) I merge the changes into the master branch of the change control \nmenu non-bare repo on the goldbox.  The menu runs from the working tree \nso now they are live.\n\nSince I won't be trying to access refs in the \"destination\" repos via \ngit commandline or gui while the rsync is running, it sounds like it \nwill be ok.  I can keep people from banging on the testbox.  I can't \nkeep them from banging on the goldbox.\n\nIn regards to the working tree of a busy \"source\" repo, it sounds like \nit could end up not matching the index.  At the end of the script I \ncould execute a \"git reset --hard && git clean -f\" on each non-bare repo \nas you suggested.  That would be pretty straightforward.\n\nThanks!\n\nv/r,\nNeal\n"}]}