{"thread":{"id":"29332","subject":"rsync a *bunch* of git repos","startedAt":"2012-01-10T21:15:48Z","lastAt":"2012-01-24T14:11:52Z","messageCount":8,"participants":["Jason","Seth Robertson","Philip Oakley","Junio C Hamano","Andreas T.Auer","Sergio","Neal Kreitzinger"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"182278","messageId":"20120110211548.GD10255@titan.lakedaemon.net","threadId":"29332","inReplyTo":null,"subject":"rsync a *bunch* of git repos","fromName":"Jason","fromEmail":"git@lakedaemon.net","sentAt":"2012-01-10T21:15:48Z","receivedAt":"2012-01-10T21:15:48Z","isPatch":false,"sender":{"key":"git@lakedaemon.net","avatar":null},"body":"All,\n\nI have a home directory with a lot of git repos in it.  Possibly over a\nhundred.  This is a good thing.  :-)  However, I'm in the process of\nchanging OS's and need to move my home directory out of my root\npartition into its own partition.\n\nThe nuts and bolts of this aren't difficult, the problem is I don't have\na complete understanding of how git stores data.  I've heard in the\npast that it uses a lot of hardlinks and softlinks.  I need to make\nsure, that once I transfer the data, and reboot the machine with the new\npartition mounted under /home, that all my git repos will be okay.\n\nGoing through the git repos one by one would be long and prone to\nerrors, so I'm looking at alternatives.  Does this look ok?\n\n## From single user mode, or live cd\n$ su -\n# mkfs.ext4 /dev/sda3\n# mount /dev/sda3 /mnt/home\n# rsync --progress -avu /home/ /mnt/home/\n# umount /mnt/home\n# mv /home /home.orig\n# mkdir /home\n\n### Sanity check?\n# mount /dev/sda3 /home\n# rsync -avun /home.orig/ /home/\n\nThat'll check that rsync thinks things are ok, but what about git?  Is\nthere a git sanity check I could run to detect that three hardlinks\nstill point to the same inode?\n\nOne alternative may be to boot to a live CD, delete everything from the\nroot partition except /home, then mv /home/* -> / .  Basically, use my\ncurrent root partition as the new /home partition.\n\nthx,\n\nJason.\n"},{"id":"182280","messageId":"201201102124.q0ALOowL026941@no.baka.org","threadId":"29332","inReplyTo":"20120110211548.GD10255@titan.lakedaemon.net","subject":"Re: rsync a *bunch* of git repos","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-01-10T21:24:50Z","receivedAt":"2012-01-10T21:24:50Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:\n\n    The nuts and bolts of this aren't difficult, the problem is I don't have\n    a complete understanding of how git stores data.  I've heard in the\n    past that it uses a lot of hardlinks and softlinks.  I need to make\n    sure, that once I transfer the data, and reboot the machine with the new\n    partition mounted under /home, that all my git repos will be okay.\n\nUnder most circumstances, git will do the right thing.  Only if you\nuse specific flags on clone (like --shared or --reference) might\nsomething go wrong, and as the manual page explains, you can use\ngit-repack to undo these.\n\nThe real solution is, after you rsync, to:\n\nfor f in */.git; do (cd $f; git fsck >&/dev/null || echo \"$f is BAD!!\"); done\n\nIf you get no output, you should be golden.  If you get output, check\nto make sure the repo you are copying from is good as well.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"182282","messageId":"25207BDC675840DEA94AA2B04D6783BD@PhilipOakley","threadId":"29332","inReplyTo":"201201102124.q0ALOowL026941@no.baka.org","subject":"Re: rsync a *bunch* of git repos","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2012-01-10T21:33:44Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"\nFrom: \"Seth Robertson\" <in-gitvger@baka.org> Sent: Tuesday, January 10, 2012 \n9:24 PM\n> In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:\n>\n>    The nuts and bolts of this aren't difficult, the problem is I don't \n> have\n>    a complete understanding of how git stores data.  I've heard in the\n>    past that it uses a lot of hardlinks and softlinks.  I need to make\n>    sure, that once I transfer the data, and reboot the machine with the \n> new\n>    partition mounted under /home, that all my git repos will be okay.\n>\n> Under most circumstances, git will do the right thing.  Only if you\n> use specific flags on clone (like --shared or --reference) might\n> something go wrong, and as the manual page explains, you can use\n> git-repack to undo these.\n\nI think there is an exception for certain git submodule setups with local \nrepos, where the gitdir link is hard coded to the absolute machine path. \nThere has been some discussion on the list recently about trying to cover \nthis case.\n\n>\n> The real solution is, after you rsync, to:\n>\n> for f in */.git; do (cd $f; git fsck >&/dev/null || echo \"$f is BAD!!\"); \n> done\n>\n> If you get no output, you should be golden.  If you get output, check\n> to make sure the repo you are copying from is good as well.\n>\n> -Seth Robertson\n"},{"id":"182285","messageId":"7v39bn9onl.fsf@alter.siamese.dyndns.org","threadId":"29332","inReplyTo":"25207BDC675840DEA94AA2B04D6783BD@PhilipOakley","subject":"Re: rsync a *bunch* of git repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-01-10T22:30:06Z","receivedAt":"2012-01-10T22:30:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" <philipoakley@iee.org> writes:\n\n> I think there is an exception for certain git submodule setups with\n> local repos, where the gitdir link is hard coded to the absolute\n> machine path. There has been some discussion on the list recently\n> about trying to cover this case.\n\nBut if you read the original post carefully, all repositories involved are\nunder /home before the transition, and they will move to the exact same\nplace under /home anyway, so I do not think the discussion you have in\nmind would apply to this case.\n\nThe only thing that is needed after the move would be to run\n\n\tgit update-index --refresh\n\nin all of the repositories, I think.\n"},{"id":"182327","messageId":"4F0D41E3.5020908@ursus.ath.cx","threadId":"29332","inReplyTo":"20120110211548.GD10255@titan.lakedaemon.net","subject":"Re: rsync a *bunch* of git repos","fromName":"Andreas T.Auer","fromEmail":"andreas.t.auer_gtml_37453@ursus.ath.cx","sentAt":"2012-01-11T08:01:39Z","receivedAt":"2012-01-11T08:01:39Z","isPatch":false,"sender":{"key":"andreas.t.auer_gtml_37453@ursus.ath.cx","avatar":null},"body":"On 10.01.2012 22:15 Jason wrote:\n> The nuts and bolts of this aren't difficult, the problem is I don't have\n> a complete understanding of how git stores data.  I've heard in the\n> past that it uses a lot of hardlinks and softlinks.  I need to make\n> sure, that once I transfer the data, and reboot the machine with the new\n> partition mounted under /home, that all my git repos will be okay.\n>\n>    \nAs far as I know git uses hardlinks to save diskspace, but it doesn't \nrely on hardlinks to work. It also works with filesystems that don't \nsupport hardlinks.\nBut without hardlinks you will probably waste disk space so I recommend \nthe -H parameter to rsync, which preserve hardlinks, if possible. If you \nhave hundreds of git repos that are cloned from each other, than this \nshould make a difference.\n"},{"id":"182351","messageId":"loom.20120111T141805-791@post.gmane.org","threadId":"29332","inReplyTo":"7v39bn9onl.fsf@alter.siamese.dyndns.org","subject":"Re: rsync a *bunch* of git repos","fromName":"Sergio","fromEmail":"sergio.callegari@gmail.com","sentAt":"2012-01-11T13:22:54Z","receivedAt":"2012-01-11T13:22:54Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Junio C Hamano <gitster <at> pobox.com> writes:\n\n> \n> \"Philip Oakley\" <philipoakley <at> iee.org> writes:\n> \n> > I think there is an exception for certain git submodule setups with\n> > local repos, where the gitdir link is hard coded to the absolute\n> > machine path. There has been some discussion on the list recently\n> > about trying to cover this case.\n> \n> But if you read the original post carefully, all repositories involved are\n> under /home before the transition, and they will move to the exact same\n> place under /home anyway, so I do not think the discussion you have in\n> mind would apply to this case.\n> \n> The only thing that is needed after the move would be to run\n> \n> \tgit update-index --refresh\n> \n> in all of the repositories, I think.\n> \n> \n\nI keep a desktop and a laptop in sync with unison (which uses rsync) and I\nconfirm that git is entirely happy about that.\n\nthe git update-index --refresh is necessary, otherwise git status will\nincorrectly report dirty trees.\n\nI wonder if it would not make sense to incorporate the update-index --refresh in\nthe git status command.\n\nAs an aside: git works fine when repos are transferred with rsync, but git packs\nare not rsync friendly nor friendly with backup strategies using binary deltas.\n"},{"id":"182359","messageId":"4F0DBE4E.30401@gmail.com","threadId":"29332","inReplyTo":"loom.20120111T141805-791@post.gmane.org","subject":"Re: rsync a *bunch* of git repos","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2012-01-11T16:52:30Z","receivedAt":"2012-01-11T16:52:30Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 1/11/2012 7:22 AM, Sergio wrote:\n\n>\n> As an aside: git works fine when repos are transferred with rsync, but git packs\n> are not rsync friendly nor friendly with backup strategies using binary deltas.\n>\nAll these answers are assuming that your source git repos are quiet \nduring the rsync.  If they are not quiet during the rsync then you are \ngoing to need a more tailored rsync to do things in a certain order. \nSee this thread for details: \nhttp://thread.gmane.org/gmane.comp.version-control.git/168699\n\nv/r,\nneal\n"},{"id":"183015","messageId":"20120124141152.GD23524@titan.lakedaemon.net","threadId":"29332","inReplyTo":"201201102124.q0ALOowL026941@no.baka.org","subject":"Re: rsync a *bunch* of git repos","fromName":"Jason","fromEmail":"git@lakedaemon.net","sentAt":"2012-01-24T14:11:52Z","receivedAt":"2012-01-24T14:11:52Z","isPatch":false,"sender":{"key":"git@lakedaemon.net","avatar":null},"body":"Hi Seth,\n\nOn Tue, Jan 10, 2012 at 04:24:50PM -0500, Seth Robertson wrote:\n> \n> In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:\n> \n>     The nuts and bolts of this aren't difficult, the problem is I don't have\n>     a complete understanding of how git stores data.  I've heard in the\n>     past that it uses a lot of hardlinks and softlinks.  I need to make\n>     sure, that once I transfer the data, and reboot the machine with the new\n>     partition mounted under /home, that all my git repos will be okay.\n> \n> Under most circumstances, git will do the right thing.  Only if you\n> use specific flags on clone (like --shared or --reference) might\n> something go wrong, and as the manual page explains, you can use\n> git-repack to undo these.\n> \n> The real solution is, after you rsync, to:\n> \n> for f in */.git; do (cd $f; git fsck >&/dev/null || echo \"$f is BAD!!\"); done\n\nOkay, I've been using the new system for a week or so now, and\neverything looks good.  Thanks!\n\nBefore the transfer, I went ahead and did the above to make sure the\nrepos were good to begin with.  All 279 of them were fine, both before\nand after.\n\nthx,\n\nJason.\n"}]}