threads / discuss / 29332

rsync a bunch of git repos

Subject: rsync a *bunch* of git repos

## tl;dr

8 messages between Jan 10, 2012 and Jan 24, 2012.

replies: 7people: 7as markdown or json

Jason· Jan 10, 2012, 21:15 UTC · lore

All,

I have a home directory with a lot of git repos in it. Possibly over a hundred. This is a good thing. :-) However, I'm in the process of changing OS's and need to move my home directory out of my root partition into its own partition.

The nuts and bolts of this aren't difficult, the problem is I don't have a complete understanding of how git stores data. I've heard in the past that it uses a lot of hardlinks and softlinks. I need to make sure, that once I transfer the data, and reboot the machine with the new partition mounted under /home, that all my git repos will be okay.

Going through the git repos one by one would be long and prone to errors, so I'm looking at alternatives. Does this look ok?

## From single user mode, or live cd $ su - # mkfs.ext4 /dev/sda3 # mount /dev/sda3 /mnt/home # rsync --progress -avu /home/ /mnt/home/ # umount /mnt/home # mv /home /home.orig # mkdir /home

### Sanity check? # mount /dev/sda3 /home # rsync -avun /home.orig/ /home/

That'll check that rsync thinks things are ok, but what about git? Is there a git sanity check I could run to detect that three hardlinks still point to the same inode?

One alternative may be to boot to a live CD, delete everything from the root partition except /home, then mv /home/* -> / . Basically, use my current root partition as the new /home partition.

thx,

Jason.

Seth Robertson· Jan 10, 2012, 21:24 UTC · re: Jason · lore

Re: rsync a *bunch* of git repos

In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:

    The nuts and bolts of this aren't difficult, the problem is I don't have
    a complete understanding of how git stores data.  I've heard in the
    past that it uses a lot of hardlinks and softlinks.  I need to make
    sure, that once I transfer the data, and reboot the machine with the new
    partition mounted under /home, that all my git repos will be okay.

Under most circumstances, git will do the right thing. Only if you use specific flags on clone (like --shared or --reference) might something go wrong, and as the manual page explains, you can use git-repack to undo these.

The real solution is, after you rsync, to:

for f in */.git; do (cd $f; git fsck >&/dev/null || echo "$f is BAD!!"); done

If you get no output, you should be golden. If you get output, check to make sure the repo you are copying from is good as well.

					-Seth Robertson

Philip Oakley· Jan 10, 2012, 21:33 UTC · re: Seth Robertson · lore

Re: rsync a *bunch* of git repos

From: "Seth Robertson" <in-gitvger@baka.org> Sent: Tuesday, January 10, 2012 
9:24 PM

Show 14 quoted lines

> In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:
>
>    The nuts and bolts of this aren't difficult, the problem is I don't 
> have
>    a complete understanding of how git stores data.  I've heard in the
>    past that it uses a lot of hardlinks and softlinks.  I need to make
>    sure, that once I transfer the data, and reboot the machine with the 
> new
>    partition mounted under /home, that all my git repos will be okay.
>
> Under most circumstances, git will do the right thing.  Only if you
> use specific flags on clone (like --shared or --reference) might
> something go wrong, and as the manual page explains, you can use
> git-repack to undo these.

I think there is an exception for certain git submodule setups with local repos, where the gitdir link is hard coded to the absolute machine path. There has been some discussion on the list recently about trying to cover this case.

Show 10 quoted lines

>
> The real solution is, after you rsync, to:
>
> for f in */.git; do (cd $f; git fsck >&/dev/null || echo "$f is BAD!!"); 
> done
>
> If you get no output, you should be golden.  If you get output, check
> to make sure the repo you are copying from is good as well.
>
> -Seth Robertson

Junio C Hamano· Jan 10, 2012, 22:30 UTC · re: Philip Oakley · lore

Re: rsync a *bunch* of git repos

"Philip Oakley" <philipoakley@iee.org> writes:

> I think there is an exception for certain git submodule setups with
> local repos, where the gitdir link is hard coded to the absolute
> machine path. There has been some discussion on the list recently
> about trying to cover this case.

But if you read the original post carefully, all repositories involved are under /home before the transition, and they will move to the exact same place under /home anyway, so I do not think the discussion you have in mind would apply to this case.

The only thing that is needed after the move would be to run

	git update-index --refresh

in all of the repositories, I think.

Sergio· Jan 11, 2012, 13:22 UTC · re: Junio C Hamano · lore

Re: rsync a *bunch* of git repos

Junio C Hamano <gitster <at> pobox.com> writes:

Show 20 quoted lines

> 
> "Philip Oakley" <philipoakley <at> iee.org> writes:
> 
> > I think there is an exception for certain git submodule setups with
> > local repos, where the gitdir link is hard coded to the absolute
> > machine path. There has been some discussion on the list recently
> > about trying to cover this case.
> 
> But if you read the original post carefully, all repositories involved are
> under /home before the transition, and they will move to the exact same
> place under /home anyway, so I do not think the discussion you have in
> mind would apply to this case.
> 
> The only thing that is needed after the move would be to run
> 
> 	git update-index --refresh
> 
> in all of the repositories, I think.
> 
>

I keep a desktop and a laptop in sync with unison (which uses rsync) and I confirm that git is entirely happy about that.

the git update-index --refresh is necessary, otherwise git status will incorrectly report dirty trees.

I wonder if it would not make sense to incorporate the update-index --refresh in the git status command.

As an aside: git works fine when repos are transferred with rsync, but git packs are not rsync friendly nor friendly with backup strategies using binary deltas.

Neal Kreitzinger· Jan 11, 2012, 16:52 UTC · re: Sergio · lore

Re: rsync a *bunch* of git repos

On 1/11/2012 7:22 AM, Sergio wrote:

>
> As an aside: git works fine when repos are transferred with rsync, but git packs
> are not rsync friendly nor friendly with backup strategies using binary deltas.
>

All these answers are assuming that your source git repos are quiet during the rsync. If they are not quiet during the rsync then you are going to need a more tailored rsync to do things in a certain order. See this thread for details: http://thread.gmane.org/gmane.comp.version-control.git/168699

v/r, neal

Jason· Jan 24, 2012, 14:11 UTC · re: Seth Robertson · lore

Re: rsync a *bunch* of git repos

Hi Seth,

On Tue, Jan 10, 2012 at 04:24:50PM -0500, Seth Robertson wrote:

Show 17 quoted lines

> 
> In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:
> 
>     The nuts and bolts of this aren't difficult, the problem is I don't have
>     a complete understanding of how git stores data.  I've heard in the
>     past that it uses a lot of hardlinks and softlinks.  I need to make
>     sure, that once I transfer the data, and reboot the machine with the new
>     partition mounted under /home, that all my git repos will be okay.
> 
> Under most circumstances, git will do the right thing.  Only if you
> use specific flags on clone (like --shared or --reference) might
> something go wrong, and as the manual page explains, you can use
> git-repack to undo these.
> 
> The real solution is, after you rsync, to:
> 
> for f in */.git; do (cd $f; git fsck >&/dev/null || echo "$f is BAD!!"); done

Okay, I've been using the new system for a week or so now, and everything looks good. Thanks!

Before the transfer, I went ahead and did the above to make sure the repos were good to begin with. All 279 of them were fine, both before and after.

thx,

Jason.

Andreas T.Auer· Jan 11, 2012, 08:01 UTC · re: Jason · lore

Re: rsync a *bunch* of git repos

On 10.01.2012 22:15 Jason wrote:

Show 7 quoted lines

> The nuts and bolts of this aren't difficult, the problem is I don't have
> a complete understanding of how git stores data.  I've heard in the
> past that it uses a lot of hardlinks and softlinks.  I need to make
> sure, that once I transfer the data, and reboot the machine with the new
> partition mounted under /home, that all my git repos will be okay.
>
>

As far as I know git uses hardlinks to save diskspace, but it doesn't rely on hardlinks to work. It also works with filesystems that don't support hardlinks. But without hardlinks you will probably waste disk space so I recommend the -H parameter to rsync, which preserve hardlinks, if possible. If you have hundreds of git repos that are cloned from each other, than this should make a difference.