Minor issues with "git pull":
1. Multiple rsync call might connect to different servers (with round-robin DNS). The effect will be interesting. One call, if possible, would be better.
2. If any rsync fails, the process should be aborted.
As long as we do have multiple calls:
3. The "grep -v '^MOTD:'" should be changed to kill blank lines after MOTDs also.
Morten
Petr BaudisRe: git pull issues...
Dear diary, on Sat, Apr 23, 2005 at 11:25:32PM CEST, I got a letter where Morten Welinder <mwelinder@gmail.com> told me that...
> Minor issues with "git pull": > > 1. Multiple rsync call might connect to different servers (with > round-robin DNS). The effect > will be interesting. One call, if possible, would be better.
If you can do it without overwriting HEAD, please go ahead and send me the patch. :-)
> 2. If any rsync fails, the process should be aborted.
Fixed.
> As long as we do have multiple calls: > > 3. The "grep -v '^MOTD:'" should be changed to kill blank lines after > MOTDs also.
Actually, I currently use this as a poor man's progress indicator. ;-) *hide*
-- Petr "Pasky" Baudis Stuff: http://pasky.or.cz/ C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
Re: git pull issues...
> > 1. Multiple rsync call might connect to different servers (with > > round-robin DNS). The effect > > will be interesting. One call, if possible, would be better. > > If you can do it without overwriting HEAD, please go ahead and send me > the patch. :-)
I'll have a go at it later, but something like this ought to work:
-d .rsync-git && die "previous pull failed -- cleanup" mkdir .rsync-git || die "cannot create .rsync-git" ln -s ../.git/objects .rsync-git/objects || die "cannot create symlink" rsync ... --keep-dirlinks ... ...
Morten
Re: git pull issues...
On 4/23/05, Petr Baudis <pasky@ucw.cz> wrote:
> > 1. Multiple rsync call might connect to different servers (with > > round-robin DNS). The effect > > will be interesting. One call, if possible, would be better. > > If you can do it without overwriting HEAD, please go ahead and send me > the patch. :-)
This ought to work. It basically saves to a different directory while still ignoring all the files we have.
Morten
Index: gitpull.sh =================================================================== --- 7de71a831508e51e0985cea173f3f7a7012c82b7/gitpull.sh (mode:100755 sha1:6abc7f5c00fd3e082d0a34a238a53b67c38b8a7f) +++ uncommitted/gitpull.sh (mode:100775) @@ -37,28 +37,32 @@ [ -s ".git/heads/$name" ] && orig_head=$(cat ".git/heads/$name") fi - -mkdir -p .git/heads -rsyncerr= -rsync $RSYNC_FLAGS -Lr "$uri/heads/$rembranch" ".git/heads/$name" 2>/dev/null || rsyncerr=1 -if [ "$rsyncerr" ] && [ "$rembranch" = "master" ]; then - rsyncerr= - rsync $RSYNC_FLAGS -Lr "$uri/HEAD" ".git/heads/$name" | grep -v '^MOTD:' || rsyncerr=1 +dotgit=dot-git +test -d "$dotgit" && die "$dotgit already exists -- please clean up." +mkdir -p "$dotgit" "$dotgit/objects" "$dotgit/tags" \ + || die "failed to create $dotgit." + +(cd .git && find objects -type f -print0 && find tags -type f -print0) | \ + rsync $RSYNC_FLAGS --from0 --exclude-from=- --whole-file -Lrv "$uri/" "$dotgit" || \ + { rm -rf "$dotgit"; die "rsync from $uri/ failed."; } + +# Move new files into place +find "$dotgit/objects" -type f -print0 | \ + perl -n0e 'chomp; $src=$_; s{^[^/]+/}{.git/}; rename $src, $_ or die "failed to move $src into place: $!\n";' +find "$dotgit/tags" -type f -print0 | \ + perl -n0e 'chomp; $src=$_; s{^[^/]+/}{.git/}; rename $src, $_ or die "failed to move $src into place: $!\n";' + +mkdir -p .git/heads || die "failed to create .git/heads" +if [ -r "$dotgit/heads/$rembranch" ]; then + mv "$dotgit/heads/$rembranch" ".git/heads/$name" +elif [ "$rembranch" = "master" -a -r "$dotgit/HEAD" ]; then + mv "$dotgit/HEAD" ".git/heads/$name" +else + rm -rf "$dotgit" + die "unable to get the head pointer of branch $rembranch" fi -[ "$rsyncerr" ] && die "unable to get the head pointer of branch $rembranch" - -[ -d .git/objects ] || mkdir -p .git/objects -# We already saw the MOTD, thank you very much. -rsync $RSYNC_FLAGS --ignore-existing --whole-file \ - -v -Lr "$uri/objects/." ".git/objects/." | grep -v '^MOTD:' || die "rsync error" - -# FIXME: Warn about conflicting tag names? -# XXX: We now throw stderr to /dev/null since not all repositories -# may have tags/ and users were confused by the harmless errors. -[ -d .git/tags ] || mkdir -p .git/tags -rsync $RSYNC_FLAGS --ignore-existing \ - -v -Lr "$uri/tags/." ".git/tags/." 2>/dev/null | grep -v '^MOTD:' || die "rsync error" +rm -rf "$dotgit" new_head=$(cat ".git/heads/$name")
Dan HolmsandRe: git pull issues...
Morten Welinder wrote:
> On 4/23/05, Petr Baudis <pasky@ucw.cz> wrote: > >>>1. Multiple rsync call might connect to different servers (with >>>round-robin DNS). The effect >>> will be interesting. One call, if possible, would be better. >> >>If you can do it without overwriting HEAD, please go ahead and send me >>the patch. :-) > > > This ought to work. It basically saves to a different directory while > still ignoring > all the files we have.
I was just looking at the same thing. Not saying this is better, or anything, bit it has, perhaps, the slight advantage that downloaded objects aren't thrown away, so that a pull can be safely resumed. Also, it doesn't download anything from an url that doesn't contain ./HEAD, ./heads or ./objects. That might be a good thing if you accidentally try to pull from, say, kernel.org/pub/...
/dan
Here goes:
Index: gitpull.sh =================================================================== --- 7de71a831508e51e0985cea173f3f7a7012c82b7/gitpull.sh (mode:100755 sha1:6abc7f5c00fd3e082d0a34a238a53b67c38b8a7f) +++ uncommitted/gitpull.sh (mode:100755) @@ -18,7 +18,7 @@ [ "$name" ] || name=$(cat .git/tracking 2>/dev/null) [ "$name" ] || die "where to pull from?" uri=$(grep $(echo -e "^$name\t" | sed 's/\./\\./g') .git/remotes | cut -f 2) -[ "$uri" ] || die "unknown remote" +[ "$uri" ] || die "unknown remote '$name'" rembranch=master if echo "$uri" | grep -q '#'; then @@ -38,28 +38,39 @@ fi -mkdir -p .git/heads -rsyncerr= -rsync $RSYNC_FLAGS -Lr "$uri/heads/$rembranch" ".git/heads/$name" 2>/dev/null || rsyncerr=1 -if [ "$rsyncerr" ] && [ "$rembranch" = "master" ]; then - rsyncerr= - rsync $RSYNC_FLAGS -Lr "$uri/HEAD" ".git/heads/$name" | grep -v '^MOTD:' || rsyncerr=1 +mkdir -p .git/heads .git/objects .git/tags || die +pulldir=$(mktemp -td gitpull.XXXXXX) +[ -d "$pulldir" ] || die "failed to create temp dir" +ln -s $(pwd)/.git/objects $(pwd)/.git/tags $pulldir || die +mkdir $pulldir/heads || die + +echo pulling $uri + +rout=$pulldir/rsync_errors +rsync $RSYNC_FLAGS -LKrv --whole-file --ignore-existing \ + --include '/HEAD' --include '/heads/' --include '/tags/' \ + --include '/objects/' --exclude='/*' \ + "$uri/." $pulldir 2>$rout || { + cat $rout; die "failed to pull from $uri" + } + +if [ -f "$pulldir/heads/$rembranch" ]; then + cp "$pulldir/heads/$rembranch" ".git/heads/$name" || die +elif [ master = "$rembranch" -a -f $pulldir/HEAD ]; then + cp $pulldir/HEAD ".git/heads/$name" || die +else + echo "unable to get the head pointer of branch $rembranch" >&2 + echo "is $uri a valid git repository?" >&2 + die "pull failed" fi -[ "$rsyncerr" ] && die "unable to get the head pointer of branch $rembranch" -[ -d .git/objects ] || mkdir -p .git/objects -# We already saw the MOTD, thank you very much. -rsync $RSYNC_FLAGS --ignore-existing --whole-file \ - -v -Lr "$uri/objects/." ".git/objects/." | grep -v '^MOTD:' || die "rsync error" +# TODO: Check that at least the head commit exists. -# FIXME: Warn about conflicting tag names? -# XXX: We now throw stderr to /dev/null since not all repositories -# may have tags/ and users were confused by the harmless errors. -[ -d .git/tags ] || mkdir -p .git/tags -rsync $RSYNC_FLAGS --ignore-existing \ - -v -Lr "$uri/tags/." ".git/tags/." 2>/dev/null | grep -v '^MOTD:' || die "rsync error" +rm -rf $pulldir +# FIXME: Warn about conflicting tag names? + new_head=$(cat ".git/heads/$name") if [ ! "$orig_head" ]; then
Re: git pull issues...
The two approaches are very much alike -- there are only so many ways to skin a cat, after all.
I actually first considered the -K approach, but then realized that only very new rsyncs support that. My version works with rsync 2.6.2 (and maybe earlier). It could be made to go further back by not using -from0 which isn't really needed here.
But I like your check-for-HEAD detail.
Morten