threads / discuss / 16652

git-bpush: Pushing to a bundle

Subject: git-bpush: Pushing to a bundle

## tl;dr

5 messages between Dec 9, 2008 and Dec 9, 2008.

replies: 4people: 3as markdown or json

Santi Béjar· Dec 9, 2008, 09:49 UTC · lore
Hi *,
  I've made a script to push to a bundle that tries to behave as the
normal push.
  It has some limitations, but for the normal cases it works fine.
The basic idea is:
- Easily create bundles with the current branch.
- Be able to push to defined bundles in remote.<remote>.url
- Only add new objects by default (do not lose objects)
- Reuse existing bundles (keep the basis and the branches)
- Check that the branches fast-forward
- Keep the pushed branches in refs/remotes/<remote>/*
But it also has some limitations:
- Do not allow refspec (git-bundle do not support them), only branch/tags names.
- Push all branches or none (consequence of the above)
- ...

Hope this helps, Santi

The scripts follows, but also attatched. #!/bin/sh

OPTIONS_KEEPDASHDASH= OPTIONS_SPEC="\ git bpush [options] [<remote> [<refs>...]] -- f,force force updates full create a full bundle v be verbose " SUBDIRECTORY_OK=Yes . git-sh-setup . git-parse-remote

cd_to_toplevel

LF=' ' IFS="$LF"

bases=
bbases=
changed=
force=
nonff=
remote=
refs=
while [ $# != 0 ] ; do
    case "$1" in
	-v) verbose=t;;
	--full) full=t;;
	-f|--force) force=t;;
	--) shift; break;;
	*) usage;;
    esac
    shift
done
[ -n "$1" ] && remote=$1 && shift
while [ $# != 0 ] ; do
    refs="$refs$LF$1" && shift
done

[ -z "$remote" ] && remote=$(get_default_remote) remoteurl=$(git config remote.${remote}.url) [ -z "$remoteurl" ] && remoteurl=$remote [ -d "$remoteurl" ] && die "$remoteurl is a directory"

# Default bases in bundle.base
# Default {refs,base} can be specified in remote.<remote>.{push,bundlebase}
if [ "$remote" != "$remoteurl" ] ; then
    [ -z "$refs" ] &&
    refs=$(git config --get-all remote.${remote}.push)
    bases=$(git config --get-all remote.${remote}.bundlebase ||
	git config --get-all bundle.base)
else
    bases=$(git config --get-all bundle.base)
fi
# git rev-parse --symbolic-full-name resolves symlinks
# Keep at least HEAD
head=
for ref in $refs ; do
    [ "$ref" = HEAD ] && head=t && break
done
[ -n "$bases" ] && bases=$(git rev-parse --revs-only $bases | sort -u)
# Full symbolic refs to be uniq
[ -n "$refs" ] && \
    refs=$(git-rev-parse --symbolic-full-name --revs-only $refs | sort -u) && \
    [ -n "$head" ] && refs="HEAD$LF$refs"
if [ -e "$remoteurl" ] ; then
    # Find the bundle's bases
    refs="$refs$LF$(git bundle list-heads $remoteurl | cut -d " " -f 2)"
    requires=
    for line in $(git bundle verify "$remoteurl" 2>/dev/null) ; do
	case "$line" in "The bundle requires"*) requires=t && continue; esac
	[ -z "$requires" ] && continue
	bbase=$(echo $line | cut -d " " -f 1)
	[ -z "$bbases" ] && bbases=$bbase && continue
	bbases="$bbases$LF$bbase"
    done
    bbases=$(echo "$bbases" | sort -u)
    [ -z "$bases" ] && bases="$bbases" || bases="$bases$LF$bbases"
elif [ -z "$refs" ] ; then
    # Push current branch
    refs="HEAD$LF$(git symbolic-ref -q HEAD)"
fi
[ -z "$refs" ] && die "No refs to push"
refs=$(echo "$refs" | sort -u)
for ref in $bases $refs ; do
    [ "$(git cat-file -t $ref^{})" != commit ] && \
	die "$(basename $0): $ref is not a commit"
done

header="To $remoteurl" [ -n "$verbose" ] && echo "Pushing to $remoteurl" && echo $header && header=

# Find what is/is not a fast-forward, up to date or new
# As "git bundle" does not support refspecs we must push all matching branches
for ref in $refs ; do
    case $ref in
	refs/tags/*) type=tags; newtext="new tag";;
	refs/heads/*|HEAD) type=heads ; newtext="new branch" ;;
    esac
    newhash=$(git rev-parse $ref)
    newshort=$(git rev-parse --short $ref)
    bshort=$(echo $ref | sed -e "s|^refs/$type/||")
    if [ -e "$remoteurl" ] ; then
	bheads="$(git bundle list-heads $remoteurl)"
	for bhead in $bheads ; do
	    bhash=$(echo $bhead | cut -d " " -f 1)
	    bref=$(echo $bhead | cut -d " " -f 2)
	    [ "$bref" != "$ref" ] && continue
	    oldshort=$(git-rev-parse --short $bhash)
	    case $type in
		tags)
		    base=$newhash;;
		heads)
		    base=$(git merge-base $bref $bhash);;
	    esac
	    if [ "$base" != $bhash ] ; then
		[ -n "$header" ] && echo $header && header=
		if [ -z "$force" ] ; then
		    nonff=t
		    echo " ! [rejected] $bshort -> $bshort (non-fast forward)"
		else
		    changed=t
		    echo " + $oldshort...$newshort $bshort -> $bshort (forced update)"
		fi
		continue 2
	    fi
	    if [ "$newhash" != "$bhash" ] ; then
		changed=t
		[ -n "$header" ] && echo $header && header=
		echo "   $oldshort..$newshort $bshort -> $bshort"
	    elif [ -n "$verbose" ] ; then
		[ -n "$header" ] && echo $header && header=
		echo " = [up to date] $bshort -> $bshort"
	    fi
	    continue 2
	done
    fi
    [ -n "$header" ] && echo $header && header=
    echo " * [$newtext] $bshort -> $bshort"
    changed=t
done
[ -n "$full" ] && bases= && [ -n "$bbases" ] && changed=t

[ -n "$nonff" ] && die "error: failed to push some refs to $remoteurl" [ -z "$changed" ] && die "Everything up-to-date" [ -n "$bases" ] && bases="--not$LF$bases"

git bundle create $remoteurl $refs $bases [ "$remote" != "$remoteurl" ] && git fetch -q "$remote" exit 0

Johannes Schindelin· Dec 9, 2008, 10:07 UTC · re: Santi Béjar · lore

Re: git-bpush: Pushing to a bundle

Hi,
On Tue, 9 Dec 2008, Santi Béjar wrote:
Show 5 quoted lines
> The basic idea is:
> 
> - Easily create bundles with the current branch.
> - Be able to push to defined bundles in remote.<remote>.url
> - Only add new objects by default (do not lose objects)

That is probably not what people need. Usually, when bundles are sent around, you need _incremental_ bundles. IOW if you already have a bundle, you want to create a new bundle that contains everything that is new, _in addition_ to the existing bundle.

> while [ $# != 0 ] ; do

Heh, I did not realize just how _used_ I got to the conventions in Git's shell programming, until I thought "Should this not use 'test' instead of brackets?"

> while [ $# != 0 ] ; do
>     refs="$refs$LF$1" && shift
> done
That is equivalent to refs="$*", no?

Anyway, I found reading your shell script quite hard, because of excessive use of brackets and single line && chains (which lack proper error handling, BTW).

Ciao, Dscho

Santi Béjar· Dec 9, 2008, 10:21 UTC · re: Johannes Schindelin · lore

Re: git-bpush: Pushing to a bundle

First, thanks for the comments.
2008/12/9 Johannes Schindelin <Johannes.Schindelin@gmx.de>:
Show 12 quoted lines
> Hi,
>
> On Tue, 9 Dec 2008, Santi Béjar wrote:
>
>> The basic idea is:
>>
>> - Easily create bundles with the current branch.
>> - Be able to push to defined bundles in remote.<remote>.url
>> - Only add new objects by default (do not lose objects)
>
> That is probably not what people need.  Usually, when bundles are sent
> around, you need _incremental_ bundles.

I do not find convenient strictly incremental bundles, because then you (or the other people) needs to fetch every single bundle. What I do is add new objects until the bundle is too big and then create a bundle with a new base. This way you don't have to worry if the other person has applied the last bundle or not.

>  IOW if you already have a bundle,
> you want to create a new bundle that contains everything that is new, _in
> addition_ to the existing bundle.
Show 5 quoted lines
>> while [ $# != 0 ] ; do
>
> Heh, I did not realize just how _used_ I got to the conventions in Git's
> shell programming, until I thought "Should this not use 'test' instead
> of brackets?"
I don't have problems either way, I'll change to follow Git's conventions.
Show 6 quoted lines
>
>> while [ $# != 0 ] ; do
>>     refs="$refs$LF$1" && shift
>> done
>
> That is equivalent to refs="$*", no?
Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.
>
> Anyway, I found reading your shell script quite hard, because of excessive
> use of brackets and single line && chains (which lack proper error
> handling, BTW).
I've tried to catch errors, but maybe not enough.
Santi
Junio C Hamano· Dec 9, 2008, 17:32 UTC · re: Santi Béjar · lore

Re: git-bpush: Pushing to a bundle

"Santi Béjar" <santi@agolina.net> writes:
Show 5 quoted lines
> I do not find convenient strictly incremental bundles, because then
> you (or the other people) needs to fetch every single bundle. What I
> do is add new objects until the bundle is too big and then create a
> bundle with a new base. This way you don't have to worry if the other
> person has applied the last bundle or not.

You both have good points. I sort of tend to side with your argument from convenience point of view, if only because that resembles the way how people traditionally arrange incremental backups "a full dump on Sunday night, and every day incremental relative to the last full dump". Dscho's suggestion is akin to "a full dump on Sunday night, and every day incremental relative to the previous day". Both form obviously can recreate the same contents, but often "incremental since the last full synchronization point", even though it may make bigger dumps, is easier to handle for humans.

Show 9 quoted lines
>>  IOW if you already have a bundle,
>> you want to create a new bundle that contains everything that is new, _in
>> addition_ to the existing bundle.
>
>>> while [ $# != 0 ] ; do
>>
>> Heh, I did not realize just how _used_ I got to the conventions in Git's
>> shell programming, until I thought "Should this not use 'test' instead
>> of brackets?"
Now I see you are improving ;-)
Show 7 quoted lines
>>> while [ $# != 0 ] ; do
>>>     refs="$refs$LF$1" && shift
>>> done
>>
>> That is equivalent to refs="$*", no?
>
> Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.

If $IFS is set to LF, "$*" will be $1, $2, $3 concatenated with LF in between. The first character in $IFS is used for that purpose..

Santi Béjar· Dec 9, 2008, 14:58 UTC · lore

Re: git-bpush: Pushing to a bundle

2008/12/9 Johannes Sixt <j.sixt@viscovery.net>:
Show 13 quoted lines
> Santi Béjar schrieb:
>> 2008/12/9 Johannes Schindelin <Johannes.Schindelin@gmx.de>:
>>> On Tue, 9 Dec 2008, Santi Béjar wrote:
>>>> while [ $# != 0 ] ; do
>>>>     refs="$refs$LF$1" && shift
>>>> done
>>> That is equivalent to refs="$*", no?
>>
>> Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.
>
> But "$*" inserts the first character of IFS (not necessarily spaces), and
> since your IFS *is* $LF, "$*" should do what you want.
>
Oh, you are right.
> Anyway, I found reading your shell script quite hard, because of excessive
> use of brackets and single line && chains (which lack proper error
> handling, BTW).

I've changed the script to follow the Git's conventions (at least I've tried), a few more error handling and some simplification. BTW, what do you find hard with single line && chains?

I do not sent a diff because it is almost as big as the script itself. #!/bin/sh

OPTIONS_KEEPDASHDASH= OPTIONS_SPEC="\ git bpush [options] [<remote> [<refs>...]] -- f,force force updates full create a full bundle v be verbose " SUBDIRECTORY_OK=Yes . git-sh-setup . git-parse-remote

cd_to_toplevel

LF=' ' IFS="$LF"

bases=
bbases=
changed=
force=
nonff=
remote=
refs=
while :
do
	case "$1" in
	-v)
		verbose=t ;;
	--full)
		full=t ;;
	-f|--force)
		force=t ;;
	--)
		shift
		break ;;
	*)
		usage ;;
	esac
	shift
done

test -n "$1" && remote=$1 && shift refs="$*"

test -z "$remote" && remote=$(get_default_remote) remoteurl=$(git config remote.${remote}.url) test -z "$remoteurl" && remoteurl=$remote test -d "$remoteurl" && die "$remoteurl is a directory"

# Default bases in bundle.base
# Default {refs,base} can be specified in remote.<remote>.{push,bundlebase}
if test "$remote" != "$remoteurl"
then
	test -z "$refs" &&
	refs=$(git config --get-all remote.${remote}.push)
	bases=$(git config --get-all remote.${remote}.bundlebase ||
		git config --get-all bundle.base)
else
	bases=$(git config --get-all bundle.base)
fi
# git rev-parse --symbolic-full-name resolves symlinks
# Keep at least HEAD
head=
for ref in $refs ; do
	test "$ref" = HEAD && head=t && break
done
test -n "$bases" && bases=$(git rev-parse --revs-only $bases | sort -u)

# Full symbolic refs need to be uniq test -n "$refs" && refs=$(git-rev-parse --symbolic-full-name --revs-only $refs | sort -u)

test -n "$head" && refs="HEAD$LF$refs"
if test -e "$remoteurl"
then
	blines=$(git bundle verify "$remoteurl" 2>/dev/null) ||
	die "Verification of \"$remoteurl\" failed"
	# Find the bundle's bases
	refs="$refs$LF$(git bundle list-heads $remoteurl | cut -d " " -f 2)"
	requires=
	for line in $blines
	do
		case "$requires,$line" in
		",The bundle requires"*)
			requires=t ;;
		t,) ;;
		t,*)
			bbase=$(echo $line | cut -d " " -f 1)
			bbases="$bbases$LF$bbase"
			;;
		esac
	done
	bases="$bases$LF$bbases"
elif test -z "$refs" ; then
	# Push current branch
	refs="HEAD$LF$(git symbolic-ref -q HEAD)"
fi
test -z "$refs" && die "No refs to push"
refs=$(echo "$refs" | sort -u)
for ref in $bases $refs
do
	test "$(git cat-file -t $ref^{})" != commit &&
	die "$(basename $0): $ref is not a commit"
done

header="To $remoteurl" test -n "$verbose" && echo "Pushing to $remoteurl" && echo $header && header=

# Find what is/is not a fast-forward, up to date or new
# As "git bundle" does not support refspecs we must push all matching branches
for ref in $refs ; do
	text=
	bchanged=
	case $ref in
	refs/tags/*)
		bshort=$(echo $ref | sed -e "s|^refs/tags/||")
		newtext="new tag";;
	refs/heads/*|HEAD)
		bshort=$(echo $ref | sed -e "s|^refs/heads/||")
		newtext="new branch" ;;
	esac
	newhash=$(git rev-parse $ref) || die "Ref $ref not valid"
	newshort=$(git rev-parse --short $ref)
	bheads=
	test -e "$remoteurl" && bheads="$(git bundle list-heads $remoteurl)"
	for bhead in $bheads
	do
		bhash=$(echo $bhead | cut -d " " -f 1)
		bref=$(echo $bhead | cut -d " " -f 2)
		# Find the matching ref in the bundle
		test "$bref" != "$ref" && continue
		oldshort=$(git rev-parse --short $bhash)
		mergebase=
		case $ref in
		refs/tags/*)
			# Only test if it is different
			mergebase=$newhash;;
		refs/heads/*|HEAD)
			mergebase=$(git merge-base $bref $bhash);;
		esac
		case $newhash,$bhash,$mergebase,$force in
		$bhash,$newhash,*)
			# No changes
			text=" = [up to date] $bshort -> $bshort"
			;;
		*,*,$bhash,*)
			# Fast-forward
			bchanged=t
			text="   $oldshort..$newshort $bshort -> $bshort"
			;;
		*,t)
			# Forced non fast-forward
			bchanged=t
			text=" + $oldshort...$newshort $bshort -> $bshort (forced update)"
			;;
		*)
			bchanged=t
			nonff=t
			text=" ! [rejected] $bshort -> $bshort (non-fast forward)"
		esac
		break
	done
	test -z "$text" && text=" * [$newtext] $bshort -> $bshort" && bchanged=t
	if test -n "$bchanged" || test -n "$verbose"
	then
		test -n "$header" && echo $header && header=
		echo $text
	fi
	test -n "$bchanged" && changed=t
done

# Recreate the bundle if --full and the current bundle is not full test -n "$full" && bases= && test -n "$bbases" && changed=t

test -n "$nonff" && die "error: failed to push some refs to $remoteurl" test -z "$changed" && die "Everything up-to-date" test -n "$bases" && bases="--not$LF$bases"

git bundle create $remoteurl $refs $bases || die "Cannot create bundle \"$remoteurl\""

test "$remote" != "$remoteurl" && { git fetch -q "$remote" ||
	die "Error fetch from bundle \"$remoteurl\"" ; }
exit 0

← back to recent threads