{"thread":{"id":"16652","subject":"git-bpush: Pushing to a bundle","startedAt":"2008-12-09T09:49:14Z","lastAt":"2008-12-09T17:32:18Z","messageCount":5,"participants":["Santi Béjar","Johannes Schindelin","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"97417","messageId":"adf1fd3d0812090149m158fcb9as15bacce58c61a1a3@mail.gmail.com","threadId":"16652","inReplyTo":null,"subject":"git-bpush: Pushing to a bundle","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2008-12-09T09:49:14Z","receivedAt":"2008-12-09T09:49:14Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"Hi *,\n\n  I've made a script to push to a bundle that tries to behave as the\nnormal push.\n\n  It has some limitations, but for the normal cases it works fine.\n\nThe basic idea is:\n\n- Easily create bundles with the current branch.\n- Be able to push to defined bundles in remote.<remote>.url\n- Only add new objects by default (do not lose objects)\n- Reuse existing bundles (keep the basis and the branches)\n- Check that the branches fast-forward\n- Keep the pushed branches in refs/remotes/<remote>/*\n\nBut it also has some limitations:\n\n- Do not allow refspec (git-bundle do not support them), only branch/tags names.\n- Push all branches or none (consequence of the above)\n- ...\n\nHope this helps,\nSanti\n\nThe scripts follows, but also attatched.\n#!/bin/sh\n\nOPTIONS_KEEPDASHDASH=\nOPTIONS_SPEC=\"\\\ngit bpush [options] [<remote> [<refs>...]]\n--\nf,force   force updates\nfull      create a full bundle\nv         be verbose\n\"\nSUBDIRECTORY_OK=Yes\n. git-sh-setup\n. git-parse-remote\n\ncd_to_toplevel\n\nLF='\n'\nIFS=\"$LF\"\n\nbases=\nbbases=\nchanged=\nforce=\nnonff=\nremote=\nrefs=\nwhile [ $# != 0 ] ; do\n    case \"$1\" in\n\t-v) verbose=t;;\n\t--full) full=t;;\n\t-f|--force) force=t;;\n\t--) shift; break;;\n\t*) usage;;\n    esac\n    shift\ndone\n\n[ -n \"$1\" ] && remote=$1 && shift\nwhile [ $# != 0 ] ; do\n    refs=\"$refs$LF$1\" && shift\ndone\n\n[ -z \"$remote\" ] && remote=$(get_default_remote)\nremoteurl=$(git config remote.${remote}.url)\n[ -z \"$remoteurl\" ] && remoteurl=$remote\n[ -d \"$remoteurl\" ] && die \"$remoteurl is a directory\"\n\n# Default bases in bundle.base\n# Default {refs,base} can be specified in remote.<remote>.{push,bundlebase}\nif [ \"$remote\" != \"$remoteurl\" ] ; then\n    [ -z \"$refs\" ] &&\n    refs=$(git config --get-all remote.${remote}.push)\n    bases=$(git config --get-all remote.${remote}.bundlebase ||\n\tgit config --get-all bundle.base)\nelse\n    bases=$(git config --get-all bundle.base)\nfi\n\n# git rev-parse --symbolic-full-name resolves symlinks\n# Keep at least HEAD\nhead=\nfor ref in $refs ; do\n    [ \"$ref\" = HEAD ] && head=t && break\ndone\n\n[ -n \"$bases\" ] && bases=$(git rev-parse --revs-only $bases | sort -u)\n# Full symbolic refs to be uniq\n[ -n \"$refs\" ] && \\\n    refs=$(git-rev-parse --symbolic-full-name --revs-only $refs | sort -u) && \\\n    [ -n \"$head\" ] && refs=\"HEAD$LF$refs\"\n\nif [ -e \"$remoteurl\" ] ; then\n    # Find the bundle's bases\n    refs=\"$refs$LF$(git bundle list-heads $remoteurl | cut -d \" \" -f 2)\"\n    requires=\n    for line in $(git bundle verify \"$remoteurl\" 2>/dev/null) ; do\n\tcase \"$line\" in \"The bundle requires\"*) requires=t && continue; esac\n\t[ -z \"$requires\" ] && continue\n\tbbase=$(echo $line | cut -d \" \" -f 1)\n\t[ -z \"$bbases\" ] && bbases=$bbase && continue\n\tbbases=\"$bbases$LF$bbase\"\n    done\n    bbases=$(echo \"$bbases\" | sort -u)\n    [ -z \"$bases\" ] && bases=\"$bbases\" || bases=\"$bases$LF$bbases\"\nelif [ -z \"$refs\" ] ; then\n    # Push current branch\n    refs=\"HEAD$LF$(git symbolic-ref -q HEAD)\"\nfi\n\n[ -z \"$refs\" ] && die \"No refs to push\"\n\nrefs=$(echo \"$refs\" | sort -u)\n\nfor ref in $bases $refs ; do\n    [ \"$(git cat-file -t $ref^{})\" != commit ] && \\\n\tdie \"$(basename $0): $ref is not a commit\"\ndone\n\nheader=\"To $remoteurl\"\n[ -n \"$verbose\" ] && echo \"Pushing to $remoteurl\" && echo $header && header=\n\n# Find what is/is not a fast-forward, up to date or new\n# As \"git bundle\" does not support refspecs we must push all matching branches\nfor ref in $refs ; do\n    case $ref in\n\trefs/tags/*) type=tags; newtext=\"new tag\";;\n\trefs/heads/*|HEAD) type=heads ; newtext=\"new branch\" ;;\n    esac\n    newhash=$(git rev-parse $ref)\n    newshort=$(git rev-parse --short $ref)\n    bshort=$(echo $ref | sed -e \"s|^refs/$type/||\")\n    if [ -e \"$remoteurl\" ] ; then\n\tbheads=\"$(git bundle list-heads $remoteurl)\"\n\tfor bhead in $bheads ; do\n\t    bhash=$(echo $bhead | cut -d \" \" -f 1)\n\t    bref=$(echo $bhead | cut -d \" \" -f 2)\n\t    [ \"$bref\" != \"$ref\" ] && continue\n\t    oldshort=$(git-rev-parse --short $bhash)\n\t    case $type in\n\t\ttags)\n\t\t    base=$newhash;;\n\t\theads)\n\t\t    base=$(git merge-base $bref $bhash);;\n\t    esac\n\t    if [ \"$base\" != $bhash ] ; then\n\t\t[ -n \"$header\" ] && echo $header && header=\n\t\tif [ -z \"$force\" ] ; then\n\t\t    nonff=t\n\t\t    echo \" ! [rejected] $bshort -> $bshort (non-fast forward)\"\n\t\telse\n\t\t    changed=t\n\t\t    echo \" + $oldshort...$newshort $bshort -> $bshort (forced update)\"\n\t\tfi\n\t\tcontinue 2\n\t    fi\n\t    if [ \"$newhash\" != \"$bhash\" ] ; then\n\t\tchanged=t\n\t\t[ -n \"$header\" ] && echo $header && header=\n\t\techo \"   $oldshort..$newshort $bshort -> $bshort\"\n\t    elif [ -n \"$verbose\" ] ; then\n\t\t[ -n \"$header\" ] && echo $header && header=\n\t\techo \" = [up to date] $bshort -> $bshort\"\n\t    fi\n\t    continue 2\n\tdone\n    fi\n    [ -n \"$header\" ] && echo $header && header=\n    echo \" * [$newtext] $bshort -> $bshort\"\n    changed=t\ndone\n\n[ -n \"$full\" ] && bases= && [ -n \"$bbases\" ] && changed=t\n\n[ -n \"$nonff\" ] && die \"error: failed to push some refs to $remoteurl\"\n[ -z \"$changed\" ] && die \"Everything up-to-date\"\n[ -n \"$bases\" ] && bases=\"--not$LF$bases\"\n\ngit bundle create $remoteurl $refs $bases\n[ \"$remote\" != \"$remoteurl\" ] && git fetch -q \"$remote\"\nexit 0\n"},{"id":"97420","messageId":"alpine.DEB.1.00.0812091100470.2916@eeepc-johanness","threadId":"16652","inReplyTo":"adf1fd3d0812090149m158fcb9as15bacce58c61a1a3@mail.gmail.com","subject":"Re: git-bpush: Pushing to a bundle","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-12-09T10:07:37Z","receivedAt":"2008-12-09T10:07:37Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 9 Dec 2008, Santi Béjar wrote:\n\n> The basic idea is:\n> \n> - Easily create bundles with the current branch.\n> - Be able to push to defined bundles in remote.<remote>.url\n> - Only add new objects by default (do not lose objects)\n\nThat is probably not what people need.  Usually, when bundles are sent \naround, you need _incremental_ bundles.  IOW if you already have a bundle, \nyou want to create a new bundle that contains everything that is new, _in \naddition_ to the existing bundle.\n\n> while [ $# != 0 ] ; do\n\nHeh, I did not realize just how _used_ I got to the conventions in Git's \nshell programming, until I thought \"Should this not use 'test' instead \nof brackets?\"\n\n> while [ $# != 0 ] ; do\n>     refs=\"$refs$LF$1\" && shift\n> done\n\nThat is equivalent to refs=\"$*\", no?\n\nAnyway, I found reading your shell script quite hard, because of excessive \nuse of brackets and single line && chains (which lack proper error \nhandling, BTW).\n\nCiao,\nDscho\n"},{"id":"97423","messageId":"adf1fd3d0812090221t2264a4f9i87b5e23be897ee84@mail.gmail.com","threadId":"16652","inReplyTo":"alpine.DEB.1.00.0812091100470.2916@eeepc-johanness","subject":"Re: git-bpush: Pushing to a bundle","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2008-12-09T10:21:36Z","receivedAt":"2008-12-09T10:21:36Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"First, thanks for the comments.\n\n2008/12/9 Johannes Schindelin <Johannes.Schindelin@gmx.de>:\n> Hi,\n>\n> On Tue, 9 Dec 2008, Santi Béjar wrote:\n>\n>> The basic idea is:\n>>\n>> - Easily create bundles with the current branch.\n>> - Be able to push to defined bundles in remote.<remote>.url\n>> - Only add new objects by default (do not lose objects)\n>\n> That is probably not what people need.  Usually, when bundles are sent\n> around, you need _incremental_ bundles.\n\nI do not find convenient strictly incremental bundles, because then\nyou (or the other people) needs to fetch every single bundle. What I\ndo is add new objects until the bundle is too big and then create a\nbundle with a new base. This way you don't have to worry if the other\nperson has applied the last bundle or not.\n\n>  IOW if you already have a bundle,\n> you want to create a new bundle that contains everything that is new, _in\n> addition_ to the existing bundle.\n\n>> while [ $# != 0 ] ; do\n>\n> Heh, I did not realize just how _used_ I got to the conventions in Git's\n> shell programming, until I thought \"Should this not use 'test' instead\n> of brackets?\"\n\nI don't have problems either way, I'll change to follow Git's conventions.\n\n>\n>> while [ $# != 0 ] ; do\n>>     refs=\"$refs$LF$1\" && shift\n>> done\n>\n> That is equivalent to refs=\"$*\", no?\n\nAlmost, IFS is set to line-feed so I needed to put $LF instead of spaces.\n\n>\n> Anyway, I found reading your shell script quite hard, because of excessive\n> use of brackets and single line && chains (which lack proper error\n> handling, BTW).\n\nI've tried to catch errors, but maybe not enough.\n\nSanti\n"},{"id":"97434","messageId":"adf1fd3d0812090658q14001019re8977b912386c256@mail.gmail.com","threadId":"16652","inReplyTo":"493E545B.6010609@viscovery.net","subject":"Re: git-bpush: Pushing to a bundle","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2008-12-09T14:58:20Z","receivedAt":"2008-12-09T14:58:20Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"2008/12/9 Johannes Sixt <j.sixt@viscovery.net>:\n> Santi Béjar schrieb:\n>> 2008/12/9 Johannes Schindelin <Johannes.Schindelin@gmx.de>:\n>>> On Tue, 9 Dec 2008, Santi Béjar wrote:\n>>>> while [ $# != 0 ] ; do\n>>>>     refs=\"$refs$LF$1\" && shift\n>>>> done\n>>> That is equivalent to refs=\"$*\", no?\n>>\n>> Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.\n>\n> But \"$*\" inserts the first character of IFS (not necessarily spaces), and\n> since your IFS *is* $LF, \"$*\" should do what you want.\n>\n\nOh, you are right.\n\n> Anyway, I found reading your shell script quite hard, because of excessive\n> use of brackets and single line && chains (which lack proper error\n> handling, BTW).\n\nI've changed the script to follow the Git's conventions (at least I've\ntried),  a few more error handling and some simplification. BTW, what\ndo you find hard with single line && chains?\n\nI do not sent a diff because it is almost as big as the script itself.\n#!/bin/sh\n\nOPTIONS_KEEPDASHDASH=\nOPTIONS_SPEC=\"\\\ngit bpush [options] [<remote> [<refs>...]]\n--\nf,force   force updates\nfull      create a full bundle\nv         be verbose\n\"\nSUBDIRECTORY_OK=Yes\n. git-sh-setup\n. git-parse-remote\n\ncd_to_toplevel\n\nLF='\n'\nIFS=\"$LF\"\n\nbases=\nbbases=\nchanged=\nforce=\nnonff=\nremote=\nrefs=\nwhile :\ndo\n\tcase \"$1\" in\n\t-v)\n\t\tverbose=t ;;\n\t--full)\n\t\tfull=t ;;\n\t-f|--force)\n\t\tforce=t ;;\n\t--)\n\t\tshift\n\t\tbreak ;;\n\t*)\n\t\tusage ;;\n\tesac\n\tshift\ndone\n\ntest -n \"$1\" && remote=$1 && shift\nrefs=\"$*\"\n\ntest -z \"$remote\" && remote=$(get_default_remote)\nremoteurl=$(git config remote.${remote}.url)\ntest -z \"$remoteurl\" && remoteurl=$remote\ntest -d \"$remoteurl\" && die \"$remoteurl is a directory\"\n\n# Default bases in bundle.base\n# Default {refs,base} can be specified in remote.<remote>.{push,bundlebase}\nif test \"$remote\" != \"$remoteurl\"\nthen\n\ttest -z \"$refs\" &&\n\trefs=$(git config --get-all remote.${remote}.push)\n\tbases=$(git config --get-all remote.${remote}.bundlebase ||\n\t\tgit config --get-all bundle.base)\nelse\n\tbases=$(git config --get-all bundle.base)\nfi\n\n# git rev-parse --symbolic-full-name resolves symlinks\n# Keep at least HEAD\nhead=\nfor ref in $refs ; do\n\ttest \"$ref\" = HEAD && head=t && break\ndone\n\ntest -n \"$bases\" && bases=$(git rev-parse --revs-only $bases | sort -u)\n\n# Full symbolic refs need to be uniq\ntest -n \"$refs\" &&\nrefs=$(git-rev-parse --symbolic-full-name --revs-only $refs | sort -u)\n\ntest -n \"$head\" && refs=\"HEAD$LF$refs\"\n\nif test -e \"$remoteurl\"\nthen\n\tblines=$(git bundle verify \"$remoteurl\" 2>/dev/null) ||\n\tdie \"Verification of \\\"$remoteurl\\\" failed\"\n\t# Find the bundle's bases\n\trefs=\"$refs$LF$(git bundle list-heads $remoteurl | cut -d \" \" -f 2)\"\n\trequires=\n\tfor line in $blines\n\tdo\n\t\tcase \"$requires,$line\" in\n\t\t\",The bundle requires\"*)\n\t\t\trequires=t ;;\n\t\tt,) ;;\n\t\tt,*)\n\t\t\tbbase=$(echo $line | cut -d \" \" -f 1)\n\t\t\tbbases=\"$bbases$LF$bbase\"\n\t\t\t;;\n\t\tesac\n\tdone\n\tbases=\"$bases$LF$bbases\"\nelif test -z \"$refs\" ; then\n\t# Push current branch\n\trefs=\"HEAD$LF$(git symbolic-ref -q HEAD)\"\nfi\n\ntest -z \"$refs\" && die \"No refs to push\"\n\nrefs=$(echo \"$refs\" | sort -u)\n\nfor ref in $bases $refs\ndo\n\ttest \"$(git cat-file -t $ref^{})\" != commit &&\n\tdie \"$(basename $0): $ref is not a commit\"\ndone\n\nheader=\"To $remoteurl\"\ntest -n \"$verbose\" && echo \"Pushing to $remoteurl\" && echo $header && header=\n\n# Find what is/is not a fast-forward, up to date or new\n# As \"git bundle\" does not support refspecs we must push all matching branches\nfor ref in $refs ; do\n\ttext=\n\tbchanged=\n\tcase $ref in\n\trefs/tags/*)\n\t\tbshort=$(echo $ref | sed -e \"s|^refs/tags/||\")\n\t\tnewtext=\"new tag\";;\n\trefs/heads/*|HEAD)\n\t\tbshort=$(echo $ref | sed -e \"s|^refs/heads/||\")\n\t\tnewtext=\"new branch\" ;;\n\tesac\n\tnewhash=$(git rev-parse $ref) || die \"Ref $ref not valid\"\n\tnewshort=$(git rev-parse --short $ref)\n\tbheads=\n\ttest -e \"$remoteurl\" && bheads=\"$(git bundle list-heads $remoteurl)\"\n\tfor bhead in $bheads\n\tdo\n\t\tbhash=$(echo $bhead | cut -d \" \" -f 1)\n\t\tbref=$(echo $bhead | cut -d \" \" -f 2)\n\t\t# Find the matching ref in the bundle\n\t\ttest \"$bref\" != \"$ref\" && continue\n\t\toldshort=$(git rev-parse --short $bhash)\n\t\tmergebase=\n\t\tcase $ref in\n\t\trefs/tags/*)\n\t\t\t# Only test if it is different\n\t\t\tmergebase=$newhash;;\n\t\trefs/heads/*|HEAD)\n\t\t\tmergebase=$(git merge-base $bref $bhash);;\n\t\tesac\n\t\tcase $newhash,$bhash,$mergebase,$force in\n\t\t$bhash,$newhash,*)\n\t\t\t# No changes\n\t\t\ttext=\" = [up to date] $bshort -> $bshort\"\n\t\t\t;;\n\t\t*,*,$bhash,*)\n\t\t\t# Fast-forward\n\t\t\tbchanged=t\n\t\t\ttext=\"   $oldshort..$newshort $bshort -> $bshort\"\n\t\t\t;;\n\t\t*,t)\n\t\t\t# Forced non fast-forward\n\t\t\tbchanged=t\n\t\t\ttext=\" + $oldshort...$newshort $bshort -> $bshort (forced update)\"\n\t\t\t;;\n\t\t*)\n\t\t\tbchanged=t\n\t\t\tnonff=t\n\t\t\ttext=\" ! [rejected] $bshort -> $bshort (non-fast forward)\"\n\t\tesac\n\t\tbreak\n\tdone\n\ttest -z \"$text\" && text=\" * [$newtext] $bshort -> $bshort\" && bchanged=t\n\tif test -n \"$bchanged\" || test -n \"$verbose\"\n\tthen\n\t\ttest -n \"$header\" && echo $header && header=\n\t\techo $text\n\tfi\n\ttest -n \"$bchanged\" && changed=t\ndone\n\n# Recreate the bundle if --full and the current bundle is not full\ntest -n \"$full\" && bases= && test -n \"$bbases\" && changed=t\n\ntest -n \"$nonff\" && die \"error: failed to push some refs to $remoteurl\"\ntest -z \"$changed\" && die \"Everything up-to-date\"\ntest -n \"$bases\" && bases=\"--not$LF$bases\"\n\ngit bundle create $remoteurl $refs $bases ||\ndie \"Cannot create bundle \\\"$remoteurl\\\"\"\n\ntest \"$remote\" != \"$remoteurl\" && { git fetch -q \"$remote\" ||\n\tdie \"Error fetch from bundle \\\"$remoteurl\\\"\" ; }\n\nexit 0\n"},{"id":"97443","messageId":"7voczl45lp.fsf@gitster.siamese.dyndns.org","threadId":"16652","inReplyTo":"adf1fd3d0812090221t2264a4f9i87b5e23be897ee84@mail.gmail.com","subject":"Re: git-bpush: Pushing to a bundle","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-12-09T17:32:18Z","receivedAt":"2008-12-09T17:32:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Santi Béjar\" <santi@agolina.net> writes:\n\n> I do not find convenient strictly incremental bundles, because then\n> you (or the other people) needs to fetch every single bundle. What I\n> do is add new objects until the bundle is too big and then create a\n> bundle with a new base. This way you don't have to worry if the other\n> person has applied the last bundle or not.\n\nYou both have good points.  I sort of tend to side with your argument from\nconvenience point of view, if only because that resembles the way how\npeople traditionally arrange incremental backups \"a full dump on Sunday\nnight, and every day incremental relative to the last full dump\".  Dscho's\nsuggestion is akin to \"a full dump on Sunday night, and every day\nincremental relative to the previous day\".  Both form obviously can\nrecreate the same contents, but often \"incremental since the last full\nsynchronization point\", even though it may make bigger dumps, is easier to\nhandle for humans.\n\n>>  IOW if you already have a bundle,\n>> you want to create a new bundle that contains everything that is new, _in\n>> addition_ to the existing bundle.\n>\n>>> while [ $# != 0 ] ; do\n>>\n>> Heh, I did not realize just how _used_ I got to the conventions in Git's\n>> shell programming, until I thought \"Should this not use 'test' instead\n>> of brackets?\"\n\nNow I see you are improving ;-)\n\n>>> while [ $# != 0 ] ; do\n>>>     refs=\"$refs$LF$1\" && shift\n>>> done\n>>\n>> That is equivalent to refs=\"$*\", no?\n>\n> Almost, IFS is set to line-feed so I needed to put $LF instead of spaces.\n\nIf $IFS is set to LF, \"$*\" will be $1, $2, $3 concatenated with LF in\nbetween.  The first character in $IFS is used for that purpose..\n"}]}