{"thread":{"id":"6872","subject":"[PATCH] Add git-bundle: move objects and references by archive.","startedAt":"2007-02-18T22:47:32Z","lastAt":"2007-02-19T15:03:41Z","messageCount":8,"participants":["Mark Levedahl","Johannes Schindelin","Junio C Hamano","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"34967","messageId":"1171838852534-git-send-email-mdl123@verizon.net","threadId":"6872","inReplyTo":null,"subject":"[PATCH] Add git-bundle: move objects and references by archive.","fromName":"Mark Levedahl","fromEmail":"mdl123@verizon.net","sentAt":"2007-02-18T22:47:32Z","receivedAt":"2007-02-18T22:47:32Z","isPatch":true,"sender":{"key":"mdl123@verizon.net","avatar":"https://avatars.githubusercontent.com/u/5302462?v=4"},"body":"Some workflows require use of repositories on machines that cannot be\nconnected, preventing use of git-fetch / git-push to transport objects\nand references between the repositories.\n\ngit-bundle provides an alternate transport mechanism, effectively allowing\ngit-fetch and git-pull to operate using sneakernet transport. git-bundle\n--create allows the user to create a bundle containing one or more branches\nor tags, but with specified basis assumed to exist on the target repository.\nAt the receiving end, git-bundle acts like git-fetch-pack, allowing the\nuser to invoke git-fetch or git-pull using the bundle file as the URL.\ngit-fetch and git-ls-remote determine they have a bundle URL by checking\nthat the URL points to a file, but are otherwise unchanged in operation\nwith bundles.\n\nSigned-off-by: Mark Levedahl <mdl123@verizon.net>\n---\n Documentation/cmd-list.perl  |    1 +\n Documentation/git-bundle.txt |  146 +++++++++++++++++++++++++++++++++++\n Makefile                     |    3 +-\n git-bundle.sh                |  174 ++++++++++++++++++++++++++++++++++++++++++\n git-fetch.sh                 |   11 ++-\n git-ls-remote.sh             |    7 ++-\n 6 files changed, 338 insertions(+), 4 deletions(-)\n mode change 100755 => 100644 Documentation/cmd-list.perl\n create mode 100644 Documentation/git-bundle.txt\n create mode 100755 git-bundle.sh\n\ndiff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl\nold mode 100755\nnew mode 100644\nindex a2d6268..f61c77a\n--- a/Documentation/cmd-list.perl\n+++ b/Documentation/cmd-list.perl\n@@ -70,6 +70,7 @@ git-archive                             mainporcelain\n git-bisect                              mainporcelain\n git-blame                               ancillaryinterrogators\n git-branch                              mainporcelain\n+git-bundle                              mainporcelain\n git-cat-file                            plumbinginterrogators\n git-checkout-index                      plumbingmanipulators\n git-checkout                            mainporcelain\ndiff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\nnew file mode 100644\nindex 0000000..27db785\n--- /dev/null\n+++ b/Documentation/git-bundle.txt\n@@ -0,0 +1,146 @@\n+git-bundle(1)\n+=============\n+\n+NAME\n+----\n+git-bundle - Move objects and refs by archive\n+\n+\n+SYNOPSIS\n+--------\n+'git-bundle' --create file <git-rev-list args> <--tar tarspec>\n+'git-bundle' --verify file  <--tar tarspec>\n+'git-bundle' --list-heads file <reflist> <--tar tarspec>\n+'git-bundle' --unbundle file <reflist> <--tar tarspec>\n+\n+DESCRIPTION\n+-----------\n+\n+Some workflows require that one or more branches of development on one\n+machine be replicated on another machine, but the two machines cannot\n+be directly connected so the interactive git protocols (git, ssh,\n+rsync, http) cannot be used.  This command provides suport for\n+git-fetch and git-pull to operate by packaging objects and references\n+in an archive at the originating machine, then importing those into\n+another repository using gitlink:git-fetch[1] and gitlink:git-pull[1]\n+after moving the archive by some means (i.e., by sneakernet).  As no\n+direct connection between repositories exists, the user must specify a\n+basis for the bundle that is held by the destination repository: the\n+bundle assumes that all objects in the basis are already in the\n+destination repository.\n+\n+OPTIONS\n+-------\n+\n+--create file::\n+       Used to create a bundle named 'file'.  This requires the\n+       git-rev-list arguments to define the bundle contents.\n+\n+--verify file::\n+       Used to check that a bundle file is valid and will apply\n+       cleanly to the current repository.  This includes checks on the\n+       bundle format itself as well as checking that the prerequisite\n+       commits exist and are fully linked in the current repository.\n+       git-bundle prints a list of missing commits, if any, and exits\n+       with non-zero status.\n+\n+--list-heads file::\n+       Lists the references defined in the bundle.  If followed by a\n+       list of references, only references matching those given are\n+       printed out.\n+\n+--unbundle file::\n+       Passes the objects in the bundle to gitlink:git-index-pack[1]\n+       for storage in the repository, then prints the names of all\n+       defined references. If a reflist is given, only references\n+       matching those in the given list are printed. This command is\n+       really plumbing, intended to be called only by\n+       gitlink:git-fetch[1].\n+\n+git-rev-list args::\n+       A list of arguments, accepatble to git-rev-parse and\n+       git-rev-list, that specify the specific objects and references\n+       to transport.  For example, \"master~10..master\" causes the\n+       current master reference to be packaged along with all objects\n+       added since its 10th ancestor commit.  There is no explicit\n+       limit to the number of references and objects that may be\n+       packaged.\n+\n+\n+reflist::\n+       A list of references used to limit the references reported as\n+       available. This is principally of use to git-fetch, which\n+       expects to recieve only those references asked for and not\n+       necessarily everything in the pack (in this case, git-bundle is\n+       acting like gitlink:git-fetch-pack[1]).\n+\n+tar tarspec::\n+\n+       git-bundle uses tar, and requires a tar supporting c, f, and\n+       -O. By default, git-bundle looks for gtar on the path, then for\n+       tar if gtar is not found. This can be overridden by explicitly\n+       defining tarspec, or by defining TAR in the environment.\n+\n+SPECIFYING REFERENCES\n+---------------------\n+\n+git-bundle will only package references that are shown by\n+git-show-ref: this includes heads, tags, and remote heads.  References\n+such as master~1 cannot be packaged, but are perfectly suitable for\n+defining the basis.  More than one reference may be packaged, and more\n+than one basis can be specified.  The objects packaged are those not\n+contained in the union of the given bases.  Each basis can be\n+specified explicitly (e.g., ^master~10), or implicitly (e.g.,\n+master~10..master, master --since=10.days.ago).\n+\n+It is very important that the basis used be held by the destination.\n+It is ok to err on the side of conservatism, causing the bundle file\n+to contain objects already in the destination as these are ignored\n+when unpacking at the destination.\n+\n+EXAMPLE\n+-------\n+\n+Assume two repositories exist as R1 on machine A, and R2 on machine B.\n+For whatever reason, direct connection between A and B is not allowed,\n+but we can move data from A to B via some mechanism (CD, email, etc).\n+We want to update R2 with developments made on branch master in R1.\n+We set a tag in R1 (lastR2bundle) after the previous such transport,\n+and move it afterwards to help build the bundle.\n+\n+in R1 on A:\n+$ git-bundle --create mybundle master ^lastR2bundle\n+$ git tag -f lastR2bundle master\n+\n+(move mybundle from A to B by some mechanism)\n+\n+in R2 on B:\n+$ git-bundle --verify mybundle\n+$ git-fetch mybundle  refspec\n+\n+where refspec is refInBundle:localRef\n+\n+\n+Also, with something like this in your config:\n+\n+[remote \"bundle\"]\n+    url = /home/me/tmp/file.bdl\n+    fetch = refs/heads/*:refs/remotes/origin/*\n+\n+You can first sneakernet the bundle file to ~/tmp/file.bdl and\n+then these commands:\n+\n+$ git ls-remote bundle\n+$ git fetch bundle\n+$ git pull bundle\n+\n+would treat it as if it is talking with a remote side over the\n+network.\n+\n+Author\n+------\n+Written by Mark Levedahl <mdl123@verizon.net>\n+\n+GIT\n+---\n+Part of the gitlink:git[7] suite\ndiff --git a/Makefile b/Makefile\nindex ebecbbd..c6d540e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -177,7 +177,8 @@ SCRIPT_SH = \\\n \tgit-applymbox.sh git-applypatch.sh git-am.sh \\\n \tgit-merge.sh git-merge-stupid.sh git-merge-octopus.sh \\\n \tgit-merge-resolve.sh git-merge-ours.sh \\\n-\tgit-lost-found.sh git-quiltimport.sh\n+\tgit-lost-found.sh git-quiltimport.sh \\\n+\tgit-bundle.sh\n \n SCRIPT_PERL = \\\n \tgit-add--interactive.perl \\\ndiff --git a/git-bundle.sh b/git-bundle.sh\nnew file mode 100755\nindex 0000000..19ac137\n--- /dev/null\n+++ b/git-bundle.sh\n@@ -0,0 +1,174 @@\n+#!/bin/sh\n+# Basic handler for bundle files to connect repositories via sneakernet.\n+# Invocation must include action.\n+# This function can create a bundle or provide information on an existing bundle\n+# supporting git-fetch, git-pull, and git-ls-remote\n+\n+USAGE='[--create bundle <git-rev-list args>] |\n+[--verify|--list-heads|--unbundle bundle] <--tar tarspec>\n+     where bundle is the name of the bundle file.'\n+\n+verify() {\n+    # Check bundle version\n+    test -f \"$bfile\" || die \"cannot find $bfile\"\n+    test \"$($TAR -xf \"\"$bfile\"\" -O version)\" = \"v1 git-bundle\" ||\n+        die \"$bfile doesn't look like a v1 bundle file.\"\n+\n+\t# do fast check, then if any prereqs are missing then go line by line\n+\t# to be verbose about the errors\n+    prereqs=$($TAR xf \"$bfile\" -O prerequisites)\n+\ttest -z \"$prereqs\" && return 0\n+\tbad=$(echo \"$prereqs\" | cut -b-40 | git-rev-list --stdin --not --all 2>&1)\n+    if test -n \"$bad\" ; then\n+\t\ttest \"$1\" = \"--silent\" && return 1\n+        echo \"error: $bfile requires the following commits you lack:\"\n+\t\techo \"$prereqs\" |\n+\t\twhile read sha1 comment ; do\n+            missing=$(git-rev-list $sha1 --not --all 2>&1)\n+            test -n \"$missing\" && echo \"$sha1 $comment\"\n+\t\tdone\n+        exit 1\n+    fi\n+    return 0\n+}\n+\n+# list all or just a subset\n+list_heads() {\n+\tif test -z \"$*\" ; then\n+\t\t$TAR -xf \"$bfile\" -O references 2>/dev/null || exit 1\n+\telse\n+\t\t($TAR -xf \"$bfile\" -O references 2>/dev/null || exit 1) |\n+\t\twhile read sha1 ref ; do\n+\t\t\tfor arg in $* ; do\n+\t\t\t\tif test \"${ref%$arg}\" != \"$ref\" ; then\n+\t\t\t\t\techo \"$sha1 $ref\"\n+\t\t\t\t\tbreak\n+\t\t\t\tfi\n+\t\t\tdone\n+\t\tdone\n+\tfi\n+}\n+\n+SUBDIRECTORY_OK=1\n+. git-sh-setup\n+\n+args=\n+action=\n+while test -n \"$1\" ; do\n+    case $1 in\n+        --create|--list-heads|--unbundle|--verify)\n+            action=${1#--}\n+            shift\n+            bfile=$1\n+            test -z \"$bfile\" && die \"$action requires filename\";;\n+        --tar=*)\n+            TAR=${1##--tar=};;\n+        --tar)\n+\t\t\tshift\n+            TAR=$1;;\n+        *)\n+            args=\"$args $1\";;\n+    esac\n+    shift\n+done\n+test -z \"$action\" && die \"No action given, what should I do?\"\n+\n+# what tar to use, prefer gtar, then tar.\n+if test -z \"$TAR\" ; then\n+    GTAR=$(which gtar 2>/dev/null)\n+    TAR=${GTAR:-tar}\n+fi\n+\n+case $action in\n+create)\n+    unknown=$(git-rev-parse --no-revs $args)\n+    test -z \"$unknown\" || die \"unknown option: $unknown\"\n+    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1\n+\n+    # find the refs to carry along and get sha1s for each.\n+    refs=\n+    fullrevargs=\n+    for arg in $gitrevargs ; do\n+        #ignore options and basis refs, get full ref name for things\n+\t\t# we will transport rejecting anything ambiguous (e.g., user\n+\t\t# gives master, have heads/master and remotes/origin/master, we\n+\t\t# keep the former).\n+        case \"$arg\" in\n+            -* | ^*) fullrevargs=\"$fullrevargs $arg\";;\n+            *)  ref=$(git-show-ref \"$arg\")\n+                test \"$(echo $ref | wc -w)\" = \"2\" || die \"Ambigous reference: $arg\n+$ref\"\n+                fullrevargs=\"$fullrevargs ${ref#* }\"\n+                refs=\"$refs $ref\";;\n+        esac\n+    done\n+    test -z \"$refs\" && die \"No references specified, I don't know what to bundle.\"\n+\n+    # git-rev-list cannot determine edge objects if a date restriction is\n+    # given...  we do things a slow way if max-age or min-age are given\n+    case \"$fullrevargs\" in\n+        *--max-age* | *--min-age*)\n+            # get a list of all commits that will be packed along with\n+            # parents of each.  A fixed git-rev-list --boundary should\n+            # replace all of this.\n+            echo \"Finding prerequisites and commits to bundle...\"\n+            commits=$(git-rev-list $fullrevargs)\n+\n+            # get immediate parents of each commit to include\n+            parents=\n+            for c in $commits ; do\n+                parents=\"$parents $(git-rev-list --parents --max-count=1 $c | cut -b42-)\"\n+            done\n+            parents=$(printf \"%s\\n\" $parents | sort | uniq)\n+\n+            # factor out what will be in this bundle, the remainder are the\n+            # bundle's prerequisites.  double up commits in this as we only\n+            # want things that are only in parents to appear once\n+            prereqs=$(printf \"%s\\n\" $parents $commits $commits | \\\n+                sort | uniq -c | sed -ne 's/^ *1 //p');;\n+        *)\n+            prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');;\n+    esac\n+\n+\t# replace prereqs with annotated version of same\n+\n+    # create refs and pack\n+    tmp=\"$GIT_DIR/bundle_tmp$$\"\n+    prerequisites=\"$tmp/prerequisites\"\n+    references=\"$tmp/references\"\n+    version=\"$tmp/version\"\n+    pack=\"$tmp/pack\"\n+    trap 'rm -rf \"$tmp\"' 0 1 2 3 15\n+\n+    mkdir \"$tmp\" &&\n+    echo \"v1 git-bundle\" > \"$version\" &&\n+\ttouch \"$prerequisites\" &&\n+\t(for p in $prereqs ; do\n+\t\tgit-rev-list --pretty=one --max-count=1 $p\n+\tdone) > \"$prerequisites\" &&\n+    git-show-ref $refs > \"$references\" &&\n+    git-rev-list --objects $fullrevargs | cut -b-40 |\n+        git pack-objects --all-progress --stdout > \"$pack\" &&\n+\n+    # create the tar file, clean up\n+    cd \"$tmp\" &&\n+    tar cf bundle prerequisites references version pack &&\n+    cd - &&\n+    mv \"$tmp/bundle\" \"$bfile\" &&\n+    rm -rf \"$tmp\"\n+\n+    # done\n+    echo \"Created $bfile\";;\n+\n+verify)\n+    verify && echo \"$bfile is ok\";;\n+\n+list-heads)\n+    list_heads $args;;\n+\n+unbundle)\n+    verify --silent || exit 1\n+    $TAR -xf \"$bfile\" -O pack | git-index-pack --stdin ||\n+        die \"error: $bfile has a corrupted pack file\"\n+    list_heads $args;;\n+esac\ndiff --git a/git-fetch.sh b/git-fetch.sh\nindex ca984e7..42cc62f 100755\n--- a/git-fetch.sh\n+++ b/git-fetch.sh\n@@ -377,8 +377,15 @@ fetch_main () {\n     ( : subshell because we muck with IFS\n       IFS=\" \t$LF\"\n       (\n-\t  git-fetch-pack --thin $exec $keep $shallow_depth \"$remote\" $rref ||\n-\t  echo failed \"$remote\"\n+\tif test -f \"$remote\" ; then\n+\t    test -n \"$shallow_depth\" &&\n+\t\tdie \"shallow clone with bundle is not supported\"\n+\t    git-bundle --unbundle \"$remote\" $rref ||\n+\t    echo failed \"$remote\"\n+\telse\n+\t    git-fetch-pack --thin $exec $keep $shallow_depth \"$remote\" $rref ||\n+\t    echo failed \"$remote\"\n+\tfi\n       ) |\n       (\n \ttrap '\ndiff --git a/git-ls-remote.sh b/git-ls-remote.sh\nindex 8ea5c5e..28bb9b8 100755\n--- a/git-ls-remote.sh\n+++ b/git-ls-remote.sh\n@@ -89,8 +89,13 @@ rsync://* )\n \t;;\n \n * )\n-\tgit-peek-remote $exec \"$peek_repo\" ||\n+\tif test -f \"$peek_repo\" ; then\n+\t\tgit bundle --list-heads \"$peek_repo\" ||\n \t\techo \"failed\tslurping\"\n+\telse\n+\t\tgit-peek-remote $exec \"$peek_repo\" ||\n+\t\techo \"failed\tslurping\"\n+\tfi\n \t;;\n esac |\n sort -t '\t' -k 2 |\n-- \n1.5.0.rc4.375.gd0938-dirty\n"},{"id":"34968","messageId":"Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6872","inReplyTo":"1171838852534-git-send-email-mdl123@verizon.net","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-19T01:07:36Z","receivedAt":"2007-02-19T01:07:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nSorry to be such a PITA, but I really, really think that it is wrong to \nmake a tar dependency here. You said your cygwin has problems with binary \nfiles. Could you please try this:\n\n\t$ echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n\nIf it returns anything else than \"<tab>0<tab>1<tab>8\", then your setup \nworks differently from mine. I fit returns what I expect it to, then we \ncan use cat directly and do not have to move the tar bloat around (it is \ninherently block based, so it wastes a lot of space, and it also stores \nother things we'll never use, like permissions for all files).\n\nOn Sun, 18 Feb 2007, Mark Levedahl wrote:\n\n> diff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl\n> old mode 100755\n> new mode 100644\n\nThis is unintended, right?\n\n> diff --git a/git-bundle.sh b/git-bundle.sh\n> new file mode 100755\n> index 0000000..19ac137\n> --- /dev/null\n> +++ b/git-bundle.sh\n> @@ -0,0 +1,174 @@\n> +#!/bin/sh\n> +# Basic handler for bundle files to connect repositories via sneakernet.\n> +# Invocation must include action.\n> +# This function can create a bundle or provide information on an existing bundle\n> +# supporting git-fetch, git-pull, and git-ls-remote\n> +\n> +USAGE='[--create bundle <git-rev-list args>] |\n> +[--verify|--list-heads|--unbundle bundle] <--tar tarspec>\n> +     where bundle is the name of the bundle file.'\n\nWe seem to prefer another (more consistent?) way to describe things: If \nthe argument is not an option, we embrace it with \"<>\", for example:\n\n\t<bundle>\n\nif it is an optional option, we put it in \"[]\", for example:\n\n\t[--tar <tarfile>]\n\nand if one of several mutual exclusive options has to be passed, we put it \nin \"()\" delimited with \"|\". Also, we put no explanation in the output of \n\"-h\", since you should read the man page if you don't know what the \noptions mean. So, it is\n\n\tgit-bundle ( --create <bundle> [<rev-list-args>...] |\n\t\t --verify | --list-heads | --unbundle <bundle> )\n\t\t[--tar <tarspec>]\n\n> +verify() {\n\nThis function is nicely done, AFAICT!\n\n> +list_heads() {\n> +\tif test -z \"$*\" ; then\n> +\t\t$TAR -xf \"$bfile\" -O references 2>/dev/null || exit 1\n> +\telse\n> +\t\t($TAR -xf \"$bfile\" -O references 2>/dev/null || exit 1) |\n\nAFAICT this will not work as expected: \"()\" opens a subshell, and \"exit 1\" \nwill only exit that subshell. Correct me if I'm wrong.\n\n> +\t\twhile read sha1 ref ; do\n> +\t\t\tfor arg in $* ; do\n> +\t\t\t\tif test \"${ref%$arg}\" != \"$ref\" ; then\n> +\t\t\t\t\techo \"$sha1 $ref\"\n> +\t\t\t\t\tbreak\n> +\t\t\t\tfi\n> +\t\t\tdone\n> +\t\tdone\n> +\tfi\n> +}\n\nThe complexity here is unnecessarily high (O(N*M)), but I guess we cannot \ndo better in shell. Maybe constructing a regexp from the args, and piping \nthe output from tar through a grep would help here. Dunno. Maybe it \ndoesn't matter much in reality, anyway.\n\n> +SUBDIRECTORY_OK=1\n> +. git-sh-setup\n\nWe tend to write this part at the very top of a git script.\n\n> +while test -n \"$1\" ; do\n> +    case $1 in\n> +        --create|--list-heads|--unbundle|--verify)\n> +            action=${1#--}\n> +            shift\n> +            bfile=$1\n\nThis can contain spaces (you are working on Windows, right? Windows users \n_love_ spaces in their filenames).\n\n> +            test -z \"$bfile\" && die \"$action requires filename\";;\n> +        --tar=*)\n> +            TAR=${1##--tar=};;\n> +        --tar)\n> +\t\t\tshift\n> +            TAR=$1;;\n> +        *)\n> +            args=\"$args $1\";;\n\nI know you want to mix the --tar option with the rev-list options (which I \nstill find totally confusing, and unnecessary at best), but you _have_ to \nquote \"$1\" in the args=... part, then. Even refs can contain spaces these \ndays.\n\n> +# what tar to use, prefer gtar, then tar.\n> +if test -z \"$TAR\" ; then\n> +    GTAR=$(which gtar 2>/dev/null)\n\nSomebody on this list said that \"which\" should not be relied upon IIRC. I \nforgot why, but I obey that hint.\n\n> +    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1\n\nHere, you rely again on the refs not containing spaces.\n\n> +    # git-rev-list cannot determine edge objects if a date restriction is\n> +    # given...  we do things a slow way if max-age or min-age are given\n\nMight make sense to teach max-age about boundary commits instead...\n\n> +            prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');;\n\nSince you are not interested in non-commits at all, why not use \n\"--boundary\" instead?\n\n> +    # create refs and pack\n> +    tmp=\"$GIT_DIR/bundle_tmp$$\"\n\nShould we really rely on $GIT_DIR being writable? For unbundling, yes, but \nfor bundling? Given the great confusion with git-status trying to write \ninto $GIT_DIR, and people _demanding_ that it does not do so, I recommend \nagainst that.\n\nIn the following part, refnames must not contain spaces again:\n\n> +\t(for p in $prereqs ; do\n> +\t\tgit-rev-list --pretty=one --max-count=1 $p\n> +\tdone) > \"$prerequisites\" &&\n> +    git-show-ref $refs > \"$references\" &&\n> +    git-rev-list --objects $fullrevargs | cut -b-40 |\n> +        git pack-objects --all-progress --stdout > \"$pack\" &&\n\npack-objects can take rev-list arguments itself these days, using \"--revs\" \nand being piped the rev-list arguments.\n\n> +verify)\n> +    verify && echo \"$bfile is ok\";;\n> +\n> +list-heads)\n> +    list_heads $args;;\n\nI like this abstraction (defining a function, and calling that). Why don't \nyou do it with all commands?\n\nRest looks fine.\n\nGiven the problems you had with \"cat\" on Cygwin, and the quoting stuff, I \nthink it might make sense to make this a builtin right away, before the \nscript stage (which is meant to speed things up, not slow them down).\n\nCiao,\nDscho\n"},{"id":"34970","messageId":"7vlkiudii3.fsf@assigned-by-dhcp.cox.net","threadId":"6872","inReplyTo":"1171838852534-git-send-email-mdl123@verizon.net","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-19T01:49:08Z","receivedAt":"2007-02-19T01:49:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mark Levedahl <mdl123@verizon.net> writes:\n\n> Some workflows require use of repositories on machines that cannot be\n> connected, preventing use of git-fetch / git-push to transport objects\n> and references between the repositories.\n\nLooks much nicer.  Thanks.\n\n>  6 files changed, 338 insertions(+), 4 deletions(-)\n>  mode change 100755 => 100644 Documentation/cmd-list.perl\n\nHmmmm?\n\n> +verify() {\n> +    # Check bundle version\n> +    test -f \"$bfile\" || die \"cannot find $bfile\"\n> +    test \"$($TAR -xf \"\"$bfile\"\" -O version)\" = \"v1 git-bundle\" ||\n> +        die \"$bfile doesn't look like a v1 bundle file.\"\n\nIt makes me feel uneasy to see these double quotes next to each\nother.  Are you sure you got your quoting right (I haven't\nchecked)?\n\nJust a minor style issue: traditional single command letter to\ntar is easier to read if you do not write leading '-'.  IOW,\n\"tar xf\" is preferred over \"tar -xf\".\n\n> +\t# do fast check, then if any prereqs are missing then go line by line\n> +\t# to be verbose about the errors\n> +    prereqs=$($TAR xf \"$bfile\" -O prerequisites)\n> +\ttest -z \"$prereqs\" && return 0\n> +\tbad=$(echo \"$prereqs\" | cut -b-40 | git-rev-list --stdin --not --all 2>&1)\n> +    if test -n \"$bad\" ; then\n> +\t\ttest \"$1\" = \"--silent\" && return 1\n> +        echo \"error: $bfile requires the following commits you lack:\"\n> +\t\techo \"$prereqs\" |\n> +\t\twhile read sha1 comment ; do\n> +            missing=$(git-rev-list $sha1 --not --all 2>&1)\n> +            test -n \"$missing\" && echo \"$sha1 $comment\"\n> +\t\tdone\n> +        exit 1\n> +    fi\n\nThis part is very hard to read.  Maybe your tabstop is not set\nto 8?\n\nLinus rightly said \"or something like that\"; I do not think the\nempty output from \"git-rev-list --stdin --not --all\" is enough.\n\nHere, you are making sure that each line of $prereqs are names\nof objects available in your repository, and well connected to\nyour existing refs.  If one of the commits listed in $prereqs\ndoes not exist in your repository, I think rev-list exits with\nnon-zero status, with a message to stderr, without spitting out\nanything to its stdout, so you would need:\n\n\tif bad=$(echo ... | rev-list ...) && test -z \"$bad\"\n        then\n        \t: happy\n\telse\n        \techo 2>&1 \"error: you lack these...\"\n\tfi\n\nIncidentally, I think you do not have to squelch error message\nfrom rev-list here.\n\nOther than that, I think Linus's check is correct and the above\nfills his \"something like that\" [*1*].\n\n\n> +list_heads() {\n> +\tif test -z \"$*\" ; then\n> +\t\t$TAR -xf \"$bfile\" -O references 2>/dev/null || exit 1\n> +\telse\n> +\t\t($TAR -xf \"$bfile\" -O references 2>/dev/null || exit 1) |\n> +\t\twhile read sha1 ref ; do\n> +\t\t\tfor arg in $* ; do\n> +\t\t\t\tif test \"${ref%$arg}\" != \"$ref\" ; then\n> +\t\t\t\t\techo \"$sha1 $ref\"\n> +\t\t\t\t\tbreak\n> +\t\t\t\tfi\n> +\t\t\tdone\n> +\t\tdone\n> +\tfi\n> +}\n\nI suspect that \"exit 1\" in the upstream of pipe does not have\nany effect on the outcome -- yup, I just tried...\n\n\t$ (tar xf no-such-file -O references || exit 1) |\n\t  while read foo; do echo $foo; done ; echo $?\n        0\n\t$\n\n> +test -z \"$action\" && die \"No action given, what should I do?\"\n> +\n> +# what tar to use, prefer gtar, then tar.\n> +if test -z \"$TAR\" ; then\n> +    GTAR=$(which gtar 2>/dev/null)\n> +    TAR=${GTAR:-tar}\n> +fi\n\n\"which\" is Ok for interactive use, but never use it in scripts.\nI've seen \"which foo\" that says \"No foo found.\" to its standard\noutput without erroring out (I think it was older Solaris).\n\n+unbundle)\n+    verify --silent || exit 1\n+    $TAR -xf \"$bfile\" -O pack | git-index-pack --stdin ||\n+        die \"error: $bfile has a corrupted pack file\"\n+    list_heads $args;;\n+esac\n\nHmm.  Your verify () shell function silently fails if prereq is\nnot met, so this makes the user of \"git --unbundle file.bdl\"\nwould get silent failure and left wondering why the bundle does\nnot unpack, doesn't it?\n\n> diff --git a/git-fetch.sh b/git-fetch.sh\n> index ca984e7..42cc62f 100755\n> --- a/git-fetch.sh\n> +++ b/git-fetch.sh\n> @@ -377,8 +377,15 @@ fetch_main () {\n\nNice.\n\n> diff --git a/git-ls-remote.sh b/git-ls-remote.sh\n> index 8ea5c5e..28bb9b8 100755\n> --- a/git-ls-remote.sh\n> +++ b/git-ls-remote.sh\n> @@ -89,8 +89,13 @@ rsync://* )\n\nLikewise.\n\n\n[footnote]\n\n*1* If somebody wants to know why it works...\n\nSuppose the \"global\" history looked like this:\n\n\t---A---B---C\n\nand your receiving repository earlier had its tip of the branch\nat commit A.  Then the user used git-http-fetch to fetch from a\nrepository that had its tip at commit B.  http-fetch (another\n\"commit walker\", local-fetch, works the same way) learns that\nthe tip commit is B, downloads commit B, reads what's in there\nto learn the tree associated with commit B, fetches that tree,\nreads what's in there to learn the subtrees and blobs in it, ...\n\nThe user can interrupt this process, at the point after commit B\nand everything that is needed to complete it was downloaded, but\nbefore it actually updated the ref to point at B.\n\nSuppose then you created a bundle with B..C to update the tip to\nC, requiring you to have B.  \"echo B | rev-list --not --all\"\nwould happily say \"Oh, B is a commit that is not reachable from\nany of your refs\", but in fact you can safely update the ref\nthat currently point at A (because the user interrupted the\nearlier fetch) with C after checking C is a fast forward of A.\nThis might make \"outputs empty\" look like a wrong check.\n\nHowever, the user could have interrupted the earlier http-fetch,\nat the point after commit B was downloaded but before finishing\nto download the tree associated with it.  A commit walker does\nnot update the ref, so the commit object B will be in your\nreceiving repository but it is not complete (it lacks its tree\nobject).  Even in this case \"echo B | rev-list --not --all\"\nwould say \"B is ahead of your repository\".  That's why \"outputs\nempty\" is a good check.\n\nBefore somebody wonders further.  An alternative good check is\nto see if \"echo B | rev-list --objects --not --all\" does _not_\nfail, and all of the objects listed in its output are available\nin the receiving repository.  This will solve the apparent\ninefficiency in the case of killing http-fetch after it\ndownloaded all the necessary objects before it updated the ref,\nwhich I described earlier.  But I think that the window for that\nto happen is very small and if you killed a commit walker it is\nmuch more likely that you lack some objects, so doing this\nalternative check is optimizing for a wrong case.\n"},{"id":"34971","messageId":"7v7iuedif9.fsf@assigned-by-dhcp.cox.net","threadId":"6872","inReplyTo":"Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-19T01:50:50Z","receivedAt":"2007-02-19T01:50:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Sorry to be such a PITA, but I really, really think that it is wrong to \n> make a tar dependency here. You said your cygwin has problems with binary \n> files. Could you please try this:\n>\n> \t$ echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n\nIt might be just me, but \"echo -e\" makes me feel much more\nuneasy than explicitly saying \"Sorry, we require GNU tar in\nthis little corner of the system\".\n\n>> +            bfile=$1\n>\n> This can contain spaces (you are working on Windows, right? Windows users \n> _love_ spaces in their filenames).\n\nWhich is fine.  Try var1=$var2 with something with SP or TAB in var2\nand later try running\n\n\t$ echo \"<$var1>\"\n\nto see what happens.\n\n>> +    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1\n>\n> Here, you rely again on the refs not containing spaces.\n\nWhich probably is fine, as refs cannot contain spaces ;-).\n\n>> +    # git-rev-list cannot determine edge objects if a date restriction is\n>> +    # given...  we do things a slow way if max-age or min-age are given\n>\n> Might make sense to teach max-age about boundary commits instead...\n\nYou are talking about a rather extensive change, but could be\ndone.  But that is a separate issue, I think.\n"},{"id":"34972","messageId":"Pine.LNX.4.63.0702190257250.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6872","inReplyTo":"7v7iuedif9.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-19T02:02:13Z","receivedAt":"2007-02-19T02:02:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 18 Feb 2007, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > Sorry to be such a PITA, but I really, really think that it is wrong to \n> > make a tar dependency here. You said your cygwin has problems with binary \n> > files. Could you please try this:\n> >\n> > \t$ echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n> \n> It might be just me, but \"echo -e\" makes me feel much more\n> uneasy than explicitly saying \"Sorry, we require GNU tar in\n> this little corner of the system\".\n\nHey, this was meant as a _test_ on why it fails for Mark, not as the end \nresult!\n\n> >> +            bfile=$1\n> >\n> > This can contain spaces (you are working on Windows, right? Windows users \n> > _love_ spaces in their filenames).\n> \n> Which is fine.  Try var1=$var2 with something with SP or TAB in var2\n> and later try running\n> \n> \t$ echo \"<$var1>\"\n> \n> to see what happens.\n\nOkay, I did not know that.\n\n> >> +    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1\n> >\n> > Here, you rely again on the refs not containing spaces.\n> \n> Which probably is fine, as refs cannot contain spaces ;-).\n\nAh! As we allow so many things in refnames, I assumed that spaces are \nallowed also.\n\n> >> +    # git-rev-list cannot determine edge objects if a date restriction is\n> >> +    # given...  we do things a slow way if max-age or min-age are given\n> >\n> > Might make sense to teach max-age about boundary commits instead...\n> \n> You are talking about a rather extensive change, but could be\n> done.  But that is a separate issue, I think.\n\nI don't think it's so hard.\n\nCiao,\nDscho\n"},{"id":"34982","messageId":"20070219075654.GB30030@spearce.org","threadId":"6872","inReplyTo":"Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-19T07:56:54Z","receivedAt":"2007-02-19T07:56:54Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> On Sun, 18 Feb 2007, Mark Levedahl wrote:\n> > +    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1\n> \n> Here, you rely again on the refs not containing spaces.\n\nThey cannot.  OK, they can if you avoid any of the Git code:\n\n  git rev-parse HEAD >\".git/refs/heads/ref with space\"\n\nbut a lot of things downstream may get annoyed by doing this.\nSpaces are not permitted in ref names.\n \n> > +    # create refs and pack\n> > +    tmp=\"$GIT_DIR/bundle_tmp$$\"\n> \n> Should we really rely on $GIT_DIR being writable? For unbundling, yes, but \n> for bundling? Given the great confusion with git-status trying to write \n> into $GIT_DIR, and people _demanding_ that it does not do so, I recommend \n> against that.\n\nI agree.  For an application like bundle creation, $GIT_DIR should be\nstrictly read-only.  There is no reason for this application to need\nto create temporary files, as the bundle can be streamed to stdout\n(or optionally to a file supplied by the user on the command line).\n\n> > +    git-rev-list --objects $fullrevargs | cut -b-40 |\n> > +        git pack-objects --all-progress --stdout > \"$pack\" &&\n> \n> pack-objects can take rev-list arguments itself these days, using \"--revs\" \n> and being piped the rev-list arguments.\n\nUh, actually it can't.  I thought it could too.  It doesn't.\nIt can only take a ref name or --not.  I'm actually not sure why\nwe limited it like that, but we did...  It may be a good idea to\nchange that restriction to not exist.\n \n-- \nShawn.\n"},{"id":"35006","messageId":"erc8nm$feg$1@sea.gmane.org","threadId":"6872","inReplyTo":"Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Mark Levedahl","fromEmail":"mdl123@verizon.net","sentAt":"2007-02-19T13:29:22Z","receivedAt":"2007-02-19T13:29:22Z","isPatch":true,"sender":{"key":"mdl123@verizon.net","avatar":"https://avatars.githubusercontent.com/u/5302462?v=4"},"body":"\"Johannes Schindelin\" <Johannes.Schindelin@gmx.de> wrote in message \nnews:Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de...\n> Hi,\n>\n> Sorry to be such a PITA, but I really, really think that it is wrong \n> to\n> make a tar dependency here. You said your cygwin has problems with \n> binary\n> files. Could you please try this:\n>\n> $ echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n>\nSame result on Cygwin and FC6:\n~>echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n      0       1       8\n\n> If it returns anything else than \"<tab>0<tab>1<tab>8\", then your setup\n> works differently from mine. I fit returns what I expect it to, then \n> we\n> can use cat directly and do not have to move the tar bloat around (it \n> is\n> inherently block based, so it wastes a lot of space, and it also \n> stores\n> other things we'll never use, like permissions for all files).\n\n\nTo repeat an earlier message, I tried the following:\n\ncat bundle | (\n      while read <header stuff> do\n        <prcess header stuff>\n      done\n      git-pack-index --stdin\n)\n\nand it worked, MOST of the time, most being the important word here. My \nmemory is already fuzzy about the failure rate, but it was something \nlike 2 of 7 bundle files I passed through resulting in git-pack-index \nreporting a corrupted pack file: when I manually edited out the header \nstuff  and passed the remains of the file directly to git-pack-index \nthings were fine. The same experiment worked perfectly on my FC6 system, \nso it was definitely some part of Cygwin, not the script. If git is \nworried about tickling \"which\" on ancient Solaris, I think it is a bit \nout of balance to simultaneously require a not yet written version of \nCygwin (that will get fixed after someone has time to reduce this \nproblem to a simple testcase and push back to the Cygwin maintainers).\n\nA bundle file *is* an archive. We can use an existing archiver or write \na new one. I don't really favor writing one but that is personal \npreference for not reinventing the wheel, nothing more. However, given \nCgywin's problems an archiver written in shell isn't going to work for \nbinary files. We could uuencode / uudecode the pack files and make \nbundles pure ascii text. Or, we could use tar. Of course, rewriting this \nas a builtin offers the ability to write a purpose built archiver as \nwell, but I was trying to avoid that.\n\n</unsolicited ignoreable gripe>Has anyone considered how much easier all \nof git would be if it were written in c + python, rather than c + every \nvariant of shell + core utils + non-core utils known to mankind since \nthe dawn of unix?</unsolicited ignorable gripe>\n\nMark\n"},{"id":"35010","messageId":"Pine.LNX.4.63.0702191557580.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6872","inReplyTo":"erc8nm$feg$1@sea.gmane.org","subject":"Re: [PATCH] Add git-bundle: move objects and references by archive.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-19T15:03:41Z","receivedAt":"2007-02-19T15:03:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 19 Feb 2007, Mark Levedahl wrote:\n\n> \"Johannes Schindelin\" <Johannes.Schindelin@gmx.de> wrote in message\n> news:Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de...\n> > Hi,\n> > \n> > Sorry to be such a PITA, but I really, really think that it is wrong to\n> > make a tar dependency here. You said your cygwin has problems with binary\n> > files. Could you please try this:\n> > \n> > $ echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n> > \n> Same result on Cygwin and FC6:\n> ~>echo -ne '\\x1a\\x1b\\x15\\x10\\0abc' | cat | wc\n>      0       1       8\n\nOkay, so I really would like to have the bundle file from this experiment:\n\n> To repeat an earlier message, I tried the following:\n> \n> cat bundle | (\n>      while read <header stuff> do\n>        <prcess header stuff>\n>      done\n>      git-pack-index --stdin\n> )\n\nbecause I cannot see how the aforementioned \"cat\" can succeed, while \npack-index --stdin cannot.\n\n> A bundle file *is* an archive.\n\nBut the archive in the bundle really is the pack. You don't need any other \narchive there. You don't need to know the name of the prerequisites file, \nfor example, if that information is in the header to begin with.\n\n> </unsolicited ignoreable gripe>Has anyone considered how much easier all \n> of git would be if it were written in c + python, rather than c + every \n> variant of shell + core utils + non-core utils known to mankind since \n> the dawn of unix?</unsolicited ignorable gripe>\n\nYou realize that Python can be a PITA, a real PITA, when it comes to \nnon-Linux systems?\n\nJust look at the fun we had with subprocess. It is included in Python 2.4, \n_works_ with Python 2.3, but _not_ Python with 2.2. If you now say \"well, \njust upgrade!\" then I SHOUT at you. There is _no_ excuse forcing users to \nupgrade when the failure is on _your_ part.\n\nCiao,\nDscho\n"}]}