# [PATCH] Add git-bundle: move objects and references by archive. 8 messages from 2007-02-18 to 2007-02-19. Participants: Mark Levedahl, Johannes Schindelin, Junio C Hamano, Shawn O. Pearce. Thread: https://gitlist.dev/t/6872 ## Mark Levedahl, 2007-02-18 22:47 Subject: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: <1171838852534-git-send-email-mdl123@verizon.net> URL: https://gitlist.dev/e/1171838852534-git-send-email-mdl123%40verizon.net ``` Some workflows require use of repositories on machines that cannot be connected, preventing use of git-fetch / git-push to transport objects and references between the repositories. git-bundle provides an alternate transport mechanism, effectively allowing git-fetch and git-pull to operate using sneakernet transport. git-bundle --create allows the user to create a bundle containing one or more branches or tags, but with specified basis assumed to exist on the target repository. At the receiving end, git-bundle acts like git-fetch-pack, allowing the user to invoke git-fetch or git-pull using the bundle file as the URL. git-fetch and git-ls-remote determine they have a bundle URL by checking that the URL points to a file, but are otherwise unchanged in operation with bundles. Signed-off-by: Mark Levedahl --- Documentation/cmd-list.perl | 1 + Documentation/git-bundle.txt | 146 +++++++++++++++++++++++++++++++++++ Makefile | 3 +- git-bundle.sh | 174 ++++++++++++++++++++++++++++++++++++++++++ git-fetch.sh | 11 ++- git-ls-remote.sh | 7 ++- 6 files changed, 338 insertions(+), 4 deletions(-) mode change 100755 => 100644 Documentation/cmd-list.perl create mode 100644 Documentation/git-bundle.txt create mode 100755 git-bundle.sh diff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl old mode 100755 new mode 100644 index a2d6268..f61c77a --- a/Documentation/cmd-list.perl +++ b/Documentation/cmd-list.perl @@ -70,6 +70,7 @@ git-archive mainporcelain git-bisect mainporcelain git-blame ancillaryinterrogators git-branch mainporcelain +git-bundle mainporcelain git-cat-file plumbinginterrogators git-checkout-index plumbingmanipulators git-checkout mainporcelain diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt new file mode 100644 index 0000000..27db785 --- /dev/null +++ b/Documentation/git-bundle.txt @@ -0,0 +1,146 @@ +git-bundle(1) +============= + +NAME +---- +git-bundle - Move objects and refs by archive + + +SYNOPSIS +-------- +'git-bundle' --create file <--tar tarspec> +'git-bundle' --verify file <--tar tarspec> +'git-bundle' --list-heads file <--tar tarspec> +'git-bundle' --unbundle file <--tar tarspec> + +DESCRIPTION +----------- + +Some workflows require that one or more branches of development on one +machine be replicated on another machine, but the two machines cannot +be directly connected so the interactive git protocols (git, ssh, +rsync, http) cannot be used. This command provides suport for +git-fetch and git-pull to operate by packaging objects and references +in an archive at the originating machine, then importing those into +another repository using gitlink:git-fetch[1] and gitlink:git-pull[1] +after moving the archive by some means (i.e., by sneakernet). As no +direct connection between repositories exists, the user must specify a +basis for the bundle that is held by the destination repository: the +bundle assumes that all objects in the basis are already in the +destination repository. + +OPTIONS +------- + +--create file:: + Used to create a bundle named 'file'. This requires the + git-rev-list arguments to define the bundle contents. + +--verify file:: + Used to check that a bundle file is valid and will apply + cleanly to the current repository. This includes checks on the + bundle format itself as well as checking that the prerequisite + commits exist and are fully linked in the current repository. + git-bundle prints a list of missing commits, if any, and exits + with non-zero status. + +--list-heads file:: + Lists the references defined in the bundle. If followed by a + list of references, only references matching those given are + printed out. + +--unbundle file:: + Passes the objects in the bundle to gitlink:git-index-pack[1] + for storage in the repository, then prints the names of all + defined references. If a reflist is given, only references + matching those in the given list are printed. This command is + really plumbing, intended to be called only by + gitlink:git-fetch[1]. + +git-rev-list args:: + A list of arguments, accepatble to git-rev-parse and + git-rev-list, that specify the specific objects and references + to transport. For example, "master~10..master" causes the + current master reference to be packaged along with all objects + added since its 10th ancestor commit. There is no explicit + limit to the number of references and objects that may be + packaged. + + +reflist:: + A list of references used to limit the references reported as + available. This is principally of use to git-fetch, which + expects to recieve only those references asked for and not + necessarily everything in the pack (in this case, git-bundle is + acting like gitlink:git-fetch-pack[1]). + +tar tarspec:: + + git-bundle uses tar, and requires a tar supporting c, f, and + -O. By default, git-bundle looks for gtar on the path, then for + tar if gtar is not found. This can be overridden by explicitly + defining tarspec, or by defining TAR in the environment. + +SPECIFYING REFERENCES +--------------------- + +git-bundle will only package references that are shown by +git-show-ref: this includes heads, tags, and remote heads. References +such as master~1 cannot be packaged, but are perfectly suitable for +defining the basis. More than one reference may be packaged, and more +than one basis can be specified. The objects packaged are those not +contained in the union of the given bases. Each basis can be +specified explicitly (e.g., ^master~10), or implicitly (e.g., +master~10..master, master --since=10.days.ago). + +It is very important that the basis used be held by the destination. +It is ok to err on the side of conservatism, causing the bundle file +to contain objects already in the destination as these are ignored +when unpacking at the destination. + +EXAMPLE +------- + +Assume two repositories exist as R1 on machine A, and R2 on machine B. +For whatever reason, direct connection between A and B is not allowed, +but we can move data from A to B via some mechanism (CD, email, etc). +We want to update R2 with developments made on branch master in R1. +We set a tag in R1 (lastR2bundle) after the previous such transport, +and move it afterwards to help build the bundle. + +in R1 on A: +$ git-bundle --create mybundle master ^lastR2bundle +$ git tag -f lastR2bundle master + +(move mybundle from A to B by some mechanism) + +in R2 on B: +$ git-bundle --verify mybundle +$ git-fetch mybundle refspec + +where refspec is refInBundle:localRef + + +Also, with something like this in your config: + +[remote "bundle"] + url = /home/me/tmp/file.bdl + fetch = refs/heads/*:refs/remotes/origin/* + +You can first sneakernet the bundle file to ~/tmp/file.bdl and +then these commands: + +$ git ls-remote bundle +$ git fetch bundle +$ git pull bundle + +would treat it as if it is talking with a remote side over the +network. + +Author +------ +Written by Mark Levedahl + +GIT +--- +Part of the gitlink:git[7] suite diff --git a/Makefile b/Makefile index ebecbbd..c6d540e 100644 --- a/Makefile +++ b/Makefile @@ -177,7 +177,8 @@ SCRIPT_SH = \ git-applymbox.sh git-applypatch.sh git-am.sh \ git-merge.sh git-merge-stupid.sh git-merge-octopus.sh \ git-merge-resolve.sh git-merge-ours.sh \ - git-lost-found.sh git-quiltimport.sh + git-lost-found.sh git-quiltimport.sh \ + git-bundle.sh SCRIPT_PERL = \ git-add--interactive.perl \ diff --git a/git-bundle.sh b/git-bundle.sh new file mode 100755 index 0000000..19ac137 --- /dev/null +++ b/git-bundle.sh @@ -0,0 +1,174 @@ +#!/bin/sh +# Basic handler for bundle files to connect repositories via sneakernet. +# Invocation must include action. +# This function can create a bundle or provide information on an existing bundle +# supporting git-fetch, git-pull, and git-ls-remote + +USAGE='[--create bundle ] | +[--verify|--list-heads|--unbundle bundle] <--tar tarspec> + where bundle is the name of the bundle file.' + +verify() { + # Check bundle version + test -f "$bfile" || die "cannot find $bfile" + test "$($TAR -xf ""$bfile"" -O version)" = "v1 git-bundle" || + die "$bfile doesn't look like a v1 bundle file." + + # do fast check, then if any prereqs are missing then go line by line + # to be verbose about the errors + prereqs=$($TAR xf "$bfile" -O prerequisites) + test -z "$prereqs" && return 0 + bad=$(echo "$prereqs" | cut -b-40 | git-rev-list --stdin --not --all 2>&1) + if test -n "$bad" ; then + test "$1" = "--silent" && return 1 + echo "error: $bfile requires the following commits you lack:" + echo "$prereqs" | + while read sha1 comment ; do + missing=$(git-rev-list $sha1 --not --all 2>&1) + test -n "$missing" && echo "$sha1 $comment" + done + exit 1 + fi + return 0 +} + +# list all or just a subset +list_heads() { + if test -z "$*" ; then + $TAR -xf "$bfile" -O references 2>/dev/null || exit 1 + else + ($TAR -xf "$bfile" -O references 2>/dev/null || exit 1) | + while read sha1 ref ; do + for arg in $* ; do + if test "${ref%$arg}" != "$ref" ; then + echo "$sha1 $ref" + break + fi + done + done + fi +} + +SUBDIRECTORY_OK=1 +. git-sh-setup + +args= +action= +while test -n "$1" ; do + case $1 in + --create|--list-heads|--unbundle|--verify) + action=${1#--} + shift + bfile=$1 + test -z "$bfile" && die "$action requires filename";; + --tar=*) + TAR=${1##--tar=};; + --tar) + shift + TAR=$1;; + *) + args="$args $1";; + esac + shift +done +test -z "$action" && die "No action given, what should I do?" + +# what tar to use, prefer gtar, then tar. +if test -z "$TAR" ; then + GTAR=$(which gtar 2>/dev/null) + TAR=${GTAR:-tar} +fi + +case $action in +create) + unknown=$(git-rev-parse --no-revs $args) + test -z "$unknown" || die "unknown option: $unknown" + gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1 + + # find the refs to carry along and get sha1s for each. + refs= + fullrevargs= + for arg in $gitrevargs ; do + #ignore options and basis refs, get full ref name for things + # we will transport rejecting anything ambiguous (e.g., user + # gives master, have heads/master and remotes/origin/master, we + # keep the former). + case "$arg" in + -* | ^*) fullrevargs="$fullrevargs $arg";; + *) ref=$(git-show-ref "$arg") + test "$(echo $ref | wc -w)" = "2" || die "Ambigous reference: $arg +$ref" + fullrevargs="$fullrevargs ${ref#* }" + refs="$refs $ref";; + esac + done + test -z "$refs" && die "No references specified, I don't know what to bundle." + + # git-rev-list cannot determine edge objects if a date restriction is + # given... we do things a slow way if max-age or min-age are given + case "$fullrevargs" in + *--max-age* | *--min-age*) + # get a list of all commits that will be packed along with + # parents of each. A fixed git-rev-list --boundary should + # replace all of this. + echo "Finding prerequisites and commits to bundle..." + commits=$(git-rev-list $fullrevargs) + + # get immediate parents of each commit to include + parents= + for c in $commits ; do + parents="$parents $(git-rev-list --parents --max-count=1 $c | cut -b42-)" + done + parents=$(printf "%s\n" $parents | sort | uniq) + + # factor out what will be in this bundle, the remainder are the + # bundle's prerequisites. double up commits in this as we only + # want things that are only in parents to appear once + prereqs=$(printf "%s\n" $parents $commits $commits | \ + sort | uniq -c | sed -ne 's/^ *1 //p');; + *) + prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');; + esac + + # replace prereqs with annotated version of same + + # create refs and pack + tmp="$GIT_DIR/bundle_tmp$$" + prerequisites="$tmp/prerequisites" + references="$tmp/references" + version="$tmp/version" + pack="$tmp/pack" + trap 'rm -rf "$tmp"' 0 1 2 3 15 + + mkdir "$tmp" && + echo "v1 git-bundle" > "$version" && + touch "$prerequisites" && + (for p in $prereqs ; do + git-rev-list --pretty=one --max-count=1 $p + done) > "$prerequisites" && + git-show-ref $refs > "$references" && + git-rev-list --objects $fullrevargs | cut -b-40 | + git pack-objects --all-progress --stdout > "$pack" && + + # create the tar file, clean up + cd "$tmp" && + tar cf bundle prerequisites references version pack && + cd - && + mv "$tmp/bundle" "$bfile" && + rm -rf "$tmp" + + # done + echo "Created $bfile";; + +verify) + verify && echo "$bfile is ok";; + +list-heads) + list_heads $args;; + +unbundle) + verify --silent || exit 1 + $TAR -xf "$bfile" -O pack | git-index-pack --stdin || + die "error: $bfile has a corrupted pack file" + list_heads $args;; +esac diff --git a/git-fetch.sh b/git-fetch.sh index ca984e7..42cc62f 100755 --- a/git-fetch.sh +++ b/git-fetch.sh @@ -377,8 +377,15 @@ fetch_main () { ( : subshell because we muck with IFS IFS=" $LF" ( - git-fetch-pack --thin $exec $keep $shallow_depth "$remote" $rref || - echo failed "$remote" + if test -f "$remote" ; then + test -n "$shallow_depth" && + die "shallow clone with bundle is not supported" + git-bundle --unbundle "$remote" $rref || + echo failed "$remote" + else + git-fetch-pack --thin $exec $keep $shallow_depth "$remote" $rref || + echo failed "$remote" + fi ) | ( trap ' diff --git a/git-ls-remote.sh b/git-ls-remote.sh index 8ea5c5e..28bb9b8 100755 --- a/git-ls-remote.sh +++ b/git-ls-remote.sh @@ -89,8 +89,13 @@ rsync://* ) ;; * ) - git-peek-remote $exec "$peek_repo" || + if test -f "$peek_repo" ; then + git bundle --list-heads "$peek_repo" || echo "failed slurping" + else + git-peek-remote $exec "$peek_repo" || + echo "failed slurping" + fi ;; esac | sort -t ' ' -k 2 | -- 1.5.0.rc4.375.gd0938-dirty ``` ## Johannes Schindelin, 2007-02-19 01:07 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0702190126220.22628%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <1171838852534-git-send-email-mdl123@verizon.net> ``` Hi, Sorry to be such a PITA, but I really, really think that it is wrong to make a tar dependency here. You said your cygwin has problems with binary files. Could you please try this: $ echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc If it returns anything else than "018", then your setup works differently from mine. I fit returns what I expect it to, then we can use cat directly and do not have to move the tar bloat around (it is inherently block based, so it wastes a lot of space, and it also stores other things we'll never use, like permissions for all files). On Sun, 18 Feb 2007, Mark Levedahl wrote: > diff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl > old mode 100755 > new mode 100644 This is unintended, right? > diff --git a/git-bundle.sh b/git-bundle.sh > new file mode 100755 > index 0000000..19ac137 > --- /dev/null > +++ b/git-bundle.sh > @@ -0,0 +1,174 @@ > +#!/bin/sh > +# Basic handler for bundle files to connect repositories via sneakernet. > +# Invocation must include action. > +# This function can create a bundle or provide information on an existing bundle > +# supporting git-fetch, git-pull, and git-ls-remote > + > +USAGE='[--create bundle ] | > +[--verify|--list-heads|--unbundle bundle] <--tar tarspec> > + where bundle is the name of the bundle file.' We seem to prefer another (more consistent?) way to describe things: If the argument is not an option, we embrace it with "<>", for example: if it is an optional option, we put it in "[]", for example: [--tar ] and if one of several mutual exclusive options has to be passed, we put it in "()" delimited with "|". Also, we put no explanation in the output of "-h", since you should read the man page if you don't know what the options mean. So, it is git-bundle ( --create [...] | --verify | --list-heads | --unbundle ) [--tar ] > +verify() { This function is nicely done, AFAICT! > +list_heads() { > + if test -z "$*" ; then > + $TAR -xf "$bfile" -O references 2>/dev/null || exit 1 > + else > + ($TAR -xf "$bfile" -O references 2>/dev/null || exit 1) | AFAICT this will not work as expected: "()" opens a subshell, and "exit 1" will only exit that subshell. Correct me if I'm wrong. > + while read sha1 ref ; do > + for arg in $* ; do > + if test "${ref%$arg}" != "$ref" ; then > + echo "$sha1 $ref" > + break > + fi > + done > + done > + fi > +} The complexity here is unnecessarily high (O(N*M)), but I guess we cannot do better in shell. Maybe constructing a regexp from the args, and piping the output from tar through a grep would help here. Dunno. Maybe it doesn't matter much in reality, anyway. > +SUBDIRECTORY_OK=1 > +. git-sh-setup We tend to write this part at the very top of a git script. > +while test -n "$1" ; do > + case $1 in > + --create|--list-heads|--unbundle|--verify) > + action=${1#--} > + shift > + bfile=$1 This can contain spaces (you are working on Windows, right? Windows users _love_ spaces in their filenames). > + test -z "$bfile" && die "$action requires filename";; > + --tar=*) > + TAR=${1##--tar=};; > + --tar) > + shift > + TAR=$1;; > + *) > + args="$args $1";; I know you want to mix the --tar option with the rev-list options (which I still find totally confusing, and unnecessary at best), but you _have_ to quote "$1" in the args=... part, then. Even refs can contain spaces these days. > +# what tar to use, prefer gtar, then tar. > +if test -z "$TAR" ; then > + GTAR=$(which gtar 2>/dev/null) Somebody on this list said that "which" should not be relied upon IIRC. I forgot why, but I obey that hint. > + gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1 Here, you rely again on the refs not containing spaces. > + # git-rev-list cannot determine edge objects if a date restriction is > + # given... we do things a slow way if max-age or min-age are given Might make sense to teach max-age about boundary commits instead... > + prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');; Since you are not interested in non-commits at all, why not use "--boundary" instead? > + # create refs and pack > + tmp="$GIT_DIR/bundle_tmp$$" Should we really rely on $GIT_DIR being writable? For unbundling, yes, but for bundling? Given the great confusion with git-status trying to write into $GIT_DIR, and people _demanding_ that it does not do so, I recommend against that. In the following part, refnames must not contain spaces again: > + (for p in $prereqs ; do > + git-rev-list --pretty=one --max-count=1 $p > + done) > "$prerequisites" && > + git-show-ref $refs > "$references" && > + git-rev-list --objects $fullrevargs | cut -b-40 | > + git pack-objects --all-progress --stdout > "$pack" && pack-objects can take rev-list arguments itself these days, using "--revs" and being piped the rev-list arguments. > +verify) > + verify && echo "$bfile is ok";; > + > +list-heads) > + list_heads $args;; I like this abstraction (defining a function, and calling that). Why don't you do it with all commands? Rest looks fine. Given the problems you had with "cat" on Cygwin, and the quoting stuff, I think it might make sense to make this a builtin right away, before the script stage (which is meant to speed things up, not slow them down). Ciao, Dscho ``` ## Junio C Hamano, 2007-02-19 01:49 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: <7vlkiudii3.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vlkiudii3.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <1171838852534-git-send-email-mdl123@verizon.net> ``` Mark Levedahl writes: > Some workflows require use of repositories on machines that cannot be > connected, preventing use of git-fetch / git-push to transport objects > and references between the repositories. Looks much nicer. Thanks. > 6 files changed, 338 insertions(+), 4 deletions(-) > mode change 100755 => 100644 Documentation/cmd-list.perl Hmmmm? > +verify() { > + # Check bundle version > + test -f "$bfile" || die "cannot find $bfile" > + test "$($TAR -xf ""$bfile"" -O version)" = "v1 git-bundle" || > + die "$bfile doesn't look like a v1 bundle file." It makes me feel uneasy to see these double quotes next to each other. Are you sure you got your quoting right (I haven't checked)? Just a minor style issue: traditional single command letter to tar is easier to read if you do not write leading '-'. IOW, "tar xf" is preferred over "tar -xf". > + # do fast check, then if any prereqs are missing then go line by line > + # to be verbose about the errors > + prereqs=$($TAR xf "$bfile" -O prerequisites) > + test -z "$prereqs" && return 0 > + bad=$(echo "$prereqs" | cut -b-40 | git-rev-list --stdin --not --all 2>&1) > + if test -n "$bad" ; then > + test "$1" = "--silent" && return 1 > + echo "error: $bfile requires the following commits you lack:" > + echo "$prereqs" | > + while read sha1 comment ; do > + missing=$(git-rev-list $sha1 --not --all 2>&1) > + test -n "$missing" && echo "$sha1 $comment" > + done > + exit 1 > + fi This part is very hard to read. Maybe your tabstop is not set to 8? Linus rightly said "or something like that"; I do not think the empty output from "git-rev-list --stdin --not --all" is enough. Here, you are making sure that each line of $prereqs are names of objects available in your repository, and well connected to your existing refs. If one of the commits listed in $prereqs does not exist in your repository, I think rev-list exits with non-zero status, with a message to stderr, without spitting out anything to its stdout, so you would need: if bad=$(echo ... | rev-list ...) && test -z "$bad" then : happy else echo 2>&1 "error: you lack these..." fi Incidentally, I think you do not have to squelch error message from rev-list here. Other than that, I think Linus's check is correct and the above fills his "something like that" [*1*]. > +list_heads() { > + if test -z "$*" ; then > + $TAR -xf "$bfile" -O references 2>/dev/null || exit 1 > + else > + ($TAR -xf "$bfile" -O references 2>/dev/null || exit 1) | > + while read sha1 ref ; do > + for arg in $* ; do > + if test "${ref%$arg}" != "$ref" ; then > + echo "$sha1 $ref" > + break > + fi > + done > + done > + fi > +} I suspect that "exit 1" in the upstream of pipe does not have any effect on the outcome -- yup, I just tried... $ (tar xf no-such-file -O references || exit 1) | while read foo; do echo $foo; done ; echo $? 0 $ > +test -z "$action" && die "No action given, what should I do?" > + > +# what tar to use, prefer gtar, then tar. > +if test -z "$TAR" ; then > + GTAR=$(which gtar 2>/dev/null) > + TAR=${GTAR:-tar} > +fi "which" is Ok for interactive use, but never use it in scripts. I've seen "which foo" that says "No foo found." to its standard output without erroring out (I think it was older Solaris). +unbundle) + verify --silent || exit 1 + $TAR -xf "$bfile" -O pack | git-index-pack --stdin || + die "error: $bfile has a corrupted pack file" + list_heads $args;; +esac Hmm. Your verify () shell function silently fails if prereq is not met, so this makes the user of "git --unbundle file.bdl" would get silent failure and left wondering why the bundle does not unpack, doesn't it? > diff --git a/git-fetch.sh b/git-fetch.sh > index ca984e7..42cc62f 100755 > --- a/git-fetch.sh > +++ b/git-fetch.sh > @@ -377,8 +377,15 @@ fetch_main () { Nice. > diff --git a/git-ls-remote.sh b/git-ls-remote.sh > index 8ea5c5e..28bb9b8 100755 > --- a/git-ls-remote.sh > +++ b/git-ls-remote.sh > @@ -89,8 +89,13 @@ rsync://* ) Likewise. [footnote] *1* If somebody wants to know why it works... Suppose the "global" history looked like this: ---A---B---C and your receiving repository earlier had its tip of the branch at commit A. Then the user used git-http-fetch to fetch from a repository that had its tip at commit B. http-fetch (another "commit walker", local-fetch, works the same way) learns that the tip commit is B, downloads commit B, reads what's in there to learn the tree associated with commit B, fetches that tree, reads what's in there to learn the subtrees and blobs in it, ... The user can interrupt this process, at the point after commit B and everything that is needed to complete it was downloaded, but before it actually updated the ref to point at B. Suppose then you created a bundle with B..C to update the tip to C, requiring you to have B. "echo B | rev-list --not --all" would happily say "Oh, B is a commit that is not reachable from any of your refs", but in fact you can safely update the ref that currently point at A (because the user interrupted the earlier fetch) with C after checking C is a fast forward of A. This might make "outputs empty" look like a wrong check. However, the user could have interrupted the earlier http-fetch, at the point after commit B was downloaded but before finishing to download the tree associated with it. A commit walker does not update the ref, so the commit object B will be in your receiving repository but it is not complete (it lacks its tree object). Even in this case "echo B | rev-list --not --all" would say "B is ahead of your repository". That's why "outputs empty" is a good check. Before somebody wonders further. An alternative good check is to see if "echo B | rev-list --objects --not --all" does _not_ fail, and all of the objects listed in its output are available in the receiving repository. This will solve the apparent inefficiency in the case of killing http-fetch after it downloaded all the necessary objects before it updated the ref, which I described earlier. But I think that the window for that to happen is very small and if you killed a commit walker it is much more likely that you lack some objects, so doing this alternative check is optimizing for a wrong case. ``` ## Junio C Hamano, 2007-02-19 01:50 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: <7v7iuedif9.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v7iuedif9.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Johannes Schindelin writes: > Sorry to be such a PITA, but I really, really think that it is wrong to > make a tar dependency here. You said your cygwin has problems with binary > files. Could you please try this: > > $ echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc It might be just me, but "echo -e" makes me feel much more uneasy than explicitly saying "Sorry, we require GNU tar in this little corner of the system". >> + bfile=$1 > > This can contain spaces (you are working on Windows, right? Windows users > _love_ spaces in their filenames). Which is fine. Try var1=$var2 with something with SP or TAB in var2 and later try running $ echo "<$var1>" to see what happens. >> + gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1 > > Here, you rely again on the refs not containing spaces. Which probably is fine, as refs cannot contain spaces ;-). >> + # git-rev-list cannot determine edge objects if a date restriction is >> + # given... we do things a slow way if max-age or min-age are given > > Might make sense to teach max-age about boundary commits instead... You are talking about a rather extensive change, but could be done. But that is a separate issue, I think. ``` ## Johannes Schindelin, 2007-02-19 02:02 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0702190257250.22628%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <7v7iuedif9.fsf@assigned-by-dhcp.cox.net> ``` Hi, On Sun, 18 Feb 2007, Junio C Hamano wrote: > Johannes Schindelin writes: > > > Sorry to be such a PITA, but I really, really think that it is wrong to > > make a tar dependency here. You said your cygwin has problems with binary > > files. Could you please try this: > > > > $ echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc > > It might be just me, but "echo -e" makes me feel much more > uneasy than explicitly saying "Sorry, we require GNU tar in > this little corner of the system". Hey, this was meant as a _test_ on why it fails for Mark, not as the end result! > >> + bfile=$1 > > > > This can contain spaces (you are working on Windows, right? Windows users > > _love_ spaces in their filenames). > > Which is fine. Try var1=$var2 with something with SP or TAB in var2 > and later try running > > $ echo "<$var1>" > > to see what happens. Okay, I did not know that. > >> + gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1 > > > > Here, you rely again on the refs not containing spaces. > > Which probably is fine, as refs cannot contain spaces ;-). Ah! As we allow so many things in refnames, I assumed that spaces are allowed also. > >> + # git-rev-list cannot determine edge objects if a date restriction is > >> + # given... we do things a slow way if max-age or min-age are given > > > > Might make sense to teach max-age about boundary commits instead... > > You are talking about a rather extensive change, but could be > done. But that is a separate issue, I think. I don't think it's so hard. Ciao, Dscho ``` ## Shawn O. Pearce, 2007-02-19 07:56 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: <20070219075654.GB30030@spearce.org> URL: https://gitlist.dev/e/20070219075654.GB30030%40spearce.org In-Reply-To: ``` Johannes Schindelin wrote: > On Sun, 18 Feb 2007, Mark Levedahl wrote: > > + gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1 > > Here, you rely again on the refs not containing spaces. They cannot. OK, they can if you avoid any of the Git code: git rev-parse HEAD >".git/refs/heads/ref with space" but a lot of things downstream may get annoyed by doing this. Spaces are not permitted in ref names. > > + # create refs and pack > > + tmp="$GIT_DIR/bundle_tmp$$" > > Should we really rely on $GIT_DIR being writable? For unbundling, yes, but > for bundling? Given the great confusion with git-status trying to write > into $GIT_DIR, and people _demanding_ that it does not do so, I recommend > against that. I agree. For an application like bundle creation, $GIT_DIR should be strictly read-only. There is no reason for this application to need to create temporary files, as the bundle can be streamed to stdout (or optionally to a file supplied by the user on the command line). > > + git-rev-list --objects $fullrevargs | cut -b-40 | > > + git pack-objects --all-progress --stdout > "$pack" && > > pack-objects can take rev-list arguments itself these days, using "--revs" > and being piped the rev-list arguments. Uh, actually it can't. I thought it could too. It doesn't. It can only take a ref name or --not. I'm actually not sure why we limited it like that, but we did... It may be a good idea to change that restriction to not exist. -- Shawn. ``` ## Mark Levedahl, 2007-02-19 13:29 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: URL: https://gitlist.dev/e/erc8nm%24feg%241%40sea.gmane.org In-Reply-To: ``` "Johannes Schindelin" wrote in message news:Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de... > Hi, > > Sorry to be such a PITA, but I really, really think that it is wrong > to > make a tar dependency here. You said your cygwin has problems with > binary > files. Could you please try this: > > $ echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc > Same result on Cygwin and FC6: ~>echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc 0 1 8 > If it returns anything else than "018", then your setup > works differently from mine. I fit returns what I expect it to, then > we > can use cat directly and do not have to move the tar bloat around (it > is > inherently block based, so it wastes a lot of space, and it also > stores > other things we'll never use, like permissions for all files). To repeat an earlier message, I tried the following: cat bundle | ( while read
do done git-pack-index --stdin ) and it worked, MOST of the time, most being the important word here. My memory is already fuzzy about the failure rate, but it was something like 2 of 7 bundle files I passed through resulting in git-pack-index reporting a corrupted pack file: when I manually edited out the header stuff and passed the remains of the file directly to git-pack-index things were fine. The same experiment worked perfectly on my FC6 system, so it was definitely some part of Cygwin, not the script. If git is worried about tickling "which" on ancient Solaris, I think it is a bit out of balance to simultaneously require a not yet written version of Cygwin (that will get fixed after someone has time to reduce this problem to a simple testcase and push back to the Cygwin maintainers). A bundle file *is* an archive. We can use an existing archiver or write a new one. I don't really favor writing one but that is personal preference for not reinventing the wheel, nothing more. However, given Cgywin's problems an archiver written in shell isn't going to work for binary files. We could uuencode / uudecode the pack files and make bundles pure ascii text. Or, we could use tar. Of course, rewriting this as a builtin offers the ability to write a purpose built archiver as well, but I was trying to avoid that. Has anyone considered how much easier all of git would be if it were written in c + python, rather than c + every variant of shell + core utils + non-core utils known to mankind since the dawn of unix? Mark ``` ## Johannes Schindelin, 2007-02-19 15:03 Subject: Re: [PATCH] Add git-bundle: move objects and references by archive. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0702191557580.22628%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: ``` Hi, On Mon, 19 Feb 2007, Mark Levedahl wrote: > "Johannes Schindelin" wrote in message > news:Pine.LNX.4.63.0702190126220.22628@wbgn013.biozentrum.uni-wuerzburg.de... > > Hi, > > > > Sorry to be such a PITA, but I really, really think that it is wrong to > > make a tar dependency here. You said your cygwin has problems with binary > > files. Could you please try this: > > > > $ echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc > > > Same result on Cygwin and FC6: > ~>echo -ne '\x1a\x1b\x15\x10\0abc' | cat | wc > 0 1 8 Okay, so I really would like to have the bundle file from this experiment: > To repeat an earlier message, I tried the following: > > cat bundle | ( > while read
do > > done > git-pack-index --stdin > ) because I cannot see how the aforementioned "cat" can succeed, while pack-index --stdin cannot. > A bundle file *is* an archive. But the archive in the bundle really is the pack. You don't need any other archive there. You don't need to know the name of the prerequisites file, for example, if that information is in the header to begin with. > Has anyone considered how much easier all > of git would be if it were written in c + python, rather than c + every > variant of shell + core utils + non-core utils known to mankind since > the dawn of unix? You realize that Python can be a PITA, a real PITA, when it comes to non-Linux systems? Just look at the fun we had with subprocess. It is included in Python 2.4, _works_ with Python 2.3, but _not_ Python with 2.2. If you now say "well, just upgrade!" then I SHOUT at you. There is _no_ excuse forcing users to upgrade when the failure is on _your_ part. Ciao, Dscho ```