threads / patch / 27656

v2, 20 partsremote-helper improvements

Subject: [PATCH v2 v2 00/20] remote-helper improvements

## tl;dr

44 messages between Jun 19, 2011 and Jul 4, 2011. Diffs are folded; open one to read it.

replies: 43people: 5as markdown or json

Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · lore
This is v2 of the series, mostly minor changes.
Jeff King (6):
  transport-helper: fix minor leak in push_refs_with_export
  t5800: factor out some ref tests
  t5800: document some non-functional parts of remote helpers
  teach remote-testgit to import non-HEAD refs
  transport-helper: don't feed bogus refs to export push
  git_remote_helpers: push all refs during a non-local export
Sverre Rabbelier (14):
  t5800: use skip_all instead of prereq
  remote-curl: accept empty line as terminator
  git-remote-testgit: only push for non-local repositories
  git-remote-testgit: fix error handling
  fast-import: introduce 'done' command
  fast-export: support done feature
  transport-helper: factor out push_update_refs_status
  transport-helper: check status code of finish_command
  transport-helper: use the new done feature where possible
  transport-helper: update ref status after push with export
  transport-helper: change import semantics
  transport-helper: export is no longer always the last command
  transport-helper: Use capname for gitdir capability too
  transport-helper: implement marks location as capability
 Documentation/git-fast-export.txt   |    4 +
 Documentation/git-fast-import.txt   |   25 ++++
 builtin/fast-export.c               |    9 ++
 fast-import.c                       |    8 ++
 git-remote-testgit.py               |   57 ++++++---
 git_remote_helpers/git/exporter.py  |   15 ++-
 git_remote_helpers/git/importer.py  |   32 +++++-
 git_remote_helpers/git/non_local.py |   20 +--
 git_remote_helpers/git/repo.py      |    7 +-
 git_remote_helpers/util.py          |   66 ++++++++++
 remote-curl.c                       |    3 +
 t/t5800-remote-helpers.sh           |   93 +++++++++++---
 t/t9300-fast-import.sh              |   42 ++++++
 transport-helper.c                  |  238 ++++++++++++++++++----------------
 14 files changed, 445 insertions(+), 174 deletions(-)
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 01/20] transport-helper: fix minor leak in push_refs_with_export

From: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 transport-helper.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
Show changes to transport-helper.c +1 −0
diff --git a/transport-helper.c b/transport-helper.c
index 660147f..b560b64 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -728,6 +728,7 @@ static int push_refs_with_export(struct transport *transport,
 			strbuf_addf(&buf, "^%s", private);
 			string_list_append(&revlist_args, strbuf_detach(&buf, NULL));
 		}
+		free(private);
 
 		string_list_append(&revlist_args, ref->name);
 
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 02/20] t5800: factor out some ref tests

From: Jeff King <peff@peff.net>

These are a little hard to read, and I'm about to add more just like them. Plus the failure output is nicer if we use test_cmp than a comparison with "test".

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 t/t5800-remote-helpers.sh |   12 ++++++++----
 1 files changed, 8 insertions(+), 4 deletions(-)
Show changes to t/t5800-remote-helpers.sh +8 −4
diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index 1fb6380..3a37ad0 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -17,6 +17,12 @@ then
 	test_set_prereq PYTHON_24
 fi
 
+compare_refs() {
+	git --git-dir="$1/.git" rev-parse --verify $2 >expect &&
+	git --git-dir="$3/.git" rev-parse --verify $4 >actual &&
+	test_cmp expect actual
+}
+
 test_expect_success PYTHON_24 'setup repository' '
 	git init --bare server/.git &&
 	git clone server public &&
@@ -59,8 +65,7 @@ test_expect_success PYTHON_24 'pushing to local repo' '
 	echo content >>file &&
 	git commit -a -m three &&
 	git push) &&
-	HEAD=$(git --git-dir=localclone/.git rev-parse --verify HEAD) &&
-	test $HEAD = $(git --git-dir=server/.git rev-parse --verify HEAD)
+	compare_refs localclone HEAD server HEAD
 '
 
 test_expect_success PYTHON_24 'synch with changes from localclone' '
@@ -73,8 +78,7 @@ test_expect_success PYTHON_24 'pushing remote local repo' '
 	echo content >>file &&
 	git commit -a -m four &&
 	git push) &&
-	HEAD=$(git --git-dir=clone/.git rev-parse --verify HEAD) &&
-	test $HEAD = $(git --git-dir=server/.git rev-parse --verify HEAD)
+	compare_refs clone HEAD server HEAD
 '
 
 test_done
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 03/20] t5800: use skip_all instead of prereq

All tests require python 2.4 or higher.
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  New in this series.
 t/t5800-remote-helpers.sh |   34 +++++++++++++++++++---------------
 1 files changed, 19 insertions(+), 15 deletions(-)
Show changes to t/t5800-remote-helpers.sh +19 −15
diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index 3a37ad0..f6796e3 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -7,15 +7,19 @@ test_description='Test remote-helper import and export commands'
 
 . ./test-lib.sh
 
-if test_have_prereq PYTHON && "$PYTHON_PATH" -c '
+if ! test_have_prereq PYTHON ; then
+	skip_all='skipping git-remote-hg tests, python not available'
+	test_done
+fi
+
+"$PYTHON_PATH" -c '
 import sys
 if sys.hexversion < 0x02040000:
     sys.exit(1)
-'
-then
-    # Requires Python 2.4 or newer
-	test_set_prereq PYTHON_24
-fi
+' || {
+	skip_all='skipping git-remote-hg tests, python version < 2.4'
+	test_done
+}
 
 compare_refs() {
 	git --git-dir="$1/.git" rev-parse --verify $2 >expect &&
@@ -23,7 +27,7 @@ compare_refs() {
 	test_cmp expect actual
 }
 
-test_expect_success PYTHON_24 'setup repository' '
+test_expect_success 'setup repository' '
 	git init --bare server/.git &&
 	git clone server public &&
 	(cd public &&
@@ -33,34 +37,34 @@ test_expect_success PYTHON_24 'setup repository' '
 	 git push origin master)
 '
 
-test_expect_success PYTHON_24 'cloning from local repo' '
+test_expect_success 'cloning from local repo' '
 	git clone "testgit::${PWD}/server" localclone &&
 	test_cmp public/file localclone/file
 '
 
-test_expect_success PYTHON_24 'cloning from remote repo' '
+test_expect_success 'cloning from remote repo' '
 	git clone "testgit::file://${PWD}/server" clone &&
 	test_cmp public/file clone/file
 '
 
-test_expect_success PYTHON_24 'create new commit on remote' '
+test_expect_success 'create new commit on remote' '
 	(cd public &&
 	 echo content >>file &&
 	 git commit -a -m two &&
 	 git push)
 '
 
-test_expect_success PYTHON_24 'pulling from local repo' '
+test_expect_success 'pulling from local repo' '
 	(cd localclone && git pull) &&
 	test_cmp public/file localclone/file
 '
 
-test_expect_success PYTHON_24 'pulling from remote remote' '
+test_expect_success 'pulling from remote remote' '
 	(cd clone && git pull) &&
 	test_cmp public/file clone/file
 '
 
-test_expect_success PYTHON_24 'pushing to local repo' '
+test_expect_success 'pushing to local repo' '
 	(cd localclone &&
 	echo content >>file &&
 	git commit -a -m three &&
@@ -68,12 +72,12 @@ test_expect_success PYTHON_24 'pushing to local repo' '
 	compare_refs localclone HEAD server HEAD
 '
 
-test_expect_success PYTHON_24 'synch with changes from localclone' '
+test_expect_success 'synch with changes from localclone' '
 	(cd clone &&
 	 git pull)
 '
 
-test_expect_success PYTHON_24 'pushing remote local repo' '
+test_expect_success 'pushing remote local repo' '
 	(cd clone &&
 	echo content >>file &&
 	git commit -a -m four &&
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 04/20] t5800: document some non-functional parts of remote helpers

From: Jeff King <peff@peff.net>

These are all things one might expect to work in a helper that is capable of handling multiple branches (which our testgit helper in theory should be able to do, as it is backed by git). All of these bugs are specific to the import/export codepaths, so they don't affect helpers like git-remote-curl that use fetch/push commands.

The first and fourth tests are about fetching and pushing new refs, and demonstrate bugs in the git_remote_helpers library (so they would be most likely to impact helpers for other VCSs which import/export git).

The second test is about importing multiple refs; it demonstrates a bug in git-remote-testgit, which is mostly for exercising the test code. Therefore it probably doesn't affect anyone in practice.

The third test demonstrates a bug in git's side of the helper code when the upstream has added refs that we do not have locally. This could impact git users who use remote helpers to access foreign VCSs.

All of those bugs have fixes later in this series.

The fifth test is the most complex, and does not have a fix in this series. It tests pushing a ref via the export mechanism to a new name on the remote side (i.e., "git push $remote old:new").

The problem is that we push all of the work of generating the export stream onto fast-export, but we have no way of communicating to fast-export that this name mapping is happening. So we tell fast-export to generate a stream with the commits for "old", but we can't tell it to label them all as "new".

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Culled the part of the commit message that is discussion of a
  possible implementation, but kept the parts describing the tests
  and why the last one is failing.
 t/t5800-remote-helpers.sh |   47 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 47 insertions(+), 0 deletions(-)
Show changes to t/t5800-remote-helpers.sh +47 −0
diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index f6796e3..9db8ca8 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -85,4 +85,51 @@ test_expect_success 'pushing remote local repo' '
 	compare_refs clone HEAD server HEAD
 '
 
+test_expect_failure 'fetch new branch' '
+	(cd public &&
+	 git checkout -b new &&
+	 echo content >>file &&
+	 git commit -a -m five &&
+	 git push origin new
+	) &&
+	(cd localclone &&
+	 git fetch origin new
+	) &&
+	compare_refs public HEAD localclone FETCH_HEAD
+'
+
+test_expect_failure 'fetch multiple branches' '
+	(cd localclone &&
+	 git fetch
+	) &&
+	compare_refs server master localclone refs/remotes/origin/master &&
+	compare_refs server new localclone refs/remotes/origin/new
+'
+
+test_expect_failure 'push when remote has extra refs' '
+	(cd clone &&
+	 echo content >>file &&
+	 git commit -a -m six &&
+	 git push
+	) &&
+	compare_refs clone master server master
+'
+
+test_expect_failure 'push new branch by name' '
+	(cd clone &&
+	 git checkout -b new-name  &&
+	 echo content >>file &&
+	 git commit -a -m seven &&
+	 git push origin new-name
+	) &&
+	compare_refs clone HEAD server refs/heads/new-name
+'
+
+test_expect_failure 'push new branch with old:new refspec' '
+	(cd clone &&
+	 git push origin new-name:new-refspec
+	) &&
+	compare_refs clone HEAD server refs/heads/new-refspec
+'
+
 test_done
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 19, 2011, 22:02 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 04/20] t5800: document some non-functional parts of remote helpers

Sverre Rabbelier wrote:
>   Culled the part of the commit message that is discussion of a
>   possible implementation, but kept the parts describing the tests
>   and why the last one is failing.

Looks good. What happened to the extra tests (e.g., push tag) mentioned in the last round?

Sverre Rabbelier· Jul 4, 2011, 11:19 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 04/20] t5800: document some non-functional parts of remote helpers

Heya,
On Mon, Jun 20, 2011 at 00:02, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Looks good.  What happened to the extra tests (e.g., push tag)
> mentioned in the last round?

Do you think they should be added as test_expect_fail targets? I didn't add them because they didn't work, but I suppose it might make sense to at least document these missing features.

-- 
Cheers,

Sverre Rabbelier
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 05/20] teach remote-testgit to import non-HEAD refs

From: Jeff King <peff@peff.net>

Upon receiving an "import" command, the testgit remote helper would ignore the ref asked for by git and generate a fast-export stream based on HEAD. Instead, we should actually give git the ref it asked for.

This requires adding a new parameter to the export_repo method in the remote-helpers python library, which may be used by code outside of git.git. We use a default parameter so that callers without the new parameter will get the same behavior as before.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Fixed spurious appending of "HEAD" noticed by Junio.
 git-remote-testgit.py              |    2 +-
 git_remote_helpers/git/exporter.py |    9 +++++++--
 t/t5800-remote-helpers.sh          |    2 +-
 3 files changed, 9 insertions(+), 4 deletions(-)
Show changes to 3 files +9 −4

git-remote-testgit.py, git_remote_helpers/git/exporter.py, t/t5800-remote-helpers.sh

diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index df9d512..e4a99a3 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -122,7 +122,7 @@ def do_import(repo, args):
         die("Need gitdir to import")
 
     repo = update_local_repo(repo)
-    repo.exporter.export_repo(repo.gitdir)
+    repo.exporter.export_repo(repo.gitdir, args)
 
 
 def do_export(repo, args):
diff --git a/git_remote_helpers/git/exporter.py b/git_remote_helpers/git/exporter.py
index f40f9d6..bc39163 100644
--- a/git_remote_helpers/git/exporter.py
+++ b/git_remote_helpers/git/exporter.py
@@ -15,7 +15,7 @@ class GitExporter(object):
 
         self.repo = repo
 
-    def export_repo(self, base):
+    def export_repo(self, base, refs=None):
         """Exports a fast-export stream for the given directory.
 
         Simply delegates to git fast-epxort and pipes it through sed
@@ -23,8 +23,13 @@ class GitExporter(object):
         default refs/heads. This is to demonstrate how the export
         data can be stored under it's own ref (using the refspec
         capability).
+
+        If None, refs defaults to ["HEAD"].
         """
 
+        if not refs:
+            refs = ["HEAD"]
+
         dirname = self.repo.get_base_path(base)
         path = os.path.abspath(os.path.join(dirname, 'testgit.marks'))
 
@@ -42,7 +47,7 @@ class GitExporter(object):
         if os.path.exists(path):
             args.append("--import-marks=" + path)
 
-        args.append("HEAD")
+        args.extend(refs)
 
         p1 = subprocess.Popen(args, stdout=subprocess.PIPE)
 
diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index 9db8ca8..ca115cc 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -85,7 +85,7 @@ test_expect_success 'pushing remote local repo' '
 	compare_refs clone HEAD server HEAD
 '
 
-test_expect_failure 'fetch new branch' '
+test_expect_success 'fetch new branch' '
 	(cd public &&
 	 git checkout -b new &&
 	 echo content >>file &&
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 06/20] transport-helper: don't feed bogus refs to export push

From: Jeff King <peff@peff.net>

When we want to push to a remote helper that has the "export" capability, we collect all of the refs we want to push and then feed them to fast-export.

However, the list of refs is actually a list of remote refs, not local refs. The mapped local refs are included via the peer_ref pointer. So when we add an argument to our fast-export command line, we must be sure to use the local peer_ref name (and if there is no local name, it is because we are not actually sending that ref, or we may not even have the ref at all).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 t/t5800-remote-helpers.sh |    2 +-
 transport-helper.c        |    3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
Show changes to 2 files +3 −2

t/t5800-remote-helpers.sh, transport-helper.c

diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index ca115cc..ceb0010 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -106,7 +106,7 @@ test_expect_failure 'fetch multiple branches' '
 	compare_refs server new localclone refs/remotes/origin/new
 '
 
-test_expect_failure 'push when remote has extra refs' '
+test_expect_success 'push when remote has extra refs' '
 	(cd clone &&
 	 echo content >>file &&
 	 git commit -a -m six &&
diff --git a/transport-helper.c b/transport-helper.c
index b560b64..34d18aa 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -730,7 +730,8 @@ static int push_refs_with_export(struct transport *transport,
 		}
 		free(private);
 
-		string_list_append(&revlist_args, ref->name);
+		if (ref->peer_ref)
+			string_list_append(&revlist_args, ref->peer_ref->name);
 
 	}
 
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 07/20] git_remote_helpers: push all refs during a non-local export

From: Jeff King <peff@peff.net>

When a remote helper exports to a non-local git repo, the steps are roughly:

  1. fast-export into a local staging area; the set of
     interesting refs is defined by what is in the fast-export
     stream
  2. git push from the staging area to the non-local repo

In the second step, we should explicitly push all refs, not just matching ones. This will let us push refs that do not yet exist in the remote repo.

This does not deal with forced (not-fast-forward) pushes.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Added last line to the commit message.
 git_remote_helpers/git/non_local.py |    2 +-
 t/t5800-remote-helpers.sh           |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Show changes to 2 files +2 −2

git_remote_helpers/git/non_local.py, t/t5800-remote-helpers.sh

diff --git a/git_remote_helpers/git/non_local.py b/git_remote_helpers/git/non_local.py
index f27389b..c53e074 100644
--- a/git_remote_helpers/git/non_local.py
+++ b/git_remote_helpers/git/non_local.py
@@ -63,7 +63,7 @@ class NonLocalGit(object):
         if not os.path.exists(path):
             die("could not find repo at %s", path)
 
-        args = ["git", "--git-dir=" + path, "push", "--quiet", self.repo.gitpath]
+        args = ["git", "--git-dir=" + path, "push", "--quiet", self.repo.gitpath, "--all"]
         child = subprocess.Popen(args)
         if child.wait() != 0:
             raise CalledProcessError
diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index ceb0010..12f471c 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -115,7 +115,7 @@ test_expect_success 'push when remote has extra refs' '
 	compare_refs clone master server master
 '
 
-test_expect_failure 'push new branch by name' '
+test_expect_success 'push new branch by name' '
 	(cd clone &&
 	 git checkout -b new-name  &&
 	 echo content >>file &&
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 08/20] remote-curl: accept empty line as terminator

This went unnoticed because the transport helper infrastructore did not check the return value of the helper, nor did the helper print anything before exiting.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 remote-curl.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
Show changes to remote-curl.c +3 −0
diff --git a/remote-curl.c b/remote-curl.c
index 17d8a9b..7e8d50f 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -857,6 +857,8 @@ int main(int argc, const char **argv)
 	do {
 		if (strbuf_getline(&buf, stdin, '\n') == EOF)
 			break;
+		if (buf.len == 0)
+			break;
 		if (!prefixcmp(buf.buf, "fetch ")) {
 			if (nongit)
 				die("Fetch attempted without a local repo");
@@ -895,6 +897,7 @@ int main(int argc, const char **argv)
 			printf("\n");
 			fflush(stdout);
 		} else {
+			fprintf(stderr, "Unknown command '%s'\n", buf.buf);
 			return 1;
 		}
 		strbuf_reset(&buf);
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 19, 2011, 22:42 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 08/20] remote-curl: accept empty line as terminator

Sverre Rabbelier wrote:
Show 8 quoted lines
> --- a/remote-curl.c
> +++ b/remote-curl.c
> @@ -857,6 +857,8 @@ int main(int argc, const char **argv)
>  	do {
>  		if (strbuf_getline(&buf, stdin, '\n') == EOF)
>  			break;
> +		if (buf.len == 0)
> +			break;
Thanks.  I wonder if that first "if" should be something like
	if (strbuf_getline(&buf, stdin, '\n') == EOF) {
		if (ferror(stdin))
			fprintf(stderr, "Error reading command stream\n");
		else
			fprintf(stderr, "Unexpected end of command stream\n");
		return 1;
	}
to catch I/O errors (e.g., the transport-helper exiting early).
Sverre Rabbelier· Jul 4, 2011, 14:11 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 08/20] remote-curl: accept empty line as terminator

Heya,
On Mon, Jun 20, 2011 at 00:42, Jonathan Nieder <jrnieder@gmail.com> wrote:
Show 11 quoted lines
> Thanks.  I wonder if that first "if" should be something like
>
>        if (strbuf_getline(&buf, stdin, '\n') == EOF) {
>                if (ferror(stdin))
>                        fprintf(stderr, "Error reading command stream\n");
>                else
>                        fprintf(stderr, "Unexpected end of command stream\n");
>                return 1;
>        }
>
> to catch I/O errors (e.g., the transport-helper exiting early).
Good idea, done. Tests still pass too ;).
-- 
Cheers,

Sverre Rabbelier
Dmitry Ivankov· Jun 20, 2011, 02:35 UTC · re: Sverre Rabbelier · lore

[PATCH v2 08/20] remote-curl: accept empty line as terminator

As a next step, won't it be better to use "exit\n" or "quit\n" as a terminator? I don't see a convention of terminating on a blank line in docs, only on EOF. Also I can imagine a blank line being read in a case of communication error, I mean a helper can miss the error and see "exit normally" terminator.

Jonathan Nieder· Jun 20, 2011, 07:55 UTC · re: Dmitry Ivankov · lore

Re: [PATCH v2 08/20] remote-curl: accept empty line as terminator

Hi,
Dmitry Ivankov wrote:
> As a next step, won't it be better to use "exit\n" or "quit\n"
> as a terminator?

If we were starting over, a "done" command (or "quit" or some other self-descriptive name) would be a no-brainer. Unfortunately, there are existing remote helpers out there --- based on a web search:

 - mediawiki
 - ccnx
 - couch
 - cvs
 - bzr
 - hg

Presumably most already ignore a final blank line and do not check for "quit".

One possibility would be to emit "done" instead of a blank line if the remote helper has declared a "done" capability. But what does it buy?

Side note: a "done" capability doesn't sound like a bad idea, though, for another reason. The transport-helper could tell fast-import to expect a "done" command at the end when importing from a remote helper declaring it, to catch situations in which the pipe prematurely closes (for example, because the remote helper has segfaulted).

> I don't see a convention of terminating on a
> blank line in docs,
Yes, this would be nice to document.
> only on EOF.  Also I can imagine a blank
> line being read in a case of communication error

A spurious NL, NL, EOF sequence does not sound likely to me. If the command stream is passing through a noisy channel, there are worse corruptions to worry about (e.g., fetching to the wrong ref).

Junio C Hamano· Jun 20, 2011, 19:41 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 08/20] remote-curl: accept empty line as terminator

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 17 quoted lines
> Side note: a "done" capability doesn't sound like a bad idea, though,
> for another reason.  The transport-helper could tell fast-import to
> expect a "done" command at the end when importing from a remote helper
> declaring it, to catch situations in which the pipe prematurely closes
> (for example, because the remote helper has segfaulted).
>
>> I don't see a convention of terminating on a
>> blank line in docs,
>
> Yes, this would be nice to document.
>
>> only on EOF.  Also I can imagine a blank
>> line being read in a case of communication error
>
> A spurious NL, NL, EOF sequence does not sound likely to me.  If the
> command stream is passing through a noisy channel, there are worse
> corruptions to worry about (e.g., fetching to the wrong ref).

I think everything you said in this message makes sense, especially the part about "premature EOF detection".

The spurious LF is an issue if you are trying to drive the backend by hand for testing, but otherwise probably not.

Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 09/20] git-remote-testgit: only push for non-local repositories

Trying to push for local repositories will fail since there is no local checkout in .git/info/... to push from as that is only used for non-local repositories (local repositories are pushed to directly).

This went unnoticed because the transport helper infrastructure does not check the return value of the helper.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 git-remote-testgit.py |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
Show changes to git-remote-testgit.py +3 −1
diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index e4a99a3..9658355 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -147,7 +147,9 @@ def do_export(repo, args):
 
     update_local_repo(repo)
     repo.importer.do_import(repo.gitdir)
-    repo.non_local.push(repo.gitdir)
+
+    if not repo.local:
+        repo.non_local.push(repo.gitdir)
 
 
 def do_gitdir(repo, args):
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 10/20] git-remote-testgit: fix error handling

If fast-export did not complete successfully the error handling code itself would error out.

This was broken in commit 23b093ee0 (Brandon Casey, Wed Jun 9 2010, Remove python 2.5'isms). Revert that commit an introduce our own copy of check_call in util.py instead.

Tested by changing 'if retcode' to 'if not retcode' temporarily.
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  This was broken in many places, not just the one place I fixed it
  in the last version of this commit.
 git_remote_helpers/git/exporter.py  |    6 +++---
 git_remote_helpers/git/importer.py  |    6 +++---
 git_remote_helpers/git/non_local.py |   18 +++++-------------
 git_remote_helpers/git/repo.py      |    7 ++++---
 git_remote_helpers/util.py          |   32 ++++++++++++++++++++++++++++++++
 5 files changed, 47 insertions(+), 22 deletions(-)
Show changes to 5 files +47 −22

git_remote_helpers/git/exporter.py, git_remote_helpers/git/importer.py, git_remote_helpers/git/non_local.py, git_remote_helpers/git/repo.py, git_remote_helpers/util.py

diff --git a/git_remote_helpers/git/exporter.py b/git_remote_helpers/git/exporter.py
index bc39163..9ee5f96 100644
--- a/git_remote_helpers/git/exporter.py
+++ b/git_remote_helpers/git/exporter.py
@@ -2,6 +2,8 @@ import os
 import subprocess
 import sys
 
+from git_remote_helpers.util import check_call
+
 
 class GitExporter(object):
     """An exporter for testgit repositories.
@@ -53,6 +55,4 @@ class GitExporter(object):
 
         args = ["sed", "s_refs/heads/_" + self.repo.prefix + "_g"]
 
-        child = subprocess.Popen(args, stdin=p1.stdout)
-        if child.wait() != 0:
-            raise CalledProcessError
+        check_call(args, stdin=p1.stdout)
diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py
index 70a7127..02a719a 100644
--- a/git_remote_helpers/git/importer.py
+++ b/git_remote_helpers/git/importer.py
@@ -1,6 +1,8 @@
 import os
 import subprocess
 
+from git_remote_helpers.util import check_call
+
 
 class GitImporter(object):
     """An importer for testgit repositories.
@@ -35,6 +37,4 @@ class GitImporter(object):
         if os.path.exists(path):
             args.append("--import-marks=" + path)
 
-        child = subprocess.Popen(args)
-        if child.wait() != 0:
-            raise CalledProcessError
+        check_call(args)
diff --git a/git_remote_helpers/git/non_local.py b/git_remote_helpers/git/non_local.py
index c53e074..e700250 100644
--- a/git_remote_helpers/git/non_local.py
+++ b/git_remote_helpers/git/non_local.py
@@ -1,7 +1,7 @@
 import os
 import subprocess
 
-from git_remote_helpers.util import die, warn
+from git_remote_helpers.util import check_call, die, warn
 
 
 class NonLocalGit(object):
@@ -29,9 +29,7 @@ class NonLocalGit(object):
         os.makedirs(path)
         args = ["git", "clone", "--bare", "--quiet", self.repo.gitpath, path]
 
-        child = subprocess.Popen(args)
-        if child.wait() != 0:
-            raise CalledProcessError
+        check_call(args)
 
         return path
 
@@ -45,14 +43,10 @@ class NonLocalGit(object):
             die("could not find repo at %s", path)
 
         args = ["git", "--git-dir=" + path, "fetch", "--quiet", self.repo.gitpath]
-        child = subprocess.Popen(args)
-        if child.wait() != 0:
-            raise CalledProcessError
+        check_call(args)
 
         args = ["git", "--git-dir=" + path, "update-ref", "refs/heads/master", "FETCH_HEAD"]
-        child = subprocess.Popen(args)
-        if child.wait() != 0:
-            raise CalledProcessError
+        child = check_call(args)
 
     def push(self, base):
         """Pushes from the non-local repo to base.
@@ -64,6 +58,4 @@ class NonLocalGit(object):
             die("could not find repo at %s", path)
 
         args = ["git", "--git-dir=" + path, "push", "--quiet", self.repo.gitpath, "--all"]
-        child = subprocess.Popen(args)
-        if child.wait() != 0:
-            raise CalledProcessError
+        child = check_call(args)
diff --git a/git_remote_helpers/git/repo.py b/git_remote_helpers/git/repo.py
index 58e1cdb..acbf8d7 100644
--- a/git_remote_helpers/git/repo.py
+++ b/git_remote_helpers/git/repo.py
@@ -1,6 +1,9 @@
 import os
 import subprocess
 
+from git_remote_helpers.util import check_call
+
+
 def sanitize(rev, sep='\t'):
     """Converts a for-each-ref line to a name/value pair.
     """
@@ -53,9 +56,7 @@ class GitRepo(object):
         path = ".cached_revs"
         ofile = open(path, "w")
 
-        child = subprocess.Popen(args, stdout=ofile)
-        if child.wait() != 0:
-            raise CalledProcessError
+        check_call(args, stdout=ofile)
         output = open(path).readlines()
         self.revmap = dict(sanitize(i) for i in output)
         if "HEAD" in self.revmap:
diff --git a/git_remote_helpers/util.py b/git_remote_helpers/util.py
index dce83e6..8b9f302 100644
--- a/git_remote_helpers/util.py
+++ b/git_remote_helpers/util.py
@@ -128,6 +128,38 @@ def run_command (args, cwd = None, shell = False, add_env = None,
     return (exit_code, output, errors)
 
 
+# from python2.7:subprocess.py
+def call(*popenargs, **kwargs):
+    """Run command with arguments.  Wait for command to complete, then
+    return the returncode attribute.
+
+    The arguments are the same as for the Popen constructor.  Example:
+
+    retcode = call(["ls", "-l"])
+    """
+    return subprocess.Popen(*popenargs, **kwargs).wait()
+
+
+# from python2.7:subprocess.py
+def check_call(*popenargs, **kwargs):
+    """Run command with arguments.  Wait for command to complete.  If
+    the exit code was zero then return, otherwise raise
+    CalledProcessError.  The CalledProcessError object will have the
+    return code in the returncode attribute.
+
+    The arguments are the same as for the Popen constructor.  Example:
+
+    check_call(["ls", "-l"])
+    """
+    retcode = call(*popenargs, **kwargs)
+    if retcode:
+        cmd = kwargs.get("args")
+        if cmd is None:
+            cmd = popenargs[0]
+        raise subprocess.CalledProcessError(retcode, cmd)
+    return 0
+
+
 def file_reader_method (missing_ok = False):
     """Decorator for simplifying reading of files.
 
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 19, 2011, 22:58 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 10/20] git-remote-testgit: fix error handling

Hi,
Sverre Rabbelier wrote:
> If fast-export did not complete successfully the error handling code
> itself would error out.
That sounds like a problem indeed.  What error message does it produce?
> This was broken in commit 23b093ee0 (Brandon Casey, Wed Jun 9 2010,
> Remove python 2.5'isms). Revert that commit an introduce our own copy
> of check_call in util.py instead.

The "an" should probably be "and". More importantly, it's probably worth mentioning that this is only a partial revert or rephrasing some other way.

Have you checked if this still works with python 2.4? Cc-ing Brandon in case he has advice.

[...]
> +++ b/git_remote_helpers/git/exporter.py
[...]
Show 8 quoted lines
> @@ -53,6 +55,4 @@ class GitExporter(object):
>  
>          args = ["sed", "s_refs/heads/_" + self.repo.prefix + "_g"]
>  
> -        child = subprocess.Popen(args, stdin=p1.stdout)
> -        if child.wait() != 0:
> -            raise CalledProcessError
> +        check_call(args, stdin=p1.stdout)
[...]
Show 7 quoted lines
> +++ b/git_remote_helpers/util.py
> @@ -128,6 +128,38 @@ def run_command (args, cwd = None, shell = False, add_env = None,
>      return (exit_code, output, errors)
>  
>  
> +# from python2.7:subprocess.py
> +def call(*popenargs, **kwargs):
[...]
Show 5 quoted lines
> +    return subprocess.Popen(*popenargs, **kwargs).wait()
> +
> +
> +# from python2.7:subprocess.py
> +def check_call(*popenargs, **kwargs):
[...]
Show 7 quoted lines
> +    retcode = call(*popenargs, **kwargs)
> +    if retcode:
> +        cmd = kwargs.get("args")
> +        if cmd is None:
> +            cmd = popenargs[0]
> +        raise subprocess.CalledProcessError(retcode, cmd)
> +    return 0

So IIUC the existing code is not providing arguments to the CalledProcessError constructor, and your patch fixes that. Good.

Based on <http://pydoc.org/2.4.1/subprocess.html>, Python 2.4 doesn't seem to have a CalledProcessError type. Maybe these code paths weren't being exercised before.

Regards, Jonathan

Brandon Casey· Jun 20, 2011, 17:50 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 10/20] git-remote-testgit: fix error handling

On 06/19/2011 05:58 PM, Jonathan Nieder wrote:
Show 5 quoted lines
> Sverre Rabbelier wrote:
> 
> Have you checked if this still works with python 2.4?  Cc-ing Brandon
> in case he has advice.
> 
> Based on <http://pydoc.org/2.4.1/subprocess.html>, Python 2.4 doesn't
> seem to have a CalledProcessError type.  Maybe these code paths
> weren't being exercised before.

Well, I don't see anywhere where CalledProccessError is actually caught. i.e. I don't see

   except subprocess.CalledProcessError:

anywhere. So, on python 2.5+ if this exception is ever raised, the script would just exit and produce a backtrace right? And the last line would look something like this:

     File "XXX", line XX, in check_call
       raise subprocess.CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['XXX', 'YYY', 'ZZZ']' returned non-zero exit status 1

On python 2.4, it would also exit and produce a backtrace that looks like this:

     File "test.py", line 11, in check_call
       raise subprocess.CalledProcessError(retcode, cmd)
   AttributeError: 'module' object has no attribute 'CalledProcessError'

So, maybe it would work on python 2.4, but for the wrong reasons. Since the CalledProcessError is never caught, couldn't we just print an error message and exit(1) within Sverre's check_call implementation?

Btw. the only reason I submitted those changes to support python 2.4 was because RHEL 5.X ships with python 2.4, and the changes were not too intrusive. So, it should be considered whether supporting 2.4 is desirable. I wouldn't want to increase the maintenance burden on the real python developers (i.e. not me).

Also, sorry about introducing this bug.  I'm definitely no python guru.
-Brandon
Sverre Rabbelier· Jun 20, 2011, 18:02 UTC · re: Brandon Casey · lore

Re: [PATCH v2 10/20] git-remote-testgit: fix error handling

Heya,

On Mon, Jun 20, 2011 at 19:50, Brandon Casey <brandon.casey.ctr@nrlssc.navy.mil> wrote:

Show 7 quoted lines
> Well, I don't see anywhere where CalledProccessError is actually caught.
> i.e. I don't see
>
>   except subprocess.CalledProcessError:
>
> anywhere.  So, on python 2.5+ if this exception is ever raised, the
> script would just exit and produce a backtrace right?

We're not trying to catch it, but we raise it (since that's what the python 2.5 implementation of check_call does). But if it's not defined in python 2.4, then my patch broke 2.4 again :).

Show 6 quoted lines
> On python 2.4, it would also exit and produce a backtrace that
> looks like this:
>
>     File "test.py", line 11, in check_call
>       raise subprocess.CalledProcessError(retcode, cmd)
>   AttributeError: 'module' object has no attribute 'CalledProcessError'

Yeah, it would, my bad :(. I'll just define a dummy CalledProcessError if it isn't defined to support python 2.4

Show 5 quoted lines
> Btw. the only reason I submitted those changes to support python 2.4
> was because RHEL 5.X ships with python 2.4, and the changes were not
> too intrusive.  So, it should be considered whether supporting 2.4
> is desirable.  I wouldn't want to increase the maintenance burden
> on the real python developers (i.e. not me).

Wikipedia says RHEL 5.x will ready End of Production 1 on Q4 2011, but RHEL 6.x has been out only a little over half a year, so it's probably worth doing.

-- 
Cheers,

Sverre Rabbelier
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 11/20] fast-import: introduce 'done' command

Add a 'done' command that causes fast-import to stop reading from the stream and exit.

If the new --done command line flag was passed on the command line (or a "feature done" declaration included at the start of the stream), make the 'done' command mandatory. So "git fast-import --done"'s input format will be prefix-free, making errors easier to detect when they show up as early termination at some convenient time of the upstream of a pipe writing to fast-import.

Another possible application of the 'done' command would to be allow a fast-import stream that is only a small part of a larger encapsulating stream to be easily parsed, leaving the file offset after the "done\n" so the other application can pick up from there. This patch does not teach fast-import to do that --- fast-import still uses buffered input (stdio).

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 Documentation/git-fast-import.txt |   25 ++++++++++++++++++++++
 fast-import.c                     |    8 +++++++
 t/t9300-fast-import.sh            |   42 +++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+), 0 deletions(-)
Show changes to 3 files +75 −0

Documentation/git-fast-import.txt, fast-import.c, t/t9300-fast-import.sh

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 249249a..0fc68a9 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -101,6 +101,12 @@ OPTIONS
 	when the `cat-blob` command is encountered in the stream.
 	The default behaviour is to write to `stdout`.
 
+--done::
+	Require a `done` command at the end of the stream.
+	This option might be useful for detecting errors that
+	cause the frontend to terminate before it has started to
+	write a stream.
+
 --export-pack-edges=<file>::
 	After creating a packfile, print a line of data to
 	<file> listing the filename of the packfile and the last
@@ -330,6 +336,11 @@ and control the current import process.  More detailed discussion
 	standard output.  This command is optional and is not needed
 	to perform an import.
 
+`done`::
+	Marks the end of the stream. This command is optional
+	unless the `done` feature was requested using the
+	`--done` command line option or `feature done` command.
+
 `cat-blob`::
 	Causes fast-import to print a blob in 'cat-file --batch'
 	format to the file descriptor set with `--cat-blob-fd` or
@@ -1015,6 +1026,11 @@ notes::
 	Versions of fast-import not supporting notes will exit
 	with a message indicating so.
 
+done::
+	Error out if the stream ends without a 'done' command.
+	Without this feature, errors causing the frontend to end
+	abruptly at a convenient point in the stream can go
+	undetected.
 
 `option`
 ~~~~~~~~
@@ -1044,6 +1060,15 @@ not be passed as option:
 * cat-blob-fd
 * force
 
+`done`
+~~~~~~
+If the `done` feature is not in use, treated as if EOF was read.
+This can be used to tell fast-import to finish early.
+
+If the `--done` command line option or `feature done` command is
+in use, the `done` command is mandatory and marks the end of the
+stream.
+
 Crash Reports
 -------------
 If fast-import is supplied invalid input it will terminate with a
diff --git a/fast-import.c b/fast-import.c
index 78d9786..8a8a915 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -354,6 +354,7 @@ static unsigned int cmd_save = 100;
 static uintmax_t next_mark;
 static struct strbuf new_data = STRBUF_INIT;
 static int seen_data_command;
+static int require_explicit_termination;
 
 /* Signal handling */
 static volatile sig_atomic_t checkpoint_requested;
@@ -3139,6 +3140,8 @@ static int parse_one_feature(const char *feature, int from_stream)
 		relative_marks_paths = 1;
 	} else if (!strcmp(feature, "no-relative-marks")) {
 		relative_marks_paths = 0;
+	} else if (!strcmp(feature, "done")) {
+		require_explicit_termination = 1;
 	} else if (!strcmp(feature, "force")) {
 		force_update = 1;
 	} else if (!strcmp(feature, "notes") || !strcmp(feature, "ls")) {
@@ -3288,6 +3291,8 @@ int main(int argc, const char **argv)
 			parse_reset_branch();
 		else if (!strcmp("checkpoint", command_buf.buf))
 			parse_checkpoint();
+		else if (!strcmp("done", command_buf.buf))
+			break;
 		else if (!prefixcmp(command_buf.buf, "progress "))
 			parse_progress();
 		else if (!prefixcmp(command_buf.buf, "feature "))
@@ -3307,6 +3312,9 @@ int main(int argc, const char **argv)
 	if (!seen_data_command)
 		parse_argv();
 
+	if (require_explicit_termination && feof(stdin))
+		die("stream ends early");
+
 	end_packfile();
 
 	dump_branches();
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 6b1ba6c..526231b 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -2197,6 +2197,48 @@ test_expect_success 'R: quiet option results in no stats being output' '
     test_cmp empty output
 '
 
+test_expect_success 'R: feature done means terminating "done" is mandatory' '
+	echo feature done | test_must_fail git fast-import &&
+	test_must_fail git fast-import --done </dev/null
+'
+
+test_expect_success 'R: terminating "done" with trailing gibberish is ok' '
+	git fast-import <<-\EOF &&
+	feature done
+	done
+	trailing gibberish
+	EOF
+	git fast-import <<-\EOF
+	done
+	more trailing gibberish
+	EOF
+'
+
+test_expect_success 'R: terminating "done" within commit' '
+	cat >expect <<-\EOF &&
+	OBJID
+	:000000 100644 OBJID OBJID A	hello.c
+	:000000 100644 OBJID OBJID A	hello2.c
+	EOF
+	git fast-import <<-EOF &&
+	commit refs/heads/done-ends
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<EOT
+	Commit terminated by "done" command
+	EOT
+	M 100644 inline hello.c
+	data <<EOT
+	Hello, world.
+	EOT
+	C hello.c hello2.c
+	done
+	EOF
+	git rev-list done-ends |
+	git diff-tree -r --stdin --root --always |
+	sed -e "s/$_x40/OBJID/g" >actual &&
+	test_cmp expect actual
+'
+
 cat >input <<EOF
 option git non-existing-option
 EOF
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 12/20] fast-export: support done feature

If fast-export is being used to generate a fast-import stream that will be used afterwards it is desirable to indicate the end of the stream with the new 'done' command.

Add a flag that causes fast-export to end with 'done'.
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 Documentation/git-fast-export.txt |    4 ++++
 builtin/fast-export.c             |    9 +++++++++
 2 files changed, 13 insertions(+), 0 deletions(-)
Show changes to 2 files +13 −0

Documentation/git-fast-export.txt, builtin/fast-export.c

diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt
index 781bd6e..e3f8453 100644
--- a/Documentation/git-fast-export.txt
+++ b/Documentation/git-fast-export.txt
@@ -82,6 +82,10 @@ marks the same across runs.
 	allow that.  So fake a tagger to be able to fast-import the
 	output.
 
+--use-done-feature::
+	Start the stream with a 'feature done' stanza, and terminate
+	it with a 'done' command.
+
 --no-data::
 	Skip output of blob objects and instead refer to blobs via
 	their original SHA-1 hash.  This is useful when rewriting the
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index daf1945..becef85 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -26,6 +26,7 @@ static int progress;
 static enum { ABORT, VERBATIM, WARN, STRIP } signed_tag_mode = ABORT;
 static enum { ERROR, DROP, REWRITE } tag_of_filtered_mode = ABORT;
 static int fake_missing_tagger;
+static int use_done_feature;
 static int no_data;
 static int full_tree;
 
@@ -627,6 +628,8 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)
 			     "Fake a tagger when tags lack one"),
 		OPT_BOOLEAN(0, "full-tree", &full_tree,
 			     "Output full tree for each commit"),
+		OPT_BOOLEAN(0, "use-done-feature", &use_done_feature,
+			     "Use the done feature to terminate the stream"),
 		{ OPTION_NEGBIT, 0, "data", &no_data, NULL,
 			"Skip output of blob data",
 			PARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },
@@ -648,6 +651,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)
 	if (argc > 1)
 		usage_with_options (fast_export_usage, options);
 
+	if (use_done_feature)
+		printf("feature done\n");
+
 	if (import_filename)
 		import_marks(import_filename);
 
@@ -675,5 +681,8 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)
 	if (export_filename)
 		export_marks(export_filename);
 
+	if (use_done_feature)
+		printf("done\n");
+
 	return 0;
 }
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 13/20] transport-helper: factor out push_update_refs_status

The update ref status part of push is useful for the export command as well, factor it out into it's own function.

Also factor out push_update_ref_status to avoid a long loop without an explicit condition with a non-trivial body.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 transport-helper.c |  153 ++++++++++++++++++++++++++++-----------------------
 1 files changed, 84 insertions(+), 69 deletions(-)
Show changes to transport-helper.c +84 −69
diff --git a/transport-helper.c b/transport-helper.c
index 34d18aa..ecb44f6 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -554,6 +554,88 @@ static int fetch(struct transport *transport,
 	return -1;
 }
 
+static void push_update_ref_status(struct strbuf *buf,
+				   struct ref **ref,
+				   struct ref *remote_refs)
+{
+	char *refname, *msg;
+	int status;
+
+	if (!prefixcmp(buf->buf, "ok ")) {
+		status = REF_STATUS_OK;
+		refname = buf->buf + 3;
+	} else if (!prefixcmp(buf->buf, "error ")) {
+		status = REF_STATUS_REMOTE_REJECT;
+		refname = buf->buf + 6;
+	} else
+		die("expected ok/error, helper said '%s'\n", buf->buf);
+
+	msg = strchr(refname, ' ');
+	if (msg) {
+		struct strbuf msg_buf = STRBUF_INIT;
+		const char *end;
+
+		*msg++ = '\0';
+		if (!unquote_c_style(&msg_buf, msg, &end))
+			msg = strbuf_detach(&msg_buf, NULL);
+		else
+			msg = xstrdup(msg);
+		strbuf_release(&msg_buf);
+
+		if (!strcmp(msg, "no match")) {
+			status = REF_STATUS_NONE;
+			free(msg);
+			msg = NULL;
+		}
+		else if (!strcmp(msg, "up to date")) {
+			status = REF_STATUS_UPTODATE;
+			free(msg);
+			msg = NULL;
+		}
+		else if (!strcmp(msg, "non-fast forward")) {
+			status = REF_STATUS_REJECT_NONFASTFORWARD;
+			free(msg);
+			msg = NULL;
+		}
+	}
+
+	if (*ref)
+		*ref = find_ref_by_name(*ref, refname);
+	if (!*ref)
+		*ref = find_ref_by_name(remote_refs, refname);
+	if (!*ref) {
+		warning("helper reported unexpected status of %s", refname);
+		return;
+	}
+
+	if ((*ref)->status != REF_STATUS_NONE) {
+		/*
+		 * Earlier, the ref was marked not to be pushed, so ignore the ref
+		 * status reported by the remote helper if the latter is 'no match'.
+		 */
+		if (status == REF_STATUS_NONE)
+			return;
+	}
+
+	(*ref)->status = status;
+	(*ref)->remote_status = msg;
+}
+
+static void push_update_refs_status(struct helper_data *data,
+				    struct ref *remote_refs)
+{
+	struct strbuf buf = STRBUF_INIT;
+	struct ref *ref = remote_refs;
+	for (;;) {
+		recvline(data, &buf);
+		if (!buf.len)
+			break;
+
+		push_update_ref_status(&buf, &ref, remote_refs);
+	}
+	strbuf_release(&buf);
+}
+
 static int push_refs_with_push(struct transport *transport,
 		struct ref *remote_refs, int flags)
 {
@@ -608,76 +690,9 @@ static int push_refs_with_push(struct transport *transport,
 
 	strbuf_addch(&buf, '\n');
 	sendline(data, &buf);
-
-	ref = remote_refs;
-	while (1) {
-		char *refname, *msg;
-		int status;
-
-		recvline(data, &buf);
-		if (!buf.len)
-			break;
-
-		if (!prefixcmp(buf.buf, "ok ")) {
-			status = REF_STATUS_OK;
-			refname = buf.buf + 3;
-		} else if (!prefixcmp(buf.buf, "error ")) {
-			status = REF_STATUS_REMOTE_REJECT;
-			refname = buf.buf + 6;
-		} else
-			die("expected ok/error, helper said '%s'\n", buf.buf);
-
-		msg = strchr(refname, ' ');
-		if (msg) {
-			struct strbuf msg_buf = STRBUF_INIT;
-			const char *end;
-
-			*msg++ = '\0';
-			if (!unquote_c_style(&msg_buf, msg, &end))
-				msg = strbuf_detach(&msg_buf, NULL);
-			else
-				msg = xstrdup(msg);
-			strbuf_release(&msg_buf);
-
-			if (!strcmp(msg, "no match")) {
-				status = REF_STATUS_NONE;
-				free(msg);
-				msg = NULL;
-			}
-			else if (!strcmp(msg, "up to date")) {
-				status = REF_STATUS_UPTODATE;
-				free(msg);
-				msg = NULL;
-			}
-			else if (!strcmp(msg, "non-fast forward")) {
-				status = REF_STATUS_REJECT_NONFASTFORWARD;
-				free(msg);
-				msg = NULL;
-			}
-		}
-
-		if (ref)
-			ref = find_ref_by_name(ref, refname);
-		if (!ref)
-			ref = find_ref_by_name(remote_refs, refname);
-		if (!ref) {
-			warning("helper reported unexpected status of %s", refname);
-			continue;
-		}
-
-		if (ref->status != REF_STATUS_NONE) {
-			/*
-			 * Earlier, the ref was marked not to be pushed, so ignore the ref
-			 * status reported by the remote helper if the latter is 'no match'.
-			 */
-			if (status == REF_STATUS_NONE)
-				continue;
-		}
-
-		ref->status = status;
-		ref->remote_status = msg;
-	}
 	strbuf_release(&buf);
+
+	push_update_refs_status(data, remote_refs);
 	return 0;
 }
 
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 14/20] transport-helper: check status code of finish_command

Previously the status code of all helpers were ignored, allowing errors that occur to go unnoticed if the error text output by the helper is not noticed.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 transport-helper.c |   23 +++++++++++++++--------
 1 files changed, 15 insertions(+), 8 deletions(-)
Show changes to transport-helper.c +15 −8
diff --git a/transport-helper.c b/transport-helper.c
index ecb44f6..7f3c6c6 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -204,6 +204,7 @@ static int disconnect_helper(struct transport *transport)
 {
 	struct helper_data *data = transport->data;
 	struct strbuf buf = STRBUF_INIT;
+	int res = 0;
 
 	if (data->helper) {
 		if (debug)
@@ -215,13 +216,13 @@ static int disconnect_helper(struct transport *transport)
 		close(data->helper->in);
 		close(data->helper->out);
 		fclose(data->out);
-		finish_command(data->helper);
+		res = finish_command(data->helper);
 		free((char *)data->helper->argv[0]);
 		free(data->helper->argv);
 		free(data->helper);
 		data->helper = NULL;
 	}
-	return 0;
+	return res;
 }
 
 static const char *unsupported_options[] = {
@@ -299,12 +300,13 @@ static void standard_options(struct transport *t)
 
 static int release_helper(struct transport *transport)
 {
+	int res = 0;
 	struct helper_data *data = transport->data;
 	free_refspec(data->refspec_nr, data->refspecs);
 	data->refspecs = NULL;
-	disconnect_helper(transport);
+	res = disconnect_helper(transport);
 	free(transport->data);
-	return 0;
+	return res;
 }
 
 static int fetch_with_fetch(struct transport *transport,
@@ -410,8 +412,11 @@ static int fetch_with_import(struct transport *transport,
 		sendline(data, &buf);
 		strbuf_reset(&buf);
 	}
-	disconnect_helper(transport);
-	finish_command(&fastimport);
+	if (disconnect_helper(transport))
+		die("Error while disconnecting helper");
+	if (finish_command(&fastimport))
+		die("Error while running fast-import");
+
 	free(fastimport.argv);
 	fastimport.argv = NULL;
 
@@ -755,8 +760,10 @@ static int push_refs_with_export(struct transport *transport,
 		die("Couldn't run fast-export");
 
 	data->no_disconnect_req = 1;
-	finish_command(&exporter);
-	disconnect_helper(transport);
+	if (finish_command(&exporter))
+		die("Error while running fast-export");
+	if (disconnect_helper(transport))
+		die("Error while disconnecting helper");
 	return 0;
 }
 
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 15/20] transport-helper: use the new done feature where possible

In other words, use fast-export --use-done-feature to add a 'done' command at the end of streams passed to remote helpers' "import" commands, and teach the remote helpers implementing "export" to use the 'done' command in turn when producing their streams.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 git-remote-testgit.py |    2 ++
 transport-helper.c    |    8 ++------
 2 files changed, 4 insertions(+), 6 deletions(-)
Show changes to 2 files +4 −6

git-remote-testgit.py, transport-helper.c

diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index 9658355..a8e47d9 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -124,6 +124,8 @@ def do_import(repo, args):
     repo = update_local_repo(repo)
     repo.exporter.export_repo(repo.gitdir, args)
 
+    print "done"
+
 
 def do_export(repo, args):
     """Imports a fast-import stream from git to testgit.
diff --git a/transport-helper.c b/transport-helper.c
index 7f3c6c6..b0361c2 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -375,8 +375,9 @@ static int get_exporter(struct transport *transport,
 	/* we need to duplicate helper->in because we want to use it after
 	 * fastexport is done with it. */
 	fastexport->out = dup(helper->in);
-	fastexport->argv = xcalloc(4 + revlist_args->nr, sizeof(*fastexport->argv));
+	fastexport->argv = xcalloc(5 + revlist_args->nr, sizeof(*fastexport->argv));
 	fastexport->argv[argc++] = "fast-export";
+	fastexport->argv[argc++] = "--use-done-feature";
 	if (export_marks)
 		fastexport->argv[argc++] = export_marks;
 	if (import_marks)
@@ -412,11 +413,8 @@ static int fetch_with_import(struct transport *transport,
 		sendline(data, &buf);
 		strbuf_reset(&buf);
 	}
-	if (disconnect_helper(transport))
-		die("Error while disconnecting helper");
 	if (finish_command(&fastimport))
 		die("Error while running fast-import");
-
 	free(fastimport.argv);
 	fastimport.argv = NULL;
 
@@ -762,8 +760,6 @@ static int push_refs_with_export(struct transport *transport,
 	data->no_disconnect_req = 1;
 	if (finish_command(&exporter))
 		die("Error while running fast-export");
-	if (disconnect_helper(transport))
-		die("Error while disconnecting helper");
 	return 0;
 }
 
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 20, 2011, 11:45 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 15/20] transport-helper: use the new done feature where possible

Hi,
Sverre Rabbelier wrote:
> In other words, use fast-export --use-done-feature to add a 'done'
> command at the end of streams passed to remote helpers' "import"
> commands, and teach the remote helpers implementing "export" to use
> the 'done' command in turn when producing their streams.
Sounds like a reasonable thing to do.  One puzzle:
[...]
Show 21 quoted lines
> +++ b/transport-helper.c
> @@ -375,8 +375,9 @@ static int get_exporter(struct transport *transport,
>  	/* we need to duplicate helper->in because we want to use it after
>  	 * fastexport is done with it. */
>  	fastexport->out = dup(helper->in);
> -	fastexport->argv = xcalloc(4 + revlist_args->nr, sizeof(*fastexport->argv));
> +	fastexport->argv = xcalloc(5 + revlist_args->nr, sizeof(*fastexport->argv));
>  	fastexport->argv[argc++] = "fast-export";
> +	fastexport->argv[argc++] = "--use-done-feature";
>  	if (export_marks)
>  		fastexport->argv[argc++] = export_marks;
>  	if (import_marks)
> @@ -412,11 +413,8 @@ static int fetch_with_import(struct transport *transport,
>  		sendline(data, &buf);
>  		strbuf_reset(&buf);
>  	}
> -	if (disconnect_helper(transport))
> -		die("Error while disconnecting helper");
>  	if (finish_command(&fastimport))
>  		die("Error while running fast-import");
> -

What is this change about? Is the plan to allow other commands after a fetch_with_import? Sounds reasonable; I think it should be advertised in the log message, though.

When does the disconnect_helper call happen (to avoid leaks)? Ah, in release_helper; phew.

The disconnect_helper call writes the blank line that terminates the list of "import %s" commands to start the import, so there would need to be a

	strbuf_reset(&buf);
	strbuf_addf(&buf, "\n");
	sendline(data, &buf);
in its place.
Junio C Hamano· Jun 20, 2011, 19:51 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 15/20] transport-helper: use the new done feature where possible

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 26 quoted lines
>> @@ -412,11 +413,8 @@ static int fetch_with_import(struct transport *transport,
>>  		sendline(data, &buf);
>>  		strbuf_reset(&buf);
>>  	}
>> -	if (disconnect_helper(transport))
>> -		die("Error while disconnecting helper");
>>  	if (finish_command(&fastimport))
>>  		die("Error while running fast-import");
>> -
>
> What is this change about?  Is the plan to allow other commands after
> a fetch_with_import?  Sounds reasonable; I think it should be
> advertised in the log message, though.
>
> When does the disconnect_helper call happen (to avoid leaks)?  Ah, in
> release_helper; phew.
>
> The disconnect_helper call writes the blank line that terminates the
> list of "import %s" commands to start the import, so there would need
> to be a
>
> 	strbuf_reset(&buf);
> 	strbuf_addf(&buf, "\n");
> 	sendline(data, &buf);
>
> in its place.
All sensible suggestions for a potential reroll.
Sverre Rabbelier· Jul 4, 2011, 13:37 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 15/20] transport-helper: use the new done feature where possible

Heya,
On Mon, Jun 20, 2011 at 13:45, Jonathan Nieder <jrnieder@gmail.com> wrote:
> What is this change about?  Is the plan to allow other commands after
> a fetch_with_import?  Sounds reasonable; I think it should be
> advertised in the log message, though.

Yeah, the order/granularity of these commits is perhaps suboptimal. The actual changes is by now explained in "export is no longer always the last command" and "change import semantics". Perhaps they should just be squashed together?

> The disconnect_helper call writes the blank line that terminates the
> list of "import %s" commands to start the import, so there would need

Terminating the list of import commands isn't needed until the "change import semantics" commit, in which a write_constant(data->helper->in, "\n") is added.

-- 
Cheers,

Sverre Rabbelier
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 16/20] transport-helper: update ref status after push with export

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  This time a proper implementation of this patch.
 git-remote-testgit.py              |    6 +++++-
 git_remote_helpers/git/importer.py |   28 +++++++++++++++++++++++++++-
 git_remote_helpers/util.py         |   34 ++++++++++++++++++++++++++++++++++
 transport-helper.c                 |    1 +
 4 files changed, 67 insertions(+), 2 deletions(-)
Show changes to 4 files +67 −2

git-remote-testgit.py, git_remote_helpers/git/importer.py, git_remote_helpers/util.py, transport-helper.c

diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index a8e47d9..854e27e 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -148,11 +148,15 @@ def do_export(repo, args):
     sys.stdout.flush()
 
     update_local_repo(repo)
-    repo.importer.do_import(repo.gitdir)
+    changed = repo.importer.do_import(repo.gitdir)
 
     if not repo.local:
         repo.non_local.push(repo.gitdir)
 
+    for ref in changed:
+        print "ok %s" % ref
+    print
+
 
 def do_gitdir(repo, args):
     """Stores the location of the gitdir.
diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py
index 02a719a..5c6b595 100644
--- a/git_remote_helpers/git/importer.py
+++ b/git_remote_helpers/git/importer.py
@@ -1,7 +1,7 @@
 import os
 import subprocess
 
-from git_remote_helpers.util import check_call
+from git_remote_helpers.util import check_call, check_output
 
 
 class GitImporter(object):
@@ -16,6 +16,18 @@ class GitImporter(object):
 
         self.repo = repo
 
+    def get_refs(self, gitdir):
+        """Returns a dictionary with refs.
+        """
+        args = ["git", "--git-dir=" + gitdir, "for-each-ref", "refs/heads"]
+        lines = check_output(args).strip().split('\n')
+        refs = {}
+        for line in lines:
+            value, name = line.split(' ')
+            name = name.strip('commit\t')
+            refs[name] = value
+        return refs
+
     def do_import(self, base):
         """Imports a fast-import stream to the given directory.
 
@@ -32,9 +44,23 @@ class GitImporter(object):
         if not os.path.exists(dirname):
             os.makedirs(dirname)
 
+        refs_before = self.get_refs(gitdir)
+
         args = ["git", "--git-dir=" + gitdir, "fast-import", "--quiet", "--export-marks=" + path]
 
         if os.path.exists(path):
             args.append("--import-marks=" + path)
 
         check_call(args)
+
+        refs_after = self.get_refs(gitdir)
+
+        changed = {}
+
+        for name, value in refs_after.iteritems():
+            if refs_before.get(name) == value:
+                continue
+
+            changed[name] = value
+
+        return changed
diff --git a/git_remote_helpers/util.py b/git_remote_helpers/util.py
index 8b9f302..6a9ec5c 100644
--- a/git_remote_helpers/util.py
+++ b/git_remote_helpers/util.py
@@ -160,6 +160,40 @@ def check_call(*popenargs, **kwargs):
     return 0
 
 
+# from python2.7:subprocess.py
+def check_output(*popenargs, **kwargs):
+    r"""Run command with arguments and return its output as a byte string.
+
+    If the exit code was non-zero it raises a CalledProcessError.  The
+    CalledProcessError object will have the return code in the returncode
+    attribute and output in the output attribute.
+
+    The arguments are the same as for the Popen constructor.  Example:
+
+    >>> check_output(["ls", "-l", "/dev/null"])
+    'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
+
+    The stdout argument is not allowed as it is used internally.
+    To capture standard error in the result, use stderr=STDOUT.
+
+    >>> check_output(["/bin/sh", "-c",
+    ...               "ls -l non_existent_file ; exit 0"],
+    ...              stderr=STDOUT)
+    'ls: non_existent_file: No such file or directory\n'
+    """
+    if 'stdout' in kwargs:
+        raise ValueError('stdout argument not allowed, it will be overridden.')
+    process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
+    output, unused_err = process.communicate()
+    retcode = process.poll()
+    if retcode:
+        cmd = kwargs.get("args")
+        if cmd is None:
+            cmd = popenargs[0]
+        raise subprocess.CalledProcessError(retcode, cmd)
+    return output
+
+
 def file_reader_method (missing_ok = False):
     """Decorator for simplifying reading of files.
 
diff --git a/transport-helper.c b/transport-helper.c
index b0361c2..bb1b97f 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -760,6 +760,7 @@ static int push_refs_with_export(struct transport *transport,
 	data->no_disconnect_req = 1;
 	if (finish_command(&exporter))
 		die("Error while running fast-export");
+	push_update_refs_status(data, remote_refs);
 	return 0;
 }
 
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 19, 2011, 23:25 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 16/20] transport-helper: update ref status after push with export

Sverre Rabbelier wrote:
Show 5 quoted lines
>
> Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
> ---
>
>   This time a proper implementation of this patch.
Nice. :)

I assume this still does not return "error <dst> <reason>" responses, instead disconnecting the helper on error, which is fine but probably worth mentioning.

Show 16 quoted lines
> --- a/git-remote-testgit.py
> +++ b/git-remote-testgit.py
> @@ -148,11 +148,15 @@ def do_export(repo, args):
>      sys.stdout.flush()
>  
>      update_local_repo(repo)
> -    repo.importer.do_import(repo.gitdir)
> +    changed = repo.importer.do_import(repo.gitdir)
>  
>      if not repo.local:
>          repo.non_local.push(repo.gitdir)
>  
> +    for ref in changed:
> +        print "ok %s" % ref
> +    print
> +
The remote-helpers(1) manual says, concerning "push":
	When the push is complete, outputs one or more ok <dst> or
	error <dst> <why>?  lines to indicate success or failure of
	each pushed ref. The status report output is terminated by a
	blank line. The option field <why> may be quoted in a C style
	string if it contains an LF.

Ideally we would want "export" to also print all refs intended for export, even if they did not actually change, but there's no obvious way to get that information. In the "not repo.local" case, seeing which refs changed in .git/info/fast-import/<repo>.git like you do here seems like a reasonable enough heuristic.

What happens if no ref changed (the "already up to date" case)?
Junio C Hamano· Jun 21, 2011, 20:05 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 16/20] transport-helper: update ref status after push with export

Sverre Rabbelier <srabbelier@gmail.com> writes:
Show 7 quoted lines
> +        changed = {}
> +
> +        for name, value in refs_after.iteritems():
> +            if refs_before.get(name) == value:
> +                continue
> +
> +            changed[name] = value

This assumes that nobody will ever _remove_ refs. I think it is a sensible assumption but somebody might want to give a power to the fast-import stream to say "I do not want that ref anymore".

I dunno; it is not a big deal to me either way. Just something to think about.

Sverre Rabbelier· Jun 21, 2011, 20:11 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 16/20] transport-helper: update ref status after push with export

Heya,
On Tue, Jun 21, 2011 at 22:05, Junio C Hamano <gitster@pobox.com> wrote:
Show 6 quoted lines
> This assumes that nobody will ever _remove_ refs. I think it is a sensible
> assumption but somebody might want to give a power to the fast-import stream
> to say "I do not want that ref anymore".
>
> I dunno; it is not a big deal to me either way.  Just something to think
> about.

I'll have a look at addressing your and Jonathan's comments (the case of 'already up to date' and 'refs removed').

-- 
Cheers,

Sverre Rabbelier
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 17/20] transport-helper: change import semantics

Currently the helper must somehow guess how many import statements to read before it starts outputting its fast-export stream. This is because the remote helper infrastructure runs fast-import only once, so the helper is forced to output one stream for all import commands it will receive. The only reason this worked in the past was because only one ref was imported at a time.

Change the semantics of the import statement such that it matches that of the list statement. That is, 'import\n' is followed by a list of refs that should be exported, followed by '\n'.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 git-remote-testgit.py     |   14 +++++++++++---
 t/t5800-remote-helpers.sh |    2 +-
 transport-helper.c        |    7 ++++++-
 3 files changed, 18 insertions(+), 5 deletions(-)
Show changes to 3 files +18 −5

git-remote-testgit.py, t/t5800-remote-helpers.sh, transport-helper.c

diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index 854e27e..4939853 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -115,14 +115,22 @@ def do_import(repo, args):
     """Exports a fast-import stream from testgit for git to import.
     """
 
-    if len(args) != 1:
-        die("Import needs exactly one ref")
+    if args:
+        die("Import expects its ref seperately")
 
     if not repo.gitdir:
         die("Need gitdir to import")
 
+    refs = []
+
+    while True:
+        line = sys.stdin.readline()
+        if line == '\n':
+            break
+        refs.append(line.strip())
+
     repo = update_local_repo(repo)
-    repo.exporter.export_repo(repo.gitdir, args)
+    repo.exporter.export_repo(repo.gitdir, refs)
 
     print "done"
 
diff --git a/t/t5800-remote-helpers.sh b/t/t5800-remote-helpers.sh
index 12f471c..1c62001 100755
--- a/t/t5800-remote-helpers.sh
+++ b/t/t5800-remote-helpers.sh
@@ -98,7 +98,7 @@ test_expect_success 'fetch new branch' '
 	compare_refs public HEAD localclone FETCH_HEAD
 '
 
-test_expect_failure 'fetch multiple branches' '
+test_expect_success 'fetch multiple branches' '
 	(cd localclone &&
 	 git fetch
 	) &&
diff --git a/transport-helper.c b/transport-helper.c
index bb1b97f..c089928 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -404,15 +404,20 @@ static int fetch_with_import(struct transport *transport,
 	if (get_importer(transport, &fastimport))
 		die("Couldn't run fast-import");
 
+	write_constant(data->helper->in, "import\n");
+
 	for (i = 0; i < nr_heads; i++) {
 		posn = to_fetch[i];
 		if (posn->status & REF_STATUS_UPTODATE)
 			continue;
 
-		strbuf_addf(&buf, "import %s\n", posn->name);
+		strbuf_addf(&buf, "%s\n", posn->name);
 		sendline(data, &buf);
 		strbuf_reset(&buf);
 	}
+
+	write_constant(data->helper->in, "\n");
+
 	if (finish_command(&fastimport))
 		die("Error while running fast-import");
 	free(fastimport.argv);
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 19, 2011, 23:38 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 17/20] transport-helper: change import semantics

Sverre Rabbelier wrote:
> Change the semantics of the import statement such that it matches
> that of the list statement. That is, 'import\n' is followed by a list
> of refs that should be exported, followed by '\n'.
I think that for learnability, it would be better to use
	import <ref>
	import <ref>
	...
	[blank line]

since that is consistent with "fetch" and "push". I'll try mocking up a patch for that tonight.

Sverre Rabbelier· Jul 4, 2011, 11:20 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 17/20] transport-helper: change import semantics

Heya,
On Mon, Jun 20, 2011 at 01:38, Jonathan Nieder <jrnieder@gmail.com> wrote:
Show 9 quoted lines
> I think that for learnability, it would be better to use
>
>        import <ref>
>        import <ref>
>        ...
>        [blank line]
>
> since that is consistent with "fetch" and "push".  I'll try mocking up
> a patch for that tonight.

That would mean the only change compared to the current behavior is the additional blank line, yes?

-- 
Cheers,

Sverre Rabbelier
Jonathan Nieder· Jul 4, 2011, 21:58 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 17/20] transport-helper: change import semantics

Sverre Rabbelier wrote:
> On Mon, Jun 20, 2011 at 01:38, Jonathan Nieder <jrnieder@gmail.com> wrote:
Show 12 quoted lines
>> I think that for learnability, it would be better to use
>>
>>        import <ref>
>>        import <ref>
>>        ...
>>        [blank line]
>>
>> since that is consistent with "fetch" and "push".  I'll try mocking up
>> a patch for that tonight.
>
> That would mean the only change compared to the current behavior is
> the additional blank line, yes?

It would mean no change compared to the current behavior. :) disconnect_helper() writes the blank line.

That's why no patch came; sorry, I should have mentioned so.
Sverre Rabbelier· Jul 4, 2011, 22:23 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 17/20] transport-helper: change import semantics

Heya,
On Mon, Jul 4, 2011 at 23:58, Jonathan Nieder <jrnieder@gmail.com> wrote:
> It would mean no change compared to the current behavior. :)
> disconnect_helper() writes the blank line.
>
> That's why no patch came; sorry, I should have mentioned so.

With the only difference that currently that \n signifies the end of the entire stream, whereas now it will mean the end of the imports?

-- 
Cheers,

Sverre Rabbelier
Jonathan Nieder· Jul 4, 2011, 22:37 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 17/20] transport-helper: change import semantics

Sverre Rabbelier wrote:
> On Mon, Jul 4, 2011 at 23:58, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> It would mean no change compared to the current behavior. :)
[...]
> With the only difference that currently that \n signifies the end of
> the entire stream, whereas now it will mean the end of the imports?

Exactly. The git-remote-testgit part of the patch would be pretty similar to what you have and could even go earlier in the series if wanted (since it does not require transport-helper changes), while the transport-helper part should be unnecessary as long as patch 15/20 "use the new done feature where possible" does not remove the \n in its output (as discussed in its subthread).

Meanwhile I should probably implement a "capability done" as Dmitry hinted, to make blank lines no longer act as a terminator as a way of helping people debugging by hand that hit <ENTER> too many times by mistake.

Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 18/20] transport-helper: export is no longer always the last command

The trailing \n in the protocol signals the helper that the connection is about to close, allowing it to do whatever cleanup neccesary.

Previously, the connection would already be closed by the time the trailing \n was to be written. Now that the remote-helper protocol uses the new done command in its fast-import streams, this is no longer the case and we can safely write the trailing \n.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Updated commit message.
 transport-helper.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)
Show changes to transport-helper.c +0 −1
diff --git a/transport-helper.c b/transport-helper.c
index c089928..f78b670 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -762,7 +762,6 @@ static int push_refs_with_export(struct transport *transport,
 			 export_marks, import_marks, &revlist_args))
 		die("Couldn't run fast-export");
 
-	data->no_disconnect_req = 1;
 	if (finish_command(&exporter))
 		die("Error while running fast-export");
 	push_update_refs_status(data, remote_refs);
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 19/20] transport-helper: Use capname for gitdir capability too

Also properly use capname in the refspec capability.

Previously the gitdir and refspec capabilities could not be listed as required or their parsing would break.

Most likely the reason the second hunk wasn't caught is because the series that added 'gitdir' as capability, and the one that added required capabilities were done in parallel.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Updated commit message.
 transport-helper.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
Show changes to transport-helper.c +2 −2
diff --git a/transport-helper.c b/transport-helper.c
index f78b670..ddf0ffa 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -171,10 +171,10 @@ static struct child_process *get_helper(struct transport *transport)
 			ALLOC_GROW(refspecs,
 				   refspec_nr + 1,
 				   refspec_alloc);
-			refspecs[refspec_nr++] = strdup(buf.buf + strlen("refspec "));
+			refspecs[refspec_nr++] = strdup(capname + strlen("refspec "));
 		} else if (!strcmp(capname, "connect")) {
 			data->connect = 1;
-		} else if (!strcmp(buf.buf, "gitdir")) {
+		} else if (!strcmp(capname, "gitdir")) {
 			struct strbuf gitdir = STRBUF_INIT;
 			strbuf_addf(&gitdir, "gitdir %s\n", get_git_dir());
 			sendline(data, &gitdir);
-- 
1.7.5.1.292.g728120
Sverre Rabbelier· Jun 19, 2011, 15:18 UTC · re: Sverre Rabbelier · lore

[PATCH v2 20/20] transport-helper: implement marks location as capability

While this requires the helper to flush stdout after listing 'gitdir' as capability, and read a command (the 'gitdir' response from the remote helper infrastructure) right after that, this is more elegant and does not require an ad-hoc exchange of values.

Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged.
 git-remote-testgit.py |   31 ++++++++++++++++++-------------
 transport-helper.c    |   47 ++++++++++++++++++-----------------------------
 2 files changed, 36 insertions(+), 42 deletions(-)
Show changes to 2 files +36 −42

git-remote-testgit.py, transport-helper.c

diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index 4939853..30f775e 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -71,8 +71,26 @@ def do_capabilities(repo, args):
     print "import"
     print "export"
     print "gitdir"
+
+    sys.stdout.flush()
+    if not read_one_line(repo):
+        die("Expected gitdir, got empty line")
+    if not repo.gitdir:
+        die("Expected gitdir, got something else")
+
     print "refspec refs/heads/*:%s*" % repo.prefix
 
+    dirname = repo.get_base_path(repo.gitdir)
+
+    if not os.path.exists(dirname):
+        os.makedirs(dirname)
+
+    path = os.path.join(dirname, 'testgit.marks')
+
+    print "*export-marks %s" % path
+    if os.path.exists(path):
+        print "*import-marks %s" % path
+
     print # end capabilities
 
 
@@ -142,19 +160,6 @@ def do_export(repo, args):
     if not repo.gitdir:
         die("Need gitdir to export")
 
-    dirname = repo.get_base_path(repo.gitdir)
-
-    if not os.path.exists(dirname):
-        os.makedirs(dirname)
-
-    path = os.path.join(dirname, 'testgit.marks')
-    print path
-    if os.path.exists(path):
-        print path
-    else:
-        print ""
-    sys.stdout.flush()
-
     update_local_repo(repo)
     changed = repo.importer.do_import(repo.gitdir)
 
diff --git a/transport-helper.c b/transport-helper.c
index ddf0ffa..12f08a1 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -23,6 +23,8 @@ struct helper_data {
 		push : 1,
 		connect : 1,
 		no_disconnect_req : 1;
+	char *export_marks;
+	char *import_marks;
 	/* These go from remote name (as in "list") to private name */
 	struct refspec *refspecs;
 	int refspec_nr;
@@ -179,6 +181,16 @@ static struct child_process *get_helper(struct transport *transport)
 			strbuf_addf(&gitdir, "gitdir %s\n", get_git_dir());
 			sendline(data, &gitdir);
 			strbuf_release(&gitdir);
+		} else if (!prefixcmp(capname, "export-marks ")) {
+			struct strbuf arg = STRBUF_INIT;
+			strbuf_addstr(&arg, "--export-marks=");
+			strbuf_addstr(&arg, capname + strlen("export-marks "));
+			data->export_marks = strbuf_detach(&arg, NULL);
+		} else if (!prefixcmp(capname, "import-marks")) {
+			struct strbuf arg = STRBUF_INIT;
+			strbuf_addstr(&arg, "--import-marks=");
+			strbuf_addstr(&arg, capname + strlen("import-marks "));
+			data->import_marks = strbuf_detach(&arg, NULL);
 		} else if (mandatory) {
 			die("Unknown mandatory capability %s. This remote "
 			    "helper probably needs newer version of Git.\n",
@@ -364,10 +376,9 @@ static int get_importer(struct transport *transport, struct child_process *fasti
 
 static int get_exporter(struct transport *transport,
 			struct child_process *fastexport,
-			const char *export_marks,
-			const char *import_marks,
 			struct string_list *revlist_args)
 {
+	struct helper_data *data = transport->data;
 	struct child_process *helper = get_helper(transport);
 	int argc = 0, i;
 	memset(fastexport, 0, sizeof(*fastexport));
@@ -378,10 +389,10 @@ static int get_exporter(struct transport *transport,
 	fastexport->argv = xcalloc(5 + revlist_args->nr, sizeof(*fastexport->argv));
 	fastexport->argv[argc++] = "fast-export";
 	fastexport->argv[argc++] = "--use-done-feature";
-	if (export_marks)
-		fastexport->argv[argc++] = export_marks;
-	if (import_marks)
-		fastexport->argv[argc++] = import_marks;
+	if (data->export_marks)
+		fastexport->argv[argc++] = data->export_marks;
+	if (data->import_marks)
+		fastexport->argv[argc++] = data->import_marks;
 
 	for (i = 0; i < revlist_args->nr; i++)
 		fastexport->argv[argc++] = revlist_args->items[i].string;
@@ -710,7 +721,6 @@ static int push_refs_with_export(struct transport *transport,
 	struct ref *ref;
 	struct child_process *helper, exporter;
 	struct helper_data *data = transport->data;
-	char *export_marks = NULL, *import_marks = NULL;
 	struct string_list revlist_args = STRING_LIST_INIT_NODUP;
 	struct strbuf buf = STRBUF_INIT;
 
@@ -718,26 +728,6 @@ static int push_refs_with_export(struct transport *transport,
 
 	write_constant(helper->in, "export\n");
 
-	recvline(data, &buf);
-	if (debug)
-		fprintf(stderr, "Debug: Got export_marks '%s'\n", buf.buf);
-	if (buf.len) {
-		struct strbuf arg = STRBUF_INIT;
-		strbuf_addstr(&arg, "--export-marks=");
-		strbuf_addbuf(&arg, &buf);
-		export_marks = strbuf_detach(&arg, NULL);
-	}
-
-	recvline(data, &buf);
-	if (debug)
-		fprintf(stderr, "Debug: Got import_marks '%s'\n", buf.buf);
-	if (buf.len) {
-		struct strbuf arg = STRBUF_INIT;
-		strbuf_addstr(&arg, "--import-marks=");
-		strbuf_addbuf(&arg, &buf);
-		import_marks = strbuf_detach(&arg, NULL);
-	}
-
 	strbuf_reset(&buf);
 
 	for (ref = remote_refs; ref; ref = ref->next) {
@@ -758,8 +748,7 @@ static int push_refs_with_export(struct transport *transport,
 
 	}
 
-	if (get_exporter(transport, &exporter,
-			 export_marks, import_marks, &revlist_args))
+	if (get_exporter(transport, &exporter, &revlist_args))
 		die("Couldn't run fast-export");
 
 	if (finish_command(&exporter))
-- 
1.7.5.1.292.g728120
Jonathan Nieder· Jun 20, 2011, 01:29 UTC · re: Sverre Rabbelier · lore

Re: [PATCH v2 20/20] transport-helper: implement marks location as capability

Sverre Rabbelier wrote:
Show 6 quoted lines
> [Subject: transport-helper: implement marks location as capability]
>
> While this requires the helper to flush stdout after listing 'gitdir'
> as capability, and read a command (the 'gitdir' response from the
> remote helper infrastructure) right after that, this is more elegant
> and does not require an ad-hoc exchange of values.

Probably I'm not thinking straight, but it's not obvious how the subject connects to the content. I suppose the pieces I'm missing are that:

 - this patch introduces new undocumented "import-marks" and
   "export-marks" capabilities that take a pathname as parameter;
 - for remote-testgit, that pathname is based on $GIT_DIR, so to
   retrieve that value we flush stdout and read the gitdir immediately
   rather than hoping the gitdir arrives on stdin in time for it to
   be used later.
But that still leaves a couple of other questions unanswered:

Why do we use a "gitdir" capability for this at all, instead of exporting the GIT_DIR environment variable as Tomas has suggested? (Not about this patch, but a separate patch explaining that in the documentation would be nice.)

How does this interact with fast-import's --relative-marks feature (if at all)?

[...]
> +++ b/transport-helper.c
[...]
Show 17 quoted lines
> @@ -718,26 +728,6 @@ static int push_refs_with_export(struct transport *transport,
>  
>  	write_constant(helper->in, "export\n");
>  
> -	recvline(data, &buf);
> -	if (debug)
> -		fprintf(stderr, "Debug: Got export_marks '%s'\n", buf.buf);
> -	if (buf.len) {
> -		struct strbuf arg = STRBUF_INIT;
> -		strbuf_addstr(&arg, "--export-marks=");
> -		strbuf_addbuf(&arg, &buf);
> -		export_marks = strbuf_detach(&arg, NULL);
> -	}
> -
> -	recvline(data, &buf);
> -	if (debug)
> -		fprintf(stderr, "Debug: Got import_marks '%s'\n", buf.buf);

Hm, it seems there is a silent (but good) change in the behavior of the "export" command, too.

Except where noted already, the patches look good, and the series seems to make the remote helper infrastructure a little saner. Thanks.

Sverre Rabbelier· Jul 4, 2011, 13:43 UTC · re: Jonathan Nieder · lore

Re: [PATCH v2 20/20] transport-helper: implement marks location as capability

Heya,
On Mon, Jun 20, 2011 at 03:29, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Why do we use a "gitdir" capability for this at all, instead of
> exporting the GIT_DIR environment variable as Tomas has suggested?
> (Not about this patch, but a separate patch explaining that in the
> documentation would be nice.)
Mainly because I didn't take the time to implement that, will do.
> How does this interact with fast-import's --relative-marks feature
> (if at all)?

They're a supplement to it, the exporter uses all three features (see git_remote_helpers/git/exporter.py).

-- 
Cheers,

Sverre Rabbelier

← back to recent threads