git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 12/23] fast-import: introduce 'done' command

From
Sverre Rabbelier <srabbelier@gmail.com>
Date
Jul 16, 2011, 13:03 UTC
Message-ID
<1310821424-4750-13-git-send-email-srabbelier@gmail.com>
In-Reply-To
<1310821424-4750-1-git-send-email-srabbelier@gmail.com>

Add a 'done' command that causes fast-import to stop reading from the stream and exit.

If the new --done command line flag was passed on the command line (or a "feature done" declaration included at the start of the stream), make the 'done' command mandatory. So "git fast-import --done"'s input format will be prefix-free, making errors easier to detect when they show up as early termination at some convenient time of the upstream of a pipe writing to fast-import.

Another possible application of the 'done' command would to be allow a fast-import stream that is only a small part of a larger encapsulating stream to be easily parsed, leaving the file offset after the "done\n" so the other application can pick up from there. This patch does not teach fast-import to do that --- fast-import still uses buffered input (stdio).

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
---
  Unchanged
 Documentation/git-fast-import.txt |   25 ++++++++++++++++++++++
 fast-import.c                     |    8 +++++++
 t/t9300-fast-import.sh            |   42 +++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+), 0 deletions(-)
diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 249249a..0fc68a9 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -101,6 +101,12 @@ OPTIONS
 	when the `cat-blob` command is encountered in the stream.
 	The default behaviour is to write to `stdout`.
 
+--done::
+	Require a `done` command at the end of the stream.
+	This option might be useful for detecting errors that
+	cause the frontend to terminate before it has started to
+	write a stream.
+
 --export-pack-edges=<file>::
 	After creating a packfile, print a line of data to
 	<file> listing the filename of the packfile and the last
@@ -330,6 +336,11 @@ and control the current import process.  More detailed discussion
 	standard output.  This command is optional and is not needed
 	to perform an import.
 
+`done`::
+	Marks the end of the stream. This command is optional
+	unless the `done` feature was requested using the
+	`--done` command line option or `feature done` command.
+
 `cat-blob`::
 	Causes fast-import to print a blob in 'cat-file --batch'
 	format to the file descriptor set with `--cat-blob-fd` or
@@ -1015,6 +1026,11 @@ notes::
 	Versions of fast-import not supporting notes will exit
 	with a message indicating so.
 
+done::
+	Error out if the stream ends without a 'done' command.
+	Without this feature, errors causing the frontend to end
+	abruptly at a convenient point in the stream can go
+	undetected.
 
 `option`
 ~~~~~~~~
@@ -1044,6 +1060,15 @@ not be passed as option:
 * cat-blob-fd
 * force
 
+`done`
+~~~~~~
+If the `done` feature is not in use, treated as if EOF was read.
+This can be used to tell fast-import to finish early.
+
+If the `--done` command line option or `feature done` command is
+in use, the `done` command is mandatory and marks the end of the
+stream.
+
 Crash Reports
 -------------
 If fast-import is supplied invalid input it will terminate with a
diff --git a/fast-import.c b/fast-import.c
index 78d9786..8a8a915 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -354,6 +354,7 @@ static unsigned int cmd_save = 100;
 static uintmax_t next_mark;
 static struct strbuf new_data = STRBUF_INIT;
 static int seen_data_command;
+static int require_explicit_termination;
 
 /* Signal handling */
 static volatile sig_atomic_t checkpoint_requested;
@@ -3139,6 +3140,8 @@ static int parse_one_feature(const char *feature, int from_stream)
 		relative_marks_paths = 1;
 	} else if (!strcmp(feature, "no-relative-marks")) {
 		relative_marks_paths = 0;
+	} else if (!strcmp(feature, "done")) {
+		require_explicit_termination = 1;
 	} else if (!strcmp(feature, "force")) {
 		force_update = 1;
 	} else if (!strcmp(feature, "notes") || !strcmp(feature, "ls")) {
@@ -3288,6 +3291,8 @@ int main(int argc, const char **argv)
 			parse_reset_branch();
 		else if (!strcmp("checkpoint", command_buf.buf))
 			parse_checkpoint();
+		else if (!strcmp("done", command_buf.buf))
+			break;
 		else if (!prefixcmp(command_buf.buf, "progress "))
 			parse_progress();
 		else if (!prefixcmp(command_buf.buf, "feature "))
@@ -3307,6 +3312,9 @@ int main(int argc, const char **argv)
 	if (!seen_data_command)
 		parse_argv();
 
+	if (require_explicit_termination && feof(stdin))
+		die("stream ends early");
+
 	end_packfile();
 
 	dump_branches();
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 2a53640..f256475 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -2197,6 +2197,48 @@ test_expect_success 'R: quiet option results in no stats being output' '
     test_cmp empty output
 '
 
+test_expect_success 'R: feature done means terminating "done" is mandatory' '
+	echo feature done | test_must_fail git fast-import &&
+	test_must_fail git fast-import --done </dev/null
+'
+
+test_expect_success 'R: terminating "done" with trailing gibberish is ok' '
+	git fast-import <<-\EOF &&
+	feature done
+	done
+	trailing gibberish
+	EOF
+	git fast-import <<-\EOF
+	done
+	more trailing gibberish
+	EOF
+'
+
+test_expect_success 'R: terminating "done" within commit' '
+	cat >expect <<-\EOF &&
+	OBJID
+	:000000 100644 OBJID OBJID A	hello.c
+	:000000 100644 OBJID OBJID A	hello2.c
+	EOF
+	git fast-import <<-EOF &&
+	commit refs/heads/done-ends
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<EOT
+	Commit terminated by "done" command
+	EOT
+	M 100644 inline hello.c
+	data <<EOT
+	Hello, world.
+	EOT
+	C hello.c hello2.c
+	done
+	EOF
+	git rev-list done-ends |
+	git diff-tree -r --stdin --root --always |
+	sed -e "s/$_x40/OBJID/g" >actual &&
+	test_cmp expect actual
+'
+
 cat >input <<EOF
 option git non-existing-option
 EOF

-- 
1.7.5.1.292.g728120
Previous: Sverre RabbelierNext: Sverre Rabbelier
Message 15 of 29 in “remote-helper improvements”
  1. 00/23 remote-helper improvementsSverre Rabbelier, Jul 16, 2011
  2. 01/23 transport-helper: fix minor leak in push_refs_with_exportSverre Rabbelier, Jul 16, 2011
  3. 02/23 t5800: factor out some ref testsSverre Rabbelier, Jul 16, 2011
  4. 03/23 t5800: use skip_all instead of prereqSverre Rabbelier, Jul 16, 2011
  5. 04/23 t5800: document some non-functional parts of remote helpersSverre Rabbelier, Jul 16, 2011
  6. 05/23 git-remote-testgit: import non-HEAD refsSverre Rabbelier, Jul 16, 2011
  7. 06/23 transport-helper: don't feed bogus refs to export pushSverre Rabbelier, Jul 16, 2011
  8. 07/23 git_remote_helpers: push all refs during a non-local exportSverre Rabbelier, Jul 16, 2011
  9. Junio C HamanoJul 17, 2011
  10. Sverre RabbelierJul 23, 2011
  11. 08/23 remote-helpers: export GIT_DIR variable to helpersSverre Rabbelier, Jul 16, 2011
  12. 09/23 remote-curl: accept empty line as terminatorSverre Rabbelier, Jul 16, 2011
  13. 10/23 git-remote-testgit: only push for non-local repositoriesSverre Rabbelier, Jul 16, 2011
  14. 11/23 git-remote-testgit: fix error handlingSverre Rabbelier, Jul 16, 2011
  15. 12/23 fast-import: introduce 'done' commandSverre Rabbelier, Jul 16, 2011
  16. 13/23 fast-export: support done featureSverre Rabbelier, Jul 16, 2011
  17. 14/23 transport-helper: factor out push_update_refs_statusSverre Rabbelier, Jul 16, 2011
  18. 15/23 transport-helper: check status code of finish_commandSverre Rabbelier, Jul 16, 2011
  19. 16/23 transport-helper: use the new done feature where possibleSverre Rabbelier, Jul 16, 2011
  20. 17/23 transport-helper: update ref status after push with exportSverre Rabbelier, Jul 16, 2011
  21. 18/23 transport-helper: change import semanticsSverre Rabbelier, Jul 16, 2011
  22. Dmitry IvankovJul 18, 2011
  23. 19/23 transport-helper: Use capname for refspec capability tooSverre Rabbelier, Jul 16, 2011
  24. 20/23 transport-helper: implement marks location as capabilitySverre Rabbelier, Jul 16, 2011
  25. 21/23 transport-helper: die early on encountering deleted refsSverre Rabbelier, Jul 16, 2011
  26. 22/23 t5800: document inability to push new branch with old contentSverre Rabbelier, Jul 16, 2011
  27. 23/23 t5800: point out that deleting branches does not workSverre Rabbelier, Jul 16, 2011
  28. Jeff KingJul 18, 2011
  29. Sverre RabbelierJul 23, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.