git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 4/7] cat-file: add --buffer option

From
Jeff King <peff@peff.net>
Date
Jun 22, 2015, 10:45 UTC
Message-ID
<20150622104517.GD14475@peff.net>
In-Reply-To
<20150622103321.GB12584@peff.net>

We use a direct write() to output the results of --batch and --batch-check. This is good for processes feeding the input and reading the output interactively, but it introduces measurable overhead if you do not want this feature. For example, on linux.git:

  $ git rev-list --objects --all | cut -d' ' -f1 >objects
  $ time git cat-file --batch-check='%(objectsize)' \
          <objects >/dev/null
  real    0m5.440s
  user    0m5.060s
  sys     0m0.384s
This patch adds an option to use regular stdio buffering:
  $ time git cat-file --batch-check='%(objectsize)' \
          --buffer <objects >/dev/null
  real    0m4.975s
  user    0m4.888s
  sys     0m0.092s
Signed-off-by: Jeff King <peff@peff.net>
---
This selectively uses fwrite or write_or_die, depending on the buffer
setting. Another option would be to just always use fwrite(), and then
selectively fflush(). It feels kind of wasteful in the non-buffered
case, as it's just another layer to write through. OTOH, the cost of
writing a line into the buffer only to flush is probably dwarfed by the
system call of actually flushing.

If we went that direction, we could probably simplify the code a bit (both getting rid of the batch_write function I call here, and dropping a bunch of existing fflush() calls, where we must flush any time we use printf for its formatting capabilities).

I also considered that the "--buffer" case is likely to be the common one. We cannot flip the default, though, as it would break any existing callers (who would need to specify "--no-buffer"). We can do the usual deprecation dance, but I don't know if it is worth it for a plumbing command like this.

 Documentation/git-cat-file.txt |  7 +++++++
 builtin/cat-file.c             | 26 +++++++++++++++++++-------
 2 files changed, 26 insertions(+), 7 deletions(-)
diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt
index 319ab4c..0058bd4 100644
--- a/Documentation/git-cat-file.txt
+++ b/Documentation/git-cat-file.txt
@@ -69,6 +69,13 @@ OPTIONS
 	not be combined with any other options or arguments.  See the
 	section `BATCH OUTPUT` below for details.
 
+--buffer::
+	Normally batch output is flushed after each object is output, so
+	that a process can interactively read and write from
+	`cat-file`. With this option, the output uses normal stdio
+	buffering; this is much more efficient when invoking
+	`--batch-check` on a large number of objects.
+
 --allow-unknown-type::
 	Allow -s or -t to query broken/corrupt objects of unknown type.
 
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index d4101b7..741e100 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -14,6 +14,7 @@ struct batch_options {
 	int enabled;
 	int follow_symlinks;
 	int print_contents;
+	int buffer_output;
 	const char *format;
 };
 
@@ -211,14 +212,25 @@ static size_t expand_format(struct strbuf *sb, const char *start, void *data)
 	return end - start + 1;
 }
 
-static void print_object_or_die(int fd, struct expand_data *data)
+static void batch_write(struct batch_options *opt, const void *data, int len)
+{
+	if (opt->buffer_output) {
+		if (fwrite(data, 1, len, stdout) != len)
+			die_errno("unable to write to stdout");
+	} else
+		write_or_die(1, data, len);
+}
+
+static void print_object_or_die(struct batch_options *opt, struct expand_data *data)
 {
 	const unsigned char *sha1 = data->sha1;
 
 	assert(data->info.typep);
 
 	if (data->type == OBJ_BLOB) {
-		if (stream_blob_to_fd(fd, sha1, NULL, 0) < 0)
+		if (opt->buffer_output)
+			fflush(stdout);
+		if (stream_blob_to_fd(1, sha1, NULL, 0) < 0)
 			die("unable to stream %s to stdout", sha1_to_hex(sha1));
 	}
 	else {
@@ -234,12 +246,11 @@ static void print_object_or_die(int fd, struct expand_data *data)
 		if (data->info.sizep && size != data->size)
 			die("object %s changed size!?", sha1_to_hex(sha1));
 
-		write_or_die(fd, contents, size);
+		batch_write(opt, contents, size);
 		free(contents);
 	}
 }
 
-
 static int batch_one_object(const char *obj_name, struct batch_options *opt,
 			    struct expand_data *data)
 {
@@ -294,12 +305,12 @@ static int batch_one_object(const char *obj_name, struct batch_options *opt,
 
 	strbuf_expand(&buf, opt->format, expand_format, data);
 	strbuf_addch(&buf, '\n');
-	write_or_die(1, buf.buf, buf.len);
+	batch_write(opt, buf.buf, buf.len);
 	strbuf_release(&buf);
 
 	if (opt->print_contents) {
-		print_object_or_die(1, data);
-		write_or_die(1, "\n", 1);
+		print_object_or_die(opt, data);
+		batch_write(opt, "\n", 1);
 	}
 	return 0;
 }
@@ -415,6 +426,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)
 			    N_("for blob objects, run textconv on object's content"), 'c'),
 		OPT_BOOL(0, "allow-unknown-type", &unknown_type,
 			  N_("allow -s and -t to work with broken/corrupt objects")),
+		OPT_BOOL(0, "buffer", &batch.buffer_output, N_("buffer --batch output")),
 		{ OPTION_CALLBACK, 0, "batch", &batch, "format",
 			N_("show info and content of objects fed from the standard input"),
 			PARSE_OPT_OPTARG, batch_option_callback },

-- 
2.4.4.719.g3984bc6
Previous: Jeff KingNext: Jeff King
Message 35 of 51 in “Improvements to parse-options and a new filter-objects command”
  1. Charles BaileyJun 19, 2015
  2. 1/3 Correct test-parse-options to handle negative intsCharles Bailey, Jun 19, 2015
  3. Junio C HamanoJun 19, 2015
  4. 2/3 Move unsigned long option parsing out of pack-objects.cCharles Bailey, Jun 19, 2015
  5. Remi Galan AlfonsoJun 19, 2015
  6. Charles BaileyJun 19, 2015
  7. Junio C HamanoJun 19, 2015
  8. Junio C HamanoJun 19, 2015
  9. Jakub NarębskiJun 20, 2015
  10. Jakub NarębskiJun 19, 2015
  11. Charles BaileyJun 20, 2015
  12. Junio C HamanoJun 20, 2015
  13. 3/3 Add filter-objects commandCharles Bailey, Jun 19, 2015
  14. Jeff KingJun 19, 2015
  15. Charles BaileyJun 19, 2015
  16. Jeff KingJun 19, 2015
  17. Junio C HamanoJun 19, 2015
  18. John KeepingJun 19, 2015
  19. Charles BaileyJun 19, 2015
  20. Improvements to integer option parsingCharles Bailey, Jun 21, 2015
  21. 1/2 Correct test-parse-options to handle negative intsCharles Bailey, Jun 21, 2015
  22. 2/2 Move unsigned long option parsing out of pack-objects.cCharles Bailey, Jun 21, 2015
  23. Charles BaileyJun 21, 2015
  24. Junio C HamanoJun 22, 2015
  25. Junio C HamanoJun 22, 2015
  26. Junio C HamanoJun 22, 2015
  27. Charles BaileyJun 22, 2015
  28. Fast enumeration of objectsCharles Bailey, Jun 21, 2015
  29. Add list-all-objects commandCharles Bailey, Jun 21, 2015
  30. Jeff KingJun 22, 2015
  31. Jeff KingJun 22, 2015
  32. 1/7 for_each_packed_object: automatically open pack indexJeff King, Jun 22, 2015
  33. 2/7 cat-file: minor style fix in options listJeff King, Jun 22, 2015
  34. 3/7 cat-file: move batch_options definition to top of fileJeff King, Jun 22, 2015
  35. 4/7 cat-file: add --buffer optionJeff King, Jun 22, 2015
  36. 5/7 cat-file: stop returning value from batch_one_objectJeff King, Jun 22, 2015
  37. 6/7 cat-file: split batch_one_object into two stagesJeff King, Jun 22, 2015
  38. 7/7 cat-file: add --batch-all-objects optionJeff King, Jun 22, 2015
  39. Eric SunshineJun 26, 2015
  40. Jeff KingJun 26, 2015
  41. 8/7 cat-file: sort and de-dup output of --batch-all-objectsJeff King, Jun 22, 2015
  42. Charles BaileyJun 22, 2015
  43. Jeff KingJun 22, 2015
  44. Charles BaileyJun 22, 2015
  45. Junio C HamanoJun 22, 2015
  46. Jeff KingJun 22, 2015
  47. Charles BaileyJun 22, 2015
  48. Duy NguyenJun 22, 2015
  49. Jeff KingJun 22, 2015
  50. Jeff KingJun 22, 2015
  51. Junio C HamanoJun 22, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.