git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] cat-file: speed up default format

From
René Scharfe <l.s.r@web.de>
Date
Jun 15, 2026, 21:53 UTC
Message-ID
<10a33614-837f-4166-aa30-6de28b052692@web.de>
In-Reply-To
<20260615170652.GB91269@coredump.intra.peff.net>
On 6/15/26 7:06 PM, Jeff King wrote:
Show 17 quoted lines
> On Mon, Jun 15, 2026 at 12:53:26PM -0400, Jeff King wrote:
> 
>> It uses per-atom callback functions which is nice and clean, though we
>> might be able to do even better with a big ugly switch() statement.
> 
> Being the curious sort, I swapped it out for a big switch statement.
> Patch below, but it does not seem to be any faster.
> 
> So the bottom line is I think you could gain a little bit of performance
> by pre-parsing (versus strbuf_expand() on each object). Around 3% for
> something that actually looks at the objects, though more like 15% if
> for just dumping the objectnames.
> 
> IMHO that is probably not worth it for a custom parsing system just for
> cat-file.  But if we were to finally unify ref-filter and cat-file (and
> even --pretty=format) then it would probably worth doing this kind of
> pre-parsing.

It could be worth it for cat-file alone if we find the right balance, as it already does do a separate parsing step, but that is awkward with its mark_query checks all over the place and remembers only object property requirements and no other format string details.

Making the opcodes small should be beneficial. We need only a handful of them, so a byte each should suffice. We can use a strbuf for that.

We can also store literal characters in there. An opcode plus with a payload char incurs an overhead of 50%, which sounds high, but at least the default format only has two of them and it's much better than storing pointer plus size for an overhead of more than 90% in case of a single char.

That gets us closer to native speed, at least on an Apple M1:
Benchmark 1: ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
  Time (mean ± σ):     992.7 ms ±   3.2 ms    [User: 967.5 ms, System: 23.8 ms]
  Range (min … max):   990.1 ms … 1000.7 ms    10 runs
Benchmark 2: ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
  Time (mean ± σ):     991.8 ms ±   1.6 ms    [User: 967.0 ms, System: 23.3 ms]
  Range (min … max):   989.3 ms … 994.4 ms    10 runs
Benchmark 3: ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
  Time (mean ± σ):     985.8 ms ±   2.9 ms    [User: 960.5 ms, System: 23.6 ms]
  Range (min … max):   982.9 ms … 993.0 ms    10 runs
Benchmark 4: ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)'
  Time (mean ± σ):     982.1 ms ±   3.2 ms    [User: 956.7 ms, System: 23.6 ms]
  Range (min … max):   979.2 ms … 989.2 ms    10 runs
Summary
  ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)' ran
    1.00 ± 0.00 times faster than ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
    1.01 ± 0.00 times faster than ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
    1.01 ± 0.00 times faster than ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'

A Ryzen laptop gives me noisy numbers that seem to suggest your switch-based code already won, but the more compact representation is at least not worse:

Benchmark 1: ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
  Time (mean ± σ):     397.5 ms ±   8.0 ms    [User: 326.9 ms, System: 39.4 ms]
  Range (min … max):   388.1 ms … 410.0 ms    10 runs
Benchmark 2: ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
  Time (mean ± σ):     388.2 ms ±   4.2 ms    [User: 318.2 ms, System: 39.2 ms]
  Range (min … max):   382.8 ms … 395.7 ms    10 runs
Benchmark 3: ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
  Time (mean ± σ):     385.5 ms ±   5.7 ms    [User: 311.2 ms, System: 43.2 ms]
  Range (min … max):   377.0 ms … 392.9 ms    10 runs
Benchmark 4: ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)'
  Time (mean ± σ):     397.5 ms ±   8.4 ms    [User: 321.9 ms, System: 45.2 ms]
  Range (min … max):   382.1 ms … 406.5 ms    10 runs
Summary
  ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' ran
    1.01 ± 0.02 times faster than ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
    1.03 ± 0.03 times faster than ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)'
    1.03 ± 0.03 times faster than ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)'
René
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 0d1998784c..5667a13e93 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -36,8 +36,6 @@ enum batch_mode {
 	BATCH_MODE_QUEUE_AND_DISPATCH,
 };
 
-struct format_item;
-
 struct batch_options {
 	struct list_objects_filter_options objects_filter;
 	int enabled;
@@ -50,7 +48,7 @@ struct batch_options {
 	char input_delim;
 	char output_delim;
 	const char *format;
-	struct format_item *parsed_format;
+	struct strbuf parsed_format;
 };
 
 static const char *force_path;
@@ -320,30 +318,16 @@ struct expand_data {
 };
 #define EXPAND_DATA_INIT  { .mode = S_IFINVALID }
 
-struct format_item {
-	enum {
-		FORMAT_TYPE_END = 0,
-		FORMAT_TYPE_LITERAL,
-		FORMAT_TYPE_OBJECTNAME,
-		FORMAT_TYPE_OBJECTTYPE,
-		FORMAT_TYPE_OBJECTSIZE,
-		FORMAT_TYPE_OBJECTSIZE_DISK,
-		FORMAT_TYPE_REST,
-		FORMAT_TYPE_DELTABASE,
-		FORMAT_TYPE_OBJECTMODE,
-	} type;
-	union {
-		struct {
-			const char *p;
-			size_t len;
-		} literal;
-	} u;
-	/*
-	 * We could make a true tree here with child/next pointers, which would
-	 * be necessary if we had recursive formats, like %(if). But for our
-	 * simple formats for now it is enough to have a linear set of items,
-	 * so we'll just allocate an array and terminate it with a NULL entry.
-	 */
+
+enum item_type {
+	FORMAT_TYPE_LITERAL,
+	FORMAT_TYPE_OBJECTNAME,
+	FORMAT_TYPE_OBJECTTYPE,
+	FORMAT_TYPE_OBJECTSIZE,
+	FORMAT_TYPE_OBJECTSIZE_DISK,
+	FORMAT_TYPE_REST,
+	FORMAT_TYPE_DELTABASE,
+	FORMAT_TYPE_OBJECTMODE,
 };
 
 static int is_atom(const char *atom, const char *s, int slen)
@@ -352,84 +336,66 @@ static int is_atom(const char *atom, const char *s, int slen)
 	return alen == slen && !memcmp(atom, s, alen);
 }
 
-static int parse_atom(struct format_item *fmt, const char *atom, int len,
+static int parse_atom(struct strbuf *parsed_format, const char *atom, int len,
 		      struct expand_data *data)
 {
 	if (is_atom("objectname", atom, len)) {
-		fmt->type = FORMAT_TYPE_OBJECTNAME;
+		strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTNAME);
 	} else if (is_atom("objecttype", atom, len)) {
 		data->info.typep = &data->type;
-		fmt->type = FORMAT_TYPE_OBJECTTYPE;
+		strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTTYPE);
 	} else if (is_atom("objectsize", atom, len)) {
 		data->info.sizep = &data->size;
-		fmt->type = FORMAT_TYPE_OBJECTSIZE;
+		strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTSIZE);
 	} else if (is_atom("objectsize:disk", atom, len)) {
 		data->info.disk_sizep = &data->disk_size;
-		fmt->type = FORMAT_TYPE_OBJECTSIZE_DISK;
+		strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTSIZE_DISK);
 	} else if (is_atom("rest", atom, len)) {
 		data->split_on_whitespace = 1;
-		fmt->type = FORMAT_TYPE_REST;
+		strbuf_addch(parsed_format, FORMAT_TYPE_REST);
 	} else if (is_atom("deltabase", atom, len)) {
 		data->info.delta_base_oid = &data->delta_base_oid;
-		fmt->type = FORMAT_TYPE_DELTABASE;
+		strbuf_addch(parsed_format, FORMAT_TYPE_DELTABASE);
 	} else if (is_atom("objectmode", atom, len)) {
-		fmt->type = FORMAT_TYPE_OBJECTMODE;
+		strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTMODE);
 	} else
 		return 0;
 	return 1;
 }
 
-static struct format_item *parse_format(const char *start,
-					struct expand_data *data)
+static void parse_format(struct strbuf *parsed_format,
+			 const char *start, struct expand_data *data)
 {
-	struct format_item *ret = NULL;
-	size_t nr = 0, alloc = 0;
-
 	while (1) {
-		const char *percent = strchrnul(start, '%');
 		const char *end;
 
-		if (percent != start) {
-			ALLOC_GROW(ret, nr + 1, alloc);
-			ret[nr].type = FORMAT_TYPE_LITERAL;
-			ret[nr].u.literal.p = start;
-			ret[nr].u.literal.len = percent - start;
-			nr++;
+		while (*start && *start != '%') {
+			strbuf_addch(parsed_format, FORMAT_TYPE_LITERAL);
+			strbuf_addch(parsed_format, *start++);
 		}
 
-		if (!*percent)
+		if (!*start)
 			break;
 
-		start = percent + 1;
+		start++;
 
-		ALLOC_GROW(ret, nr + 1, alloc);
 		if (skip_prefix(start, "%", &start) || *start != '(') {
-			ret[nr].type = FORMAT_TYPE_LITERAL;
-			ret[nr].u.literal.p = "%";
-			ret[nr].u.literal.len = 1;
+			strbuf_addch(parsed_format, FORMAT_TYPE_LITERAL);
+			strbuf_addch(parsed_format, '%');
 		} else if ((end = strchr(start + 1, ')')) &&
-			   parse_atom(&ret[nr], start + 1, end - start - 1, data)) {
+			   parse_atom(parsed_format, start + 1, end - start - 1, data)) {
 			start = end + 1;
 		} else {
 			strbuf_expand_bad_format(start, "cat-file");
 		}
-		nr++;
 	}
-
-	ALLOC_GROW(ret, nr + 1, alloc);
-	ret[nr].type = FORMAT_TYPE_END;
-
-	return ret;
 }
 
-static void expand_format(struct strbuf *sb, struct format_item *fmt,
+static void expand_format(struct strbuf *sb, struct strbuf *parsed_format,
 			  struct expand_data *data)
 {
-	for (; fmt->type; fmt++)
-		switch (fmt->type) {
-		case FORMAT_TYPE_END:
-			BUG("we should have already left the loop!");
-			break;
+	for (size_t i = 0; i < parsed_format->len; i++)
+		switch (parsed_format->buf[i]) {
 		case FORMAT_TYPE_OBJECTNAME:
 			strbuf_add_oid_hex(sb, &data->oid);
 			break;
@@ -453,7 +419,7 @@ static void expand_format(struct strbuf *sb, struct format_item *fmt,
 				strbuf_addf(sb, "%06o", data->mode);
 			break;
 		case FORMAT_TYPE_LITERAL:
-			strbuf_add(sb, fmt->u.literal.p, fmt->u.literal.len);
+			strbuf_addch(sb, parsed_format->buf[++i]);
 		}
 }
 
@@ -641,7 +607,7 @@ static void batch_object_write(const char *obj_name,
 	if (!opt->format) {
 		print_default_format(scratch, data, opt);
 	} else {
-		expand_format(scratch, opt->parsed_format, data);
+		expand_format(scratch, &opt->parsed_format, data);
 		strbuf_addch(scratch, opt->output_delim);
 	}
 
@@ -1010,9 +976,8 @@ static int batch_objects(struct batch_options *opt)
 	int save_warning;
 	int retval = 0;
 
-	opt->parsed_format = parse_format(opt->format ?
-					  opt->format : DEFAULT_FORMAT,
-					  &data);
+	parse_format(&opt->parsed_format,
+		     opt->format ? opt->format : DEFAULT_FORMAT, &data);
 	if (opt->transform_mode)
 		data.split_on_whitespace = 1;
 
@@ -1152,6 +1117,7 @@ int cmd_cat_file(int argc,
 	const char *exp_type = NULL, *obj_name = NULL;
 	struct batch_options batch = {
 		.objects_filter = LIST_OBJECTS_FILTER_INIT,
+		.parsed_format = STRBUF_INIT,
 	};
 	int unknown_type = 0;
 	int input_nul_terminated = 0;
Previous: Jeff KingNext: Jeff King
Message 5 of 10 in “cat-file: speed up default format”
  1. cat-file: speed up default formatRené Scharfe, Jun 14, 2026
  2. Patrick SteinhardtJun 15, 2026
  3. Jeff KingJun 15, 2026
  4. Jeff KingJun 15, 2026
  5. René ScharfeJun 15, 2026
  6. Jeff KingJun 16, 2026
  7. René ScharfeJun 16, 2026
  8. Junio C HamanoJun 16, 2026
  9. René ScharfeJun 15, 2026
  10. Jeff KingJun 16, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.