From: René Scharfe Date: Mon, 15 Jun 2026 21:53:07 GMT Subject: Re: [PATCH] cat-file: speed up default format Message-ID: <10a33614-837f-4166-aa30-6de28b052692@web.de> In-Reply-To: <20260615170652.GB91269@coredump.intra.peff.net> On 6/15/26 7:06 PM, Jeff King wrote: > On Mon, Jun 15, 2026 at 12:53:26PM -0400, Jeff King wrote: > >> It uses per-atom callback functions which is nice and clean, though we >> might be able to do even better with a big ugly switch() statement. > > Being the curious sort, I swapped it out for a big switch statement. > Patch below, but it does not seem to be any faster. > > So the bottom line is I think you could gain a little bit of performance > by pre-parsing (versus strbuf_expand() on each object). Around 3% for > something that actually looks at the objects, though more like 15% if > for just dumping the objectnames. > > IMHO that is probably not worth it for a custom parsing system just for > cat-file. But if we were to finally unify ref-filter and cat-file (and > even --pretty=format) then it would probably worth doing this kind of > pre-parsing. It could be worth it for cat-file alone if we find the right balance, as it already does do a separate parsing step, but that is awkward with its mark_query checks all over the place and remembers only object property requirements and no other format string details. Making the opcodes small should be beneficial. We need only a handful of them, so a byte each should suffice. We can use a strbuf for that. We can also store literal characters in there. An opcode plus with a payload char incurs an overhead of 50%, which sounds high, but at least the default format only has two of them and it's much better than storing pointer plus size for an overhead of more than 90% in case of a single char. That gets us closer to native speed, at least on an Apple M1: Benchmark 1: ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' Time (mean ± σ): 992.7 ms ± 3.2 ms [User: 967.5 ms, System: 23.8 ms] Range (min … max): 990.1 ms … 1000.7 ms 10 runs Benchmark 2: ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' Time (mean ± σ): 991.8 ms ± 1.6 ms [User: 967.0 ms, System: 23.3 ms] Range (min … max): 989.3 ms … 994.4 ms 10 runs Benchmark 3: ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' Time (mean ± σ): 985.8 ms ± 2.9 ms [User: 960.5 ms, System: 23.6 ms] Range (min … max): 982.9 ms … 993.0 ms 10 runs Benchmark 4: ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)' Time (mean ± σ): 982.1 ms ± 3.2 ms [User: 956.7 ms, System: 23.6 ms] Range (min … max): 979.2 ms … 989.2 ms 10 runs Summary ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)' ran 1.00 ± 0.00 times faster than ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' 1.01 ± 0.00 times faster than ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' 1.01 ± 0.00 times faster than ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' A Ryzen laptop gives me noisy numbers that seem to suggest your switch-based code already won, but the more compact representation is at least not worse: Benchmark 1: ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' Time (mean ± σ): 397.5 ms ± 8.0 ms [User: 326.9 ms, System: 39.4 ms] Range (min … max): 388.1 ms … 410.0 ms 10 runs Benchmark 2: ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' Time (mean ± σ): 388.2 ms ± 4.2 ms [User: 318.2 ms, System: 39.2 ms] Range (min … max): 382.8 ms … 395.7 ms 10 runs Benchmark 3: ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' Time (mean ± σ): 385.5 ms ± 5.7 ms [User: 311.2 ms, System: 43.2 ms] Range (min … max): 377.0 ms … 392.9 ms 10 runs Benchmark 4: ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)' Time (mean ± σ): 397.5 ms ± 8.4 ms [User: 321.9 ms, System: 45.2 ms] Range (min … max): 382.1 ms … 406.5 ms 10 runs Summary ./git cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' ran 1.01 ± 0.02 times faster than ./git_switch cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' 1.03 ± 0.03 times faster than ./git_fp cat-file --batch-all-objects --batch-check='%(objectname)-%(objecttype)-%(objectsize)' 1.03 ± 0.03 times faster than ./git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype) %(objectsize)' René diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 0d1998784c..5667a13e93 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -36,8 +36,6 @@ enum batch_mode { BATCH_MODE_QUEUE_AND_DISPATCH, }; -struct format_item; - struct batch_options { struct list_objects_filter_options objects_filter; int enabled; @@ -50,7 +48,7 @@ struct batch_options { char input_delim; char output_delim; const char *format; - struct format_item *parsed_format; + struct strbuf parsed_format; }; static const char *force_path; @@ -320,30 +318,16 @@ struct expand_data { }; #define EXPAND_DATA_INIT { .mode = S_IFINVALID } -struct format_item { - enum { - FORMAT_TYPE_END = 0, - FORMAT_TYPE_LITERAL, - FORMAT_TYPE_OBJECTNAME, - FORMAT_TYPE_OBJECTTYPE, - FORMAT_TYPE_OBJECTSIZE, - FORMAT_TYPE_OBJECTSIZE_DISK, - FORMAT_TYPE_REST, - FORMAT_TYPE_DELTABASE, - FORMAT_TYPE_OBJECTMODE, - } type; - union { - struct { - const char *p; - size_t len; - } literal; - } u; - /* - * We could make a true tree here with child/next pointers, which would - * be necessary if we had recursive formats, like %(if). But for our - * simple formats for now it is enough to have a linear set of items, - * so we'll just allocate an array and terminate it with a NULL entry. - */ + +enum item_type { + FORMAT_TYPE_LITERAL, + FORMAT_TYPE_OBJECTNAME, + FORMAT_TYPE_OBJECTTYPE, + FORMAT_TYPE_OBJECTSIZE, + FORMAT_TYPE_OBJECTSIZE_DISK, + FORMAT_TYPE_REST, + FORMAT_TYPE_DELTABASE, + FORMAT_TYPE_OBJECTMODE, }; static int is_atom(const char *atom, const char *s, int slen) @@ -352,84 +336,66 @@ static int is_atom(const char *atom, const char *s, int slen) return alen == slen && !memcmp(atom, s, alen); } -static int parse_atom(struct format_item *fmt, const char *atom, int len, +static int parse_atom(struct strbuf *parsed_format, const char *atom, int len, struct expand_data *data) { if (is_atom("objectname", atom, len)) { - fmt->type = FORMAT_TYPE_OBJECTNAME; + strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTNAME); } else if (is_atom("objecttype", atom, len)) { data->info.typep = &data->type; - fmt->type = FORMAT_TYPE_OBJECTTYPE; + strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTTYPE); } else if (is_atom("objectsize", atom, len)) { data->info.sizep = &data->size; - fmt->type = FORMAT_TYPE_OBJECTSIZE; + strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTSIZE); } else if (is_atom("objectsize:disk", atom, len)) { data->info.disk_sizep = &data->disk_size; - fmt->type = FORMAT_TYPE_OBJECTSIZE_DISK; + strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTSIZE_DISK); } else if (is_atom("rest", atom, len)) { data->split_on_whitespace = 1; - fmt->type = FORMAT_TYPE_REST; + strbuf_addch(parsed_format, FORMAT_TYPE_REST); } else if (is_atom("deltabase", atom, len)) { data->info.delta_base_oid = &data->delta_base_oid; - fmt->type = FORMAT_TYPE_DELTABASE; + strbuf_addch(parsed_format, FORMAT_TYPE_DELTABASE); } else if (is_atom("objectmode", atom, len)) { - fmt->type = FORMAT_TYPE_OBJECTMODE; + strbuf_addch(parsed_format, FORMAT_TYPE_OBJECTMODE); } else return 0; return 1; } -static struct format_item *parse_format(const char *start, - struct expand_data *data) +static void parse_format(struct strbuf *parsed_format, + const char *start, struct expand_data *data) { - struct format_item *ret = NULL; - size_t nr = 0, alloc = 0; - while (1) { - const char *percent = strchrnul(start, '%'); const char *end; - if (percent != start) { - ALLOC_GROW(ret, nr + 1, alloc); - ret[nr].type = FORMAT_TYPE_LITERAL; - ret[nr].u.literal.p = start; - ret[nr].u.literal.len = percent - start; - nr++; + while (*start && *start != '%') { + strbuf_addch(parsed_format, FORMAT_TYPE_LITERAL); + strbuf_addch(parsed_format, *start++); } - if (!*percent) + if (!*start) break; - start = percent + 1; + start++; - ALLOC_GROW(ret, nr + 1, alloc); if (skip_prefix(start, "%", &start) || *start != '(') { - ret[nr].type = FORMAT_TYPE_LITERAL; - ret[nr].u.literal.p = "%"; - ret[nr].u.literal.len = 1; + strbuf_addch(parsed_format, FORMAT_TYPE_LITERAL); + strbuf_addch(parsed_format, '%'); } else if ((end = strchr(start + 1, ')')) && - parse_atom(&ret[nr], start + 1, end - start - 1, data)) { + parse_atom(parsed_format, start + 1, end - start - 1, data)) { start = end + 1; } else { strbuf_expand_bad_format(start, "cat-file"); } - nr++; } - - ALLOC_GROW(ret, nr + 1, alloc); - ret[nr].type = FORMAT_TYPE_END; - - return ret; } -static void expand_format(struct strbuf *sb, struct format_item *fmt, +static void expand_format(struct strbuf *sb, struct strbuf *parsed_format, struct expand_data *data) { - for (; fmt->type; fmt++) - switch (fmt->type) { - case FORMAT_TYPE_END: - BUG("we should have already left the loop!"); - break; + for (size_t i = 0; i < parsed_format->len; i++) + switch (parsed_format->buf[i]) { case FORMAT_TYPE_OBJECTNAME: strbuf_add_oid_hex(sb, &data->oid); break; @@ -453,7 +419,7 @@ static void expand_format(struct strbuf *sb, struct format_item *fmt, strbuf_addf(sb, "%06o", data->mode); break; case FORMAT_TYPE_LITERAL: - strbuf_add(sb, fmt->u.literal.p, fmt->u.literal.len); + strbuf_addch(sb, parsed_format->buf[++i]); } } @@ -641,7 +607,7 @@ static void batch_object_write(const char *obj_name, if (!opt->format) { print_default_format(scratch, data, opt); } else { - expand_format(scratch, opt->parsed_format, data); + expand_format(scratch, &opt->parsed_format, data); strbuf_addch(scratch, opt->output_delim); } @@ -1010,9 +976,8 @@ static int batch_objects(struct batch_options *opt) int save_warning; int retval = 0; - opt->parsed_format = parse_format(opt->format ? - opt->format : DEFAULT_FORMAT, - &data); + parse_format(&opt->parsed_format, + opt->format ? opt->format : DEFAULT_FORMAT, &data); if (opt->transform_mode) data.split_on_whitespace = 1; @@ -1152,6 +1117,7 @@ int cmd_cat_file(int argc, const char *exp_type = NULL, *obj_name = NULL; struct batch_options batch = { .objects_filter = LIST_OBJECTS_FILTER_INIT, + .parsed_format = STRBUF_INIT, }; int unknown_type = 0; int input_nul_terminated = 0;