{"thread":{"id":"55998","subject":"[PATCH] speed up alt_odb_usable() with many alternates","startedAt":"2021-06-24T00:58:08Z","lastAt":"2021-09-29T15:55:52Z","messageCount":99,"participants":["Eric Wong","René Scharfe","Junio C Hamano","Ævar Arnfjörð Bjarmason","Andrzej Hunt","Carlo Arenas","Carlo Marcelo Arenas Belón","Bagas Sanjaya","Phillip Wood","Jeff King","Eric Sunshine","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"428352","messageId":"20210624005806.12079-1-e@80x24.org","threadId":"55998","inReplyTo":null,"subject":"[PATCH] speed up alt_odb_usable() with many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-24T00:58:06Z","receivedAt":"2021-06-24T00:58:08Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"With many alternates, the duplicate check in alt_odb_usable()\nwastes many cycles doing repeated fspathcmp() on every existing\nalternate. Use a khash to speed up lookups by odb->path.\n\nSince the kh_put_* API uses the supplied key without\nduplicating it, we also take advantage of it to replace both\nxstrdup() and strbuf_release() in link_alt_odb_entry() with\nstrbuf_detach() to avoid the allocation and copy.\n\nIn a test repository with 50K alternates and each of those 50K\nalternates having one alternate each (for a total of 100K total\nalternates); this speeds up lookup of a non-existent blob from\nover 16 minutes to roughly 8 seconds on my busy workstation.\n\nNote: all underlying git object directories were small and\nunpacked with only loose objects and no packs. Having to load\npacks increases times significantly.\n\nSigned-off-by: Eric Wong \n---\n Note: this project I'm doing this for probably won't have 100K\n alternates yet, but ~60K is a possibility. I hope to find\n more speedups along these lines.\n\n object-file.c | 33 ++++++++++++++++++++++-----------\n object-store.h | 17 +++++++++++++++++\n object.c | 2 ++\n 3 files changed, 41 insertions(+), 11 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..304af3a172 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -517,9 +517,9 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n */\n static int alt_odb_usable(struct raw_object_store *o,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir)\n+\t\t\t const char *normalized_objdir, khiter_t *pos)\n {\n-\tstruct object_directory *odb;\n+\tint r;\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path->buf)) {\n@@ -533,14 +533,22 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n \t */\n-\tfor (odb = o->odb; odb; odb = odb->next) {\n-\t\tif (!fspathcmp(path->buf, odb->path))\n-\t\t\treturn 0;\n+\tif (!o->odb_by_path) {\n+\t\tkhiter_t p;\n+\n+\t\to->odb_by_path = kh_init_odb_path_map();\n+\t\tassert(!o->odb->next);\n+\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n+\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t\tassert(r == 1); /* never used */\n+\t\tkh_value(o->odb_by_path, p) = o->odb;\n \t}\n \tif (!fspathcmp(path->buf, normalized_objdir))\n \t\treturn 0;\n-\n-\treturn 1;\n+\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n+\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n+\treturn r == 0 ? 0 : 1;\n }\n \n /*\n@@ -566,6 +574,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n+\tkhiter_t pos;\n \n \tif (!is_absolute_path(entry) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -587,23 +596,25 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \twhile (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')\n \t\tstrbuf_setlen(&pathbuf, pathbuf.len - 1);\n \n-\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {\n+\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir, &pos)) {\n \t\tstrbuf_release(&pathbuf);\n \t\treturn -1;\n \t}\n \n \tCALLOC_ARRAY(ent, 1);\n-\tent->path = xstrdup(pathbuf.buf);\n+\t/* pathbuf.buf is already in r->objects->odb_by_path */\n+\tent->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*r->objects->odb_tail = ent;\n \tr->objects->odb_tail = &(ent->next);\n \tent->next = NULL;\n+\tassert(r->objects->odb_by_path);\n+\tkh_value(r->objects->odb_by_path, pos) = ent;\n \n \t/* recursively add alternates */\n-\tread_info_alternates(r, pathbuf.buf, depth + 1);\n+\tread_info_alternates(r, ent->path, depth + 1);\n \n-\tstrbuf_release(&pathbuf);\n \treturn 0;\n }\n \ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb..20c1cedb75 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -7,6 +7,8 @@\n #include \"oid-array.h\"\n #include \"strbuf.h\"\n #include \"thread-utils.h\"\n+#include \"khash.h\"\n+#include \"dir.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -30,6 +32,19 @@ struct object_directory {\n \tchar *path;\n };\n \n+static inline int odb_path_eq(const char *a, const char *b)\n+{\n+\treturn !fspathcmp(a, b);\n+}\n+\n+static inline int odb_path_hash(const char *str)\n+{\n+\treturn ignore_case ? strihash(str) : __ac_X31_hash_string(str);\n+}\n+\n+KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n+\tstruct object_directory *, 1, odb_path_hash, odb_path_eq);\n+\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct object_directory *, void *);\n@@ -116,6 +131,8 @@ struct raw_object_store {\n \t */\n \tstruct object_directory *odb;\n \tstruct object_directory **odb_tail;\n+\tkh_odb_path_map_t *odb_by_path;\n+\n \tint loaded_alternates;\n \n \t/*\ndiff --git a/object.c b/object.c\nindex 14188453c5..2b3c075a15 100644\n--- a/object.c\n+++ b/object.c\n@@ -511,6 +511,8 @@ static void free_object_directories(struct raw_object_store *o)\n \t\tfree_object_directory(o->odb);\n \t\to->odb = next;\n \t}\n+\tkh_destroy_odb_path_map(o->odb_by_path);\n+\to->odb_by_path = NULL;\n }\n \n void raw_object_store_clear(struct raw_object_store *o)\n"},{"id":"428537","messageId":"20210627024718.25383-1-e@80x24.org","threadId":"55998","inReplyTo":"20210624005806.12079-1-e@80x24.org","subject":"[PATCH 0/5] optimizations for many odb alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-27T02:47:13Z","receivedAt":"2021-06-27T02:47:21Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Cc-ing Rene and Peff for their previous work on loose object\ncaching speedups (and also Peff on crit-bit trees).\n\nI'm expecting a use case involving tens of thousands of\nrepos being tied together by alternates. I realize this is\nan odd case, but there's some fairly small changes that\ngive significant speedups and memory savings.\n\nI can't seem to get consistent benchmarks on my workstation\n(since it doubles as a public-facing server :x), but things\nseem generally in the ballpark...\n\n1/5 is a resend and the biggest obvious time improvement\n(at some cost to space).\n\n2/5 and 4/5 are pretty obvious; 3/5 should be obvious, too,\nbut my arithmetic is terrible :x\n\n5/5 is a big (and easily measured) space improvement that\nwill negate space regression caused by 1/5 (and then some).\nI'm not sure if there's much or any change in time in\neither direction, though...\n\nEric Wong (5):\n speed up alt_odb_usable() with many alternates\n avoid strlen via strbuf_addstr in link_alt_odb_entry\n make object_directory.loose_objects_subdir_seen a bitmap\n oidcpy_with_padding: constify `src' arg\n oidtree: a crit-bit tree for odb_loose_cache\n\n Makefile | 3 +\n alloc.c | 6 ++\n alloc.h | 1 +\n cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n cbtree.h | 56 ++++++++++++++\n hash.h | 2 +-\n object-file.c | 68 +++++++++-------\n object-name.c | 28 +++----\n object-store.h | 24 +++++-\n object.c | 2 +\n oidtree.c | 94 ++++++++++++++++++++++\n oidtree.h | 29 +++++++\n t/helper/test-oidtree.c | 45 +++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0069-oidtree.sh | 52 +++++++++++++\n 16 files changed, 530 insertions(+), 49 deletions(-)\n create mode 100644 cbtree.c\n create mode 100644 cbtree.h\n create mode 100644 oidtree.c\n create mode 100644 oidtree.h\n create mode 100644 t/helper/test-oidtree.c\n create mode 100755 t/t0069-oidtree.sh\n"},{"id":"428538","messageId":"20210627024718.25383-3-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH 2/5] avoid strlen via strbuf_addstr in link_alt_odb_entry","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-27T02:47:15Z","receivedAt":"2021-06-27T02:47:27Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"We can save a few milliseconds (across 100K odbs) by using\nstrbuf_addbuf() instead of strbuf_addstr() by passing `entry' as\na strbuf pointer rather than a \"const char *\".\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 304af3a172..6be43c2b60 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -569,18 +569,18 @@ static int alt_odb_usable(struct raw_object_store *o,\n static void read_info_alternates(struct repository *r,\n \t\t\t\t const char *relative_base,\n \t\t\t\t int depth);\n-static int link_alt_odb_entry(struct repository *r, const char *entry,\n+static int link_alt_odb_entry(struct repository *r, const struct strbuf *entry,\n \tconst char *relative_base, int depth, const char *normalized_objdir)\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tkhiter_t pos;\n \n-\tif (!is_absolute_path(entry) && relative_base) {\n+\tif (!is_absolute_path(entry->buf) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n \t\tstrbuf_addch(&pathbuf, '/');\n \t}\n-\tstrbuf_addstr(&pathbuf, entry);\n+\tstrbuf_addbuf(&pathbuf, entry);\n \n \tif (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {\n \t\terror(_(\"unable to normalize alternate object path: %s\"),\n@@ -671,7 +671,7 @@ static void link_alt_odb_entries(struct repository *r, const char *alt,\n \t\talt = parse_alt_odb_entry(alt, sep, &entry);\n \t\tif (!entry.len)\n \t\t\tcontinue;\n-\t\tlink_alt_odb_entry(r, entry.buf,\n+\t\tlink_alt_odb_entry(r, &entry,\n \t\t\t\t relative_base, depth, objdirbuf.buf);\n \t}\n \tstrbuf_release(&entry);\n"},{"id":"428539","messageId":"20210627024718.25383-2-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH 1/5] speed up alt_odb_usable() with many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-27T02:47:14Z","receivedAt":"2021-06-27T02:47:27Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"With many alternates, the duplicate check in alt_odb_usable()\nwastes many cycles doing repeated fspathcmp() on every existing\nalternate. Use a khash to speed up lookups by odb->path.\n\nSince the kh_put_* API uses the supplied key without\nduplicating it, we also take advantage of it to replace both\nxstrdup() and strbuf_release() in link_alt_odb_entry() with\nstrbuf_detach() to avoid the allocation and copy.\n\nIn a test repository with 50K alternates and each of those 50K\nalternates having one alternate each (for a total of 100K total\nalternates); this speeds up lookup of a non-existent blob from\nover 16 minutes to roughly 2.7 seconds on my busy workstation.\n\nNote: all underlying git object directories were small and\nunpacked with only loose objects and no packs. Having to load\npacks increases times significantly.\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 33 ++++++++++++++++++++++-----------\n object-store.h | 17 +++++++++++++++++\n object.c | 2 ++\n 3 files changed, 41 insertions(+), 11 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..304af3a172 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -517,9 +517,9 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n */\n static int alt_odb_usable(struct raw_object_store *o,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir)\n+\t\t\t const char *normalized_objdir, khiter_t *pos)\n {\n-\tstruct object_directory *odb;\n+\tint r;\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path->buf)) {\n@@ -533,14 +533,22 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n \t */\n-\tfor (odb = o->odb; odb; odb = odb->next) {\n-\t\tif (!fspathcmp(path->buf, odb->path))\n-\t\t\treturn 0;\n+\tif (!o->odb_by_path) {\n+\t\tkhiter_t p;\n+\n+\t\to->odb_by_path = kh_init_odb_path_map();\n+\t\tassert(!o->odb->next);\n+\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n+\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t\tassert(r == 1); /* never used */\n+\t\tkh_value(o->odb_by_path, p) = o->odb;\n \t}\n \tif (!fspathcmp(path->buf, normalized_objdir))\n \t\treturn 0;\n-\n-\treturn 1;\n+\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n+\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n+\treturn r == 0 ? 0 : 1;\n }\n \n /*\n@@ -566,6 +574,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n+\tkhiter_t pos;\n \n \tif (!is_absolute_path(entry) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -587,23 +596,25 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \twhile (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')\n \t\tstrbuf_setlen(&pathbuf, pathbuf.len - 1);\n \n-\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {\n+\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir, &pos)) {\n \t\tstrbuf_release(&pathbuf);\n \t\treturn -1;\n \t}\n \n \tCALLOC_ARRAY(ent, 1);\n-\tent->path = xstrdup(pathbuf.buf);\n+\t/* pathbuf.buf is already in r->objects->odb_by_path */\n+\tent->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*r->objects->odb_tail = ent;\n \tr->objects->odb_tail = &(ent->next);\n \tent->next = NULL;\n+\tassert(r->objects->odb_by_path);\n+\tkh_value(r->objects->odb_by_path, pos) = ent;\n \n \t/* recursively add alternates */\n-\tread_info_alternates(r, pathbuf.buf, depth + 1);\n+\tread_info_alternates(r, ent->path, depth + 1);\n \n-\tstrbuf_release(&pathbuf);\n \treturn 0;\n }\n \ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb..20c1cedb75 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -7,6 +7,8 @@\n #include \"oid-array.h\"\n #include \"strbuf.h\"\n #include \"thread-utils.h\"\n+#include \"khash.h\"\n+#include \"dir.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -30,6 +32,19 @@ struct object_directory {\n \tchar *path;\n };\n \n+static inline int odb_path_eq(const char *a, const char *b)\n+{\n+\treturn !fspathcmp(a, b);\n+}\n+\n+static inline int odb_path_hash(const char *str)\n+{\n+\treturn ignore_case ? strihash(str) : __ac_X31_hash_string(str);\n+}\n+\n+KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n+\tstruct object_directory *, 1, odb_path_hash, odb_path_eq);\n+\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct object_directory *, void *);\n@@ -116,6 +131,8 @@ struct raw_object_store {\n \t */\n \tstruct object_directory *odb;\n \tstruct object_directory **odb_tail;\n+\tkh_odb_path_map_t *odb_by_path;\n+\n \tint loaded_alternates;\n \n \t/*\ndiff --git a/object.c b/object.c\nindex 14188453c5..2b3c075a15 100644\n--- a/object.c\n+++ b/object.c\n@@ -511,6 +511,8 @@ static void free_object_directories(struct raw_object_store *o)\n \t\tfree_object_directory(o->odb);\n \t\to->odb = next;\n \t}\n+\tkh_destroy_odb_path_map(o->odb_by_path);\n+\to->odb_by_path = NULL;\n }\n \n void raw_object_store_clear(struct raw_object_store *o)\n"},{"id":"428540","messageId":"20210627024718.25383-4-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH 3/5] make object_directory.loose_objects_subdir_seen a bitmap","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-27T02:47:16Z","receivedAt":"2021-06-27T02:47:28Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"There's no point in using 8 bits per-directory when 1 bit\nwill do. This saves us 224 bytes per object directory, which\nends up being 22MB when dealing with 100K alternates.\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 10 +++++++---\n object-store.h | 2 +-\n 2 files changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 6be43c2b60..2c8b9c05f9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2463,12 +2463,16 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t BM_SIZE = sizeof(odb->loose_objects_subdir_seen[0]) * CHAR_BIT;\n+\tuint32_t *bitmap;\n+\tuint32_t bit = 1 << (subdir_nr % BM_SIZE);\n \n \tif (subdir_nr < 0 ||\n-\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n+\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen) * BM_SIZE)\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tif (odb->loose_objects_subdir_seen[subdir_nr])\n+\tbitmap = &odb->loose_objects_subdir_seen[subdir_nr / BM_SIZE];\n+\tif (*bitmap & bit)\n \t\treturn &odb->loose_objects_cache[subdir_nr];\n \n \tstrbuf_addstr(&buf, odb->path);\n@@ -2476,7 +2480,7 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n \t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n-\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n+\t*bitmap |= bit;\n \tstrbuf_release(&buf);\n \treturn &odb->loose_objects_cache[subdir_nr];\n }\ndiff --git a/object-store.h b/object-store.h\nindex 20c1cedb75..8fcddf3e65 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -22,7 +22,7 @@ struct object_directory {\n \t *\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n-\tchar loose_objects_subdir_seen[256];\n+\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n \tstruct oid_array loose_objects_cache[256];\n \n \t/*\n"},{"id":"428541","messageId":"20210627024718.25383-5-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH 4/5] oidcpy_with_padding: constify `src' arg","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-27T02:47:17Z","receivedAt":"2021-06-27T02:47:34Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"As with `oidcpy', the source struct will not be modified and\nthis will allow an upcoming const-correct caller to use it.\n\nSigned-off-by: Eric Wong \n---\n hash.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 9c6df4d952..27a180248f 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -265,7 +265,7 @@ static inline void oidcpy(struct object_id *dst, const struct object_id *src)\n \n /* Like oidcpy() but zero-pads the unused bytes in dst's hash array. */\n static inline void oidcpy_with_padding(struct object_id *dst,\n-\t\t\t\t struct object_id *src)\n+\t\t\t\t const struct object_id *src)\n {\n \tsize_t hashsz;\n \n"},{"id":"428542","messageId":"20210627024718.25383-6-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-27T02:47:18Z","receivedAt":"2021-06-27T02:47:35Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This saves 8K per `struct object_directory', meaning it saves\naround 800MB in my case involving 100K alternates (half or more\nof those alternates are unlikely to hold loose objects).\n\nThis is implemented in two parts: a generic, allocation-free\n`cbtree' and the `oidtree' wrapper on top of it. The latter\nprovides allocation using alloc_state as a memory pool to\nimprove locality and reduce free(3) overhead.\n\nUnlike oid-array, the crit-bit tree does not require sorting.\nPerformance is bound by the key length, for oidtree that is\nfixed at sizeof(struct object_id). There's no need to have\n256 oidtrees to mitigate the O(n log n) overhead like we did\nwith oid-array.\n\nBeing a prefix trie, it is natively suited for expanding short\nobject IDs via prefix-limited iteration in\n`find_short_object_filename'.\n\nOn my busy workstation, p4205 performance seems to be roughly\nunchanged (+/-8%). Startup with 100K total alternates with no\nloose objects seems around 10-20% faster on a hot cache.\n(800MB in memory savings means more memory for the kernel FS\ncache).\n\nThe generic cbtree implementation does impose some extra\noverhead for oidtree in that it uses memcmp(3) on\n\"struct object_id\" so it wastes cycles comparing 12 extra bytes\non SHA-1 repositories. I've not yet explored reducing this\noverhead, but I expect there are many places in our code base\nwhere we'd want to investigate this.\n\nMore information on crit-bit trees: https://cr.yp.to/critbit.html\n\nSigned-off-by: Eric Wong \n---\n Makefile | 3 +\n alloc.c | 6 ++\n alloc.h | 1 +\n cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n cbtree.h | 56 ++++++++++++++\n object-file.c | 17 ++--\n object-name.c | 28 +++----\n object-store.h | 5 +-\n oidtree.c | 94 ++++++++++++++++++++++\n oidtree.h | 29 +++++++\n t/helper/test-oidtree.c | 45 +++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0069-oidtree.sh | 52 +++++++++++++\n 14 files changed, 476 insertions(+), 29 deletions(-)\n create mode 100644 cbtree.c\n create mode 100644 cbtree.h\n create mode 100644 oidtree.c\n create mode 100644 oidtree.h\n create mode 100644 t/helper/test-oidtree.c\n create mode 100755 t/t0069-oidtree.sh\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..a1525978fb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n TEST_BUILTINS_OBJS += test-oid-array.o\n TEST_BUILTINS_OBJS += test-oidmap.o\n+TEST_BUILTINS_OBJS += test-oidtree.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n@@ -845,6 +846,7 @@ LIB_OBJS += branch.o\n LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n+LIB_OBJS += cbtree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n LIB_OBJS += chunk-format.o\n@@ -940,6 +942,7 @@ LIB_OBJS += object.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += oidtree.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-check.o\ndiff --git a/alloc.c b/alloc.c\nindex 957a0af362..ca1e178c5a 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -14,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"oidtree.h\"\n #include \"alloc.h\"\n \n #define BLOCKING 1024\n@@ -123,6 +124,11 @@ void *alloc_commit_node(struct repository *r)\n \treturn c;\n }\n \n+void *alloc_from_state(struct alloc_state *alloc_state, size_t n)\n+{\n+\treturn alloc_node(alloc_state, n);\n+}\n+\n static void report(const char *name, unsigned int count, size_t size)\n {\n \tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\ndiff --git a/alloc.h b/alloc.h\nindex 371d388b55..4032375aa1 100644\n--- a/alloc.h\n+++ b/alloc.h\n@@ -13,6 +13,7 @@ void init_commit_node(struct commit *c);\n void *alloc_commit_node(struct repository *r);\n void *alloc_tag_node(struct repository *r);\n void *alloc_object_node(struct repository *r);\n+void *alloc_from_state(struct alloc_state *, size_t n);\n void alloc_report(struct repository *r);\n \n struct alloc_state *allocate_alloc_state(void);\ndiff --git a/cbtree.c b/cbtree.c\nnew file mode 100644\nindex 0000000000..b0c65d810f\n--- /dev/null\n+++ b/cbtree.c\n@@ -0,0 +1,167 @@\n+/*\n+ * crit-bit tree implementation, does no allocations internally\n+ * For more information on crit-bit trees: https://cr.yp.to/critbit.html\n+ * Based on Adam Langley's adaptation of Dan Bernstein's public domain code\n+ * git clone https://github.com/agl/critbit.git\n+ */\n+#include \"cbtree.h\"\n+\n+static struct cb_node *cb_node_of(const void *p)\n+{\n+\treturn (struct cb_node *)((uintptr_t)p - 1);\n+}\n+\n+/* locate the best match, does not do a final comparision */\n+static struct cb_node *cb_internal_best_match(struct cb_node *p,\n+\t\t\t\t\tconst uint8_t *k, size_t klen)\n+{\n+\twhile (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tuint8_t c = q->byte < klen ? k[q->byte] : 0;\n+\t\tsize_t direction = (1 + (q->otherbits | c)) >> 8;\n+\n+\t\tp = q->child[direction];\n+\t}\n+\treturn p;\n+}\n+\n+/* returns NULL if successful, existing cb_node if duplicate */\n+struct cb_node *cb_insert(struct cb_tree *t, struct cb_node *node, size_t klen)\n+{\n+\tsize_t newbyte, newotherbits;\n+\tuint8_t c;\n+\tint newdirection;\n+\tstruct cb_node **wherep, *p;\n+\n+\tassert(!((uintptr_t)node & 1)); /* allocations must be aligned */\n+\n+\tif (!t->root) {\t\t/* insert into empty tree */\n+\t\tt->root = node;\n+\t\treturn NULL;\t/* success */\n+\t}\n+\n+\t/* see if a node already exists */\n+\tp = cb_internal_best_match(t->root, node->k, klen);\n+\n+\t/* find first differing byte */\n+\tfor (newbyte = 0; newbyte < klen; newbyte++) {\n+\t\tif (p->k[newbyte] != node->k[newbyte])\n+\t\t\tgoto different_byte_found;\n+\t}\n+\treturn p;\t/* element exists, let user deal with it */\n+\n+different_byte_found:\n+\tnewotherbits = p->k[newbyte] ^ node->k[newbyte];\n+\tnewotherbits |= newotherbits >> 1;\n+\tnewotherbits |= newotherbits >> 2;\n+\tnewotherbits |= newotherbits >> 4;\n+\tnewotherbits = (newotherbits & ~(newotherbits >> 1)) ^ 255;\n+\tc = p->k[newbyte];\n+\tnewdirection = (1 + (newotherbits | c)) >> 8;\n+\n+\tnode->byte = newbyte;\n+\tnode->otherbits = newotherbits;\n+\tnode->child[1 - newdirection] = node;\n+\n+\t/* find a place to insert it */\n+\twherep = &t->root;\n+\tfor (;;) {\n+\t\tstruct cb_node *q;\n+\t\tsize_t direction;\n+\n+\t\tp = *wherep;\n+\t\tif (!(1 & (uintptr_t)p))\n+\t\t\tbreak;\n+\t\tq = cb_node_of(p);\n+\t\tif (q->byte > newbyte)\n+\t\t\tbreak;\n+\t\tif (q->byte == newbyte && q->otherbits > newotherbits)\n+\t\t\tbreak;\n+\t\tc = q->byte < klen ? node->k[q->byte] : 0;\n+\t\tdirection = (1 + (q->otherbits | c)) >> 8;\n+\t\twherep = q->child + direction;\n+\t}\n+\n+\tnode->child[newdirection] = *wherep;\n+\t*wherep = (struct cb_node *)(1 + (uintptr_t)node);\n+\n+\treturn NULL; /* success */\n+}\n+\n+struct cb_node *cb_lookup(struct cb_tree *t, const uint8_t *k, size_t klen)\n+{\n+\tstruct cb_node *p = cb_internal_best_match(t->root, k, klen);\n+\n+\treturn p && !memcmp(p->k, k, klen) ? p : NULL;\n+}\n+\n+struct cb_node *cb_unlink(struct cb_tree *t, const uint8_t *k, size_t klen)\n+{\n+\tstruct cb_node **wherep = &t->root;\n+\tstruct cb_node **whereq = NULL;\n+\tstruct cb_node *q = NULL;\n+\tsize_t direction = 0;\n+\tuint8_t c;\n+\tstruct cb_node *p = t->root;\n+\n+\tif (!p) return NULL;\t/* empty tree, nothing to delete */\n+\n+\t/* traverse to find best match, keeping link to parent */\n+\twhile (1 & (uintptr_t)p) {\n+\t\twhereq = wherep;\n+\t\tq = cb_node_of(p);\n+\t\tc = q->byte < klen ? k[q->byte] : 0;\n+\t\tdirection = (1 + (q->otherbits | c)) >> 8;\n+\t\twherep = q->child + direction;\n+\t\tp = *wherep;\n+\t}\n+\n+\tif (memcmp(p->k, k, klen))\n+\t\treturn NULL;\t\t/* no match, nothing unlinked */\n+\n+\t/* found an exact match */\n+\tif (whereq)\t/* update parent */\n+\t\t*whereq = q->child[1 - direction];\n+\telse\n+\t\tt->root = NULL;\n+\treturn p;\n+}\n+\n+static enum cb_next cb_descend(struct cb_node *p, cb_iter fn, void *arg)\n+{\n+\tif (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tenum cb_next n = cb_descend(q->child[0], fn, arg);\n+\n+\t\treturn n == CB_BREAK ? n : cb_descend(q->child[1], fn, arg);\n+\t} else {\n+\t\treturn fn(p, arg);\n+\t}\n+}\n+\n+void cb_each(struct cb_tree *t, const uint8_t *kpfx, size_t klen,\n+\t\t\tcb_iter fn, void *arg)\n+{\n+\tstruct cb_node *p = t->root;\n+\tstruct cb_node *top = p;\n+\tsize_t i = 0;\n+\n+\tif (!p) return; /* empty tree */\n+\n+\t/* Walk tree, maintaining top pointer */\n+\twhile (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tuint8_t c = q->byte < klen ? kpfx[q->byte] : 0;\n+\t\tsize_t direction = (1 + (q->otherbits | c)) >> 8;\n+\n+\t\tp = q->child[direction];\n+\t\tif (q->byte < klen)\n+\t\t\ttop = p;\n+\t}\n+\n+\tfor (i = 0; i < klen; i++) {\n+\t\tif (p->k[i] != kpfx[i])\n+\t\t\treturn; /* \"best\" match failed */\n+\t}\n+\tcb_descend(top, fn, arg);\n+}\ndiff --git a/cbtree.h b/cbtree.h\nnew file mode 100644\nindex 0000000000..fe4587087e\n--- /dev/null\n+++ b/cbtree.h\n@@ -0,0 +1,56 @@\n+/*\n+ * crit-bit tree implementation, does no allocations internally\n+ * For more information on crit-bit trees: https://cr.yp.to/critbit.html\n+ * Based on Adam Langley's adaptation of Dan Bernstein's public domain code\n+ * git clone https://github.com/agl/critbit.git\n+ *\n+ * This is adapted to store arbitrary data (not just NUL-terminated C strings\n+ * and allocates no memory internally. The user needs to allocate\n+ * \"struct cb_node\" and fill cb_node.k[] with arbitrary match data\n+ * for memcmp.\n+ * If \"klen\" is variable, then it should be embedded into \"c_node.k[]\"\n+ * Recursion is bound by the maximum value of \"klen\" used.\n+ */\n+#ifndef CBTREE_H\n+#define CBTREE_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct cb_node;\n+struct cb_node {\n+\tstruct cb_node *child[2];\n+\t/*\n+\t * n.b. uint32_t for `byte' is excessive for OIDs,\n+\t * we may consider shorter variants if nothing else gets stored.\n+\t */\n+\tuint32_t byte;\n+\tuint8_t otherbits;\n+\tuint8_t k[FLEX_ARRAY]; /* arbitrary data */\n+};\n+\n+struct cb_tree {\n+\tstruct cb_node *root;\n+};\n+\n+enum cb_next {\n+\tCB_CONTINUE = 0,\n+\tCB_BREAK = 1\n+};\n+\n+#define CBTREE_INIT { .root = NULL }\n+\n+static inline void cb_init(struct cb_tree *t)\n+{\n+\tt->root = NULL;\n+}\n+\n+struct cb_node *cb_lookup(struct cb_tree *, const uint8_t *k, size_t klen);\n+struct cb_node *cb_insert(struct cb_tree *, struct cb_node *, size_t klen);\n+struct cb_node *cb_unlink(struct cb_tree *t, const uint8_t *k, size_t klen);\n+\n+typedef enum cb_next (*cb_iter)(struct cb_node *, void *arg);\n+\n+void cb_each(struct cb_tree *, const uint8_t *kpfx, size_t klen,\n+\t\tcb_iter, void *arg);\n+\n+#endif /* CBTREE_H */\ndiff --git a/object-file.c b/object-file.c\nindex 2c8b9c05f9..d33b84c4a4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1175,7 +1175,7 @@ static int quick_has_loose(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\tif (oid_array_lookup(odb_loose_cache(odb, oid), oid) >= 0)\n+\t\tif (oidtree_contains(odb_loose_cache(odb, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -2454,11 +2454,11 @@ int for_each_loose_object(each_loose_object_fn cb, void *data,\n static int append_loose_object(const struct object_id *oid, const char *path,\n \t\t\t void *data)\n {\n-\toid_array_append(data, oid);\n+\toidtree_insert(data, oid);\n \treturn 0;\n }\n \n-struct oid_array *odb_loose_cache(struct object_directory *odb,\n+struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n@@ -2473,24 +2473,21 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n \n \tbitmap = &odb->loose_objects_subdir_seen[subdir_nr / BM_SIZE];\n \tif (*bitmap & bit)\n-\t\treturn &odb->loose_objects_cache[subdir_nr];\n+\t\treturn &odb->loose_objects_cache;\n \n \tstrbuf_addstr(&buf, odb->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n+\t\t\t\t &odb->loose_objects_cache);\n \t*bitmap |= bit;\n \tstrbuf_release(&buf);\n-\treturn &odb->loose_objects_cache[subdir_nr];\n+\treturn &odb->loose_objects_cache;\n }\n \n void odb_clear_loose_cache(struct object_directory *odb)\n {\n-\tint i;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(odb->loose_objects_cache); i++)\n-\t\toid_array_clear(&odb->loose_objects_cache[i]);\n+\toidtree_destroy(&odb->loose_objects_cache);\n \tmemset(&odb->loose_objects_subdir_seen, 0,\n \t sizeof(odb->loose_objects_subdir_seen));\n }\ndiff --git a/object-name.c b/object-name.c\nindex 64202de60b..3263c19457 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -87,27 +87,21 @@ static void update_candidates(struct disambiguate_state *ds, const struct object\n \n static int match_hash(unsigned, const unsigned char *, const unsigned char *);\n \n+static enum cb_next match_prefix(const struct object_id *oid, void *arg)\n+{\n+\tstruct disambiguate_state *ds = arg;\n+\t/* no need to call match_hash, oidtree_each did prefix match */\n+\tupdate_candidates(ds, oid);\n+\treturn ds->ambiguous ? CB_BREAK : CB_CONTINUE;\n+}\n+\n static void find_short_object_filename(struct disambiguate_state *ds)\n {\n \tstruct object_directory *odb;\n \n-\tfor (odb = ds->repo->objects->odb; odb && !ds->ambiguous; odb = odb->next) {\n-\t\tint pos;\n-\t\tstruct oid_array *loose_objects;\n-\n-\t\tloose_objects = odb_loose_cache(odb, &ds->bin_pfx);\n-\t\tpos = oid_array_lookup(loose_objects, &ds->bin_pfx);\n-\t\tif (pos < 0)\n-\t\t\tpos = -1 - pos;\n-\t\twhile (!ds->ambiguous && pos < loose_objects->nr) {\n-\t\t\tconst struct object_id *oid;\n-\t\t\toid = loose_objects->oid + pos;\n-\t\t\tif (!match_hash(ds->len, ds->bin_pfx.hash, oid->hash))\n-\t\t\t\tbreak;\n-\t\t\tupdate_candidates(ds, oid);\n-\t\t\tpos++;\n-\t\t}\n-\t}\n+\tfor (odb = ds->repo->objects->odb; odb && !ds->ambiguous; odb = odb->next)\n+\t\toidtree_each(odb_loose_cache(odb, &ds->bin_pfx),\n+\t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \n static int match_hash(unsigned len, const unsigned char *a, const unsigned char *b)\ndiff --git a/object-store.h b/object-store.h\nindex 8fcddf3e65..b507108d18 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -9,6 +9,7 @@\n #include \"thread-utils.h\"\n #include \"khash.h\"\n #include \"dir.h\"\n+#include \"oidtree.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -23,7 +24,7 @@ struct object_directory {\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n \tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oid_array loose_objects_cache[256];\n+\tstruct oidtree loose_objects_cache;\n \n \t/*\n \t * Path to the alternative object store. If this is a relative path,\n@@ -69,7 +70,7 @@ void add_to_alternates_memory(const char *dir);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n-struct oid_array *odb_loose_cache(struct object_directory *odb,\n+struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\ndiff --git a/oidtree.c b/oidtree.c\nnew file mode 100644\nindex 0000000000..c1188d8f48\n--- /dev/null\n+++ b/oidtree.c\n@@ -0,0 +1,94 @@\n+/*\n+ * A wrapper around cbtree which stores oids\n+ * May be used to replace oid-array for prefix (abbreviation) matches\n+ */\n+#include \"oidtree.h\"\n+#include \"alloc.h\"\n+#include \"hash.h\"\n+\n+struct oidtree_node {\n+\t/* n.k[] is used to store \"struct object_id\" */\n+\tstruct cb_node n;\n+};\n+\n+struct oidtree_iter_data {\n+\toidtree_iter fn;\n+\tvoid *arg;\n+\tsize_t *last_nibble_at;\n+\tint algo;\n+\tuint8_t last_byte;\n+};\n+\n+void oidtree_destroy(struct oidtree *ot)\n+{\n+\tif (ot->mempool) {\n+\t\tclear_alloc_state(ot->mempool);\n+\t\tFREE_AND_NULL(ot->mempool);\n+\t}\n+\toidtree_init(ot);\n+}\n+\n+void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n+{\n+\tstruct oidtree_node *on;\n+\n+\tif (!ot->mempool)\n+\t\tot->mempool = allocate_alloc_state();\n+\tif (!oid->algo)\n+\t\tBUG(\"oidtree_insert requires oid->algo\");\n+\n+\ton = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n+\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n+\n+\t/*\n+\t * n.b. we shouldn't get duplicates, here, but we'll have\n+\t * a small leak that won't be freed until oidtree_destroy\n+\t */\n+\tcb_insert(&ot->t, &on->n, sizeof(*oid));\n+}\n+\n+int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n+{\n+\tstruct object_id k = { 0 };\n+\tsize_t klen = sizeof(k);\n+\toidcpy_with_padding(&k, oid);\n+\n+\tif (oid->algo == GIT_HASH_UNKNOWN) {\n+\t\tk.algo = hash_algo_by_ptr(the_hash_algo);\n+\t\tklen -= sizeof(oid->algo);\n+\t}\n+\n+\treturn cb_lookup(&ot->t, (const uint8_t *)&k, klen) ? 1 : 0;\n+}\n+\n+static enum cb_next iter(struct cb_node *n, void *arg)\n+{\n+\tstruct oidtree_iter_data *x = arg;\n+\tconst struct object_id *oid = (const struct object_id *)n->k;\n+\n+\tif (x->algo != GIT_HASH_UNKNOWN && x->algo != oid->algo)\n+\t\treturn CB_CONTINUE;\n+\n+\tif (x->last_nibble_at) {\n+\t\tif ((oid->hash[*x->last_nibble_at] ^ x->last_byte) & 0xf0)\n+\t\t\treturn CB_CONTINUE;\n+\t}\n+\n+\treturn x->fn(oid, x->arg);\n+}\n+\n+void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n+\t\t\tsize_t oidhexlen, oidtree_iter fn, void *arg)\n+{\n+\tsize_t klen = oidhexlen / 2;\n+\tstruct oidtree_iter_data x = { 0 };\n+\n+\tx.fn = fn;\n+\tx.arg = arg;\n+\tx.algo = oid->algo;\n+\tif (oidhexlen & 1) {\n+\t\tx.last_byte = oid->hash[klen];\n+\t\tx.last_nibble_at = &klen;\n+\t}\n+\tcb_each(&ot->t, (const uint8_t *)oid, klen, iter, &x);\n+}\ndiff --git a/oidtree.h b/oidtree.h\nnew file mode 100644\nindex 0000000000..73399bb978\n--- /dev/null\n+++ b/oidtree.h\n@@ -0,0 +1,29 @@\n+#ifndef OIDTREE_H\n+#define OIDTREE_H\n+\n+#include \"cbtree.h\"\n+#include \"hash.h\"\n+\n+struct alloc_state;\n+struct oidtree {\n+\tstruct cb_tree t;\n+\tstruct alloc_state *mempool;\n+};\n+\n+#define OIDTREE_INIT { .t = CBTREE_INIT, .mempool = NULL }\n+\n+static inline void oidtree_init(struct oidtree *ot)\n+{\n+\tcb_init(&ot->t);\n+\tot->mempool = NULL;\n+}\n+\n+void oidtree_destroy(struct oidtree *);\n+void oidtree_insert(struct oidtree *, const struct object_id *);\n+int oidtree_contains(struct oidtree *, const struct object_id *);\n+\n+typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *arg);\n+void oidtree_each(struct oidtree *, const struct object_id *,\n+\t\t\tsize_t oidhexlen, oidtree_iter, void *arg);\n+\n+#endif /* OIDTREE_H */\ndiff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\nnew file mode 100644\nindex 0000000000..44bb2e7c29\n--- /dev/null\n+++ b/t/helper/test-oidtree.c\n@@ -0,0 +1,45 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"oidtree.h\"\n+\n+static enum cb_next print_oid(const struct object_id *oid, void *data)\n+{\n+\tputs(oid_to_hex(oid));\n+\treturn CB_CONTINUE;\n+}\n+\n+int cmd__oidtree(int argc, const char **argv)\n+{\n+\tstruct oidtree ot = OIDTREE_INIT;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint nongit_ok;\n+\n+\tsetup_git_directory_gently(&nongit_ok);\n+\n+\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\tconst char *arg;\n+\t\tstruct object_id oid;\n+\n+\t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(\"not a hexadecimal oid: %s\", arg);\n+\t\t\toidtree_insert(&ot, &oid);\n+\t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(\"not a hexadecimal oid: %s\", arg);\n+\t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n+\t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n+\t\t\tchar buf[GIT_SHA1_HEXSZ + 1] = { '0' };\n+\t\t\tmemset(&oid, 0, sizeof(oid));\n+\t\t\tmemcpy(buf, arg, strlen(arg));\n+\t\t\tbuf[GIT_SHA1_HEXSZ] = 0;\n+\t\t\tget_oid_hex_any(buf, &oid);\n+\t\t\toid.algo = GIT_HASH_SHA1;\n+\t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n+\t\t} else if (!strcmp(line.buf, \"destroy\"))\n+\t\t\toidtree_destroy(&ot);\n+\t\telse\n+\t\t\tdie(\"unknown command: %s\", line.buf);\n+\t}\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..9d37debf28 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n \t{ \"mktemp\", cmd__mktemp },\n \t{ \"oid-array\", cmd__oid_array },\n \t{ \"oidmap\", cmd__oidmap },\n+\t{ \"oidtree\", cmd__oidtree },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f683a2f59c 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -32,6 +32,7 @@ int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n int cmd__oidmap(int argc, const char **argv);\n+int cmd__oidtree(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\ndiff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\nnew file mode 100755\nindex 0000000000..bb4229210c\n--- /dev/null\n+++ b/t/t0069-oidtree.sh\n@@ -0,0 +1,52 @@\n+#!/bin/sh\n+\n+test_description='basic tests for the oidtree implementation'\n+. ./test-lib.sh\n+\n+echoid () {\n+\tprefix=\"${1:+$1 }\"\n+\tshift\n+\twhile test $# -gt 0\n+\tdo\n+\t\techo \"$1\"\n+\t\tshift\n+\tdone | awk -v prefix=\"$prefix\" '{\n+\t\tprintf(\"%s%s\", prefix, $0);\n+\t\tneed = 40 - length($0);\n+\t\tfor (i = 0; i < need; i++)\n+\t\t\tprintf(\"0\");\n+\t\tprintf \"\\n\";\n+\t}'\n+}\n+\n+test_expect_success 'oidtree insert and contains' '\n+\tcat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'oidtree each' '\n+\techoid \"\" 123 321 321 >expect &&\n+\t{\n+\t\techoid insert f 9 8 123 321 a b c d e\n+\t\techo each 12300\n+\t\techo each 3211\n+\t\techo each 3210\n+\t\techo each 32100\n+\t\techo destroy\n+\t} | test-tool oidtree >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n"},{"id":"428548","messageId":"496545dc-e372-401c-13f4-daa7ee765d39@web.de","threadId":"55998","inReplyTo":"20210627024718.25383-4-e@80x24.org","subject":"Re: [PATCH 3/5] make object_directory.loose_objects_subdir_seen a bitmap","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-06-27T10:23:18Z","receivedAt":"2021-06-27T10:23:36Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 27.06.21 um 04:47 schrieb Eric Wong:\n> There's no point in using 8 bits per-directory when 1 bit\n> will do. This saves us 224 bytes per object directory, which\n> ends up being 22MB when dealing with 100K alternates.\n\nThe point was simplicity under the assumption that the number of\nrepositories is low -- for most users it's only one. That obviously\ndoesn't hold for your use case anymore. :)\n\nA compact representation should also reduce dcache misses, so this\nshould be a win for the single-repo case as well.\n\n> Signed-off-by: Eric Wong \n> ---\n> object-file.c | 10 +++++++---\n> object-store.h | 2 +-\n> 2 files changed, 8 insertions(+), 4 deletions(-)\n>\n> diff --git a/object-file.c b/object-file.c\n> index 6be43c2b60..2c8b9c05f9 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2463,12 +2463,16 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n> {\n> \tint subdir_nr = oid->hash[0];\n> \tstruct strbuf buf = STRBUF_INIT;\n> +\tsize_t BM_SIZE = sizeof(odb->loose_objects_subdir_seen[0]) * CHAR_BIT;\n\nWith that name I'd expect the variable to contain the number of bytes or\nbits in the whole bitmap. And to not be a variable at all, but rather a\nmacro. Perhaps word_bits?\n\nbitsizeof() does the same and is slightly shorter.\n\n> +\tuint32_t *bitmap;\n\nAh, you call the array items bitmap, which they are. Hmm. I rather\nthink of the whole thing as a bitmap and its uint32_t elements as words.\nDoes it matter? Not sure.\n\n> +\tuint32_t bit = 1 << (subdir_nr % BM_SIZE);\n\nI'd call that mask, but bit is fine as well..\n\nAnyway, it would look something like this:\n\n\tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n\tsize_t word_index = subdir_nr / word_bits;\n\tsize_t mask = 1 << (subdir_nr % word_bits);\n\n>\n> \tif (subdir_nr < 0 ||\n> -\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n> +\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen) * BM_SIZE)\n\nbitsizeof(odb->loose_objects_subdir_seen) would be easier to read and\nunderstand, I think.\n\n> \t\tBUG(\"subdir_nr out of range\");\n>\n> -\tif (odb->loose_objects_subdir_seen[subdir_nr])\n> +\tbitmap = &odb->loose_objects_subdir_seen[subdir_nr / BM_SIZE];\n> +\tif (*bitmap & bit)\n> \t\treturn &odb->loose_objects_cache[subdir_nr];\n>\n> \tstrbuf_addstr(&buf, odb->path);\n> @@ -2476,7 +2480,7 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n> \t\t\t\t append_loose_object,\n> \t\t\t\t NULL, NULL,\n> \t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n> -\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n> +\t*bitmap |= bit;\n> \tstrbuf_release(&buf);\n> \treturn &odb->loose_objects_cache[subdir_nr];\n> }\n> diff --git a/object-store.h b/object-store.h\n> index 20c1cedb75..8fcddf3e65 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -22,7 +22,7 @@ struct object_directory {\n> \t *\n> \t * Be sure to call odb_load_loose_cache() before using.\n> \t */\n> -\tchar loose_objects_subdir_seen[256];\n> +\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n\nPerhaps\tDIV_ROUND_UP(256, bitsizeof(uint32_t))? The comment explains\nit nicely already, though.\n\n> \tstruct oid_array loose_objects_cache[256];\n>\n> \t/*\n>\n\nSummary: Good idea, the implementation looks correct, I stumbled\nover some of the names, bitsizeof() could be used.\n\nRené\n"},{"id":"428648","messageId":"20210628230953.GA9830@dcvr","threadId":"55998","inReplyTo":"496545dc-e372-401c-13f4-daa7ee765d39@web.de","subject":"Re: [PATCH 3/5] make object_directory.loose_objects_subdir_seen a bitmap","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-28T23:09:53Z","receivedAt":"2021-06-28T23:09:56Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"René Scharfe wrote:\n> Am 27.06.21 um 04:47 schrieb Eric Wong:\n> Anyway, it would look something like this:\n> \n> \tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n> \tsize_t word_index = subdir_nr / word_bits;\n> \tsize_t mask = 1 << (subdir_nr % word_bits);\n\n yeah, I missed bitsizeof :x\n\n> > --- a/object-store.h\n> > +++ b/object-store.h\n> > @@ -22,7 +22,7 @@ struct object_directory {\n> > \t *\n> > \t * Be sure to call odb_load_loose_cache() before using.\n> > \t */\n> > -\tchar loose_objects_subdir_seen[256];\n> > +\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n> \n> Perhaps\tDIV_ROUND_UP(256, bitsizeof(uint32_t))? The comment explains\n> it nicely already, though.\n\nI think I'll keep my original there... IMHO the macros\nobfuscate the meaning for those less familiar with our codebase\n(myself included :x)\n\n> > \tstruct oid_array loose_objects_cache[256];\n> >\n> > \t/*\n> >\n> \n> Summary: Good idea, the implementation looks correct, I stumbled\n> over some of the names, bitsizeof() could be used.\n\nThanks for the review.\n\nI'll squash up the following for v2 while awaiting feedback\nfor the rest of the series:\n\ndiff --git a/object-file.c b/object-file.c\nindex d33b84c4a4..6c397fb4f1 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2463,16 +2463,17 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t BM_SIZE = sizeof(odb->loose_objects_subdir_seen[0]) * CHAR_BIT;\n+\tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n+\tsize_t word_index = subdir_nr / word_bits;\n+\tsize_t mask = 1 << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n-\tuint32_t bit = 1 << (subdir_nr % BM_SIZE);\n \n \tif (subdir_nr < 0 ||\n-\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen) * BM_SIZE)\n+\t subdir_nr >= bitsizeof(odb->loose_objects_subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &odb->loose_objects_subdir_seen[subdir_nr / BM_SIZE];\n-\tif (*bitmap & bit)\n+\tbitmap = &odb->loose_objects_subdir_seen[word_index];\n+\tif (*bitmap & mask)\n \t\treturn &odb->loose_objects_cache;\n \n \tstrbuf_addstr(&buf, odb->path);\n@@ -2480,7 +2481,7 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n \t\t\t\t &odb->loose_objects_cache);\n-\t*bitmap |= bit;\n+\t*bitmap |= mask;\n \tstrbuf_release(&buf);\n \treturn &odb->loose_objects_cache;\n }\n"},{"id":"428754","messageId":"xmqqzgv8ybpg.fsf@gitster.g","threadId":"55998","inReplyTo":"20210627024718.25383-6-e@80x24.org","subject":"Re: [PATCH 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-29T14:42:03Z","receivedAt":"2021-06-29T14:42:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> This saves 8K per `struct object_directory', meaning it saves\n> around 800MB in my case involving 100K alternates (half or more\n> of those alternates are unlikely to hold loose objects).\n>\n> This is implemented in two parts: a generic, allocation-free\n> `cbtree' and the `oidtree' wrapper on top of it. The latter\n> provides allocation using alloc_state as a memory pool to\n> improve locality and reduce free(3) overhead.\n\nThis seems to break CI test, with \"fatal: not a hexadecimal oid\",\nperhaps because there is hardcoded 40 here?\n\n> index 0000000000..bb4229210c\n> --- /dev/null\n> +++ b/t/t0069-oidtree.sh\n> @@ -0,0 +1,52 @@\n> +#!/bin/sh\n> +\n> +test_description='basic tests for the oidtree implementation'\n> +. ./test-lib.sh\n> +\n> +echoid () {\n> +\tprefix=\"${1:+$1 }\"\n> +\tshift\n> +\twhile test $# -gt 0\n> +\tdo\n> +\t\techo \"$1\"\n> +\t\tshift\n> +\tdone | awk -v prefix=\"$prefix\" '{\n> +\t\tprintf(\"%s%s\", prefix, $0);\n> +\t\tneed = 40 - length($0);\n> +\t\tfor (i = 0; i < need; i++)\n> +\t\t\tprintf(\"0\");\n> +\t\tprintf \"\\n\";\n> +\t}'\n"},{"id":"428773","messageId":"20210629201720.GA18039@dcvr","threadId":"55998","inReplyTo":"xmqqzgv8ybpg.fsf@gitster.g","subject":"Re: [PATCH 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:17:20Z","receivedAt":"2021-06-29T20:17:22Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Eric Wong writes:\n> \n> > This saves 8K per `struct object_directory', meaning it saves\n> > around 800MB in my case involving 100K alternates (half or more\n> > of those alternates are unlikely to hold loose objects).\n> >\n> > This is implemented in two parts: a generic, allocation-free\n> > `cbtree' and the `oidtree' wrapper on top of it. The latter\n> > provides allocation using alloc_state as a memory pool to\n> > improve locality and reduce free(3) overhead.\n> \n> This seems to break CI test, with \"fatal: not a hexadecimal oid\",\n> perhaps because there is hardcoded 40 here?\n\nYes, I think this needs to be squashed in:\n--------8<------\nSubject: [PATCH] t0069: make oidtree test hash-agnostic\n\nTested with both:\n\nmake -C t t0069-oidtree.sh GIT_TEST_DEFAULT_HASH=sha1\nmake -C t t0069-oidtree.sh GIT_TEST_DEFAULT_HASH=sha256\n\nSigned-off-by: Eric Wong \n---\n t/helper/test-oidtree.c | 14 ++++++++------\n t/t0069-oidtree.sh | 4 ++--\n 2 files changed, 10 insertions(+), 8 deletions(-)\n\ndiff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\nindex 44bb2e7c29..e0da13eea3 100644\n--- a/t/helper/test-oidtree.c\n+++ b/t/helper/test-oidtree.c\n@@ -13,6 +13,7 @@ int cmd__oidtree(int argc, const char **argv)\n \tstruct oidtree ot = OIDTREE_INIT;\n \tstruct strbuf line = STRBUF_INIT;\n \tint nongit_ok;\n+\tint algo = GIT_HASH_UNKNOWN;\n \n \tsetup_git_directory_gently(&nongit_ok);\n \n@@ -21,20 +22,21 @@ int cmd__oidtree(int argc, const char **argv)\n \t\tstruct object_id oid;\n \n \t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"not a hexadecimal oid: %s\", arg);\n+\t\t\tif (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"insert not a hexadecimal oid: %s\", arg);\n+\t\t\talgo = oid.algo;\n \t\t\toidtree_insert(&ot, &oid);\n \t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n \t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"not a hexadecimal oid: %s\", arg);\n+\t\t\t\tdie(\"contains not a hexadecimal oid: %s\", arg);\n \t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n \t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n-\t\t\tchar buf[GIT_SHA1_HEXSZ + 1] = { '0' };\n+\t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n \t\t\tmemset(&oid, 0, sizeof(oid));\n \t\t\tmemcpy(buf, arg, strlen(arg));\n-\t\t\tbuf[GIT_SHA1_HEXSZ] = 0;\n+\t\t\tbuf[hash_algos[algo].hexsz] = 0;\n \t\t\tget_oid_hex_any(buf, &oid);\n-\t\t\toid.algo = GIT_HASH_SHA1;\n+\t\t\toid.algo = algo;\n \t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n \t\t} else if (!strcmp(line.buf, \"destroy\"))\n \t\t\toidtree_destroy(&ot);\ndiff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\nindex bb4229210c..0594f57c81 100755\n--- a/t/t0069-oidtree.sh\n+++ b/t/t0069-oidtree.sh\n@@ -10,9 +10,9 @@ echoid () {\n \tdo\n \t\techo \"$1\"\n \t\tshift\n-\tdone | awk -v prefix=\"$prefix\" '{\n+\tdone | awk -v prefix=\"$prefix\" -v ZERO_OID=$ZERO_OID '{\n \t\tprintf(\"%s%s\", prefix, $0);\n-\t\tneed = 40 - length($0);\n+\t\tneed = length(ZERO_OID) - length($0);\n \t\tfor (i = 0; i < need; i++)\n \t\t\tprintf(\"0\");\n \t\tprintf \"\\n\";\n"},{"id":"428776","messageId":"20210629205305.7100-1-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH v2 0/5] optimizations for many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:53:00Z","receivedAt":"2021-06-29T20:53:07Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"v2 has better naming for 3/5, fix test for sha256 in 5/5\nThanks to René and Junio for feedback so far.\n\nEric Wong (5):\n speed up alt_odb_usable() with many alternates\n avoid strlen via strbuf_addstr in link_alt_odb_entry\n make object_directory.loose_objects_subdir_seen a bitmap\n oidcpy_with_padding: constify `src' arg\n oidtree: a crit-bit tree for odb_loose_cache\n\n Makefile | 3 +\n alloc.c | 6 ++\n alloc.h | 1 +\n cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n cbtree.h | 56 ++++++++++++++\n hash.h | 2 +-\n object-file.c | 69 ++++++++++-------\n object-name.c | 28 +++----\n object-store.h | 24 +++++-\n object.c | 2 +\n oidtree.c | 94 ++++++++++++++++++++++\n oidtree.h | 29 +++++++\n t/helper/test-oidtree.c | 47 +++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0069-oidtree.sh | 52 +++++++++++++\n 16 files changed, 533 insertions(+), 49 deletions(-)\n create mode 100644 cbtree.c\n create mode 100644 cbtree.h\n create mode 100644 oidtree.c\n create mode 100644 oidtree.h\n create mode 100644 t/helper/test-oidtree.c\n create mode 100755 t/t0069-oidtree.sh\n\nInterdiff against v1:\ndiff --git a/object-file.c b/object-file.c\nindex d33b84c4a4..6c397fb4f1 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2463,16 +2463,17 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t BM_SIZE = sizeof(odb->loose_objects_subdir_seen[0]) * CHAR_BIT;\n+\tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n+\tsize_t word_index = subdir_nr / word_bits;\n+\tsize_t mask = 1 << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n-\tuint32_t bit = 1 << (subdir_nr % BM_SIZE);\n \n \tif (subdir_nr < 0 ||\n-\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen) * BM_SIZE)\n+\t subdir_nr >= bitsizeof(odb->loose_objects_subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tbitmap = &odb->loose_objects_subdir_seen[subdir_nr / BM_SIZE];\n-\tif (*bitmap & bit)\n+\tbitmap = &odb->loose_objects_subdir_seen[word_index];\n+\tif (*bitmap & mask)\n \t\treturn &odb->loose_objects_cache;\n \n \tstrbuf_addstr(&buf, odb->path);\n@@ -2480,7 +2481,7 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n \t\t\t\t &odb->loose_objects_cache);\n-\t*bitmap |= bit;\n+\t*bitmap |= mask;\n \tstrbuf_release(&buf);\n \treturn &odb->loose_objects_cache;\n }\ndiff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\nindex 44bb2e7c29..e0da13eea3 100644\n--- a/t/helper/test-oidtree.c\n+++ b/t/helper/test-oidtree.c\n@@ -13,6 +13,7 @@ int cmd__oidtree(int argc, const char **argv)\n \tstruct oidtree ot = OIDTREE_INIT;\n \tstruct strbuf line = STRBUF_INIT;\n \tint nongit_ok;\n+\tint algo = GIT_HASH_UNKNOWN;\n \n \tsetup_git_directory_gently(&nongit_ok);\n \n@@ -21,20 +22,21 @@ int cmd__oidtree(int argc, const char **argv)\n \t\tstruct object_id oid;\n \n \t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"not a hexadecimal oid: %s\", arg);\n+\t\t\tif (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"insert not a hexadecimal oid: %s\", arg);\n+\t\t\talgo = oid.algo;\n \t\t\toidtree_insert(&ot, &oid);\n \t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n \t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"not a hexadecimal oid: %s\", arg);\n+\t\t\t\tdie(\"contains not a hexadecimal oid: %s\", arg);\n \t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n \t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n-\t\t\tchar buf[GIT_SHA1_HEXSZ + 1] = { '0' };\n+\t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n \t\t\tmemset(&oid, 0, sizeof(oid));\n \t\t\tmemcpy(buf, arg, strlen(arg));\n-\t\t\tbuf[GIT_SHA1_HEXSZ] = 0;\n+\t\t\tbuf[hash_algos[algo].hexsz] = 0;\n \t\t\tget_oid_hex_any(buf, &oid);\n-\t\t\toid.algo = GIT_HASH_SHA1;\n+\t\t\toid.algo = algo;\n \t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n \t\t} else if (!strcmp(line.buf, \"destroy\"))\n \t\t\toidtree_destroy(&ot);\ndiff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\nindex bb4229210c..0594f57c81 100755\n--- a/t/t0069-oidtree.sh\n+++ b/t/t0069-oidtree.sh\n@@ -10,9 +10,9 @@ echoid () {\n \tdo\n \t\techo \"$1\"\n \t\tshift\n-\tdone | awk -v prefix=\"$prefix\" '{\n+\tdone | awk -v prefix=\"$prefix\" -v ZERO_OID=$ZERO_OID '{\n \t\tprintf(\"%s%s\", prefix, $0);\n-\t\tneed = 40 - length($0);\n+\t\tneed = length(ZERO_OID) - length($0);\n \t\tfor (i = 0; i < need; i++)\n \t\t\tprintf(\"0\");\n \t\tprintf \"\\n\";\n"},{"id":"428777","messageId":"20210629205305.7100-2-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH v2 1/5] speed up alt_odb_usable() with many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:53:01Z","receivedAt":"2021-06-29T20:53:08Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"With many alternates, the duplicate check in alt_odb_usable()\nwastes many cycles doing repeated fspathcmp() on every existing\nalternate. Use a khash to speed up lookups by odb->path.\n\nSince the kh_put_* API uses the supplied key without\nduplicating it, we also take advantage of it to replace both\nxstrdup() and strbuf_release() in link_alt_odb_entry() with\nstrbuf_detach() to avoid the allocation and copy.\n\nIn a test repository with 50K alternates and each of those 50K\nalternates having one alternate each (for a total of 100K total\nalternates); this speeds up lookup of a non-existent blob from\nover 16 minutes to roughly 2.7 seconds on my busy workstation.\n\nNote: all underlying git object directories were small and\nunpacked with only loose objects and no packs. Having to load\npacks increases times significantly.\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 33 ++++++++++++++++++++++-----------\n object-store.h | 17 +++++++++++++++++\n object.c | 2 ++\n 3 files changed, 41 insertions(+), 11 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..304af3a172 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -517,9 +517,9 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n */\n static int alt_odb_usable(struct raw_object_store *o,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir)\n+\t\t\t const char *normalized_objdir, khiter_t *pos)\n {\n-\tstruct object_directory *odb;\n+\tint r;\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path->buf)) {\n@@ -533,14 +533,22 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n \t */\n-\tfor (odb = o->odb; odb; odb = odb->next) {\n-\t\tif (!fspathcmp(path->buf, odb->path))\n-\t\t\treturn 0;\n+\tif (!o->odb_by_path) {\n+\t\tkhiter_t p;\n+\n+\t\to->odb_by_path = kh_init_odb_path_map();\n+\t\tassert(!o->odb->next);\n+\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n+\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t\tassert(r == 1); /* never used */\n+\t\tkh_value(o->odb_by_path, p) = o->odb;\n \t}\n \tif (!fspathcmp(path->buf, normalized_objdir))\n \t\treturn 0;\n-\n-\treturn 1;\n+\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n+\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n+\treturn r == 0 ? 0 : 1;\n }\n \n /*\n@@ -566,6 +574,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n+\tkhiter_t pos;\n \n \tif (!is_absolute_path(entry) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -587,23 +596,25 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \twhile (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')\n \t\tstrbuf_setlen(&pathbuf, pathbuf.len - 1);\n \n-\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {\n+\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir, &pos)) {\n \t\tstrbuf_release(&pathbuf);\n \t\treturn -1;\n \t}\n \n \tCALLOC_ARRAY(ent, 1);\n-\tent->path = xstrdup(pathbuf.buf);\n+\t/* pathbuf.buf is already in r->objects->odb_by_path */\n+\tent->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*r->objects->odb_tail = ent;\n \tr->objects->odb_tail = &(ent->next);\n \tent->next = NULL;\n+\tassert(r->objects->odb_by_path);\n+\tkh_value(r->objects->odb_by_path, pos) = ent;\n \n \t/* recursively add alternates */\n-\tread_info_alternates(r, pathbuf.buf, depth + 1);\n+\tread_info_alternates(r, ent->path, depth + 1);\n \n-\tstrbuf_release(&pathbuf);\n \treturn 0;\n }\n \ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb..20c1cedb75 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -7,6 +7,8 @@\n #include \"oid-array.h\"\n #include \"strbuf.h\"\n #include \"thread-utils.h\"\n+#include \"khash.h\"\n+#include \"dir.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -30,6 +32,19 @@ struct object_directory {\n \tchar *path;\n };\n \n+static inline int odb_path_eq(const char *a, const char *b)\n+{\n+\treturn !fspathcmp(a, b);\n+}\n+\n+static inline int odb_path_hash(const char *str)\n+{\n+\treturn ignore_case ? strihash(str) : __ac_X31_hash_string(str);\n+}\n+\n+KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n+\tstruct object_directory *, 1, odb_path_hash, odb_path_eq);\n+\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct object_directory *, void *);\n@@ -116,6 +131,8 @@ struct raw_object_store {\n \t */\n \tstruct object_directory *odb;\n \tstruct object_directory **odb_tail;\n+\tkh_odb_path_map_t *odb_by_path;\n+\n \tint loaded_alternates;\n \n \t/*\ndiff --git a/object.c b/object.c\nindex 14188453c5..2b3c075a15 100644\n--- a/object.c\n+++ b/object.c\n@@ -511,6 +511,8 @@ static void free_object_directories(struct raw_object_store *o)\n \t\tfree_object_directory(o->odb);\n \t\to->odb = next;\n \t}\n+\tkh_destroy_odb_path_map(o->odb_by_path);\n+\to->odb_by_path = NULL;\n }\n \n void raw_object_store_clear(struct raw_object_store *o)\n"},{"id":"428778","messageId":"20210629205305.7100-3-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH v2 2/5] avoid strlen via strbuf_addstr in link_alt_odb_entry","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:53:02Z","receivedAt":"2021-06-29T20:53:11Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"We can save a few milliseconds (across 100K odbs) by using\nstrbuf_addbuf() instead of strbuf_addstr() by passing `entry' as\na strbuf pointer rather than a \"const char *\".\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 304af3a172..6be43c2b60 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -569,18 +569,18 @@ static int alt_odb_usable(struct raw_object_store *o,\n static void read_info_alternates(struct repository *r,\n \t\t\t\t const char *relative_base,\n \t\t\t\t int depth);\n-static int link_alt_odb_entry(struct repository *r, const char *entry,\n+static int link_alt_odb_entry(struct repository *r, const struct strbuf *entry,\n \tconst char *relative_base, int depth, const char *normalized_objdir)\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tkhiter_t pos;\n \n-\tif (!is_absolute_path(entry) && relative_base) {\n+\tif (!is_absolute_path(entry->buf) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n \t\tstrbuf_addch(&pathbuf, '/');\n \t}\n-\tstrbuf_addstr(&pathbuf, entry);\n+\tstrbuf_addbuf(&pathbuf, entry);\n \n \tif (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {\n \t\terror(_(\"unable to normalize alternate object path: %s\"),\n@@ -671,7 +671,7 @@ static void link_alt_odb_entries(struct repository *r, const char *alt,\n \t\talt = parse_alt_odb_entry(alt, sep, &entry);\n \t\tif (!entry.len)\n \t\t\tcontinue;\n-\t\tlink_alt_odb_entry(r, entry.buf,\n+\t\tlink_alt_odb_entry(r, &entry,\n \t\t\t\t relative_base, depth, objdirbuf.buf);\n \t}\n \tstrbuf_release(&entry);\n"},{"id":"428779","messageId":"20210629205305.7100-4-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH v2 3/5] make object_directory.loose_objects_subdir_seen a bitmap","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:53:03Z","receivedAt":"2021-06-29T20:53:14Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"There's no point in using 8 bits per-directory when 1 bit\nwill do. This saves us 224 bytes per object directory, which\nends up being 22MB when dealing with 100K alternates.\n\nv2: use bitsizeof() macro and better variable names\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 11 ++++++++---\n object-store.h | 2 +-\n 2 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 6be43c2b60..91183d1297 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2463,12 +2463,17 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n+\tsize_t word_index = subdir_nr / word_bits;\n+\tsize_t mask = 1 << (subdir_nr % word_bits);\n+\tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n+\t subdir_nr >= bitsizeof(odb->loose_objects_subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tif (odb->loose_objects_subdir_seen[subdir_nr])\n+\tbitmap = &odb->loose_objects_subdir_seen[word_index];\n+\tif (*bitmap & mask)\n \t\treturn &odb->loose_objects_cache[subdir_nr];\n \n \tstrbuf_addstr(&buf, odb->path);\n@@ -2476,7 +2481,7 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n \t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n-\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n+\t*bitmap |= mask;\n \tstrbuf_release(&buf);\n \treturn &odb->loose_objects_cache[subdir_nr];\n }\ndiff --git a/object-store.h b/object-store.h\nindex 20c1cedb75..8fcddf3e65 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -22,7 +22,7 @@ struct object_directory {\n \t *\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n-\tchar loose_objects_subdir_seen[256];\n+\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n \tstruct oid_array loose_objects_cache[256];\n \n \t/*\n"},{"id":"428780","messageId":"20210629205305.7100-5-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH v2 4/5] oidcpy_with_padding: constify `src' arg","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:53:04Z","receivedAt":"2021-06-29T20:53:16Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"As with `oidcpy', the source struct will not be modified and\nthis will allow an upcoming const-correct caller to use it.\n\nSigned-off-by: Eric Wong \n---\n hash.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 9c6df4d952..27a180248f 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -265,7 +265,7 @@ static inline void oidcpy(struct object_id *dst, const struct object_id *src)\n \n /* Like oidcpy() but zero-pads the unused bytes in dst's hash array. */\n static inline void oidcpy_with_padding(struct object_id *dst,\n-\t\t\t\t struct object_id *src)\n+\t\t\t\t const struct object_id *src)\n {\n \tsize_t hashsz;\n \n"},{"id":"428781","messageId":"20210629205305.7100-6-e@80x24.org","threadId":"55998","inReplyTo":"20210627024718.25383-1-e@80x24.org","subject":"[PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-06-29T20:53:05Z","receivedAt":"2021-06-29T20:53:20Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This saves 8K per `struct object_directory', meaning it saves\naround 800MB in my case involving 100K alternates (half or more\nof those alternates are unlikely to hold loose objects).\n\nThis is implemented in two parts: a generic, allocation-free\n`cbtree' and the `oidtree' wrapper on top of it. The latter\nprovides allocation using alloc_state as a memory pool to\nimprove locality and reduce free(3) overhead.\n\nUnlike oid-array, the crit-bit tree does not require sorting.\nPerformance is bound by the key length, for oidtree that is\nfixed at sizeof(struct object_id). There's no need to have\n256 oidtrees to mitigate the O(n log n) overhead like we did\nwith oid-array.\n\nBeing a prefix trie, it is natively suited for expanding short\nobject IDs via prefix-limited iteration in\n`find_short_object_filename'.\n\nOn my busy workstation, p4205 performance seems to be roughly\nunchanged (+/-8%). Startup with 100K total alternates with no\nloose objects seems around 10-20% faster on a hot cache.\n(800MB in memory savings means more memory for the kernel FS\ncache).\n\nThe generic cbtree implementation does impose some extra\noverhead for oidtree in that it uses memcmp(3) on\n\"struct object_id\" so it wastes cycles comparing 12 extra bytes\non SHA-1 repositories. I've not yet explored reducing this\noverhead, but I expect there are many places in our code base\nwhere we'd want to investigate this.\n\nMore information on crit-bit trees: https://cr.yp.to/critbit.html\n\nv2: make oidtree test hash-agnostic\n\nSigned-off-by: Eric Wong \n---\n Makefile | 3 +\n alloc.c | 6 ++\n alloc.h | 1 +\n cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n cbtree.h | 56 ++++++++++++++\n object-file.c | 17 ++--\n object-name.c | 28 +++----\n object-store.h | 5 +-\n oidtree.c | 94 ++++++++++++++++++++++\n oidtree.h | 29 +++++++\n t/helper/test-oidtree.c | 47 +++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0069-oidtree.sh | 52 +++++++++++++\n 14 files changed, 478 insertions(+), 29 deletions(-)\n create mode 100644 cbtree.c\n create mode 100644 cbtree.h\n create mode 100644 oidtree.c\n create mode 100644 oidtree.h\n create mode 100644 t/helper/test-oidtree.c\n create mode 100755 t/t0069-oidtree.sh\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..a1525978fb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n TEST_BUILTINS_OBJS += test-oid-array.o\n TEST_BUILTINS_OBJS += test-oidmap.o\n+TEST_BUILTINS_OBJS += test-oidtree.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n@@ -845,6 +846,7 @@ LIB_OBJS += branch.o\n LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n+LIB_OBJS += cbtree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n LIB_OBJS += chunk-format.o\n@@ -940,6 +942,7 @@ LIB_OBJS += object.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += oidtree.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-check.o\ndiff --git a/alloc.c b/alloc.c\nindex 957a0af362..ca1e178c5a 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -14,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"oidtree.h\"\n #include \"alloc.h\"\n \n #define BLOCKING 1024\n@@ -123,6 +124,11 @@ void *alloc_commit_node(struct repository *r)\n \treturn c;\n }\n \n+void *alloc_from_state(struct alloc_state *alloc_state, size_t n)\n+{\n+\treturn alloc_node(alloc_state, n);\n+}\n+\n static void report(const char *name, unsigned int count, size_t size)\n {\n \tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\ndiff --git a/alloc.h b/alloc.h\nindex 371d388b55..4032375aa1 100644\n--- a/alloc.h\n+++ b/alloc.h\n@@ -13,6 +13,7 @@ void init_commit_node(struct commit *c);\n void *alloc_commit_node(struct repository *r);\n void *alloc_tag_node(struct repository *r);\n void *alloc_object_node(struct repository *r);\n+void *alloc_from_state(struct alloc_state *, size_t n);\n void alloc_report(struct repository *r);\n \n struct alloc_state *allocate_alloc_state(void);\ndiff --git a/cbtree.c b/cbtree.c\nnew file mode 100644\nindex 0000000000..b0c65d810f\n--- /dev/null\n+++ b/cbtree.c\n@@ -0,0 +1,167 @@\n+/*\n+ * crit-bit tree implementation, does no allocations internally\n+ * For more information on crit-bit trees: https://cr.yp.to/critbit.html\n+ * Based on Adam Langley's adaptation of Dan Bernstein's public domain code\n+ * git clone https://github.com/agl/critbit.git\n+ */\n+#include \"cbtree.h\"\n+\n+static struct cb_node *cb_node_of(const void *p)\n+{\n+\treturn (struct cb_node *)((uintptr_t)p - 1);\n+}\n+\n+/* locate the best match, does not do a final comparision */\n+static struct cb_node *cb_internal_best_match(struct cb_node *p,\n+\t\t\t\t\tconst uint8_t *k, size_t klen)\n+{\n+\twhile (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tuint8_t c = q->byte < klen ? k[q->byte] : 0;\n+\t\tsize_t direction = (1 + (q->otherbits | c)) >> 8;\n+\n+\t\tp = q->child[direction];\n+\t}\n+\treturn p;\n+}\n+\n+/* returns NULL if successful, existing cb_node if duplicate */\n+struct cb_node *cb_insert(struct cb_tree *t, struct cb_node *node, size_t klen)\n+{\n+\tsize_t newbyte, newotherbits;\n+\tuint8_t c;\n+\tint newdirection;\n+\tstruct cb_node **wherep, *p;\n+\n+\tassert(!((uintptr_t)node & 1)); /* allocations must be aligned */\n+\n+\tif (!t->root) {\t\t/* insert into empty tree */\n+\t\tt->root = node;\n+\t\treturn NULL;\t/* success */\n+\t}\n+\n+\t/* see if a node already exists */\n+\tp = cb_internal_best_match(t->root, node->k, klen);\n+\n+\t/* find first differing byte */\n+\tfor (newbyte = 0; newbyte < klen; newbyte++) {\n+\t\tif (p->k[newbyte] != node->k[newbyte])\n+\t\t\tgoto different_byte_found;\n+\t}\n+\treturn p;\t/* element exists, let user deal with it */\n+\n+different_byte_found:\n+\tnewotherbits = p->k[newbyte] ^ node->k[newbyte];\n+\tnewotherbits |= newotherbits >> 1;\n+\tnewotherbits |= newotherbits >> 2;\n+\tnewotherbits |= newotherbits >> 4;\n+\tnewotherbits = (newotherbits & ~(newotherbits >> 1)) ^ 255;\n+\tc = p->k[newbyte];\n+\tnewdirection = (1 + (newotherbits | c)) >> 8;\n+\n+\tnode->byte = newbyte;\n+\tnode->otherbits = newotherbits;\n+\tnode->child[1 - newdirection] = node;\n+\n+\t/* find a place to insert it */\n+\twherep = &t->root;\n+\tfor (;;) {\n+\t\tstruct cb_node *q;\n+\t\tsize_t direction;\n+\n+\t\tp = *wherep;\n+\t\tif (!(1 & (uintptr_t)p))\n+\t\t\tbreak;\n+\t\tq = cb_node_of(p);\n+\t\tif (q->byte > newbyte)\n+\t\t\tbreak;\n+\t\tif (q->byte == newbyte && q->otherbits > newotherbits)\n+\t\t\tbreak;\n+\t\tc = q->byte < klen ? node->k[q->byte] : 0;\n+\t\tdirection = (1 + (q->otherbits | c)) >> 8;\n+\t\twherep = q->child + direction;\n+\t}\n+\n+\tnode->child[newdirection] = *wherep;\n+\t*wherep = (struct cb_node *)(1 + (uintptr_t)node);\n+\n+\treturn NULL; /* success */\n+}\n+\n+struct cb_node *cb_lookup(struct cb_tree *t, const uint8_t *k, size_t klen)\n+{\n+\tstruct cb_node *p = cb_internal_best_match(t->root, k, klen);\n+\n+\treturn p && !memcmp(p->k, k, klen) ? p : NULL;\n+}\n+\n+struct cb_node *cb_unlink(struct cb_tree *t, const uint8_t *k, size_t klen)\n+{\n+\tstruct cb_node **wherep = &t->root;\n+\tstruct cb_node **whereq = NULL;\n+\tstruct cb_node *q = NULL;\n+\tsize_t direction = 0;\n+\tuint8_t c;\n+\tstruct cb_node *p = t->root;\n+\n+\tif (!p) return NULL;\t/* empty tree, nothing to delete */\n+\n+\t/* traverse to find best match, keeping link to parent */\n+\twhile (1 & (uintptr_t)p) {\n+\t\twhereq = wherep;\n+\t\tq = cb_node_of(p);\n+\t\tc = q->byte < klen ? k[q->byte] : 0;\n+\t\tdirection = (1 + (q->otherbits | c)) >> 8;\n+\t\twherep = q->child + direction;\n+\t\tp = *wherep;\n+\t}\n+\n+\tif (memcmp(p->k, k, klen))\n+\t\treturn NULL;\t\t/* no match, nothing unlinked */\n+\n+\t/* found an exact match */\n+\tif (whereq)\t/* update parent */\n+\t\t*whereq = q->child[1 - direction];\n+\telse\n+\t\tt->root = NULL;\n+\treturn p;\n+}\n+\n+static enum cb_next cb_descend(struct cb_node *p, cb_iter fn, void *arg)\n+{\n+\tif (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tenum cb_next n = cb_descend(q->child[0], fn, arg);\n+\n+\t\treturn n == CB_BREAK ? n : cb_descend(q->child[1], fn, arg);\n+\t} else {\n+\t\treturn fn(p, arg);\n+\t}\n+}\n+\n+void cb_each(struct cb_tree *t, const uint8_t *kpfx, size_t klen,\n+\t\t\tcb_iter fn, void *arg)\n+{\n+\tstruct cb_node *p = t->root;\n+\tstruct cb_node *top = p;\n+\tsize_t i = 0;\n+\n+\tif (!p) return; /* empty tree */\n+\n+\t/* Walk tree, maintaining top pointer */\n+\twhile (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tuint8_t c = q->byte < klen ? kpfx[q->byte] : 0;\n+\t\tsize_t direction = (1 + (q->otherbits | c)) >> 8;\n+\n+\t\tp = q->child[direction];\n+\t\tif (q->byte < klen)\n+\t\t\ttop = p;\n+\t}\n+\n+\tfor (i = 0; i < klen; i++) {\n+\t\tif (p->k[i] != kpfx[i])\n+\t\t\treturn; /* \"best\" match failed */\n+\t}\n+\tcb_descend(top, fn, arg);\n+}\ndiff --git a/cbtree.h b/cbtree.h\nnew file mode 100644\nindex 0000000000..fe4587087e\n--- /dev/null\n+++ b/cbtree.h\n@@ -0,0 +1,56 @@\n+/*\n+ * crit-bit tree implementation, does no allocations internally\n+ * For more information on crit-bit trees: https://cr.yp.to/critbit.html\n+ * Based on Adam Langley's adaptation of Dan Bernstein's public domain code\n+ * git clone https://github.com/agl/critbit.git\n+ *\n+ * This is adapted to store arbitrary data (not just NUL-terminated C strings\n+ * and allocates no memory internally. The user needs to allocate\n+ * \"struct cb_node\" and fill cb_node.k[] with arbitrary match data\n+ * for memcmp.\n+ * If \"klen\" is variable, then it should be embedded into \"c_node.k[]\"\n+ * Recursion is bound by the maximum value of \"klen\" used.\n+ */\n+#ifndef CBTREE_H\n+#define CBTREE_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct cb_node;\n+struct cb_node {\n+\tstruct cb_node *child[2];\n+\t/*\n+\t * n.b. uint32_t for `byte' is excessive for OIDs,\n+\t * we may consider shorter variants if nothing else gets stored.\n+\t */\n+\tuint32_t byte;\n+\tuint8_t otherbits;\n+\tuint8_t k[FLEX_ARRAY]; /* arbitrary data */\n+};\n+\n+struct cb_tree {\n+\tstruct cb_node *root;\n+};\n+\n+enum cb_next {\n+\tCB_CONTINUE = 0,\n+\tCB_BREAK = 1\n+};\n+\n+#define CBTREE_INIT { .root = NULL }\n+\n+static inline void cb_init(struct cb_tree *t)\n+{\n+\tt->root = NULL;\n+}\n+\n+struct cb_node *cb_lookup(struct cb_tree *, const uint8_t *k, size_t klen);\n+struct cb_node *cb_insert(struct cb_tree *, struct cb_node *, size_t klen);\n+struct cb_node *cb_unlink(struct cb_tree *t, const uint8_t *k, size_t klen);\n+\n+typedef enum cb_next (*cb_iter)(struct cb_node *, void *arg);\n+\n+void cb_each(struct cb_tree *, const uint8_t *kpfx, size_t klen,\n+\t\tcb_iter, void *arg);\n+\n+#endif /* CBTREE_H */\ndiff --git a/object-file.c b/object-file.c\nindex 91183d1297..6c397fb4f1 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1175,7 +1175,7 @@ static int quick_has_loose(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\tif (oid_array_lookup(odb_loose_cache(odb, oid), oid) >= 0)\n+\t\tif (oidtree_contains(odb_loose_cache(odb, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -2454,11 +2454,11 @@ int for_each_loose_object(each_loose_object_fn cb, void *data,\n static int append_loose_object(const struct object_id *oid, const char *path,\n \t\t\t void *data)\n {\n-\toid_array_append(data, oid);\n+\toidtree_insert(data, oid);\n \treturn 0;\n }\n \n-struct oid_array *odb_loose_cache(struct object_directory *odb,\n+struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n@@ -2474,24 +2474,21 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n \n \tbitmap = &odb->loose_objects_subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn &odb->loose_objects_cache[subdir_nr];\n+\t\treturn &odb->loose_objects_cache;\n \n \tstrbuf_addstr(&buf, odb->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n+\t\t\t\t &odb->loose_objects_cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn &odb->loose_objects_cache[subdir_nr];\n+\treturn &odb->loose_objects_cache;\n }\n \n void odb_clear_loose_cache(struct object_directory *odb)\n {\n-\tint i;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(odb->loose_objects_cache); i++)\n-\t\toid_array_clear(&odb->loose_objects_cache[i]);\n+\toidtree_destroy(&odb->loose_objects_cache);\n \tmemset(&odb->loose_objects_subdir_seen, 0,\n \t sizeof(odb->loose_objects_subdir_seen));\n }\ndiff --git a/object-name.c b/object-name.c\nindex 64202de60b..3263c19457 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -87,27 +87,21 @@ static void update_candidates(struct disambiguate_state *ds, const struct object\n \n static int match_hash(unsigned, const unsigned char *, const unsigned char *);\n \n+static enum cb_next match_prefix(const struct object_id *oid, void *arg)\n+{\n+\tstruct disambiguate_state *ds = arg;\n+\t/* no need to call match_hash, oidtree_each did prefix match */\n+\tupdate_candidates(ds, oid);\n+\treturn ds->ambiguous ? CB_BREAK : CB_CONTINUE;\n+}\n+\n static void find_short_object_filename(struct disambiguate_state *ds)\n {\n \tstruct object_directory *odb;\n \n-\tfor (odb = ds->repo->objects->odb; odb && !ds->ambiguous; odb = odb->next) {\n-\t\tint pos;\n-\t\tstruct oid_array *loose_objects;\n-\n-\t\tloose_objects = odb_loose_cache(odb, &ds->bin_pfx);\n-\t\tpos = oid_array_lookup(loose_objects, &ds->bin_pfx);\n-\t\tif (pos < 0)\n-\t\t\tpos = -1 - pos;\n-\t\twhile (!ds->ambiguous && pos < loose_objects->nr) {\n-\t\t\tconst struct object_id *oid;\n-\t\t\toid = loose_objects->oid + pos;\n-\t\t\tif (!match_hash(ds->len, ds->bin_pfx.hash, oid->hash))\n-\t\t\t\tbreak;\n-\t\t\tupdate_candidates(ds, oid);\n-\t\t\tpos++;\n-\t\t}\n-\t}\n+\tfor (odb = ds->repo->objects->odb; odb && !ds->ambiguous; odb = odb->next)\n+\t\toidtree_each(odb_loose_cache(odb, &ds->bin_pfx),\n+\t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \n static int match_hash(unsigned len, const unsigned char *a, const unsigned char *b)\ndiff --git a/object-store.h b/object-store.h\nindex 8fcddf3e65..b507108d18 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -9,6 +9,7 @@\n #include \"thread-utils.h\"\n #include \"khash.h\"\n #include \"dir.h\"\n+#include \"oidtree.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -23,7 +24,7 @@ struct object_directory {\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n \tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oid_array loose_objects_cache[256];\n+\tstruct oidtree loose_objects_cache;\n \n \t/*\n \t * Path to the alternative object store. If this is a relative path,\n@@ -69,7 +70,7 @@ void add_to_alternates_memory(const char *dir);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n-struct oid_array *odb_loose_cache(struct object_directory *odb,\n+struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\ndiff --git a/oidtree.c b/oidtree.c\nnew file mode 100644\nindex 0000000000..c1188d8f48\n--- /dev/null\n+++ b/oidtree.c\n@@ -0,0 +1,94 @@\n+/*\n+ * A wrapper around cbtree which stores oids\n+ * May be used to replace oid-array for prefix (abbreviation) matches\n+ */\n+#include \"oidtree.h\"\n+#include \"alloc.h\"\n+#include \"hash.h\"\n+\n+struct oidtree_node {\n+\t/* n.k[] is used to store \"struct object_id\" */\n+\tstruct cb_node n;\n+};\n+\n+struct oidtree_iter_data {\n+\toidtree_iter fn;\n+\tvoid *arg;\n+\tsize_t *last_nibble_at;\n+\tint algo;\n+\tuint8_t last_byte;\n+};\n+\n+void oidtree_destroy(struct oidtree *ot)\n+{\n+\tif (ot->mempool) {\n+\t\tclear_alloc_state(ot->mempool);\n+\t\tFREE_AND_NULL(ot->mempool);\n+\t}\n+\toidtree_init(ot);\n+}\n+\n+void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n+{\n+\tstruct oidtree_node *on;\n+\n+\tif (!ot->mempool)\n+\t\tot->mempool = allocate_alloc_state();\n+\tif (!oid->algo)\n+\t\tBUG(\"oidtree_insert requires oid->algo\");\n+\n+\ton = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n+\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n+\n+\t/*\n+\t * n.b. we shouldn't get duplicates, here, but we'll have\n+\t * a small leak that won't be freed until oidtree_destroy\n+\t */\n+\tcb_insert(&ot->t, &on->n, sizeof(*oid));\n+}\n+\n+int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n+{\n+\tstruct object_id k = { 0 };\n+\tsize_t klen = sizeof(k);\n+\toidcpy_with_padding(&k, oid);\n+\n+\tif (oid->algo == GIT_HASH_UNKNOWN) {\n+\t\tk.algo = hash_algo_by_ptr(the_hash_algo);\n+\t\tklen -= sizeof(oid->algo);\n+\t}\n+\n+\treturn cb_lookup(&ot->t, (const uint8_t *)&k, klen) ? 1 : 0;\n+}\n+\n+static enum cb_next iter(struct cb_node *n, void *arg)\n+{\n+\tstruct oidtree_iter_data *x = arg;\n+\tconst struct object_id *oid = (const struct object_id *)n->k;\n+\n+\tif (x->algo != GIT_HASH_UNKNOWN && x->algo != oid->algo)\n+\t\treturn CB_CONTINUE;\n+\n+\tif (x->last_nibble_at) {\n+\t\tif ((oid->hash[*x->last_nibble_at] ^ x->last_byte) & 0xf0)\n+\t\t\treturn CB_CONTINUE;\n+\t}\n+\n+\treturn x->fn(oid, x->arg);\n+}\n+\n+void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n+\t\t\tsize_t oidhexlen, oidtree_iter fn, void *arg)\n+{\n+\tsize_t klen = oidhexlen / 2;\n+\tstruct oidtree_iter_data x = { 0 };\n+\n+\tx.fn = fn;\n+\tx.arg = arg;\n+\tx.algo = oid->algo;\n+\tif (oidhexlen & 1) {\n+\t\tx.last_byte = oid->hash[klen];\n+\t\tx.last_nibble_at = &klen;\n+\t}\n+\tcb_each(&ot->t, (const uint8_t *)oid, klen, iter, &x);\n+}\ndiff --git a/oidtree.h b/oidtree.h\nnew file mode 100644\nindex 0000000000..73399bb978\n--- /dev/null\n+++ b/oidtree.h\n@@ -0,0 +1,29 @@\n+#ifndef OIDTREE_H\n+#define OIDTREE_H\n+\n+#include \"cbtree.h\"\n+#include \"hash.h\"\n+\n+struct alloc_state;\n+struct oidtree {\n+\tstruct cb_tree t;\n+\tstruct alloc_state *mempool;\n+};\n+\n+#define OIDTREE_INIT { .t = CBTREE_INIT, .mempool = NULL }\n+\n+static inline void oidtree_init(struct oidtree *ot)\n+{\n+\tcb_init(&ot->t);\n+\tot->mempool = NULL;\n+}\n+\n+void oidtree_destroy(struct oidtree *);\n+void oidtree_insert(struct oidtree *, const struct object_id *);\n+int oidtree_contains(struct oidtree *, const struct object_id *);\n+\n+typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *arg);\n+void oidtree_each(struct oidtree *, const struct object_id *,\n+\t\t\tsize_t oidhexlen, oidtree_iter, void *arg);\n+\n+#endif /* OIDTREE_H */\ndiff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\nnew file mode 100644\nindex 0000000000..e0da13eea3\n--- /dev/null\n+++ b/t/helper/test-oidtree.c\n@@ -0,0 +1,47 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"oidtree.h\"\n+\n+static enum cb_next print_oid(const struct object_id *oid, void *data)\n+{\n+\tputs(oid_to_hex(oid));\n+\treturn CB_CONTINUE;\n+}\n+\n+int cmd__oidtree(int argc, const char **argv)\n+{\n+\tstruct oidtree ot = OIDTREE_INIT;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint nongit_ok;\n+\tint algo = GIT_HASH_UNKNOWN;\n+\n+\tsetup_git_directory_gently(&nongit_ok);\n+\n+\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\tconst char *arg;\n+\t\tstruct object_id oid;\n+\n+\t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n+\t\t\tif (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"insert not a hexadecimal oid: %s\", arg);\n+\t\t\talgo = oid.algo;\n+\t\t\toidtree_insert(&ot, &oid);\n+\t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(\"contains not a hexadecimal oid: %s\", arg);\n+\t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n+\t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n+\t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n+\t\t\tmemset(&oid, 0, sizeof(oid));\n+\t\t\tmemcpy(buf, arg, strlen(arg));\n+\t\t\tbuf[hash_algos[algo].hexsz] = 0;\n+\t\t\tget_oid_hex_any(buf, &oid);\n+\t\t\toid.algo = algo;\n+\t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n+\t\t} else if (!strcmp(line.buf, \"destroy\"))\n+\t\t\toidtree_destroy(&ot);\n+\t\telse\n+\t\t\tdie(\"unknown command: %s\", line.buf);\n+\t}\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..9d37debf28 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n \t{ \"mktemp\", cmd__mktemp },\n \t{ \"oid-array\", cmd__oid_array },\n \t{ \"oidmap\", cmd__oidmap },\n+\t{ \"oidtree\", cmd__oidtree },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f683a2f59c 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -32,6 +32,7 @@ int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n int cmd__oidmap(int argc, const char **argv);\n+int cmd__oidtree(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\ndiff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\nnew file mode 100755\nindex 0000000000..0594f57c81\n--- /dev/null\n+++ b/t/t0069-oidtree.sh\n@@ -0,0 +1,52 @@\n+#!/bin/sh\n+\n+test_description='basic tests for the oidtree implementation'\n+. ./test-lib.sh\n+\n+echoid () {\n+\tprefix=\"${1:+$1 }\"\n+\tshift\n+\twhile test $# -gt 0\n+\tdo\n+\t\techo \"$1\"\n+\t\tshift\n+\tdone | awk -v prefix=\"$prefix\" -v ZERO_OID=$ZERO_OID '{\n+\t\tprintf(\"%s%s\", prefix, $0);\n+\t\tneed = length(ZERO_OID) - length($0);\n+\t\tfor (i = 0; i < need; i++)\n+\t\t\tprintf(\"0\");\n+\t\tprintf \"\\n\";\n+\t}'\n+}\n+\n+test_expect_success 'oidtree insert and contains' '\n+\tcat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'oidtree each' '\n+\techoid \"\" 123 321 321 >expect &&\n+\t{\n+\t\techoid insert f 9 8 123 321 a b c d e\n+\t\techo each 12300\n+\t\techo each 3211\n+\t\techo each 3210\n+\t\techo each 32100\n+\t\techo destroy\n+\t} | test-tool oidtree >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n"},{"id":"429105","messageId":"fc342ddd-1b35-62cc-dd4b-e0462d595819@web.de","threadId":"55998","inReplyTo":"20210629205305.7100-2-e@80x24.org","subject":"Re: [PATCH v2 1/5] speed up alt_odb_usable() with many alternates","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-07-03T10:05:44Z","receivedAt":"2021-07-03T10:06:07Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 29.06.21 um 22:53 schrieb Eric Wong:\n> With many alternates, the duplicate check in alt_odb_usable()\n> wastes many cycles doing repeated fspathcmp() on every existing\n> alternate. Use a khash to speed up lookups by odb->path.\n>\n> Since the kh_put_* API uses the supplied key without\n> duplicating it, we also take advantage of it to replace both\n> xstrdup() and strbuf_release() in link_alt_odb_entry() with\n> strbuf_detach() to avoid the allocation and copy.\n>\n> In a test repository with 50K alternates and each of those 50K\n> alternates having one alternate each (for a total of 100K total\n> alternates); this speeds up lookup of a non-existent blob from\n> over 16 minutes to roughly 2.7 seconds on my busy workstation.\n\nYay for hashmaps! :)\n\n> Note: all underlying git object directories were small and\n> unpacked with only loose objects and no packs. Having to load\n> packs increases times significantly.\n>\n> Signed-off-by: Eric Wong \n> ---\n> object-file.c | 33 ++++++++++++++++++++++-----------\n> object-store.h | 17 +++++++++++++++++\n> object.c | 2 ++\n> 3 files changed, 41 insertions(+), 11 deletions(-)\n>\n> diff --git a/object-file.c b/object-file.c\n> index f233b440b2..304af3a172 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -517,9 +517,9 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n> */\n> static int alt_odb_usable(struct raw_object_store *o,\n> \t\t\t struct strbuf *path,\n> -\t\t\t const char *normalized_objdir)\n> +\t\t\t const char *normalized_objdir, khiter_t *pos)\n> {\n> -\tstruct object_directory *odb;\n> +\tint r;\n>\n> \t/* Detect cases where alternate disappeared */\n> \tif (!is_directory(path->buf)) {\n> @@ -533,14 +533,22 @@ static int alt_odb_usable(struct raw_object_store *o,\n> \t * Prevent the common mistake of listing the same\n> \t * thing twice, or object directory itself.\n> \t */\n> -\tfor (odb = o->odb; odb; odb = odb->next) {\n> -\t\tif (!fspathcmp(path->buf, odb->path))\n> -\t\t\treturn 0;\n> +\tif (!o->odb_by_path) {\n> +\t\tkhiter_t p;\n> +\n> +\t\to->odb_by_path = kh_init_odb_path_map();\n> +\t\tassert(!o->odb->next);\n> +\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n\nSo on the first run you not just create the hashmap, but you also\npre-populate it with the main object directory. Makes sense. The\nhashmap wouldn't even be created in repositories without alternates.\n\n> +\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n\nOur other callers don't handle a negative return code because it would\nindicate an allocation failure, and in our version we use ALLOC_ARRAY,\nwhich dies on error. So you don't need that check here, but we better\nclarify that in khash.h.\n\n> +\t\tassert(r == 1); /* never used */\n> +\t\tkh_value(o->odb_by_path, p) = o->odb;\n> \t}\n> \tif (!fspathcmp(path->buf, normalized_objdir))\n> \t\treturn 0;\n> -\n> -\treturn 1;\n> +\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n> +\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n\nDito.\n\n> +\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n> +\treturn r == 0 ? 0 : 1;\n\nThe comment indicates that khash would be nicer to use if it had an\nenum for the kh_put return values. Perhaps, but that should be done in\nanother series.\n\nI like the solution in oidset.c to make this more readable, though: Call\nthe return value \"added\" instead of \"r\" and then a \"return !added;\"\nmakes sense without additional comments.\n\n> }\n>\n> /*\n> @@ -566,6 +574,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n> {\n> \tstruct object_directory *ent;\n> \tstruct strbuf pathbuf = STRBUF_INIT;\n> +\tkhiter_t pos;\n>\n> \tif (!is_absolute_path(entry) && relative_base) {\n> \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n> @@ -587,23 +596,25 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n> \twhile (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')\n> \t\tstrbuf_setlen(&pathbuf, pathbuf.len - 1);\n>\n> -\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {\n> +\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir, &pos)) {\n> \t\tstrbuf_release(&pathbuf);\n> \t\treturn -1;\n> \t}\n>\n> \tCALLOC_ARRAY(ent, 1);\n> -\tent->path = xstrdup(pathbuf.buf);\n> +\t/* pathbuf.buf is already in r->objects->odb_by_path */\n\nTricky stuff (to me), important comment.\n\n> +\tent->path = strbuf_detach(&pathbuf, NULL);\n>\n> \t/* add the alternate entry */\n> \t*r->objects->odb_tail = ent;\n> \tr->objects->odb_tail = &(ent->next);\n> \tent->next = NULL;\n> +\tassert(r->objects->odb_by_path);\n> +\tkh_value(r->objects->odb_by_path, pos) = ent;\n>\n> \t/* recursively add alternates */\n> -\tread_info_alternates(r, pathbuf.buf, depth + 1);\n> +\tread_info_alternates(r, ent->path, depth + 1);\n>\n> -\tstrbuf_release(&pathbuf);\n> \treturn 0;\n> }\n>\n> diff --git a/object-store.h b/object-store.h\n> index ec32c23dcb..20c1cedb75 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -7,6 +7,8 @@\n> #include \"oid-array.h\"\n> #include \"strbuf.h\"\n> #include \"thread-utils.h\"\n> +#include \"khash.h\"\n> +#include \"dir.h\"\n>\n> struct object_directory {\n> \tstruct object_directory *next;\n> @@ -30,6 +32,19 @@ struct object_directory {\n> \tchar *path;\n> };\n>\n> +static inline int odb_path_eq(const char *a, const char *b)\n> +{\n> +\treturn !fspathcmp(a, b);\n> +}\n\nThis is not specific to the object store. It could be called fspatheq\nand live in dir.h. Or dir.c -- a surprising amount of code seems to\nnecessary for that negation (https://godbolt.org/z/MY7Wda3a7). Anyway,\nit's just an idea for another series.\n\n> +\n> +static inline int odb_path_hash(const char *str)\n> +{\n> +\treturn ignore_case ? strihash(str) : __ac_X31_hash_string(str);\n> +}\n\nThe internal Attractive Chaos (__ac_*) macros should be left confined\nto khash.h, I think. Its alias kh_str_hash_func would be better\nsuited here.\n\nDo we want to use the K&R hash function here at all, though? If we\nuse FNV-1 when ignoring case, why not also use it (i.e. strhash) when\nrespecting it? At least that's done in builtin/sparse-checkout.c,\ndir.c and merge-recursive.c. This is just handwaving and yammering\nabout lack of symmetry, but I do wonder how your performance numbers\nlook with strhash. If it's fine then we could package this up as\nfspathhash..\n\nAnd I also wonder how it looks if you use strihash unconditionally.\nI guess case collisions are usually rare and branching based on a\nglobal variable may be more expensive than case folding..\n\nAnyway, just ideas; kh_str_hash_func would be OK as well.\n\n> +\n> +KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n> +\tstruct object_directory *, 1, odb_path_hash, odb_path_eq);\n> +\n> void prepare_alt_odb(struct repository *r);\n> char *compute_alternate_path(const char *path, struct strbuf *err);\n> typedef int alt_odb_fn(struct object_directory *, void *);\n> @@ -116,6 +131,8 @@ struct raw_object_store {\n> \t */\n> \tstruct object_directory *odb;\n> \tstruct object_directory **odb_tail;\n> +\tkh_odb_path_map_t *odb_by_path;\n> +\n> \tint loaded_alternates;\n>\n> \t/*\n> diff --git a/object.c b/object.c\n> index 14188453c5..2b3c075a15 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -511,6 +511,8 @@ static void free_object_directories(struct raw_object_store *o)\n> \t\tfree_object_directory(o->odb);\n> \t\to->odb = next;\n> \t}\n> +\tkh_destroy_odb_path_map(o->odb_by_path);\n> +\to->odb_by_path = NULL;\n> }\n>\n> void raw_object_store_clear(struct raw_object_store *o)\n>\n"},{"id":"429156","messageId":"d069bc0b-f989-0d15-61ec-19f85841bc51@web.de","threadId":"55998","inReplyTo":"fc342ddd-1b35-62cc-dd4b-e0462d595819@web.de","subject":"Re: [PATCH v2 1/5] speed up alt_odb_usable() with many alternates","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-07-04T09:02:19Z","receivedAt":"2021-07-04T09:02:29Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.07.21 um 12:05 schrieb René Scharfe:\n> Am 29.06.21 um 22:53 schrieb Eric Wong:\n>> +\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n>> +\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n\n>> +\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n>> +\treturn r == 0 ? 0 : 1;\n\n> I like the solution in oidset.c to make this more readable, though: Call\n> the return value \"added\" instead of \"r\" and then a \"return !added;\"\n> makes sense without additional comments.\n\nThat's probably because I wrote that part; see 8b2f8cbcb1 (oidset: use\nkhash, 2018-10-04) -- I had somehow forgotten about that. o_O\n\nAnd here we wouldn't negate. Passing on the value verbatim, without\nnormalizing 2 to 1, would work fine.\n\nalt_odb_usable() and its caller become quite entangled due to the\nhashmap insert operation being split between them. I suspect the code\nwould improve by inlining the function in a follow-up patch, making\nreturn code considerations moot. The improvement is not significant\nenough to hold up this series in case you don't like the idea, though.\n\nRough demo:\n\n object-file.c | 82 +++++++++++++++++++++++++++--------------------------------\n 1 file changed, 37 insertions(+), 45 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 304af3a172..a5e91091ee 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -512,45 +512,6 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n \treturn odb_loose_path(r->objects->odb, buf, oid);\n }\n\n-/*\n- * Return non-zero iff the path is usable as an alternate object database.\n- */\n-static int alt_odb_usable(struct raw_object_store *o,\n-\t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir, khiter_t *pos)\n-{\n-\tint r;\n-\n-\t/* Detect cases where alternate disappeared */\n-\tif (!is_directory(path->buf)) {\n-\t\terror(_(\"object directory %s does not exist; \"\n-\t\t\t\"check .git/objects/info/alternates\"),\n-\t\t path->buf);\n-\t\treturn 0;\n-\t}\n-\n-\t/*\n-\t * Prevent the common mistake of listing the same\n-\t * thing twice, or object directory itself.\n-\t */\n-\tif (!o->odb_by_path) {\n-\t\tkhiter_t p;\n-\n-\t\to->odb_by_path = kh_init_odb_path_map();\n-\t\tassert(!o->odb->next);\n-\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n-\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n-\t\tassert(r == 1); /* never used */\n-\t\tkh_value(o->odb_by_path, p) = o->odb;\n-\t}\n-\tif (!fspathcmp(path->buf, normalized_objdir))\n-\t\treturn 0;\n-\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n-\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n-\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n-\treturn r == 0 ? 0 : 1;\n-}\n-\n /*\n * Prepare alternate object database registry.\n *\n@@ -575,6 +536,9 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tkhiter_t pos;\n+\tint ret = -1;\n+\tint added;\n+\tstruct raw_object_store *o = r->objects;\n\n \tif (!is_absolute_path(entry) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -585,8 +549,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \tif (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {\n \t\terror(_(\"unable to normalize alternate object path: %s\"),\n \t\t pathbuf.buf);\n-\t\tstrbuf_release(&pathbuf);\n-\t\treturn -1;\n+\t\tgoto out;\n \t}\n\n \t/*\n@@ -596,11 +559,37 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \twhile (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')\n \t\tstrbuf_setlen(&pathbuf, pathbuf.len - 1);\n\n-\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir, &pos)) {\n-\t\tstrbuf_release(&pathbuf);\n-\t\treturn -1;\n+\t/* Detect cases where alternate disappeared */\n+\tif (!is_directory(pathbuf.buf)) {\n+\t\terror(_(\"object directory %s does not exist; \"\n+\t\t\t\"check .git/objects/info/alternates\"),\n+\t\t pathbuf.buf);\n+\t\tgoto out;\n+\t}\n+\n+\t/*\n+\t * Prevent the common mistake of listing the same\n+\t * thing twice, or object directory itself.\n+\t */\n+\tif (!o->odb_by_path) {\n+\t\tkhiter_t p;\n+\n+\t\to->odb_by_path = kh_init_odb_path_map();\n+\t\tassert(!o->odb->next);\n+\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &added);\n+\t\tif (added < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\t\tassert(added);\n+\t\tkh_value(o->odb_by_path, p) = o->odb;\n \t}\n\n+\tif (!fspathcmp(pathbuf.buf, normalized_objdir))\n+\t\tgoto out;\n+\n+\tpos = kh_put_odb_path_map(o->odb_by_path, pathbuf.buf, &added);\n+\tif (added < 0) die_errno(_(\"kh_put_odb_path_map\"));\n+\tif (!added)\n+\t\tgoto out;\n+\n \tCALLOC_ARRAY(ent, 1);\n \t/* pathbuf.buf is already in r->objects->odb_by_path */\n \tent->path = strbuf_detach(&pathbuf, NULL);\n@@ -615,7 +604,10 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \t/* recursively add alternates */\n \tread_info_alternates(r, ent->path, depth + 1);\n\n-\treturn 0;\n+\tret = 0;\n+out:\n+\tstrbuf_release(&pathbuf);\n+\treturn ret;\n }\n\n static const char *parse_alt_odb_entry(const char *string,\n\n"},{"id":"429157","messageId":"ab757bce-3b51-afac-312c-ea2e883cf0bf@web.de","threadId":"55998","inReplyTo":"20210629205305.7100-6-e@80x24.org","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-07-04T09:02:29Z","receivedAt":"2021-07-04T09:02:41Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 29.06.21 um 22:53 schrieb Eric Wong:\n> This saves 8K per `struct object_directory', meaning it saves\n> around 800MB in my case involving 100K alternates (half or more\n> of those alternates are unlikely to hold loose objects).\n>\n> This is implemented in two parts: a generic, allocation-free\n> `cbtree' and the `oidtree' wrapper on top of it. The latter\n> provides allocation using alloc_state as a memory pool to\n> improve locality and reduce free(3) overhead.\n>\n> Unlike oid-array, the crit-bit tree does not require sorting.\n> Performance is bound by the key length, for oidtree that is\n> fixed at sizeof(struct object_id). There's no need to have\n> 256 oidtrees to mitigate the O(n log n) overhead like we did\n> with oid-array.\n>\n> Being a prefix trie, it is natively suited for expanding short\n> object IDs via prefix-limited iteration in\n> `find_short_object_filename'.\n\nSounds like a good match.\n\n>\n> On my busy workstation, p4205 performance seems to be roughly\n> unchanged (+/-8%). Startup with 100K total alternates with no\n> loose objects seems around 10-20% faster on a hot cache.\n> (800MB in memory savings means more memory for the kernel FS\n> cache).\n>\n> The generic cbtree implementation does impose some extra\n> overhead for oidtree in that it uses memcmp(3) on\n> \"struct object_id\" so it wastes cycles comparing 12 extra bytes\n> on SHA-1 repositories. I've not yet explored reducing this\n> overhead, but I expect there are many places in our code base\n> where we'd want to investigate this.\n>\n> More information on crit-bit trees: https://cr.yp.to/critbit.html\n>\n> v2: make oidtree test hash-agnostic\n>\n> Signed-off-by: Eric Wong \n> ---\n> Makefile | 3 +\n> alloc.c | 6 ++\n> alloc.h | 1 +\n> cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n> cbtree.h | 56 ++++++++++++++\n> object-file.c | 17 ++--\n> object-name.c | 28 +++----\n> object-store.h | 5 +-\n> oidtree.c | 94 ++++++++++++++++++++++\n> oidtree.h | 29 +++++++\n> t/helper/test-oidtree.c | 47 +++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t0069-oidtree.sh | 52 +++++++++++++\n> 14 files changed, 478 insertions(+), 29 deletions(-)\n> create mode 100644 cbtree.c\n> create mode 100644 cbtree.h\n> create mode 100644 oidtree.c\n> create mode 100644 oidtree.h\n> create mode 100644 t/helper/test-oidtree.c\n> create mode 100755 t/t0069-oidtree.sh\n>\n> diff --git a/Makefile b/Makefile\n> index c3565fc0f8..a1525978fb 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -722,6 +722,7 @@ TEST_BUILTINS_OBJS += test-mergesort.o\n> TEST_BUILTINS_OBJS += test-mktemp.o\n> TEST_BUILTINS_OBJS += test-oid-array.o\n> TEST_BUILTINS_OBJS += test-oidmap.o\n> +TEST_BUILTINS_OBJS += test-oidtree.o\n> TEST_BUILTINS_OBJS += test-online-cpus.o\n> TEST_BUILTINS_OBJS += test-parse-options.o\n> TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n> @@ -845,6 +846,7 @@ LIB_OBJS += branch.o\n> LIB_OBJS += bulk-checkin.o\n> LIB_OBJS += bundle.o\n> LIB_OBJS += cache-tree.o\n> +LIB_OBJS += cbtree.o\n> LIB_OBJS += chdir-notify.o\n> LIB_OBJS += checkout.o\n> LIB_OBJS += chunk-format.o\n> @@ -940,6 +942,7 @@ LIB_OBJS += object.o\n> LIB_OBJS += oid-array.o\n> LIB_OBJS += oidmap.o\n> LIB_OBJS += oidset.o\n> +LIB_OBJS += oidtree.o\n> LIB_OBJS += pack-bitmap-write.o\n> LIB_OBJS += pack-bitmap.o\n> LIB_OBJS += pack-check.o\n> diff --git a/alloc.c b/alloc.c\n> index 957a0af362..ca1e178c5a 100644\n> --- a/alloc.c\n> +++ b/alloc.c\n> @@ -14,6 +14,7 @@\n> #include \"tree.h\"\n> #include \"commit.h\"\n> #include \"tag.h\"\n> +#include \"oidtree.h\"\n> #include \"alloc.h\"\n>\n> #define BLOCKING 1024\n> @@ -123,6 +124,11 @@ void *alloc_commit_node(struct repository *r)\n> \treturn c;\n> }\n>\n> +void *alloc_from_state(struct alloc_state *alloc_state, size_t n)\n> +{\n> +\treturn alloc_node(alloc_state, n);\n> +}\n> +\n\nWhy extend alloc.c instead of using mem-pool.c? (I don't know which fits\nbetter, but when you say \"memory pool\" and not use mem-pool.c I just have\nto ask..)\n\n> diff --git a/oidtree.c b/oidtree.c\n> new file mode 100644\n> index 0000000000..c1188d8f48\n> --- /dev/null\n> +++ b/oidtree.c\n> @@ -0,0 +1,94 @@\n> +/*\n> + * A wrapper around cbtree which stores oids\n> + * May be used to replace oid-array for prefix (abbreviation) matches\n> + */\n> +#include \"oidtree.h\"\n> +#include \"alloc.h\"\n> +#include \"hash.h\"\n> +\n> +struct oidtree_node {\n> +\t/* n.k[] is used to store \"struct object_id\" */\n> +\tstruct cb_node n;\n> +};\n> +\n> +struct oidtree_iter_data {\n> +\toidtree_iter fn;\n> +\tvoid *arg;\n> +\tsize_t *last_nibble_at;\n> +\tint algo;\n> +\tuint8_t last_byte;\n> +};\n> +\n> +void oidtree_destroy(struct oidtree *ot)\n> +{\n> +\tif (ot->mempool) {\n> +\t\tclear_alloc_state(ot->mempool);\n> +\t\tFREE_AND_NULL(ot->mempool);\n> +\t}\n> +\toidtree_init(ot);\n> +}\n> +\n> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n> +{\n> +\tstruct oidtree_node *on;\n> +\n> +\tif (!ot->mempool)\n> +\t\tot->mempool = allocate_alloc_state();\n> +\tif (!oid->algo)\n> +\t\tBUG(\"oidtree_insert requires oid->algo\");\n> +\n> +\ton = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n> +\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n> +\n> +\t/*\n> +\t * n.b. we shouldn't get duplicates, here, but we'll have\n> +\t * a small leak that won't be freed until oidtree_destroy\n> +\t */\n\nWhy shouldn't we get duplicates? That depends on the usage of oidtree,\nright? The current user is fine because we avoid reading the same loose\nobject directory twice using the loose_objects_subdir_seen bitmap.\n\nThe leak comes from the allocation above, which is not used in case we\nalready have the key in the oidtree. So we need memory for all\ncandidates, not just the inserted candidates. That's probably\nacceptable in most use cases.\n\nWe can do better by keeping track of the unnecessary allocation in\nstruct oidtree and recycling it at the next insert attempt, however.\nThat way we'd only waste at most one slot.\n\n> +\tcb_insert(&ot->t, &on->n, sizeof(*oid));\n> +}\n> +\n> +int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n> +{\n> +\tstruct object_id k = { 0 };\n> +\tsize_t klen = sizeof(k);\n> +\toidcpy_with_padding(&k, oid);\n\nWhy initialize k; isn't oidcpy_with_padding() supposed to overwrite it\ncompletely?\n\n> +\n> +\tif (oid->algo == GIT_HASH_UNKNOWN) {\n> +\t\tk.algo = hash_algo_by_ptr(the_hash_algo);\n> +\t\tklen -= sizeof(oid->algo);\n> +\t}\n\nThis relies on the order of the members hash and algo in struct\nobject_id to find a matching hash if we don't actually know algo. It\nalso relies on the absence of padding after algo. Would something like\nthis make sense?\n\n BUILD_ASSERT_OR_ZERO(offsetof(struct object_id, algo) + sizeof(k.algo) == sizeof(k));\n\nAnd why set k.algo to some arbitrary value if we ignore it anyway? I.e.\nwhy not keep it GIT_HASH_UNKNOWN, as set by oidcpy_with_padding()?\n\n> +\n> +\treturn cb_lookup(&ot->t, (const uint8_t *)&k, klen) ? 1 : 0;\n> +}\n> +\n> +static enum cb_next iter(struct cb_node *n, void *arg)\n> +{\n> +\tstruct oidtree_iter_data *x = arg;\n> +\tconst struct object_id *oid = (const struct object_id *)n->k;\n> +\n> +\tif (x->algo != GIT_HASH_UNKNOWN && x->algo != oid->algo)\n> +\t\treturn CB_CONTINUE;\n> +\n> +\tif (x->last_nibble_at) {\n> +\t\tif ((oid->hash[*x->last_nibble_at] ^ x->last_byte) & 0xf0)\n> +\t\t\treturn CB_CONTINUE;\n> +\t}\n> +\n> +\treturn x->fn(oid, x->arg);\n> +}\n> +\n> +void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n> +\t\t\tsize_t oidhexlen, oidtree_iter fn, void *arg)\n> +{\n> +\tsize_t klen = oidhexlen / 2;\n> +\tstruct oidtree_iter_data x = { 0 };\n> +\n> +\tx.fn = fn;\n> +\tx.arg = arg;\n> +\tx.algo = oid->algo;\n> +\tif (oidhexlen & 1) {\n> +\t\tx.last_byte = oid->hash[klen];\n> +\t\tx.last_nibble_at = &klen;\n> +\t}\n> +\tcb_each(&ot->t, (const uint8_t *)oid, klen, iter, &x);\n> +}\n\nClamp oidhexlen at GIT_MAX_HEXSZ? Or die?\n\nRené\n"},{"id":"429160","messageId":"87zgv276lf.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"20210629205305.7100-6-e@80x24.org","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-04T09:32:50Z","receivedAt":"2021-07-04T09:48:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jun 29 2021, Eric Wong wrote:\n\n> +struct alloc_state;\n> +struct oidtree {\n> +\tstruct cb_tree t;\n\ns/t/tree/? Too short a name for an interface IMO.\n\n> +\tstruct alloc_state *mempool;\n> +};\n> +\n> +#define OIDTREE_INIT { .t = CBTREE_INIT, .mempool = NULL }\n\nLet's use designated initilaizers for new code. Just:\n\n\t#define OIDTREE_init { \\\n\t\t.tere = CBTREE_INIT, \\\n\t}\n\nWill do, no need for the \".mempool = NULL\"\n\n> +static inline void oidtree_init(struct oidtree *ot)\n> +{\n> +\tcb_init(&ot->t);\n> +\tot->mempool = NULL;\n> +}\n\nYou can use the \"memcpy() a blank\" trick/idiom here:\nhttps://lore.kernel.org/git/patch-2.5-955dbd1693d-20210701T104855Z-avarab@gmail.com/\n\nAlso, is this even needed? Why have the \"destroy\" re-initialize it?\n\n> +void oidtree_destroy(struct oidtree *);\n\nMaybe s/destroy/release/, or if you actually need that reset behavior\noidtree_reset(). We've got\n\n> +void oidtree_insert(struct oidtree *, const struct object_id *);\n> +int oidtree_contains(struct oidtree *, const struct object_id *);\n> +\n> +typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *arg);\n\nAn \"arg\" name for some arguments, but none for others, if there's a name\nhere call it \"data\" like you do elswhere?\n\n> +void oidtree_each(struct oidtree *, const struct object_id *,\n> +\t\t\tsize_t oidhexlen, oidtree_iter, void *arg);\n\ns/oidhexlen/hexsz/, like in git_hash_algo.a\n\n> +\n> +#endif /* OIDTREE_H */\n> diff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\n> new file mode 100644\n> index 0000000000..e0da13eea3\n> --- /dev/null\n> +++ b/t/helper/test-oidtree.c\n> @@ -0,0 +1,47 @@\n> +#include \"test-tool.h\"\n> +#include \"cache.h\"\n> +#include \"oidtree.h\"\n> +\n> +static enum cb_next print_oid(const struct object_id *oid, void *data)\n> +{\n> +\tputs(oid_to_hex(oid));\n> +\treturn CB_CONTINUE;\n> +}\n> +\n> +int cmd__oidtree(int argc, const char **argv)\n> +{\n> +\tstruct oidtree ot = OIDTREE_INIT;\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\tint nongit_ok;\n> +\tint algo = GIT_HASH_UNKNOWN;\n> +\n> +\tsetup_git_directory_gently(&nongit_ok);\n> +\n> +\twhile (strbuf_getline(&line, stdin) != EOF) {\n> +\t\tconst char *arg;\n> +\t\tstruct object_id oid;\n> +\n> +\t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n> +\t\t\tif (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN)\n> +\t\t\t\tdie(\"insert not a hexadecimal oid: %s\", arg);\n> +\t\t\talgo = oid.algo;\n> +\t\t\toidtree_insert(&ot, &oid);\n> +\t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n> +\t\t\tif (get_oid_hex(arg, &oid))\n> +\t\t\t\tdie(\"contains not a hexadecimal oid: %s\", arg);\n> +\t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n> +\t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n> +\t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n> +\t\t\tmemset(&oid, 0, sizeof(oid));\n> +\t\t\tmemcpy(buf, arg, strlen(arg));\n> +\t\t\tbuf[hash_algos[algo].hexsz] = 0;\n\n= '\\0' if it's the intent to have a NULL-terminated string is more\nreadable.\n\n> +\t\t\tget_oid_hex_any(buf, &oid);\n> +\t\t\toid.algo = algo;\n> +\t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n> +\t\t} else if (!strcmp(line.buf, \"destroy\"))\n> +\t\t\toidtree_destroy(&ot);\n> +\t\telse\n> +\t\t\tdie(\"unknown command: %s\", line.buf);\n\nMissing braces.\n\n> +\t}\n> +\treturn 0;\n> +}\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index c5bd0c6d4c..9d37debf28 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n> \t{ \"mktemp\", cmd__mktemp },\n> \t{ \"oid-array\", cmd__oid_array },\n> \t{ \"oidmap\", cmd__oidmap },\n> +\t{ \"oidtree\", cmd__oidtree },\n> \t{ \"online-cpus\", cmd__online_cpus },\n> \t{ \"parse-options\", cmd__parse_options },\n> \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index e8069a3b22..f683a2f59c 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -32,6 +32,7 @@ int cmd__match_trees(int argc, const char **argv);\n> int cmd__mergesort(int argc, const char **argv);\n> int cmd__mktemp(int argc, const char **argv);\n> int cmd__oidmap(int argc, const char **argv);\n> +int cmd__oidtree(int argc, const char **argv);\n> int cmd__online_cpus(int argc, const char **argv);\n> int cmd__parse_options(int argc, const char **argv);\n> int cmd__parse_pathspec_file(int argc, const char** argv);\n> diff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\n> new file mode 100755\n> index 0000000000..0594f57c81\n> --- /dev/null\n> +++ b/t/t0069-oidtree.sh\n> @@ -0,0 +1,52 @@\n> +#!/bin/sh\n> +\n> +test_description='basic tests for the oidtree implementation'\n> +. ./test-lib.sh\n> +\n> +echoid () {\n> +\tprefix=\"${1:+$1 }\"\n> +\tshift\n> +\twhile test $# -gt 0\n> +\tdo\n> +\t\techo \"$1\"\n> +\t\tshift\n> +\tdone | awk -v prefix=\"$prefix\" -v ZERO_OID=$ZERO_OID '{\n> +\t\tprintf(\"%s%s\", prefix, $0);\n> +\t\tneed = length(ZERO_OID) - length($0);\n> +\t\tfor (i = 0; i < need; i++)\n> +\t\t\tprintf(\"0\");\n> +\t\tprintf \"\\n\";\n> +\t}'\n> +}\n\nLooks fairly easy to do in pure-shell, first of all you don't need a\nlength() on $ZERO_OID, use $(test_oid hexsz) instead. That applies for\nthe awk version too.\n\nBut once you have that and the N arguments just do a wc -c on the\nargument, use $(()) to compute the $difference, and a loop with:\n\n printf \"%s%s%0${difference}d\" \"$prefix\" \"$shortoid\" \"0\"\n\n> +\n> +test_expect_success 'oidtree insert and contains' '\n> +\tcat >expect < +0\n> +0\n> +0\n> +1\n> +1\n> +0\n> +EOF\n\nuse \"<<-\\EOF\" and indent it.\n\n> +\t{\n> +\t\techoid insert 444 1 2 3 4 5 a b c d e &&\n> +\t\techoid contains 44 441 440 444 4440 4444\n> +\t\techo destroy\n> +\t} | test-tool oidtree >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'oidtree each' '\n> +\techoid \"\" 123 321 321 >expect &&\n> +\t{\n> +\t\techoid insert f 9 8 123 321 a b c d e\n> +\t\techo each 12300\n> +\t\techo each 3211\n> +\t\techo each 3210\n> +\t\techo each 32100\n> +\t\techo destroy\n> +\t} | test-tool oidtree >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n\n"},{"id":"429376","messageId":"20210706230115.GA8624@dcvr","threadId":"55998","inReplyTo":"fc342ddd-1b35-62cc-dd4b-e0462d595819@web.de","subject":"Re: [PATCH v2 1/5] speed up alt_odb_usable() with many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-06T23:01:15Z","receivedAt":"2021-07-06T23:01:16Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"René Scharfe wrote:\n> Am 29.06.21 um 22:53 schrieb Eric Wong:\n> > With many alternates, the duplicate check in alt_odb_usable()\n> > wastes many cycles doing repeated fspathcmp() on every existing\n> > alternate. Use a khash to speed up lookups by odb->path.\n> >\n> > Since the kh_put_* API uses the supplied key without\n> > duplicating it, we also take advantage of it to replace both\n> > xstrdup() and strbuf_release() in link_alt_odb_entry() with\n> > strbuf_detach() to avoid the allocation and copy.\n> >\n> > In a test repository with 50K alternates and each of those 50K\n> > alternates having one alternate each (for a total of 100K total\n> > alternates); this speeds up lookup of a non-existent blob from\n> > over 16 minutes to roughly 2.7 seconds on my busy workstation.\n> \n> Yay for hashmaps! :)\n> \n> > Note: all underlying git object directories were small and\n> > unpacked with only loose objects and no packs. Having to load\n> > packs increases times significantly.\n> >\n> > Signed-off-by: Eric Wong \n> > ---\n> > object-file.c | 33 ++++++++++++++++++++++-----------\n> > object-store.h | 17 +++++++++++++++++\n> > object.c | 2 ++\n> > 3 files changed, 41 insertions(+), 11 deletions(-)\n> >\n> > diff --git a/object-file.c b/object-file.c\n> > index f233b440b2..304af3a172 100644\n> > --- a/object-file.c\n> > +++ b/object-file.c\n> > @@ -517,9 +517,9 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n> > */\n> > static int alt_odb_usable(struct raw_object_store *o,\n> > \t\t\t struct strbuf *path,\n> > -\t\t\t const char *normalized_objdir)\n> > +\t\t\t const char *normalized_objdir, khiter_t *pos)\n> > {\n> > -\tstruct object_directory *odb;\n> > +\tint r;\n> >\n> > \t/* Detect cases where alternate disappeared */\n> > \tif (!is_directory(path->buf)) {\n> > @@ -533,14 +533,22 @@ static int alt_odb_usable(struct raw_object_store *o,\n> > \t * Prevent the common mistake of listing the same\n> > \t * thing twice, or object directory itself.\n> > \t */\n> > -\tfor (odb = o->odb; odb; odb = odb->next) {\n> > -\t\tif (!fspathcmp(path->buf, odb->path))\n> > -\t\t\treturn 0;\n> > +\tif (!o->odb_by_path) {\n> > +\t\tkhiter_t p;\n> > +\n> > +\t\to->odb_by_path = kh_init_odb_path_map();\n> > +\t\tassert(!o->odb->next);\n> > +\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n> \n> So on the first run you not just create the hashmap, but you also\n> pre-populate it with the main object directory. Makes sense. The\n> hashmap wouldn't even be created in repositories without alternates.\n> \n> > +\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n> \n> Our other callers don't handle a negative return code because it would\n> indicate an allocation failure, and in our version we use ALLOC_ARRAY,\n> which dies on error. So you don't need that check here, but we better\n> clarify that in khash.h.\n> \n> > +\t\tassert(r == 1); /* never used */\n> > +\t\tkh_value(o->odb_by_path, p) = o->odb;\n> > \t}\n> > \tif (!fspathcmp(path->buf, normalized_objdir))\n> > \t\treturn 0;\n> > -\n> > -\treturn 1;\n> > +\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n> > +\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n> \n> Dito.\n> \n> > +\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n> > +\treturn r == 0 ? 0 : 1;\n> \n> The comment indicates that khash would be nicer to use if it had an\n> enum for the kh_put return values. Perhaps, but that should be done in\n> another series.\n\nAgreed for another series. I've also found myself wishing khash\nused enums. But I'm also not sure how much changing of 3rd\nparty code we should be doing...\n\n> I like the solution in oidset.c to make this more readable, though: Call\n> the return value \"added\" instead of \"r\" and then a \"return !added;\"\n> makes sense without additional comments.\n> \n> > }\n> >\n> > /*\n> > diff --git a/object-store.h b/object-store.h\n> > index ec32c23dcb..20c1cedb75 100644\n> > --- a/object-store.h\n> > +++ b/object-store.h\n> > @@ -7,6 +7,8 @@\n> > #include \"oid-array.h\"\n> > #include \"strbuf.h\"\n> > #include \"thread-utils.h\"\n> > +#include \"khash.h\"\n> > +#include \"dir.h\"\n> >\n> > struct object_directory {\n> > \tstruct object_directory *next;\n> > @@ -30,6 +32,19 @@ struct object_directory {\n> > \tchar *path;\n> > };\n> >\n> > +static inline int odb_path_eq(const char *a, const char *b)\n> > +{\n> > +\treturn !fspathcmp(a, b);\n> > +}\n> \n> This is not specific to the object store. It could be called fspatheq\n> and live in dir.h. Or dir.c -- a surprising amount of code seems to\n> necessary for that negation (https://godbolt.org/z/MY7Wda3a7). Anyway,\n> it's just an idea for another series.\n\nNo JS here for godbolt, but there's also a bunch of \"!fspathcmp\"\nhere that could probably be changed to fspatheq.\n\n> > +\n> > +static inline int odb_path_hash(const char *str)\n> > +{\n> > +\treturn ignore_case ? strihash(str) : __ac_X31_hash_string(str);\n> > +}\n> \n> The internal Attractive Chaos (__ac_*) macros should be left confined\n> to khash.h, I think. Its alias kh_str_hash_func would be better\n> suited here.\n> \n> Do we want to use the K&R hash function here at all, though? If we\n> use FNV-1 when ignoring case, why not also use it (i.e. strhash) when\n> respecting it? At least that's done in builtin/sparse-checkout.c,\n> dir.c and merge-recursive.c. This is just handwaving and yammering\n> about lack of symmetry, but I do wonder how your performance numbers\n> look with strhash. If it's fine then we could package this up as\n> fspathhash..\n\nYeah, I think fspathhash should be path_hash in merge-recursive.c\n(and path_hash eliminated).\n\nI don't have performance numbers, and I doubt hash function\nperformance is much overhead, here. I used X31 since it was\nlocal to khash.\n\nI would prefer we only have one non-cryptographic hash\nimplementation to reduce cognitive overhead, so maybe we can\ndrop X31 entirely for FNV-1. I'd also prefer we only have khash\nor hashmap, not both.\n\n> And I also wonder how it looks if you use strihash unconditionally.\n> I guess case collisions are usually rare and branching based on a\n> global variable may be more expensive than case folding.\n\n*shrug* I'll let somebody with more appropriate systems do\nbenchmarks, there. But it could be an easy switch once\nfspathhash is in place.\n"},{"id":"429378","messageId":"20210706232159.GB8624@dcvr","threadId":"55998","inReplyTo":"ab757bce-3b51-afac-312c-ea2e883cf0bf@web.de","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-06T23:21:59Z","receivedAt":"2021-07-06T23:22:00Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"René Scharfe wrote:\n> Am 29.06.21 um 22:53 schrieb Eric Wong:\n> > --- a/alloc.c\n> > +++ b/alloc.c\n> > @@ -14,6 +14,7 @@\n> > #include \"tree.h\"\n> > #include \"commit.h\"\n> > #include \"tag.h\"\n> > +#include \"oidtree.h\"\n> > #include \"alloc.h\"\n> >\n> > #define BLOCKING 1024\n> > @@ -123,6 +124,11 @@ void *alloc_commit_node(struct repository *r)\n> > \treturn c;\n> > }\n> >\n> > +void *alloc_from_state(struct alloc_state *alloc_state, size_t n)\n> > +{\n> > +\treturn alloc_node(alloc_state, n);\n> > +}\n> > +\n> \n> Why extend alloc.c instead of using mem-pool.c? (I don't know which fits\n> better, but when you say \"memory pool\" and not use mem-pool.c I just have\n> to ask..)\n\nI didn't know mem-pool.c existed :x (And I've always known\nabout alloc.c).\n\nPerhaps we could merge them in another series to avoid further\nconfusion.\n\n> > +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n> > +{\n> > +\tstruct oidtree_node *on;\n> > +\n> > +\tif (!ot->mempool)\n> > +\t\tot->mempool = allocate_alloc_state();\n> > +\tif (!oid->algo)\n> > +\t\tBUG(\"oidtree_insert requires oid->algo\");\n> > +\n> > +\ton = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n> > +\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n> > +\n> > +\t/*\n> > +\t * n.b. we shouldn't get duplicates, here, but we'll have\n> > +\t * a small leak that won't be freed until oidtree_destroy\n> > +\t */\n> \n> Why shouldn't we get duplicates? That depends on the usage of oidtree,\n> right? The current user is fine because we avoid reading the same loose\n> object directory twice using the loose_objects_subdir_seen bitmap.\n\nYes, it reflects the current caller.\n\n> The leak comes from the allocation above, which is not used in case we\n> already have the key in the oidtree. So we need memory for all\n> candidates, not just the inserted candidates. That's probably\n> acceptable in most use cases.\n\nYes, I think the small, impossible-due-to-current-usage leak is\nan acceptable trade off.\n\n> We can do better by keeping track of the unnecessary allocation in\n> struct oidtree and recycling it at the next insert attempt, however.\n> That way we'd only waste at most one slot.\n\nIt'd involve maintaining a free list; which may be better\nsuited to being in alloc_state or mem_pool. That would also\nincrease the size of a struct *somewhere* and add a small\namount of code complexity, too.\n\n> > +\tcb_insert(&ot->t, &on->n, sizeof(*oid));\n> > +}\n> > +\n> > +int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n> > +{\n> > +\tstruct object_id k = { 0 };\n> > +\tsize_t klen = sizeof(k);\n> > +\toidcpy_with_padding(&k, oid);\n> \n> Why initialize k; isn't oidcpy_with_padding() supposed to overwrite it\n> completely?\n\nAh, I only added oidcpy_with_padding later into the development\nof this patch.\n\n> > +\n> > +\tif (oid->algo == GIT_HASH_UNKNOWN) {\n> > +\t\tk.algo = hash_algo_by_ptr(the_hash_algo);\n> > +\t\tklen -= sizeof(oid->algo);\n> > +\t}\n> \n> This relies on the order of the members hash and algo in struct\n> object_id to find a matching hash if we don't actually know algo. It\n> also relies on the absence of padding after algo. Would something like\n> this make sense?\n> \n> BUILD_ASSERT_OR_ZERO(offsetof(struct object_id, algo) + sizeof(k.algo) == sizeof(k));\n\nMaybe... I think a static assertion that object_id.hash be the\nfirst element of \"struct object_id\" is definitely needed, at\nleast.\n\n> And why set k.algo to some arbitrary value if we ignore it anyway? I.e.\n> why not keep it GIT_HASH_UNKNOWN, as set by oidcpy_with_padding()?\n\nGood point, shortening klen would've been all that was needed.\n\n> > +void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n> > +\t\t\tsize_t oidhexlen, oidtree_iter fn, void *arg)\n> > +{\n> > +\tsize_t klen = oidhexlen / 2;\n> > +\tstruct oidtree_iter_data x = { 0 };\n> > +\n> > +\tx.fn = fn;\n> > +\tx.arg = arg;\n> > +\tx.algo = oid->algo;\n> > +\tif (oidhexlen & 1) {\n> > +\t\tx.last_byte = oid->hash[klen];\n> > +\t\tx.last_nibble_at = &klen;\n> > +\t}\n> > +\tcb_each(&ot->t, (const uint8_t *)oid, klen, iter, &x);\n> > +}\n> \n> Clamp oidhexlen at GIT_MAX_HEXSZ? Or die?\n\nI think an assertion would be enough, here.\ninit_object_disambiguation already clamps to the_hash_algo->hexsz\n"},{"id":"429457","messageId":"20210707231019.14738-1-e@80x24.org","threadId":"55998","inReplyTo":"20210629205305.7100-1-e@80x24.org","subject":"[PATCH v3 0/5] optimizations for many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:10:14Z","receivedAt":"2021-07-07T23:10:22Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Implemented suggestions from Ævar and René, noticed a few more\nthings that's probably worth exploring at some point...\n\nTODO items unrelated to this series (probably for somebody else):\n\n* try fspatheq and fspathhash in more places in hopes it can\n reduce binary + icache size\n\n* favor ${#var} (instead of \"wc -c\" as as suggested by Ævar)\n to reduce fork+execve overhead in tests. We already use ${#var}\n in t/test-lib-functions.sh, and it works in shells I've tried\n (dash, posh, ksh93, bash --posix)\n\n* reduce internal redundancies (hash functions,\n hash table and memory pool implementations, etc.)\n\nEric Wong (5):\n speed up alt_odb_usable() with many alternates\n avoid strlen via strbuf_addstr in link_alt_odb_entry\n make object_directory.loose_objects_subdir_seen a bitmap\n oidcpy_with_padding: constify `src' arg\n oidtree: a crit-bit tree for odb_loose_cache\n\n Makefile | 3 +\n cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n cbtree.h | 56 ++++++++++++++\n dir.c | 10 +++\n dir.h | 2 +\n hash.h | 2 +-\n object-file.c | 75 ++++++++++--------\n object-name.c | 28 +++----\n object-store.h | 14 +++-\n object.c | 2 +\n oidtree.c | 104 +++++++++++++++++++++++++\n oidtree.h | 22 ++++++\n t/helper/test-oidtree.c | 49 ++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0069-oidtree.sh | 49 ++++++++++++\n 16 files changed, 534 insertions(+), 51 deletions(-)\n create mode 100644 cbtree.c\n create mode 100644 cbtree.h\n create mode 100644 oidtree.c\n create mode 100644 oidtree.h\n create mode 100644 t/helper/test-oidtree.c\n create mode 100755 t/t0069-oidtree.sh\n\nInterdiff against v2:\ndiff --git a/alloc.c b/alloc.c\nindex ca1e178c5a..957a0af362 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -14,7 +14,6 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n-#include \"oidtree.h\"\n #include \"alloc.h\"\n \n #define BLOCKING 1024\n@@ -124,11 +123,6 @@ void *alloc_commit_node(struct repository *r)\n \treturn c;\n }\n \n-void *alloc_from_state(struct alloc_state *alloc_state, size_t n)\n-{\n-\treturn alloc_node(alloc_state, n);\n-}\n-\n static void report(const char *name, unsigned int count, size_t size)\n {\n \tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\ndiff --git a/alloc.h b/alloc.h\nindex 4032375aa1..371d388b55 100644\n--- a/alloc.h\n+++ b/alloc.h\n@@ -13,7 +13,6 @@ void init_commit_node(struct commit *c);\n void *alloc_commit_node(struct repository *r);\n void *alloc_tag_node(struct repository *r);\n void *alloc_object_node(struct repository *r);\n-void *alloc_from_state(struct alloc_state *, size_t n);\n void alloc_report(struct repository *r);\n \n struct alloc_state *allocate_alloc_state(void);\ndiff --git a/dir.c b/dir.c\nindex ebe5ec046e..20b942d161 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -84,11 +84,21 @@ int fspathcmp(const char *a, const char *b)\n \treturn ignore_case ? strcasecmp(a, b) : strcmp(a, b);\n }\n \n+int fspatheq(const char *a, const char *b)\n+{\n+\treturn !fspathcmp(a, b);\n+}\n+\n int fspathncmp(const char *a, const char *b, size_t count)\n {\n \treturn ignore_case ? strncasecmp(a, b, count) : strncmp(a, b, count);\n }\n \n+unsigned int fspathhash(const char *str)\n+{\n+\treturn ignore_case ? strihash(str) : strhash(str);\n+}\n+\n int git_fnmatch(const struct pathspec_item *item,\n \t\tconst char *pattern, const char *string,\n \t\tint prefix)\ndiff --git a/dir.h b/dir.h\nindex e3db9b9ec6..2af7bcd7e5 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -489,7 +489,9 @@ int remove_dir_recursively(struct strbuf *path, int flag);\n int remove_path(const char *path);\n \n int fspathcmp(const char *a, const char *b);\n+int fspatheq(const char *a, const char *b);\n int fspathncmp(const char *a, const char *b, size_t count);\n+unsigned int fspathhash(const char *str);\n \n /*\n * The prefix part of pattern must not contains wildcards.\ndiff --git a/object-file.c b/object-file.c\nindex 6c397fb4f1..35f3e7e9bb 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -539,14 +539,12 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t\to->odb_by_path = kh_init_odb_path_map();\n \t\tassert(!o->odb->next);\n \t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n-\t\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n \t\tassert(r == 1); /* never used */\n \t\tkh_value(o->odb_by_path, p) = o->odb;\n \t}\n-\tif (!fspathcmp(path->buf, normalized_objdir))\n+\tif (fspatheq(path->buf, normalized_objdir))\n \t\treturn 0;\n \t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n-\tif (r < 0) die_errno(_(\"kh_put_odb_path_map\"));\n \t/* r: 0 = exists, 1 = never used, 2 = deleted */\n \treturn r == 0 ? 0 : 1;\n }\n@@ -2474,21 +2472,25 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n \n \tbitmap = &odb->loose_objects_subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn &odb->loose_objects_cache;\n-\n+\t\treturn odb->loose_objects_cache;\n+\tif (!odb->loose_objects_cache) {\n+\t\tALLOC_ARRAY(odb->loose_objects_cache, 1);\n+\t\toidtree_init(odb->loose_objects_cache);\n+\t}\n \tstrbuf_addstr(&buf, odb->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t &odb->loose_objects_cache);\n+\t\t\t\t odb->loose_objects_cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn &odb->loose_objects_cache;\n+\treturn odb->loose_objects_cache;\n }\n \n void odb_clear_loose_cache(struct object_directory *odb)\n {\n-\toidtree_destroy(&odb->loose_objects_cache);\n+\toidtree_clear(odb->loose_objects_cache);\n+\tFREE_AND_NULL(odb->loose_objects_cache);\n \tmemset(&odb->loose_objects_subdir_seen, 0,\n \t sizeof(odb->loose_objects_subdir_seen));\n }\ndiff --git a/object-store.h b/object-store.h\nindex b507108d18..e679acc4c3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -24,7 +24,7 @@ struct object_directory {\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n \tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oidtree loose_objects_cache;\n+\tstruct oidtree *loose_objects_cache;\n \n \t/*\n \t * Path to the alternative object store. If this is a relative path,\n@@ -33,18 +33,8 @@ struct object_directory {\n \tchar *path;\n };\n \n-static inline int odb_path_eq(const char *a, const char *b)\n-{\n-\treturn !fspathcmp(a, b);\n-}\n-\n-static inline int odb_path_hash(const char *str)\n-{\n-\treturn ignore_case ? strihash(str) : __ac_X31_hash_string(str);\n-}\n-\n KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n-\tstruct object_directory *, 1, odb_path_hash, odb_path_eq);\n+\tstruct object_directory *, 1, fspathhash, fspatheq);\n \n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\ndiff --git a/oidtree.c b/oidtree.c\nindex c1188d8f48..7eb0e9ba05 100644\n--- a/oidtree.c\n+++ b/oidtree.c\n@@ -19,46 +19,55 @@ struct oidtree_iter_data {\n \tuint8_t last_byte;\n };\n \n-void oidtree_destroy(struct oidtree *ot)\n+void oidtree_init(struct oidtree *ot)\n {\n-\tif (ot->mempool) {\n-\t\tclear_alloc_state(ot->mempool);\n-\t\tFREE_AND_NULL(ot->mempool);\n+\tcb_init(&ot->tree);\n+\tmem_pool_init(&ot->mem_pool, 0);\n+}\n+\n+void oidtree_clear(struct oidtree *ot)\n+{\n+\tif (ot) {\n+\t\tmem_pool_discard(&ot->mem_pool, 0);\n+\t\toidtree_init(ot);\n \t}\n-\toidtree_init(ot);\n }\n \n void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n {\n \tstruct oidtree_node *on;\n \n-\tif (!ot->mempool)\n-\t\tot->mempool = allocate_alloc_state();\n \tif (!oid->algo)\n \t\tBUG(\"oidtree_insert requires oid->algo\");\n \n-\ton = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n+\ton = mem_pool_alloc(&ot->mem_pool, sizeof(*on) + sizeof(*oid));\n \toidcpy_with_padding((struct object_id *)on->n.k, oid);\n \n \t/*\n-\t * n.b. we shouldn't get duplicates, here, but we'll have\n-\t * a small leak that won't be freed until oidtree_destroy\n+\t * n.b. Current callers won't get us duplicates, here. If a\n+\t * future caller causes duplicates, there'll be a a small leak\n+\t * that won't be freed until oidtree_clear. Currently it's not\n+\t * worth maintaining a free list\n \t */\n-\tcb_insert(&ot->t, &on->n, sizeof(*oid));\n+\tcb_insert(&ot->tree, &on->n, sizeof(*oid));\n }\n \n+\n int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n {\n-\tstruct object_id k = { 0 };\n+\tstruct object_id k;\n \tsize_t klen = sizeof(k);\n+\n \toidcpy_with_padding(&k, oid);\n \n-\tif (oid->algo == GIT_HASH_UNKNOWN) {\n-\t\tk.algo = hash_algo_by_ptr(the_hash_algo);\n+\tif (oid->algo == GIT_HASH_UNKNOWN)\n \t\tklen -= sizeof(oid->algo);\n-\t}\n \n-\treturn cb_lookup(&ot->t, (const uint8_t *)&k, klen) ? 1 : 0;\n+\t/* cb_lookup relies on memcmp on the struct, so order matters: */\n+\tklen += BUILD_ASSERT_OR_ZERO(offsetof(struct object_id, hash) <\n+\t\t\t\toffsetof(struct object_id, algo));\n+\n+\treturn cb_lookup(&ot->tree, (const uint8_t *)&k, klen) ? 1 : 0;\n }\n \n static enum cb_next iter(struct cb_node *n, void *arg)\n@@ -78,17 +87,18 @@ static enum cb_next iter(struct cb_node *n, void *arg)\n }\n \n void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n-\t\t\tsize_t oidhexlen, oidtree_iter fn, void *arg)\n+\t\t\tsize_t oidhexsz, oidtree_iter fn, void *arg)\n {\n-\tsize_t klen = oidhexlen / 2;\n+\tsize_t klen = oidhexsz / 2;\n \tstruct oidtree_iter_data x = { 0 };\n+\tassert(oidhexsz <= GIT_MAX_HEXSZ);\n \n \tx.fn = fn;\n \tx.arg = arg;\n \tx.algo = oid->algo;\n-\tif (oidhexlen & 1) {\n+\tif (oidhexsz & 1) {\n \t\tx.last_byte = oid->hash[klen];\n \t\tx.last_nibble_at = &klen;\n \t}\n-\tcb_each(&ot->t, (const uint8_t *)oid, klen, iter, &x);\n+\tcb_each(&ot->tree, (const uint8_t *)oid, klen, iter, &x);\n }\ndiff --git a/oidtree.h b/oidtree.h\nindex 73399bb978..77898f510a 100644\n--- a/oidtree.h\n+++ b/oidtree.h\n@@ -3,27 +3,20 @@\n \n #include \"cbtree.h\"\n #include \"hash.h\"\n+#include \"mem-pool.h\"\n \n-struct alloc_state;\n struct oidtree {\n-\tstruct cb_tree t;\n-\tstruct alloc_state *mempool;\n+\tstruct cb_tree tree;\n+\tstruct mem_pool mem_pool;\n };\n \n-#define OIDTREE_INIT { .t = CBTREE_INIT, .mempool = NULL }\n-\n-static inline void oidtree_init(struct oidtree *ot)\n-{\n-\tcb_init(&ot->t);\n-\tot->mempool = NULL;\n-}\n-\n-void oidtree_destroy(struct oidtree *);\n+void oidtree_init(struct oidtree *);\n+void oidtree_clear(struct oidtree *);\n void oidtree_insert(struct oidtree *, const struct object_id *);\n int oidtree_contains(struct oidtree *, const struct object_id *);\n \n-typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *arg);\n+typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *data);\n void oidtree_each(struct oidtree *, const struct object_id *,\n-\t\t\tsize_t oidhexlen, oidtree_iter, void *arg);\n+\t\t\tsize_t oidhexsz, oidtree_iter, void *data);\n \n #endif /* OIDTREE_H */\ndiff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\nindex e0da13eea3..180ee28dd9 100644\n--- a/t/helper/test-oidtree.c\n+++ b/t/helper/test-oidtree.c\n@@ -10,11 +10,12 @@ static enum cb_next print_oid(const struct object_id *oid, void *data)\n \n int cmd__oidtree(int argc, const char **argv)\n {\n-\tstruct oidtree ot = OIDTREE_INIT;\n+\tstruct oidtree ot;\n \tstruct strbuf line = STRBUF_INIT;\n \tint nongit_ok;\n \tint algo = GIT_HASH_UNKNOWN;\n \n+\toidtree_init(&ot);\n \tsetup_git_directory_gently(&nongit_ok);\n \n \twhile (strbuf_getline(&line, stdin) != EOF) {\n@@ -34,14 +35,15 @@ int cmd__oidtree(int argc, const char **argv)\n \t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n \t\t\tmemset(&oid, 0, sizeof(oid));\n \t\t\tmemcpy(buf, arg, strlen(arg));\n-\t\t\tbuf[hash_algos[algo].hexsz] = 0;\n+\t\t\tbuf[hash_algos[algo].hexsz] = '\\0';\n \t\t\tget_oid_hex_any(buf, &oid);\n \t\t\toid.algo = algo;\n \t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n-\t\t} else if (!strcmp(line.buf, \"destroy\"))\n-\t\t\toidtree_destroy(&ot);\n-\t\telse\n+\t\t} else if (!strcmp(line.buf, \"clear\")) {\n+\t\t\toidtree_clear(&ot);\n+\t\t} else {\n \t\t\tdie(\"unknown command: %s\", line.buf);\n+\t\t}\n \t}\n \treturn 0;\n }\ndiff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\nindex 0594f57c81..bfb1397d7b 100755\n--- a/t/t0069-oidtree.sh\n+++ b/t/t0069-oidtree.sh\n@@ -3,35 +3,32 @@\n test_description='basic tests for the oidtree implementation'\n . ./test-lib.sh\n \n+maxhexsz=$(test_oid hexsz)\n echoid () {\n \tprefix=\"${1:+$1 }\"\n \tshift\n \twhile test $# -gt 0\n \tdo\n-\t\techo \"$1\"\n+\t\tshortoid=\"$1\"\n \t\tshift\n-\tdone | awk -v prefix=\"$prefix\" -v ZERO_OID=$ZERO_OID '{\n-\t\tprintf(\"%s%s\", prefix, $0);\n-\t\tneed = length(ZERO_OID) - length($0);\n-\t\tfor (i = 0; i < need; i++)\n-\t\t\tprintf(\"0\");\n-\t\tprintf \"\\n\";\n-\t}'\n+\t\tdifference=$(($maxhexsz - ${#shortoid}))\n+\t\tprintf \"%s%s%0${difference}d\\\\n\" \"$prefix\" \"$shortoid\" \"0\"\n+\tdone\n }\n \n test_expect_success 'oidtree insert and contains' '\n-\tcat >expect <expect <<-\\EOF &&\n+\t\t0\n+\t\t0\n+\t\t0\n+\t\t1\n+\t\t1\n+\t\t0\n+\tEOF\n \t{\n \t\techoid insert 444 1 2 3 4 5 a b c d e &&\n \t\techoid contains 44 441 440 444 4440 4444\n-\t\techo destroy\n+\t\techo clear\n \t} | test-tool oidtree >actual &&\n \ttest_cmp expect actual\n '\n@@ -44,7 +41,7 @@ test_expect_success 'oidtree each' '\n \t\techo each 3211\n \t\techo each 3210\n \t\techo each 32100\n-\t\techo destroy\n+\t\techo clear\n \t} | test-tool oidtree >actual &&\n \ttest_cmp expect actual\n '\n"},{"id":"429458","messageId":"20210707231019.14738-2-e@80x24.org","threadId":"55998","inReplyTo":"20210629205305.7100-1-e@80x24.org","subject":"[PATCH v3 1/5] speed up alt_odb_usable() with many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:10:15Z","receivedAt":"2021-07-07T23:10:39Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"With many alternates, the duplicate check in alt_odb_usable()\nwastes many cycles doing repeated fspathcmp() on every existing\nalternate. Use a khash to speed up lookups by odb->path.\n\nSince the kh_put_* API uses the supplied key without\nduplicating it, we also take advantage of it to replace both\nxstrdup() and strbuf_release() in link_alt_odb_entry() with\nstrbuf_detach() to avoid the allocation and copy.\n\nIn a test repository with 50K alternates and each of those 50K\nalternates having one alternate each (for a total of 100K total\nalternates); this speeds up lookup of a non-existent blob from\nover 16 minutes to roughly 2.7 seconds on my busy workstation.\n\nNote: all underlying git object directories were small and\nunpacked with only loose objects and no packs. Having to load\npacks increases times significantly.\n\nv3: Introduce and use fspatheq and fspathhash functions;\n avoid unnecessary checks for allocation failures already\n handled by our own *alloc wrappers.\n\nSigned-off-by: Eric Wong \n---\n dir.c | 10 ++++++++++\n dir.h | 2 ++\n object-file.c | 33 +++++++++++++++++++++------------\n object-store.h | 7 +++++++\n object.c | 2 ++\n 5 files changed, 42 insertions(+), 12 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex ebe5ec046e..20b942d161 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -84,11 +84,21 @@ int fspathcmp(const char *a, const char *b)\n \treturn ignore_case ? strcasecmp(a, b) : strcmp(a, b);\n }\n \n+int fspatheq(const char *a, const char *b)\n+{\n+\treturn !fspathcmp(a, b);\n+}\n+\n int fspathncmp(const char *a, const char *b, size_t count)\n {\n \treturn ignore_case ? strncasecmp(a, b, count) : strncmp(a, b, count);\n }\n \n+unsigned int fspathhash(const char *str)\n+{\n+\treturn ignore_case ? strihash(str) : strhash(str);\n+}\n+\n int git_fnmatch(const struct pathspec_item *item,\n \t\tconst char *pattern, const char *string,\n \t\tint prefix)\ndiff --git a/dir.h b/dir.h\nindex e3db9b9ec6..2af7bcd7e5 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -489,7 +489,9 @@ int remove_dir_recursively(struct strbuf *path, int flag);\n int remove_path(const char *path);\n \n int fspathcmp(const char *a, const char *b);\n+int fspatheq(const char *a, const char *b);\n int fspathncmp(const char *a, const char *b, size_t count);\n+unsigned int fspathhash(const char *str);\n \n /*\n * The prefix part of pattern must not contains wildcards.\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..a13f49b192 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -517,9 +517,9 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf,\n */\n static int alt_odb_usable(struct raw_object_store *o,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *normalized_objdir)\n+\t\t\t const char *normalized_objdir, khiter_t *pos)\n {\n-\tstruct object_directory *odb;\n+\tint r;\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path->buf)) {\n@@ -533,14 +533,20 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n \t */\n-\tfor (odb = o->odb; odb; odb = odb->next) {\n-\t\tif (!fspathcmp(path->buf, odb->path))\n-\t\t\treturn 0;\n+\tif (!o->odb_by_path) {\n+\t\tkhiter_t p;\n+\n+\t\to->odb_by_path = kh_init_odb_path_map();\n+\t\tassert(!o->odb->next);\n+\t\tp = kh_put_odb_path_map(o->odb_by_path, o->odb->path, &r);\n+\t\tassert(r == 1); /* never used */\n+\t\tkh_value(o->odb_by_path, p) = o->odb;\n \t}\n-\tif (!fspathcmp(path->buf, normalized_objdir))\n+\tif (fspatheq(path->buf, normalized_objdir))\n \t\treturn 0;\n-\n-\treturn 1;\n+\t*pos = kh_put_odb_path_map(o->odb_by_path, path->buf, &r);\n+\t/* r: 0 = exists, 1 = never used, 2 = deleted */\n+\treturn r == 0 ? 0 : 1;\n }\n \n /*\n@@ -566,6 +572,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n+\tkhiter_t pos;\n \n \tif (!is_absolute_path(entry) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n@@ -587,23 +594,25 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \twhile (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')\n \t\tstrbuf_setlen(&pathbuf, pathbuf.len - 1);\n \n-\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {\n+\tif (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir, &pos)) {\n \t\tstrbuf_release(&pathbuf);\n \t\treturn -1;\n \t}\n \n \tCALLOC_ARRAY(ent, 1);\n-\tent->path = xstrdup(pathbuf.buf);\n+\t/* pathbuf.buf is already in r->objects->odb_by_path */\n+\tent->path = strbuf_detach(&pathbuf, NULL);\n \n \t/* add the alternate entry */\n \t*r->objects->odb_tail = ent;\n \tr->objects->odb_tail = &(ent->next);\n \tent->next = NULL;\n+\tassert(r->objects->odb_by_path);\n+\tkh_value(r->objects->odb_by_path, pos) = ent;\n \n \t/* recursively add alternates */\n-\tread_info_alternates(r, pathbuf.buf, depth + 1);\n+\tread_info_alternates(r, ent->path, depth + 1);\n \n-\tstrbuf_release(&pathbuf);\n \treturn 0;\n }\n \ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb..6077065d90 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -7,6 +7,8 @@\n #include \"oid-array.h\"\n #include \"strbuf.h\"\n #include \"thread-utils.h\"\n+#include \"khash.h\"\n+#include \"dir.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -30,6 +32,9 @@ struct object_directory {\n \tchar *path;\n };\n \n+KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n+\tstruct object_directory *, 1, fspathhash, fspatheq);\n+\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct object_directory *, void *);\n@@ -116,6 +121,8 @@ struct raw_object_store {\n \t */\n \tstruct object_directory *odb;\n \tstruct object_directory **odb_tail;\n+\tkh_odb_path_map_t *odb_by_path;\n+\n \tint loaded_alternates;\n \n \t/*\ndiff --git a/object.c b/object.c\nindex 14188453c5..2b3c075a15 100644\n--- a/object.c\n+++ b/object.c\n@@ -511,6 +511,8 @@ static void free_object_directories(struct raw_object_store *o)\n \t\tfree_object_directory(o->odb);\n \t\to->odb = next;\n \t}\n+\tkh_destroy_odb_path_map(o->odb_by_path);\n+\to->odb_by_path = NULL;\n }\n \n void raw_object_store_clear(struct raw_object_store *o)\n"},{"id":"429459","messageId":"20210707231019.14738-3-e@80x24.org","threadId":"55998","inReplyTo":"20210629205305.7100-1-e@80x24.org","subject":"[PATCH v3 2/5] avoid strlen via strbuf_addstr in link_alt_odb_entry","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:10:16Z","receivedAt":"2021-07-07T23:10:50Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"We can save a few milliseconds (across 100K odbs) by using\nstrbuf_addbuf() instead of strbuf_addstr() by passing `entry' as\na strbuf pointer rather than a \"const char *\".\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex a13f49b192..2dd70ddf3a 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -567,18 +567,18 @@ static int alt_odb_usable(struct raw_object_store *o,\n static void read_info_alternates(struct repository *r,\n \t\t\t\t const char *relative_base,\n \t\t\t\t int depth);\n-static int link_alt_odb_entry(struct repository *r, const char *entry,\n+static int link_alt_odb_entry(struct repository *r, const struct strbuf *entry,\n \tconst char *relative_base, int depth, const char *normalized_objdir)\n {\n \tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \tkhiter_t pos;\n \n-\tif (!is_absolute_path(entry) && relative_base) {\n+\tif (!is_absolute_path(entry->buf) && relative_base) {\n \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n \t\tstrbuf_addch(&pathbuf, '/');\n \t}\n-\tstrbuf_addstr(&pathbuf, entry);\n+\tstrbuf_addbuf(&pathbuf, entry);\n \n \tif (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {\n \t\terror(_(\"unable to normalize alternate object path: %s\"),\n@@ -669,7 +669,7 @@ static void link_alt_odb_entries(struct repository *r, const char *alt,\n \t\talt = parse_alt_odb_entry(alt, sep, &entry);\n \t\tif (!entry.len)\n \t\t\tcontinue;\n-\t\tlink_alt_odb_entry(r, entry.buf,\n+\t\tlink_alt_odb_entry(r, &entry,\n \t\t\t\t relative_base, depth, objdirbuf.buf);\n \t}\n \tstrbuf_release(&entry);\n"},{"id":"429460","messageId":"20210707231019.14738-4-e@80x24.org","threadId":"55998","inReplyTo":"20210629205305.7100-1-e@80x24.org","subject":"[PATCH v3 3/5] make object_directory.loose_objects_subdir_seen a bitmap","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:10:17Z","receivedAt":"2021-07-07T23:10:51Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"There's no point in using 8 bits per-directory when 1 bit\nwill do. This saves us 224 bytes per object directory, which\nends up being 22MB when dealing with 100K alternates.\n\nv2: use bitsizeof() macro and better variable names\n\nSigned-off-by: Eric Wong \n---\n object-file.c | 11 ++++++++---\n object-store.h | 2 +-\n 2 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 2dd70ddf3a..91ded8c22a 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2461,12 +2461,17 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n {\n \tint subdir_nr = oid->hash[0];\n \tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n+\tsize_t word_index = subdir_nr / word_bits;\n+\tsize_t mask = 1 << (subdir_nr % word_bits);\n+\tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n+\t subdir_nr >= bitsizeof(odb->loose_objects_subdir_seen))\n \t\tBUG(\"subdir_nr out of range\");\n \n-\tif (odb->loose_objects_subdir_seen[subdir_nr])\n+\tbitmap = &odb->loose_objects_subdir_seen[word_index];\n+\tif (*bitmap & mask)\n \t\treturn &odb->loose_objects_cache[subdir_nr];\n \n \tstrbuf_addstr(&buf, odb->path);\n@@ -2474,7 +2479,7 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n \t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n-\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n+\t*bitmap |= mask;\n \tstrbuf_release(&buf);\n \treturn &odb->loose_objects_cache[subdir_nr];\n }\ndiff --git a/object-store.h b/object-store.h\nindex 6077065d90..ab6d469970 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -22,7 +22,7 @@ struct object_directory {\n \t *\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n-\tchar loose_objects_subdir_seen[256];\n+\tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n \tstruct oid_array loose_objects_cache[256];\n \n \t/*\n"},{"id":"429461","messageId":"20210707231019.14738-5-e@80x24.org","threadId":"55998","inReplyTo":"20210629205305.7100-1-e@80x24.org","subject":"[PATCH v3 4/5] oidcpy_with_padding: constify `src' arg","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:10:18Z","receivedAt":"2021-07-07T23:10:52Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"As with `oidcpy', the source struct will not be modified and\nthis will allow an upcoming const-correct caller to use it.\n\nSigned-off-by: Eric Wong \n---\n hash.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 9c6df4d952..27a180248f 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -265,7 +265,7 @@ static inline void oidcpy(struct object_id *dst, const struct object_id *src)\n \n /* Like oidcpy() but zero-pads the unused bytes in dst's hash array. */\n static inline void oidcpy_with_padding(struct object_id *dst,\n-\t\t\t\t struct object_id *src)\n+\t\t\t\t const struct object_id *src)\n {\n \tsize_t hashsz;\n \n"},{"id":"429462","messageId":"20210707231019.14738-6-e@80x24.org","threadId":"55998","inReplyTo":"20210629205305.7100-1-e@80x24.org","subject":"[PATCH v3 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:10:19Z","receivedAt":"2021-07-07T23:10:56Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This saves 8K per `struct object_directory', meaning it saves\naround 800MB in my case involving 100K alternates (half or more\nof those alternates are unlikely to hold loose objects).\n\nThis is implemented in two parts: a generic, allocation-free\n`cbtree' and the `oidtree' wrapper on top of it. The latter\nprovides allocation using alloc_state as a memory pool to\nimprove locality and reduce free(3) overhead.\n\nUnlike oid-array, the crit-bit tree does not require sorting.\nPerformance is bound by the key length, for oidtree that is\nfixed at sizeof(struct object_id). There's no need to have\n256 oidtrees to mitigate the O(n log n) overhead like we did\nwith oid-array.\n\nBeing a prefix trie, it is natively suited for expanding short\nobject IDs via prefix-limited iteration in\n`find_short_object_filename'.\n\nOn my busy workstation, p4205 performance seems to be roughly\nunchanged (+/-8%). Startup with 100K total alternates with no\nloose objects seems around 10-20% faster on a hot cache.\n(800MB in memory savings means more memory for the kernel FS\ncache).\n\nThe generic cbtree implementation does impose some extra\noverhead for oidtree in that it uses memcmp(3) on\n\"struct object_id\" so it wastes cycles comparing 12 extra bytes\non SHA-1 repositories. I've not yet explored reducing this\noverhead, but I expect there are many places in our code base\nwhere we'd want to investigate this.\n\nMore information on crit-bit trees: https://cr.yp.to/critbit.html\n\nv2: make oidtree test hash-agnostic\n\nv3: Implement suggestions by René and Ævar\n use mem_pool instead of alloc_state\n s/oidtree.t/oidtree.tree/\n lazy-allocate entire loose_objects_state struct\n remove no-longer-used OIDTREE_INIT macro, uninline oidtree_init\n s/oidtree_destroy/oidtree_clear/\n simplify and add extra assertions\n s/hexlen/hexsz/\n minor style and naming fixes\n\nSigned-off-by: Eric Wong \n---\n Makefile | 3 +\n cbtree.c | 167 ++++++++++++++++++++++++++++++++++++++++\n cbtree.h | 56 ++++++++++++++\n object-file.c | 23 +++---\n object-name.c | 28 +++----\n object-store.h | 5 +-\n oidtree.c | 104 +++++++++++++++++++++++++\n oidtree.h | 22 ++++++\n t/helper/test-oidtree.c | 49 ++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0069-oidtree.sh | 49 ++++++++++++\n 12 files changed, 478 insertions(+), 30 deletions(-)\n create mode 100644 cbtree.c\n create mode 100644 cbtree.h\n create mode 100644 oidtree.c\n create mode 100644 oidtree.h\n create mode 100644 t/helper/test-oidtree.c\n create mode 100755 t/t0069-oidtree.sh\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..a1525978fb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -722,6 +722,7 @@ TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n TEST_BUILTINS_OBJS += test-oid-array.o\n TEST_BUILTINS_OBJS += test-oidmap.o\n+TEST_BUILTINS_OBJS += test-oidtree.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n@@ -845,6 +846,7 @@ LIB_OBJS += branch.o\n LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n+LIB_OBJS += cbtree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n LIB_OBJS += chunk-format.o\n@@ -940,6 +942,7 @@ LIB_OBJS += object.o\n LIB_OBJS += oid-array.o\n LIB_OBJS += oidmap.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += oidtree.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-check.o\ndiff --git a/cbtree.c b/cbtree.c\nnew file mode 100644\nindex 0000000000..b0c65d810f\n--- /dev/null\n+++ b/cbtree.c\n@@ -0,0 +1,167 @@\n+/*\n+ * crit-bit tree implementation, does no allocations internally\n+ * For more information on crit-bit trees: https://cr.yp.to/critbit.html\n+ * Based on Adam Langley's adaptation of Dan Bernstein's public domain code\n+ * git clone https://github.com/agl/critbit.git\n+ */\n+#include \"cbtree.h\"\n+\n+static struct cb_node *cb_node_of(const void *p)\n+{\n+\treturn (struct cb_node *)((uintptr_t)p - 1);\n+}\n+\n+/* locate the best match, does not do a final comparision */\n+static struct cb_node *cb_internal_best_match(struct cb_node *p,\n+\t\t\t\t\tconst uint8_t *k, size_t klen)\n+{\n+\twhile (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tuint8_t c = q->byte < klen ? k[q->byte] : 0;\n+\t\tsize_t direction = (1 + (q->otherbits | c)) >> 8;\n+\n+\t\tp = q->child[direction];\n+\t}\n+\treturn p;\n+}\n+\n+/* returns NULL if successful, existing cb_node if duplicate */\n+struct cb_node *cb_insert(struct cb_tree *t, struct cb_node *node, size_t klen)\n+{\n+\tsize_t newbyte, newotherbits;\n+\tuint8_t c;\n+\tint newdirection;\n+\tstruct cb_node **wherep, *p;\n+\n+\tassert(!((uintptr_t)node & 1)); /* allocations must be aligned */\n+\n+\tif (!t->root) {\t\t/* insert into empty tree */\n+\t\tt->root = node;\n+\t\treturn NULL;\t/* success */\n+\t}\n+\n+\t/* see if a node already exists */\n+\tp = cb_internal_best_match(t->root, node->k, klen);\n+\n+\t/* find first differing byte */\n+\tfor (newbyte = 0; newbyte < klen; newbyte++) {\n+\t\tif (p->k[newbyte] != node->k[newbyte])\n+\t\t\tgoto different_byte_found;\n+\t}\n+\treturn p;\t/* element exists, let user deal with it */\n+\n+different_byte_found:\n+\tnewotherbits = p->k[newbyte] ^ node->k[newbyte];\n+\tnewotherbits |= newotherbits >> 1;\n+\tnewotherbits |= newotherbits >> 2;\n+\tnewotherbits |= newotherbits >> 4;\n+\tnewotherbits = (newotherbits & ~(newotherbits >> 1)) ^ 255;\n+\tc = p->k[newbyte];\n+\tnewdirection = (1 + (newotherbits | c)) >> 8;\n+\n+\tnode->byte = newbyte;\n+\tnode->otherbits = newotherbits;\n+\tnode->child[1 - newdirection] = node;\n+\n+\t/* find a place to insert it */\n+\twherep = &t->root;\n+\tfor (;;) {\n+\t\tstruct cb_node *q;\n+\t\tsize_t direction;\n+\n+\t\tp = *wherep;\n+\t\tif (!(1 & (uintptr_t)p))\n+\t\t\tbreak;\n+\t\tq = cb_node_of(p);\n+\t\tif (q->byte > newbyte)\n+\t\t\tbreak;\n+\t\tif (q->byte == newbyte && q->otherbits > newotherbits)\n+\t\t\tbreak;\n+\t\tc = q->byte < klen ? node->k[q->byte] : 0;\n+\t\tdirection = (1 + (q->otherbits | c)) >> 8;\n+\t\twherep = q->child + direction;\n+\t}\n+\n+\tnode->child[newdirection] = *wherep;\n+\t*wherep = (struct cb_node *)(1 + (uintptr_t)node);\n+\n+\treturn NULL; /* success */\n+}\n+\n+struct cb_node *cb_lookup(struct cb_tree *t, const uint8_t *k, size_t klen)\n+{\n+\tstruct cb_node *p = cb_internal_best_match(t->root, k, klen);\n+\n+\treturn p && !memcmp(p->k, k, klen) ? p : NULL;\n+}\n+\n+struct cb_node *cb_unlink(struct cb_tree *t, const uint8_t *k, size_t klen)\n+{\n+\tstruct cb_node **wherep = &t->root;\n+\tstruct cb_node **whereq = NULL;\n+\tstruct cb_node *q = NULL;\n+\tsize_t direction = 0;\n+\tuint8_t c;\n+\tstruct cb_node *p = t->root;\n+\n+\tif (!p) return NULL;\t/* empty tree, nothing to delete */\n+\n+\t/* traverse to find best match, keeping link to parent */\n+\twhile (1 & (uintptr_t)p) {\n+\t\twhereq = wherep;\n+\t\tq = cb_node_of(p);\n+\t\tc = q->byte < klen ? k[q->byte] : 0;\n+\t\tdirection = (1 + (q->otherbits | c)) >> 8;\n+\t\twherep = q->child + direction;\n+\t\tp = *wherep;\n+\t}\n+\n+\tif (memcmp(p->k, k, klen))\n+\t\treturn NULL;\t\t/* no match, nothing unlinked */\n+\n+\t/* found an exact match */\n+\tif (whereq)\t/* update parent */\n+\t\t*whereq = q->child[1 - direction];\n+\telse\n+\t\tt->root = NULL;\n+\treturn p;\n+}\n+\n+static enum cb_next cb_descend(struct cb_node *p, cb_iter fn, void *arg)\n+{\n+\tif (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tenum cb_next n = cb_descend(q->child[0], fn, arg);\n+\n+\t\treturn n == CB_BREAK ? n : cb_descend(q->child[1], fn, arg);\n+\t} else {\n+\t\treturn fn(p, arg);\n+\t}\n+}\n+\n+void cb_each(struct cb_tree *t, const uint8_t *kpfx, size_t klen,\n+\t\t\tcb_iter fn, void *arg)\n+{\n+\tstruct cb_node *p = t->root;\n+\tstruct cb_node *top = p;\n+\tsize_t i = 0;\n+\n+\tif (!p) return; /* empty tree */\n+\n+\t/* Walk tree, maintaining top pointer */\n+\twhile (1 & (uintptr_t)p) {\n+\t\tstruct cb_node *q = cb_node_of(p);\n+\t\tuint8_t c = q->byte < klen ? kpfx[q->byte] : 0;\n+\t\tsize_t direction = (1 + (q->otherbits | c)) >> 8;\n+\n+\t\tp = q->child[direction];\n+\t\tif (q->byte < klen)\n+\t\t\ttop = p;\n+\t}\n+\n+\tfor (i = 0; i < klen; i++) {\n+\t\tif (p->k[i] != kpfx[i])\n+\t\t\treturn; /* \"best\" match failed */\n+\t}\n+\tcb_descend(top, fn, arg);\n+}\ndiff --git a/cbtree.h b/cbtree.h\nnew file mode 100644\nindex 0000000000..fe4587087e\n--- /dev/null\n+++ b/cbtree.h\n@@ -0,0 +1,56 @@\n+/*\n+ * crit-bit tree implementation, does no allocations internally\n+ * For more information on crit-bit trees: https://cr.yp.to/critbit.html\n+ * Based on Adam Langley's adaptation of Dan Bernstein's public domain code\n+ * git clone https://github.com/agl/critbit.git\n+ *\n+ * This is adapted to store arbitrary data (not just NUL-terminated C strings\n+ * and allocates no memory internally. The user needs to allocate\n+ * \"struct cb_node\" and fill cb_node.k[] with arbitrary match data\n+ * for memcmp.\n+ * If \"klen\" is variable, then it should be embedded into \"c_node.k[]\"\n+ * Recursion is bound by the maximum value of \"klen\" used.\n+ */\n+#ifndef CBTREE_H\n+#define CBTREE_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct cb_node;\n+struct cb_node {\n+\tstruct cb_node *child[2];\n+\t/*\n+\t * n.b. uint32_t for `byte' is excessive for OIDs,\n+\t * we may consider shorter variants if nothing else gets stored.\n+\t */\n+\tuint32_t byte;\n+\tuint8_t otherbits;\n+\tuint8_t k[FLEX_ARRAY]; /* arbitrary data */\n+};\n+\n+struct cb_tree {\n+\tstruct cb_node *root;\n+};\n+\n+enum cb_next {\n+\tCB_CONTINUE = 0,\n+\tCB_BREAK = 1\n+};\n+\n+#define CBTREE_INIT { .root = NULL }\n+\n+static inline void cb_init(struct cb_tree *t)\n+{\n+\tt->root = NULL;\n+}\n+\n+struct cb_node *cb_lookup(struct cb_tree *, const uint8_t *k, size_t klen);\n+struct cb_node *cb_insert(struct cb_tree *, struct cb_node *, size_t klen);\n+struct cb_node *cb_unlink(struct cb_tree *t, const uint8_t *k, size_t klen);\n+\n+typedef enum cb_next (*cb_iter)(struct cb_node *, void *arg);\n+\n+void cb_each(struct cb_tree *, const uint8_t *kpfx, size_t klen,\n+\t\tcb_iter, void *arg);\n+\n+#endif /* CBTREE_H */\ndiff --git a/object-file.c b/object-file.c\nindex 91ded8c22a..35f3e7e9bb 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1173,7 +1173,7 @@ static int quick_has_loose(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\tif (oid_array_lookup(odb_loose_cache(odb, oid), oid) >= 0)\n+\t\tif (oidtree_contains(odb_loose_cache(odb, oid), oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -2452,11 +2452,11 @@ int for_each_loose_object(each_loose_object_fn cb, void *data,\n static int append_loose_object(const struct object_id *oid, const char *path,\n \t\t\t void *data)\n {\n-\toid_array_append(data, oid);\n+\toidtree_insert(data, oid);\n \treturn 0;\n }\n \n-struct oid_array *odb_loose_cache(struct object_directory *odb,\n+struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t const struct object_id *oid)\n {\n \tint subdir_nr = oid->hash[0];\n@@ -2472,24 +2472,25 @@ struct oid_array *odb_loose_cache(struct object_directory *odb,\n \n \tbitmap = &odb->loose_objects_subdir_seen[word_index];\n \tif (*bitmap & mask)\n-\t\treturn &odb->loose_objects_cache[subdir_nr];\n-\n+\t\treturn odb->loose_objects_cache;\n+\tif (!odb->loose_objects_cache) {\n+\t\tALLOC_ARRAY(odb->loose_objects_cache, 1);\n+\t\toidtree_init(odb->loose_objects_cache);\n+\t}\n \tstrbuf_addstr(&buf, odb->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n+\t\t\t\t odb->loose_objects_cache);\n \t*bitmap |= mask;\n \tstrbuf_release(&buf);\n-\treturn &odb->loose_objects_cache[subdir_nr];\n+\treturn odb->loose_objects_cache;\n }\n \n void odb_clear_loose_cache(struct object_directory *odb)\n {\n-\tint i;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(odb->loose_objects_cache); i++)\n-\t\toid_array_clear(&odb->loose_objects_cache[i]);\n+\toidtree_clear(odb->loose_objects_cache);\n+\tFREE_AND_NULL(odb->loose_objects_cache);\n \tmemset(&odb->loose_objects_subdir_seen, 0,\n \t sizeof(odb->loose_objects_subdir_seen));\n }\ndiff --git a/object-name.c b/object-name.c\nindex 64202de60b..3263c19457 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -87,27 +87,21 @@ static void update_candidates(struct disambiguate_state *ds, const struct object\n \n static int match_hash(unsigned, const unsigned char *, const unsigned char *);\n \n+static enum cb_next match_prefix(const struct object_id *oid, void *arg)\n+{\n+\tstruct disambiguate_state *ds = arg;\n+\t/* no need to call match_hash, oidtree_each did prefix match */\n+\tupdate_candidates(ds, oid);\n+\treturn ds->ambiguous ? CB_BREAK : CB_CONTINUE;\n+}\n+\n static void find_short_object_filename(struct disambiguate_state *ds)\n {\n \tstruct object_directory *odb;\n \n-\tfor (odb = ds->repo->objects->odb; odb && !ds->ambiguous; odb = odb->next) {\n-\t\tint pos;\n-\t\tstruct oid_array *loose_objects;\n-\n-\t\tloose_objects = odb_loose_cache(odb, &ds->bin_pfx);\n-\t\tpos = oid_array_lookup(loose_objects, &ds->bin_pfx);\n-\t\tif (pos < 0)\n-\t\t\tpos = -1 - pos;\n-\t\twhile (!ds->ambiguous && pos < loose_objects->nr) {\n-\t\t\tconst struct object_id *oid;\n-\t\t\toid = loose_objects->oid + pos;\n-\t\t\tif (!match_hash(ds->len, ds->bin_pfx.hash, oid->hash))\n-\t\t\t\tbreak;\n-\t\t\tupdate_candidates(ds, oid);\n-\t\t\tpos++;\n-\t\t}\n-\t}\n+\tfor (odb = ds->repo->objects->odb; odb && !ds->ambiguous; odb = odb->next)\n+\t\toidtree_each(odb_loose_cache(odb, &ds->bin_pfx),\n+\t\t\t\t&ds->bin_pfx, ds->len, match_prefix, ds);\n }\n \n static int match_hash(unsigned len, const unsigned char *a, const unsigned char *b)\ndiff --git a/object-store.h b/object-store.h\nindex ab6d469970..e679acc4c3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -9,6 +9,7 @@\n #include \"thread-utils.h\"\n #include \"khash.h\"\n #include \"dir.h\"\n+#include \"oidtree.h\"\n \n struct object_directory {\n \tstruct object_directory *next;\n@@ -23,7 +24,7 @@ struct object_directory {\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n \tuint32_t loose_objects_subdir_seen[8]; /* 256 bits */\n-\tstruct oid_array loose_objects_cache[256];\n+\tstruct oidtree *loose_objects_cache;\n \n \t/*\n \t * Path to the alternative object store. If this is a relative path,\n@@ -59,7 +60,7 @@ void add_to_alternates_memory(const char *dir);\n * Populate and return the loose object cache array corresponding to the\n * given object ID.\n */\n-struct oid_array *odb_loose_cache(struct object_directory *odb,\n+struct oidtree *odb_loose_cache(struct object_directory *odb,\n \t\t\t\t const struct object_id *oid);\n \n /* Empty the loose object cache for the specified object directory. */\ndiff --git a/oidtree.c b/oidtree.c\nnew file mode 100644\nindex 0000000000..7eb0e9ba05\n--- /dev/null\n+++ b/oidtree.c\n@@ -0,0 +1,104 @@\n+/*\n+ * A wrapper around cbtree which stores oids\n+ * May be used to replace oid-array for prefix (abbreviation) matches\n+ */\n+#include \"oidtree.h\"\n+#include \"alloc.h\"\n+#include \"hash.h\"\n+\n+struct oidtree_node {\n+\t/* n.k[] is used to store \"struct object_id\" */\n+\tstruct cb_node n;\n+};\n+\n+struct oidtree_iter_data {\n+\toidtree_iter fn;\n+\tvoid *arg;\n+\tsize_t *last_nibble_at;\n+\tint algo;\n+\tuint8_t last_byte;\n+};\n+\n+void oidtree_init(struct oidtree *ot)\n+{\n+\tcb_init(&ot->tree);\n+\tmem_pool_init(&ot->mem_pool, 0);\n+}\n+\n+void oidtree_clear(struct oidtree *ot)\n+{\n+\tif (ot) {\n+\t\tmem_pool_discard(&ot->mem_pool, 0);\n+\t\toidtree_init(ot);\n+\t}\n+}\n+\n+void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n+{\n+\tstruct oidtree_node *on;\n+\n+\tif (!oid->algo)\n+\t\tBUG(\"oidtree_insert requires oid->algo\");\n+\n+\ton = mem_pool_alloc(&ot->mem_pool, sizeof(*on) + sizeof(*oid));\n+\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n+\n+\t/*\n+\t * n.b. Current callers won't get us duplicates, here. If a\n+\t * future caller causes duplicates, there'll be a a small leak\n+\t * that won't be freed until oidtree_clear. Currently it's not\n+\t * worth maintaining a free list\n+\t */\n+\tcb_insert(&ot->tree, &on->n, sizeof(*oid));\n+}\n+\n+\n+int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n+{\n+\tstruct object_id k;\n+\tsize_t klen = sizeof(k);\n+\n+\toidcpy_with_padding(&k, oid);\n+\n+\tif (oid->algo == GIT_HASH_UNKNOWN)\n+\t\tklen -= sizeof(oid->algo);\n+\n+\t/* cb_lookup relies on memcmp on the struct, so order matters: */\n+\tklen += BUILD_ASSERT_OR_ZERO(offsetof(struct object_id, hash) <\n+\t\t\t\toffsetof(struct object_id, algo));\n+\n+\treturn cb_lookup(&ot->tree, (const uint8_t *)&k, klen) ? 1 : 0;\n+}\n+\n+static enum cb_next iter(struct cb_node *n, void *arg)\n+{\n+\tstruct oidtree_iter_data *x = arg;\n+\tconst struct object_id *oid = (const struct object_id *)n->k;\n+\n+\tif (x->algo != GIT_HASH_UNKNOWN && x->algo != oid->algo)\n+\t\treturn CB_CONTINUE;\n+\n+\tif (x->last_nibble_at) {\n+\t\tif ((oid->hash[*x->last_nibble_at] ^ x->last_byte) & 0xf0)\n+\t\t\treturn CB_CONTINUE;\n+\t}\n+\n+\treturn x->fn(oid, x->arg);\n+}\n+\n+void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n+\t\t\tsize_t oidhexsz, oidtree_iter fn, void *arg)\n+{\n+\tsize_t klen = oidhexsz / 2;\n+\tstruct oidtree_iter_data x = { 0 };\n+\tassert(oidhexsz <= GIT_MAX_HEXSZ);\n+\n+\tx.fn = fn;\n+\tx.arg = arg;\n+\tx.algo = oid->algo;\n+\tif (oidhexsz & 1) {\n+\t\tx.last_byte = oid->hash[klen];\n+\t\tx.last_nibble_at = &klen;\n+\t}\n+\tcb_each(&ot->tree, (const uint8_t *)oid, klen, iter, &x);\n+}\ndiff --git a/oidtree.h b/oidtree.h\nnew file mode 100644\nindex 0000000000..77898f510a\n--- /dev/null\n+++ b/oidtree.h\n@@ -0,0 +1,22 @@\n+#ifndef OIDTREE_H\n+#define OIDTREE_H\n+\n+#include \"cbtree.h\"\n+#include \"hash.h\"\n+#include \"mem-pool.h\"\n+\n+struct oidtree {\n+\tstruct cb_tree tree;\n+\tstruct mem_pool mem_pool;\n+};\n+\n+void oidtree_init(struct oidtree *);\n+void oidtree_clear(struct oidtree *);\n+void oidtree_insert(struct oidtree *, const struct object_id *);\n+int oidtree_contains(struct oidtree *, const struct object_id *);\n+\n+typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *data);\n+void oidtree_each(struct oidtree *, const struct object_id *,\n+\t\t\tsize_t oidhexsz, oidtree_iter, void *data);\n+\n+#endif /* OIDTREE_H */\ndiff --git a/t/helper/test-oidtree.c b/t/helper/test-oidtree.c\nnew file mode 100644\nindex 0000000000..180ee28dd9\n--- /dev/null\n+++ b/t/helper/test-oidtree.c\n@@ -0,0 +1,49 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"oidtree.h\"\n+\n+static enum cb_next print_oid(const struct object_id *oid, void *data)\n+{\n+\tputs(oid_to_hex(oid));\n+\treturn CB_CONTINUE;\n+}\n+\n+int cmd__oidtree(int argc, const char **argv)\n+{\n+\tstruct oidtree ot;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint nongit_ok;\n+\tint algo = GIT_HASH_UNKNOWN;\n+\n+\toidtree_init(&ot);\n+\tsetup_git_directory_gently(&nongit_ok);\n+\n+\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\tconst char *arg;\n+\t\tstruct object_id oid;\n+\n+\t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n+\t\t\tif (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"insert not a hexadecimal oid: %s\", arg);\n+\t\t\talgo = oid.algo;\n+\t\t\toidtree_insert(&ot, &oid);\n+\t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(\"contains not a hexadecimal oid: %s\", arg);\n+\t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n+\t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n+\t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n+\t\t\tmemset(&oid, 0, sizeof(oid));\n+\t\t\tmemcpy(buf, arg, strlen(arg));\n+\t\t\tbuf[hash_algos[algo].hexsz] = '\\0';\n+\t\t\tget_oid_hex_any(buf, &oid);\n+\t\t\toid.algo = algo;\n+\t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n+\t\t} else if (!strcmp(line.buf, \"clear\")) {\n+\t\t\toidtree_clear(&ot);\n+\t\t} else {\n+\t\t\tdie(\"unknown command: %s\", line.buf);\n+\t\t}\n+\t}\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..9d37debf28 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n \t{ \"mktemp\", cmd__mktemp },\n \t{ \"oid-array\", cmd__oid_array },\n \t{ \"oidmap\", cmd__oidmap },\n+\t{ \"oidtree\", cmd__oidtree },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f683a2f59c 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -32,6 +32,7 @@ int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n int cmd__oidmap(int argc, const char **argv);\n+int cmd__oidtree(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\ndiff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\nnew file mode 100755\nindex 0000000000..bfb1397d7b\n--- /dev/null\n+++ b/t/t0069-oidtree.sh\n@@ -0,0 +1,49 @@\n+#!/bin/sh\n+\n+test_description='basic tests for the oidtree implementation'\n+. ./test-lib.sh\n+\n+maxhexsz=$(test_oid hexsz)\n+echoid () {\n+\tprefix=\"${1:+$1 }\"\n+\tshift\n+\twhile test $# -gt 0\n+\tdo\n+\t\tshortoid=\"$1\"\n+\t\tshift\n+\t\tdifference=$(($maxhexsz - ${#shortoid}))\n+\t\tprintf \"%s%s%0${difference}d\\\\n\" \"$prefix\" \"$shortoid\" \"0\"\n+\tdone\n+}\n+\n+test_expect_success 'oidtree insert and contains' '\n+\tcat >expect <<-\\EOF &&\n+\t\t0\n+\t\t0\n+\t\t0\n+\t\t1\n+\t\t1\n+\t\t0\n+\tEOF\n+\t{\n+\t\techoid insert 444 1 2 3 4 5 a b c d e &&\n+\t\techoid contains 44 441 440 444 4440 4444\n+\t\techo clear\n+\t} | test-tool oidtree >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'oidtree each' '\n+\techoid \"\" 123 321 321 >expect &&\n+\t{\n+\t\techoid insert f 9 8 123 321 a b c d e\n+\t\techo each 12300\n+\t\techo each 3211\n+\t\techo each 3210\n+\t\techo each 32100\n+\t\techo clear\n+\t} | test-tool oidtree >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n"},{"id":"429463","messageId":"20210707231222.GA27550@dcvr","threadId":"55998","inReplyTo":"87zgv276lf.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-07T23:12:22Z","receivedAt":"2021-07-07T23:12:23Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Jun 29 2021, Eric Wong wrote:\n> \n> > +struct alloc_state;\n> > +struct oidtree {\n> > +\tstruct cb_tree t;\n> \n> s/t/tree/? Too short a name for an interface IMO.\n\nDone. I was keeping `t' to match agl's published version\n(and it remains that way in cbtree.[ch])\n\n> > +\tstruct alloc_state *mempool;\n> > +};\n> > +\n> > +#define OIDTREE_INIT { .t = CBTREE_INIT, .mempool = NULL }\n> \n> Let's use designated initilaizers for new code. Just:\n> \n> \t#define OIDTREE_init { \\\n> \t\t.tere = CBTREE_INIT, \\\n> \t}\n> \n> Will do, no need for the \".mempool = NULL\"\n \n> > +static inline void oidtree_init(struct oidtree *ot)\n> > +{\n> > +\tcb_init(&ot->t);\n> > +\tot->mempool = NULL;\n> > +}\n> \n> You can use the \"memcpy() a blank\" trick/idiom here:\n> https://lore.kernel.org/git/patch-2.5-955dbd1693d-20210701T104855Z-avarab@gmail.com/\n> \n> Also, is this even needed? Why have the \"destroy\" re-initialize it?\n\nI'm using mem_pool, now. With the way mem_pool_init works,\nI've decided to do away with OIDTREE_INIT and only use\noidtree_init (and lazy-malloc the entire loose_objects_cache)\n\n> > +void oidtree_destroy(struct oidtree *);\n> \n> Maybe s/destroy/release/, or if you actually need that reset behavior\n> oidtree_reset(). We've got\n\nI'm renaming it oidtree_clear to match oid_array_clear.\n\n> > +void oidtree_insert(struct oidtree *, const struct object_id *);\n> > +int oidtree_contains(struct oidtree *, const struct object_id *);\n> > +\n> > +typedef enum cb_next (*oidtree_iter)(const struct object_id *, void *arg);\n> \n> An \"arg\" name for some arguments, but none for others, if there's a name\n> here call it \"data\" like you do elswhere?\n\nOK, using \"data\". To reduce noise, I prefer to only name\nvariables in prototypes if the usage can't be easily inferred\nfrom its type and function name.\n\n> > +void oidtree_each(struct oidtree *, const struct object_id *,\n> > +\t\t\tsize_t oidhexlen, oidtree_iter, void *arg);\n> \n> s/oidhexlen/hexsz/, like in git_hash_algo.a\n\ndone\n\n> > +++ b/t/helper/test-oidtree.c\n> > @@ -0,0 +1,47 @@\n> > +#include \"test-tool.h\"\n> > +#include \"cache.h\"\n> > +#include \"oidtree.h\"\n> > +\n> > +static enum cb_next print_oid(const struct object_id *oid, void *data)\n> > +{\n> > +\tputs(oid_to_hex(oid));\n> > +\treturn CB_CONTINUE;\n> > +}\n> > +\n> > +int cmd__oidtree(int argc, const char **argv)\n> > +{\n> > +\tstruct oidtree ot = OIDTREE_INIT;\n> > +\tstruct strbuf line = STRBUF_INIT;\n> > +\tint nongit_ok;\n> > +\tint algo = GIT_HASH_UNKNOWN;\n> > +\n> > +\tsetup_git_directory_gently(&nongit_ok);\n> > +\n> > +\twhile (strbuf_getline(&line, stdin) != EOF) {\n> > +\t\tconst char *arg;\n> > +\t\tstruct object_id oid;\n> > +\n> > +\t\tif (skip_prefix(line.buf, \"insert \", &arg)) {\n> > +\t\t\tif (get_oid_hex_any(arg, &oid) == GIT_HASH_UNKNOWN)\n> > +\t\t\t\tdie(\"insert not a hexadecimal oid: %s\", arg);\n> > +\t\t\talgo = oid.algo;\n> > +\t\t\toidtree_insert(&ot, &oid);\n> > +\t\t} else if (skip_prefix(line.buf, \"contains \", &arg)) {\n> > +\t\t\tif (get_oid_hex(arg, &oid))\n> > +\t\t\t\tdie(\"contains not a hexadecimal oid: %s\", arg);\n> > +\t\t\tprintf(\"%d\\n\", oidtree_contains(&ot, &oid));\n> > +\t\t} else if (skip_prefix(line.buf, \"each \", &arg)) {\n> > +\t\t\tchar buf[GIT_MAX_HEXSZ + 1] = { '0' };\n> > +\t\t\tmemset(&oid, 0, sizeof(oid));\n> > +\t\t\tmemcpy(buf, arg, strlen(arg));\n> > +\t\t\tbuf[hash_algos[algo].hexsz] = 0;\n> \n> = '\\0' if it's the intent to have a NULL-terminated string is more\n> readable.\n\ndone\n\n> > +\t\t\tget_oid_hex_any(buf, &oid);\n> > +\t\t\toid.algo = algo;\n> > +\t\t\toidtree_each(&ot, &oid, strlen(arg), print_oid, NULL);\n> > +\t\t} else if (!strcmp(line.buf, \"destroy\"))\n> > +\t\t\toidtree_destroy(&ot);\n> > +\t\telse\n> > +\t\t\tdie(\"unknown command: %s\", line.buf);\n> \n> Missing braces.\n\nAdded.\n\n> > +\t}\n> > +\treturn 0;\n> > +}\n> > diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> > index c5bd0c6d4c..9d37debf28 100644\n> > --- a/t/helper/test-tool.c\n> > +++ b/t/helper/test-tool.c\n> > @@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n> > \t{ \"mktemp\", cmd__mktemp },\n> > \t{ \"oid-array\", cmd__oid_array },\n> > \t{ \"oidmap\", cmd__oidmap },\n> > +\t{ \"oidtree\", cmd__oidtree },\n> > \t{ \"online-cpus\", cmd__online_cpus },\n> > \t{ \"parse-options\", cmd__parse_options },\n> > \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n> > diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> > index e8069a3b22..f683a2f59c 100644\n> > --- a/t/helper/test-tool.h\n> > +++ b/t/helper/test-tool.h\n> > @@ -32,6 +32,7 @@ int cmd__match_trees(int argc, const char **argv);\n> > int cmd__mergesort(int argc, const char **argv);\n> > int cmd__mktemp(int argc, const char **argv);\n> > int cmd__oidmap(int argc, const char **argv);\n> > +int cmd__oidtree(int argc, const char **argv);\n> > int cmd__online_cpus(int argc, const char **argv);\n> > int cmd__parse_options(int argc, const char **argv);\n> > int cmd__parse_pathspec_file(int argc, const char** argv);\n> > diff --git a/t/t0069-oidtree.sh b/t/t0069-oidtree.sh\n> > new file mode 100755\n> > index 0000000000..0594f57c81\n> > --- /dev/null\n> > +++ b/t/t0069-oidtree.sh\n> > @@ -0,0 +1,52 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='basic tests for the oidtree implementation'\n> > +. ./test-lib.sh\n> > +\n> > +echoid () {\n> > +\tprefix=\"${1:+$1 }\"\n> > +\tshift\n> > +\twhile test $# -gt 0\n> > +\tdo\n> > +\t\techo \"$1\"\n> > +\t\tshift\n> > +\tdone | awk -v prefix=\"$prefix\" -v ZERO_OID=$ZERO_OID '{\n> > +\t\tprintf(\"%s%s\", prefix, $0);\n> > +\t\tneed = length(ZERO_OID) - length($0);\n> > +\t\tfor (i = 0; i < need; i++)\n> > +\t\t\tprintf(\"0\");\n> > +\t\tprintf \"\\n\";\n> > +\t}'\n> > +}\n> \n> Looks fairly easy to do in pure-shell, first of all you don't need a\n> length() on $ZERO_OID, use $(test_oid hexsz) instead. That applies for\n> the awk version too.\n\nAh, I didn't know about test_oid, using it, now.\n\n> But once you have that and the N arguments just do a wc -c on the\n> argument, use $(()) to compute the $difference, and a loop with:\n> \n> printf \"%s%s%0${difference}d\" \"$prefix\" \"$shortoid\" \"0\"\n\nI also wanted to avoid repeated 'wc -c' and figured awk was\nportable enough since we use it elsewhere in tests. I've now\nnoticed \"${#var}\" is portable and we're already relying on it in\npacketize(), so I'm using that.\n\n> > +\n> > +test_expect_success 'oidtree insert and contains' '\n> > +\tcat >expect < > +0\n> > +0\n> > +0\n> > +1\n> > +1\n> > +0\n> > +EOF\n> \n> use \"<<-\\EOF\" and indent it.\n\ndone\n\nThanks all for the reviews.\n"},{"id":"429467","messageId":"xmqq8s2hd5ay.fsf@gitster.g","threadId":"55998","inReplyTo":"20210707231019.14738-2-e@80x24.org","subject":"Re: [PATCH v3 1/5] speed up alt_odb_usable() with many alternates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-08T00:20:53Z","receivedAt":"2021-07-08T00:20:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> With many alternates, the duplicate check in alt_odb_usable()\n> wastes many cycles doing repeated fspathcmp() on every existing\n> alternate. Use a khash to speed up lookups by odb->path.\n>\n> Since the kh_put_* API uses the supplied key without\n> duplicating it, we also take advantage of it to replace both\n> xstrdup() and strbuf_release() in link_alt_odb_entry() with\n> strbuf_detach() to avoid the allocation and copy.\n>\n> In a test repository with 50K alternates and each of those 50K\n> alternates having one alternate each (for a total of 100K total\n> alternates); this speeds up lookup of a non-existent blob from\n> over 16 minutes to roughly 2.7 seconds on my busy workstation.\n>\n> Note: all underlying git object directories were small and\n> unpacked with only loose objects and no packs. Having to load\n> packs increases times significantly.\n>\n> v3: Introduce and use fspatheq and fspathhash functions;\n> avoid unnecessary checks for allocation failures already\n> handled by our own *alloc wrappers.\n\nThe last one does not belong to the commit log message, as \"git log\"\nreaders do not care about and will not have access to v2 and earlier.\n"},{"id":"429468","messageId":"20210708011446.GA15899@dcvr","threadId":"55998","inReplyTo":"xmqq8s2hd5ay.fsf@gitster.g","subject":"Re: [PATCH v3 1/5] speed up alt_odb_usable() with many alternates","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-08T01:14:46Z","receivedAt":"2021-07-08T01:14:47Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Eric Wong writes:\n> > v3: Introduce and use fspatheq and fspathhash functions;\n> > avoid unnecessary checks for allocation failures already\n> > handled by our own *alloc wrappers.\n> \n> The last one does not belong to the commit log message, as \"git log\"\n> readers do not care about and will not have access to v2 and earlier.\n\nOops :x Are you going to remove that on your end or do you want a resend?\nLooking at \"git log\", it looks like I've done it a bit over the years :x\n"},{"id":"429471","messageId":"xmqq1r89ctqh.fsf@gitster.g","threadId":"55998","inReplyTo":"20210708011446.GA15899@dcvr","subject":"Re: [PATCH v3 1/5] speed up alt_odb_usable() with many alternates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-08T04:30:46Z","receivedAt":"2021-07-08T04:30:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> Junio C Hamano wrote:\n>> Eric Wong writes:\n>> > v3: Introduce and use fspatheq and fspathhash functions;\n>> > avoid unnecessary checks for allocation failures already\n>> > handled by our own *alloc wrappers.\n>> \n>> The last one does not belong to the commit log message, as \"git log\"\n>> readers do not care about and will not have access to v2 and earlier.\n>\n> Oops :x Are you going to remove that on your end or do you want a resend?\n> Looking at \"git log\", it looks like I've done it a bit over the years :x\n\nOK. I guess there are a few more patches in the series with the\nsame issue. \"rebase -i\" is our friend ;-)\n"},{"id":"429472","messageId":"xmqqwnq1bdwq.fsf@gitster.g","threadId":"55998","inReplyTo":"20210707231019.14738-3-e@80x24.org","subject":"Re: [PATCH v3 2/5] avoid strlen via strbuf_addstr in link_alt_odb_entry","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-08T04:57:57Z","receivedAt":"2021-07-08T04:58:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> We can save a few milliseconds (across 100K odbs) by using\n> strbuf_addbuf() instead of strbuf_addstr() by passing `entry' as\n> a strbuf pointer rather than a \"const char *\".\n\nOK; trivially corect ;-)\n\n>\n> Signed-off-by: Eric Wong \n> ---\n> object-file.c | 8 ++++----\n> 1 file changed, 4 insertions(+), 4 deletions(-)\n>\n> diff --git a/object-file.c b/object-file.c\n> index a13f49b192..2dd70ddf3a 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -567,18 +567,18 @@ static int alt_odb_usable(struct raw_object_store *o,\n> static void read_info_alternates(struct repository *r,\n> \t\t\t\t const char *relative_base,\n> \t\t\t\t int depth);\n> -static int link_alt_odb_entry(struct repository *r, const char *entry,\n> +static int link_alt_odb_entry(struct repository *r, const struct strbuf *entry,\n> \tconst char *relative_base, int depth, const char *normalized_objdir)\n> {\n> \tstruct object_directory *ent;\n> \tstruct strbuf pathbuf = STRBUF_INIT;\n> \tkhiter_t pos;\n> \n> -\tif (!is_absolute_path(entry) && relative_base) {\n> +\tif (!is_absolute_path(entry->buf) && relative_base) {\n> \t\tstrbuf_realpath(&pathbuf, relative_base, 1);\n> \t\tstrbuf_addch(&pathbuf, '/');\n> \t}\n> -\tstrbuf_addstr(&pathbuf, entry);\n> +\tstrbuf_addbuf(&pathbuf, entry);\n> \n> \tif (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {\n> \t\terror(_(\"unable to normalize alternate object path: %s\"),\n> @@ -669,7 +669,7 @@ static void link_alt_odb_entries(struct repository *r, const char *alt,\n> \t\talt = parse_alt_odb_entry(alt, sep, &entry);\n> \t\tif (!entry.len)\n> \t\t\tcontinue;\n> -\t\tlink_alt_odb_entry(r, entry.buf,\n> +\t\tlink_alt_odb_entry(r, &entry,\n> \t\t\t\t relative_base, depth, objdirbuf.buf);\n> \t}\n> \tstrbuf_release(&entry);\n"},{"id":"432180","messageId":"3cbec773-cd99-cf9f-a713-45ef8e6746c3@ahunt.org","threadId":"55998","inReplyTo":"20210629205305.7100-6-e@80x24.org","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-08-06T15:31:19Z","receivedAt":"2021-08-06T15:31:30Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 29/06/2021 22:53, Eric Wong wrote:\n> [...snip...]\n> diff --git a/oidtree.c b/oidtree.c\n> new file mode 100644\n> index 0000000000..c1188d8f48\n> --- /dev/null\n> +++ b/oidtree.c\n> @@ -0,0 +1,94 @@\n> +/*\n> + * A wrapper around cbtree which stores oids\n> + * May be used to replace oid-array for prefix (abbreviation) matches\n> + */\n> +#include \"oidtree.h\"\n> +#include \"alloc.h\"\n> +#include \"hash.h\"\n> +\n> +struct oidtree_node {\n> +\t/* n.k[] is used to store \"struct object_id\" */\n> +\tstruct cb_node n;\n> +};\n> +\n> [... snip ...]\n> +\n> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n> +{\n> +\tstruct oidtree_node *on;\n> +\n> +\tif (!ot->mempool)\n> +\t\tot->mempool = allocate_alloc_state();\n> +\tif (!oid->algo)\n> +\t\tBUG(\"oidtree_insert requires oid->algo\");\n> +\n> +\ton = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n> +\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n\nI think this object_id cast introduced undefined behaviour - here's my \nlayperson's interepretation of what's going on (full UBSAN output is \npasted below):\n\ncb_node.k is a uint8_t[], and hence can be 1-byte aligned (on my \nmachine: offsetof(struct cb_node, k) == 21). We're casting its pointer \nto \"struct object_id *\", and later try to access object_id.hash within \noidcpy_with_padding. My compiler assumes that an object_id pointer needs \nto be 4-byte aligned, and reading from a misaligned pointer means we hit \nundefined behaviour. (I think the 4-byte alignment requirement comes \nfrom the fact that object_id's largest member is an int?)\n\nI'm not sure what an elegant and idiomatic fix might be - IIUC it's hard \nto guarantee misaligned access can't happen with a flex array that's \nbeing used for arbitrary data (you would presumably have to declare it \nas an array of whatever the largest supported type is, so that you can \nguarantee correct alignment even when cbtree is used with that type) - \nwhich might imply that k needs to be declared as a void pointer? That in \nturn would make cbtree.c harder to read.\n\nAnyhow, here's the UBSAN output from t0000 running against next:\n\nhash.h:277:14: runtime error: member access within misaligned address \n0x7fcb31c4103d for type 'struct object_id', which requires 4 byte alignment\n0x7fcb31c4103d: note: pointer points here\n 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a \n 5a 5a 5a 5a 5a 5a 5a 5a 5a\n ^\n #0 0xc76d9d in oidcpy_with_padding hash.h:277:14\n #1 0xc768f5 in oidtree_insert oidtree.c:44:2\n #2 0xc418e3 in append_loose_object object-file.c:2398:2\n #3 0xc3fbdc in for_each_file_in_obj_subdir object-file.c:2316:9\n #4 0xc41785 in odb_loose_cache object-file.c:2424:2\n #5 0xc50336 in find_short_object_filename object-name.c:103:16\n #6 0xc50e04 in repo_find_unique_abbrev_r object-name.c:712:2\n #7 0xc519a9 in repo_find_unique_abbrev object-name.c:727:2\n #8 0x9b6ce2 in diff_abbrev_oid diff.c:4208:10\n #9 0x9f13d0 in fill_metainfo diff.c:4286:8\n #10 0x9f02d6 in run_diff_cmd diff.c:4322:3\n #11 0x9efbef in run_diff diff.c:4422:3\n #12 0x9c9ac9 in diff_flush_patch diff.c:5765:2\n #13 0x9c9e74 in diff_flush_patch_all_file_pairs diff.c:6246:4\n #14 0x9be33e in diff_flush diff.c:6387:3\n #15 0xb8864e in log_tree_diff_flush log-tree.c:895:2\n #16 0xb8987b in log_tree_diff log-tree.c:933:4\n #17 0xb88c9a in log_tree_commit log-tree.c:988:10\n #18 0x5b4257 in cmd_log_walk log.c:426:8\n #19 0x5b6224 in cmd_show log.c:698:10\n #20 0x42ec48 in run_builtin git.c:461:11\n #21 0x4295e0 in handle_builtin git.c:714:3\n #22 0x42d043 in run_argv git.c:781:4\n #23 0x428cc2 in cmd_main git.c:912:19\n #24 0x7791ce in main common-main.c:52:11\n #25 0x7fcb30aab349 in __libc_start_main (/lib64/libc.so.6+0x24349)\n #26 0x4074a9 in _start start.S:120\n\nSUMMARY: UndefinedBehaviorSanitizer: undefined-behavior hash.h:277:14 in\n\n\n> +\n> +\t/*\n> +\t * n.b. we shouldn't get duplicates, here, but we'll have\n> +\t * a small leak that won't be freed until oidtree_destroy\n> +\t */\n> +\tcb_insert(&ot->t, &on->n, sizeof(*oid));\n> +}\n> +\n\n\nATB,\n\nAndrzej\n"},{"id":"432190","messageId":"bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de","threadId":"55998","inReplyTo":"3cbec773-cd99-cf9f-a713-45ef8e6746c3@ahunt.org","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-06T17:53:47Z","receivedAt":"2021-08-06T17:54:22Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 06.08.21 um 17:31 schrieb Andrzej Hunt:\n>\n>\n> On 29/06/2021 22:53, Eric Wong wrote:\n>> [...snip...]\n>> diff --git a/oidtree.c b/oidtree.c\n>> new file mode 100644\n>> index 0000000000..c1188d8f48\n>> --- /dev/null\n>> +++ b/oidtree.c\n>> @@ -0,0 +1,94 @@\n>> +/*\n>> + * A wrapper around cbtree which stores oids\n>> + * May be used to replace oid-array for prefix (abbreviation) matches\n>> + */\n>> +#include \"oidtree.h\"\n>> +#include \"alloc.h\"\n>> +#include \"hash.h\"\n>> +\n>> +struct oidtree_node {\n>> +    /* n.k[] is used to store \"struct object_id\" */\n>> +    struct cb_node n;\n>> +};\n>> +\n>> [... snip ...]\n>> +\n>> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n>> +{\n>> +    struct oidtree_node *on;\n>> +\n>> +    if (!ot->mempool)\n>> +        ot->mempool = allocate_alloc_state();\n>> +    if (!oid->algo)\n>> +        BUG(\"oidtree_insert requires oid->algo\");\n>> +\n>> +    on = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n>> +    oidcpy_with_padding((struct object_id *)on->n.k, oid);\n>\n> I think this object_id cast introduced undefined behaviour - here's\n> my layperson's interepretation of what's going on (full UBSAN output\n> is pasted below):\n>\n> cb_node.k is a uint8_t[], and hence can be 1-byte aligned (on my\n> machine: offsetof(struct cb_node, k) == 21). We're casting its\n> pointer to \"struct object_id *\", and later try to access\n> object_id.hash within oidcpy_with_padding. My compiler assumes that\n> an object_id pointer needs to be 4-byte aligned, and reading from a\n> misaligned pointer means we hit undefined behaviour. (I think the\n> 4-byte alignment requirement comes from the fact that object_id's\n> largest member is an int?)\n>\n> I'm not sure what an elegant and idiomatic fix might be - IIUC it's\n> hard to guarantee misaligned access can't happen with a flex array\n> that's being used for arbitrary data (you would presumably have to\n> declare it as an array of whatever the largest supported type is, so\n> that you can guarantee correct alignment even when cbtree is used\n> with that type) - which might imply that k needs to be declared as a\n> void pointer? That in turn would make cbtree.c harder to read.\n\nC11 has alignas. We could also make the member before the flex array,\notherbits, wider, e.g. promote it to uint32_t.\n\nA more parsimonious solution would be to turn the int member of struct\nobject_id, algo, into an unsigned char for now and reconsider the issue\nonce we support our 200th algorithm or so. This breaks notes, though.\nIts GET_PTR_TYPE seems to require struct leaf_node to have 4-byte\nalignment for some reason. That can be ensured by adding an int member.\n\nAnyway, with either of these fixes UBSan is still unhappy about a\ndifferent issue. Here's a patch for that:\n\n--- >8 ---\nSubject: [PATCH] object-file: use unsigned arithmetic with bit mask\n\n33f379eee6 (make object_directory.loose_objects_subdir_seen a bitmap,\n2021-07-07) replaced a wasteful 256-byte array with a 32-byte array\nand bit operations. The mask calculation shifts a literal 1 of type\nint left by anything between 0 and 31. UndefinedBehaviorSanitizer\ndoesn't like that and reports:\n\nobject-file.c:2477:18: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'\n\nMake sure to use an unsigned 1 instead to avoid the issue.\n\nSigned-off-by: René Scharfe \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 3d27dc8dea..a8be899481 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2474,7 +2474,7 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n-\tsize_t mask = 1 << (subdir_nr % word_bits);\n+\tsize_t mask = 1u << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n\n \tif (subdir_nr < 0 ||\n--\n2.32.0\n"},{"id":"432261","messageId":"20210807224957.GA5068@dcvr","threadId":"55998","inReplyTo":"bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-08-07T22:49:57Z","receivedAt":"2021-08-07T22:50:02Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"René Scharfe wrote:\n> Am 06.08.21 um 17:31 schrieb Andrzej Hunt:\n> > On 29/06/2021 22:53, Eric Wong wrote:\n> >> [...snip...]\n> >> diff --git a/oidtree.c b/oidtree.c\n> >> new file mode 100644\n> >> index 0000000000..c1188d8f48\n> >> --- /dev/null\n> >> +++ b/oidtree.c\n\n> >> +struct oidtree_node {\n> >> +    /* n.k[] is used to store \"struct object_id\" */\n> >> +    struct cb_node n;\n> >> +};\n> >> +\n> >> [... snip ...]\n> >> +\n> >> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n> >> +{\n> >> +    struct oidtree_node *on;\n> >> +\n> >> +    if (!ot->mempool)\n> >> +        ot->mempool = allocate_alloc_state();\n> >> +    if (!oid->algo)\n> >> +        BUG(\"oidtree_insert requires oid->algo\");\n> >> +\n> >> +    on = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n> >> +    oidcpy_with_padding((struct object_id *)on->n.k, oid);\n> >\n> > I think this object_id cast introduced undefined behaviour - here's\n> > my layperson's interepretation of what's going on (full UBSAN output\n> > is pasted below):\n> >\n> > cb_node.k is a uint8_t[], and hence can be 1-byte aligned (on my\n> > machine: offsetof(struct cb_node, k) == 21). We're casting its\n> > pointer to \"struct object_id *\", and later try to access\n> > object_id.hash within oidcpy_with_padding. My compiler assumes that\n> > an object_id pointer needs to be 4-byte aligned, and reading from a\n> > misaligned pointer means we hit undefined behaviour. (I think the\n> > 4-byte alignment requirement comes from the fact that object_id's\n> > largest member is an int?)\n\nI seem to recall struct alignment requirements being\narchitecture-dependent; and x86/x86-64 are the most liberal\nw.r.t alignment requirements.\n\n> > I'm not sure what an elegant and idiomatic fix might be - IIUC it's\n> > hard to guarantee misaligned access can't happen with a flex array\n> > that's being used for arbitrary data (you would presumably have to\n> > declare it as an array of whatever the largest supported type is, so\n> > that you can guarantee correct alignment even when cbtree is used\n> > with that type) - which might imply that k needs to be declared as a\n> > void pointer? That in turn would make cbtree.c harder to read.\n> \n> C11 has alignas. We could also make the member before the flex array,\n> otherbits, wider, e.g. promote it to uint32_t.\n\nUgh, no. cb_node should be as small as possible and (for our\ncurrent purposes) ->byte could be uint8_t.\n\n> A more parsimonious solution would be to turn the int member of struct\n> object_id, algo, into an unsigned char for now and reconsider the issue\n> once we support our 200th algorithm or so.\n\nYes, making struct object_id smaller would benefit all git users\n(at least for the next few centuries :P).\n\n> This breaks notes, though.\n> Its GET_PTR_TYPE seems to require struct leaf_node to have 4-byte\n> alignment for some reason. That can be ensured by adding an int member.\n\nAdding a 4-byte int to leaf_node after shaving 6-bytes off two\nobject_id structs would mean a net savings of 2 bytes;\nsounds good to me.\n\nI don't know much about notes nor the associated code,\nbut I also wonder if crit-bit tree can be used there, too.\n\n> Anyway, with either of these fixes UBSan is still unhappy about a\n> different issue. Here's a patch for that:\n\nThanks \n"},{"id":"432279","messageId":"CAPUEsphf9F1+=zOMKx3j=jH8xqDwQX99+9uHiYUpXhFE1nervg@mail.gmail.com","threadId":"55998","inReplyTo":"20210807224957.GA5068@dcvr","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T01:35:59Z","receivedAt":"2021-08-09T01:36:13Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Sat, Aug 7, 2021 at 3:51 PM Eric Wong wrote:\n>\n> René Scharfe wrote:\n> > Am 06.08.21 um 17:31 schrieb Andrzej Hunt:\n> > > On 29/06/2021 22:53, Eric Wong wrote:\n> > >> [...snip...]\n> > >> diff --git a/oidtree.c b/oidtree.c\n> > >> new file mode 100644\n> > >> index 0000000000..c1188d8f48\n> > >> --- /dev/null\n> > >> +++ b/oidtree.c\n>\n> > >> +struct oidtree_node {\n> > >> + /* n.k[] is used to store \"struct object_id\" */\n> > >> + struct cb_node n;\n> > >> +};\n> > >> +\n> > >> [... snip ...]\n> > >> +\n> > >> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n> > >> +{\n> > >> + struct oidtree_node *on;\n> > >> +\n> > >> + if (!ot->mempool)\n> > >> + ot->mempool = allocate_alloc_state();\n> > >> + if (!oid->algo)\n> > >> + BUG(\"oidtree_insert requires oid->algo\");\n> > >> +\n> > >> + on = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n> > >> + oidcpy_with_padding((struct object_id *)on->n.k, oid);\n> > >\n> > > I think this object_id cast introduced undefined behaviour - here's\n> > > my layperson's interepretation of what's going on (full UBSAN output\n> > > is pasted below):\n> > >\n> > > cb_node.k is a uint8_t[], and hence can be 1-byte aligned (on my\n> > > machine: offsetof(struct cb_node, k) == 21). We're casting its\n> > > pointer to \"struct object_id *\", and later try to access\n> > > object_id.hash within oidcpy_with_padding. My compiler assumes that\n> > > an object_id pointer needs to be 4-byte aligned, and reading from a\n> > > misaligned pointer means we hit undefined behaviour. (I think the\n> > > 4-byte alignment requirement comes from the fact that object_id's\n> > > largest member is an int?)\n>\n> I seem to recall struct alignment requirements being\n> architecture-dependent; and x86/x86-64 are the most liberal\n> w.r.t alignment requirements.\n\nI think the problem here is not the alignment though, but the fact that\nthe nesting of structs with flexible arrays is forbidden by ISO/IEC\n9899:2011 6.7.2.1¶3 that reads :\n\n6.7.2.1 Structure and union specifiers\n\n¶3 A structure or union shall not contain a member with incomplete or\nfunction type (hence, a structure shall not contain an instance of\nitself, but may contain a pointer to an instance of itself), except\nthat the last member of a structure with more than one named member\nmay have incomplete array type; such a structure (and any union\ncontaining, possibly recursively, a member that is such a structure)\nshall not be a member of a structure or an element of an array.\n\nand it will throw a warning with clang 12\n(-Wflexible-array-extensions) or gcc 11 (-Werror=pedantic) when using\nDEVOPTS=pedantic\n\nMy somewhat naive suggestion was to avoid the struct nesting by\nremoving struct oidtree_node and using a struct cb_node directly.\n\nWill reply with a small series of patches that fix pedantic related\nwarnings in ew/many-alternate-optim on top of next.\n\nCarlo\n"},{"id":"432280","messageId":"20210809013833.58110-1-carenas@gmail.com","threadId":"55998","inReplyTo":"CAPUEsphf9F1+=zOMKx3j=jH8xqDwQX99+9uHiYUpXhFE1nervg@mail.gmail.com","subject":"[PATCH/RFC 0/3] pedantic errors in next","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T01:38:30Z","receivedAt":"2021-08-09T01:38:52Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"Building next with pedantic enabled shows the following 2 issues that\nwere originally in ew/many-alternate-optim, apologies for not catching\nthem earlier.\n\nthe second one could be skipped, and has indeed another similar case\nalready in seen which will be send separately.\n\nthe third patch adds a CI job that could be used to detect this issues\nearly and that adds about 5m of computing time.\n\nCarlo Marcelo Arenas Belón (3):\n oidtree: avoid nested struct oidtree_node\n object-store: avoid extra ';' from KHASH_INIT\n ci: run a pedantic build as part of the GitHub workflow\n\n .github/workflows/main.yml | 2 ++\n ci/install-docker-dependencies.sh | 4 ++++\n ci/run-build-and-tests.sh | 10 +++++++---\n object-store.h | 2 +-\n oidtree.c | 11 +++--------\n 5 files changed, 17 insertions(+), 12 deletions(-)\n\n-- \n2.33.0.rc1.373.gc715f1a457\n\n"},{"id":"432281","messageId":"20210809013833.58110-3-carenas@gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-1-carenas@gmail.com","subject":"[PATCH/RFC 2/3] object-store: avoid extra ';' from KHASH_INIT","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T01:38:32Z","receivedAt":"2021-08-09T01:38:54Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"cf2dc1c238 (speed up alt_odb_usable() with many alternates, 2021-07-07)\nintroduces a KHASH_INIT invocation with a trailing ';', which while\ncommonly expected will trigger warnings with pedantic on both\nclang[-Wextra-semi] and gcc[-Wpedantic], because that macro has already\na semicolon and is meant to be invoked without one.\n\nwhile fixing the macro would be a worthy solution (specially considering\nthis is a common recurring problem), remove the extra ';' for now to\nminimize churn.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n object-store.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex e679acc4c3..d24915ced1 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -34,7 +34,7 @@ struct object_directory {\n };\n \n KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n-\tstruct object_directory *, 1, fspathhash, fspatheq);\n+\tstruct object_directory *, 1, fspathhash, fspatheq)\n \n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n-- \n2.33.0.rc1.373.gc715f1a457\n\n"},{"id":"432282","messageId":"20210809013833.58110-2-carenas@gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-1-carenas@gmail.com","subject":"[PATCH/RFC 1/3] oidtree: avoid nested struct oidtree_node","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T01:38:31Z","receivedAt":"2021-08-09T01:38:56Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"92d8ed8ac1 (oidtree: a crit-bit tree for odb_loose_cache, 2021-07-07)\nadds a struct oidtree_node that contains only an n field with a\nstruct cb_node.\n\nunfortunately, while building in pedantic mode witch clang 12 (as well\nas a similar error from gcc 11) it will show:\n\n oidtree.c:11:17: error: 'n' may not be nested in a struct due to flexible array member [-Werror,-Wflexible-array-extensions]\n struct cb_node n;\n ^\n\nbecause of a constrain coded in ISO C 11 6.7.2.1¶3 that forbids using\nstructs that contain a flexible array as part of another struct.\n\nuse a strict cb_node directly instead.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n oidtree.c | 11 +++--------\n 1 file changed, 3 insertions(+), 8 deletions(-)\n\ndiff --git a/oidtree.c b/oidtree.c\nindex 7eb0e9ba05..580cab8ae2 100644\n--- a/oidtree.c\n+++ b/oidtree.c\n@@ -6,11 +6,6 @@\n #include \"alloc.h\"\n #include \"hash.h\"\n \n-struct oidtree_node {\n-\t/* n.k[] is used to store \"struct object_id\" */\n-\tstruct cb_node n;\n-};\n-\n struct oidtree_iter_data {\n \toidtree_iter fn;\n \tvoid *arg;\n@@ -35,13 +30,13 @@ void oidtree_clear(struct oidtree *ot)\n \n void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n {\n-\tstruct oidtree_node *on;\n+\tstruct cb_node *on;\n \n \tif (!oid->algo)\n \t\tBUG(\"oidtree_insert requires oid->algo\");\n \n \ton = mem_pool_alloc(&ot->mem_pool, sizeof(*on) + sizeof(*oid));\n-\toidcpy_with_padding((struct object_id *)on->n.k, oid);\n+\toidcpy_with_padding((struct object_id *)on->k, oid);\n \n \t/*\n \t * n.b. Current callers won't get us duplicates, here. If a\n@@ -49,7 +44,7 @@ void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n \t * that won't be freed until oidtree_clear. Currently it's not\n \t * worth maintaining a free list\n \t */\n-\tcb_insert(&ot->tree, &on->n, sizeof(*oid));\n+\tcb_insert(&ot->tree, on, sizeof(*oid));\n }\n \n \n-- \n2.33.0.rc1.373.gc715f1a457\n\n"},{"id":"432283","messageId":"20210809013833.58110-4-carenas@gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-1-carenas@gmail.com","subject":"[PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T01:38:33Z","receivedAt":"2021-08-09T01:38:57Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"similar to the recently added sparse task, it is nice to know as early\nas possible.\n\nadd a dockerized build using fedora (that usually has the latest gcc)\nto be ahead of the curve and avoid older ISO C issues at the same time.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n .github/workflows/main.yml | 2 ++\n ci/install-docker-dependencies.sh | 4 ++++\n ci/run-build-and-tests.sh | 10 +++++++---\n 3 files changed, 13 insertions(+), 3 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 47876a4f02..6b9427eff1 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -259,6 +259,8 @@ jobs:\n image: alpine\n - jobname: Linux32\n image: daald/ubuntu32:xenial\n+ - jobname: pedantic\n+ image: fedora\n env:\n jobname: ${{matrix.vector.jobname}}\n runs-on: ubuntu-latest\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 26a6689766..07a8c6b199 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -15,4 +15,8 @@ linux-musl)\n \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n \t;;\n+pedantic)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n esac\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 3ce81ffee9..f3aba5d6cb 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -10,6 +10,11 @@ windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n *) ln -s \"$cache_dir/.prove\" t/.prove;;\n esac\n \n+if test \"$jobname\" = \"pedantic\"\n+then\n+\texport DEVOPTS=pedantic\n+fi\n+\n make\n case \"$jobname\" in\n linux-gcc)\n@@ -35,10 +40,9 @@ linux-clang)\n \texport GIT_TEST_DEFAULT_HASH=sha256\n \tmake test\n \t;;\n-linux-gcc-4.8)\n+linux-gcc-4.8|pedantic)\n \t# Don't run the tests; we only care about whether Git can be\n-\t# built with GCC 4.8, as it errors out on some undesired (C99)\n-\t# constructs that newer compilers seem to quietly accept.\n+\t# built with GCC 4.8 or with pedantic\n \t;;\n *)\n \tmake test\n-- \n2.33.0.rc1.373.gc715f1a457\n\n"},{"id":"432302","messageId":"c903b477-8438-7c9a-bbbc-ec87d1b05451@gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-4-carenas@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-08-09T10:50:35Z","receivedAt":"2021-08-09T10:50:42Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 09/08/21 08.38, Carlo Marcelo Arenas Belón wrote:\n> similar to the recently added sparse task, it is nice to know as early\n> as possible.\n> \n> add a dockerized build using fedora (that usually has the latest gcc)\n> to be ahead of the curve and avoid older ISO C issues at the same time.\n> \n\nBut from GCC manual [1], the default C dialect used is `-std=gnu17`, \nwhile `-pedantic` is only relevant for ISO C (such as `-std=c17`).\n\nAnd why not using `-pedantic-errors`, so that non-ISO features are \ntreated as errors?\n\nNewcomers contributing to Git may think that based on what our CI do, \nthey can submit patches with C17 features (perhaps with GNU extensions). \nThen at some time there is casual users that complain that Git doesn't \ncompile with their default older compiler (maybe they run LTS \ndistributions or pre-C17 compiler). Thus we want Git to be compiled \nsuccessfully using wide variety of compilers (maybe as old as GCC 4.8).\n\n[1]: https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gcc/Standards.html#Standards\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"432304","messageId":"1b096830-3e01-efbe-25dc-c0505c8bac7b@gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-4-carenas@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-08-09T14:56:01Z","receivedAt":"2021-08-09T14:56:09Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Carlo\n\nOn 09/08/2021 02:38, Carlo Marcelo Arenas Belón wrote:\n> similar to the recently added sparse task, it is nice to know as early\n> as possible.\n> \n> add a dockerized build using fedora (that usually has the latest gcc)\n> to be ahead of the curve and avoid older ISO C issues at the same time.\n\nIf we want to be able to compile with -Wpedantic then it might be better \njust to turn it on unconditionally in config.mak.dev. Then developers \nwill see any errors before they push and the ci builds will all use it \nrather than having to run an extra job. I had a quick scan of the mail \narchive threads starting at [1,2] and it's not clear to me why \n-Wpedaintic was added as an optional extra.\n\nTotally unrelated to this patch but while looking at the ci scripts I \nnoticed that we only run the linux-gcc-4.8 job on travis, not on github.\n\nBest Wishes\n\nPhillip\n\n[1] https://lore.kernel.org/git/20180721185933.32377-1-dev+git@drbeat.li/\n[2] https://lore.kernel.org/git/20180721203647.2619-1-dev+git@drbeat.li/\n\n> Signed-off-by: Carlo Marcelo Arenas Belón \n> ---\n> .github/workflows/main.yml | 2 ++\n> ci/install-docker-dependencies.sh | 4 ++++\n> ci/run-build-and-tests.sh | 10 +++++++---\n> 3 files changed, 13 insertions(+), 3 deletions(-)\n> \n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index 47876a4f02..6b9427eff1 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -259,6 +259,8 @@ jobs:\n> image: alpine\n> - jobname: Linux32\n> image: daald/ubuntu32:xenial\n> + - jobname: pedantic\n> + image: fedora\n> env:\n> jobname: ${{matrix.vector.jobname}}\n> runs-on: ubuntu-latest\n> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> index 26a6689766..07a8c6b199 100755\n> --- a/ci/install-docker-dependencies.sh\n> +++ b/ci/install-docker-dependencies.sh\n> @@ -15,4 +15,8 @@ linux-musl)\n> \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> \t;;\n> +pedantic)\n> +\tdnf -yq update >/dev/null &&\n> +\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n> +\t;;\n> esac\n> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> index 3ce81ffee9..f3aba5d6cb 100755\n> --- a/ci/run-build-and-tests.sh\n> +++ b/ci/run-build-and-tests.sh\n> @@ -10,6 +10,11 @@ windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n> *) ln -s \"$cache_dir/.prove\" t/.prove;;\n> esac\n> \n> +if test \"$jobname\" = \"pedantic\"\n> +then\n> +\texport DEVOPTS=pedantic\n> +fi\n> +\n> make\n> case \"$jobname\" in\n> linux-gcc)\n> @@ -35,10 +40,9 @@ linux-clang)\n> \texport GIT_TEST_DEFAULT_HASH=sha256\n> \tmake test\n> \t;;\n> -linux-gcc-4.8)\n> +linux-gcc-4.8|pedantic)\n> \t# Don't run the tests; we only care about whether Git can be\n> -\t# built with GCC 4.8, as it errors out on some undesired (C99)\n> -\t# constructs that newer compilers seem to quietly accept.\n> +\t# built with GCC 4.8 or with pedantic\n> \t;;\n> *)\n> \tmake test\n> \n"},{"id":"432309","messageId":"xmqq1r72haww.fsf@gitster.g","threadId":"55998","inReplyTo":"20210809013833.58110-3-carenas@gmail.com","subject":"Re: [PATCH/RFC 2/3] object-store: avoid extra ';' from KHASH_INIT","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-09T15:53:35Z","receivedAt":"2021-08-09T15:53:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Carlo Marcelo Arenas Belón writes:\n\n> cf2dc1c238 (speed up alt_odb_usable() with many alternates, 2021-07-07)\n> introduces a KHASH_INIT invocation with a trailing ';', which while\n> commonly expected will trigger warnings with pedantic on both\n> clang[-Wextra-semi] and gcc[-Wpedantic], because that macro has already\n> a semicolon and is meant to be invoked without one.\n>\n> while fixing the macro would be a worthy solution (specially considering\n> this is a common recurring problem), remove the extra ';' for now to\n> minimize churn.\n\nThanks. I fully agree with the reasoning.\n\n>\n> Signed-off-by: Carlo Marcelo Arenas Belón \n> ---\n> object-store.h | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/object-store.h b/object-store.h\n> index e679acc4c3..d24915ced1 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -34,7 +34,7 @@ struct object_directory {\n> };\n> \n> KHASH_INIT(odb_path_map, const char * /* key: odb_path */,\n> -\tstruct object_directory *, 1, fspathhash, fspatheq);\n> +\tstruct object_directory *, 1, fspathhash, fspatheq)\n> \n> void prepare_alt_odb(struct repository *r);\n> char *compute_alternate_path(const char *path, struct strbuf *err);\n"},{"id":"432311","messageId":"xmqqtujyftzx.fsf@gitster.g","threadId":"55998","inReplyTo":"20210809013833.58110-1-carenas@gmail.com","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-09T16:44:18Z","receivedAt":"2021-08-09T16:44:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Carlo Marcelo Arenas Belón writes:\n\n> Building next with pedantic enabled shows the following 2 issues that\n> were originally in ew/many-alternate-optim, apologies for not catching\n> them earlier.\n\nThis of course affects 'master'.\n\nThe first two look trivially correct and I am tempted to take them\nin -rc2; the last one, from my cursory look, I didn't see anything\nwrong in it, but is not all that urgent, either.\n\n> the second one could be skipped, and has indeed another similar case\n> already in seen which will be send separately.\n\nThanks.\n"},{"id":"432331","messageId":"20210809201050.GA8077@dcvr","threadId":"55998","inReplyTo":"xmqqtujyftzx.fsf@gitster.g","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-08-09T20:10:50Z","receivedAt":"2021-08-09T20:10:52Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Carlo Marcelo Arenas Belón writes:\n> \n> > Building next with pedantic enabled shows the following 2 issues that\n> > were originally in ew/many-alternate-optim, apologies for not catching\n> > them earlier.\n> \n> This of course affects 'master'.\n> \n> The first two look trivially correct and I am tempted to take them\n> in -rc2; the last one, from my cursory look, I didn't see anything\n> wrong in it, but is not all that urgent, either.\n\nAgreed on all counts.\n\nI've been starting to think oidtree/cbtree would be better\ndone as BSD-style macro-defined functions (similar to how\nkhash.h is, or sys/{queue,tree}.h on *BSD systems).\n\nI prefer Linux(kernel)-style container_of generics since I find\nthem easier-to-follow and have extra type-checking, but with a\nflex-array it's not pedantically correct.\n\nSo I guess using CPP like khash does might be a better way to\ngo, here.\n\nThoughts?\n\n\nSide note: I've also been considering Perl as a more powerful\nCPP replacement so I could use the same code for a persistent\non-disk store (it would be easier to swap in pread/mmap use).\nAn on-disk format could make it good for refs and pre-packed\nobject storage (perhaps replacing loose objects).\n"},{"id":"432341","messageId":"CAPUEspjtOzc-kis1P-3nhGXQR20vVP+ny2-o_BojG8mx2fo+Jg@mail.gmail.com","threadId":"55998","inReplyTo":"c903b477-8438-7c9a-bbbc-ec87d1b05451@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T22:03:54Z","receivedAt":"2021-08-09T22:04:09Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Mon, Aug 9, 2021 at 3:50 AM Bagas Sanjaya wrote:\n>\n> On 09/08/21 08.38, Carlo Marcelo Arenas Belón wrote:\n> > add a dockerized build using fedora (that usually has the latest gcc)\n> > to be ahead of the curve and avoid older ISO C issues at the same time.\n>\n> But from GCC manual [1], the default C dialect used is `-std=gnu17`,\n> while `-pedantic` is only relevant for ISO C (such as `-std=c17`).\n\nsorry about that, my comment was confusing\n\nI only meant to imply that newer compilers were not throwing any more\nwarnings than the ones that were fixed unlike what you would get if\nusing older compilers or targeting an older standard. This implies that\nit will likely not have many false positives and the few breaks that would\ncome with newer compiled might be worth investigating or adding to the\nignore list.\n\na strict C89 compiler won't even build (ex: inline is a gnu extension\nand the codebase has\nbeen adding those officially since fe9dc6b08c (Merge branch\n'jc/post-c89-rules-doc', 2019-07-25))\n\nand so the pedantic check implied you would target at least gnu89 and\ngenerate lots of warnings (so don't expect to build with DEVELOPER=1\nthat adds -Werror)\n\nare you suggesting we need a more aggresive target like strict C99? at\nleast gcc 11.2.0\nseems to be able to still build next without warnings.\n\n> And why not using `-pedantic-errors`, so that non-ISO features are\n> treated as errors?\n\nwarnings are already treated as errors, if you want to see all\nwarnings need DEVOPTS=\"no-error pedantic\"\n\n> Newcomers contributing to Git may think that based on what our CI do,\n> they can submit patches with C17 features (perhaps with GNU extensions).\n> Then at some time there is casual users that complain that Git doesn't\n> compile with their default older compiler (maybe they run LTS\n> distributions or pre-C17 compiler). Thus we want Git to be compiled\n> successfully using wide variety of compilers (maybe as old as GCC 4.8).\n\nthe codebase was meant to be C89 compatible (as described in\nDocumentation/CodingGuidelines).\n\ngcc-4 is a good target because AFAIK was the last one that defaulted\nto gnu89 mode\nand was also used as the system compiler for several really old\nsystems that still have support.\n\nI tested with 4.9.4, which was the oldest I could get a hold off from\ngcc's docker hub, but I suspect\nwill work the same in that old gcc from centos or debian as well.\n\nCarlo\n"},{"id":"432346","messageId":"CAPUEspgh54AywgqMuOXJf5uPZdR2AN9JLrzJwcOtoec7sRnN7w@mail.gmail.com","threadId":"55998","inReplyTo":"1b096830-3e01-efbe-25dc-c0505c8bac7b@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-09T22:48:48Z","receivedAt":"2021-08-09T22:49:05Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Mon, Aug 9, 2021 at 7:56 AM Phillip Wood wrote:\n>\n> Totally unrelated to this patch but while looking at the ci scripts I\n> noticed that we only run the linux-gcc-4.8 job on travis, not on github.\n\nit is actually related and part of the reason why I sent this as an RFC.\ntravis[1] itself is not running, probably because it broke when\ntravis-ci.org was\nshutdown some time ago.\n\nmaybe wasn't as useful as a CI job using valuable CPU minutes when it could\nrun in the development environment before the code was submitted? the same\ncould apply to this request if you consider that unlike the other\nsimilar jobs (ex: sparse or \"Static Analysis\")\nthere is no need to install an additional (probably tricky to get tool)\n\nCarlo\n\n[1] https://travis-ci.com/github/git\n"},{"id":"432352","messageId":"YRIZsOaguDW0HaeI@carlos-mbp.lan","threadId":"55998","inReplyTo":"xmqqtujyftzx.fsf@gitster.g","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-08-10T06:16:16Z","receivedAt":"2021-08-10T06:16:22Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"Thanks,\n\nin the discussion above René[1] proposed a fix for UBsan issues that were\nreported and that it is still missing.\n\nmy version of it didn't require the extra 4 bytes or showed issues with\nnotes so is probably incomplete and should be replaced from the original\nif possible, but follows below:\n\nCarlo\n\n[1] https://lore.kernel.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/\n\n+CC René for advise \n--- >8 ---\nDate: Sun, 8 Aug 2021 20:45:56 -0700\nSubject: [PATCH] build: fixes for SANITIZE=undefined (WIP)\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nmostly from instructions/code provided by René in :\n\n https://lore.kernel.org/git/20210807224957.GA5068@dcvr/\n\ntested with Xcode in macOS 11.5.1 (x86_64)\n---\n hash.h | 2 +-\n object-file.c | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 27a180248f..3127ba1ef8 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -115,7 +115,7 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\n-\tint algo;\n+\tuint8_t algo;\n };\n \n /* A suitably aligned type for stack allocations of hash contexts. */\ndiff --git a/object-file.c b/object-file.c\nindex 374f3c26bf..2fa282a9b4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2406,7 +2406,7 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n \tsize_t word_index = subdir_nr / word_bits;\n-\tsize_t mask = 1 << (subdir_nr % word_bits);\n+\tsize_t mask = 1U << (subdir_nr % word_bits);\n \tuint32_t *bitmap;\n \n \tif (subdir_nr < 0 ||\n-- \n2.33.0.rc1.379.g2890ef5eb6\n\n"},{"id":"432382","messageId":"6fac977d-80c5-d123-d232-2df5e5966c04@gmail.com","threadId":"55998","inReplyTo":"CAPUEspgh54AywgqMuOXJf5uPZdR2AN9JLrzJwcOtoec7sRnN7w@mail.gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-08-10T15:24:07Z","receivedAt":"2021-08-10T15:24:17Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 09/08/2021 23:48, Carlo Arenas wrote:\n> On Mon, Aug 9, 2021 at 7:56 AM Phillip Wood wrote:\n>>\n>> Totally unrelated to this patch but while looking at the ci scripts I\n>> noticed that we only run the linux-gcc-4.8 job on travis, not on github.\n> \n> it is actually related and part of the reason why I sent this as an RFC.\n> travis[1] itself is not running, probably because it broke when\n> travis-ci.org was\n> shutdown some time ago.\n> \n> maybe wasn't as useful as a CI job using valuable CPU minutes when it could\n> run in the development environment before the code was submitted? the same\n> could apply to this request if you consider that unlike the other\n> similar jobs (ex: sparse or \"Static Analysis\")\n> there is no need to install an additional (probably tricky to get tool)\n\nI think there is value in running the CI jobs with -Wpedantic otherwise \nwe'll continually fixing patches up after they've been merged, I just \nwonder if we need a separate job to do it. We could export \nDEVOPTS=pedantic in ci/build-and-run-tests.sh or change config.mak.dev \nto turn on -Wpedantic with DEVELOPER=1. Having said all that your commit \nmessage also mentioned using a recent compiler to pick up any problem \nearly, I'm not sure how common that is but perhaps that makes a new job \nworth it. If so there is a gcc docker image[1] which always has the \nlatest compiler.\n\nBest Wishes\n\nPhillip\n\n[1] https://hub.docker.com/_/gcc\n\n> Carlo\n> \n> [1] https://travis-ci.com/github/git\n> \n"},{"id":"432391","messageId":"xmqqeeb1dumx.fsf@gitster.g","threadId":"55998","inReplyTo":"6fac977d-80c5-d123-d232-2df5e5966c04@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-10T18:25:42Z","receivedAt":"2021-08-10T18:25:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I think there is value in running the CI jobs with -Wpedantic\n> otherwise we'll continually fixing patches up after they've been\n> merged, I just wonder if we need a separate job to do it.\n\nSeeing https://github.com/git/git/actions/runs/1114402527 for\nexample, I notice that we run three dockerized jobs and this one\ntakes about 3 minutes to compile everything. The other two take\nabout 9 minutes and compile and run tests.\n\nShouldn't we be running tests in this job, too? I know the new\ncomment in ci/run-build-and-tests.sh says \"Don't run the tests; we\nonly care about ...\", but I do not see a reason to declare that we\ndo not care if the resulting binary passes the tests or not.\n\nIt also seems that the environment does not have an access to any\nworking \"git\" binary and \"save_good_tree\" helper seems to fail\nbecause of it.\n\nhttps://github.com/git/git/runs/3284998605?check_suite_focus=true#step:5:692\n\n\nI wonder if this untested patch on top of the patch under discussion\nis sufficient, if we wanted to also run tests on the fedora image?\n\n\ndiff --git c/ci/install-docker-dependencies.sh w/ci/install-docker-dependencies.sh\nindex 07a8c6b199..f139c0632b 100755\n--- c/ci/install-docker-dependencies.sh\n+++ w/ci/install-docker-dependencies.sh\n@@ -17,6 +17,8 @@ linux-musl)\n \t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\tdnf -yq install git make gcc findutils diffutils perl python3 \\\n+\t\tgettext zlib-devel expat-devel openssl-devel \\\n+\t\tcurl-devel pcre2-devel >/dev/null\n \t;;\n esac\ndiff --git c/ci/run-build-and-tests.sh w/ci/run-build-and-tests.sh\nindex f3aba5d6cb..5b2fcf3428 100755\n--- c/ci/run-build-and-tests.sh\n+++ w/ci/run-build-and-tests.sh\n@@ -40,9 +40,10 @@ linux-clang)\n \texport GIT_TEST_DEFAULT_HASH=sha256\n \tmake test\n \t;;\n-linux-gcc-4.8|pedantic)\n+linux-gcc-4.8)\n \t# Don't run the tests; we only care about whether Git can be\n-\t# built with GCC 4.8 or with pedantic\n+\t# built with GCC 4.8, as it errors out on some undesired (C99)\n+\t# constructs that newer compilers seem to quietly accept.\n \t;;\n *)\n \tmake test\n\n\n\n"},{"id":"432403","messageId":"eaf4eeb8-bd5f-5817-7a07-f72b735e1ad6@web.de","threadId":"55998","inReplyTo":"CAPUEsphf9F1+=zOMKx3j=jH8xqDwQX99+9uHiYUpXhFE1nervg@mail.gmail.com","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-10T18:59:21Z","receivedAt":"2021-08-10T18:59:56Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 09.08.21 um 03:35 schrieb Carlo Arenas:\n> On Sat, Aug 7, 2021 at 3:51 PM Eric Wong wrote:\n>>\n>> René Scharfe wrote:\n>>> Am 06.08.21 um 17:31 schrieb Andrzej Hunt:\n>>>> On 29/06/2021 22:53, Eric Wong wrote:\n>>>>> [...snip...]\n>>>>> diff --git a/oidtree.c b/oidtree.c\n>>>>> new file mode 100644\n>>>>> index 0000000000..c1188d8f48\n>>>>> --- /dev/null\n>>>>> +++ b/oidtree.c\n>>\n>>>>> +struct oidtree_node {\n>>>>> + /* n.k[] is used to store \"struct object_id\" */\n>>>>> + struct cb_node n;\n>>>>> +};\n>>>>> +\n>>>>> [... snip ...]\n>>>>> +\n>>>>> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n>>>>> +{\n>>>>> + struct oidtree_node *on;\n>>>>> +\n>>>>> + if (!ot->mempool)\n>>>>> + ot->mempool = allocate_alloc_state();\n>>>>> + if (!oid->algo)\n>>>>> + BUG(\"oidtree_insert requires oid->algo\");\n>>>>> +\n>>>>> + on = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n>>>>> + oidcpy_with_padding((struct object_id *)on->n.k, oid);\n>>>>\n>>>> I think this object_id cast introduced undefined behaviour - here's\n>>>> my layperson's interepretation of what's going on (full UBSAN output\n>>>> is pasted below):\n>>>>\n>>>> cb_node.k is a uint8_t[], and hence can be 1-byte aligned (on my\n>>>> machine: offsetof(struct cb_node, k) == 21). We're casting its\n>>>> pointer to \"struct object_id *\", and later try to access\n>>>> object_id.hash within oidcpy_with_padding. My compiler assumes that\n>>>> an object_id pointer needs to be 4-byte aligned, and reading from a\n>>>> misaligned pointer means we hit undefined behaviour. (I think the\n>>>> 4-byte alignment requirement comes from the fact that object_id's\n>>>> largest member is an int?)\n>>\n>> I seem to recall struct alignment requirements being\n>> architecture-dependent; and x86/x86-64 are the most liberal\n>> w.r.t alignment requirements.\n>\n> I think the problem here is not the alignment though, but the fact that\n> the nesting of structs with flexible arrays is forbidden by ISO/IEC\n> 9899:2011 6.7.2.1¶3 that reads :\n>\n> 6.7.2.1 Structure and union specifiers\n>\n> ¶3 A structure or union shall not contain a member with incomplete or\n> function type (hence, a structure shall not contain an instance of\n> itself, but may contain a pointer to an instance of itself), except\n> that the last member of a structure with more than one named member\n> may have incomplete array type; such a structure (and any union\n> containing, possibly recursively, a member that is such a structure)\n> shall not be a member of a structure or an element of an array.\n>\n> and it will throw a warning with clang 12\n> (-Wflexible-array-extensions) or gcc 11 (-Werror=pedantic) when using\n> DEVOPTS=pedantic\n\nThat's an additional problem. UBSan still reports the alignment error\nwith your patches.\n\nRené\n"},{"id":"432406","messageId":"0b973579-748e-ce2f-20aa-a967765cce83@web.de","threadId":"55998","inReplyTo":"YRIZsOaguDW0HaeI@carlos-mbp.lan","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-10T19:30:34Z","receivedAt":"2021-08-10T19:30:44Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 10.08.21 um 08:16 schrieb Carlo Marcelo Arenas Belón:\n> Thanks,\n>\n> in the discussion above René[1] proposed a fix for UBsan issues that were\n> reported and that it is still missing.\n>\n> my version of it didn't require the extra 4 bytes or showed issues with\n> notes so is probably incomplete and should be replaced from the original\n> if possible, but follows below:\n\nWith your three patches plus the one below t3301-notes.sh and several more\nfail on an Apple M1. Adding an unused int member to struct leaf_node fixes\nthat. I didn't dig deeper into the notes code to understand the actual\nissue, though.\n\n>\n> Carlo\n>\n> [1] https://lore.kernel.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/\n>\n> +CC René for advise\n> --- >8 ---\n> Date: Sun, 8 Aug 2021 20:45:56 -0700\n> Subject: [PATCH] build: fixes for SANITIZE=undefined (WIP)\n> MIME-Version: 1.0\n> Content-Type: text/plain; charset=UTF-8\n> Content-Transfer-Encoding: 8bit\n>\n> mostly from instructions/code provided by René in :\n>\n> https://lore.kernel.org/git/20210807224957.GA5068@dcvr/\n>\n> tested with Xcode in macOS 11.5.1 (x86_64)\n> ---\n> hash.h | 2 +-\n> object-file.c | 2 +-\n> 2 files changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/hash.h b/hash.h\n> index 27a180248f..3127ba1ef8 100644\n> --- a/hash.h\n> +++ b/hash.h\n> @@ -115,7 +115,7 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n>\n> struct object_id {\n> \tunsigned char hash[GIT_MAX_RAWSZ];\n> -\tint algo;\n> +\tuint8_t algo;\n> };\n>\n> /* A suitably aligned type for stack allocations of hash contexts. */\n> diff --git a/object-file.c b/object-file.c\n> index 374f3c26bf..2fa282a9b4 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2406,7 +2406,7 @@ struct oidtree *odb_loose_cache(struct object_directory *odb,\n> \tstruct strbuf buf = STRBUF_INIT;\n> \tsize_t word_bits = bitsizeof(odb->loose_objects_subdir_seen[0]);\n> \tsize_t word_index = subdir_nr / word_bits;\n> -\tsize_t mask = 1 << (subdir_nr % word_bits);\n> +\tsize_t mask = 1U << (subdir_nr % word_bits);\n> \tuint32_t *bitmap;\n>\n> \tif (subdir_nr < 0 ||\n>\n\nThe first hunk is about alignment (and missing the notes fix, as mentioned).\nThe second hunk is about shifting a signed 32-bit value 31 places to the\nleft, which is undefined (because technically there are only 31 value bits).\nThose are different issues and they should be addressed by separate patches,\nI think. That's why I submitted a patch for the the second one in\nhttp://public-inbox.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/.\n\nRené\n"},{"id":"432407","messageId":"d7e11b65-cd20-6a0f-7440-6003578aa680@web.de","threadId":"55998","inReplyTo":"20210807224957.GA5068@dcvr","subject":"Re: [PATCH v2 5/5] oidtree: a crit-bit tree for odb_loose_cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-10T19:40:45Z","receivedAt":"2021-08-10T19:41:05Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 08.08.21 um 00:49 schrieb Eric Wong:\n> René Scharfe wrote:\n>> Am 06.08.21 um 17:31 schrieb Andrzej Hunt:\n>>> On 29/06/2021 22:53, Eric Wong wrote:\n>>>> [...snip...]\n>>>> diff --git a/oidtree.c b/oidtree.c\n>>>> new file mode 100644\n>>>> index 0000000000..c1188d8f48\n>>>> --- /dev/null\n>>>> +++ b/oidtree.c\n>\n>>>> +struct oidtree_node {\n>>>> +    /* n.k[] is used to store \"struct object_id\" */\n>>>> +    struct cb_node n;\n>>>> +};\n>>>> +\n>>>> [... snip ...]\n>>>> +\n>>>> +void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n>>>> +{\n>>>> +    struct oidtree_node *on;\n>>>> +\n>>>> +    if (!ot->mempool)\n>>>> +        ot->mempool = allocate_alloc_state();\n>>>> +    if (!oid->algo)\n>>>> +        BUG(\"oidtree_insert requires oid->algo\");\n>>>> +\n>>>> +    on = alloc_from_state(ot->mempool, sizeof(*on) + sizeof(*oid));\n>>>> +    oidcpy_with_padding((struct object_id *)on->n.k, oid);\n>>>\n>>> I think this object_id cast introduced undefined behaviour - here's\n>>> my layperson's interepretation of what's going on (full UBSAN output\n>>> is pasted below):\n>>>\n>>> cb_node.k is a uint8_t[], and hence can be 1-byte aligned (on my\n>>> machine: offsetof(struct cb_node, k) == 21). We're casting its\n>>> pointer to \"struct object_id *\", and later try to access\n>>> object_id.hash within oidcpy_with_padding. My compiler assumes that\n>>> an object_id pointer needs to be 4-byte aligned, and reading from a\n>>> misaligned pointer means we hit undefined behaviour. (I think the\n>>> 4-byte alignment requirement comes from the fact that object_id's\n>>> largest member is an int?)\n>\n> I seem to recall struct alignment requirements being\n> architecture-dependent; and x86/x86-64 are the most liberal\n> w.r.t alignment requirements.\n>\n>>> I'm not sure what an elegant and idiomatic fix might be - IIUC it's\n>>> hard to guarantee misaligned access can't happen with a flex array\n>>> that's being used for arbitrary data (you would presumably have to\n>>> declare it as an array of whatever the largest supported type is, so\n>>> that you can guarantee correct alignment even when cbtree is used\n>>> with that type) - which might imply that k needs to be declared as a\n>>> void pointer? That in turn would make cbtree.c harder to read.\n>>\n>> C11 has alignas. We could also make the member before the flex array,\n>> otherbits, wider, e.g. promote it to uint32_t.\n>\n> Ugh, no. cb_node should be as small as possible and (for our\n> current purposes) ->byte could be uint8_t.\n\nWell, we can make both byte and otherbits uint16_t. That would require\na good comment explaining the reasoning and probably some rework later,\nbut might be the least intrusive solution for now.\n\n>> A more parsimonious solution would be to turn the int member of struct\n>> object_id, algo, into an unsigned char for now and reconsider the issue\n>> once we support our 200th algorithm or so.\n>\n> Yes, making struct object_id smaller would benefit all git users\n> (at least for the next few centuries :P).\n\nTrue, we're currently using 4 bytes to distinguish between SHA-1 and\nSHA-256, i.e. to represent a single bit. Reducing the size of struct\nobject_id from 36 bytes to 33 bytes seems quite significant.\n\nI don't know how important the 4-byte alignment is, though. cf0983213c\n(hash: add an algo member to struct object_id, 2021-04-26) doesn't\nmention it, but the notes code seems to rely on it -- strange.\n\nOverall this seems to be a good way to go -- after the next release.\n\nRené\n"},{"id":"432427","messageId":"CAPUEspiWdGRQoBnpn_uwjkqV7ffMm+MkzbNVU1rZ6yCwkpmNaA@mail.gmail.com","threadId":"55998","inReplyTo":"0b973579-748e-ce2f-20aa-a967765cce83@web.de","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-10T23:49:36Z","receivedAt":"2021-08-10T23:49:50Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Tue, Aug 10, 2021 at 12:30 PM René Scharfe wrote:\n>\n> Those are different issues and they should be addressed by separate patches,\n> I think. That's why I submitted a patch for the second one in\n> http://public-inbox.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/.\n\nagree, and that is why I mentioned not to merge mine but use your\nwhole series instead when it is published (mine was just a stopgap to\nsee if I could get SANITIZE=undefined to behave meanwhile, but that I\nthought would be worth making public so anyone else affected might\nhave something to start with)\n\nwould at least the two included in the chunks above be safe enough for\nRC2 as I hope?, is the one with the additional int too hacky to be\nconsidered for release?; FWIW hadn't been able to reproduce that issue\nyou reported in t3301 even with an Apple M1 with macOS 11.5.1 (I use\nNO_GETTEXT=1 though, not sure that might be why)\n\nAnything I can help with?\n\nCarlo\n"},{"id":"432430","messageId":"CAPUEsphbV_wGKPZP4T6tSO+qe89xmGpcpPozq+pZkHhsoup2gg@mail.gmail.com","threadId":"55998","inReplyTo":"CAPUEspiWdGRQoBnpn_uwjkqV7ffMm+MkzbNVU1rZ6yCwkpmNaA@mail.gmail.com","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-11T00:57:40Z","receivedAt":"2021-08-11T00:57:53Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Tue, Aug 10, 2021 at 4:49 PM Carlo Arenas wrote:\n> FWIW hadn't been able to reproduce that issue\n> you reported in t3301 even with an Apple M1 with macOS 11.5.1 (I use\n> NO_GETTEXT=1 though, not sure that might be why)\n\nbut it seems to be reproducible in the 32bit linux job from CI[1] :\n\n git: notes.c:206: note_tree_remove: Assertion `GET_PTR_TYPE(entry)\n== 0' failed.\n\nCarlo\n\n[1] https://github.com/carenas/git/runs/3295672049?check_suite_focus=true\n"},{"id":"432486","messageId":"1a18a701-7d14-d6c5-6929-30636e688006@web.de","threadId":"55998","inReplyTo":"CAPUEspiWdGRQoBnpn_uwjkqV7ffMm+MkzbNVU1rZ6yCwkpmNaA@mail.gmail.com","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-11T14:57:52Z","receivedAt":"2021-08-11T14:58:04Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Hi Carlo,\n\nAm 11.08.21 um 01:49 schrieb Carlo Arenas:\n> On Tue, Aug 10, 2021 at 12:30 PM René Scharfe wrote:\n>>\n>> Those are different issues and they should be addressed by separate patches,\n>> I think. That's why I submitted a patch for the second one in\n>> http://public-inbox.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/.\n>\n> agree, and that is why I mentioned not to merge mine but use your\n> whole series instead when it is published (mine was just a stopgap to\n> see if I could get SANITIZE=undefined to behave meanwhile, but that I\n> thought would be worth making public so anyone else affected might\n> have something to start with)\n>\n> would at least the two included in the chunks above be safe enough for\n> RC2 as I hope?, is the one with the additional int too hacky to be\n> considered for release?;\n\nI think your -pedantic fixes 1 and 2 should go into the next possible\nrelease candidate because they fix regressions.\n\nSame for my signed-left-shift fix in\nhttp://public-inbox.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/\n(or some improved version if it's lacking in some way) and the yet to be\npublished fix for the alignment issue. I assume Andrzej as the reporter\nor Eric as the original author would like to have a shot at the latter.\n\n> FWIW hadn't been able to reproduce that issue\n> you reported in t3301 even with an Apple M1 with macOS 11.5.1 (I use\n> NO_GETTEXT=1 though, not sure that might be why)\n\nStrange. I use Apple clang version 12.0.5 (clang-1205.0.22.11) on the\nsame OS.\n\nAt least the reproduction on Linux that you mentioned in your reply\nmeans I can calm down a bit because it's not just a problem on my\nsystem..\n\nThank you,\nRené\n"},{"id":"432497","messageId":"xmqq5ywbaoey.fsf@gitster.g","threadId":"55998","inReplyTo":"1a18a701-7d14-d6c5-6929-30636e688006@web.de","subject":"Re: [PATCH/RFC 0/3] pedantic errors in next","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-11T17:20:37Z","receivedAt":"2021-08-11T17:20:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n>> would at least the two included in the chunks above be safe enough for\n>> RC2 as I hope?, is the one with the additional int too hacky to be\n>> considered for release?;\n>\n> I think your -pedantic fixes 1 and 2 should go into the next possible\n> release candidate because they fix regressions.\n\nI agree. They are at the bottom of 'seen' just above 'master' in\nlast night's pushout for this exact reason.\n\n> Same for my signed-left-shift fix in\n> http://public-inbox.org/git/bab9f889-ee2e-d3c3-0319-e297b59261a0@web.de/\n> (or some improved version if it's lacking in some way) and the yet to be\n> published fix for the alignment issue. I assume Andrzej as the reporter\n> or Eric as the original author would like to have a shot at the latter.\n\nThanks. It was missed as it was buried in the discussion exchange.\nWill queue together with cb/many-alternate-optim-fixup topic.\n\n"},{"id":"432742","messageId":"9583052d-9181-7532-304a-4bacfb9e1147@web.de","threadId":"55998","inReplyTo":"3cbec773-cd99-cf9f-a713-45ef8e6746c3@ahunt.org","subject":"[PATCH] oidtree: avoid unaligned access to crit-bit tree","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-14T20:00:38Z","receivedAt":"2021-08-14T20:01:18Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"The flexible array member \"k\" of struct cb_node is used to store the key\nof the crit-bit tree node. It offers no alignment guarantees -- in fact\nthe current struct layout puts it one byte after a 4-byte aligned\naddress, i.e. guaranteed to be misaligned.\n\noidtree uses a struct object_id as cb_node key. Since cf0983213c (hash:\nadd an algo member to struct object_id, 2021-04-26) it requires 4-byte\nalignment. The mismatch is reported by UndefinedBehaviorSanitizer at\nruntime like this:\n\nhash.h:277:2: runtime error: member access within misaligned address 0x00015000802d for type 'struct object_id', which requires 4 byte alignment\n0x00015000802d: note: pointer points here\n 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00\n ^\nSUMMARY: UndefinedBehaviorSanitizer: undefined-behavior hash.h:277:2 in\n\nWe can fix that by:\n\n1. eliminating the alignment requirement of struct object_id,\n2. providing the alignment in struct cb_node, or\n3. avoiding the issue by only using memcpy to access \"k\".\n\nCurrently we only store one of two values in \"algo\" in struct object_id.\nWe could use a uint8_t for that instead and widen it only once we add\nsupport for our twohundredth algorithm or so. That would not only avoid\nalignment issues, but also reduce the memory requirements for each\ninstance of struct object_id by ca. 9%.\n\nSupporting keys with alignment requirements might be useful to spread\nthe use of crit-bit trees. It can be achieved by using a wider type for\n\"k\" (e.g. uintmax_t), using different types for the members \"byte\" and\n\"otherbits\" (e.g. uint16_t or uint32_t for each), or by avoiding the use\nof flexible arrays like khash.h does.\n\nThis patch implements the third option, though, because it has the least\npotential for causing side-effects and we're close to the next release.\nIf one of the other options is implemented later as well to get their\nadditional benefits we can get rid of the extra copies introduced here.\n\nReported-by: Andrzej Hunt \nSigned-off-by: René Scharfe \n---\n cbtree.h | 2 +-\n hash.h | 2 +-\n oidtree.c | 20 +++++++++++++++-----\n 3 files changed, 17 insertions(+), 7 deletions(-)\n\ndiff --git a/cbtree.h b/cbtree.h\nindex fe4587087e..a04a312c3f 100644\n--- a/cbtree.h\n+++ b/cbtree.h\n@@ -25,7 +25,7 @@ struct cb_node {\n \t */\n \tuint32_t byte;\n \tuint8_t otherbits;\n-\tuint8_t k[FLEX_ARRAY]; /* arbitrary data */\n+\tuint8_t k[FLEX_ARRAY]; /* arbitrary data, unaligned */\n };\n\n struct cb_tree {\ndiff --git a/hash.h b/hash.h\nindex 27a180248f..9e25c40e9a 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -115,7 +115,7 @@ static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *s\n\n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\n-\tint algo;\n+\tint algo;\t/* XXX requires 4-byte alignment */\n };\n\n /* A suitably aligned type for stack allocations of hash contexts. */\ndiff --git a/oidtree.c b/oidtree.c\nindex 580cab8ae2..0d39389bee 100644\n--- a/oidtree.c\n+++ b/oidtree.c\n@@ -31,12 +31,19 @@ void oidtree_clear(struct oidtree *ot)\n void oidtree_insert(struct oidtree *ot, const struct object_id *oid)\n {\n \tstruct cb_node *on;\n+\tstruct object_id k;\n\n \tif (!oid->algo)\n \t\tBUG(\"oidtree_insert requires oid->algo\");\n\n \ton = mem_pool_alloc(&ot->mem_pool, sizeof(*on) + sizeof(*oid));\n-\toidcpy_with_padding((struct object_id *)on->k, oid);\n+\n+\t/*\n+\t * Clear the padding and copy the result in separate steps to\n+\t * respect the 4-byte alignment needed by struct object_id.\n+\t */\n+\toidcpy_with_padding(&k, oid);\n+\tmemcpy(on->k, &k, sizeof(k));\n\n \t/*\n \t * n.b. Current callers won't get us duplicates, here. If a\n@@ -68,17 +75,20 @@ int oidtree_contains(struct oidtree *ot, const struct object_id *oid)\n static enum cb_next iter(struct cb_node *n, void *arg)\n {\n \tstruct oidtree_iter_data *x = arg;\n-\tconst struct object_id *oid = (const struct object_id *)n->k;\n+\tstruct object_id k;\n+\n+\t/* Copy to provide 4-byte alignment needed by struct object_id. */\n+\tmemcpy(&k, n->k, sizeof(k));\n\n-\tif (x->algo != GIT_HASH_UNKNOWN && x->algo != oid->algo)\n+\tif (x->algo != GIT_HASH_UNKNOWN && x->algo != k.algo)\n \t\treturn CB_CONTINUE;\n\n \tif (x->last_nibble_at) {\n-\t\tif ((oid->hash[*x->last_nibble_at] ^ x->last_byte) & 0xf0)\n+\t\tif ((k.hash[*x->last_nibble_at] ^ x->last_byte) & 0xf0)\n \t\t\treturn CB_CONTINUE;\n \t}\n\n-\treturn x->fn(oid, x->arg);\n+\treturn x->fn(&k, x->arg);\n }\n\n void oidtree_each(struct oidtree *ot, const struct object_id *oid,\n--\n2.32.0\n\n"},{"id":"432833","messageId":"xmqqv945qk5o.fsf@gitster.g","threadId":"55998","inReplyTo":"9583052d-9181-7532-304a-4bacfb9e1147@web.de","subject":"Re: [PATCH] oidtree: avoid unaligned access to crit-bit tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-16T19:11:47Z","receivedAt":"2021-08-16T19:12:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> The flexible array member \"k\" of struct cb_node is used to store the key\n> of the crit-bit tree node. It offers no alignment guarantees -- in fact\n> the current struct layout puts it one byte after a 4-byte aligned\n> address, i.e. guaranteed to be misaligned.\n>\n> oidtree uses a struct object_id as cb_node key. Since cf0983213c (hash:\n> add an algo member to struct object_id, 2021-04-26) it requires 4-byte\n> alignment. The mismatch is reported by UndefinedBehaviorSanitizer at\n> runtime like this:\n>\n> hash.h:277:2: runtime error: member access within misaligned address 0x00015000802d for type 'struct object_id', which requires 4 byte alignment\n> 0x00015000802d: note: pointer points here\n> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00\n> ^\n> SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior hash.h:277:2 in\n>\n> We can fix that by:\n>\n> 1. eliminating the alignment requirement of struct object_id,\n> 2. providing the alignment in struct cb_node, or\n> 3. avoiding the issue by only using memcpy to access \"k\".\n>\n> Currently we only store one of two values in \"algo\" in struct object_id.\n> We could use a uint8_t for that instead and widen it only once we add\n> support for our twohundredth algorithm or so. That would not only avoid\n> alignment issues, but also reduce the memory requirements for each\n> instance of struct object_id by ca. 9%.\n>\n> Supporting keys with alignment requirements might be useful to spread\n> the use of crit-bit trees. It can be achieved by using a wider type for\n> \"k\" (e.g. uintmax_t), using different types for the members \"byte\" and\n> \"otherbits\" (e.g. uint16_t or uint32_t for each), or by avoiding the use\n> of flexible arrays like khash.h does.\n>\n> This patch implements the third option, though, because it has the least\n> potential for causing side-effects and we're close to the next release.\n> If one of the other options is implemented later as well to get their\n> additional benefits we can get rid of the extra copies introduced here.\n>\n> Reported-by: Andrzej Hunt \n> Signed-off-by: René Scharfe \n> ---\n> cbtree.h | 2 +-\n> hash.h | 2 +-\n> oidtree.c | 20 +++++++++++++++-----\n> 3 files changed, 17 insertions(+), 7 deletions(-)\n\nThanks. Among the choices you considered (and I agree that each of\nthem is a solution that goes in a reasonable direction), the one\nchosen here certainly is the least risky one.\n\n"},{"id":"434063","messageId":"87zgszxirn.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"1b096830-3e01-efbe-25dc-c0505c8bac7b@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-30T11:36:50Z","receivedAt":"2021-08-30T11:40:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 09 2021, Phillip Wood wrote:\n\n> Hi Carlo\n>\n> On 09/08/2021 02:38, Carlo Marcelo Arenas Belón wrote:\n>> similar to the recently added sparse task, it is nice to know as early\n>> as possible.\n>> add a dockerized build using fedora (that usually has the latest\n>> gcc)\n>> to be ahead of the curve and avoid older ISO C issues at the same time.\n>\n> If we want to be able to compile with -Wpedantic then it might be\n> better just to turn it on unconditionally in config.mak.dev. Then\n> developers will see any errors before they push and the ci builds will\n> all use it rather than having to run an extra job. I had a quick scan\n> of the mail archive threads starting at [1,2] and it's not clear to me\n> why -Wpedaintic was added as an optional extra.\n\nThis is from wetware memory, so maybe it's wrong: But I recall that with\nDEVOPTS=pedantic we used to have a giant wall of warnings not too long\nago (i.e. 1-3 years), and not just that referenced\nUSE_PARENS_AROUND_GETTEXT_N issue.\n\nSo yeah, I take and agree with your point that perhaps we should turn\nthis on by default for DEVELOPER if that's not the case.\n\nOn the other hand we can't combine that with\nUSE_PARENS_AROUND_GETTEXT_N, and to the extent that we think DEVELOPER\nis useful, the entire point of having USE_PARENS_AROUND_GETTEXT_N seems\nto be to catch exactly that sort of in-development issue.\n\nSo if we turn pedantic on in DEVOPTS by default, wouldn't it make sense\nto at least have a CI job where we test that we compile with\nUSE_PARENS_AROUND_GETTEXT_N (which at that point would no be the default\nanymore).\n\nOr maybe the existing CI config matrix would already cover that,\ni.e. we've got some entry point to it that doesn't go through\nci/lib.sh's DEVELOPER=1 that I've missed, if so nevermind the last two\nparagraphs (three, including this one).\n"},{"id":"434064","messageId":"87wno3xioa.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-4-carenas@gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-30T11:40:53Z","receivedAt":"2021-08-30T11:42:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Aug 08 2021, Carlo Marcelo Arenas Belón wrote:\n\n> -linux-gcc-4.8)\n> +linux-gcc-4.8|pedantic)\n> \t# Don't run the tests; we only care about whether Git can be\n> -\t# built with GCC 4.8, as it errors out on some undesired (C99)\n> -\t# constructs that newer compilers seem to quietly accept.\n> +\t# built with GCC 4.8 or with pedantic\n> \t;;\n> *)\n> \tmake test\n\nAside from Junio's suggested squash in \ndownthread, which would obsolete this comment:\n\nI think this would be clearer by not combining these two, i.e. just:\n\n linux-gcc-4.8)\n # \n ;;\n pedantic)\n # \n ;;\n\nWe'll surely eventually end up with not just one, but N setups that want\nto compile-only, so not having to reword one big comment referring to\nthem all when we do so leads to less churn...\n"},{"id":"434333","messageId":"CAPUEspj43=z8nSdh8UAiqZ+UR8UAZkSsQr1WviGtasQ7d-fHTQ@mail.gmail.com","threadId":"55998","inReplyTo":"87zgszxirn.fsf@evledraar.gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-31T20:28:19Z","receivedAt":"2021-08-31T20:28:34Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Mon, Aug 30, 2021 at 4:40 AM Ævar Arnfjörð Bjarmason\n wrote:\n> On Mon, Aug 09 2021, Phillip Wood wrote:\n> > On 09/08/2021 02:38, Carlo Marcelo Arenas Belón wrote:\n> >> similar to the recently added sparse task, it is nice to know as early\n> >> as possible.\n> >> add a dockerized build using fedora (that usually has the latest\n> >> gcc)\n> >> to be ahead of the curve and avoid older ISO C issues at the same time.\n> >\n> > If we want to be able to compile with -Wpedantic then it might be\n> > better just to turn it on unconditionally in config.mak.dev. Then\n> > developers will see any errors before they push and the ci builds will\n> > all use it rather than having to run an extra job. I had a quick scan\n> > of the mail archive threads starting at [1,2] and it's not clear to me\n> > why -Wpedaintic was added as an optional extra.\n>\n> This is from wetware memory, so maybe it's wrong: But I recall that with\n> DEVOPTS=pedantic we used to have a giant wall of warnings not too long\n> ago (i.e. 1-3 years), and not just that referenced\n> USE_PARENS_AROUND_GETTEXT_N issue.\n\nwhen gcc (and clang) moved to target C99 by default (after version 5)\nthen that wall of errors went away. Indeed git can build cleanly in a\nstrict C99 compiler and until reftable was able to build even with gcc\n2.95.3\n\nthe nostalgic can get it back with `CC=gcc -std=gnu89`, and indeed I\nwas considering this might be a good alternative to the defunct\ngcc-4.8 job, where the weather balloons breaking with strict C89\ncompatibility could be explicitly coded.\n\n> So if we turn pedantic on in DEVOPTS by default, wouldn't it make sense\n> to at least have a CI job where we test that we compile with\n> USE_PARENS_AROUND_GETTEXT_N (which at that point would not be the default\n> anymore).\n\nagree, and indeed was thinking it might be worth combining this job\nwith the SANITIZE one for efficiency.\n\nCarlo\n"},{"id":"434364","messageId":"87bl5dwcx0.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"CAPUEspj43=z8nSdh8UAiqZ+UR8UAZkSsQr1WviGtasQ7d-fHTQ@mail.gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T20:51:55Z","receivedAt":"2021-08-31T20:57:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 31 2021, Carlo Arenas wrote:\n\n> On Mon, Aug 30, 2021 at 4:40 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>> On Mon, Aug 09 2021, Phillip Wood wrote:\n>> > On 09/08/2021 02:38, Carlo Marcelo Arenas Belón wrote:\n>> >> similar to the recently added sparse task, it is nice to know as early\n>> >> as possible.\n>> >> add a dockerized build using fedora (that usually has the latest\n>> >> gcc)\n>> >> to be ahead of the curve and avoid older ISO C issues at the same time.\n>> >\n>> > If we want to be able to compile with -Wpedantic then it might be\n>> > better just to turn it on unconditionally in config.mak.dev. Then\n>> > developers will see any errors before they push and the ci builds will\n>> > all use it rather than having to run an extra job. I had a quick scan\n>> > of the mail archive threads starting at [1,2] and it's not clear to me\n>> > why -Wpedaintic was added as an optional extra.\n>>\n>> This is from wetware memory, so maybe it's wrong: But I recall that with\n>> DEVOPTS=pedantic we used to have a giant wall of warnings not too long\n>> ago (i.e. 1-3 years), and not just that referenced\n>> USE_PARENS_AROUND_GETTEXT_N issue.\n>\n> when gcc (and clang) moved to target C99 by default (after version 5)\n> then that wall of errors went away. Indeed git can build cleanly in a\n> strict C99 compiler and until reftable was able to build even with gcc\n> 2.95.3\n>\n> the nostalgic can get it back with `CC=gcc -std=gnu89`, and indeed I\n> was considering this might be a good alternative to the defunct\n> gcc-4.8 job, where the weather balloons breaking with strict C89\n> compatibility could be explicitly coded.\n>\n>> So if we turn pedantic on in DEVOPTS by default, wouldn't it make sense\n>> to at least have a CI job where we test that we compile with\n>> USE_PARENS_AROUND_GETTEXT_N (which at that point would not be the default\n>> anymore).\n>\n> agree, and indeed was thinking it might be worth combining this job\n> with the SANITIZE one for efficiency.\n\nOn the other hand maybe we should just remove\nUSE_PARENS_AROUND_GETTEXT_N entirely, i.e. always use the parens.\n\nThat facility seems to have been added in response to a one-off mistake\nin 9c9b4f2f8b7 (standardize usage info string format, 2015-01-13). See\nhttps://lore.kernel.org/git/ecb18f9d6ac56da0a61c3b98f8f2236@74d39fa044aa309eaea14b9f57fe79c/. That\nlater landed as 290c8e7a3fe (gettext.h: add parentheses around N_\nexpansion if supported, 2015-01-11).\n\nIt doesn't seem worth the effort to forever maintain this special case\nand use CI resources etc. to catch what was effectively a one-off typo.\n"},{"id":"434371","messageId":"CAPUEsphbUHgt09A2qdy-5U0L1y13pkeNjBEUbjkE6JOPeqDVMA@mail.gmail.com","threadId":"55998","inReplyTo":"87bl5dwcx0.fsf@evledraar.gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-08-31T23:54:52Z","receivedAt":"2021-08-31T23:55:07Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Tue, Aug 31, 2021 at 1:57 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On the other hand maybe we should just remove\n> USE_PARENS_AROUND_GETTEXT_N entirely, i.e. always use the parens.\n\nthat would break pedantic in all versions of gcc since it is a GNU\nextension and is not valid in any C standard.\n(unlike the ones we are using with weather balloons and that are valid C99)\n\nthe C standard says arrays can be initialized by a string literal\n(obviously with quotes) and allows only optional {} which would avoid\nthe accidental concatenation that triggered this, but can't be used as\nan alternative.\n\n> It doesn't seem worth the effort to forever maintain this special case\n> and use CI resources etc. to catch what was effectively a one-off typo.\n\nunder that argument, removing this safeguard might be also possible.\n\nCarlo\n"},{"id":"434375","messageId":"YS7c3169x5Wk4PlA@coredump.intra.peff.net","threadId":"55998","inReplyTo":"CAPUEsphbUHgt09A2qdy-5U0L1y13pkeNjBEUbjkE6JOPeqDVMA@mail.gmail.com","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T01:52:31Z","receivedAt":"2021-09-01T01:52:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 04:54:52PM -0700, Carlo Arenas wrote:\n\n> On Tue, Aug 31, 2021 at 1:57 PM Ævar Arnfjörð Bjarmason\n> wrote:\n> >\n> > On the other hand maybe we should just remove\n> > USE_PARENS_AROUND_GETTEXT_N entirely, i.e. always use the parens.\n> \n> that would break pedantic in all versions of gcc since it is a GNU\n> extension and is not valid in any C standard.\n> (unlike the ones we are using with weather balloons and that are valid C99)\n\nI think Ævar might have mis-spoke there. It would make sense to get rid\nof the feature and _never_ use parens, which is always valid C (and does\nnot tickle pedantic, but also does not catch any accidental string\nconcatenation).\n\nThat actually seems quite reasonable to me.\n\nSomething like this, I guess?\n\ndiff --git a/Makefile b/Makefile\nindex d1feab008f..4936e234bc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -409,15 +409,6 @@ all::\n # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version\n # before 2.17) for clock_gettime and CLOCK_MONOTONIC.\n #\n-# Define USE_PARENS_AROUND_GETTEXT_N to \"yes\" if your compiler happily\n-# compiles the following initialization:\n-#\n-# static const char s[] = (\"FOO\");\n-#\n-# and define it to \"no\" if you need to remove the parentheses () around the\n-# constant. The default is \"auto\", which means to use parentheses if your\n-# compiler is detected to support it.\n-#\n # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n #\n # Define HAVE_GETDELIM if your system has the getdelim() function.\n@@ -497,8 +488,7 @@ all::\n #\n # pedantic:\n #\n-# Enable -pedantic compilation. This also disables\n-# USE_PARENS_AROUND_GETTEXT_N to produce only relevant warnings.\n+# Enable -pedantic compilation.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -1347,14 +1337,6 @@ ifneq (,$(SOCKLEN_T))\n \tBASIC_CFLAGS += -Dsocklen_t=$(SOCKLEN_T)\n endif\n \n-ifeq (yes,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=1\n-else\n-ifeq (no,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n-endif\n-endif\n-\n ifeq ($(uname_S),Darwin)\n \tifndef NO_FINK\n \t\tifeq ($(shell test -d /sw/lib && echo y),y)\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 022fb58218..41d6345bc0 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -4,8 +4,6 @@ SPARSE_FLAGS += -Wsparse-error\n endif\n ifneq ($(filter pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n-# don't warn for each N_ use\n-DEVELOPER_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n endif\n DEVELOPER_CFLAGS += -Wall\n DEVELOPER_CFLAGS += -Wdeclaration-after-statement\ndiff --git a/gettext.h b/gettext.h\nindex c8b34fd612..d209911ebb 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -55,31 +55,7 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n }\n \n /* Mark msgid for translation but do not translate it. */\n-#if !USE_PARENS_AROUND_GETTEXT_N\n #define N_(msgid) msgid\n-#else\n-/*\n- * Strictly speaking, this will lead to invalid C when\n- * used this way:\n- *\tstatic const char s[] = N_(\"FOO\");\n- * which will expand to\n- *\tstatic const char s[] = (\"FOO\");\n- * and in valid C, the initializer on the right hand side must\n- * be without the parentheses. But many compilers do accept it\n- * as a language extension and it will allow us to catch mistakes\n- * like:\n- *\tstatic const char *msgs[] = {\n- *\t\tN_(\"one\")\n- *\t\tN_(\"two\"),\n- *\t\tN_(\"three\"),\n- *\t\tNULL\n- *\t};\n- * (notice the missing comma on one of the lines) by forcing\n- * a compilation error, because parenthesised (\"one\") (\"two\")\n- * will not get silently turned into (\"onetwo\").\n- */\n-#define N_(msgid) (msgid)\n-#endif\n \n const char *get_preferred_languages(void);\n int is_utf8_locale(void);\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b46605300a..ddc65ff61d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1253,10 +1253,6 @@ int warn_on_fopen_errors(const char *path);\n */\n int open_nofollow(const char *path, int flags);\n \n-#if !defined(USE_PARENS_AROUND_GETTEXT_N) && defined(__GNUC__)\n-#define USE_PARENS_AROUND_GETTEXT_N 1\n-#endif\n-\n #ifndef SHELL_PATH\n # define SHELL_PATH \"/bin/sh\"\n #endif\n"},{"id":"434399","messageId":"20210901091941.34886-1-carenas@gmail.com","threadId":"55998","inReplyTo":"20210809013833.58110-4-carenas@gmail.com","subject":"[RFC PATCH v2 0/4] developer: support pedantic","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-01T09:19:37Z","receivedAt":"2021-09-01T09:20:09Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"WARNING: this will break CI with seen when merged and unless the\nkwown pedantic issue still remaining from fsmonitor[0] is merged\nfirst (expected to come in a reroll)\n\nthis series has a different subject than v1 and that is currently\ntracked as cb/ci-build-pedantic, but is a reroll (even if it discards\nall changes from v1) and was originally suggested by Phillip as an\nalternative.\n\nbecause of that, it might conflict with changes proposed by Ævar[2]\nbut that are still not in \"seen\" AFAIK and merges cleanly otherwise.\n\nfirst patch was suggested[1] by Peff, so hopefully my commit message\nand his assumed SoB are still worth not mixing it with patch 2 (which\nhas a slight different but related focus and touches the same files)\nbut since it is no longer a single patch, lets go wild.\n\npatches 3 and 4 are optional and mostly for RFC, so that a solution\nto any possible issue that the retiring of USE_PARENS_AROUND_GETTEXT_N\nare addressed.\n\nCarlo Marcelo Arenas Belón (3):\n developer: enable pedantic by default\n developer: add an alternative script for detecting broken N_()\n developer: move detect-compiler out of the main directory\n\nJeff King (1):\n developer: retire USE_PARENS_AROUND_GETTEXT_N support\n\n Makefile | 22 +-----\n config.mak.dev | 7 +-\n detect-compiler => devtools/detect-compiler | 0\n .../find_accidentally_concat_i18n_strings.pl | 69 +++++++++++++++++++\n gettext.h | 24 -------\n git-compat-util.h | 4 --\n 6 files changed, 74 insertions(+), 52 deletions(-)\n rename detect-compiler => devtools/detect-compiler (100%)\n create mode 100755 devtools/find_accidentally_concat_i18n_strings.pl\n\n[0] https://lore.kernel.org/git/20210809063004.73736-3-carenas@gmail.com/\n[1] https://lore.kernel.org/git/YS7c3169x5Wk4PlA@coredump.intra.peff.net/\n[2] https://lore.kernel.org/git/cover-v3-0.8-00000000000-20210831T132546Z-avarab@gmail.com/\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434400","messageId":"20210901091941.34886-2-carenas@gmail.com","threadId":"55998","inReplyTo":"20210901091941.34886-1-carenas@gmail.com","subject":"[RFC PATCH v2 1/4] developer: retire USE_PARENS_AROUND_GETTEXT_N support","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-01T09:19:38Z","receivedAt":"2021-09-01T09:20:23Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"From: Jeff King \n\n290c8e7a3f (gettext.h: add parentheses around N_ expansion if supported,\n2015-01-11) adds a trick for GNU compilers that breaks the build, if an\naccidental concatenation of i18n strings is used, but relies on invalid\nC that gcc/clang just happen to allow (unless in pedantic mode).\n\nremove that code and all subsequent fixes so that pedantic can run.\n\nan alternative will be provided in a future patch.\n\nSigned-off-by: Jeff King \nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n Makefile | 20 +-------------------\n config.mak.dev | 2 --\n gettext.h | 24 ------------------------\n git-compat-util.h | 4 ----\n 4 files changed, 1 insertion(+), 49 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 9573190f1d..4e94073c2a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -409,15 +409,6 @@ all::\n # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version\n # before 2.17) for clock_gettime and CLOCK_MONOTONIC.\n #\n-# Define USE_PARENS_AROUND_GETTEXT_N to \"yes\" if your compiler happily\n-# compiles the following initialization:\n-#\n-# static const char s[] = (\"FOO\");\n-#\n-# and define it to \"no\" if you need to remove the parentheses () around the\n-# constant. The default is \"auto\", which means to use parentheses if your\n-# compiler is detected to support it.\n-#\n # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n #\n # Define HAVE_GETDELIM if your system has the getdelim() function.\n@@ -497,8 +488,7 @@ all::\n #\n # pedantic:\n #\n-# Enable -pedantic compilation. This also disables\n-# USE_PARENS_AROUND_GETTEXT_N to produce only relevant warnings.\n+# Enable -pedantic compilation.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -1347,14 +1337,6 @@ ifneq (,$(SOCKLEN_T))\n \tBASIC_CFLAGS += -Dsocklen_t=$(SOCKLEN_T)\n endif\n \n-ifeq (yes,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=1\n-else\n-ifeq (no,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n-endif\n-endif\n-\n ifeq ($(uname_S),Darwin)\n \tifndef NO_FINK\n \t\tifeq ($(shell test -d /sw/lib && echo y),y)\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 022fb58218..41d6345bc0 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -4,8 +4,6 @@ SPARSE_FLAGS += -Wsparse-error\n endif\n ifneq ($(filter pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n-# don't warn for each N_ use\n-DEVELOPER_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n endif\n DEVELOPER_CFLAGS += -Wall\n DEVELOPER_CFLAGS += -Wdeclaration-after-statement\ndiff --git a/gettext.h b/gettext.h\nindex c8b34fd612..d209911ebb 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -55,31 +55,7 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n }\n \n /* Mark msgid for translation but do not translate it. */\n-#if !USE_PARENS_AROUND_GETTEXT_N\n #define N_(msgid) msgid\n-#else\n-/*\n- * Strictly speaking, this will lead to invalid C when\n- * used this way:\n- *\tstatic const char s[] = N_(\"FOO\");\n- * which will expand to\n- *\tstatic const char s[] = (\"FOO\");\n- * and in valid C, the initializer on the right hand side must\n- * be without the parentheses. But many compilers do accept it\n- * as a language extension and it will allow us to catch mistakes\n- * like:\n- *\tstatic const char *msgs[] = {\n- *\t\tN_(\"one\")\n- *\t\tN_(\"two\"),\n- *\t\tN_(\"three\"),\n- *\t\tNULL\n- *\t};\n- * (notice the missing comma on one of the lines) by forcing\n- * a compilation error, because parenthesised (\"one\") (\"two\")\n- * will not get silently turned into (\"onetwo\").\n- */\n-#define N_(msgid) (msgid)\n-#endif\n \n const char *get_preferred_languages(void);\n int is_utf8_locale(void);\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b46605300a..ddc65ff61d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1253,10 +1253,6 @@ int warn_on_fopen_errors(const char *path);\n */\n int open_nofollow(const char *path, int flags);\n \n-#if !defined(USE_PARENS_AROUND_GETTEXT_N) && defined(__GNUC__)\n-#define USE_PARENS_AROUND_GETTEXT_N 1\n-#endif\n-\n #ifndef SHELL_PATH\n # define SHELL_PATH \"/bin/sh\"\n #endif\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434401","messageId":"20210901091941.34886-3-carenas@gmail.com","threadId":"55998","inReplyTo":"20210901091941.34886-1-carenas@gmail.com","subject":"[RFC PATCH v2 2/4] developer: enable pedantic by default","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-01T09:19:39Z","receivedAt":"2021-09-01T09:20:28Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"with the codebase firmly C99 compatible and most compilers supporting\nnewer versions by default, could help bring visibility to problems.\n\nreverse the DEVOPTS=pedantic flag to provide a fallback for people stuck\nwith gcc < 5 or some other compiler that either doesn't support this flag\nor has issues with it, and while at it also enable -Wpedantic which used\nto be controversial when Apple compilers and clang had widely divergent\nversion numbers.\n\nideally any compiler found to have issues with these flags will be added\nto an exception, but leaving it open for now as a weather balloon.\n\n[1] https://lore.kernel.org/git/20181127100557.53891-1-carenas@gmail.com/\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n Makefile | 4 ++--\n config.mak.dev | 3 ++-\n 2 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4e94073c2a..f7a2b20c77 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -486,9 +486,9 @@ all::\n # setting this flag the exceptions are removed, and all of\n # -Wextra is used.\n #\n-# pedantic:\n+# no-pedantic:\n #\n-# Enable -pedantic compilation.\n+# Disable -pedantic compilation.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 41d6345bc0..76f43dea3f 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -2,8 +2,9 @@ ifeq ($(filter no-error,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -Werror\n SPARSE_FLAGS += -Wsparse-error\n endif\n-ifneq ($(filter pedantic,$(DEVOPTS)),)\n+ifeq ($(filter no-pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n+DEVELOPER_CFLAGS += -Wpedantic\n endif\n DEVELOPER_CFLAGS += -Wall\n DEVELOPER_CFLAGS += -Wdeclaration-after-statement\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434402","messageId":"20210901091941.34886-4-carenas@gmail.com","threadId":"55998","inReplyTo":"20210901091941.34886-1-carenas@gmail.com","subject":"[RFC PATCH v2 3/4] developer: add an alternative script for detecting broken N_()","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-01T09:19:40Z","receivedAt":"2021-09-01T09:20:33Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"obviously incomplete and buggy (ex: won't detect two overlapping matches)\n\nit could be added to some makefile target or documented better as an\nalternative to the compilation errors the previous implementation did,\nbut I have to admit, I haven't found any place in the codebase where\na valid concatenation could take place, so at least the tracking of\nexceptions might not be worthy, even if it might be the best part.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n .../find_accidentally_concat_i18n_strings.pl | 69 +++++++++++++++++++\n 1 file changed, 69 insertions(+)\n create mode 100755 devtools/find_accidentally_concat_i18n_strings.pl\n\ndiff --git a/devtools/find_accidentally_concat_i18n_strings.pl b/devtools/find_accidentally_concat_i18n_strings.pl\nnew file mode 100755\nindex 0000000000..82bffb2477\n--- /dev/null\n+++ b/devtools/find_accidentally_concat_i18n_strings.pl\n@@ -0,0 +1,69 @@\n+#!/usr/bin/perl\n+\n+#\n+# find .. \\( -name \"*.c\" -o -name \"*.h\" \\) -exec ./find_accidentally_concat_i18n_strings.pl {} \\;\n+#\n+# this will help find places in the code that might have strings that\n+# are marked for translation but are not correctly separated, causing\n+# problems like the one reported in :\n+#\n+# https://lore.kernel.org/git/ecb18f9d6ac56da0a61c3b98f8f2236@74d39fa044aa309eaea14b9f57fe79c/\n+# \n+\n+use strict;\n+use warnings;\n+\n+my $myself = $0;\n+my $file = $ARGV[0];\n+my $errors = 0;\n+my $key;\n+\n+chomp(my @exceptions = );\n+\n+sub ask {\n+\tlocal $| = 1;\n+\tprint \"possible bug found in $key:\\n\";\n+\tprint \"\\n$&\\n\";\n+\tprint \"\\nadd exception (y/N): \";\n+\tchomp(my $answer = );\n+\tif (lc($answer) ne 'y') {\n+\t\t++$errors;\n+\t\treturn;\n+\t}\n+\treturn 1;\n+}\n+\n+sub process_file {\n+\tmy $content;\n+\topen(my $fh, '<', $file) or die;\n+\t{\n+\t\tlocal $/;\n+\t\t$content = <$fh>;\n+\t}\n+\tclose($fh);\n+\twhile ($content =~ /N_\\(.*?\\)[ \\t\\n]+N_\\(.*?\\)/g) {\n+\t\tmy $index = length($`);\n+\t\t$key = \"$file:$index\";\n+\t\tif (!grep {/^$key$/} @exceptions) {\n+\t\t\tpush @exceptions, $key if ask();\n+\t\t}\n+\t}\n+}\n+\n+&process_file;\n+\n+{\n+\tlocal *MYSELF;\n+\tlocal $/ = \"\\n__END__\";\n+\topen (MYSELF, $myself);\n+\tchomp(my $file = );\n+\tclose MYSELF;\n+\topen (MYSELF, \">$myself\") || die \"can't update myself\";\n+\tprint MYSELF $file, \"\\n__END__\\n\";\n+\tforeach (@exceptions) {\n+\t\tprint MYSELF \"$_\\n\";\n+\t}\n+\tclose MYSELF;\n+}\n+exit($errors);\n+__END__\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434403","messageId":"20210901091941.34886-5-carenas@gmail.com","threadId":"55998","inReplyTo":"20210901091941.34886-1-carenas@gmail.com","subject":"[RFC PATCH v2 4/4] developer: move detect-compiler out of the main directory","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-01T09:19:41Z","receivedAt":"2021-09-01T09:21:04Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"as suggested by Junio[1], and using the newly created subdirectory for\ndev helpers that was introduced in a previous patch.\n\n[1] https://lore.kernel.org/git/xmqqva4gpits.fsf@gitster-ct.c.googlers.com/\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n config.mak.dev | 2 +-\n detect-compiler => devtools/detect-compiler | 0\n 2 files changed, 1 insertion(+), 1 deletion(-)\n rename detect-compiler => devtools/detect-compiler (100%)\n\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 76f43dea3f..e382c65aff 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -18,7 +18,7 @@ DEVELOPER_CFLAGS += -Wvla\n DEVELOPER_CFLAGS += -fno-common\n \n ifndef COMPILER_FEATURES\n-COMPILER_FEATURES := $(shell ./detect-compiler $(CC))\n+COMPILER_FEATURES := $(shell ./devtools/detect-compiler $(CC))\n endif\n \n ifneq ($(filter clang4,$(COMPILER_FEATURES)),)\ndiff --git a/detect-compiler b/devtools/detect-compiler\nsimilarity index 100%\nrename from detect-compiler\nrename to devtools/detect-compiler\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434407","messageId":"YS9RieTeJSFmd6M7@coredump.intra.peff.net","threadId":"55998","inReplyTo":"20210901091941.34886-1-carenas@gmail.com","subject":"Re: [RFC PATCH v2 0/4] developer: support pedantic","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T10:10:17Z","receivedAt":"2021-09-01T10:10:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 02:19:37AM -0700, Carlo Marcelo Arenas Belón wrote:\n\n> first patch was suggested[1] by Peff, so hopefully my commit message\n> and his assumed SoB are still worth not mixing it with patch 2 (which\n> has a slight different but related focus and touches the same files)\n> but since it is no longer a single patch, lets go wild.\n\nMy SoB is fine there (though really Ævar did the actual thinking; I just\ndeleted a lot of lines in vim :) ).\n\nPatch 2 looks good to me, though I kind of wonder if it is even worth\nhaving an option to turn it off.\n\n> patches 3 and 4 are optional and mostly for RFC, so that a solution\n> to any possible issue that the retiring of USE_PARENS_AROUND_GETTEXT_N\n> are addressed.\n\nIMHO the issue it is trying to find is not worth the inevitable problems\nthat hacky perl parsing of C will cause (both false positives and\nnegatives). Not a statement on your perl code, but just based on\nprevious experience.\n\nSo I'd probably take the first two patches, and leave the others.\n\n-Peff\n"},{"id":"434413","messageId":"patch-1.1-d24f1df5d49-20210901T112248Z-avarab@gmail.com","threadId":"55998","inReplyTo":"YS9RieTeJSFmd6M7@coredump.intra.peff.net","subject":"[PATCH] gettext: remove optional non-standard parens in N_() definition","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-01T11:25:52Z","receivedAt":"2021-09-01T11:25:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the USE_PARENS_AROUND_GETTEXT_N compile-time option which was\nmeant to catch an inadvertent mistakes which is too obscure to\nmaintain this facility.\n\nThe backstory of how USE_PARENS_AROUND_GETTEXT_N came about is: When I\nadded the N_() macro in 65784830366 (i18n: add no-op _() and N_()\nwrappers, 2011-02-22) it was defined as:\n\n #define N_(msgid) (msgid)\n\nThis is non-standard C, as was noticed and fixed in 642f85faab2 (i18n:\navoid parenthesized string as array initializer,\n2011-04-07). I.e. this needed to be defined as:\n\n #define N_(msgid) msgid\n\nThen in e62cd35a3e8 (i18n: log: mark parseopt strings for translation,\n2012-08-20) when \"builtin_log_usage\" was marked for translation the\nstring concatenation the string concatenation for passing to usage()\nadded in 1c370ea4e51 (Show usage string for 'git log -h', 'git show\n-h' and 'git diff -h', 2009-08-06) was faithfully preserved:\n\n- \"git log [] [..] [[--] ...]\\n\"\n- \" or: git show [options] ...\",\n+ N_(\"git log [] [..] [[--] ...]\\n\")\n+ N_(\" or: git show [options] ...\"),\n\nThis was then fixed to be the expected array of usage strings in\ne66dc0cc4b1 (log.c: fix translation markings, 2015-01-06) rather than\na string with multiple \"\\n\"-delimited usage strings, and finally in\n290c8e7a3fe (gettext.h: add parentheses around N_ expansion if\nsupported, 2015-01-11) USE_PARENS_AROUND_GETTEXT_N was added to ensure\nthis mistake didn't happen again.\n\nI think that even if this was a N_()-specific issue this\nUSE_PARENS_AROUND_GETTEXT_N facility wouldn't be worth it, the issue\nwould be too rare to worry about.\n\nBut I also think that 290c8e7a3fe which introduced\nUSE_PARENS_AROUND_GETTEXT_N misattributed the problem. The issue\nwasn't with the N_() macro added in e62cd35a3e8, but that before the\nN_() macro existed in the codebase the initial migration to\nparse_options() in 1c370ea4e51 continued passsing in a \"\\n\"-delimited\nstring, when the new API it was migrating to supported and expected\nthe passing of an array.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nOn Wed, Sep 01 2021, Jeff King wrote:\n\n> On Wed, Sep 01, 2021 at 02:19:37AM -0700, Carlo Marcelo Arenas Belón wrote:\n>\n>> first patch was suggested[1] by Peff, so hopefully my commit message\n>> and his assumed SoB are still worth not mixing it with patch 2 (which\n>> has a slight different but related focus and touches the same files)\n>> but since it is no longer a single patch, lets go wild.\n>\n> My SoB is fine there (though really Ævar did the actual thinking; I just\n> deleted a lot of lines in vim :) ).\n>\n> Patch 2 looks good to me, though I kind of wonder if it is even worth\n> having an option to turn it off.\n>\n>> patches 3 and 4 are optional and mostly for RFC, so that a solution\n>> to any possible issue that the retiring of USE_PARENS_AROUND_GETTEXT_N\n>> are addressed.\n>\n> IMHO the issue it is trying to find is not worth the inevitable problems\n> that hacky perl parsing of C will cause (both false positives and\n> negatives). Not a statement on your perl code, but just based on\n> previous experience.\n>\n> So I'd probably take the first two patches, and leave the others.\n\nI came up with this after reading your\n (the patch content is the\nsame) but before seeing that Carlo had beaten me to it here in\n<20210901091941.34886-1-carenas@gmail.com> upthread.\n\nI don't care how this lands exactly, but thin (eye of the beholder and\nall that) that the commit message above is better. Carlo: Feel free to\nsteal it partially or entirely, I also made this a \"PATCH\" instead of\n\"RFC PATCH\" in case Junio feels like queuing this, then you could\nbuild your DEVOPTS=pedantic by default here on top.\n\n Makefile | 20 +-------------------\n config.mak.dev | 2 --\n gettext.h | 24 ------------------------\n git-compat-util.h | 4 ----\n 4 files changed, 1 insertion(+), 49 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex d1feab008fc..4936e234bc1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -409,15 +409,6 @@ all::\n # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version\n # before 2.17) for clock_gettime and CLOCK_MONOTONIC.\n #\n-# Define USE_PARENS_AROUND_GETTEXT_N to \"yes\" if your compiler happily\n-# compiles the following initialization:\n-#\n-# static const char s[] = (\"FOO\");\n-#\n-# and define it to \"no\" if you need to remove the parentheses () around the\n-# constant. The default is \"auto\", which means to use parentheses if your\n-# compiler is detected to support it.\n-#\n # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n #\n # Define HAVE_GETDELIM if your system has the getdelim() function.\n@@ -497,8 +488,7 @@ all::\n #\n # pedantic:\n #\n-# Enable -pedantic compilation. This also disables\n-# USE_PARENS_AROUND_GETTEXT_N to produce only relevant warnings.\n+# Enable -pedantic compilation.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -1347,14 +1337,6 @@ ifneq (,$(SOCKLEN_T))\n \tBASIC_CFLAGS += -Dsocklen_t=$(SOCKLEN_T)\n endif\n \n-ifeq (yes,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=1\n-else\n-ifeq (no,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n-endif\n-endif\n-\n ifeq ($(uname_S),Darwin)\n \tifndef NO_FINK\n \t\tifeq ($(shell test -d /sw/lib && echo y),y)\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 022fb582180..41d6345bc0a 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -4,8 +4,6 @@ SPARSE_FLAGS += -Wsparse-error\n endif\n ifneq ($(filter pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n-# don't warn for each N_ use\n-DEVELOPER_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n endif\n DEVELOPER_CFLAGS += -Wall\n DEVELOPER_CFLAGS += -Wdeclaration-after-statement\ndiff --git a/gettext.h b/gettext.h\nindex c8b34fd6122..d209911ebb8 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -55,31 +55,7 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n }\n \n /* Mark msgid for translation but do not translate it. */\n-#if !USE_PARENS_AROUND_GETTEXT_N\n #define N_(msgid) msgid\n-#else\n-/*\n- * Strictly speaking, this will lead to invalid C when\n- * used this way:\n- *\tstatic const char s[] = N_(\"FOO\");\n- * which will expand to\n- *\tstatic const char s[] = (\"FOO\");\n- * and in valid C, the initializer on the right hand side must\n- * be without the parentheses. But many compilers do accept it\n- * as a language extension and it will allow us to catch mistakes\n- * like:\n- *\tstatic const char *msgs[] = {\n- *\t\tN_(\"one\")\n- *\t\tN_(\"two\"),\n- *\t\tN_(\"three\"),\n- *\t\tNULL\n- *\t};\n- * (notice the missing comma on one of the lines) by forcing\n- * a compilation error, because parenthesised (\"one\") (\"two\")\n- * will not get silently turned into (\"onetwo\").\n- */\n-#define N_(msgid) (msgid)\n-#endif\n \n const char *get_preferred_languages(void);\n int is_utf8_locale(void);\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b46605300ab..ddc65ff61d9 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1253,10 +1253,6 @@ int warn_on_fopen_errors(const char *path);\n */\n int open_nofollow(const char *path, int flags);\n \n-#if !defined(USE_PARENS_AROUND_GETTEXT_N) && defined(__GNUC__)\n-#define USE_PARENS_AROUND_GETTEXT_N 1\n-#endif\n-\n #ifndef SHELL_PATH\n # define SHELL_PATH \"/bin/sh\"\n #endif\n-- \n2.33.0.807.gf14ecf9c2e9\n\n"},{"id":"434415","messageId":"878s0gwmvq.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"YS9RieTeJSFmd6M7@coredump.intra.peff.net","subject":"Re: [RFC PATCH v2 0/4] developer: support pedantic","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-01T11:27:41Z","receivedAt":"2021-09-01T11:34:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\n[I should have included this in my just-sent [1], but forgot]\n\nOn Wed, Sep 01 2021, Jeff King wrote:\n\n> On Wed, Sep 01, 2021 at 02:19:37AM -0700, Carlo Marcelo Arenas Belón wrote:\n>\n>> first patch was suggested[1] by Peff, so hopefully my commit message\n>> and his assumed SoB are still worth not mixing it with patch 2 (which\n>> has a slight different but related focus and touches the same files)\n>> but since it is no longer a single patch, lets go wild.\n>\n> My SoB is fine there (though really Ævar did the actual thinking; I just\n> deleted a lot of lines in vim :) ).\n>\n> Patch 2 looks good to me, though I kind of wonder if it is even worth\n> having an option to turn it off.\n>\n>> patches 3 and 4 are optional and mostly for RFC, so that a solution\n>> to any possible issue that the retiring of USE_PARENS_AROUND_GETTEXT_N\n>> are addressed.\n>\n> IMHO the issue it is trying to find is not worth the inevitable problems\n> that hacky perl parsing of C will cause (both false positives and\n> negatives). Not a statement on your perl code, but just based on\n> previous experience.\n>\n> So I'd probably take the first two patches, and leave the others.\n\nAgreed. Per the rationale in my version of the commit messsage for\nCarlo's 1/4 at [1] I don't think this was ever worth it.\n\nI.e. it wasn't even an N_()-specific issue to begin with, but just a\nmigration from usage() (takes a string) to usage_with_options() (takes\nan array of strings).\n\nI just submitted a related series at [2] to fix the alignment of\ncontinued strings containing \"\\n\" in parse-options.c, which is the\nreason we need to support \"\\n\"-continued strings at all in\nparse-options.c.\n\nSo I think (per [1]) that we should just remove\nUSE_PARENS_AROUND_GETTEXT_N, and that the 3/4 here isn't needed at all\n(aside from concerns about parsing C with Perl).\n\nBut in the future we needed any assertion for this sort of thing at all\nit would be better built on top of my [2]. I.e. parse-options.c could do\nsome basic sanity checking on the usage array it takes, we'd then end up\ndetecting the issue USE_PARENS_AROUND_GETTEXT_N was trying to address,\nand more (such as the alignment problems I fixed in 1/2 of my [2]).\n\n1. https://lore.kernel.org/git/patch-1.1-d24f1df5d49-20210901T112248Z-avarab@gmail.com\n2. https://lore.kernel.org/git/cover-0.2-00000000000-20210901T110917Z-avarab@gmail.com\n"},{"id":"434459","messageId":"CAPig+cRHwK=m0jpuTgsTG+cNVpGASHJP6v3kEQrJFS6Qt=biwQ@mail.gmail.com","threadId":"55998","inReplyTo":"patch-1.1-d24f1df5d49-20210901T112248Z-avarab@gmail.com","subject":"Re: [PATCH] gettext: remove optional non-standard parens in N_() definition","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-09-01T17:31:12Z","receivedAt":"2021-09-01T17:31:27Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Sep 1, 2021 at 7:26 AM Ævar Arnfjörð Bjarmason wrote:\n> [...]\n> Then in e62cd35a3e8 (i18n: log: mark parseopt strings for translation,\n> 2012-08-20) when \"builtin_log_usage\" was marked for translation the\n> string concatenation the string concatenation for passing to usage()\n> added in 1c370ea4e51 (Show usage string for 'git log -h', 'git show\n> -h' and 'git diff -h', 2009-08-06) was faithfully preserved:\n\n\"...the string concatenation the string concatenation...\"\n"},{"id":"434461","messageId":"xmqqbl5cqixu.fsf@gitster.g","threadId":"55998","inReplyTo":"YS7c3169x5Wk4PlA@coredump.intra.peff.net","subject":"Re: [PATCH/RFC 3/3] ci: run a pedantic build as part of the GitHub workflow","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-01T17:55:41Z","receivedAt":"2021-09-01T17:55:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Tue, Aug 31, 2021 at 04:54:52PM -0700, Carlo Arenas wrote:\n>\n>> On Tue, Aug 31, 2021 at 1:57 PM Ævar Arnfjörð Bjarmason\n>> wrote:\n>> >\n>> > On the other hand maybe we should just remove\n>> > USE_PARENS_AROUND_GETTEXT_N entirely, i.e. always use the parens.\n>> \n>> that would break pedantic in all versions of gcc since it is a GNU\n>> extension and is not valid in any C standard.\n>> (unlike the ones we are using with weather balloons and that are valid C99)\n>\n> I think Ævar might have mis-spoke there. It would make sense to get rid\n> of the feature and _never_ use parens, which is always valid C (and does\n> not tickle pedantic, but also does not catch any accidental string\n> concatenation).\n>\n> That actually seems quite reasonable to me.\n\nThat does sound sensible.\n\n> Something like this, I guess?\n\nLooks good. We could give a warning when the now defunct knob is\nused, but I don't think there is anything gained by doing so (over\njust silently ignoring it).\n"},{"id":"434462","messageId":"CAPUEspj2PNCjM2cLU6WzQ+rV7ftRBTfptKwht7N-=zVFTzan-A@mail.gmail.com","threadId":"55998","inReplyTo":"878s0gwmvq.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH v2 0/4] developer: support pedantic","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-09-01T18:03:02Z","receivedAt":"2021-09-01T18:03:21Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Wed, Sep 1, 2021 at 4:34 AM Ævar Arnfjörð Bjarmason wrote:\n>\n> On Wed, Sep 01 2021, Jeff King wrote:\n>\n> > Patch 2 looks good to me, though I kind of wonder if it is even worth\n> > having an option to turn it off.\n\nI failed to mention \"Patch 2\" isn't ready as it will break the mingw64\nbuilds in Windows.\n\nWhile I also hope there is no need to have an option to turn it off,\nrealistically I expect the wall of errors is still there for non\ngcc/clang compilers and I am curious if some developer still using\nRHEL 7 (or a clone) will report back and will be forced to use it.\n\n> > IMHO the issue it is trying to find is not worth the inevitable problems\n> > that hacky perl parsing of C will cause (both false positives and\n> > negatives). Not a statement on your perl code, but just based on\n> > previous experience.\n> >\n> > So I'd probably take the first two patches, and leave the others.\n\nMaybe better to discard the whole series and rebase it on top of Ævar's then\n\n> So I think (per [1]) that we should just remove\n> USE_PARENS_AROUND_GETTEXT_N, and that the 3/4 here isn't needed at all\n> (aside from concerns about parsing C with Perl).\n>\n> But in the future we need any assertion for this sort of thing at all\n> it would be better built on top of my [2]. I.e. parse-options.c could do\n> some basic sanity checking on the usage array it takes, we'd then end up\n> detecting the issue USE_PARENS_AROUND_GETTEXT_N was trying to address,\n> and more (such as the alignment problems I fixed in 1/2 of my [2]).\n\nRegardless of how ugly my perl script was, I don't think this specific\nissue could be\nhandled by the C code, as it needs to be done with preprocessed sources.\n\nnote also, it is not really parsing C, but just looking at a regex\nwhich could have been as well handled with a simple grep.\n\nThe script was built under the incorrect assumption it would be useful\nto track exceptions and have a way to keep that state (as well as the\ncode) cleanly out of the way (which is why patch 4 is also there).\n\nCarlo\n"},{"id":"434505","messageId":"YTCVzUKfjQTBFmr5@coredump.intra.peff.net","threadId":"55998","inReplyTo":"patch-1.1-d24f1df5d49-20210901T112248Z-avarab@gmail.com","subject":"Re: [PATCH] gettext: remove optional non-standard parens in N_() definition","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-02T09:13:49Z","receivedAt":"2021-09-02T09:13:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 01:25:52PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> I don't care how this lands exactly, but thin (eye of the beholder and\n> all that) that the commit message above is better. Carlo: Feel free to\n> steal it partially or entirely, I also made this a \"PATCH\" instead of\n> \"RFC PATCH\" in case Junio feels like queuing this, then you could\n> build your DEVOPTS=pedantic by default here on top.\n\nFWIW, I think it is better, too. :)\n\nOne small typo (in addition to the one Eric noted):\n\n> Remove the USE_PARENS_AROUND_GETTEXT_N compile-time option which was\n> meant to catch an inadvertent mistakes which is too obscure to\n> maintain this facility.\n\ns/mistakes/mistake/\n\n-Peff\n"},{"id":"434592","messageId":"xmqqilzikcp7.fsf@gitster.g","threadId":"55998","inReplyTo":"patch-1.1-d24f1df5d49-20210901T112248Z-avarab@gmail.com","subject":"Re: [PATCH] gettext: remove optional non-standard parens in N_() definition","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-02T19:19:16Z","receivedAt":"2021-09-02T19:19:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Remove the USE_PARENS_AROUND_GETTEXT_N compile-time option which was\n> meant to catch an inadvertent mistakes which is too obscure to\n> maintain this facility.\n> ...\n> I don't care how this lands exactly, but thin (eye of the beholder and\n> all that) that the commit message above is better. Carlo: Feel free to\n> steal it partially or entirely, I also made this a \"PATCH\" instead of\n> \"RFC PATCH\" in case Junio feels like queuing this, then you could\n> build your DEVOPTS=pedantic by default here on top.\n\nFWIW, I think this goes in the right direction, and I'd rather not\nto have to handle too many multi-patch topics in which one step\ntakes hostage the other steps.\n\nThanks, all.\n"},{"id":"434641","messageId":"20210903170232.57646-1-carenas@gmail.com","threadId":"55998","inReplyTo":"20210901091941.34886-1-carenas@gmail.com","subject":"[PATCH v3 0/3] support pedantic in developer mode","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-03T17:02:29Z","receivedAt":"2021-09-03T17:03:03Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"This series enables pedantic mode for building when DEVELOPER=1 is\nused and as an alternative to only enabling it in one CI job, that\nwas merged to \"seen\" as part of cb/ci-build-pedantic.\n\nThe second patch is really an independent prerequisite to ensure\nthat it doesn't break the build for Windows and is the minimal change\npossible.\n\nAdditional changes needed for the git-for-windows/git fork main to be\nposted independently.\n\nIt merges and builds successfully all the way to \"seen\" IF the known\nproblem reported earlier[1] and expected as part of a reroll of \njh/builtin-fsmonitor is merged first.\n\n[1] https://lore.kernel.org/git/20210809063004.73736-3-carenas@gmail.com/\n\nCarlo Marcelo Arenas Belón (2):\n win32: allow building with pedantic mode enabled\n developer: enable pedantic by default\n\nÆvar Arnfjörð Bjarmason (1):\n gettext: remove optional non-standard parens in N_() definition\n\n Makefile | 22 ++--------------------\n compat/nedmalloc/nedmalloc.c | 2 +-\n compat/win32/lazyload.h | 2 +-\n config.mak.dev | 19 +++++++++++--------\n gettext.h | 24 ------------------------\n git-compat-util.h | 4 ----\n 6 files changed, 15 insertions(+), 58 deletions(-)\n\n--\nv3\n- replace the first patch with an even better worded one from Ævar\n- include minor changes needed for Windows\n- version check new flags to avoid risk of breaking old compilers\n- drop alternative\nv2\n- enable pedantic globally instead of single job as suggested by Phillip\n- propose an alternative solution for USE_PARENS_AROUND_GETTEXT_N\nv1\n- create job to check for pedantic compilation (only in Linux, using\n Fedora)\n\n2.33.0.481.g26d3bed244\n\n"},{"id":"434642","messageId":"20210903170232.57646-2-carenas@gmail.com","threadId":"55998","inReplyTo":"20210903170232.57646-1-carenas@gmail.com","subject":"[PATCH v3 1/3] gettext: remove optional non-standard parens in N_() definition","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-03T17:02:30Z","receivedAt":"2021-09-03T17:03:04Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nRemove the USE_PARENS_AROUND_GETTEXT_N compile-time option which was\nmeant to catch an inadvertent mistake which is too obscure to\nmaintain this facility.\n\nThe backstory of how USE_PARENS_AROUND_GETTEXT_N came about is: When I\nadded the N_() macro in 65784830366 (i18n: add no-op _() and N_()\nwrappers, 2011-02-22) it was defined as:\n\n #define N_(msgid) (msgid)\n\nThis is non-standard C, as was noticed and fixed in 642f85faab2 (i18n:\navoid parenthesized string as array initializer, 2011-04-07).\nI.e. this needed to be defined as:\n\n #define N_(msgid) msgid\n\nThen in e62cd35a3e8 (i18n: log: mark parseopt strings for translation,\n2012-08-20) when \"builtin_log_usage\" was marked for translation the\nstring concatenation for passing to usage() added in 1c370ea4e51\n(Show usage string for 'git log -h', 'git show -h' and 'git diff -h',\n2009-08-06) was faithfully preserved:\n\n- \"git log [] [..] [[--] ...]\\n\"\n- \" or: git show [options] ...\",\n+ N_(\"git log [] [..] [[--] ...]\\n\")\n+ N_(\" or: git show [options] ...\"),\n\nThis was then fixed to be the expected array of usage strings in\ne66dc0cc4b1 (log.c: fix translation markings, 2015-01-06) rather than\na string with multiple \"\\n\"-delimited usage strings, and finally in\n290c8e7a3fe (gettext.h: add parentheses around N_ expansion if\nsupported, 2015-01-11) USE_PARENS_AROUND_GETTEXT_N was added to ensure\nthis mistake didn't happen again.\n\nI think that even if this was a N_()-specific issue this\nUSE_PARENS_AROUND_GETTEXT_N facility wouldn't be worth it, the issue\nwould be too rare to worry about.\n\nBut I also think that 290c8e7a3fe which introduced\nUSE_PARENS_AROUND_GETTEXT_N misattributed the problem. The issue\nwasn't with the N_() macro added in e62cd35a3e8, but that before the\nN_() macro existed in the codebase the initial migration to\nparse_options() in 1c370ea4e51 continued passsing in a \"\\n\"-delimited\nstring, when the new API it was migrating to supported and expected\nthe passing of an array.\n\nHelped-by: Eric Sunshine \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n Makefile | 20 +-------------------\n config.mak.dev | 2 --\n gettext.h | 24 ------------------------\n git-compat-util.h | 4 ----\n 4 files changed, 1 insertion(+), 49 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 9573190f1d..4e94073c2a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -409,15 +409,6 @@ all::\n # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version\n # before 2.17) for clock_gettime and CLOCK_MONOTONIC.\n #\n-# Define USE_PARENS_AROUND_GETTEXT_N to \"yes\" if your compiler happily\n-# compiles the following initialization:\n-#\n-# static const char s[] = (\"FOO\");\n-#\n-# and define it to \"no\" if you need to remove the parentheses () around the\n-# constant. The default is \"auto\", which means to use parentheses if your\n-# compiler is detected to support it.\n-#\n # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n #\n # Define HAVE_GETDELIM if your system has the getdelim() function.\n@@ -497,8 +488,7 @@ all::\n #\n # pedantic:\n #\n-# Enable -pedantic compilation. This also disables\n-# USE_PARENS_AROUND_GETTEXT_N to produce only relevant warnings.\n+# Enable -pedantic compilation.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -1347,14 +1337,6 @@ ifneq (,$(SOCKLEN_T))\n \tBASIC_CFLAGS += -Dsocklen_t=$(SOCKLEN_T)\n endif\n \n-ifeq (yes,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=1\n-else\n-ifeq (no,$(USE_PARENS_AROUND_GETTEXT_N))\n-\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n-endif\n-endif\n-\n ifeq ($(uname_S),Darwin)\n \tifndef NO_FINK\n \t\tifeq ($(shell test -d /sw/lib && echo y),y)\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 022fb58218..41d6345bc0 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -4,8 +4,6 @@ SPARSE_FLAGS += -Wsparse-error\n endif\n ifneq ($(filter pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n-# don't warn for each N_ use\n-DEVELOPER_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n endif\n DEVELOPER_CFLAGS += -Wall\n DEVELOPER_CFLAGS += -Wdeclaration-after-statement\ndiff --git a/gettext.h b/gettext.h\nindex c8b34fd612..d209911ebb 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -55,31 +55,7 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n }\n \n /* Mark msgid for translation but do not translate it. */\n-#if !USE_PARENS_AROUND_GETTEXT_N\n #define N_(msgid) msgid\n-#else\n-/*\n- * Strictly speaking, this will lead to invalid C when\n- * used this way:\n- *\tstatic const char s[] = N_(\"FOO\");\n- * which will expand to\n- *\tstatic const char s[] = (\"FOO\");\n- * and in valid C, the initializer on the right hand side must\n- * be without the parentheses. But many compilers do accept it\n- * as a language extension and it will allow us to catch mistakes\n- * like:\n- *\tstatic const char *msgs[] = {\n- *\t\tN_(\"one\")\n- *\t\tN_(\"two\"),\n- *\t\tN_(\"three\"),\n- *\t\tNULL\n- *\t};\n- * (notice the missing comma on one of the lines) by forcing\n- * a compilation error, because parenthesised (\"one\") (\"two\")\n- * will not get silently turned into (\"onetwo\").\n- */\n-#define N_(msgid) (msgid)\n-#endif\n \n const char *get_preferred_languages(void);\n int is_utf8_locale(void);\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b46605300a..ddc65ff61d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1253,10 +1253,6 @@ int warn_on_fopen_errors(const char *path);\n */\n int open_nofollow(const char *path, int flags);\n \n-#if !defined(USE_PARENS_AROUND_GETTEXT_N) && defined(__GNUC__)\n-#define USE_PARENS_AROUND_GETTEXT_N 1\n-#endif\n-\n #ifndef SHELL_PATH\n # define SHELL_PATH \"/bin/sh\"\n #endif\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434643","messageId":"20210903170232.57646-3-carenas@gmail.com","threadId":"55998","inReplyTo":"20210903170232.57646-1-carenas@gmail.com","subject":"[PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-03T17:02:31Z","receivedAt":"2021-09-03T17:03:06Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"In preparation to building with pedantic mode enabled, change a couple\nof places where the current mingw gcc compiler provided with the SDK\nreports issues.\n\nA full fix for the incompatible use of (void *) to store function\npointers has been punted, with the minimal change to instead use a\ngeneric function pointer (FARPROC), and therefore the (hopefully)\ntemporary need to disable incompatible pointer warnings.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\nThis is all that is needed to build cleanly once merged to maint/master/next\n\nThere is at least one fix needed on top for seen, that was sent already\nand is expected as part of a different reroll as well of several more for\ngit-for-windows/main that will be send independently.\n\n compat/nedmalloc/nedmalloc.c | 2 +-\n compat/win32/lazyload.h | 2 +-\n config.mak.dev | 13 ++++++++-----\n 3 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\nindex 1cc31c3502..edb438a777 100644\n--- a/compat/nedmalloc/nedmalloc.c\n+++ b/compat/nedmalloc/nedmalloc.c\n@@ -510,7 +510,7 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n \tassert(idx<=THREADCACHEMAXBINS);\n \tif(tck==*binsptr)\n \t{\n-\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n+\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n \t\tabort();\n \t}\n #ifdef FULLSANITYCHECKS\ndiff --git a/compat/win32/lazyload.h b/compat/win32/lazyload.h\nindex 9e631c8593..d2056cdadf 100644\n--- a/compat/win32/lazyload.h\n+++ b/compat/win32/lazyload.h\n@@ -37,7 +37,7 @@ struct proc_addr {\n #define INIT_PROC_ADDR(function) \\\n \t(function = get_proc_addr(&proc_addr_##function))\n \n-static inline void *get_proc_addr(struct proc_addr *proc)\n+static inline FARPROC get_proc_addr(struct proc_addr *proc)\n {\n \t/* only do this once */\n \tif (!proc->initialized) {\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 41d6345bc0..5424db5c22 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -1,11 +1,18 @@\n+ifndef COMPILER_FEATURES\n+COMPILER_FEATURES := $(shell ./detect-compiler $(CC))\n+endif\n+\n ifeq ($(filter no-error,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -Werror\n SPARSE_FLAGS += -Wsparse-error\n endif\n+DEVELOPER_CFLAGS += -Wall\n ifneq ($(filter pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n+ifneq ($(filter gcc5,$(COMPILER_FEATURES)),)\n+DEVELOPER_CFLAGS += -Wno-incompatible-pointer-types\n+endif\n endif\n-DEVELOPER_CFLAGS += -Wall\n DEVELOPER_CFLAGS += -Wdeclaration-after-statement\n DEVELOPER_CFLAGS += -Wformat-security\n DEVELOPER_CFLAGS += -Wold-style-definition\n@@ -16,10 +23,6 @@ DEVELOPER_CFLAGS += -Wunused\n DEVELOPER_CFLAGS += -Wvla\n DEVELOPER_CFLAGS += -fno-common\n \n-ifndef COMPILER_FEATURES\n-COMPILER_FEATURES := $(shell ./detect-compiler $(CC))\n-endif\n-\n ifneq ($(filter clang4,$(COMPILER_FEATURES)),)\n DEVELOPER_CFLAGS += -Wtautological-constant-out-of-range-compare\n endif\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434644","messageId":"20210903170232.57646-4-carenas@gmail.com","threadId":"55998","inReplyTo":"20210903170232.57646-1-carenas@gmail.com","subject":"[PATCH v3 3/3] developer: enable pedantic by default","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-03T17:02:32Z","receivedAt":"2021-09-03T17:03:07Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"With the codebase firmly C99 compatible and most compilers supporting\nnewer versions by default, it could help bring visibility to problems.\n\nReverse the DEVOPTS=pedantic flag to provide a fallback for people stuck\nwith gcc < 5 or some other compiler that either doesn't support this flag\nor has issues with it, and while at it also enable -Wpedantic which used\nto be controversial[1] when Apple compilers and clang had widely divergent\nversion numbers.\n\nIdeally any compiler found to have issues with these flags will be added\nto an exception, and indeed, one was added to safely process windows\nheaders that would use non standard print identifiers, but it is expected\nthat more will be needed, so it could be considered a weather balloon.\n\n[1] https://lore.kernel.org/git/20181127100557.53891-1-carenas@gmail.com/\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n\n Makefile | 4 ++--\n config.mak.dev | 4 +++-\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 4e94073c2a..f7a2b20c77 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -486,9 +486,9 @@ all::\n # setting this flag the exceptions are removed, and all of\n # -Wextra is used.\n #\n-# pedantic:\n+# no-pedantic:\n #\n-# Enable -pedantic compilation.\n+# Disable -pedantic compilation.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\ndiff --git a/config.mak.dev b/config.mak.dev\nindex 5424db5c22..c080ac0231 100644\n--- a/config.mak.dev\n+++ b/config.mak.dev\n@@ -7,9 +7,11 @@ DEVELOPER_CFLAGS += -Werror\n SPARSE_FLAGS += -Wsparse-error\n endif\n DEVELOPER_CFLAGS += -Wall\n-ifneq ($(filter pedantic,$(DEVOPTS)),)\n+ifeq ($(filter no-pedantic,$(DEVOPTS)),)\n DEVELOPER_CFLAGS += -pedantic\n+DEVELOPER_CFLAGS += -Wpedantic\n ifneq ($(filter gcc5,$(COMPILER_FEATURES)),)\n+DEVELOPER_CFLAGS += -Wno-pedantic-ms-format\n DEVELOPER_CFLAGS += -Wno-incompatible-pointer-types\n endif\n endif\n-- \n2.33.0.481.g26d3bed244\n\n"},{"id":"434667","messageId":"bc4789a0-ae80-c1dd-35b1-86949a807490@web.de","threadId":"55998","inReplyTo":"20210903170232.57646-3-carenas@gmail.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-09-03T18:47:02Z","receivedAt":"2021-09-03T18:47:45Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.09.21 um 19:02 schrieb Carlo Marcelo Arenas Belón:\n> In preparation to building with pedantic mode enabled, change a couple\n> of places where the current mingw gcc compiler provided with the SDK\n> reports issues.\n>\n> A full fix for the incompatible use of (void *) to store function\n> pointers has been punted, with the minimal change to instead use a\n> generic function pointer (FARPROC), and therefore the (hopefully)\n> temporary need to disable incompatible pointer warnings.\n>\n> Signed-off-by: Carlo Marcelo Arenas Belón \n> ---\n> This is all that is needed to build cleanly once merged to maint/master/next\n>\n> There is at least one fix needed on top for seen, that was sent already\n> and is expected as part of a different reroll as well of several more for\n> git-for-windows/main that will be send independently.\n>\n> compat/nedmalloc/nedmalloc.c | 2 +-\n> compat/win32/lazyload.h | 2 +-\n> config.mak.dev | 13 ++++++++-----\n> 3 files changed, 10 insertions(+), 7 deletions(-)\n>\n> diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n> index 1cc31c3502..edb438a777 100644\n> --- a/compat/nedmalloc/nedmalloc.c\n> +++ b/compat/nedmalloc/nedmalloc.c\n> @@ -510,7 +510,7 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n> \tassert(idx<=THREADCACHEMAXBINS);\n> \tif(tck==*binsptr)\n> \t{\n> -\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n> +\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n\nThis change is not mentioned in the commit message. Clang on MacOS\ndoesn't like the original code either and report if USE_NED_ALLOCATOR is\nenabled it reports:\n\ncompat/nedmalloc/nedmalloc.c:513:82: error: format specifies type 'void *' but the argument has type 'threadcacheblk *' (aka 'struct threadcacheblk_t *') [-Werror,-Wformat-pedantic]\n fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n ~~ ^~~\nThis makes no sense to me, though: Any pointer can be converted to a\nvoid pointer without a cast in C. GCC doesn't require void pointers\nfor %p even with -pedantic.\n\nA slightly shorter fix would be to replace \"tck\" with \"mem\". Not as\nobvious without further context, though.\n\nRené\n"},{"id":"434669","messageId":"YTKBzi3z5AotirNO@carlos-mbp.lan","threadId":"55998","inReplyTo":"bc4789a0-ae80-c1dd-35b1-86949a807490@web.de","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-03T20:13:02Z","receivedAt":"2021-09-03T20:13:16Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Fri, Sep 03, 2021 at 08:47:02PM +0200, René Scharfe wrote:\n> Am 03.09.21 um 19:02 schrieb Carlo Marcelo Arenas Belón:\n> > diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n> > index 1cc31c3502..edb438a777 100644\n> > --- a/compat/nedmalloc/nedmalloc.c\n> > +++ b/compat/nedmalloc/nedmalloc.c\n> > @@ -510,7 +510,7 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n> > \tassert(idx<=THREADCACHEMAXBINS);\n> > \tif(tck==*binsptr)\n> > \t{\n> > -\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n> > +\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n> \n> This change is not mentioned in the commit message.\n\ngot me there, I was intentionally trying to ignore it since nedmalloc gives\nme PTSD and is obsoleted AFAIK[1], so just adding a casting to void (while\nugly) was also less intrusive.\n\n> compat/nedmalloc/nedmalloc.c:513:82: error: format specifies type 'void *' but the argument has type 'threadcacheblk *' (aka 'struct threadcacheblk_t *') [-Werror,-Wformat-pedantic]\n> fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n> ~~ ^~~\n> This makes no sense to me, though: Any pointer can be converted to a\n> void pointer without a cast in C. GCC doesn't require void pointers\n> for %p even with -pedantic.\n\nstrange, gcc-11 prints the following in MacOS for me:\n\ncompat/nedmalloc/nedmalloc.c: In function 'threadcache_free':\ncompat/nedmalloc/nedmalloc.c:522:78: warning: format '%p' expects argument of type 'void *', but argument 3 has type 'threadcacheblk *' {aka 'struct threadcacheblk_t *'} [-Wformat=]\n 522 | fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n | ~^ ~~~\n | | |\n | void * threadcacheblk * {aka struct threadcacheblk_t *}\n\nI think the rationale is that it is better to be safe than sorry, and since\nthe parameter is variadic there is no chance for the compiler to do any\nimplicit type casting (unless one is provided explicitly).\n\nclang 14 does also trigger a warning, so IMHO this code will be needed\nuntil nedmalloc is retired.\n\n> A slightly shorter fix would be to replace \"tck\" with \"mem\". Not as\n> obvious without further context, though.\n\nso something like this on top?\n\nCarlo\n---- > 8 ----\ndiff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\nindex edb438a777..14e8c4df4f 100644\n--- a/compat/nedmalloc/nedmalloc.c\n+++ b/compat/nedmalloc/nedmalloc.c\n@@ -510,7 +510,15 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n \tassert(idx<=THREADCACHEMAXBINS);\n \tif(tck==*binsptr)\n \t{\n-\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n+\t\t/*\n+\t\t * Original code used tck instead of mem, but that was changed\n+\t\t * to workaround a pedantic warning from mingw64 gcc 10.3 that\n+\t\t * requires %p to have a explicit (void *) as a parameter.\n+\t\t *\n+\t\t * This might seem to be a compiler bug or limitation that\n+\t\t * should be changed back if fixed for maintanability.\n+\t\t */\n+\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", mem);\n \t\tabort();\n \t}\n #ifdef FULLSANITYCHECKS\n"},{"id":"434670","messageId":"xmqq7dfxfli5.fsf@gitster.g","threadId":"55998","inReplyTo":"YTKBzi3z5AotirNO@carlos-mbp.lan","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-03T20:32:34Z","receivedAt":"2021-09-03T20:32:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Carlo Marcelo Arenas Belón writes:\n\n>> A slightly shorter fix would be to replace \"tck\" with \"mem\". Not as\n>> obvious without further context, though.\n>\n> so something like this on top?\n\n> Carlo\n> ---- > 8 ----\n> diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n> index edb438a777..14e8c4df4f 100644\n> --- a/compat/nedmalloc/nedmalloc.c\n> +++ b/compat/nedmalloc/nedmalloc.c\n> @@ -510,7 +510,15 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n> \tassert(idx<=THREADCACHEMAXBINS);\n> \tif(tck==*binsptr)\n> \t{\n> -\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n> +\t\t/*\n> +\t\t * Original code used tck instead of mem, but that was changed\n> +\t\t * to workaround a pedantic warning from mingw64 gcc 10.3 that\n> +\t\t * requires %p to have a explicit (void *) as a parameter.\n> +\t\t *\n> +\t\t * This might seem to be a compiler bug or limitation that\n> +\t\t * should be changed back if fixed for maintanability.\n> +\t\t */\n> +\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", mem);\n> \t\tabort();\n> \t}\n\nThe new comment explains why the original (i.e. unadorned 'tck'),\nwhich should work fine, needs to be changed. The reason is because\na version of compiler wants an explict (void *) cast to go with the\nplaceholder \"%p\".\n\nGiven that, it would be much better to pass (void *)tck instead of\nmem, no? Especially since the comment does not say tck and mem have\nthe same pointer value.\n\nHaving said lal that, I have to wonder if how much help the\ndeveloper who is hunting for allocation bug is getting out of a raw\npointer value in this message, though.\n\n"},{"id":"434672","messageId":"e20dc0b7-8925-1ccf-3adf-c52a892cc3f0@web.de","threadId":"55998","inReplyTo":"YTKBzi3z5AotirNO@carlos-mbp.lan","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-09-03T20:38:30Z","receivedAt":"2021-09-03T20:39:03Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.09.21 um 22:13 schrieb Carlo Marcelo Arenas Belón:\n> On Fri, Sep 03, 2021 at 08:47:02PM +0200, René Scharfe wrote:\n>> Am 03.09.21 um 19:02 schrieb Carlo Marcelo Arenas Belón:\n>>> diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n>>> index 1cc31c3502..edb438a777 100644\n>>> --- a/compat/nedmalloc/nedmalloc.c\n>>> +++ b/compat/nedmalloc/nedmalloc.c\n>>> @@ -510,7 +510,7 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n>>> \tassert(idx<=THREADCACHEMAXBINS);\n>>> \tif(tck==*binsptr)\n>>> \t{\n>>> -\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n>>> +\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n>>\n>> This change is not mentioned in the commit message.\n>\n> got me there, I was intentionally trying to ignore it since nedmalloc gives\n> me PTSD and is obsoleted AFAIK[1], so just adding a casting to void (while\n> ugly) was also less intrusive.\n>\n>> compat/nedmalloc/nedmalloc.c:513:82: error: format specifies type 'void *' but the argument has type 'threadcacheblk *' (aka 'struct threadcacheblk_t *') [-Werror,-Wformat-pedantic]\n>> fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n>> ~~ ^~~\n>> This makes no sense to me, though: Any pointer can be converted to a\n>> void pointer without a cast in C. GCC doesn't require void pointers\n>> for %p even with -pedantic.\n>\n> strange, gcc-11 prints the following in MacOS for me:\n>\n> compat/nedmalloc/nedmalloc.c: In function 'threadcache_free':\n> compat/nedmalloc/nedmalloc.c:522:78: warning: format '%p' expects argument of type 'void *', but argument 3 has type 'threadcacheblk *' {aka 'struct threadcacheblk_t *'} [-Wformat=]\n> 522 | fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n> | ~^ ~~~\n> | | |\n> | void * threadcacheblk * {aka struct threadcacheblk_t *}\n>\n> I think the rationale is that it is better to be safe than sorry, and since\n> the parameter is variadic there is no chance for the compiler to do any\n> implicit type casting (unless one is provided explicitly).\n\nTrue, other pointers could be smaller on some machines.\n\n> clang 14 does also trigger a warning, so IMHO this code will be needed\n> until nedmalloc is retired.\n>\n>> A slightly shorter fix would be to replace \"tck\" with \"mem\". Not as\n>> obvious without further context, though.\n>\n> so something like this on top?\n\nNah, I like your original version better now that I understand the warning..\n\nThough for upstream it would make more sense to report the caller-supplied\npointer value in the error message than a casted one..\n\n>\n> Carlo\n> ---- > 8 ----\n> diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n> index edb438a777..14e8c4df4f 100644\n> --- a/compat/nedmalloc/nedmalloc.c\n> +++ b/compat/nedmalloc/nedmalloc.c\n> @@ -510,7 +510,15 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n> \tassert(idx<=THREADCACHEMAXBINS);\n> \tif(tck==*binsptr)\n> \t{\n> -\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n> +\t\t/*\n> +\t\t * Original code used tck instead of mem, but that was changed\n> +\t\t * to workaround a pedantic warning from mingw64 gcc 10.3 that\n> +\t\t * requires %p to have a explicit (void *) as a parameter.\n> +\t\t *\n> +\t\t * This might seem to be a compiler bug or limitation that\n> +\t\t * should be changed back if fixed for maintanability.\n> +\t\t */\n> +\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", mem);\n> \t\tabort();\n> \t}\n> #ifdef FULLSANITYCHECKS\n>\n"},{"id":"434683","messageId":"5983c238-e926-3b08-ed10-1de1343a8d00@web.de","threadId":"55998","inReplyTo":"e20dc0b7-8925-1ccf-3adf-c52a892cc3f0@web.de","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-09-04T09:37:28Z","receivedAt":"2021-09-04T09:37:48Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.09.21 um 22:38 schrieb René Scharfe:\n> Am 03.09.21 um 22:13 schrieb Carlo Marcelo Arenas Belón:\n>> On Fri, Sep 03, 2021 at 08:47:02PM +0200, René Scharfe wrote:\n>>> Am 03.09.21 um 19:02 schrieb Carlo Marcelo Arenas Belón:\n>>>> diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n>>>> index 1cc31c3502..edb438a777 100644\n>>>> --- a/compat/nedmalloc/nedmalloc.c\n>>>> +++ b/compat/nedmalloc/nedmalloc.c\n>>>> @@ -510,7 +510,7 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n>>>> \tassert(idx<=THREADCACHEMAXBINS);\n>>>> \tif(tck==*binsptr)\n>>>> \t{\n>>>> -\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n>>>> +\t\tfprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n>>>\n>>> This change is not mentioned in the commit message.\n>>\n>> got me there, I was intentionally trying to ignore it since nedmalloc gives\n>> me PTSD and is obsoleted AFAIK[1], so just adding a casting to void (while\n>> ugly) was also less intrusive.\n\nExpected your [1] to stand for a footnote, and got confused when I found none.\nThe last commit in https://github.com/ned14/nedmalloc is from seven years ago\nand this repository is archived, with the author still being active on GitHub.\nSeems like nedmalloc reached its end of life. Has there been an official\nannouncement?\n\n>> strange, gcc-11 prints the following in MacOS for me:\n>>\n>> compat/nedmalloc/nedmalloc.c: In function 'threadcache_free':\n>> compat/nedmalloc/nedmalloc.c:522:78: warning: format '%p' expects argument of type 'void *', but argument 3 has type 'threadcacheblk *' {aka 'struct threadcacheblk_t *'} [-Wformat=]\n>> 522 | fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n>> | ~^ ~~~\n>> | | |\n>> | void * threadcacheblk * {aka struct threadcacheblk_t *}\n\nI don't have GCC installed, only checked with https://godbolt.org/z/jc356vqb4\n\nRené\n"},{"id":"434695","messageId":"CAPUEspheuPPfbCv59ouNcq4Ac8-6LvOAwDO3V3F9UJGHm+Qwyg@mail.gmail.com","threadId":"55998","inReplyTo":"5983c238-e926-3b08-ed10-1de1343a8d00@web.de","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-09-04T14:42:12Z","receivedAt":"2021-09-04T14:42:30Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Sat, Sep 4, 2021 at 2:37 AM René Scharfe wrote:\n>\n> Am 03.09.21 um 22:38 schrieb René Scharfe:\n> > Am 03.09.21 um 22:13 schrieb Carlo Marcelo Arenas Belón:\n> >> On Fri, Sep 03, 2021 at 08:47:02PM +0200, René Scharfe wrote:\n> >>> Am 03.09.21 um 19:02 schrieb Carlo Marcelo Arenas Belón:\n> >>>> diff --git a/compat/nedmalloc/nedmalloc.c b/compat/nedmalloc/nedmalloc.c\n> >>>> index 1cc31c3502..edb438a777 100644\n> >>>> --- a/compat/nedmalloc/nedmalloc.c\n> >>>> +++ b/compat/nedmalloc/nedmalloc.c\n> >>>> @@ -510,7 +510,7 @@ static void threadcache_free(nedpool *p, threadcache *tc, int mymspace, void *me\n> >>>> assert(idx<=THREADCACHEMAXBINS);\n> >>>> if(tck==*binsptr)\n> >>>> {\n> >>>> - fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", tck);\n> >>>> + fprintf(stderr, \"Attempt to free already freed memory block %p - aborting!\\n\", (void *)tck);\n> >>>\n> >>> This change is not mentioned in the commit message.\n> >>\n> >> got me there, I was intentionally trying to ignore it since nedmalloc gives\n> >> me PTSD and is obsoleted AFAIK[1], so just adding a casting to void (while\n> >> ugly) was also less intrusive.\n>\n> Expected your [1] to stand for a footnote, and got confused when I found none.\n> The last commit in https://github.com/ned14/nedmalloc is from seven years ago\n> and this repository is archived, with the author still being active on GitHub.\n\n> Seems like nedmalloc reached its end of life. Has there been an official\n> announcement?\n\nApologies; this is the [1] I was referring to:\n\n[1] https://lore.kernel.org/git/nycvar.QRO.7.76.6.1908082213400.46@tvgsbejva\nqbjf.bet/\n\nTLDR; nedmalloc works but is only stable in Windows, and indeed shows other\nwarnings in macOS that would have broken a DEVELOPER=1 build as well\nwhich I am ignoring.\n\n compat/nedmalloc/nedmalloc.c:326:8: warning: address of array\n'p->caches' will always evaluate to 'true' [-Wpointer-bool-conversion]\n if(p->caches)\n ~~ ~~~^~~~~~\n1 warning generated.\n\nCarlo\n"},{"id":"434718","messageId":"87o897pi7x.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"20210903170232.57646-1-carenas@gmail.com","subject":"Re: [PATCH v3 0/3] support pedantic in developer mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-05T07:54:00Z","receivedAt":"2021-09-05T07:58:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 03 2021, Carlo Marcelo Arenas Belón wrote:\n\n> This series enables pedantic mode for building when DEVELOPER=1 is\n> used and as an alternative to only enabling it in one CI job, that\n> was merged to \"seen\" as part of cb/ci-build-pedantic.\n>\n> The second patch is really an independent prerequisite to ensure\n> that it doesn't break the build for Windows and is the minimal change\n> possible.\n>\n> Additional changes needed for the git-for-windows/git fork main to be\n> posted independently.\n>\n> It merges and builds successfully all the way to \"seen\" IF the known\n> problem reported earlier[1] and expected as part of a reroll of \n> jh/builtin-fsmonitor is merged first.\n>\n> [1] https://lore.kernel.org/git/20210809063004.73736-3-carenas@gmail.com/\n>\n> Carlo Marcelo Arenas Belón (2):\n> win32: allow building with pedantic mode enabled\n> developer: enable pedantic by default\n>\n> Ævar Arnfjörð Bjarmason (1):\n> gettext: remove optional non-standard parens in N_() definition\n\nThis whole series looks good to me, thanks for picking up my patch as\nthe 1/3. The only comment I have on it (doesn't need a re-roll) is that\nI found the first paragraph in 2/3 slightly confusing, i.e.:\n \n In preparation to building with pedantic mode enabled, change a couple\n of places where the current mingw gcc compiler provided with the SDK\n reports issues.\n\nWith \"the SDK\" we're talking about the Win32 SDK, which is implicit from\nthe subject line. I'd find something like this less confusing:\n\n In preparation for building with DEVOPTS=pedantic enabled\n everywhere, change a couple of places where we'd get Win32 breakes\n under the GCC version provided wit hthe current MinGW version.\n\nOr something. I'm not sure if this /only/ impacts Win32, or just that\ncompiler version. Some of the diffstat is win32-only, but nod nedmalloc,\nbut I see there's some parallel discussion about whether that's in\neffect win32-specific.\n\nAnyway, that's all a tiny nit. In general I like the change. I also\nchecked that an existing DEVOPTS=pedantic wouldn't accidentally enable\nDEVOPTS=no-pedantic (i.e. that it wasn't a glob), but it doesn't, since\nthat's not how $(filter) works.\n"},{"id":"435441","messageId":"87h7escu9m.fsf@evledraar.gmail.com","threadId":"55998","inReplyTo":"20210903170232.57646-2-carenas@gmail.com","subject":"Re: [PATCH v3 1/3] gettext: remove optional non-standard parens in N_() definition","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-10T15:39:57Z","receivedAt":"2021-09-10T15:42:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 03 2021, Carlo Marcelo Arenas Belón wrote:\n\n> From: Ævar Arnfjörð Bjarmason \n>\n> Remove the USE_PARENS_AROUND_GETTEXT_N compile-time option which was\n> meant to catch an inadvertent mistake which is too obscure to\n> maintain this facility.\n>\n> The backstory of how USE_PARENS_AROUND_GETTEXT_N came about is: When I\n> added the N_() macro in 65784830366 (i18n: add no-op _() and N_()\n> wrappers, 2011-02-22) it was defined as:\n>\n> #define N_(msgid) (msgid)\n>\n> This is non-standard C, as was noticed and fixed in 642f85faab2 (i18n:\n> avoid parenthesized string as array initializer, 2011-04-07).\n> I.e. this needed to be defined as:\n>\n> #define N_(msgid) msgid\n>\n> Then in e62cd35a3e8 (i18n: log: mark parseopt strings for translation,\n> 2012-08-20) when \"builtin_log_usage\" was marked for translation the\n> string concatenation for passing to usage() added in 1c370ea4e51\n> (Show usage string for 'git log -h', 'git show -h' and 'git diff -h',\n> 2009-08-06) was faithfully preserved:\n>\n> - \"git log [] [..] [[--] ...]\\n\"\n> - \" or: git show [options] ...\",\n> + N_(\"git log [] [..] [[--] ...]\\n\")\n> + N_(\" or: git show [options] ...\"),\n>\n> This was then fixed to be the expected array of usage strings in\n> e66dc0cc4b1 (log.c: fix translation markings, 2015-01-06) rather than\n> a string with multiple \"\\n\"-delimited usage strings, and finally in\n> 290c8e7a3fe (gettext.h: add parentheses around N_ expansion if\n> supported, 2015-01-11) USE_PARENS_AROUND_GETTEXT_N was added to ensure\n> this mistake didn't happen again.\n>\n> I think that even if this was a N_()-specific issue this\n> USE_PARENS_AROUND_GETTEXT_N facility wouldn't be worth it, the issue\n> would be too rare to worry about.\n>\n> But I also think that 290c8e7a3fe which introduced\n> USE_PARENS_AROUND_GETTEXT_N misattributed the problem. The issue\n> wasn't with the N_() macro added in e62cd35a3e8, but that before the\n> N_() macro existed in the codebase the initial migration to\n> parse_options() in 1c370ea4e51 continued passsing in a \"\\n\"-delimited\n> string, when the new API it was migrating to supported and expected\n> the passing of an array.\n>\n> Helped-by: Eric Sunshine \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Carlo Marcelo Arenas Belón \n> ---\n> Makefile | 20 +-------------------\n> config.mak.dev | 2 --\n> gettext.h | 24 ------------------------\n> git-compat-util.h | 4 ----\n> 4 files changed, 1 insertion(+), 49 deletions(-)\n>\n> diff --git a/Makefile b/Makefile\n> index 9573190f1d..4e94073c2a 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -409,15 +409,6 @@ all::\n> # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version\n> # before 2.17) for clock_gettime and CLOCK_MONOTONIC.\n> #\n> -# Define USE_PARENS_AROUND_GETTEXT_N to \"yes\" if your compiler happily\n> -# compiles the following initialization:\n> -#\n> -# static const char s[] = (\"FOO\");\n> -#\n> -# and define it to \"no\" if you need to remove the parentheses () around the\n> -# constant. The default is \"auto\", which means to use parentheses if your\n> -# compiler is detected to support it.\n> -#\n> # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n> #\n> # Define HAVE_GETDELIM if your system has the getdelim() function.\n> @@ -497,8 +488,7 @@ all::\n> #\n> # pedantic:\n> #\n> -# Enable -pedantic compilation. This also disables\n> -# USE_PARENS_AROUND_GETTEXT_N to produce only relevant warnings.\n> +# Enable -pedantic compilation.\n> \n> GIT-VERSION-FILE: FORCE\n> \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n> @@ -1347,14 +1337,6 @@ ifneq (,$(SOCKLEN_T))\n> \tBASIC_CFLAGS += -Dsocklen_t=$(SOCKLEN_T)\n> endif\n> \n> -ifeq (yes,$(USE_PARENS_AROUND_GETTEXT_N))\n> -\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=1\n> -else\n> -ifeq (no,$(USE_PARENS_AROUND_GETTEXT_N))\n> -\tBASIC_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n> -endif\n> -endif\n> -\n> ifeq ($(uname_S),Darwin)\n> \tifndef NO_FINK\n> \t\tifeq ($(shell test -d /sw/lib && echo y),y)\n> diff --git a/config.mak.dev b/config.mak.dev\n> index 022fb58218..41d6345bc0 100644\n> --- a/config.mak.dev\n> +++ b/config.mak.dev\n> @@ -4,8 +4,6 @@ SPARSE_FLAGS += -Wsparse-error\n> endif\n> ifneq ($(filter pedantic,$(DEVOPTS)),)\n> DEVELOPER_CFLAGS += -pedantic\n> -# don't warn for each N_ use\n> -DEVELOPER_CFLAGS += -DUSE_PARENS_AROUND_GETTEXT_N=0\n> endif\n> DEVELOPER_CFLAGS += -Wall\n> DEVELOPER_CFLAGS += -Wdeclaration-after-statement\n> diff --git a/gettext.h b/gettext.h\n> index c8b34fd612..d209911ebb 100644\n> --- a/gettext.h\n> +++ b/gettext.h\n> @@ -55,31 +55,7 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n> }\n> \n> /* Mark msgid for translation but do not translate it. */\n> -#if !USE_PARENS_AROUND_GETTEXT_N\n> #define N_(msgid) msgid\n> -#else\n> -/*\n> - * Strictly speaking, this will lead to invalid C when\n> - * used this way:\n> - *\tstatic const char s[] = N_(\"FOO\");\n> - * which will expand to\n> - *\tstatic const char s[] = (\"FOO\");\n> - * and in valid C, the initializer on the right hand side must\n> - * be without the parentheses. But many compilers do accept it\n> - * as a language extension and it will allow us to catch mistakes\n> - * like:\n> - *\tstatic const char *msgs[] = {\n> - *\t\tN_(\"one\")\n> - *\t\tN_(\"two\"),\n> - *\t\tN_(\"three\"),\n> - *\t\tNULL\n> - *\t};\n> - * (notice the missing comma on one of the lines) by forcing\n> - * a compilation error, because parenthesised (\"one\") (\"two\")\n> - * will not get silently turned into (\"onetwo\").\n> - */\n> -#define N_(msgid) (msgid)\n> -#endif\n> \n> const char *get_preferred_languages(void);\n> int is_utf8_locale(void);\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index b46605300a..ddc65ff61d 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -1253,10 +1253,6 @@ int warn_on_fopen_errors(const char *path);\n> */\n> int open_nofollow(const char *path, int flags);\n> \n> -#if !defined(USE_PARENS_AROUND_GETTEXT_N) && defined(__GNUC__)\n> -#define USE_PARENS_AROUND_GETTEXT_N 1\n> -#endif\n> -\n> #ifndef SHELL_PATH\n> # define SHELL_PATH \"/bin/sh\"\n> #endif\n\nA note & cross-link: I've submitted a v2 of another series where we'll\neffectively duplicate the check being removed here, see\nhttps://lore.kernel.org/git/cover-v2-0.6-00000000000-20210910T153146Z-avarab@gmail.com/\n\nWell, it's not a general N_() multi-line checker like the proposed\nhttps://lore.kernel.org/git/20210901091941.34886-4-carenas@gmail.com/\nand this USE_PARENS_AROUND_GETTEXT_N, but we added\nUSE_PARENS_AROUND_GETTEXT_N to begin with for these usage strings. The\nbad usage of that usage API (phew!, that's a mouthful) will be caught by\nthat usage CAPI being stricter now.\n"},{"id":"437202","messageId":"20210927230438.3759964-1-jonathantanmy@google.com","threadId":"55998","inReplyTo":"20210903170232.57646-3-carenas@gmail.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-27T23:04:38Z","receivedAt":"2021-09-27T23:04:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +ifneq ($(filter gcc5,$(COMPILER_FEATURES)),)\n> +DEVELOPER_CFLAGS += -Wno-incompatible-pointer-types\n> +endif\n\nI noticed today that I wasn't warned about some incompatible function\npointer signatures (that I expected to be warned about) due to this\nline - could the condition of adding this compiler flag be further\nnarrowed down? gcc -v says:\n\n gcc version 10.3.0 (Debian 10.3.0-9+build2) \n\nOn my system, if I remove that line, \"make DEVELOPER=1\" is still\nsuccessful.\n"},{"id":"437215","messageId":"CAPUEsphk9b0TpUDgW9qkG=ehKx+hPi5GNtqTP2o2MeL1VpHHPQ@mail.gmail.com","threadId":"55998","inReplyTo":"20210927230438.3759964-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-09-28T00:30:34Z","receivedAt":"2021-09-28T00:30:48Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Mon, Sep 27, 2021 at 4:04 PM Jonathan Tan wrote:\n>\n> > +ifneq ($(filter gcc5,$(COMPILER_FEATURES)),)\n> > +DEVELOPER_CFLAGS += -Wno-incompatible-pointer-types\n> > +endif\n>\n> I noticed today that I wasn't warned about some incompatible function\n> pointer signatures (that I expected to be warned about) due to this\n> line - could the condition of adding this compiler flag be further\n> narrowed down? gcc -v says:\n\nApologies; it is gone already in \"seen\" (and hopefully soon in \"next\")\nby merging js/win-lazyload-buildfix[1]\n\n> gcc version 10.3.0 (Debian 10.3.0-9+build2)\n>\n> On my system, if I remove that line, \"make DEVELOPER=1\" is still\n> successful.\n\nCorrect; it was only needed in Windows, will narrow it further.\n\nCarlo\n\n[1] https://github.com/gitster/git/tree/js/win-lazyload-buildfix\n"},{"id":"437327","messageId":"20210928165005.228922-1-jonathantanmy@google.com","threadId":"55998","inReplyTo":"CAPUEsphk9b0TpUDgW9qkG=ehKx+hPi5GNtqTP2o2MeL1VpHHPQ@mail.gmail.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T16:50:04Z","receivedAt":"2021-09-28T16:50:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Apologies; it is gone already in \"seen\" (and hopefully soon in \"next\")\n> by merging js/win-lazyload-buildfix[1]\n\nAh, thanks for having fixed this.\n"},{"id":"437337","messageId":"xmqq35polhye.fsf@gitster.g","threadId":"55998","inReplyTo":"CAPUEsphk9b0TpUDgW9qkG=ehKx+hPi5GNtqTP2o2MeL1VpHHPQ@mail.gmail.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T17:37:29Z","receivedAt":"2021-09-28T17:37:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Carlo Arenas writes:\n\n> On Mon, Sep 27, 2021 at 4:04 PM Jonathan Tan wrote:\n>>\n>> > +ifneq ($(filter gcc5,$(COMPILER_FEATURES)),)\n>> > +DEVELOPER_CFLAGS += -Wno-incompatible-pointer-types\n>> > +endif\n>>\n>> I noticed today that I wasn't warned about some incompatible function\n>> pointer signatures (that I expected to be warned about) due to this\n>> line - could the condition of adding this compiler flag be further\n>> narrowed down? gcc -v says:\n>\n> Apologies; it is gone already in \"seen\" (and hopefully soon in \"next\")\n> by merging js/win-lazyload-buildfix[1]\n\nI will mark it not ready for 'next', while waiting for a fix-up.\nThanks for stopping me.\n"},{"id":"437359","messageId":"20210928201613.1110573-1-jonathantanmy@google.com","threadId":"55998","inReplyTo":"xmqq35polhye.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:16:13Z","receivedAt":"2021-09-28T20:16:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Carlo Arenas writes:\n> \n> > On Mon, Sep 27, 2021 at 4:04 PM Jonathan Tan wrote:\n> >>\n> >> > +ifneq ($(filter gcc5,$(COMPILER_FEATURES)),)\n> >> > +DEVELOPER_CFLAGS += -Wno-incompatible-pointer-types\n> >> > +endif\n> >>\n> >> I noticed today that I wasn't warned about some incompatible function\n> >> pointer signatures (that I expected to be warned about) due to this\n> >> line - could the condition of adding this compiler flag be further\n> >> narrowed down? gcc -v says:\n> >\n> > Apologies; it is gone already in \"seen\" (and hopefully soon in \"next\")\n> > by merging js/win-lazyload-buildfix[1]\n> \n> I will mark it not ready for 'next', while waiting for a fix-up.\n> Thanks for stopping me.\n\nJust checking - which branch is not ready for next? The issue I noticed\nis already in master, and js/win-lazyload-buildfix contains the fix for\nthe issue (which ideally would be merged as soon as possible, but\nmerging according to the usual schedule is fine).\n"},{"id":"437408","messageId":"CAPUEsphVJBCPLSfOyH2bqTCpcDvtPuOVGQsD3XaWVuZbFiVUeA@mail.gmail.com","threadId":"55998","inReplyTo":"20210928201613.1110573-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-09-29T01:00:36Z","receivedAt":"2021-09-29T01:00:53Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Tue, Sep 28, 2021 at 1:16 PM Jonathan Tan wrote:\n>\n> > Carlo Arenas writes:\n> >\n> > > Apologies; it is gone already in \"seen\" (and hopefully soon in \"next\")\n> > > by merging js/win-lazyload-buildfix[1]\n> >\n> > I will mark it not ready for 'next', while waiting for a fix-up.\n> > Thanks for stopping me.\n>\n> Just checking - which branch is not ready for next? The issue I noticed\n> is already in master, and js/win-lazyload-buildfix contains the fix for\n> the issue (which ideally would be merged as soon as possible, but\n> merging according to the usual schedule is fine).\n\nMy guess was that he meant to have also the windows specific check\nadded as part of this branch, and that would have prevented the issue\nyou reported originally as well.\n\nEither way, a v4 has been posted[1] (sorry, forgot to CC you), and\nhopefully that is now ready for next ;)\n\nCarlo\n\n[1] https://lore.kernel.org/git/20210929004832.96304-1-carenas@gmail.com/\n"},{"id":"437463","messageId":"xmqq35pnfkax.fsf@gitster.g","threadId":"55998","inReplyTo":"CAPUEsphVJBCPLSfOyH2bqTCpcDvtPuOVGQsD3XaWVuZbFiVUeA@mail.gmail.com","subject":"Re: [PATCH v3 2/3] win32: allow building with pedantic mode enabled","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-29T15:55:34Z","receivedAt":"2021-09-29T15:55:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Carlo Arenas writes:\n\n> Either way, a v4 has been posted[1] (sorry, forgot to CC you), and\n> hopefully that is now ready for next ;)\n\nThanks, both.\n"}]}