{"thread":{"id":"55053","subject":"[PATCH 02/17] chunk-format: create chunk format write API","startedAt":"2021-01-26T16:02:33Z","lastAt":"2021-02-24T19:44:58Z","messageCount":120,"participants":["Derrick Stolee via GitGitGadget","Taylor Blau","Chris Torek","Junio C Hamano","Derrick Stolee","SZEDER Gábor"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"415289","messageId":"9bd273f8c94fdb0c3adf8aedef3480ff5f4232b8.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:11Z","receivedAt":"2021-01-26T16:02:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of combining the logic from the commit-graph and\nmulti-pack-index file formats, create a new chunk-format API. Use a\n'struct chunkfile' pointer to keep track of data that has been\nregistered for writes. This struct is anonymous outside of\nchunk-format.c to ensure no user attempts to interfere with the data.\n\nThe next change will use this API in commit-graph.c, but the general\napproach is:\n\n 1. initialize the chunkfile with init_chunkfile(f).\n 2. add chunks in the intended writing order with add_chunk().\n 3. write any header information to the hashfile f.\n 4. write the chunkfile data using write_chunkfile().\n 5. free the chunkfile struct using free_chunkfile().\n\nHelped-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n chunk-format.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 20 +++++++++++\n 3 files changed, 112 insertions(+)\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\ndiff --git a/Makefile b/Makefile\nindex 7b64106930a..50a7663841e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -854,6 +854,7 @@ LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n+LIB_OBJS += chunk-format.o\n LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\ndiff --git a/chunk-format.c b/chunk-format.c\nnew file mode 100644\nindex 00000000000..2ce37ecc6bb\n--- /dev/null\n+++ b/chunk-format.c\n@@ -0,0 +1,91 @@\n+#include \"cache.h\"\n+#include \"chunk-format.h\"\n+#include \"csum-file.h\"\n+#define CHUNK_LOOKUP_WIDTH 12\n+\n+/*\n+ * When writing a chunk-based file format, collect the chunks in\n+ * an array of chunk_info structs. The size stores the _expected_\n+ * amount of data that will be written by write_fn.\n+ */\n+struct chunk_info {\n+\tuint32_t id;\n+\tuint64_t size;\n+\tchunk_write_fn write_fn;\n+};\n+\n+struct chunkfile {\n+\tstruct hashfile *f;\n+\n+\tstruct chunk_info *chunks;\n+\tsize_t chunks_nr;\n+\tsize_t chunks_alloc;\n+};\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f)\n+{\n+\tstruct chunkfile *cf = xcalloc(1, sizeof(*cf));\n+\tcf->f = f;\n+\treturn cf;\n+}\n+\n+void free_chunkfile(struct chunkfile *cf)\n+{\n+\tif (!cf)\n+\t\treturn;\n+\tfree(cf->chunks);\n+\tfree(cf);\n+}\n+\n+int get_num_chunks(struct chunkfile *cf)\n+{\n+\treturn cf->chunks_nr;\n+}\n+\n+void add_chunk(struct chunkfile *cf,\n+\t uint64_t id,\n+\t chunk_write_fn fn,\n+\t size_t size)\n+{\n+\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n+\n+\tcf->chunks[cf->chunks_nr].id = id;\n+\tcf->chunks[cf->chunks_nr].write_fn = fn;\n+\tcf->chunks[cf->chunks_nr].size = size;\n+\tcf->chunks_nr++;\n+}\n+\n+int write_chunkfile(struct chunkfile *cf, void *data)\n+{\n+\tint i;\n+\tsize_t cur_offset = cf->f->offset + cf->f->total;\n+\n+\t/* Add the table of contents to the current offset */\n+\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n+\t\thashwrite_be64(cf->f, cur_offset);\n+\n+\t\tcur_offset += cf->chunks[i].size;\n+\t}\n+\n+\t/* Trailing entry marks the end of the chunks */\n+\thashwrite_be32(cf->f, 0);\n+\thashwrite_be64(cf->f, cur_offset);\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tuint64_t start_offset = cf->f->total + cf->f->offset;\n+\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\n+\t\tif (cf->f->total + cf->f->offset != start_offset + cf->chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n+\t\t\t cf->f->total + cf->f->offset - start_offset);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nnew file mode 100644\nindex 00000000000..bfaed672813\n--- /dev/null\n+++ b/chunk-format.h\n@@ -0,0 +1,20 @@\n+#ifndef CHUNK_FORMAT_H\n+#define CHUNK_FORMAT_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct hashfile;\n+struct chunkfile;\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f);\n+void free_chunkfile(struct chunkfile *cf);\n+int get_num_chunks(struct chunkfile *cf);\n+typedef int (*chunk_write_fn)(struct hashfile *f,\n+\t\t\t void *data);\n+void add_chunk(struct chunkfile *cf,\n+\t uint64_t id,\n+\t chunk_write_fn fn,\n+\t size_t size);\n+int write_chunkfile(struct chunkfile *cf, void *data);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"415290","messageId":"pull.848.git.1611676886.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":null,"subject":"[PATCH 00/17] Refactor chunk-format into an API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:09Z","receivedAt":"2021-01-26T16:03:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a restart on the topic previously submitted [1] but dropped because\nak/corrected-commit-date was still in progress. This version is based on\nthat branch.\n\n[1]\nhttps://lore.kernel.org/git/pull.804.git.1607012215.gitgitgadget@gmail.com/\n\nThis version also changes the approach to use a more dynamic interaction\nwith a struct chunkfile pointer. This idea is credited to Taylor Blau [2],\nbut I started again from scratch. I also go further to make struct chunkfile\nanonymous to API consumers. It is defined only in chunk-format.c, which\nshould hopefully deter future users from interacting with that data\ndirectly.\n\n[2] https://lore.kernel.org/git/X8%2FI%2FRzXZksio+ri@nand.local/\n\nThis combined API is beneficial to reduce duplicated logic. Or rather, to\nensure that similar file formats have similar protections against bad data.\nThe multi-pack-index code did not have as many guards as the commit-graph\ncode did, but now they both share a common base that checks for things like\nduplicate chunks or offsets outside the size of the file.\n\nHere are some stats for the end-to-end change:\n\n * 638 insertions(+), 456 deletions(-).\n * commit-graph.c: 171 insertions(+), 192 deletions(-)\n * midx.c: 196 insertions(+), 260 deletions(-)\n\nWhile there is an overall increase to the code size, the consumers do get a\nbit smaller. Boilerplate things like abstracting method to match\nchunk_write_fn and chunk_read_fn make up a lot of these insertions. The\n\"interesting\" code gets a lot smaller and cleaner.\n\nThanks, -Stolee\n\nDerrick Stolee (17):\n commit-graph: anonymize data in chunk_write_fn\n chunk-format: create chunk format write API\n commit-graph: use chunk-format write API\n midx: rename pack_info to write_midx_context\n midx: use context in write_midx_pack_names()\n midx: add entries to write_midx_context\n midx: add pack_perm to write_midx_context\n midx: add num_large_offsets to write_midx_context\n midx: return success/failure in chunk write methods\n midx: drop chunk progress during write\n midx: use chunk-format API in write_midx_internal()\n chunk-format: create read chunk API\n commit-graph: use chunk-format read API\n midx: use chunk-format read API\n midx: use 64-bit multiplication for chunk sizes\n chunk-format: restore duplicate chunk checks\n chunk-format: add technical docs\n\n Documentation/technical/chunk-format.txt | 54 +++\n .../technical/commit-graph-format.txt | 3 +\n Documentation/technical/pack-format.txt | 3 +\n Makefile | 1 +\n chunk-format.c | 165 +++++++\n chunk-format.h | 41 ++\n commit-graph.c | 363 +++++++-------\n midx.c | 456 ++++++++----------\n t/t5318-commit-graph.sh | 2 +-\n t/t5319-multi-pack-index.sh | 6 +-\n 10 files changed, 638 insertions(+), 456 deletions(-)\n create mode 100644 Documentation/technical/chunk-format.txt\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\n\nbase-commit: 5a3b130cad0d5c770f766e3af6d32b41766374c0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-848%2Fderrickstolee%2Fchunk-format%2Frefactor-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-848/derrickstolee/chunk-format/refactor-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/848\n-- \ngitgitgadget\n"},{"id":"415291","messageId":"09b32829e4ff2384cb35afaf1a34385e25bac8d8.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 01/17] commit-graph: anonymize data in chunk_write_fn","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:10Z","receivedAt":"2021-01-26T16:03:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for creating an API around file formats using chunks and\ntables of contents, prepare the commit-graph write code to use\nprototypes that will match this new API.\n\nSpecifically, convert chunk_write_fn to take a \"void *data\" parameter\ninstead of the commit-graph-specific \"struct write_commit_graph_context\"\npointer.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 38 ++++++++++++++++++++++++++++----------\n 1 file changed, 28 insertions(+), 10 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f3bde2ad95a..b26ed72396e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1040,8 +1040,10 @@ struct write_commit_graph_context {\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n \n@@ -1066,8 +1068,10 @@ static int write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static int write_graph_chunk_oids(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n@@ -1085,8 +1089,10 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static int write_graph_chunk_data(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n@@ -1187,8 +1193,10 @@ static int write_graph_chunk_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tint i, num_generation_data_overflows = 0;\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n@@ -1208,8 +1216,10 @@ static int write_graph_chunk_generation_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n-\t\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tint i;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1226,8 +1236,10 @@ static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n }\n \n static int write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n@@ -1280,8 +1292,10 @@ static int write_graph_chunk_extra_edges(struct hashfile *f,\n }\n \n static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t cur_pos = 0;\n@@ -1315,8 +1329,10 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n }\n \n static int write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\tstruct write_commit_graph_context *ctx)\n+\t\t\t\t\tvoid *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \n@@ -1737,8 +1753,10 @@ static int write_graph_chunk_base_1(struct hashfile *f,\n }\n \n static int write_graph_chunk_base(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx =\n+\t\t(struct write_commit_graph_context *)data;\n \tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n \n \tif (num != ctx->num_commit_graphs_after - 1) {\n@@ -1750,7 +1768,7 @@ static int write_graph_chunk_base(struct hashfile *f,\n }\n \n typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t struct write_commit_graph_context *ctx);\n+\t\t\t void *data);\n \n struct chunk_info {\n \tuint32_t id;\n-- \ngitgitgadget\n\n"},{"id":"415292","messageId":"14a0246b98257cf0eb00de88b1a04409fc60138f.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 05/17] midx: use context in write_midx_pack_names()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:14Z","receivedAt":"2021-01-26T16:03:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align the write_midx_internal() to use the chunk-format\nAPI, start converting chunk writing methods to match chunk_write_fn. The\nfirst case is to convert write_midx_pack_names() to take \"void *data\".\nWe already have the necessary data in \"struct write_midx_context\", so\nthis conversion is rather mechanical.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 21 ++++++++++-----------\n 1 file changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dfc1a289246..f348a70e018 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -643,27 +643,26 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t struct pack_info *info,\n-\t\t\t\t uint32_t num_packs)\n+static size_t write_midx_pack_names(struct hashfile *f, void *data)\n {\n+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \tuint32_t i;\n \tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < num_packs; i++) {\n+\tfor (i = 0; i < ctx->nr; i++) {\n \t\tsize_t writelen;\n \n-\t\tif (info[i].expired)\n+\t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n+\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t info[i - 1].pack_name,\n-\t\t\t info[i].pack_name);\n+\t\t\t ctx->info[i - 1].pack_name,\n+\t\t\t ctx->info[i].pack_name);\n \n-\t\twritelen = strlen(info[i].pack_name) + 1;\n-\t\thashwrite(f, info[i].pack_name, writelen);\n+\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n+\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -990,7 +989,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-- \ngitgitgadget\n\n"},{"id":"415293","messageId":"79f479ef7d16bd5fd4be058eed5fe9d06291fd65.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 06/17] midx: add entries to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:15Z","receivedAt":"2021-01-26T16:03:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"struct pack_midx_entry *entries\" list and its count\ninto the context.\n\nUpdate write_midx_oid_fanout() and write_midx_oid_lookup() to take the\ncontext directly, as these are easy conversions with this new data.\n\nOnly the callers of write_midx_object_offsets() and\nwrite_midx_large_offsets() are updated here, since additional data in\nthe context before those methods can match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 49 ++++++++++++++++++++++++++-----------------------\n 1 file changed, 26 insertions(+), 23 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f348a70e018..ee6f3504c6a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -458,6 +458,9 @@ struct write_midx_context {\n \tstruct multi_pack_index *m;\n \tstruct progress *progress;\n \tunsigned pack_paths_checked;\n+\n+\tstruct pack_midx_entry *entries;\n+\tuint32_t entries_nr;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -678,11 +681,11 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n }\n \n static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n-\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n \tuint32_t count = 0;\n \tuint32_t i;\n \n@@ -706,18 +709,19 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \treturn MIDX_CHUNK_FANOUT_SIZE;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+static size_t write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n+\tunsigned char hash_len = the_hash_algo->rawsz;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (i < nr_objects - 1) {\n+\t\tif (i < ctx->entries_nr - 1) {\n \t\t\tstruct pack_midx_entry *next = list;\n \t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n \t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n@@ -805,8 +809,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries, num_large_offsets = 0;\n-\tstruct pack_midx_entry *entries = NULL;\n+\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n@@ -852,12 +855,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n-\tfor (i = 0; i < nr_entries; i++) {\n-\t\tif (entries[i].offset > 0x7fffffff)\n+\tfor (i = 0; i < ctx.entries_nr; i++) {\n+\t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n-\t\tif (entries[i].offset > 0xffffffff)\n+\t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n@@ -947,10 +950,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * the_hash_algo->rawsz;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n \n \tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n \tif (large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n@@ -993,19 +996,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, the_hash_algo->rawsz, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1035,7 +1038,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tfree(ctx.info);\n-\tfree(entries);\n+\tfree(ctx.entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n \treturn result;\n-- \ngitgitgadget\n\n"},{"id":"415294","messageId":"9fe5ee8611c3a18daaa9701bb5eefc8c408a7e76.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 04/17] midx: rename pack_info to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:13Z","receivedAt":"2021-01-26T16:03:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to streamline our chunk-based file formats, align some of\nthe code structure in write_midx_internal() to be similar to the\npatterns in write_commit_graph_file().\n\nSpecifically, let's create a \"struct write_midx_context\" that can be\nused as a data parameter to abstract function types.\n\nThis change only renames \"struct pack_info\" to \"struct\nwrite_midx_context\" and the names of instances from \"packs\" to \"ctx\". In\nfuture changes, we will expand the data inside \"struct\nwrite_midx_context\" and align our chunk-writing method with the\nchunk-format API.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 130 ++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 65 insertions(+), 65 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 79c282b070d..dfc1a289246 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -451,7 +451,7 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n-struct pack_list {\n+struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n \tuint32_t alloc;\n@@ -463,37 +463,37 @@ struct pack_list {\n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t const char *file_name, void *data)\n {\n-\tstruct pack_list *packs = (struct pack_list *)data;\n+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \n \tif (ends_with(file_name, \".idx\")) {\n-\t\tdisplay_progress(packs->progress, ++packs->pack_paths_checked);\n-\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n+\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n-\t\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t\t 0);\n+\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->info[packs->nr].p) {\n+\t\tif (!ctx->info[ctx->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n+\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->info[packs->nr].p);\n-\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n+\t\t\tclose_pack(ctx->info[ctx->nr].p);\n+\t\t\tFREE_AND_NULL(ctx->info[ctx->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n-\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n-\t\tpacks->info[packs->nr].expired = 0;\n-\t\tpacks->nr++;\n+\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n+\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n+\t\tctx->info[ctx->nr].expired = 0;\n+\t\tctx->nr++;\n \t}\n }\n \n@@ -801,7 +801,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n-\tstruct pack_list packs;\n+\tstruct write_midx_context ctx = { 0 };\n \tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n@@ -820,40 +820,40 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t midx_name);\n \n \tif (m)\n-\t\tpacks.m = m;\n+\t\tctx.m = m;\n \telse\n-\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n-\n-\tpacks.nr = 0;\n-\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n-\tpacks.info = NULL;\n-\tALLOC_ARRAY(packs.info, packs.alloc);\n-\n-\tif (packs.m) {\n-\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n-\n-\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n-\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.info[packs.nr].p = NULL;\n-\t\t\tpacks.info[packs.nr].expired = 0;\n-\t\t\tpacks.nr++;\n+\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\n+\tctx.nr = 0;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.info = NULL;\n+\tALLOC_ARRAY(ctx.info, ctx.alloc);\n+\n+\tif (ctx.m) {\n+\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n+\n+\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n+\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n+\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].expired = 0;\n+\t\t\tctx.nr++;\n \t\t}\n \t}\n \n-\tpacks.pack_paths_checked = 0;\n+\tctx.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n-\t\tpacks.progress = NULL;\n+\t\tctx.progress = NULL;\n \n-\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n-\tstop_progress(&packs.progress);\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n+\tstop_progress(&ctx.progress);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -862,19 +862,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n-\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n \tif (packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n-\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n \t\t\t\t\t packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n-\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n \t\t\t\t packs_to_drop->items[drop_index].string);\n@@ -882,7 +882,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n \t\t\t} else {\n-\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t\tctx.info[i].expired = 0;\n \t\t\t}\n \t\t}\n \n@@ -898,19 +898,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].expired) {\n+\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (!packs.info[i].expired)\n-\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (!ctx.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n@@ -921,19 +921,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\tif (packs.m)\n-\t\tclose_midx(packs.m);\n+\tif (ctx.m)\n+\t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\tif (packs.nr - dropped_packs == 0) {\n+\tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n+\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -990,7 +990,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -1027,15 +1027,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tcommit_lock_file(&lk);\n \n cleanup:\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].p) {\n-\t\t\tclose_pack(packs.info[i].p);\n-\t\t\tfree(packs.info[i].p);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].p) {\n+\t\t\tclose_pack(ctx.info[i].p);\n+\t\t\tfree(ctx.info[i].p);\n \t\t}\n-\t\tfree(packs.info[i].pack_name);\n+\t\tfree(ctx.info[i].pack_name);\n \t}\n \n-\tfree(packs.info);\n+\tfree(ctx.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \ngitgitgadget\n\n"},{"id":"415295","messageId":"a3d6177a352643721fdc07512629b48d1213157a.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 03/17] commit-graph: use chunk-format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:12Z","receivedAt":"2021-01-26T16:04:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph write logic is ready to make use of the chunk-format\nwrite API. Each chunk write method is already in the correct prototype.\nWe only need to use the 'struct chunkfile' pointer and the correct API\ncalls.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 118 ++++++++++++++++---------------------------------\n 1 file changed, 37 insertions(+), 81 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b26ed72396e..b2c0f233eab 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -19,6 +19,7 @@\n #include \"shallow.h\"\n #include \"json-writer.h\"\n #include \"trace2.h\"\n+#include \"chunk-format.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -1767,27 +1768,17 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t void *data);\n-\n-struct chunk_info {\n-\tuint32_t id;\n-\tuint64_t size;\n-\tchunk_write_fn write_fn;\n-};\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n \tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tstruct chunk_info chunks[MAX_NUM_CHUNKS + 1];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n-\tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n+\tstruct chunkfile *cf;\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1833,76 +1824,50 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \t}\n \n-\tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunks[0].size = GRAPH_FANOUT_SIZE;\n-\tchunks[0].write_fn = write_graph_chunk_fanout;\n-\tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunks[1].size = hashsz * ctx->commits.nr;\n-\tchunks[1].write_fn = write_graph_chunk_oids;\n-\tchunks[2].id = GRAPH_CHUNKID_DATA;\n-\tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n-\tchunks[2].write_fn = write_graph_chunk_data;\n+\tcf = init_chunkfile(f);\n+\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n+\t\t write_graph_chunk_fanout, GRAPH_FANOUT_SIZE);\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP,\n+\t\t write_graph_chunk_oids, hashsz * ctx->commits.nr);\n+\tadd_chunk(cf, GRAPH_CHUNKID_DATA,\n+\t\t write_graph_chunk_data, (hashsz + 16) * ctx->commits.nr);\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_NO_GDAT, 0))\n \t\tctx->write_generation_data = 0;\n-\tif (ctx->write_generation_data) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_generation_data_overflows) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW;\n-\t\tchunks[num_chunks].size = sizeof(timestamp_t) * ctx->num_generation_data_overflows;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data_overflow;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_extra_edges) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n-\t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n-\t\tnum_chunks++;\n-\t}\n+\tif (ctx->write_generation_data)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t\t write_graph_chunk_generation_data,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr);\n+\tif (ctx->num_generation_data_overflows)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t\t write_graph_chunk_generation_data_overflow,\n+\t\t\t sizeof(timestamp_t) * ctx->num_generation_data_overflows);\n+\tif (ctx->num_extra_edges)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES,\n+\t\t\t write_graph_chunk_extra_edges,\n+\t\t\t 4 * ctx->num_extra_edges);\n \tif (ctx->changed_paths) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n-\t\tnum_chunks++;\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n-\t\t\t\t\t + ctx->total_bloom_filter_data_size;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n-\t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n-\t\tnum_chunks++;\n-\t}\n-\n-\tchunks[num_chunks].id = 0;\n-\tchunks[num_chunks].size = 0;\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t write_graph_chunk_bloom_indexes,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr);\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t write_graph_chunk_bloom_data,\n+\t\t\t sizeof(uint32_t) * 3\n+\t\t\t\t+ ctx->total_bloom_filter_data_size);\n+\t}\n+\tif (ctx->num_commit_graphs_after > 1)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BASE,\n+\t\t\t write_graph_chunk_base,\n+\t\t\t hashsz * (ctx->num_commit_graphs_after - 1));\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n-\tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\n-\t\tchunk_offset += chunks[i].size;\n-\t}\n-\n \tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n@@ -1914,17 +1879,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tuint64_t start_offset = f->total + f->offset;\n-\n-\t\tif (chunks[i].write_fn(f, ctx))\n-\t\t\treturn -1;\n-\n-\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n-\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n-\t\t\t chunks[i].size, chunks[i].id,\n-\t\t\t f->total + f->offset - start_offset);\n-\t}\n+\twrite_chunkfile(cf, ctx);\n \n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n@@ -1941,6 +1896,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tclose_commit_graph(ctx->r->objects);\n \tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfree_chunkfile(cf);\n \n \tif (ctx->split) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n-- \ngitgitgadget\n\n"},{"id":"415296","messageId":"cb145e0e32afed99b9bfa822c76f48bee18885ba.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 14/17] midx: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:23Z","receivedAt":"2021-01-26T16:04:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). In particular, we\ncan use the return value of pair_chunk() to generate an error when a\nrequired chunk is missing.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 103 ++++++++++++++++++++----------------\n t/t5319-multi-pack-index.sh | 6 +--\n 2 files changed, 60 insertions(+), 49 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 0bfd2d802b6..dd019c00795 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -54,6 +54,51 @@ static char *get_midx_filename(const char *object_dir)\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+static int midx_read_pack_names(const unsigned char *chunk_start,\n+\t\t\t\tsize_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n+\tm->chunk_pack_names = chunk_start;\n+\treturn 0;\n+}\n+\n+static int midx_read_oid_fanout(const unsigned char *chunk_start,\n+\t\t\t\tsize_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n+\tm->chunk_oid_fanout = (uint32_t *)chunk_start;\n+\n+\tif (chunk_size != 4 * 256) {\n+\t\terror(_(\"multi-pack-index OID fanout is of the wrong size\"));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int midx_read_oid_lookup(const unsigned char *chunk_start,\n+\t\t\t\tsize_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n+\tm->chunk_oid_lookup = chunk_start;\n+\treturn 0;\n+}\n+\n+static int midx_read_offsets(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n+\tm->chunk_object_offsets = chunk_start;\n+\treturn 0;\n+}\n+\n+static int midx_read_large_offsets(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n+\tm->chunk_large_offsets = chunk_start;\n+\treturn 0;\n+}\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\n@@ -65,6 +110,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n \tconst char *cur_pack_name;\n+\tstruct chunkfile *cf = NULL;\n \n \tfd = git_open(midx_name);\n \n@@ -114,58 +160,23 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n-\tfor (i = 0; i < m->num_chunks; i++) {\n-\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n-\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n-\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\n-\t\tif (chunk_offset >= m->data_len)\n-\t\t\tdie(_(\"invalid chunk offset (too large)\"));\n-\n-\t\tswitch (chunk_id) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase 0:\n-\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\t/*\n-\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n-\t\t\t\t * extensions to add optional chunks.\n-\t\t\t\t */\n-\t\t\t\tbreak;\n-\t\t}\n-\t}\n+\tcf = init_chunkfile(NULL);\n \n-\tif (!m->chunk_pack_names)\n+\tif (read_table_of_contents(cf, m->data, midx_size,\n+\t\t\t\t MIDX_HEADER_SIZE, m->num_chunks))\n+\t\tgoto cleanup_fail;\n+\n+\tif (pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, midx_read_pack_names, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n-\tif (!m->chunk_oid_fanout)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDFANOUT, midx_read_oid_fanout, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n-\tif (!m->chunk_oid_lookup)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDLOOKUP, midx_read_oid_lookup, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n-\tif (!m->chunk_object_offsets)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS, midx_read_offsets, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n+\tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, midx_read_large_offsets, m);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a94..ad4e878b65b 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -314,12 +314,12 @@ test_expect_success 'verify bad OID version' '\n \n test_expect_success 'verify truncated chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\01\" $objdir \\\n-\t\t\"missing required\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'verify extended chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\07\" $objdir \\\n-\t\t\"terminating multi-pack-index chunk id appears earlier than expected\"\n+\t\t\"terminating chunk id appears earlier than expected\"\n '\n \n test_expect_success 'verify missing required chunk' '\n@@ -329,7 +329,7 @@ test_expect_success 'verify missing required chunk' '\n \n test_expect_success 'verify invalid chunk offset' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_OFFSET \"\\01\" $objdir \\\n-\t\t\"invalid chunk offset (too large)\"\n+\t\t\"improper chunk offset(s)\"\n '\n \n test_expect_success 'verify packnames out of order' '\n-- \ngitgitgadget\n\n"},{"id":"415297","messageId":"05cbd0a8d93a3e54d868a549fe76e16cb75ba6d6.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 17/17] chunk-format: add technical docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:26Z","receivedAt":"2021-01-26T16:04:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-based file format is now an API in the code, but we should\nalso take time to document it as a file format. Specifically, it matches\nthe CHUNK LOOKUP sections of the commit-graph and multi-pack-index\nfiles, but there are some commonalities that should be grouped in this\ndocument.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/chunk-format.txt | 54 +++++++++++++++++++\n .../technical/commit-graph-format.txt | 3 ++\n Documentation/technical/pack-format.txt | 3 ++\n 3 files changed, 60 insertions(+)\n create mode 100644 Documentation/technical/chunk-format.txt\n\ndiff --git a/Documentation/technical/chunk-format.txt b/Documentation/technical/chunk-format.txt\nnew file mode 100644\nindex 00000000000..3db3792dea2\n--- /dev/null\n+++ b/Documentation/technical/chunk-format.txt\n@@ -0,0 +1,54 @@\n+Chunk-based file formats\n+========================\n+\n+Some file formats in Git use a common concept of \"chunks\" to describe\n+sections of the file. This allows structured access to a large file by\n+scanning a small \"table of contents\" for the remaining data. This common\n+format is used by the `commit-graph` and `multi-pack-index` files. See\n+link:technical/pack-format.html[the `multi-pack-index` format] and\n+link:technical/commit-graph-format.html[the `commit-graph` format] for\n+how they use the chunks to describe structured data.\n+\n+A chunk-based file format begins with some header information custom to\n+that format. That header should include enough information to identify\n+the file type, format version, and number of chunks in the file. From this\n+information, that file can determine the start of the chunk-based region.\n+\n+The chunk-based region starts with a table of contents describing where\n+each chunk starts and ends. This consists of (C+1) rows of 12 bytes each,\n+where C is the number of chunks. Consider the following table:\n+\n+ | Chunk ID (4 bytes) | Chunk Offset (8 bytes) |\n+ |--------------------|------------------------|\n+ | ID[0] | OFFSET[0] |\n+ | ... | ... |\n+ | ID[C] | OFFSET[C] |\n+ | 0x0000 | OFFSET[C+1] |\n+\n+Each row consists of a 4-byte chunk identifier (ID) and an 8-byte offset.\n+Each integer is stored in network-byte order.\n+\n+The chunk identifier `ID[i]` is a label for the data stored within this\n+fill from `OFFSET[i]` (inclusive) to `OFFSET[i+1]` (exclusive). Thus, the\n+size of the `i`th chunk is equal to the difference between `OFFSET[i+1]`\n+and `OFFSET[i]`. This requires that the chunk data appears contiguously\n+in the same order as the table of contents.\n+\n+The final entry in the table of contents must be four zero bytes. This\n+confirms that the table of contents is ending and provides the offset for\n+the end of the chunk-based data.\n+\n+Note: The chunk-based format expects that the file contains _at least_ a\n+trailing hash after `OFFSET[C+1]`.\n+\n+Functions for working with chunk-based file formats are declared in\n+`chunk-format.h`. Using these methods provide extra checks that assist\n+developers when creating new file formats, including:\n+\n+ 1. Writing and reading the table of contents.\n+\n+ 2. Verifying that the data written in a chunk matches the expected size\n+ that was recorded in the table of contents.\n+\n+ 3. Checking that a table of contents describes offsets properly within\n+ the file boundaries.\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex b6658eff188..87971c27dd7 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -61,6 +61,9 @@ CHUNK LOOKUP:\n the length using the next chunk position if necessary.) Each chunk\n ID appears at most once.\n \n+ The CHUNK LOOKUP matches the table of contents from\n+ link:technical/chunk-format.html[the chunk-based file format].\n+\n The remaining data in the body is described one chunk at a time, and\n these chunks may be given in any order. Chunks are required unless\n otherwise specified.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex f96b2e605f3..2fb1e60d29e 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -301,6 +301,9 @@ CHUNK LOOKUP:\n \t (Chunks are provided in file-order, so you can infer the length\n \t using the next chunk position if necessary.)\n \n+\tThe CHUNK LOOKUP matches the table of contents from\n+\tlink:technical/chunk-format.html[the chunk-based file format].\n+\n \tThe remaining data in the body is described one chunk at a time, and\n \tthese chunks may be given in any order. Chunks are required unless\n \totherwise specified.\n-- \ngitgitgadget\n"},{"id":"415298","messageId":"49cfb4f63e275ce70b20dd6d3f156971b33ddcec.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 11/17] midx: use chunk-format API in write_midx_internal()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:20Z","receivedAt":"2021-01-26T16:05:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-format API allows writing the table of contents and all chunks\nusing the anonymous 'struct chunkfile' type. We only need to convert our\nlocal chunk logic to this API for the multi-pack-index writes to share\nthat logic with the commit-graph file writes.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 104 +++++++++++----------------------------------------------\n 1 file changed, 19 insertions(+), 85 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 145c6bd0913..0bfd2d802b6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -11,6 +11,7 @@\n #include \"trace2.h\"\n #include \"run-command.h\"\n #include \"repository.h\"\n+#include \"chunk-format.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -799,18 +800,15 @@ static int write_midx_large_offsets(struct hashfile *f,\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop, unsigned flags)\n {\n-\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t header_size = 0;\n-\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n-\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tstruct chunkfile *cf;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name))\n@@ -923,98 +921,34 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n \n-\tcur_chunk = 0;\n-\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n-\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n-\n-\tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (ctx.large_offsets_needed) {\n-\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n-\n-\t\tcur_chunk++;\n-\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n-\t}\n-\n-\tchunk_ids[cur_chunk] = 0;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n-\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n-\t\t\t chunk_offsets[i - 1],\n-\t\t\t chunk_offsets[i]);\n-\n-\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n-\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n-\t\t\t chunk_offsets[i]);\n-\n-\t\thashwrite_be32(f, chunk_ids[i]);\n-\t\thashwrite_be64(f, chunk_offsets[i]);\n-\t}\n-\n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (f->total + f->offset != chunk_offsets[i])\n-\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n-\t\t\t chunk_offsets[i],\n-\t\t\t f->total + f->offset,\n-\t\t\t chunk_ids[i]);\n+\tcf = init_chunkfile(f);\n \n-\t\tswitch (chunk_ids[i]) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twrite_midx_pack_names(f, &ctx);\n-\t\t\t\tbreak;\n+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES,\n+\t\t write_midx_pack_names, pack_name_concat_len);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n+\t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n+\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n+\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n+\t\t write_midx_object_offsets,\n+\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n \n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twrite_midx_object_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twrite_midx_large_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n-\t\t\t\t chunk_ids[i]);\n-\t\t}\n-\t}\n+\tif (ctx.large_offsets_needed)\n+\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n+\t\t\twrite_midx_large_offsets,\n+\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n \n-\tif (f->total + f->offset != chunk_offsets[num_chunks])\n-\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t f->total + f->offset,\n-\t\t chunk_offsets[num_chunks]);\n+\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n+\twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n cleanup:\n-- \ngitgitgadget\n\n"},{"id":"415299","messageId":"62a23842aa650d6f56d5d258cc76fa56a547c088.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 16/17] chunk-format: restore duplicate chunk checks","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:25Z","receivedAt":"2021-01-26T16:05:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore refactoring into the chunk-format API, the commit-graph parsing\nlogic included checks for duplicate chunks. It is unlikely that we would\ndesire a chunk-based file format that allows duplicate chunk IDs in the\ntable of contents, so add duplicate checks into\nread_table_of_contents().\n\nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 674d31d5e58..3c833038096 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -14,6 +14,7 @@ struct chunk_info {\n \tchunk_write_fn write_fn;\n \n \tconst void *start;\n+\tunsigned found:1;\n };\n \n struct chunkfile {\n@@ -98,6 +99,7 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\t uint64_t toc_offset,\n \t\t\t int toc_length)\n {\n+\tint i;\n \tuint32_t chunk_id;\n \tconst unsigned char *table_of_contents = mfile + toc_offset;\n \n@@ -124,6 +126,14 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\treturn -1;\n \t\t}\n \n+\t\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\t\tif (cf->chunks[i].id == chunk_id) {\n+\t\t\t\terror(_(\"duplicate chunk ID %\"PRIx32\" found\"),\n+\t\t\t\t\tchunk_id);\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t}\n+\n \t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n \t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n \t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n-- \ngitgitgadget\n\n"},{"id":"415300","messageId":"f6c58ff72d24ec166d5d9b661c433bcb13377fd2.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:24Z","receivedAt":"2021-01-26T16:06:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen calculating the sizes of certain chunks, we should use 64-bit\nmultiplication always. This allows us to properly predict the chunk\nsizes without risk of overflow.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex dd019c00795..47aaeb804b8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -945,7 +945,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n \t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n \tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n-\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n+\t\t write_midx_oid_lookup, (uint64_t)ctx.entries_nr * the_hash_algo->rawsz);\n \tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n \t\t write_midx_object_offsets,\n \t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n@@ -953,7 +953,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.large_offsets_needed)\n \t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n \t\t\twrite_midx_large_offsets,\n-\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n+\t\t\t(uint64_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n-- \ngitgitgadget\n\n"},{"id":"415301","messageId":"0b4ce3f1732a6b7297473ac3bae035f555a9268a.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 07/17] midx: add pack_perm to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:16Z","receivedAt":"2021-01-26T16:06:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t *pack_perm\" and large_offsets_needed bit\ninto the context.\n\nUpdate write_midx_object_offsets() to match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 40 +++++++++++++++++++++-------------------\n 1 file changed, 21 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex ee6f3504c6a..66feff096e8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -461,6 +461,9 @@ struct write_midx_context {\n \n \tstruct pack_midx_entry *entries;\n \tuint32_t entries_nr;\n+\n+\tuint32_t *pack_perm;\n+\tunsigned large_offsets_needed:1;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -736,27 +739,27 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n-\t\t\t\t\tuint32_t *perm,\n-\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t\tvoid *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n \n-\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n-\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n \t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->offset);\n@@ -805,13 +808,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n-\tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -857,11 +858,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n+\tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n-\t\t\tlarge_offsets_needed = 1;\n+\t\t\tctx.large_offsets_needed = 1;\n \t}\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n@@ -900,13 +902,13 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n@@ -927,7 +929,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n-\tnum_chunks = large_offsets_needed ? 5 : 4;\n+\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -954,7 +956,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (large_offsets_needed) {\n+\tif (ctx.large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n \t\tcur_chunk++;\n@@ -1004,7 +1006,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -1039,7 +1041,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfree(ctx.info);\n \tfree(ctx.entries);\n-\tfree(pack_perm);\n+\tfree(ctx.pack_perm);\n \tfree(midx_name);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"415302","messageId":"e613ffa9ac639664c37989019bdc6b749dc00e4f.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 10/17] midx: drop chunk progress during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:19Z","receivedAt":"2021-01-26T16:06:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMost expensive operations in write_midx_internal() use the context\nstruct's progress member, and these indicate the process of the\nexpensive operations within the chunk writing methods. However, there is\na competing progress struct that counts the progress over all chunks.\nThis is not very helpful compared to the others, so drop it.\n\nThis also reduces our barriers to combining the chunk writing code with\nchunk-format.c.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 852dd5b776e..145c6bd0913 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -808,7 +808,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -976,9 +975,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\thashwrite_be64(f, chunk_offsets[i]);\n \t}\n \n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n-\t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n@@ -1011,10 +1007,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\n \t\t}\n-\n-\t\tdisplay_progress(progress, i + 1);\n \t}\n-\tstop_progress(&progress);\n \n \tif (f->total + f->offset != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-- \ngitgitgadget\n\n"},{"id":"415303","messageId":"909ca28e0ba23bed307a01e8851f9132581417b7.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 09/17] midx: return success/failure in chunk write methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:18Z","receivedAt":"2021-01-26T16:06:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHistorically, the chunk-writing methods in midx.c have returned the\namount of data written so the writer method could compare this with the\ntable of contents. This presents with some interesting issues:\n\n1. If a chunk writing method has a bug that miscalculates the written\n bytes, then we can satisfy the table of contents without actually\n writing the right amount of data to the hashfile. The commit-graph\n writing code checks the hashfile struct directly for a more robust\n verification.\n\n2. There is no way for a chunk writing method to gracefully fail.\n Returning an int presents an opportunity to fail without a die().\n\n3. The current pattern doesn't match chunk_write_fn type exactly, so we\n cannot share code with commit-graph.c\n\nFor these reasons, convert the midx chunk writer methods to return an\n'int'. Since none of them fail at the moment, they all return 0.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 63 +++++++++++++++++++++++++---------------------------------\n 1 file changed, 27 insertions(+), 36 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 40b815f8877..852dd5b776e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -650,7 +650,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f, void *data)\n+static int write_midx_pack_names(struct hashfile *f, void *data)\n {\n \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \tuint32_t i;\n@@ -678,14 +678,13 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n \tif (i < MIDX_CHUNK_ALIGNMENT) {\n \t\tmemset(padding, 0, sizeof(padding));\n \t\thashwrite(f, padding, i);\n-\t\twritten += i;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \tstruct pack_midx_entry *list = ctx->entries;\n@@ -710,17 +709,16 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \t\tlist = next;\n \t}\n \n-\treturn MIDX_CHUNK_FANOUT_SIZE;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \tunsigned char hash_len = the_hash_algo->rawsz;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -734,19 +732,17 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \t\t}\n \n \t\thashwrite(f, obj->oid.hash, (int)hash_len);\n-\t\twritten += hash_len;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f,\n-\t\t\t\t\tvoid *data)\n+static int write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -766,20 +762,17 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \t\t\t obj->offset);\n \t\telse\n \t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n-\n-\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n-\tsize_t written = 0;\n \tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n@@ -795,12 +788,12 @@ static size_t write_midx_large_offsets(struct hashfile *f,\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\twritten += hashwrite_be64(f, offset);\n+\t\thashwrite_be64(f, offset);\n \n \t\tnr_large_offset--;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n@@ -812,7 +805,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t written = 0;\n+\tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tstruct progress *progress = NULL;\n@@ -940,10 +933,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n+\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n@@ -981,39 +974,37 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\thashwrite_be32(f, chunk_ids[i]);\n \t\thashwrite_be64(f, chunk_offsets[i]);\n-\n-\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (written != chunk_offsets[i])\n+\t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n \t\t\t chunk_offsets[i],\n-\t\t\t written,\n+\t\t\t f->total + f->offset,\n \t\t\t chunk_ids[i]);\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n+\t\t\t\twrite_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n+\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n+\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n+\t\t\t\twrite_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n+\t\t\t\twrite_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1025,9 +1016,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \tstop_progress(&progress);\n \n-\tif (written != chunk_offsets[num_chunks])\n+\tif (f->total + f->offset != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t written,\n+\t\t f->total + f->offset,\n \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n-- \ngitgitgadget\n\n"},{"id":"415304","messageId":"eabc7b736478e68c6e7d0af73df8e3e73e17bc34.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 08/17] midx: add num_large_offsets to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:17Z","receivedAt":"2021-01-26T16:07:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t num_large_offsets\" into the context. With\nthis new data, write_midx_large_offsets() now matches the\nchunk_write_fn type.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 17 ++++++++++-------\n 1 file changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 66feff096e8..40b815f8877 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -464,6 +464,7 @@ struct write_midx_context {\n \n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n+\tuint32_t num_large_offsets;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -772,11 +773,14 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n-\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects, *end = objects + nr_objects;\n+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n \tsize_t written = 0;\n+\tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n \t\tstruct pack_midx_entry *obj;\n@@ -811,7 +815,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n@@ -861,7 +864,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n-\t\t\tnum_large_offsets++;\n+\t\t\tctx.num_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n@@ -961,7 +964,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tcur_chunk++;\n \t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n \t}\n \n \tchunk_ids[cur_chunk] = 0;\n@@ -1010,7 +1013,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n-- \ngitgitgadget\n\n"},{"id":"415305","messageId":"7339990f07db81897a30cb12e2d33ac1ef855d20.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 13/17] commit-graph: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:22Z","receivedAt":"2021-01-26T16:07:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). While the current\nimplementation loses the duplicate-chunk detection, that will be added\nin a future change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 209 ++++++++++++++++++++--------------------\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 108 insertions(+), 103 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b2c0f233eab..44c06d0fb67 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -299,15 +299,99 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n+static int graph_read_oid_fanout(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_oid_fanout = (uint32_t*)chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_oid_lookup(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_oid_lookup = chunk_start;\n+\tg->num_commits = chunk_size / g->hash_len;\n+\treturn 0;\n+}\n+\n+static int graph_read_data(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_commit_data = chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_extra_edges(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_extra_edges = chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_base_graphs(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_base_graphs = chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_generation_data(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_generation_data = chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_generation_overflow(const unsigned char *chunk_start,\n+\t\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_generation_data_overflow = chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_bloom_indices(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tg->chunk_bloom_indexes = chunk_start;\n+\treturn 0;\n+}\n+\n+static int graph_read_bloom_data(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = (struct commit_graph *)data;\n+\tuint32_t hash_version;\n+\tg->chunk_bloom_data = chunk_start;\n+\thash_version = get_be32(chunk_start);\n+\n+\tif (hash_version != 1)\n+\t\treturn 0;\n+\n+\tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n+\tg->bloom_filter_settings->hash_version = hash_version;\n+\tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n+\tg->bloom_filter_settings->bits_per_entry = get_be32(chunk_start + 8);\n+\tg->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n+\n+\treturn 0;\n+}\n+\n struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n-\tconst unsigned char *data, *chunk_lookup;\n-\tuint32_t i;\n+\tconst unsigned char *data;\n \tstruct commit_graph *graph;\n-\tuint64_t next_chunk_offset;\n \tuint32_t graph_signature;\n \tunsigned char graph_version, hash_version;\n+\tstruct chunkfile *cf = NULL;\n \n \tif (!graph_map)\n \t\treturn NULL;\n@@ -356,108 +440,27 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n-\tchunk_lookup = data + 8;\n-\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\tfor (i = 0; i < graph->num_chunks; i++) {\n-\t\tuint32_t chunk_id;\n-\t\tuint64_t chunk_offset = next_chunk_offset;\n-\t\tint chunk_repeated = 0;\n-\n-\t\tchunk_id = get_be32(chunk_lookup + 0);\n-\n-\t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n-\t\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\n-\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n-\t\t\terror(_(\"commit-graph improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n-\t\t\t (uint32_t)chunk_offset);\n-\t\t\tgoto free_and_return;\n-\t\t}\n-\n-\t\tswitch (chunk_id) {\n-\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n-\t\t\tif (graph->chunk_oid_fanout)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_oid_fanout = (uint32_t*)(data + chunk_offset);\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n-\t\t\tif (graph->chunk_oid_lookup)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n-\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n-\t\t\t\tgraph->num_commits = (next_chunk_offset - chunk_offset)\n-\t\t\t\t\t\t / graph->hash_len;\n-\t\t\t}\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_DATA:\n-\t\t\tif (graph->chunk_commit_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA:\n-\t\t\tif (graph->chunk_generation_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data = data + chunk_offset;\n-\t\t\tbreak;\n+\tcf = init_chunkfile(NULL);\n \n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW:\n-\t\t\tif (graph->chunk_generation_data_overflow)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data_overflow = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_EXTRAEDGES:\n-\t\t\tif (graph->chunk_extra_edges)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BASE:\n-\t\t\tif (graph->chunk_base_graphs)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n-\t\t\tif (graph->chunk_bloom_indexes)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n-\t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMDATA:\n-\t\t\tif (graph->chunk_bloom_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n-\t\t\t\tuint32_t hash_version;\n-\t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n-\t\t\t\thash_version = get_be32(data + chunk_offset);\n-\n-\t\t\t\tif (hash_version != 1)\n-\t\t\t\t\tbreak;\n+\tif (read_table_of_contents(cf, graph->data, graph_size,\n+\t\t\t\t GRAPH_HEADER_SIZE, graph->num_chunks))\n+\t\tgoto free_and_return;\n \n-\t\t\t\tgraph->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n-\t\t\t\tgraph->bloom_filter_settings->hash_version = hash_version;\n-\t\t\t\tgraph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n-\t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n-\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n+\tpair_chunk(cf, GRAPH_CHUNKID_OIDFANOUT, graph_read_oid_fanout, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, graph_read_oid_lookup, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_DATA, graph_read_data, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, graph_read_extra_edges, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_BASE, graph_read_base_graphs, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t graph_read_generation_data, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t graph_read_generation_overflow, graph);\n \n-\t\tif (chunk_repeated) {\n-\t\t\terror(_(\"commit-graph chunk id %08x appears multiple times\"), chunk_id);\n-\t\t\tgoto free_and_return;\n-\t\t}\n+\tif (r->settings.commit_graph_read_changed_paths) {\n+\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t graph_read_bloom_indices, graph);\n+\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -474,9 +477,11 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \tif (verify_commit_graph_lite(graph))\n \t\tgoto free_and_return;\n \n+\tfree_chunkfile(cf);\n \treturn graph;\n \n free_and_return:\n+\tfree_chunkfile(cf);\n \tfree(graph->bloom_filter_settings);\n \tfree(graph);\n \treturn NULL;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex fa27df579a5..c7da741284e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -564,7 +564,7 @@ test_expect_success 'detect bad hash version' '\n \n test_expect_success 'detect low chunk count' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\01\" \\\n-\t\t\"missing the .* chunk\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'detect missing OID fanout chunk' '\n-- \ngitgitgadget\n\n"},{"id":"415306","messageId":"e3475633e1d2e397c5095f34c23a19fccb2c8428.1611676886.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-26T16:01:21Z","receivedAt":"2021-01-26T16:09:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd the capability to read the table of contents, then pair the chunks\nwith necessary logic using read_chunk_fn pointers. Callers will be added\nin future changes, but the typical outline will be:\n\n 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n 2. call read_table_of_contents().\n 3. for each chunk to parse, call pair_chunk() with appropriate pointers.\n 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n\nWe are re-using the anonymous 'struct chunkfile' data, as it is internal\nto the chunk-format API. This gives it essentially two modes: write and\nread. If the same struct instance was used for both reads and writes,\nthen there would be failures.\n\nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 21 +++++++++++++++++\n 2 files changed, 85 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 2ce37ecc6bb..674d31d5e58 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -12,6 +12,8 @@ struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n \tchunk_write_fn write_fn;\n+\n+\tconst void *start;\n };\n \n struct chunkfile {\n@@ -89,3 +91,65 @@ int write_chunkfile(struct chunkfile *cf, void *data)\n \n \treturn 0;\n }\n+\n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length)\n+{\n+\tuint32_t chunk_id;\n+\tconst unsigned char *table_of_contents = mfile + toc_offset;\n+\n+\tALLOC_GROW(cf->chunks, toc_length, cf->chunks_alloc);\n+\n+\twhile (toc_length--) {\n+\t\tuint64_t chunk_offset, next_chunk_offset;\n+\n+\t\tchunk_id = get_be32(table_of_contents);\n+\t\tchunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (!chunk_id) {\n+\t\t\terror(_(\"terminating chunk id appears earlier than expected\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\ttable_of_contents += CHUNK_LOOKUP_WIDTH;\n+\t\tnext_chunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (next_chunk_offset < chunk_offset ||\n+\t\t next_chunk_offset > mfile_size - the_hash_algo->rawsz) {\n+\t\t\terror(_(\"improper chunk offset(s) %\"PRIx64\" and %\"PRIx64\"\"),\n+\t\t\t chunk_offset, next_chunk_offset);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n+\t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n+\t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n+\t\tcf->chunks_nr++;\n+\t}\n+\n+\tchunk_id = get_be32(table_of_contents);\n+\tif (chunk_id) {\n+\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tif (cf->chunks[i].id == chunk_id)\n+\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n+\t}\n+\n+\treturn CHUNK_NOT_FOUND;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nindex bfaed672813..250e08b8e6a 100644\n--- a/chunk-format.h\n+++ b/chunk-format.h\n@@ -17,4 +17,25 @@ void add_chunk(struct chunkfile *cf,\n \t size_t size);\n int write_chunkfile(struct chunkfile *cf, void *data);\n \n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length);\n+\n+/*\n+ * When reading a table of contents, we find the chunk with matching 'id'\n+ * then call its read_fn to populate the necessary 'data' based on the\n+ * chunk start and size.\n+ */\n+typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size, void *data);\n+\n+\n+#define CHUNK_NOT_FOUND (-2)\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"415368","messageId":"YBDRoEzRKO6GIVCb@nand.local","threadId":"55053","inReplyTo":"CAPx1GvfQ4K+2mt0BC-1o7e0omHAvC6XhmpOuGa41v-b850w14w@mail.gmail.com","subject":"Re: [PATCH 01/17] commit-graph: anonymize data in chunk_write_fn","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T02:36:16Z","receivedAt":"2021-01-27T03:59:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 05:53:39PM -0800, Chris Torek wrote:\n> Note: this is purely style, and minor, but I'll ask...\n>\n> On Tue, Jan 26, 2021 at 8:08 AM Derrick Stolee via GitGitGadget\n> wrote:\n> > static int write_graph_chunk_fanout(struct hashfile *f,\n> > - struct write_commit_graph_context *ctx)\n> > + void *data)\n> > {\n> > + struct write_commit_graph_context *ctx =\n> > + (struct write_commit_graph_context *)data;\n>\n> Why bother with the cast on the last line here? In C,\n> conversion from `void *` to `struct whatever *` is fine.\n>\n> (the change itself looks fine, btw)\n\nAgreed. It's not a correctness issue, but I find these unnecessary casts\nto detract from readability. If you do end up rerolling this series,\nI'd rather see\n\n struct write_commit_graph_context *ctx = data;\n\n...but I don't think that this (non-)issue alone is worth a reroll.\n\nThanks,\nTaylor\n"},{"id":"415370","messageId":"YBDU1i6+wEyGL0TQ@nand.local","threadId":"55053","inReplyTo":"9fe5ee8611c3a18daaa9701bb5eefc8c408a7e76.1611676886.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/17] midx: rename pack_info to write_midx_context","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T02:49:58Z","receivedAt":"2021-01-27T03:59:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 04:01:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n> This change only renames \"struct pack_info\" to \"struct\n> write_midx_context\" and the names of instances from \"packs\" to \"ctx\". In\n> future changes, we will expand the data inside \"struct\n> write_midx_context\" and align our chunk-writing method with the\n> chunk-format API.\n\nThanks for saying that; that makes clear what is (and isn't) going on\nhere.\n\n> @@ -463,37 +463,37 @@ struct pack_list {\n> static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> \t\t\t const char *file_name, void *data)\n> {\n> -\tstruct pack_list *packs = (struct pack_list *)data;\n> +\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n\nSame comments as earlier about the unnecessary cast on the right-hand\nside of this (and the below) assignment.\n\nOtherwise this patch looks obviously fine to me.\n\n\nThanks,\nTaylor\n"},{"id":"415371","messageId":"YBDX0wD4wwzuCyYh@nand.local","threadId":"55053","inReplyTo":"e3475633e1d2e397c5095f34c23a19fccb2c8428.1611676886.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 12/17] chunk-format: create read chunk API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T03:02:43Z","receivedAt":"2021-01-27T03:59:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 04:01:21PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/chunk-format.c b/chunk-format.c\n> index 2ce37ecc6bb..674d31d5e58 100644\n> --- a/chunk-format.c\n> +++ b/chunk-format.c\n> @@ -12,6 +12,8 @@ struct chunk_info {\n> \tuint32_t id;\n> \tuint64_t size;\n> \tchunk_write_fn write_fn;\n> +\n> +\tconst void *start;\n\nIt may be clearer to fold both of these into an anonymous union along\nwith an enum to indicate which mode we're in. But, I could also buy that\nthat is more error prone, so perhaps just a comment along the lines of\n\"exactly one of these is NULL\" would suffice, too.\n\n> };\n>\n> struct chunkfile {\n> @@ -89,3 +91,65 @@ int write_chunkfile(struct chunkfile *cf, void *data)\n>\n> \treturn 0;\n> }\n> +\n> +int read_table_of_contents(struct chunkfile *cf,\n> +\t\t\t const unsigned char *mfile,\n> +\t\t\t size_t mfile_size,\n\nAssuming that mfile and mfile_size are a pointer to a memory mapped\nregion and its size? If so, a nit is that I'd expect \"data\" and \"size\"\ninstead of \"mfile\".\n\nI think that it's probably going too far to have the chunkfile API\nhandle mapping its own memory, so in that way I don't think it's wrong\nfor the callers to be handling that.\n\nOTOH, it does seem a little weird to temporarily hand off ownership like\nthis. I don't think I have a better suggestion, though.\n\nThe implementation of this function looks good to me.\n\n> +int pair_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t chunk_read_fn fn,\n> +\t void *data)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\tif (cf->chunks[i].id == chunk_id)\n> +\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n> +\t}\n> +\n> +\treturn CHUNK_NOT_FOUND;\n> +}\n> diff --git a/chunk-format.h b/chunk-format.h\n> index bfaed672813..250e08b8e6a 100644\n> --- a/chunk-format.h\n> +++ b/chunk-format.h\n> @@ -17,4 +17,25 @@ void add_chunk(struct chunkfile *cf,\n> \t size_t size);\n> int write_chunkfile(struct chunkfile *cf, void *data);\n>\n> +int read_table_of_contents(struct chunkfile *cf,\n> +\t\t\t const unsigned char *mfile,\n> +\t\t\t size_t mfile_size,\n> +\t\t\t uint64_t toc_offset,\n> +\t\t\t int toc_length);\n> +\n> +/*\n> + * When reading a table of contents, we find the chunk with matching 'id'\n> + * then call its read_fn to populate the necessary 'data' based on the\n> + * chunk start and size.\n> + */\n> +typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n> +\t\t\t size_t chunk_size, void *data);\n> +\n> +\n> +#define CHUNK_NOT_FOUND (-2)\n> +int pair_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t chunk_read_fn fn,\n> +\t void *data);\n\nFrom reading the implementation, I take it that this function calls fn\nwith the location and size of the requested chunk, along with the user\nsupplied data.\n\nI'm not sure that \"pair\" gives me that same sense. Maybe \"read\" or\n\"lookup\" would be better?\n\nDunno.\n\nThanks,\nTaylor\n"},{"id":"415374","messageId":"YBDQBkmCFUTMM5/C@nand.local","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/17] Refactor chunk-format into an API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T02:29:26Z","receivedAt":"2021-01-27T04:16:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 04:01:09PM +0000, Derrick Stolee via GitGitGadget wrote:\n> This version also changes the approach to use a more dynamic interaction\n> with a struct chunkfile pointer. This idea is credited to Taylor Blau [2],\n> but I started again from scratch. I also go further to make struct chunkfile\n> anonymous to API consumers. It is defined only in chunk-format.c, which\n> should hopefully deter future users from interacting with that data\n> directly.\n>\n> [2] https://lore.kernel.org/git/X8%2FI%2FRzXZksio+ri@nand.local/\n\nGreat; I am very happy that you found my patch to be useful. I'm glad\nthat you decided to start from scratch, too, since as I recall there\nwere some unresolved test issues that I punted on in case you decided to\nabandon the topic altogether.\n\n> This combined API is beneficial to reduce duplicated logic. Or rather, to\n> ensure that similar file formats have similar protections against bad data.\n> The multi-pack-index code did not have as many guards as the commit-graph\n> code did, but now they both share a common base that checks for things like\n> duplicate chunks or offsets outside the size of the file.\n\nDefinitely good.\n\n> Here are some stats for the end-to-end change:\n>\n> * 638 insertions(+), 456 deletions(-).\n> * commit-graph.c: 171 insertions(+), 192 deletions(-)\n> * midx.c: 196 insertions(+), 260 deletions(-)\n>\n> While there is an overall increase to the code size, the consumers do get a\n> bit smaller. Boilerplate things like abstracting method to match\n> chunk_write_fn and chunk_read_fn make up a lot of these insertions. The\n> \"interesting\" code gets a lot smaller and cleaner.\n\nLike I said in [1], I don't think a net +182 line diff is reason alone\nnot to pursue this topic. I don't think that an chunked index v3 will\ncome as part of my work on the on-disk revindex format, but I do think\nthat it's something brian may be interested in. So, I'm feeling rather\ncertain that we'll eventually have new callers, at which point this will\nreduce duplication overall.\n\n[1]: https://lore.kernel.org/git/X8%2FK1dUgUmwp8ZOv@nand.local/\n\nThanks,\nTaylor\n"},{"id":"415375","messageId":"YBDTKqiCEkWmxEFr@nand.local","threadId":"55053","inReplyTo":"9bd273f8c94fdb0c3adf8aedef3480ff5f4232b8.1611676886.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/17] chunk-format: create chunk format write API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T02:42:50Z","receivedAt":"2021-01-27T04:16:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 04:01:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n> +/*\n> + * When writing a chunk-based file format, collect the chunks in\n> + * an array of chunk_info structs. The size stores the _expected_\n> + * amount of data that will be written by write_fn.\n> + */\n> +struct chunk_info {\n> +\tuint32_t id;\n> +\tuint64_t size;\n\nHmm. Would we not want an off_t to indicate the size here?\n\nI wondered briefly if we even needed a size field at all, since calling\nwrite_fn would tell us the number of bytes written. But I suppose you\nwant to know ahead of time so that you can write the file in one pass\n(beginning with the table of contents, which certainly needs to know the\nsize).\n\n> +\t/* Trailing entry marks the end of the chunks */\n> +\thashwrite_be32(cf->f, 0);\n> +\thashwrite_be64(cf->f, cur_offset);\n> +\n> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\tuint64_t start_offset = cf->f->total + cf->f->offset;\n> +\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n> +\n> +\t\tif (result)\n> +\t\t\treturn result;\n> +\n> +\t\tif (cf->f->total + cf->f->offset != start_offset + cf->chunks[i].size)\n\nI don't think this is a practical concern, but a malicious caller could\noverflow this by passing a bogus \"size\" parameter. Maybe:\n\n uint64_t end_offset = ...;\n\n if (end_offset - start_offset != cf->chunks[i].size)\n BUG(...)\n\n?\n\n> diff --git a/chunk-format.h b/chunk-format.h\n> new file mode 100644\n> index 00000000000..bfaed672813\n> --- /dev/null\n> +++ b/chunk-format.h\n> @@ -0,0 +1,20 @@\n> +#ifndef CHUNK_FORMAT_H\n> +#define CHUNK_FORMAT_H\n> +\n> +#include \"git-compat-util.h\"\n> +\n> +struct hashfile;\n> +struct chunkfile;\n> +\n> +struct chunkfile *init_chunkfile(struct hashfile *f);\n> +void free_chunkfile(struct chunkfile *cf);\n> +int get_num_chunks(struct chunkfile *cf);\n> +typedef int (*chunk_write_fn)(struct hashfile *f,\n> +\t\t\t void *data);\n> +void add_chunk(struct chunkfile *cf,\n> +\t uint64_t id,\n> +\t chunk_write_fn fn,\n> +\t size_t size);\n> +int write_chunkfile(struct chunkfile *cf, void *data);\n\nVery clean API.\n\nThanks,\nTaylor\n"},{"id":"415376","messageId":"CAPx1GvfQ4K+2mt0BC-1o7e0omHAvC6XhmpOuGa41v-b850w14w@mail.gmail.com","threadId":"55053","inReplyTo":"09b32829e4ff2384cb35afaf1a34385e25bac8d8.1611676886.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/17] commit-graph: anonymize data in chunk_write_fn","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2021-01-27T01:53:39Z","receivedAt":"2021-01-27T04:17:50Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"Note: this is purely style, and minor, but I'll ask...\n\nOn Tue, Jan 26, 2021 at 8:08 AM Derrick Stolee via GitGitGadget\n wrote:\n> static int write_graph_chunk_fanout(struct hashfile *f,\n> - struct write_commit_graph_context *ctx)\n> + void *data)\n> {\n> + struct write_commit_graph_context *ctx =\n> + (struct write_commit_graph_context *)data;\n\nWhy bother with the cast on the last line here? In C,\nconversion from `void *` to `struct whatever *` is fine.\n\n(the change itself looks fine, btw)\n\nChris\n"},{"id":"415378","messageId":"YBDUOGLkiQ7ocl+D@nand.local","threadId":"55053","inReplyTo":"a3d6177a352643721fdc07512629b48d1213157a.1611676886.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 03/17] commit-graph: use chunk-format write API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T02:47:20Z","receivedAt":"2021-01-27T04:17:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 04:01:12PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> The commit-graph write logic is ready to make use of the chunk-format\n> write API. Each chunk write method is already in the correct prototype.\n> We only need to use the 'struct chunkfile' pointer and the correct API\n> calls.\n>\n> Signed-off-by: Derrick Stolee \n\nNicely done. The majority of this patch was remarkably easy to read,\nwhich I attribute to you doing the necessary prep work to make the\ncallbacks usable by the new API. Thank you.\n\n> @@ -1941,6 +1896,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>\n> \tclose_commit_graph(ctx->r->objects);\n> \tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n> +\tfree_chunkfile(cf);\n\nSince chunkfiles are so tightly coupled to hashfiles (i.e., you can only\n\"construct\" a chunkfile given a 'struct hashfile*'), I wonder whether\nthis should be:\n\n finalize_chunkfile(cf, ...)\n\ninstead. It seems kind of weird to give up ownership of 'f' down to the\nchunkfile API only to reach down into it again.\n\nI could even buy that you'd always want to finalize and free a chunkfile\nat the same time, and so perhaps the calls could be combined, but that\nmay be a step too far.\n\nThanks,\nTaylor\n"},{"id":"415394","messageId":"YBDY0Me2QMcUMD2e@nand.local","threadId":"55053","inReplyTo":"cb145e0e32afed99b9bfa822c76f48bee18885ba.1611676886.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 14/17] midx: use chunk-format read API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T03:06:56Z","receivedAt":"2021-01-27T06:50:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 26, 2021 at 04:01:23PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> Instead of parsing the table of contents directly, use the chunk-format\n> API methods read_table_of_contents() and pair_chunk(). In particular, we\n> can use the return value of pair_chunk() to generate an error when a\n> required chunk is missing.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> midx.c | 103 ++++++++++++++++++++----------------\n> t/t5319-multi-pack-index.sh | 6 +--\n> 2 files changed, 60 insertions(+), 49 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 0bfd2d802b6..dd019c00795 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -54,6 +54,51 @@ static char *get_midx_filename(const char *object_dir)\n> \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n> }\n>\n> +static int midx_read_pack_names(const unsigned char *chunk_start,\n> +\t\t\t\tsize_t chunk_size, void *data)\n> +{\n> +\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n> +\tm->chunk_pack_names = chunk_start;\n> +\treturn 0;\n> +}\n\nThere are a lot of these callbacks that just assign some 'void **' to\npoint at chunk_start.\n\nMaybe a good use of the \"pair_chunk\" name would be something like:\n\n int pair_chunk(struct chunkfile *cf, uint32_t id, const unsigned char **p);\n\nwhich does the same as what you wrote here. So instead of what you\nwrote, you could instead:\n\n pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, &m->chunk_pack_names);\n\nThis would be in addition to the richer callback-style function which\nallows the caller greater flexibility (e.g., for the Bloom filter\nrelated readers in the commit-graph code).\n\nThanks,\nTaylor\n"},{"id":"415415","messageId":"xmqqr1m7z5nl.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/17] Refactor chunk-format into an API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-26T22:37:18Z","receivedAt":"2021-01-27T13:21:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This is a restart on the topic previously submitted [1] but dropped because\n> ak/corrected-commit-date was still in progress. This version is based on\n> that branch.\n\nNice to see that we have an endorsement on ak/corrected-commit-date\ntopic ;-)\n\nI've scanned this round of the topic and they were pleasant read.\n\nI may have other comments after a more careful reading, but so far,\nI am happy with what I see here.\n\nThanks.\n"},{"id":"415421","messageId":"c1cd1184-a254-7e39-5432-ecaf6e20ff90@gmail.com","threadId":"55053","inReplyTo":"YBDTKqiCEkWmxEFr@nand.local","subject":"Re: [PATCH 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-27T13:49:16Z","receivedAt":"2021-01-27T13:50:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/26/2021 9:42 PM, Taylor Blau wrote:\n> On Tue, Jan 26, 2021 at 04:01:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> +/*\n>> + * When writing a chunk-based file format, collect the chunks in\n>> + * an array of chunk_info structs. The size stores the _expected_\n>> + * amount of data that will be written by write_fn.\n>> + */\n>> +struct chunk_info {\n>> +\tuint32_t id;\n>> +\tuint64_t size;\n> \n> Hmm. Would we not want an off_t to indicate the size here?\n> \n> I wondered briefly if we even needed a size field at all, since calling\n> write_fn would tell us the number of bytes written. But I suppose you\n> want to know ahead of time so that you can write the file in one pass\n> (beginning with the table of contents, which certainly needs to know the\n> size).\n\nIs off_t 64-bits on a 32-bit machine? This is intentionally typed\nto be \"64 bits no matter what\" because it correlates with the file\nformat's size for the chunk offsets.\n\n>> +\t\tif (cf->f->total + cf->f->offset != start_offset + cf->chunks[i].size)\n> \n> I don't think this is a practical concern, but a malicious caller could\n> overflow this by passing a bogus \"size\" parameter. Maybe:\n> \n> uint64_t end_offset = ...;\n> \n> if (end_offset - start_offset != cf->chunks[i].size)\n> BUG(...)\n\nSure.\n\nThanks,\n-Stolee\n"},{"id":"415422","messageId":"0fd9f692-ff8c-f4b9-33ac-975da93e9591@gmail.com","threadId":"55053","inReplyTo":"YBDY0Me2QMcUMD2e@nand.local","subject":"Re: [PATCH 14/17] midx: use chunk-format read API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-27T13:50:55Z","receivedAt":"2021-01-27T13:52:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/26/2021 10:06 PM, Taylor Blau wrote:\n> On Tue, Jan 26, 2021 at 04:01:23PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> Instead of parsing the table of contents directly, use the chunk-format\n>> API methods read_table_of_contents() and pair_chunk(). In particular, we\n>> can use the return value of pair_chunk() to generate an error when a\n>> required chunk is missing.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> midx.c | 103 ++++++++++++++++++++----------------\n>> t/t5319-multi-pack-index.sh | 6 +--\n>> 2 files changed, 60 insertions(+), 49 deletions(-)\n>>\n>> diff --git a/midx.c b/midx.c\n>> index 0bfd2d802b6..dd019c00795 100644\n>> --- a/midx.c\n>> +++ b/midx.c\n>> @@ -54,6 +54,51 @@ static char *get_midx_filename(const char *object_dir)\n>> \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n>> }\n>>\n>> +static int midx_read_pack_names(const unsigned char *chunk_start,\n>> +\t\t\t\tsize_t chunk_size, void *data)\n>> +{\n>> +\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n>> +\tm->chunk_pack_names = chunk_start;\n>> +\treturn 0;\n>> +}\n> \n> There are a lot of these callbacks that just assign some 'void **' to\n> point at chunk_start.\n> \n> Maybe a good use of the \"pair_chunk\" name would be something like:\n> \n> int pair_chunk(struct chunkfile *cf, uint32_t id, const unsigned char **p);\n> \n> which does the same as what you wrote here. So instead of what you\n> wrote, you could instead:\n> \n> pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, &m->chunk_pack_names);\n> \n> This would be in addition to the richer callback-style function which\n> allows the caller greater flexibility (e.g., for the Bloom filter\n> related readers in the commit-graph code).\n\nYou're right that _most_ callers just want to assign a pointer,\nso this mechanism would be better. I'll make a different function,\nread_chunk() perhaps, that relies on a callback for advanced users.\n\nThanks,\n-Stolee\n"},{"id":"415426","messageId":"4a4e90b129aefd6a186561d5e814dc7695bae7cb.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 05/17] midx: use context in write_midx_pack_names()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:44Z","receivedAt":"2021-01-27T15:07:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align the write_midx_internal() to use the chunk-format\nAPI, start converting chunk writing methods to match chunk_write_fn. The\nfirst case is to convert write_midx_pack_names() to take \"void *data\".\nWe already have the necessary data in \"struct write_midx_context\", so\nthis conversion is rather mechanical.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 21 ++++++++++-----------\n 1 file changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 561f65a63a5..88452b04433 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -643,27 +643,26 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t struct pack_info *info,\n-\t\t\t\t uint32_t num_packs)\n+static size_t write_midx_pack_names(struct hashfile *f, void *data)\n {\n+\tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n \tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < num_packs; i++) {\n+\tfor (i = 0; i < ctx->nr; i++) {\n \t\tsize_t writelen;\n \n-\t\tif (info[i].expired)\n+\t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n+\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t info[i - 1].pack_name,\n-\t\t\t info[i].pack_name);\n+\t\t\t ctx->info[i - 1].pack_name,\n+\t\t\t ctx->info[i].pack_name);\n \n-\t\twritelen = strlen(info[i].pack_name) + 1;\n-\t\thashwrite(f, info[i].pack_name, writelen);\n+\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n+\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -990,7 +989,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-- \ngitgitgadget\n\n"},{"id":"415427","messageId":"70af6e3083f4f5e2b921c1c9817c790c8b5f66ce.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 03/17] commit-graph: use chunk-format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:42Z","receivedAt":"2021-01-27T15:07:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph write logic is ready to make use of the chunk-format\nwrite API. Each chunk write method is already in the correct prototype.\nWe only need to use the 'struct chunkfile' pointer and the correct API\ncalls.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 118 ++++++++++++++++---------------------------------\n 1 file changed, 37 insertions(+), 81 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fae7d1b6393..ba33777dcb8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -19,6 +19,7 @@\n #include \"shallow.h\"\n #include \"json-writer.h\"\n #include \"trace2.h\"\n+#include \"chunk-format.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -1758,27 +1759,17 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t void *data);\n-\n-struct chunk_info {\n-\tuint32_t id;\n-\tuint64_t size;\n-\tchunk_write_fn write_fn;\n-};\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n \tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tstruct chunk_info chunks[MAX_NUM_CHUNKS + 1];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n-\tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n+\tstruct chunkfile *cf;\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1824,76 +1815,50 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \t}\n \n-\tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunks[0].size = GRAPH_FANOUT_SIZE;\n-\tchunks[0].write_fn = write_graph_chunk_fanout;\n-\tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunks[1].size = hashsz * ctx->commits.nr;\n-\tchunks[1].write_fn = write_graph_chunk_oids;\n-\tchunks[2].id = GRAPH_CHUNKID_DATA;\n-\tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n-\tchunks[2].write_fn = write_graph_chunk_data;\n+\tcf = init_chunkfile(f);\n+\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n+\t\t write_graph_chunk_fanout, GRAPH_FANOUT_SIZE);\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP,\n+\t\t write_graph_chunk_oids, hashsz * ctx->commits.nr);\n+\tadd_chunk(cf, GRAPH_CHUNKID_DATA,\n+\t\t write_graph_chunk_data, (hashsz + 16) * ctx->commits.nr);\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_NO_GDAT, 0))\n \t\tctx->write_generation_data = 0;\n-\tif (ctx->write_generation_data) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_generation_data_overflows) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW;\n-\t\tchunks[num_chunks].size = sizeof(timestamp_t) * ctx->num_generation_data_overflows;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data_overflow;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_extra_edges) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n-\t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n-\t\tnum_chunks++;\n-\t}\n+\tif (ctx->write_generation_data)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t\t write_graph_chunk_generation_data,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr);\n+\tif (ctx->num_generation_data_overflows)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t\t write_graph_chunk_generation_data_overflow,\n+\t\t\t sizeof(timestamp_t) * ctx->num_generation_data_overflows);\n+\tif (ctx->num_extra_edges)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES,\n+\t\t\t write_graph_chunk_extra_edges,\n+\t\t\t 4 * ctx->num_extra_edges);\n \tif (ctx->changed_paths) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n-\t\tnum_chunks++;\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n-\t\t\t\t\t + ctx->total_bloom_filter_data_size;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n-\t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n-\t\tnum_chunks++;\n-\t}\n-\n-\tchunks[num_chunks].id = 0;\n-\tchunks[num_chunks].size = 0;\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t write_graph_chunk_bloom_indexes,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr);\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t write_graph_chunk_bloom_data,\n+\t\t\t sizeof(uint32_t) * 3\n+\t\t\t\t+ ctx->total_bloom_filter_data_size);\n+\t}\n+\tif (ctx->num_commit_graphs_after > 1)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BASE,\n+\t\t\t write_graph_chunk_base,\n+\t\t\t hashsz * (ctx->num_commit_graphs_after - 1));\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n-\tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\n-\t\tchunk_offset += chunks[i].size;\n-\t}\n-\n \tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n@@ -1905,17 +1870,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tuint64_t start_offset = f->total + f->offset;\n-\n-\t\tif (chunks[i].write_fn(f, ctx))\n-\t\t\treturn -1;\n-\n-\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n-\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n-\t\t\t chunks[i].size, chunks[i].id,\n-\t\t\t f->total + f->offset - start_offset);\n-\t}\n+\twrite_chunkfile(cf, ctx);\n \n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n@@ -1932,6 +1887,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tclose_commit_graph(ctx->r->objects);\n \tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfree_chunkfile(cf);\n \n \tif (ctx->split) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n-- \ngitgitgadget\n\n"},{"id":"415428","messageId":"2f1c496f3ab537820861fb8563b65f2c2fe15136.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 07/17] midx: add pack_perm to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:46Z","receivedAt":"2021-01-27T15:07:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t *pack_perm\" and large_offsets_needed bit\ninto the context.\n\nUpdate write_midx_object_offsets() to match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 40 +++++++++++++++++++++-------------------\n 1 file changed, 21 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4520ef82b91..cd994e333ec 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -461,6 +461,9 @@ struct write_midx_context {\n \n \tstruct pack_midx_entry *entries;\n \tuint32_t entries_nr;\n+\n+\tuint32_t *pack_perm;\n+\tunsigned large_offsets_needed:1;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -736,27 +739,27 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n-\t\t\t\t\tuint32_t *perm,\n-\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t\tvoid *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n \n-\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n-\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n \t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->offset);\n@@ -805,13 +808,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n-\tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -857,11 +858,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n+\tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n-\t\t\tlarge_offsets_needed = 1;\n+\t\t\tctx.large_offsets_needed = 1;\n \t}\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n@@ -900,13 +902,13 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n@@ -927,7 +929,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n-\tnum_chunks = large_offsets_needed ? 5 : 4;\n+\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -954,7 +956,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (large_offsets_needed) {\n+\tif (ctx.large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n \t\tcur_chunk++;\n@@ -1004,7 +1006,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -1039,7 +1041,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfree(ctx.info);\n \tfree(ctx.entries);\n-\tfree(pack_perm);\n+\tfree(ctx.pack_perm);\n \tfree(midx_name);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"415429","messageId":"814512f216719d89f41822753d5c71df5e49385d.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:41Z","receivedAt":"2021-01-27T15:08:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of combining the logic from the commit-graph and\nmulti-pack-index file formats, create a new chunk-format API. Use a\n'struct chunkfile' pointer to keep track of data that has been\nregistered for writes. This struct is anonymous outside of\nchunk-format.c to ensure no user attempts to interfere with the data.\n\nThe next change will use this API in commit-graph.c, but the general\napproach is:\n\n 1. initialize the chunkfile with init_chunkfile(f).\n 2. add chunks in the intended writing order with add_chunk().\n 3. write any header information to the hashfile f.\n 4. write the chunkfile data using write_chunkfile().\n 5. free the chunkfile struct using free_chunkfile().\n\nHelped-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n chunk-format.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 20 +++++++++++\n 3 files changed, 112 insertions(+)\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\ndiff --git a/Makefile b/Makefile\nindex 7b64106930a..50a7663841e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -854,6 +854,7 @@ LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n+LIB_OBJS += chunk-format.o\n LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\ndiff --git a/chunk-format.c b/chunk-format.c\nnew file mode 100644\nindex 00000000000..ab914c55856\n--- /dev/null\n+++ b/chunk-format.c\n@@ -0,0 +1,91 @@\n+#include \"cache.h\"\n+#include \"chunk-format.h\"\n+#include \"csum-file.h\"\n+#define CHUNK_LOOKUP_WIDTH 12\n+\n+/*\n+ * When writing a chunk-based file format, collect the chunks in\n+ * an array of chunk_info structs. The size stores the _expected_\n+ * amount of data that will be written by write_fn.\n+ */\n+struct chunk_info {\n+\tuint32_t id;\n+\tuint64_t size;\n+\tchunk_write_fn write_fn;\n+};\n+\n+struct chunkfile {\n+\tstruct hashfile *f;\n+\n+\tstruct chunk_info *chunks;\n+\tsize_t chunks_nr;\n+\tsize_t chunks_alloc;\n+};\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f)\n+{\n+\tstruct chunkfile *cf = xcalloc(1, sizeof(*cf));\n+\tcf->f = f;\n+\treturn cf;\n+}\n+\n+void free_chunkfile(struct chunkfile *cf)\n+{\n+\tif (!cf)\n+\t\treturn;\n+\tfree(cf->chunks);\n+\tfree(cf);\n+}\n+\n+int get_num_chunks(struct chunkfile *cf)\n+{\n+\treturn cf->chunks_nr;\n+}\n+\n+void add_chunk(struct chunkfile *cf,\n+\t uint64_t id,\n+\t chunk_write_fn fn,\n+\t size_t size)\n+{\n+\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n+\n+\tcf->chunks[cf->chunks_nr].id = id;\n+\tcf->chunks[cf->chunks_nr].write_fn = fn;\n+\tcf->chunks[cf->chunks_nr].size = size;\n+\tcf->chunks_nr++;\n+}\n+\n+int write_chunkfile(struct chunkfile *cf, void *data)\n+{\n+\tint i;\n+\tsize_t cur_offset = cf->f->offset + cf->f->total;\n+\n+\t/* Add the table of contents to the current offset */\n+\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n+\t\thashwrite_be64(cf->f, cur_offset);\n+\n+\t\tcur_offset += cf->chunks[i].size;\n+\t}\n+\n+\t/* Trailing entry marks the end of the chunks */\n+\thashwrite_be32(cf->f, 0);\n+\thashwrite_be64(cf->f, cur_offset);\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tuint64_t start_offset = cf->f->total + cf->f->offset;\n+\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\n+\t\tif (cf->f->total + cf->f->offset - start_offset != cf->chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n+\t\t\t cf->f->total + cf->f->offset - start_offset);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nnew file mode 100644\nindex 00000000000..bfaed672813\n--- /dev/null\n+++ b/chunk-format.h\n@@ -0,0 +1,20 @@\n+#ifndef CHUNK_FORMAT_H\n+#define CHUNK_FORMAT_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct hashfile;\n+struct chunkfile;\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f);\n+void free_chunkfile(struct chunkfile *cf);\n+int get_num_chunks(struct chunkfile *cf);\n+typedef int (*chunk_write_fn)(struct hashfile *f,\n+\t\t\t void *data);\n+void add_chunk(struct chunkfile *cf,\n+\t uint64_t id,\n+\t chunk_write_fn fn,\n+\t size_t size);\n+int write_chunkfile(struct chunkfile *cf, void *data);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"415430","messageId":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.git.1611676886.gitgitgadget@gmail.com","subject":"[PATCH v2 00/17] Refactor chunk-format into an API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:39Z","receivedAt":"2021-01-27T15:08:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a restart on the topic previously submitted [1] but dropped because\nak/corrected-commit-date was still in progress. This version is based on\nthat branch.\n\n[1]\nhttps://lore.kernel.org/git/pull.804.git.1607012215.gitgitgadget@gmail.com/\n\nThis version also changes the approach to use a more dynamic interaction\nwith a struct chunkfile pointer. This idea is credited to Taylor Blau [2],\nbut I started again from scratch. I also go further to make struct chunkfile\nanonymous to API consumers. It is defined only in chunk-format.c, which\nshould hopefully deter future users from interacting with that data\ndirectly.\n\n[2] https://lore.kernel.org/git/X8%2FI%2FRzXZksio+ri@nand.local/\n\nThis combined API is beneficial to reduce duplicated logic. Or rather, to\nensure that similar file formats have similar protections against bad data.\nThe multi-pack-index code did not have as many guards as the commit-graph\ncode did, but now they both share a common base that checks for things like\nduplicate chunks or offsets outside the size of the file.\n\nHere are some stats for the end-to-end change:\n\n * 570 insertions(+), 456 deletions(-).\n * commit-graph.c: 107 insertions(+), 192 deletions(-)\n * midx.c: 164 insertions(+), 260 deletions(-)\n\nWhile there is an overall increase to the code size, the consumers do get\nsmaller. Boilerplate things like abstracting method to match chunk_write_fn\nand chunk_read_fn make up a lot of these insertions. The \"interesting\" code\ngets a lot smaller and cleaner.\n\n\nUpdates in V2\n=============\n\n * The method pair_chunk() now automatically sets a pointer while\n read_chunk() uses the callback. This greatly reduces the code size.\n\n * Pointer casts are now implicit instead of explicit.\n\n * Extra care is taken to not overflow when verifying chunk sizes on write.\n\nThanks, -Stolee\n\nDerrick Stolee (17):\n commit-graph: anonymize data in chunk_write_fn\n chunk-format: create chunk format write API\n commit-graph: use chunk-format write API\n midx: rename pack_info to write_midx_context\n midx: use context in write_midx_pack_names()\n midx: add entries to write_midx_context\n midx: add pack_perm to write_midx_context\n midx: add num_large_offsets to write_midx_context\n midx: return success/failure in chunk write methods\n midx: drop chunk progress during write\n midx: use chunk-format API in write_midx_internal()\n chunk-format: create read chunk API\n commit-graph: use chunk-format read API\n midx: use chunk-format read API\n midx: use 64-bit multiplication for chunk sizes\n chunk-format: restore duplicate chunk checks\n chunk-format: add technical docs\n\n Documentation/technical/chunk-format.txt | 54 +++\n .../technical/commit-graph-format.txt | 3 +\n Documentation/technical/pack-format.txt | 3 +\n Makefile | 1 +\n chunk-format.c | 181 ++++++++\n chunk-format.h | 53 +++\n commit-graph.c | 299 +++++-------\n midx.c | 424 +++++++-----------\n t/t5318-commit-graph.sh | 2 +-\n t/t5319-multi-pack-index.sh | 6 +-\n 10 files changed, 570 insertions(+), 456 deletions(-)\n create mode 100644 Documentation/technical/chunk-format.txt\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\n\nbase-commit: 5a3b130cad0d5c770f766e3af6d32b41766374c0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-848%2Fderrickstolee%2Fchunk-format%2Frefactor-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-848/derrickstolee/chunk-format/refactor-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/848\n\nRange-diff vs v1:\n\n 1: 09b32829e4f ! 1: 243dcec9436 commit-graph: anonymize data in chunk_write_fn\n @@ commit-graph.c: struct write_commit_graph_context {\n -\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n \n @@ commit-graph.c: static int write_graph_chunk_fanout(struct hashfile *f,\n -\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n @@ commit-graph.c: static const unsigned char *commit_to_sha1(size_t index, void *t\n -\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n @@ commit-graph.c: static int write_graph_chunk_data(struct hashfile *f,\n -\t\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tint i, num_generation_data_overflows = 0;\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n @@ commit-graph.c: static int write_graph_chunk_generation_data(struct hashfile *f,\n -\t\t\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tint i;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n @@ commit-graph.c: static int write_graph_chunk_generation_data_overflow(struct has\n -\t\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n @@ commit-graph.c: static int write_graph_chunk_extra_edges(struct hashfile *f,\n -\t\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t cur_pos = 0;\n @@ commit-graph.c: static void trace2_bloom_filter_settings(struct write_commit_gra\n -\t\t\t\t\tstruct write_commit_graph_context *ctx)\n +\t\t\t\t\tvoid *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \n @@ commit-graph.c: static int write_graph_chunk_base_1(struct hashfile *f,\n -\t\t\t\t struct write_commit_graph_context *ctx)\n +\t\t\t\t void *data)\n {\n -+\tstruct write_commit_graph_context *ctx =\n -+\t\t(struct write_commit_graph_context *)data;\n ++\tstruct write_commit_graph_context *ctx = data;\n \tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n \n \tif (num != ctx->num_commit_graphs_after - 1) {\n 2: 9bd273f8c94 ! 2: 814512f2167 chunk-format: create chunk format write API\n @@ chunk-format.c (new)\n +\t\tif (result)\n +\t\t\treturn result;\n +\n -+\t\tif (cf->f->total + cf->f->offset != start_offset + cf->chunks[i].size)\n ++\t\tif (cf->f->total + cf->f->offset - start_offset != cf->chunks[i].size)\n +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n +\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n +\t\t\t cf->f->total + cf->f->offset - start_offset);\n 3: a3d6177a352 = 3: 70af6e3083f commit-graph: use chunk-format write API\n 4: 9fe5ee8611c ! 4: 0cac7890bed midx: rename pack_info to write_midx_context\n @@ midx.c: struct pack_list {\n \t\t\t const char *file_name, void *data)\n {\n -\tstruct pack_list *packs = (struct pack_list *)data;\n -+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n ++\tstruct write_midx_context *ctx = data;\n \n \tif (ends_with(file_name, \".idx\")) {\n -\t\tdisplay_progress(packs->progress, ++packs->pack_paths_checked);\n 5: 14a0246b982 ! 5: 4a4e90b129a midx: use context in write_midx_pack_names()\n @@ midx.c: static struct pack_midx_entry *get_sorted_entries(struct multi_pack_inde\n -\t\t\t\t uint32_t num_packs)\n +static size_t write_midx_pack_names(struct hashfile *f, void *data)\n {\n -+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n ++\tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n \tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n \tsize_t written = 0;\n 6: 79f479ef7d1 ! 6: 30ad423997b midx: add entries to write_midx_context\n @@ midx.c: static size_t write_midx_pack_names(struct hashfile *f, void *data)\n {\n -\tstruct pack_midx_entry *list = objects;\n -\tstruct pack_midx_entry *last = objects + nr_objects;\n -+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n ++\tstruct write_midx_context *ctx = data;\n +\tstruct pack_midx_entry *list = ctx->entries;\n +\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n \tuint32_t count = 0;\n @@ midx.c: static size_t write_midx_oid_fanout(struct hashfile *f,\n +\t\t\t\t void *data)\n {\n -\tstruct pack_midx_entry *list = objects;\n -+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n ++\tstruct write_midx_context *ctx = data;\n +\tunsigned char hash_len = the_hash_algo->rawsz;\n +\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n 7: 0b4ce3f1732 ! 7: 2f1c496f3ab midx: add pack_perm to write_midx_context\n @@ midx.c: static size_t write_midx_oid_lookup(struct hashfile *f,\n +\t\t\t\t\tvoid *data)\n {\n -\tstruct pack_midx_entry *list = objects;\n -+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n ++\tstruct write_midx_context *ctx = data;\n +\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n \tsize_t written = 0;\n 8: eabc7b73647 ! 8: c4939548e51 midx: add num_large_offsets to write_midx_context\n @@ midx.c: static size_t write_midx_object_offsets(struct hashfile *f,\n +\t\t\t\t void *data)\n {\n -\tstruct pack_midx_entry *list = objects, *end = objects + nr_objects;\n -+\tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n ++\tstruct write_midx_context *ctx = data;\n +\tstruct pack_midx_entry *list = ctx->entries;\n +\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n \tsize_t written = 0;\n 9: 909ca28e0ba ! 9: b3cc73c2256 midx: return success/failure in chunk write methods\n @@ midx.c: static struct pack_midx_entry *get_sorted_entries(struct multi_pack_inde\n -static size_t write_midx_pack_names(struct hashfile *f, void *data)\n +static int write_midx_pack_names(struct hashfile *f, void *data)\n {\n - \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n + \tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n @@ midx.c: static size_t write_midx_pack_names(struct hashfile *f, void *data)\n \tif (i < MIDX_CHUNK_ALIGNMENT) {\n @@ midx.c: static size_t write_midx_pack_names(struct hashfile *f, void *data)\n +static int write_midx_oid_fanout(struct hashfile *f,\n +\t\t\t\t void *data)\n {\n - \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n + \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n @@ midx.c: static size_t write_midx_oid_fanout(struct hashfile *f,\n \t\tlist = next;\n @@ midx.c: static size_t write_midx_oid_fanout(struct hashfile *f,\n +static int write_midx_oid_lookup(struct hashfile *f,\n +\t\t\t\t void *data)\n {\n - \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n + \tstruct write_midx_context *ctx = data;\n \tunsigned char hash_len = the_hash_algo->rawsz;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n @@ midx.c: static size_t write_midx_oid_lookup(struct hashfile *f,\n +static int write_midx_object_offsets(struct hashfile *f,\n +\t\t\t\t void *data)\n {\n - \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n + \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n -\tsize_t written = 0;\n @@ midx.c: static size_t write_midx_object_offsets(struct hashfile *f,\n +static int write_midx_large_offsets(struct hashfile *f,\n +\t\t\t\t void *data)\n {\n - \tstruct write_midx_context *ctx = (struct write_midx_context *)data;\n + \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n -\tsize_t written = 0;\n 10: e613ffa9ac6 = 10: 78744d3b701 midx: drop chunk progress during write\n 11: 49cfb4f63e2 = 11: 07dc0cf8c68 midx: use chunk-format API in write_midx_internal()\n 12: e3475633e1d ! 12: d8d8e9e2aa3 chunk-format: create read chunk API\n @@ Commit message\n \n 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n 2. call read_table_of_contents().\n - 3. for each chunk to parse, call pair_chunk() with appropriate pointers.\n + 3. for each chunk to parse,\n + a. call pair_chunk() to assign a pointer with the chunk position, or\n + b. call read_chunk() to run a callback on the chunk start and size.\n 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n \n We are re-using the anonymous 'struct chunkfile' data, as it is internal\n @@ chunk-format.c: int write_chunkfile(struct chunkfile *cf, void *data)\n +\n +int pair_chunk(struct chunkfile *cf,\n +\t uint32_t chunk_id,\n ++\t const unsigned char **p)\n ++{\n ++\tint i;\n ++\n ++\tfor (i = 0; i < cf->chunks_nr; i++) {\n ++\t\tif (cf->chunks[i].id == chunk_id) {\n ++\t\t\t*p = cf->chunks[i].start;\n ++\t\t\treturn 0;\n ++\t\t}\n ++\t}\n ++\n ++\treturn CHUNK_NOT_FOUND;\n ++}\n ++\n ++int read_chunk(struct chunkfile *cf,\n ++\t uint32_t chunk_id,\n +\t chunk_read_fn fn,\n +\t void *data)\n +{\n @@ chunk-format.h: void add_chunk(struct chunkfile *cf,\n +\t\t\t uint64_t toc_offset,\n +\t\t\t int toc_length);\n +\n ++#define CHUNK_NOT_FOUND (-2)\n ++\n +/*\n -+ * When reading a table of contents, we find the chunk with matching 'id'\n -+ * then call its read_fn to populate the necessary 'data' based on the\n -+ * chunk start and size.\n ++ * Find 'chunk_id' in the given chunkfile and assign the\n ++ * given pointer to the position in the mmap'd file where\n ++ * that chunk begins.\n ++ *\n ++ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n + */\n ++int pair_chunk(struct chunkfile *cf,\n ++\t uint32_t chunk_id,\n ++\t const unsigned char **p);\n ++\n +typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n +\t\t\t size_t chunk_size, void *data);\n -+\n -+\n -+#define CHUNK_NOT_FOUND (-2)\n -+int pair_chunk(struct chunkfile *cf,\n ++/*\n ++ * Find 'chunk_id' in the given chunkfile and call the\n ++ * given chunk_read_fn method with the information for\n ++ * that chunk.\n ++ *\n ++ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n ++ */\n ++int read_chunk(struct chunkfile *cf,\n +\t uint32_t chunk_id,\n +\t chunk_read_fn fn,\n +\t void *data);\n 13: 7339990f07d ! 13: 8744d278596 commit-graph: use chunk-format read API\n @@ commit-graph.c: static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n -+static int graph_read_oid_fanout(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_oid_fanout = (uint32_t*)chunk_start;\n -+\treturn 0;\n -+}\n -+\n +static int graph_read_oid_lookup(const unsigned char *chunk_start,\n +\t\t\t\t size_t chunk_size, void *data)\n +{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n ++\tstruct commit_graph *g = data;\n +\tg->chunk_oid_lookup = chunk_start;\n +\tg->num_commits = chunk_size / g->hash_len;\n +\treturn 0;\n +}\n +\n -+static int graph_read_data(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_commit_data = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int graph_read_extra_edges(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_extra_edges = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int graph_read_base_graphs(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_base_graphs = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int graph_read_generation_data(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_generation_data = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int graph_read_generation_overflow(const unsigned char *chunk_start,\n -+\t\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_generation_data_overflow = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int graph_read_bloom_indices(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n -+\tg->chunk_bloom_indexes = chunk_start;\n -+\treturn 0;\n -+}\n -+\n +static int graph_read_bloom_data(const unsigned char *chunk_start,\n +\t\t\t\t size_t chunk_size, void *data)\n +{\n -+\tstruct commit_graph *g = (struct commit_graph *)data;\n ++\tstruct commit_graph *g = data;\n +\tuint32_t hash_version;\n +\tg->chunk_bloom_data = chunk_start;\n +\thash_version = get_be32(chunk_start);\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n -\t\t\t\t\t\t / graph->hash_len;\n -\t\t\t}\n -\t\t\tbreak;\n --\n ++\tcf = init_chunkfile(NULL);\n + \n -\t\tcase GRAPH_CHUNKID_DATA:\n -\t\t\tif (graph->chunk_commit_data)\n -\t\t\t\tchunk_repeated = 1;\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n -\t\t\telse\n -\t\t\t\tgraph->chunk_generation_data = data + chunk_offset;\n -\t\t\tbreak;\n -+\tcf = init_chunkfile(NULL);\n - \n +-\n -\t\tcase GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW:\n -\t\t\tif (graph->chunk_generation_data_overflow)\n -\t\t\t\tchunk_repeated = 1;\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n -\t\t\t}\n -\t\t\tbreak;\n -\t\t}\n -+\tpair_chunk(cf, GRAPH_CHUNKID_OIDFANOUT, graph_read_oid_fanout, graph);\n -+\tpair_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, graph_read_oid_lookup, graph);\n -+\tpair_chunk(cf, GRAPH_CHUNKID_DATA, graph_read_data, graph);\n -+\tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, graph_read_extra_edges, graph);\n -+\tpair_chunk(cf, GRAPH_CHUNKID_BASE, graph_read_base_graphs, graph);\n ++\tpair_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n ++\t\t (const unsigned char **)&graph->chunk_oid_fanout);\n ++\tread_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, graph_read_oid_lookup, graph);\n ++\tpair_chunk(cf, GRAPH_CHUNKID_DATA, &graph->chunk_commit_data);\n ++\tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, &graph->chunk_extra_edges);\n ++\tpair_chunk(cf, GRAPH_CHUNKID_BASE, &graph->chunk_base_graphs);\n +\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n -+\t\t graph_read_generation_data, graph);\n ++\t\t &graph->chunk_generation_data);\n +\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n -+\t\t graph_read_generation_overflow, graph);\n ++\t\t &graph->chunk_generation_data_overflow);\n \n -\t\tif (chunk_repeated) {\n -\t\t\terror(_(\"commit-graph chunk id %08x appears multiple times\"), chunk_id);\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n -\t\t}\n +\tif (r->settings.commit_graph_read_changed_paths) {\n +\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n -+\t\t\t graph_read_bloom_indices, graph);\n -+\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n ++\t\t\t &graph->chunk_bloom_indexes);\n ++\t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n +\t\t\t graph_read_bloom_data, graph);\n \t}\n \n 14: cb145e0e32a ! 14: 750c03253c9 midx: use chunk-format read API\n @@ midx.c: static char *get_midx_filename(const char *object_dir)\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n -+static int midx_read_pack_names(const unsigned char *chunk_start,\n -+\t\t\t\tsize_t chunk_size, void *data)\n -+{\n -+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n -+\tm->chunk_pack_names = chunk_start;\n -+\treturn 0;\n -+}\n -+\n +static int midx_read_oid_fanout(const unsigned char *chunk_start,\n +\t\t\t\tsize_t chunk_size, void *data)\n +{\n -+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n ++\tstruct multi_pack_index *m = data;\n +\tm->chunk_oid_fanout = (uint32_t *)chunk_start;\n +\n +\tif (chunk_size != 4 * 256) {\n @@ midx.c: static char *get_midx_filename(const char *object_dir)\n +\t}\n +\treturn 0;\n +}\n -+\n -+static int midx_read_oid_lookup(const unsigned char *chunk_start,\n -+\t\t\t\tsize_t chunk_size, void *data)\n -+{\n -+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n -+\tm->chunk_oid_lookup = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int midx_read_offsets(const unsigned char *chunk_start,\n -+\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n -+\tm->chunk_object_offsets = chunk_start;\n -+\treturn 0;\n -+}\n -+\n -+static int midx_read_large_offsets(const unsigned char *chunk_start,\n -+\t\t\t\t size_t chunk_size, void *data)\n -+{\n -+\tstruct multi_pack_index *m = (struct multi_pack_index *)data;\n -+\tm->chunk_large_offsets = chunk_start;\n -+\treturn 0;\n -+}\n +\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n @@ midx.c: struct multi_pack_index *load_multi_pack_index(const char *object_dir, i\n +\t\t\t\t MIDX_HEADER_SIZE, m->num_chunks))\n +\t\tgoto cleanup_fail;\n +\n -+\tif (pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, midx_read_pack_names, m) == CHUNK_NOT_FOUND)\n ++\tif (pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, &m->chunk_pack_names) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n -\tif (!m->chunk_oid_fanout)\n -+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDFANOUT, midx_read_oid_fanout, m) == CHUNK_NOT_FOUND)\n ++\tif (read_chunk(cf, MIDX_CHUNKID_OIDFANOUT, midx_read_oid_fanout, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n -\tif (!m->chunk_oid_lookup)\n -+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDLOOKUP, midx_read_oid_lookup, m) == CHUNK_NOT_FOUND)\n ++\tif (pair_chunk(cf, MIDX_CHUNKID_OIDLOOKUP, &m->chunk_oid_lookup) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n -\tif (!m->chunk_object_offsets)\n -+\tif (pair_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS, midx_read_offsets, m) == CHUNK_NOT_FOUND)\n ++\tif (pair_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS, &m->chunk_object_offsets) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n -+\tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, midx_read_large_offsets, m);\n ++\tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n +\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n 15: f6c58ff72d2 = 15: 83d292532a0 midx: use 64-bit multiplication for chunk sizes\n 16: 62a23842aa6 = 16: 669eeec707a chunk-format: restore duplicate chunk checks\n 17: 05cbd0a8d93 = 17: 8f3985ab5df chunk-format: add technical docs\n\n-- \ngitgitgadget\n"},{"id":"415431","messageId":"83d292532a0fa3f3a0ad343421be4a99a03471d0.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:54Z","receivedAt":"2021-01-27T15:35:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen calculating the sizes of certain chunks, we should use 64-bit\nmultiplication always. This allows us to properly predict the chunk\nsizes without risk of overflow.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e94dcd34b7f..a365dac6bbc 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -913,7 +913,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n \t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n \tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n-\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n+\t\t write_midx_oid_lookup, (uint64_t)ctx.entries_nr * the_hash_algo->rawsz);\n \tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n \t\t write_midx_object_offsets,\n \t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n@@ -921,7 +921,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.large_offsets_needed)\n \t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n \t\t\twrite_midx_large_offsets,\n-\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n+\t\t\t(uint64_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n-- \ngitgitgadget\n\n"},{"id":"415432","messageId":"8f3985ab5df3e4abc6de6db7f71f1adcbc16b4a8.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 17/17] chunk-format: add technical docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:56Z","receivedAt":"2021-01-27T15:35:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-based file format is now an API in the code, but we should\nalso take time to document it as a file format. Specifically, it matches\nthe CHUNK LOOKUP sections of the commit-graph and multi-pack-index\nfiles, but there are some commonalities that should be grouped in this\ndocument.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/chunk-format.txt | 54 +++++++++++++++++++\n .../technical/commit-graph-format.txt | 3 ++\n Documentation/technical/pack-format.txt | 3 ++\n 3 files changed, 60 insertions(+)\n create mode 100644 Documentation/technical/chunk-format.txt\n\ndiff --git a/Documentation/technical/chunk-format.txt b/Documentation/technical/chunk-format.txt\nnew file mode 100644\nindex 00000000000..3db3792dea2\n--- /dev/null\n+++ b/Documentation/technical/chunk-format.txt\n@@ -0,0 +1,54 @@\n+Chunk-based file formats\n+========================\n+\n+Some file formats in Git use a common concept of \"chunks\" to describe\n+sections of the file. This allows structured access to a large file by\n+scanning a small \"table of contents\" for the remaining data. This common\n+format is used by the `commit-graph` and `multi-pack-index` files. See\n+link:technical/pack-format.html[the `multi-pack-index` format] and\n+link:technical/commit-graph-format.html[the `commit-graph` format] for\n+how they use the chunks to describe structured data.\n+\n+A chunk-based file format begins with some header information custom to\n+that format. That header should include enough information to identify\n+the file type, format version, and number of chunks in the file. From this\n+information, that file can determine the start of the chunk-based region.\n+\n+The chunk-based region starts with a table of contents describing where\n+each chunk starts and ends. This consists of (C+1) rows of 12 bytes each,\n+where C is the number of chunks. Consider the following table:\n+\n+ | Chunk ID (4 bytes) | Chunk Offset (8 bytes) |\n+ |--------------------|------------------------|\n+ | ID[0] | OFFSET[0] |\n+ | ... | ... |\n+ | ID[C] | OFFSET[C] |\n+ | 0x0000 | OFFSET[C+1] |\n+\n+Each row consists of a 4-byte chunk identifier (ID) and an 8-byte offset.\n+Each integer is stored in network-byte order.\n+\n+The chunk identifier `ID[i]` is a label for the data stored within this\n+fill from `OFFSET[i]` (inclusive) to `OFFSET[i+1]` (exclusive). Thus, the\n+size of the `i`th chunk is equal to the difference between `OFFSET[i+1]`\n+and `OFFSET[i]`. This requires that the chunk data appears contiguously\n+in the same order as the table of contents.\n+\n+The final entry in the table of contents must be four zero bytes. This\n+confirms that the table of contents is ending and provides the offset for\n+the end of the chunk-based data.\n+\n+Note: The chunk-based format expects that the file contains _at least_ a\n+trailing hash after `OFFSET[C+1]`.\n+\n+Functions for working with chunk-based file formats are declared in\n+`chunk-format.h`. Using these methods provide extra checks that assist\n+developers when creating new file formats, including:\n+\n+ 1. Writing and reading the table of contents.\n+\n+ 2. Verifying that the data written in a chunk matches the expected size\n+ that was recorded in the table of contents.\n+\n+ 3. Checking that a table of contents describes offsets properly within\n+ the file boundaries.\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex b6658eff188..87971c27dd7 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -61,6 +61,9 @@ CHUNK LOOKUP:\n the length using the next chunk position if necessary.) Each chunk\n ID appears at most once.\n \n+ The CHUNK LOOKUP matches the table of contents from\n+ link:technical/chunk-format.html[the chunk-based file format].\n+\n The remaining data in the body is described one chunk at a time, and\n these chunks may be given in any order. Chunks are required unless\n otherwise specified.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex f96b2e605f3..2fb1e60d29e 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -301,6 +301,9 @@ CHUNK LOOKUP:\n \t (Chunks are provided in file-order, so you can infer the length\n \t using the next chunk position if necessary.)\n \n+\tThe CHUNK LOOKUP matches the table of contents from\n+\tlink:technical/chunk-format.html[the chunk-based file format].\n+\n \tThe remaining data in the body is described one chunk at a time, and\n \tthese chunks may be given in any order. Chunks are required unless\n \totherwise specified.\n-- \ngitgitgadget\n"},{"id":"415433","messageId":"750c03253c95cf9fdbcf41bb65058956920ee83e.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 14/17] midx: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:53Z","receivedAt":"2021-01-27T15:35:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). In particular, we\ncan use the return value of pair_chunk() to generate an error when a\nrequired chunk is missing.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 71 +++++++++++++------------------------\n t/t5319-multi-pack-index.sh | 6 ++--\n 2 files changed, 28 insertions(+), 49 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3585e04a706..e94dcd34b7f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -54,6 +54,19 @@ static char *get_midx_filename(const char *object_dir)\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+static int midx_read_oid_fanout(const unsigned char *chunk_start,\n+\t\t\t\tsize_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = data;\n+\tm->chunk_oid_fanout = (uint32_t *)chunk_start;\n+\n+\tif (chunk_size != 4 * 256) {\n+\t\terror(_(\"multi-pack-index OID fanout is of the wrong size\"));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\n@@ -65,6 +78,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n \tconst char *cur_pack_name;\n+\tstruct chunkfile *cf = NULL;\n \n \tfd = git_open(midx_name);\n \n@@ -114,58 +128,23 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n-\tfor (i = 0; i < m->num_chunks; i++) {\n-\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n-\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n-\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\n-\t\tif (chunk_offset >= m->data_len)\n-\t\t\tdie(_(\"invalid chunk offset (too large)\"));\n-\n-\t\tswitch (chunk_id) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase 0:\n-\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\t/*\n-\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n-\t\t\t\t * extensions to add optional chunks.\n-\t\t\t\t */\n-\t\t\t\tbreak;\n-\t\t}\n-\t}\n+\tcf = init_chunkfile(NULL);\n \n-\tif (!m->chunk_pack_names)\n+\tif (read_table_of_contents(cf, m->data, midx_size,\n+\t\t\t\t MIDX_HEADER_SIZE, m->num_chunks))\n+\t\tgoto cleanup_fail;\n+\n+\tif (pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, &m->chunk_pack_names) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n-\tif (!m->chunk_oid_fanout)\n+\tif (read_chunk(cf, MIDX_CHUNKID_OIDFANOUT, midx_read_oid_fanout, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n-\tif (!m->chunk_oid_lookup)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDLOOKUP, &m->chunk_oid_lookup) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n-\tif (!m->chunk_object_offsets)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS, &m->chunk_object_offsets) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n+\tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a94..ad4e878b65b 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -314,12 +314,12 @@ test_expect_success 'verify bad OID version' '\n \n test_expect_success 'verify truncated chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\01\" $objdir \\\n-\t\t\"missing required\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'verify extended chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\07\" $objdir \\\n-\t\t\"terminating multi-pack-index chunk id appears earlier than expected\"\n+\t\t\"terminating chunk id appears earlier than expected\"\n '\n \n test_expect_success 'verify missing required chunk' '\n@@ -329,7 +329,7 @@ test_expect_success 'verify missing required chunk' '\n \n test_expect_success 'verify invalid chunk offset' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_OFFSET \"\\01\" $objdir \\\n-\t\t\"invalid chunk offset (too large)\"\n+\t\t\"improper chunk offset(s)\"\n '\n \n test_expect_success 'verify packnames out of order' '\n-- \ngitgitgadget\n\n"},{"id":"415434","messageId":"8744d2785965773a2b561e9e1e91170530d052b1.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 13/17] commit-graph: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:52Z","receivedAt":"2021-01-27T15:36:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). While the current\nimplementation loses the duplicate-chunk detection, that will be added\nin a future change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 154 ++++++++++++++--------------------------\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 53 insertions(+), 103 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ba33777dcb8..8aa4881d85d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -299,15 +299,43 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n+static int graph_read_oid_lookup(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = data;\n+\tg->chunk_oid_lookup = chunk_start;\n+\tg->num_commits = chunk_size / g->hash_len;\n+\treturn 0;\n+}\n+\n+static int graph_read_bloom_data(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = data;\n+\tuint32_t hash_version;\n+\tg->chunk_bloom_data = chunk_start;\n+\thash_version = get_be32(chunk_start);\n+\n+\tif (hash_version != 1)\n+\t\treturn 0;\n+\n+\tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n+\tg->bloom_filter_settings->hash_version = hash_version;\n+\tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n+\tg->bloom_filter_settings->bits_per_entry = get_be32(chunk_start + 8);\n+\tg->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n+\n+\treturn 0;\n+}\n+\n struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n-\tconst unsigned char *data, *chunk_lookup;\n-\tuint32_t i;\n+\tconst unsigned char *data;\n \tstruct commit_graph *graph;\n-\tuint64_t next_chunk_offset;\n \tuint32_t graph_signature;\n \tunsigned char graph_version, hash_version;\n+\tstruct chunkfile *cf = NULL;\n \n \tif (!graph_map)\n \t\treturn NULL;\n@@ -356,108 +384,28 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n-\tchunk_lookup = data + 8;\n-\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\tfor (i = 0; i < graph->num_chunks; i++) {\n-\t\tuint32_t chunk_id;\n-\t\tuint64_t chunk_offset = next_chunk_offset;\n-\t\tint chunk_repeated = 0;\n-\n-\t\tchunk_id = get_be32(chunk_lookup + 0);\n-\n-\t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n-\t\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\n-\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n-\t\t\terror(_(\"commit-graph improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n-\t\t\t (uint32_t)chunk_offset);\n-\t\t\tgoto free_and_return;\n-\t\t}\n-\n-\t\tswitch (chunk_id) {\n-\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n-\t\t\tif (graph->chunk_oid_fanout)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_oid_fanout = (uint32_t*)(data + chunk_offset);\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n-\t\t\tif (graph->chunk_oid_lookup)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n-\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n-\t\t\t\tgraph->num_commits = (next_chunk_offset - chunk_offset)\n-\t\t\t\t\t\t / graph->hash_len;\n-\t\t\t}\n-\t\t\tbreak;\n+\tcf = init_chunkfile(NULL);\n \n-\t\tcase GRAPH_CHUNKID_DATA:\n-\t\t\tif (graph->chunk_commit_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA:\n-\t\t\tif (graph->chunk_generation_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW:\n-\t\t\tif (graph->chunk_generation_data_overflow)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data_overflow = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_EXTRAEDGES:\n-\t\t\tif (graph->chunk_extra_edges)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BASE:\n-\t\t\tif (graph->chunk_base_graphs)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n-\t\t\tif (graph->chunk_bloom_indexes)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n-\t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMDATA:\n-\t\t\tif (graph->chunk_bloom_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n-\t\t\t\tuint32_t hash_version;\n-\t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n-\t\t\t\thash_version = get_be32(data + chunk_offset);\n-\n-\t\t\t\tif (hash_version != 1)\n-\t\t\t\t\tbreak;\n+\tif (read_table_of_contents(cf, graph->data, graph_size,\n+\t\t\t\t GRAPH_HEADER_SIZE, graph->num_chunks))\n+\t\tgoto free_and_return;\n \n-\t\t\t\tgraph->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n-\t\t\t\tgraph->bloom_filter_settings->hash_version = hash_version;\n-\t\t\t\tgraph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n-\t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n-\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n+\tpair_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n+\t\t (const unsigned char **)&graph->chunk_oid_fanout);\n+\tread_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, graph_read_oid_lookup, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_DATA, &graph->chunk_commit_data);\n+\tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, &graph->chunk_extra_edges);\n+\tpair_chunk(cf, GRAPH_CHUNKID_BASE, &graph->chunk_base_graphs);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t &graph->chunk_generation_data);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t &graph->chunk_generation_data_overflow);\n \n-\t\tif (chunk_repeated) {\n-\t\t\terror(_(\"commit-graph chunk id %08x appears multiple times\"), chunk_id);\n-\t\t\tgoto free_and_return;\n-\t\t}\n+\tif (r->settings.commit_graph_read_changed_paths) {\n+\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t &graph->chunk_bloom_indexes);\n+\t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -474,9 +422,11 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \tif (verify_commit_graph_lite(graph))\n \t\tgoto free_and_return;\n \n+\tfree_chunkfile(cf);\n \treturn graph;\n \n free_and_return:\n+\tfree_chunkfile(cf);\n \tfree(graph->bloom_filter_settings);\n \tfree(graph);\n \treturn NULL;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex fa27df579a5..c7da741284e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -564,7 +564,7 @@ test_expect_success 'detect bad hash version' '\n \n test_expect_success 'detect low chunk count' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\01\" \\\n-\t\t\"missing the .* chunk\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'detect missing OID fanout chunk' '\n-- \ngitgitgadget\n\n"},{"id":"415435","messageId":"669eeec707ab92a3e5983ad12baddc2c15012d43.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 16/17] chunk-format: restore duplicate chunk checks","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:55Z","receivedAt":"2021-01-27T15:36:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore refactoring into the chunk-format API, the commit-graph parsing\nlogic included checks for duplicate chunks. It is unlikely that we would\ndesire a chunk-based file format that allows duplicate chunk IDs in the\ntable of contents, so add duplicate checks into\nread_table_of_contents().\n\nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 74501084cf8..1ee875df423 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -14,6 +14,7 @@ struct chunk_info {\n \tchunk_write_fn write_fn;\n \n \tconst void *start;\n+\tunsigned found:1;\n };\n \n struct chunkfile {\n@@ -98,6 +99,7 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\t uint64_t toc_offset,\n \t\t\t int toc_length)\n {\n+\tint i;\n \tuint32_t chunk_id;\n \tconst unsigned char *table_of_contents = mfile + toc_offset;\n \n@@ -124,6 +126,14 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\treturn -1;\n \t\t}\n \n+\t\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\t\tif (cf->chunks[i].id == chunk_id) {\n+\t\t\t\terror(_(\"duplicate chunk ID %\"PRIx32\" found\"),\n+\t\t\t\t\tchunk_id);\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t}\n+\n \t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n \t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n \t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n-- \ngitgitgadget\n\n"},{"id":"415436","messageId":"07dc0cf8c683676d304ac16fde2338f49e5cc483.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 11/17] midx: use chunk-format API in write_midx_internal()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:50Z","receivedAt":"2021-01-27T15:36:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-format API allows writing the table of contents and all chunks\nusing the anonymous 'struct chunkfile' type. We only need to convert our\nlocal chunk logic to this API for the multi-pack-index writes to share\nthat logic with the commit-graph file writes.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 104 +++++++++++----------------------------------------------\n 1 file changed, 19 insertions(+), 85 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 6ee262aab79..3585e04a706 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -11,6 +11,7 @@\n #include \"trace2.h\"\n #include \"run-command.h\"\n #include \"repository.h\"\n+#include \"chunk-format.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -799,18 +800,15 @@ static int write_midx_large_offsets(struct hashfile *f,\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop, unsigned flags)\n {\n-\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t header_size = 0;\n-\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n-\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tstruct chunkfile *cf;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name))\n@@ -923,98 +921,34 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n \n-\tcur_chunk = 0;\n-\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n-\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n-\n-\tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (ctx.large_offsets_needed) {\n-\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n-\n-\t\tcur_chunk++;\n-\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n-\t}\n-\n-\tchunk_ids[cur_chunk] = 0;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n-\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n-\t\t\t chunk_offsets[i - 1],\n-\t\t\t chunk_offsets[i]);\n-\n-\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n-\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n-\t\t\t chunk_offsets[i]);\n-\n-\t\thashwrite_be32(f, chunk_ids[i]);\n-\t\thashwrite_be64(f, chunk_offsets[i]);\n-\t}\n-\n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (f->total + f->offset != chunk_offsets[i])\n-\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n-\t\t\t chunk_offsets[i],\n-\t\t\t f->total + f->offset,\n-\t\t\t chunk_ids[i]);\n+\tcf = init_chunkfile(f);\n \n-\t\tswitch (chunk_ids[i]) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twrite_midx_pack_names(f, &ctx);\n-\t\t\t\tbreak;\n+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES,\n+\t\t write_midx_pack_names, pack_name_concat_len);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n+\t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n+\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n+\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n+\t\t write_midx_object_offsets,\n+\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n \n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twrite_midx_object_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twrite_midx_large_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n-\t\t\t\t chunk_ids[i]);\n-\t\t}\n-\t}\n+\tif (ctx.large_offsets_needed)\n+\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n+\t\t\twrite_midx_large_offsets,\n+\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n \n-\tif (f->total + f->offset != chunk_offsets[num_chunks])\n-\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t f->total + f->offset,\n-\t\t chunk_offsets[num_chunks]);\n+\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n+\twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n cleanup:\n-- \ngitgitgadget\n\n"},{"id":"415437","messageId":"d8d8e9e2aa3faf0fdda5dc688fb92e924fec423a.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:51Z","receivedAt":"2021-01-27T15:36:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd the capability to read the table of contents, then pair the chunks\nwith necessary logic using read_chunk_fn pointers. Callers will be added\nin future changes, but the typical outline will be:\n\n 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n 2. call read_table_of_contents().\n 3. for each chunk to parse,\n a. call pair_chunk() to assign a pointer with the chunk position, or\n b. call read_chunk() to run a callback on the chunk start and size.\n 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n\nWe are re-using the anonymous 'struct chunkfile' data, as it is internal\nto the chunk-format API. This gives it essentially two modes: write and\nread. If the same struct instance was used for both reads and writes,\nthen there would be failures.\n\nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 33 +++++++++++++++++++++\n 2 files changed, 113 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex ab914c55856..74501084cf8 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -12,6 +12,8 @@ struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n \tchunk_write_fn write_fn;\n+\n+\tconst void *start;\n };\n \n struct chunkfile {\n@@ -89,3 +91,81 @@ int write_chunkfile(struct chunkfile *cf, void *data)\n \n \treturn 0;\n }\n+\n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length)\n+{\n+\tuint32_t chunk_id;\n+\tconst unsigned char *table_of_contents = mfile + toc_offset;\n+\n+\tALLOC_GROW(cf->chunks, toc_length, cf->chunks_alloc);\n+\n+\twhile (toc_length--) {\n+\t\tuint64_t chunk_offset, next_chunk_offset;\n+\n+\t\tchunk_id = get_be32(table_of_contents);\n+\t\tchunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (!chunk_id) {\n+\t\t\terror(_(\"terminating chunk id appears earlier than expected\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\ttable_of_contents += CHUNK_LOOKUP_WIDTH;\n+\t\tnext_chunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (next_chunk_offset < chunk_offset ||\n+\t\t next_chunk_offset > mfile_size - the_hash_algo->rawsz) {\n+\t\t\terror(_(\"improper chunk offset(s) %\"PRIx64\" and %\"PRIx64\"\"),\n+\t\t\t chunk_offset, next_chunk_offset);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n+\t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n+\t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n+\t\tcf->chunks_nr++;\n+\t}\n+\n+\tchunk_id = get_be32(table_of_contents);\n+\tif (chunk_id) {\n+\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t const unsigned char **p)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tif (cf->chunks[i].id == chunk_id) {\n+\t\t\t*p = cf->chunks[i].start;\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn CHUNK_NOT_FOUND;\n+}\n+\n+int read_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tif (cf->chunks[i].id == chunk_id)\n+\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n+\t}\n+\n+\treturn CHUNK_NOT_FOUND;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nindex bfaed672813..b62c9bf8ba1 100644\n--- a/chunk-format.h\n+++ b/chunk-format.h\n@@ -17,4 +17,37 @@ void add_chunk(struct chunkfile *cf,\n \t size_t size);\n int write_chunkfile(struct chunkfile *cf, void *data);\n \n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length);\n+\n+#define CHUNK_NOT_FOUND (-2)\n+\n+/*\n+ * Find 'chunk_id' in the given chunkfile and assign the\n+ * given pointer to the position in the mmap'd file where\n+ * that chunk begins.\n+ *\n+ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n+ */\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t const unsigned char **p);\n+\n+typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size, void *data);\n+/*\n+ * Find 'chunk_id' in the given chunkfile and call the\n+ * given chunk_read_fn method with the information for\n+ * that chunk.\n+ *\n+ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n+ */\n+int read_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"415438","messageId":"b3cc73c22567ed89e2537ffaaa04f82a436d9802.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 09/17] midx: return success/failure in chunk write methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:48Z","receivedAt":"2021-01-27T15:36:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHistorically, the chunk-writing methods in midx.c have returned the\namount of data written so the writer method could compare this with the\ntable of contents. This presents with some interesting issues:\n\n1. If a chunk writing method has a bug that miscalculates the written\n bytes, then we can satisfy the table of contents without actually\n writing the right amount of data to the hashfile. The commit-graph\n writing code checks the hashfile struct directly for a more robust\n verification.\n\n2. There is no way for a chunk writing method to gracefully fail.\n Returning an int presents an opportunity to fail without a die().\n\n3. The current pattern doesn't match chunk_write_fn type exactly, so we\n cannot share code with commit-graph.c\n\nFor these reasons, convert the midx chunk writer methods to return an\n'int'. Since none of them fail at the moment, they all return 0.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 63 +++++++++++++++++++++++++---------------------------------\n 1 file changed, 27 insertions(+), 36 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 5be081f229a..e23a5fc4903 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -650,7 +650,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f, void *data)\n+static int write_midx_pack_names(struct hashfile *f, void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n@@ -678,14 +678,13 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n \tif (i < MIDX_CHUNK_ALIGNMENT) {\n \t\tmemset(padding, 0, sizeof(padding));\n \t\thashwrite(f, padding, i);\n-\t\twritten += i;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n@@ -710,17 +709,16 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \t\tlist = next;\n \t}\n \n-\treturn MIDX_CHUNK_FANOUT_SIZE;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tunsigned char hash_len = the_hash_algo->rawsz;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -734,19 +732,17 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \t\t}\n \n \t\thashwrite(f, obj->oid.hash, (int)hash_len);\n-\t\twritten += hash_len;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f,\n-\t\t\t\t\tvoid *data)\n+static int write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -766,20 +762,17 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \t\t\t obj->offset);\n \t\telse\n \t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n-\n-\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n-\tsize_t written = 0;\n \tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n@@ -795,12 +788,12 @@ static size_t write_midx_large_offsets(struct hashfile *f,\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\twritten += hashwrite_be64(f, offset);\n+\t\thashwrite_be64(f, offset);\n \n \t\tnr_large_offset--;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n@@ -812,7 +805,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t written = 0;\n+\tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tstruct progress *progress = NULL;\n@@ -940,10 +933,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n+\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n@@ -981,39 +974,37 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\thashwrite_be32(f, chunk_ids[i]);\n \t\thashwrite_be64(f, chunk_offsets[i]);\n-\n-\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (written != chunk_offsets[i])\n+\t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n \t\t\t chunk_offsets[i],\n-\t\t\t written,\n+\t\t\t f->total + f->offset,\n \t\t\t chunk_ids[i]);\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n+\t\t\t\twrite_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n+\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n+\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n+\t\t\t\twrite_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n+\t\t\t\twrite_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1025,9 +1016,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \tstop_progress(&progress);\n \n-\tif (written != chunk_offsets[num_chunks])\n+\tif (f->total + f->offset != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t written,\n+\t\t f->total + f->offset,\n \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n-- \ngitgitgadget\n\n"},{"id":"415439","messageId":"78744d3b7016520cfc946e858eb1f6233003bffc.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 10/17] midx: drop chunk progress during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:49Z","receivedAt":"2021-01-27T15:36:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMost expensive operations in write_midx_internal() use the context\nstruct's progress member, and these indicate the process of the\nexpensive operations within the chunk writing methods. However, there is\na competing progress struct that counts the progress over all chunks.\nThis is not very helpful compared to the others, so drop it.\n\nThis also reduces our barriers to combining the chunk writing code with\nchunk-format.c.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e23a5fc4903..6ee262aab79 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -808,7 +808,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -976,9 +975,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\thashwrite_be64(f, chunk_offsets[i]);\n \t}\n \n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n-\t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n@@ -1011,10 +1007,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\n \t\t}\n-\n-\t\tdisplay_progress(progress, i + 1);\n \t}\n-\tstop_progress(&progress);\n \n \tif (f->total + f->offset != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-- \ngitgitgadget\n\n"},{"id":"415442","messageId":"30ad423997b71645c928b1b6f3cbec71e712d31c.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 06/17] midx: add entries to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:45Z","receivedAt":"2021-01-27T15:57:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"struct pack_midx_entry *entries\" list and its count\ninto the context.\n\nUpdate write_midx_oid_fanout() and write_midx_oid_lookup() to take the\ncontext directly, as these are easy conversions with this new data.\n\nOnly the callers of write_midx_object_offsets() and\nwrite_midx_large_offsets() are updated here, since additional data in\nthe context before those methods can match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 49 ++++++++++++++++++++++++++-----------------------\n 1 file changed, 26 insertions(+), 23 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 88452b04433..4520ef82b91 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -458,6 +458,9 @@ struct write_midx_context {\n \tstruct multi_pack_index *m;\n \tstruct progress *progress;\n \tunsigned pack_paths_checked;\n+\n+\tstruct pack_midx_entry *entries;\n+\tuint32_t entries_nr;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -678,11 +681,11 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n }\n \n static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n-\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n \tuint32_t count = 0;\n \tuint32_t i;\n \n@@ -706,18 +709,19 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \treturn MIDX_CHUNK_FANOUT_SIZE;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+static size_t write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = data;\n+\tunsigned char hash_len = the_hash_algo->rawsz;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (i < nr_objects - 1) {\n+\t\tif (i < ctx->entries_nr - 1) {\n \t\t\tstruct pack_midx_entry *next = list;\n \t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n \t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n@@ -805,8 +809,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries, num_large_offsets = 0;\n-\tstruct pack_midx_entry *entries = NULL;\n+\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n@@ -852,12 +855,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n-\tfor (i = 0; i < nr_entries; i++) {\n-\t\tif (entries[i].offset > 0x7fffffff)\n+\tfor (i = 0; i < ctx.entries_nr; i++) {\n+\t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n-\t\tif (entries[i].offset > 0xffffffff)\n+\t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n@@ -947,10 +950,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * the_hash_algo->rawsz;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n \n \tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n \tif (large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n@@ -993,19 +996,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, the_hash_algo->rawsz, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1035,7 +1038,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tfree(ctx.info);\n-\tfree(entries);\n+\tfree(ctx.entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n \treturn result;\n-- \ngitgitgadget\n\n"},{"id":"415443","messageId":"c4939548e51cd22fdf32609224deecd57d49b690.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 08/17] midx: add num_large_offsets to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:47Z","receivedAt":"2021-01-27T15:58:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t num_large_offsets\" into the context. With\nthis new data, write_midx_large_offsets() now matches the\nchunk_write_fn type.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 17 ++++++++++-------\n 1 file changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex cd994e333ec..5be081f229a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -464,6 +464,7 @@ struct write_midx_context {\n \n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n+\tuint32_t num_large_offsets;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -772,11 +773,14 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n-\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects, *end = objects + nr_objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n \tsize_t written = 0;\n+\tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n \t\tstruct pack_midx_entry *obj;\n@@ -811,7 +815,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n@@ -861,7 +864,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n-\t\t\tnum_large_offsets++;\n+\t\t\tctx.num_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n@@ -961,7 +964,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tcur_chunk++;\n \t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n \t}\n \n \tchunk_ids[cur_chunk] = 0;\n@@ -1010,7 +1013,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n-- \ngitgitgadget\n\n"},{"id":"415444","messageId":"0cac7890bed7774296e9984f9891c3e765a9a11d.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 04/17] midx: rename pack_info to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:43Z","receivedAt":"2021-01-27T15:59:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to streamline our chunk-based file formats, align some of\nthe code structure in write_midx_internal() to be similar to the\npatterns in write_commit_graph_file().\n\nSpecifically, let's create a \"struct write_midx_context\" that can be\nused as a data parameter to abstract function types.\n\nThis change only renames \"struct pack_info\" to \"struct\nwrite_midx_context\" and the names of instances from \"packs\" to \"ctx\". In\nfuture changes, we will expand the data inside \"struct\nwrite_midx_context\" and align our chunk-writing method with the\nchunk-format API.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 130 ++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 65 insertions(+), 65 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 79c282b070d..561f65a63a5 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -451,7 +451,7 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n-struct pack_list {\n+struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n \tuint32_t alloc;\n@@ -463,37 +463,37 @@ struct pack_list {\n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t const char *file_name, void *data)\n {\n-\tstruct pack_list *packs = (struct pack_list *)data;\n+\tstruct write_midx_context *ctx = data;\n \n \tif (ends_with(file_name, \".idx\")) {\n-\t\tdisplay_progress(packs->progress, ++packs->pack_paths_checked);\n-\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n+\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n-\t\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t\t 0);\n+\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->info[packs->nr].p) {\n+\t\tif (!ctx->info[ctx->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n+\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->info[packs->nr].p);\n-\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n+\t\t\tclose_pack(ctx->info[ctx->nr].p);\n+\t\t\tFREE_AND_NULL(ctx->info[ctx->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n-\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n-\t\tpacks->info[packs->nr].expired = 0;\n-\t\tpacks->nr++;\n+\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n+\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n+\t\tctx->info[ctx->nr].expired = 0;\n+\t\tctx->nr++;\n \t}\n }\n \n@@ -801,7 +801,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n-\tstruct pack_list packs;\n+\tstruct write_midx_context ctx = { 0 };\n \tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n@@ -820,40 +820,40 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t midx_name);\n \n \tif (m)\n-\t\tpacks.m = m;\n+\t\tctx.m = m;\n \telse\n-\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n-\n-\tpacks.nr = 0;\n-\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n-\tpacks.info = NULL;\n-\tALLOC_ARRAY(packs.info, packs.alloc);\n-\n-\tif (packs.m) {\n-\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n-\n-\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n-\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.info[packs.nr].p = NULL;\n-\t\t\tpacks.info[packs.nr].expired = 0;\n-\t\t\tpacks.nr++;\n+\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\n+\tctx.nr = 0;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.info = NULL;\n+\tALLOC_ARRAY(ctx.info, ctx.alloc);\n+\n+\tif (ctx.m) {\n+\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n+\n+\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n+\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n+\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].expired = 0;\n+\t\t\tctx.nr++;\n \t\t}\n \t}\n \n-\tpacks.pack_paths_checked = 0;\n+\tctx.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n-\t\tpacks.progress = NULL;\n+\t\tctx.progress = NULL;\n \n-\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n-\tstop_progress(&packs.progress);\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n+\tstop_progress(&ctx.progress);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -862,19 +862,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n-\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n \tif (packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n-\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n \t\t\t\t\t packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n-\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n \t\t\t\t packs_to_drop->items[drop_index].string);\n@@ -882,7 +882,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n \t\t\t} else {\n-\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t\tctx.info[i].expired = 0;\n \t\t\t}\n \t\t}\n \n@@ -898,19 +898,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].expired) {\n+\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (!packs.info[i].expired)\n-\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (!ctx.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n@@ -921,19 +921,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\tif (packs.m)\n-\t\tclose_midx(packs.m);\n+\tif (ctx.m)\n+\t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\tif (packs.nr - dropped_packs == 0) {\n+\tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n+\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -990,7 +990,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -1027,15 +1027,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tcommit_lock_file(&lk);\n \n cleanup:\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].p) {\n-\t\t\tclose_pack(packs.info[i].p);\n-\t\t\tfree(packs.info[i].p);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].p) {\n+\t\t\tclose_pack(ctx.info[i].p);\n+\t\t\tfree(ctx.info[i].p);\n \t\t}\n-\t\tfree(packs.info[i].pack_name);\n+\t\tfree(ctx.info[i].pack_name);\n \t}\n \n-\tfree(packs.info);\n+\tfree(ctx.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \ngitgitgadget\n\n"},{"id":"415445","messageId":"243dcec9436853ff8d1bf2580e76ab909b7cb324.1611759716.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v2 01/17] commit-graph: anonymize data in chunk_write_fn","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-01-27T15:01:40Z","receivedAt":"2021-01-27T15:59:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for creating an API around file formats using chunks and\ntables of contents, prepare the commit-graph write code to use\nprototypes that will match this new API.\n\nSpecifically, convert chunk_write_fn to take a \"void *data\" parameter\ninstead of the commit-graph-specific \"struct write_commit_graph_context\"\npointer.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++----------\n 1 file changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f3bde2ad95a..fae7d1b6393 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1040,8 +1040,9 @@ struct write_commit_graph_context {\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n \n@@ -1066,8 +1067,9 @@ static int write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static int write_graph_chunk_oids(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n@@ -1085,8 +1087,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static int write_graph_chunk_data(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n@@ -1187,8 +1190,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i, num_generation_data_overflows = 0;\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n@@ -1208,8 +1212,9 @@ static int write_graph_chunk_generation_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n-\t\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1226,8 +1231,9 @@ static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n }\n \n static int write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n@@ -1280,8 +1286,9 @@ static int write_graph_chunk_extra_edges(struct hashfile *f,\n }\n \n static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t cur_pos = 0;\n@@ -1315,8 +1322,9 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n }\n \n static int write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\tstruct write_commit_graph_context *ctx)\n+\t\t\t\t\tvoid *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \n@@ -1737,8 +1745,9 @@ static int write_graph_chunk_base_1(struct hashfile *f,\n }\n \n static int write_graph_chunk_base(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n \n \tif (num != ctx->num_commit_graphs_after - 1) {\n@@ -1750,7 +1759,7 @@ static int write_graph_chunk_base(struct hashfile *f,\n }\n \n typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t struct write_commit_graph_context *ctx);\n+\t\t\t void *data);\n \n struct chunk_info {\n \tuint32_t id;\n-- \ngitgitgadget\n\n"},{"id":"415446","messageId":"YBGO10nRSCQ78gHW@nand.local","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/17] Refactor chunk-format into an API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-27T16:03:35Z","receivedAt":"2021-01-27T16:06:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 27, 2021 at 03:01:39PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Updates in V2\n> =============\n>\n> * The method pair_chunk() now automatically sets a pointer while\n> read_chunk() uses the callback. This greatly reduces the code size.\n>\n> * Pointer casts are now implicit instead of explicit.\n>\n> * Extra care is taken to not overflow when verifying chunk sizes on write.\n\nThanks, I read the range-diff between this version and the last and\nappreciate you taking the time to address all of my concerns.\n\nI think that this is ready to go, so please have my:\n\n Reviewed-by: Taylor Blau \n\nThanks,\nTaylor\n"},{"id":"416209","messageId":"xmqq5z375xxf.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"814512f216719d89f41822753d5c71df5e49385d.1611759716.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/17] chunk-format: create chunk format write API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-04T21:24:12Z","receivedAt":"2021-02-04T21:25:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +/*\n> + * When writing a chunk-based file format, collect the chunks in\n> + * an array of chunk_info structs. The size stores the _expected_\n> + * amount of data that will be written by write_fn.\n> + */\n> +struct chunk_info {\n> +\tuint32_t id;\n> +\tuint64_t size;\n> +\tchunk_write_fn write_fn;\n> +};\n> +...\n> +void add_chunk(struct chunkfile *cf,\n> +\t uint64_t id,\n> +\t chunk_write_fn fn,\n> +\t size_t size)\n> +{\n> +\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n> +\n> +\tcf->chunks[cf->chunks_nr].id = id;\n> +\tcf->chunks[cf->chunks_nr].write_fn = fn;\n> +\tcf->chunks[cf->chunks_nr].size = size;\n> +\tcf->chunks_nr++;\n> +}\n\nSomebody somewhere between the caller in the higher part of the\ncallchain (that has to work with platform native types) and the\non-disk format at the bottom of the callchain (that has to work\nwith fixed size data fields) needs to make sure that the size that\nthe higher level caller has fits on-disk data structure we define,\nand the data we read from disk fits the in-core structure our caller\nuse on the reading side.\n\nIf there is a function at the one level closer to the disk than\n\"struct chunk_info\" that takes a \"struct chunk_info\" and writes the\nid and size to disk (and fills \"struct chunk_info\" from what is read\nfrom the disk, on the reading side), it would be a good place to do\nthe size_t to uint64_t conversion.\n\nIt is OK to do the conversion-while-checking in add_chunk(), too.\n\nBut a silent type casting from size_t to uint64_t done silently by\nassignment bothers me. Also, I think you meant to make the incoming\nID uint32_t; am I missing something, or did nobody notice it in the\nreview of the previous round?\n\n> +int write_chunkfile(struct chunkfile *cf, void *data)\n> +{\n> +\tint i;\n> +\tsize_t cur_offset = cf->f->offset + cf->f->total;\n\nThat ought to be off_t, as it is a seek position inside a file\n(struct hashfile.total is already off_t).\n\nUse csum-file.h::hashfile_total() instead, perhaps? .offset member\nis an implementation detail of the hashfile API (i.e. some leftover\nbits are kept in in-core buffer, until we accumulate enough to make\nit worth flushing to the disk), and by using the helper, this code\ndoes not have to know about it.\n\n> +\t/* Add the table of contents to the current offset */\n> +\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n\nIs that 12 == sizeof(chunk_info.id) + sizeof(chunk_info.size)?\nIf so, this makes sense.\n\n> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n> +\t\thashwrite_be64(cf->f, cur_offset);\n> +\n> +\t\tcur_offset += cf->chunks[i].size;\n> +\t}\n> +\n> +\t/* Trailing entry marks the end of the chunks */\n> +\thashwrite_be32(cf->f, 0);\n> +\thashwrite_be64(cf->f, cur_offset);\n\nOK. This helper does not tell us anything about what comes in the\non-disk file before this point, but we write a table of contents\nthat says \"chunk with this ID has this size, chunk with that ID has\nthat size, ...\", concluded by something that looks like another\nentry with chunk ID 0 that records the current offset as its size.\n\n> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\tuint64_t start_offset = cf->f->total + cf->f->offset;\n\nLikewise about the type and use of hashfile_total().\n\n> +\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n> +\n> +\t\tif (result)\n> +\t\t\treturn result;\n> +\n> +\t\tif (cf->f->total + cf->f->offset - start_offset != cf->chunks[i].size)\n> +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n> +\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n> +\t\t\t cf->f->total + cf->f->offset - start_offset);\n\nI won't complain, as this apparently is sufficient to abstract out\nthe two existing chunked format files, but it somehow feels a bit\nlimiting that the one that calls add_chunk() is required to know\nwhat the size of generated data would be, way before .write_fn() is\ncalled to produce the actual data here.\n\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> diff --git a/chunk-format.h b/chunk-format.h\n> new file mode 100644\n> index 00000000000..bfaed672813\n> --- /dev/null\n> +++ b/chunk-format.h\n> @@ -0,0 +1,20 @@\n> +#ifndef CHUNK_FORMAT_H\n> +#define CHUNK_FORMAT_H\n> +\n> +#include \"git-compat-util.h\"\n> +\n> +struct hashfile;\n> +struct chunkfile;\n> +\n> +struct chunkfile *init_chunkfile(struct hashfile *f);\n> +void free_chunkfile(struct chunkfile *cf);\n> +int get_num_chunks(struct chunkfile *cf);\n> +typedef int (*chunk_write_fn)(struct hashfile *f,\n> +\t\t\t void *data);\n\nWrite this on a single line.\n\n> +void add_chunk(struct chunkfile *cf,\n> +\t uint64_t id,\n> +\t chunk_write_fn fn,\n> +\t size_t size);\n\nShouldn't this match the order of members in chunk_info struct?\n\n> +int write_chunkfile(struct chunkfile *cf, void *data);\n> +\n> +#endif\n"},{"id":"416213","messageId":"xmqq1rdv5uej.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"xmqq5z375xxf.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 02/17] chunk-format: create chunk format write API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-04T22:40:20Z","receivedAt":"2021-02-04T22:41:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +void add_chunk(struct chunkfile *cf,\n>> +\t uint64_t id,\n>> +\t chunk_write_fn fn,\n>> +\t size_t size);\n>\n> Shouldn't this match the order of members in chunk_info struct?\n\nNah. Unless there are other reasons to touch numerous add_chunk()\ncalls that have already been written (like we find a need to add a\nnew parameter to the call), I do not think this matters too much.\n\nThanks.\n"},{"id":"416217","messageId":"xmqqlfc34eyd.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"b3cc73c22567ed89e2537ffaaa04f82a436d9802.1611759716.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/17] midx: return success/failure in chunk write methods","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-04T22:59:22Z","receivedAt":"2021-02-04T23:00:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Historically, the chunk-writing methods in midx.c have returned the\n> amount of data written so the writer method could compare this with the\n> table of contents. This presents with some interesting issues:\n>\n> 1. If a chunk writing method has a bug that miscalculates the written\n> bytes, then we can satisfy the table of contents without actually\n> writing the right amount of data to the hashfile. The commit-graph\n> writing code checks the hashfile struct directly for a more robust\n> verification.\n>\n> 2. There is no way for a chunk writing method to gracefully fail.\n> Returning an int presents an opportunity to fail without a die().\n>\n> 3. The current pattern doesn't match chunk_write_fn type exactly, so we\n> cannot share code with commit-graph.c\n>\n> For these reasons, convert the midx chunk writer methods to return an\n> 'int'. Since none of them fail at the moment, they all return 0.\n\nOK, that makes sense.\n\nIn this step I see the same \"f->total + f->offset\" sprinkled in some\nplaces, but they will all go away in the \"midx: use chunk-format API\nin write_midx_internal()\" step, so it is not such a huge deal.\n\nThanks.\n"},{"id":"416222","messageId":"xmqqczxf4d2k.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"d8d8e9e2aa3faf0fdda5dc688fb92e924fec423a.1611759716.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-04T23:40:03Z","receivedAt":"2021-02-04T23:41:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> We are re-using the anonymous 'struct chunkfile' data, as it is internal\n> to the chunk-format API. This gives it essentially two modes: write and\n> read. If the same struct instance was used for both reads and writes,\n> then there would be failures.\n\nWriting it here won't help future developers very much. I think\nthat belongs to API/function doc for init_chunkfile().\n\n> diff --git a/chunk-format.c b/chunk-format.c\n> index ab914c55856..74501084cf8 100644\n> --- a/chunk-format.c\n> +++ b/chunk-format.c\n> @@ -12,6 +12,8 @@ struct chunk_info {\n> \tuint32_t id;\n> \tuint64_t size;\n> \tchunk_write_fn write_fn;\n> +\n> +\tconst void *start;\n> };\n> \n> struct chunkfile {\n> @@ -89,3 +91,81 @@ int write_chunkfile(struct chunkfile *cf, void *data)\n> \n> \treturn 0;\n> }\n> +\n> +int read_table_of_contents(struct chunkfile *cf,\n> +\t\t\t const unsigned char *mfile,\n> +\t\t\t size_t mfile_size,\n> +\t\t\t uint64_t toc_offset,\n> +\t\t\t int toc_length)\n\nIt's a bit of mystery, having seen how the table-of-contents is laid\nout by reading the writing side of the code, how toc_offset and\ntoc_length are discovered by the caller. IIRC, the size to cover\neverything from the beginning of the file to the end of the\ntable-of-contents was recorded as the length of a non-existent chunk\nwith id 0, but we need to be able to somehow find it to use that as\na way to get to the (end of) table-of-contents from the beginning of\nthe file. I guess we'll learn enough when we get to read the code\nthat calls this function.\n\n> +{\n> +\tuint32_t chunk_id;\n> +\tconst unsigned char *table_of_contents = mfile + toc_offset;\n> +\n> +\tALLOC_GROW(cf->chunks, toc_length, cf->chunks_alloc);\n> +\n> +\twhile (toc_length--) {\n> +\t\tuint64_t chunk_offset, next_chunk_offset;\n> +\n> +\t\tchunk_id = get_be32(table_of_contents);\n> +\t\tchunk_offset = get_be64(table_of_contents + 4);\n> +\n> +\t\tif (!chunk_id) {\n> +\t\t\terror(_(\"terminating chunk id appears earlier than expected\"));\n> +\t\t\treturn 1;\n> +\t\t}\n> +\n> +\t\ttable_of_contents += CHUNK_LOOKUP_WIDTH;\n> +\t\tnext_chunk_offset = get_be64(table_of_contents + 4);\n> +\n> +\t\tif (next_chunk_offset < chunk_offset ||\n> +\t\t next_chunk_offset > mfile_size - the_hash_algo->rawsz) {\n\nThe chunks have to be recorded in toc in the order they appear, and\nthere must be enough room to store the hashfile trailer checksum\nafter the last chunk. OK.\n\n> +\t\t\terror(_(\"improper chunk offset(s) %\"PRIx64\" and %\"PRIx64\"\"),\n> +\t\t\t chunk_offset, next_chunk_offset);\n> +\t\t\treturn -1;\n> +\t\t}\n> +\n> +\t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n> +\t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n> +\t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n> +\t\tcf->chunks_nr++;\n> +\t}\n> +\n> +\tchunk_id = get_be32(table_of_contents);\n> +\tif (chunk_id) {\n> +\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n> +\t\treturn -1;\n> +\t}\n\nShouldn't we be validating the size component associated with this\n\"id=0\" fake chunk that appears at the end as well? IIRC, the size\nfield should be pointing at the byte immediately after the TOC entry\nfor the last true chunk, immediately before this zero chunk id?\n\n> +int pair_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t const unsigned char **p)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\tif (cf->chunks[i].id == chunk_id) {\n> +\t\t\t*p = cf->chunks[i].start;\n> +\t\t\treturn 0;\n> +\t\t}\n\nOK, the assumption here is that there will be at most one chunk that\nhas the chunk_id we seek to find (or putting it differently, second\nand subsequent chunks with the same ID are ignored). We may want to\nwrite it down somewhere.\n\n> +\t}\n> +\n> +\treturn CHUNK_NOT_FOUND;\n> +}\n> +\n> +int read_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t chunk_read_fn fn,\n> +\t void *data)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\tif (cf->chunks[i].id == chunk_id)\n> +\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n\nIt is curious why pair_chunk() exists in the first place. With\nsomething like this:\n\n int pair_chunk_fn(const unsigned char *chunk_start,\n size_t chunk_size,\n void *data)\n {\n const unsigned char **p = data;\n *p = chunk_start;\n return 0;\n }\n\ninstead of\n\n\tconst unsigned char *location;\n\n\tpair_chunk(cf, chunk_id, &location);\n\nwe can write\n\n\tconst unsigned char *location;\n\n\tread_chunk(cf, chunk_id, pair_chunk_fn, &location);\n\nno? That would allow us to reorganize the in-core TOC more easily\nif it turns out to be necessary in the future.\n\n> diff --git a/chunk-format.h b/chunk-format.h\n> index bfaed672813..b62c9bf8ba1 100644\n> --- a/chunk-format.h\n> +++ b/chunk-format.h\n> @@ -17,4 +17,37 @@ void add_chunk(struct chunkfile *cf,\n> \t size_t size);\n> int write_chunkfile(struct chunkfile *cf, void *data);\n> \n> +int read_table_of_contents(struct chunkfile *cf,\n> +\t\t\t const unsigned char *mfile,\n> +\t\t\t size_t mfile_size,\n> +\t\t\t uint64_t toc_offset,\n> +\t\t\t int toc_length);\n> +\n> +#define CHUNK_NOT_FOUND (-2)\n> +\n> +/*\n> + * Find 'chunk_id' in the given chunkfile and assign the\n> + * given pointer to the position in the mmap'd file where\n> + * that chunk begins.\n> + *\n> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n> + */\n> +int pair_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t const unsigned char **p);\n> +\n> +typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n> +\t\t\t size_t chunk_size, void *data);\n\nIs the assumption throughout the chunked file API that reading is\nactually not done as a \"seek+read+process\", but as a normal memory\naccess over mmapped region? The reason I ask is because the answer\naffects the choice of the right type for the offset. The function\nsignature of read_table_of_contents() uses size_t to represent the\nlength of the entire mmapped region that holds the data from the\nfile, and that is better than off_t, especially if size_t were\nsmaller than off_t (i.e. we may not be able to mmap a huge size that\nfilesystem can handle and let us access with seek+read).\n\nBut the assumption that the whole mfile can be mmapped in as a whole\nis only in read_table_of_contents(), and users of read_chunk() API\ncan be oblivious, I think---IOW, we could \"page in\" the chunk's data\nin read_chunk() while the callback function works on it in-core, and\nthen discard it, if we wanted to change the implementation [*].\n\n\tSide note: for that to work, the API must say that the\n\tcallback function MUST NOT assume that the memory region\n\tstarting at chunk_start it is given will stay in memory\n\tafter it returns. Otherwise, we cannot \"page in\" and \"page\n\tout\".\n\nI am not advocating that we should not assume the entire file can be\nmapped in. I would however advocate to be explicit in documenting\nwhat the users of API can and cannot do (e.g. if we want the \"read\"\ncallbacks to take advantage of the fact that mfile will stay mapped\nuntil the chunkfile is discarded, we should say so, so that they will\nnot make unnecessary copies out of the mmapped region).\n\n> +/*\n> + * Find 'chunk_id' in the given chunkfile and call the\n> + * given chunk_read_fn method with the information for\n> + * that chunk.\n> + *\n> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n> + */\n> +int read_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t chunk_read_fn fn,\n> +\t void *data);\n\nDid I miss an update to free_chunkfile() to release resources used\nto read this file? For some reason, unlike the writing side, the\nreading side of this API feels a bit incomplete to me.\n\nThanks.\n"},{"id":"416224","messageId":"xmqq8s834c4s.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"83d292532a0fa3f3a0ad343421be4a99a03471d0.1611759716.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T00:00:19Z","receivedAt":"2021-02-05T00:01:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When calculating the sizes of certain chunks, we should use 64-bit\n> multiplication always. This allows us to properly predict the chunk\n> sizes without risk of overflow.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> midx.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n\nThis one I find somewhat questionable for multiple reasons.\n\n * the fourth parameter of add_chunk() is of size_t, not uint64_t;\n shouldn't the multiplication be done in type size_t instead?\n\n * these mutiplications were introduced in \"midx: use chunk-format\n API in write_midx_internal()\"; that step should use the\n arithmetic with cast (if necessary) from the start, no?\n\n * There is \"ctx.entries_nr * MIDX_CHUNKID_OFFSET_WIDTH\" passed to\n add_chunk(), in the post-context of the first hunk. Shouldn't\n that be covered as well? I didn't grep for all uses of\n add_chunk(), but I wouldn't be surprised if this patch missed\n some of the calls that need the same treatment.\n\n> diff --git a/midx.c b/midx.c\n> index e94dcd34b7f..a365dac6bbc 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -913,7 +913,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n> \t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n> \tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n> -\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n> +\t\t write_midx_oid_lookup, (uint64_t)ctx.entries_nr * the_hash_algo->rawsz);\n> \tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n> \t\t write_midx_object_offsets,\n> \t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n> @@ -921,7 +921,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \tif (ctx.large_offsets_needed)\n> \t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n> \t\t\twrite_midx_large_offsets,\n> -\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n> +\t\t\t(uint64_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n> \n> \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n> \twrite_chunkfile(cf, &ctx);\n"},{"id":"416225","messageId":"xmqq4kir4bw7.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"669eeec707ab92a3e5983ad12baddc2c15012d43.1611759716.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 16/17] chunk-format: restore duplicate chunk checks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T00:05:28Z","receivedAt":"2021-02-05T00:06:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Before refactoring into the chunk-format API, the commit-graph parsing\n> logic included checks for duplicate chunks. It is unlikely that we would\n> desire a chunk-based file format that allows duplicate chunk IDs in the\n> table of contents, so add duplicate checks into\n> read_table_of_contents().\n\nMakes sense. This answers a question I had while reading one of the\nprevious steps about the design, I think.\n\nHowever...\n\n> diff --git a/chunk-format.c b/chunk-format.c\n> index 74501084cf8..1ee875df423 100644\n> --- a/chunk-format.c\n> +++ b/chunk-format.c\n> @@ -14,6 +14,7 @@ struct chunk_info {\n> \tchunk_write_fn write_fn;\n> \n> \tconst void *start;\n> +\tunsigned found:1;\n\nThis defines a .found member ...\n\n> @@ -98,6 +99,7 @@ int read_table_of_contents(struct chunkfile *cf,\n> \t\t\t uint64_t toc_offset,\n> \t\t\t int toc_length)\n> {\n> +\tint i;\n> \tuint32_t chunk_id;\n> \tconst unsigned char *table_of_contents = mfile + toc_offset;\n> \n> @@ -124,6 +126,14 @@ int read_table_of_contents(struct chunkfile *cf,\n> \t\t\treturn -1;\n> \t\t}\n> \n> +\t\tfor (i = 0; i < cf->chunks_nr; i++) {\n> +\t\t\tif (cf->chunks[i].id == chunk_id) {\n> +\t\t\t\terror(_(\"duplicate chunk ID %\"PRIx32\" found\"),\n> +\t\t\t\t\tchunk_id);\n> +\t\t\t\treturn -1;\n> +\t\t\t}\n> +\t\t}\n> +\n> \t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n> \t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n> \t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n\n... and no new code touches it.\n\nThe way duplicate is found is by having a inner loop that checks the\nIDs of chunks we've seen so far (quadratic, but presumably that\nwould not matter as long as we'd be dealing with just half a dozen\nchunk types).\n\nIs the .found bit used for something else and needs to be added in a\ndifferent step?\n"},{"id":"416226","messageId":"xmqqy2g32wvj.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"8f3985ab5df3e4abc6de6db7f71f1adcbc16b4a8.1611759716.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 17/17] chunk-format: add technical docs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T00:15:12Z","receivedAt":"2021-02-05T00:15:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +Chunk-based file formats\n> +========================\n> +\n> +Some file formats in Git use a common concept of \"chunks\" to describe\n> +sections of the file. This allows structured access to a large file by\n> +scanning a small \"table of contents\" for the remaining data. This common\n> +format is used by the `commit-graph` and `multi-pack-index` files. See\n> +link:technical/pack-format.html[the `multi-pack-index` format] and\n> +link:technical/commit-graph-format.html[the `commit-graph` format] for\n> +how they use the chunks to describe structured data.\n> +\n> +A chunk-based file format begins with some header information custom to\n> +that format. That header should include enough information to identify\n> +the file type, format version, and number of chunks in the file. From this\n> +information, that file can determine the start of the chunk-based region.\n> +\n> +The chunk-based region starts with a table of contents describing where\n> +each chunk starts and ends. This consists of (C+1) rows of 12 bytes each,\n> +where C is the number of chunks. Consider the following table:\n> +\n> + | Chunk ID (4 bytes) | Chunk Offset (8 bytes) |\n> + |--------------------|------------------------|\n> + | ID[0] | OFFSET[0] |\n> + | ... | ... |\n> + | ID[C] | OFFSET[C] |\n> + | 0x0000 | OFFSET[C+1] |\n> +\n> +Each row consists of a 4-byte chunk identifier (ID) and an 8-byte offset.\n> +Each integer is stored in network-byte order.\n> +\n> +The chunk identifier `ID[i]` is a label for the data stored within this\n> +fill from `OFFSET[i]` (inclusive) to `OFFSET[i+1]` (exclusive). Thus, the\n> +size of the `i`th chunk is equal to the difference between `OFFSET[i+1]`\n> +and `OFFSET[i]`. This requires that the chunk data appears contiguously\n> +in the same order as the table of contents.\n> +\n> +The final entry in the table of contents must be four zero bytes. This\n> +confirms that the table of contents is ending and provides the offset for\n> +the end of the chunk-based data.\n> +\n> +Note: The chunk-based format expects that the file contains _at least_ a\n> +trailing hash after `OFFSET[C+1]`.\n\nI think the above describes what I saw in the writing side of the\ncode quite clearly and very well. I misread that the OFFSET[C+1]\nwas pointing elsewhere in my review of [2/17] somehow, but the code\nis clear that it points at the end of the last chunk from the code,\nand the above documents it well.\n\nMy comments on the need to document the reading side API, on what\nthe read_chunk callback should be able to assume (namely, the whole\nthing stays in memory until the caller that decided to use chunkfile\nAPI decides to discard it), still stands, I would think.\n\nThanks.\n\n"},{"id":"416232","messageId":"xmqqa6sj2rm1.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/17] Refactor chunk-format into an API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T02:08:54Z","receivedAt":"2021-02-05T02:09:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This is a restart on the topic previously submitted [1] but dropped because\n> ak/corrected-commit-date was still in progress. This version is based on\n> that branch.\n\nI've read the topic through, and found it a pleasant read. There\nare some questionable use of integer types, some uneven application\nof casts, and the reading side API is somewhat underdocumented, but\nthe overall direction looked quite sane.\n\nI am undecided if I should expect a reroll, or declare what we have\n\"already good enough\" for 'next' and expect incremental refinements,\nthough. A reroll that catches all nits would certainly make the\nresulting topic's history nicer for future developers to work with,\nbut its also a pain to re-read essentially the same patches again,\nso...\n\n\n"},{"id":"416233","messageId":"6fec49f1-3ea6-f365-ed54-b1e7d44bcefe@gmail.com","threadId":"55053","inReplyTo":"xmqqa6sj2rm1.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/17] Refactor chunk-format into an API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-05T02:27:04Z","receivedAt":"2021-02-05T02:28:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/4/2021 9:08 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> This is a restart on the topic previously submitted [1] but dropped because\n>> ak/corrected-commit-date was still in progress. This version is based on\n>> that branch.\n> \n> I've read the topic through, and found it a pleasant read. There\n> are some questionable use of integer types, some uneven application\n> of casts, and the reading side API is somewhat underdocumented, but\n> the overall direction looked quite sane.\n> \n> I am undecided if I should expect a reroll, or declare what we have\n> \"already good enough\" for 'next' and expect incremental refinements,\n> though. A reroll that catches all nits would certainly make the\n> resulting topic's history nicer for future developers to work with,\n> but its also a pain to re-read essentially the same patches again,\n> so...\n\nI appreciate the willingness to take the topic as-is, but I think\nyou've given me enough interesting comments to deserve a re-roll.\nI will provide one tomorrow and hope that the range-diff is a\npleasant read ;).\n\nThanks,\n-Stolee\n"},{"id":"416252","messageId":"CAPx1GvcNTkXo-6rdaRc4YNGAq8AoCtwT86AEUT+A6c22erVW0g@mail.gmail.com","threadId":"55053","inReplyTo":"xmqq8s834c4s.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2021-02-05T10:59:06Z","receivedAt":"2021-02-05T11:06:26Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Thu, Feb 4, 2021 at 4:00 PM Junio C Hamano wrote:\n> * the fourth parameter of add_chunk() is of size_t, not uint64_t;\n> shouldn't the multiplication be done in type size_t instead?\n\nThere are (still) systems with 32-bit size_t (but 64-bit\noff_t / file sizes), so ... probably not. Is size_t ever more than\n64 bits these days?\n\nChris\n"},{"id":"416254","messageId":"d5097b51-e096-16b3-5b7b-2c7dd1cbe918@gmail.com","threadId":"55053","inReplyTo":"xmqq5z375xxf.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-05T11:37:48Z","receivedAt":"2021-02-05T11:41:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/4/2021 4:24 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +/*\n>> + * When writing a chunk-based file format, collect the chunks in\n>> + * an array of chunk_info structs. The size stores the _expected_\n>> + * amount of data that will be written by write_fn.\n>> + */\n>> +struct chunk_info {\n>> +\tuint32_t id;\n>> +\tuint64_t size;\n>> +\tchunk_write_fn write_fn;\n>> +};\n>> +...\n>> +void add_chunk(struct chunkfile *cf,\n>> +\t uint64_t id,\n>> +\t chunk_write_fn fn,\n>> +\t size_t size)\n>> +{\n>> +\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n>> +\n>> +\tcf->chunks[cf->chunks_nr].id = id;\n>> +\tcf->chunks[cf->chunks_nr].write_fn = fn;\n>> +\tcf->chunks[cf->chunks_nr].size = size;\n>> +\tcf->chunks_nr++;\n>> +}\n> \n> Somebody somewhere between the caller in the higher part of the\n> callchain (that has to work with platform native types) and the\n> on-disk format at the bottom of the callchain (that has to work\n> with fixed size data fields) needs to make sure that the size that\n> the higher level caller has fits on-disk data structure we define,\n> and the data we read from disk fits the in-core structure our caller\n> use on the reading side.\n> \n> If there is a function at the one level closer to the disk than\n> \"struct chunk_info\" that takes a \"struct chunk_info\" and writes the\n> id and size to disk (and fills \"struct chunk_info\" from what is read\n> from the disk, on the reading side), it would be a good place to do\n> the size_t to uint64_t conversion.\n\nI'm fine with keeping the external interface focused on size_t\ninstead of uint64_t.\n\n> It is OK to do the conversion-while-checking in add_chunk(), too.\n> \n> But a silent type casting from size_t to uint64_t done silently by\n> assignment bothers me. \n\nDoes this bother you only because its part of the external interface?\nIf I understand correctly, uint64_t will always be at least as big\nas size_t, so this doesn't need any protections for overflow or\nanything. Is there something I should be doing before casting?\n\n> Also, I think you meant to make the incoming\n> ID uint32_t; am I missing something, or did nobody notice it in the\n> review of the previous round?\n\nYes, this should be 32-bits. Will fix.\n \n>> +int write_chunkfile(struct chunkfile *cf, void *data)\n>> +{\n>> +\tint i;\n>> +\tsize_t cur_offset = cf->f->offset + cf->f->total;\n> \n> That ought to be off_t, as it is a seek position inside a file\n> (struct hashfile.total is already off_t).\n\nI can use off_t for the other offsets in this computation, but\ncur_offset will be used in hashwrite_be64(), so maybe it is best\nto use uint64_t here?\n\n> Use csum-file.h::hashfile_total() instead, perhaps? .offset member\n> is an implementation detail of the hashfile API (i.e. some leftover\n> bits are kept in in-core buffer, until we accumulate enough to make\n> it worth flushing to the disk), and by using the helper, this code\n> does not have to know about it.\n\nThanks! This is cleaner.\n\n>> +\t/* Add the table of contents to the current offset */\n>> +\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n> \n> Is that 12 == sizeof(chunk_info.id) + sizeof(chunk_info.size)?\n> If so, this makes sense.\n\nYes.\n\n>> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n>> +\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n>> +\t\thashwrite_be64(cf->f, cur_offset);\n>> +\n>> +\t\tcur_offset += cf->chunks[i].size;\n>> +\t}\n>> +\n>> +\t/* Trailing entry marks the end of the chunks */\n>> +\thashwrite_be32(cf->f, 0);\n>> +\thashwrite_be64(cf->f, cur_offset);\n> \n> OK. This helper does not tell us anything about what comes in the\n> on-disk file before this point, but we write a table of contents\n> that says \"chunk with this ID has this size, chunk with that ID has\n> that size, ...\", concluded by something that looks like another\n> entry with chunk ID 0 that records the current offset as its size.\n\nRight. The table of contents gives us enough information to find\nthe start _and_ end of each chunk (and hence compute their size).\n\n>> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n>> +\t\tuint64_t start_offset = cf->f->total + cf->f->offset;\n> \n> Likewise about the type and use of hashfile_total().\n\nThis one can definitely be off_t.\n\n>> +\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n>> +\n>> +\t\tif (result)\n>> +\t\t\treturn result;\n>> +\n>> +\t\tif (cf->f->total + cf->f->offset - start_offset != cf->chunks[i].size)\n>> +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n>> +\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n>> +\t\t\t cf->f->total + cf->f->offset - start_offset);\n> \n> I won't complain, as this apparently is sufficient to abstract out\n> the two existing chunked format files, but it somehow feels a bit\n> limiting that the one that calls add_chunk() is required to know\n> what the size of generated data would be, way before .write_fn() is\n> called to produce the actual data here.\n\nThis was pointed out earlier, but it _is_ part of the existing users\nof the format. The table of contents is written at the start of the\nfile instead of the end (such as in the .zip format).\n\nThe current chunk format API makes the same assumption (ToC comes\nfirst) but could be adjusted later to let this part of the method\ndynamically compute the chunk sizes and fill a ToC at the end. The\nway to modify this API would be to add a 'flags' parameter.\n\nSo far, this has not been necessary, but might be in the future.\n\n>> +\t}\n>> +\n>> +\treturn 0;\n>> +}\n>> diff --git a/chunk-format.h b/chunk-format.h\n>> new file mode 100644\n>> index 00000000000..bfaed672813\n>> --- /dev/null\n>> +++ b/chunk-format.h\n>> @@ -0,0 +1,20 @@\n>> +#ifndef CHUNK_FORMAT_H\n>> +#define CHUNK_FORMAT_H\n>> +\n>> +#include \"git-compat-util.h\"\n>> +\n>> +struct hashfile;\n>> +struct chunkfile;\n>> +\n>> +struct chunkfile *init_chunkfile(struct hashfile *f);\n>> +void free_chunkfile(struct chunkfile *cf);\n>> +int get_num_chunks(struct chunkfile *cf);\n>> +typedef int (*chunk_write_fn)(struct hashfile *f,\n>> +\t\t\t void *data);\n> \n> Write this on a single line.\n\nWill do.\n\nThanks,\n-Stolee\n"},{"id":"416255","messageId":"1278de82-417c-a6ee-a5fe-055fa0ef1903@gmail.com","threadId":"55053","inReplyTo":"xmqqczxf4d2k.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-05T12:19:52Z","receivedAt":"2021-02-05T12:23:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/4/2021 6:40 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> We are re-using the anonymous 'struct chunkfile' data, as it is internal\n>> to the chunk-format API. This gives it essentially two modes: write and\n>> read. If the same struct instance was used for both reads and writes,\n>> then there would be failures.\n> \n> Writing it here won't help future developers very much. I think\n> that belongs to API/function doc for init_chunkfile().\n\nI will add a clear comment there. Thanks.\n\n>> +int read_table_of_contents(struct chunkfile *cf,\n>> +\t\t\t const unsigned char *mfile,\n>> +\t\t\t size_t mfile_size,\n>> +\t\t\t uint64_t toc_offset,\n>> +\t\t\t int toc_length)\n> \n> It's a bit of mystery, having seen how the table-of-contents is laid\n> out by reading the writing side of the code, how toc_offset and\n> toc_length are discovered by the caller. IIRC, the size to cover\n> everything from the beginning of the file to the end of the\n> table-of-contents was recorded as the length of a non-existent chunk\n> with id 0, but we need to be able to somehow find it to use that as\n> a way to get to the (end of) table-of-contents from the beginning of\n> the file. I guess we'll learn enough when we get to read the code\n> that calls this function.\n\nThe existing formats have a byte in their header specifying how\nmany chunks are in the table of contents. That's how this information\nis known in advance.\n\nIf we want to instead rely on the terminating chunk id with value 0,\nthen this method could be modified in the future. It does complicate\nthe allocation of cf->chunks slightly.\n\n>> +\tchunk_id = get_be32(table_of_contents);\n>> +\tif (chunk_id) {\n>> +\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n>> +\t\treturn -1;\n>> +\t}\n> \n> Shouldn't we be validating the size component associated with this\n> \"id=0\" fake chunk that appears at the end as well? IIRC, the size\n> field should be pointing at the byte immediately after the TOC entry\n> for the last true chunk, immediately before this zero chunk id?\n\nDuring the loop, we scanned ahead to find the offset of the\nterminating chunk and compute the size of the last \"real\" chunk.\n\nAny size validation here would be to check that the offset points\n_exactly_ to the terminating hash, but that might be too restrictive\non the format. (Who knows if there is a legit reason to have non-\nchunked data between the chunked data and the trailing hash?)\n\nAs I re-roll, I plan to skip this final check. It would be easy to\nadd it in a forward fix, thought.\n\n>> +int pair_chunk(struct chunkfile *cf,\n>> +\t uint32_t chunk_id,\n>> +\t const unsigned char **p)\n>> +{\n>> +\tint i;\n>> +\n>> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n>> +\t\tif (cf->chunks[i].id == chunk_id) {\n>> +\t\t\t*p = cf->chunks[i].start;\n>> +\t\t\treturn 0;\n>> +\t\t}\n> \n> OK, the assumption here is that there will be at most one chunk that\n> has the chunk_id we seek to find (or putting it differently, second\n> and subsequent chunks with the same ID are ignored). We may want to\n> write it down somewhere.\n\nThis is enforced by a later patch, as you discovered.\n\n>> +\tfor (i = 0; i < cf->chunks_nr; i++) {\n>> +\t\tif (cf->chunks[i].id == chunk_id)\n>> +\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n> \n> It is curious why pair_chunk() exists in the first place. With\n> something like this:\n> \n> int pair_chunk_fn(const unsigned char *chunk_start,\n> size_t chunk_size,\n> void *data)\n> {\n> const unsigned char **p = data;\n> *p = chunk_start;\n> return 0;\n> }\n> \n> instead of\n> \n> \tconst unsigned char *location;\n> \n> \tpair_chunk(cf, chunk_id, &location);\n> \n> we can write\n> \n> \tconst unsigned char *location;\n> \n> \tread_chunk(cf, chunk_id, pair_chunk_fn, &location);\n> \n> no? That would allow us to reorganize the in-core TOC more easily\n> if it turns out to be necessary in the future.\n\nI like this, but why not just use pair_chunk_fn inside of\nthe implementation of pair_chunk() so callers have an easy\ninterface. This results in the implementation:\n\nstatic int pair_chunk_fn(const unsigned char *chunk_start,\n\t\t\t size_t chunk_size,\n\t\t\t void *data)\n{\n\tconst unsigned char **p = data;\n\t*p = chunk_start;\n\treturn 0;\n}\n\nint pair_chunk(struct chunkfile *cf,\n\t uint32_t chunk_id,\n\t const unsigned char **p)\n{\n\treturn read_chunk(cf, chunk_id, pair_chunk_fn, p);\n}\n\n>> +typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n>> +\t\t\t size_t chunk_size, void *data);\n> \n> Is the assumption throughout the chunked file API that reading is\n> actually not done as a \"seek+read+process\", but as a normal memory\n> access over mmapped region? The reason I ask is because the answer\n> affects the choice of the right type for the offset. The function\n> signature of read_table_of_contents() uses size_t to represent the\n> length of the entire mmapped region that holds the data from the\n> file, and that is better than off_t, especially if size_t were\n> smaller than off_t (i.e. we may not be able to mmap a huge size that\n> filesystem can handle and let us access with seek+read).\n> \n> But the assumption that the whole mfile can be mmapped in as a whole\n> is only in read_table_of_contents(), and users of read_chunk() API\n> can be oblivious, I think---IOW, we could \"page in\" the chunk's data\n> in read_chunk() while the callback function works on it in-core, and\n> then discard it, if we wanted to change the implementation [*].\n> \n> \tSide note: for that to work, the API must say that the\n> \tcallback function MUST NOT assume that the memory region\n> \tstarting at chunk_start it is given will stay in memory\n> \tafter it returns. Otherwise, we cannot \"page in\" and \"page\n> \tout\".\n> \n> I am not advocating that we should not assume the entire file can be\n> mapped in. I would however advocate to be explicit in documenting\n> what the users of API can and cannot do (e.g. if we want the \"read\"\n> callbacks to take advantage of the fact that mfile will stay mapped\n> until the chunkfile is discarded, we should say so, so that they will\n> not make unnecessary copies out of the mmapped region).\n\nI will add this expectation to the documentation.\n\n>> +/*\n>> + * Find 'chunk_id' in the given chunkfile and call the\n>> + * given chunk_read_fn method with the information for\n>> + * that chunk.\n>> + *\n>> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n>> + */\n>> +int read_chunk(struct chunkfile *cf,\n>> +\t uint32_t chunk_id,\n>> +\t chunk_read_fn fn,\n>> +\t void *data);\n> \n> Did I miss an update to free_chunkfile() to release resources used\n> to read this file? For some reason, unlike the writing side, the\n> reading side of this API feels a bit incomplete to me.\n\nfree_chunkfile(cf) already frees cf->chunks (and cf itself). No\nother resources are allocated during the read. The caller is\nresponsible for the mmap'd file resource.\n\nThe reading side could use more documentation. I will think about\nthis.\n\nThanks,\n-Stolee\n"},{"id":"416256","messageId":"7c209432-82e6-7785-a5c1-1068ec3f0378@gmail.com","threadId":"55053","inReplyTo":"xmqq8s834c4s.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-05T12:30:42Z","receivedAt":"2021-02-05T12:34:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/4/2021 7:00 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> When calculating the sizes of certain chunks, we should use 64-bit\n>> multiplication always. This allows us to properly predict the chunk\n>> sizes without risk of overflow.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> midx.c | 4 ++--\n>> 1 file changed, 2 insertions(+), 2 deletions(-)\n> \n> This one I find somewhat questionable for multiple reasons.\n> \n> * the fourth parameter of add_chunk() is of size_t, not uint64_t;\n> shouldn't the multiplication be done in type size_t instead?\n\nThis is probably appropriate because we will truncate to size_t if\nit is smaller than uint64_t.\n\n> * these mutiplications were introduced in \"midx: use chunk-format\n> API in write_midx_internal()\"; that step should use the\n> arithmetic with cast (if necessary) from the start, no?\n\nI wanted to isolate these changes specifically so we could be\ncareful about the multiplications and not be distracted by them\nwhen converting to the chunk-format API. The multiplications were\n\"moved\" by that patch, not \"introduced\".\n\n> * There is \"ctx.entries_nr * MIDX_CHUNKID_OFFSET_WIDTH\" passed to\n> add_chunk(), in the post-context of the first hunk. Shouldn't\n> that be covered as well? I didn't grep for all uses of\n> add_chunk(), but I wouldn't be surprised if this patch missed\n> some of the calls that need the same treatment.\n\nAnd here is a great example of why it was good to call out these\nmultiplications in their own patch.\n\nI did a full inspection of all multiplications in midx.c and\nfound a few more instances of possible overflow. Two are on the\nread side, but they require the object lookup chunk to have size\n4gb or larger. This is not _that_ far off from possibility! My\nmulti-pack-index for the Windows repository is currently ~1.6 GB\n(in total, including the other chunks).\n\nThanks,\n-Stolee\n"},{"id":"416260","messageId":"70f68c95e479b9d6476e070922514a4a1f05fd3b.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 10/17] midx: drop chunk progress during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:45Z","receivedAt":"2021-02-05T16:26:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMost expensive operations in write_midx_internal() use the context\nstruct's progress member, and these indicate the process of the\nexpensive operations within the chunk writing methods. However, there is\na competing progress struct that counts the progress over all chunks.\nThis is not very helpful compared to the others, so drop it.\n\nThis also reduces our barriers to combining the chunk writing code with\nchunk-format.c.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex c92a6c47be01..4f4aa351e60e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -808,7 +808,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -976,9 +975,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\thashwrite_be64(f, chunk_offsets[i]);\n \t}\n \n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n-\t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n@@ -1011,10 +1007,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\n \t\t}\n-\n-\t\tdisplay_progress(progress, i + 1);\n \t}\n-\tstop_progress(&progress);\n \n \tif (hashfile_total(f) != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-- \ngitgitgadget\n\n"},{"id":"416261","messageId":"4c7d751f1e39df30fe9ba3b3e9e0829ddead0340.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 17/17] chunk-format: add technical docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:52Z","receivedAt":"2021-02-05T16:33:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-based file format is now an API in the code, but we should\nalso take time to document it as a file format. Specifically, it matches\nthe CHUNK LOOKUP sections of the commit-graph and multi-pack-index\nfiles, but there are some commonalities that should be grouped in this\ndocument.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/chunk-format.txt | 116 ++++++++++++++++++\n .../technical/commit-graph-format.txt | 3 +\n Documentation/technical/pack-format.txt | 3 +\n 3 files changed, 122 insertions(+)\n create mode 100644 Documentation/technical/chunk-format.txt\n\ndiff --git a/Documentation/technical/chunk-format.txt b/Documentation/technical/chunk-format.txt\nnew file mode 100644\nindex 000000000000..593614fcedab\n--- /dev/null\n+++ b/Documentation/technical/chunk-format.txt\n@@ -0,0 +1,116 @@\n+Chunk-based file formats\n+========================\n+\n+Some file formats in Git use a common concept of \"chunks\" to describe\n+sections of the file. This allows structured access to a large file by\n+scanning a small \"table of contents\" for the remaining data. This common\n+format is used by the `commit-graph` and `multi-pack-index` files. See\n+link:technical/pack-format.html[the `multi-pack-index` format] and\n+link:technical/commit-graph-format.html[the `commit-graph` format] for\n+how they use the chunks to describe structured data.\n+\n+A chunk-based file format begins with some header information custom to\n+that format. That header should include enough information to identify\n+the file type, format version, and number of chunks in the file. From this\n+information, that file can determine the start of the chunk-based region.\n+\n+The chunk-based region starts with a table of contents describing where\n+each chunk starts and ends. This consists of (C+1) rows of 12 bytes each,\n+where C is the number of chunks. Consider the following table:\n+\n+ | Chunk ID (4 bytes) | Chunk Offset (8 bytes) |\n+ |--------------------|------------------------|\n+ | ID[0] | OFFSET[0] |\n+ | ... | ... |\n+ | ID[C] | OFFSET[C] |\n+ | 0x0000 | OFFSET[C+1] |\n+\n+Each row consists of a 4-byte chunk identifier (ID) and an 8-byte offset.\n+Each integer is stored in network-byte order.\n+\n+The chunk identifier `ID[i]` is a label for the data stored within this\n+fill from `OFFSET[i]` (inclusive) to `OFFSET[i+1]` (exclusive). Thus, the\n+size of the `i`th chunk is equal to the difference between `OFFSET[i+1]`\n+and `OFFSET[i]`. This requires that the chunk data appears contiguously\n+in the same order as the table of contents.\n+\n+The final entry in the table of contents must be four zero bytes. This\n+confirms that the table of contents is ending and provides the offset for\n+the end of the chunk-based data.\n+\n+Note: The chunk-based format expects that the file contains _at least_ a\n+trailing hash after `OFFSET[C+1]`.\n+\n+Functions for working with chunk-based file formats are declared in\n+`chunk-format.h`. Using these methods provide extra checks that assist\n+developers when creating new file formats.\n+\n+Writing chunk-based file formats\n+--------------------------------\n+\n+To write a chunk-based file format, create a `struct chunkfile` by\n+calling `init_chunkfile()` and pass a `struct hashfile` pointer. The\n+caller is responsible for opening the `hashfile` and writing header\n+information so the file format is identifiable before the chunk-based\n+format begins.\n+\n+Then, call `add_chunk()` for each chunk that is intended for write. This\n+populates the `chunkfile` with information about the order and size of\n+each chunk to write. Provide a `chunk_write_fn` function pointer to\n+perform the write of the chunk data upon request.\n+\n+Call `write_chunkfile()` to write the table of contents to the `hashfile`\n+followed by each of the chunks. This will verify that each chunk wrote\n+the expected amount of data so the table of contents is correct.\n+\n+Finally, call `free_chunkfile()` to clear the `struct chunkfile` data. The\n+caller is responsible for finalizing the `hashfile` by writing the trailing\n+hash and closing the file.\n+\n+Reading chunk-based file formats\n+--------------------------------\n+\n+To read a chunk-based file format, the file must be opened as a\n+memory-mapped region. The chunk-format API expects that the entire file\n+is mapped as a contiguous memory region.\n+\n+Initialize a `struct chunkfile` pointer with `init_chunkfile(NULL)`.\n+\n+After reading the header information from the beginning of the file,\n+including the chunk count, call `read_table_of_contents()` to populate\n+the `struct chunkfile` with the list of chunks, their offsets, and their\n+sizes.\n+\n+Extract the data information for each chunk using `pair_chunk()` or\n+`read_chunk()`:\n+\n+* `pair_chunk()` assigns a given pointer with the location inside the\n+ memory-mapped file corresponding to that chunk's offset. If the chunk\n+ does not exist, then the pointer is not modified.\n+\n+* `read_chunk()` takes a `chunk_read_fn` function pointer and calls it\n+ with the appropriate initial pointer and size information. The function\n+ is not called if the chunk does not exist. Use this method to read chunks\n+ if you need to perform immediate parsing or if you need to execute logic\n+ based on the size of the chunk.\n+\n+After calling these methods, call `free_chunkfile()` to clear the\n+`struct chunkfile` data. This will not close the memory-mapped region.\n+Callers are expected to own that data for the timeframe the pointers into\n+the region are needed.\n+\n+Examples\n+--------\n+\n+These file formats use the chunk-format API, and can be used as examples\n+for future formats:\n+\n+* *commit-graph:* see `write_commit_graph_file()` and `parse_commit_graph()`\n+ in `commit-graph.c` for how the chunk-format API is used to write and\n+ parse the commit-graph file format documented in\n+ link:technical/commit-graph-format.html[the commit-graph file format].\n+\n+* *multi-pack-index:* see `write_midx_internal()` and `load_multi_pack_index()`\n+ in `midx.c` for how the chunk-format API is used to write and\n+ parse the multi-pack-index file format documented in\n+ link:technical/pack-format.html[the multi-pack-index file format].\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex b6658eff1882..87971c27dd73 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -61,6 +61,9 @@ CHUNK LOOKUP:\n the length using the next chunk position if necessary.) Each chunk\n ID appears at most once.\n \n+ The CHUNK LOOKUP matches the table of contents from\n+ link:technical/chunk-format.html[the chunk-based file format].\n+\n The remaining data in the body is described one chunk at a time, and\n these chunks may be given in any order. Chunks are required unless\n otherwise specified.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex f96b2e605f34..2fb1e60d29ec 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -301,6 +301,9 @@ CHUNK LOOKUP:\n \t (Chunks are provided in file-order, so you can infer the length\n \t using the next chunk position if necessary.)\n \n+\tThe CHUNK LOOKUP matches the table of contents from\n+\tlink:technical/chunk-format.html[the chunk-based file format].\n+\n \tThe remaining data in the body is described one chunk at a time, and\n \tthese chunks may be given in any order. Chunks are required unless\n \totherwise specified.\n-- \ngitgitgadget\n"},{"id":"416263","messageId":"7aa3242e15b7a0cca7b7d4394311697ed84bc05a.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 09/17] midx: return success/failure in chunk write methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:44Z","receivedAt":"2021-02-05T16:41:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHistorically, the chunk-writing methods in midx.c have returned the\namount of data written so the writer method could compare this with the\ntable of contents. This presents with some interesting issues:\n\n1. If a chunk writing method has a bug that miscalculates the written\n bytes, then we can satisfy the table of contents without actually\n writing the right amount of data to the hashfile. The commit-graph\n writing code checks the hashfile struct directly for a more robust\n verification.\n\n2. There is no way for a chunk writing method to gracefully fail.\n Returning an int presents an opportunity to fail without a die().\n\n3. The current pattern doesn't match chunk_write_fn type exactly, so we\n cannot share code with commit-graph.c\n\nFor these reasons, convert the midx chunk writer methods to return an\n'int'. Since none of them fail at the moment, they all return 0.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 63 +++++++++++++++++++++++++---------------------------------\n 1 file changed, 27 insertions(+), 36 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 5be081f229ad..c92a6c47be01 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -650,7 +650,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f, void *data)\n+static int write_midx_pack_names(struct hashfile *f, void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n@@ -678,14 +678,13 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n \tif (i < MIDX_CHUNK_ALIGNMENT) {\n \t\tmemset(padding, 0, sizeof(padding));\n \t\thashwrite(f, padding, i);\n-\t\twritten += i;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n@@ -710,17 +709,16 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \t\tlist = next;\n \t}\n \n-\treturn MIDX_CHUNK_FANOUT_SIZE;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tunsigned char hash_len = the_hash_algo->rawsz;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -734,19 +732,17 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \t\t}\n \n \t\thashwrite(f, obj->oid.hash, (int)hash_len);\n-\t\twritten += hash_len;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f,\n-\t\t\t\t\tvoid *data)\n+static int write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -766,20 +762,17 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \t\t\t obj->offset);\n \t\telse\n \t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n-\n-\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n-\tsize_t written = 0;\n \tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n@@ -795,12 +788,12 @@ static size_t write_midx_large_offsets(struct hashfile *f,\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\twritten += hashwrite_be64(f, offset);\n+\t\thashwrite_be64(f, offset);\n \n \t\tnr_large_offset--;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n@@ -812,7 +805,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t written = 0;\n+\tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tstruct progress *progress = NULL;\n@@ -940,10 +933,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n+\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n@@ -981,39 +974,37 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\thashwrite_be32(f, chunk_ids[i]);\n \t\thashwrite_be64(f, chunk_offsets[i]);\n-\n-\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (written != chunk_offsets[i])\n+\t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n \t\t\t chunk_offsets[i],\n-\t\t\t written,\n+\t\t\t f->total + f->offset,\n \t\t\t chunk_ids[i]);\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n+\t\t\t\twrite_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n+\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n+\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n+\t\t\t\twrite_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n+\t\t\t\twrite_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1025,9 +1016,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \tstop_progress(&progress);\n \n-\tif (written != chunk_offsets[num_chunks])\n+\tif (hashfile_total(f) != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t written,\n+\t\t hashfile_total(f),\n \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n-- \ngitgitgadget\n\n"},{"id":"416277","messageId":"xmqqim761flz.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"d5097b51-e096-16b3-5b7b-2c7dd1cbe918@gmail.com","subject":"Re: [PATCH v2 02/17] chunk-format: create chunk format write API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T19:25:44Z","receivedAt":"2021-02-05T19:29:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> But a silent type casting from size_t to uint64_t done silently by\n>> assignment bothers me. \n>\n> Does this bother you only because its part of the external interface?\n> If I understand correctly, uint64_t will always be at least as big\n> as size_t, so this doesn't need any protections for overflow or\n> anything. Is there something I should be doing before casting?\n\nI am OK to use uint64_t on the caller-facing side, as long as we\nexplicitly declare that uint64_t ought to be large enough for\neverybody. Struct members and variables that are closer to the\non-disk data need to be of sized type to avoid repeating the pain\ncaused by our early \"unsigned long ought to be large enough for\neverybody\" attitude, but it is nicer to be working with more\nabstract types in the layer higher up. And if there is a risk of\ntruncation in either direction, we should be defensive. That's all.\n\n>>> +int write_chunkfile(struct chunkfile *cf, void *data)\n>>> +{\n>>> +\tint i;\n>>> +\tsize_t cur_offset = cf->f->offset + cf->f->total;\n>> \n>> That ought to be off_t, as it is a seek position inside a file\n>> (struct hashfile.total is already off_t).\n>\n> I can use off_t for the other offsets in this computation, but\n> cur_offset will be used in hashwrite_be64(), so maybe it is best\n> to use uint64_t here?\n\nAs I discovered in the later parts, I think off_t makes less sense\nthan size_t in the context of this design, so size_t is fine, as\nlong as we keep the \"users of chunkfile API work on a mmapped region\nof contiguous memory\" (which I think is OK). uint64_t is also fine,\nas long as this one is an internal implementation detail; iow,\ncallers of the API do not have to be responsible for casting their\nmore abstract and platform neutral types down to these fixed-sized\ntypes even if we choose to use uint64_t here.\n\n> The current chunk format API makes the same assumption (ToC comes\n> first) but could be adjusted later to let this part of the method\n> dynamically compute the chunk sizes and fill a ToC at the end. The\n> way to modify this API would be to add a 'flags' parameter.\n>\n> So far, this has not been necessary, but might be in the future.\n\nYup, and I am happy with the current design for now. Thanks for\nclarifying the thinking behind it.\n"},{"id":"416279","messageId":"xmqqeehu1f1q.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"1278de82-417c-a6ee-a5fe-055fa0ef1903@gmail.com","subject":"Re: [PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T19:37:53Z","receivedAt":"2021-02-05T19:43:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>> +\tchunk_id = get_be32(table_of_contents);\n>>> +\tif (chunk_id) {\n>>> +\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n>>> +\t\treturn -1;\n>>> +\t}\n>> \n>> Shouldn't we be validating the size component associated with this\n>> \"id=0\" fake chunk that appears at the end as well?\n\nNo, please disregard this comment, which was based on my incorrect\nunderstanding of the \"size\" field associated with this fake ID==0\nchunk (I incorrectly thought the size had something to do with the\nfile header plus TOC, but it is not---it is to allow discovering the\nsize of the last chunk by being a sentinel that records the offset\nof an extra chunk at the end that does not actually exist).\n\n> I like this, but why not just use pair_chunk_fn inside of\n> the implementation of pair_chunk() so callers have an easy\n> interface.\n\nYes, I didn't realize that earlier design iteration resulted in the\nintroduction of the \"pair_chunk()\" after discovering that it often\nis necessary to just note the address where the data begins, so I\nam OK to leave something like pair_chunk() as a public interface,\nand implementing the pair_chunk() helper like you suggest would be a\nperfectly fine way to do so.\n\nIt however is curious that the callers who use pair_chunk() do not\nget the same quality of data as read_chunk() callers.\n\nThe users of pair_chunk() presumably are not ready to (or simply do\nnot want to) process the data immediately by using read_chunk() with\ncallback, but when they get ready to process the data, unlike\nread_chunk callbacks, they do not get to learn how much they ought\nto process---all they learn is the address of the beginning of the\nchunk. I do not see a way to write pair_chunk() users safely to\nguarantee that they do not overrun at the tail end of the chunk they\nare processing.\n\nThanks.\n"},{"id":"416280","messageId":"xmqqa6si1ety.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"7c209432-82e6-7785-a5c1-1068ec3f0378@gmail.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T19:42:33Z","receivedAt":"2021-02-05T19:45:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 2/4/2021 7:00 PM, Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" writes:\n>> \n>>> From: Derrick Stolee \n>>>\n>>> When calculating the sizes of certain chunks, we should use 64-bit\n>>> multiplication always. This allows us to properly predict the chunk\n>>> sizes without risk of overflow.\n>>>\n>>> Signed-off-by: Derrick Stolee \n>>> ---\n>>> midx.c | 4 ++--\n>>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>> \n>> This one I find somewhat questionable for multiple reasons.\n>> \n>> * the fourth parameter of add_chunk() is of size_t, not uint64_t;\n>> shouldn't the multiplication be done in type size_t instead?\n>\n> This is probably appropriate because we will truncate to size_t if\n> it is smaller than uint64_t.\n\nIn other words, if size_t turns out to be too small, doing\nmultiplication in uint64_t would not help at all and add_chunk() API\nneeds its parameter types updated [*].\n\n side note: I really wish that the language and the compiler\n helped us so that we didn't have to do this---after all, our\n function prototype says the result will be passed as a certain\n type, so it would be nice if the arithmetic to compute that\n result were automatically carried out in a way not to cause\n truncation.\n\n>> * these mutiplications were introduced in \"midx: use chunk-format\n>> API in write_midx_internal()\"; that step should use the\n>> arithmetic with cast (if necessary) from the start, no?\n>\n> I wanted to isolate these changes specifically so we could be\n> careful about the multiplications and not be distracted by them\n> when converting to the chunk-format API. The multiplications were\n> \"moved\" by that patch, not \"introduced\".\n\nHmph, I somehow had an impression that they did not have truncation\nissue in the original context, but perhaps I was wrong. OK.\n\n> I did a full inspection of all multiplications in midx.c and\n> found a few more instances of possible overflow. Two are on the\n> read side, but they require the object lookup chunk to have size\n> 4gb or larger. This is not _that_ far off from possibility! My\n> multi-pack-index for the Windows repository is currently ~1.6 GB\n> (in total, including the other chunks).\n\nThanks.\n"},{"id":"416293","messageId":"xmqqwnvmz1pv.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"CAPx1GvcNTkXo-6rdaRc4YNGAq8AoCtwT86AEUT+A6c22erVW0g@mail.gmail.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-05T20:41:48Z","receivedAt":"2021-02-05T21:18:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Chris Torek writes:\n\n> On Thu, Feb 4, 2021 at 4:00 PM Junio C Hamano wrote:\n>> * the fourth parameter of add_chunk() is of size_t, not uint64_t;\n>> shouldn't the multiplication be done in type size_t instead?\n>\n> There are (still) systems with 32-bit size_t (but 64-bit\n> off_t / file sizes), so ... probably not. Is size_t ever more than\n> 64 bits these days?\n\nSorry, you lost me. I do not see how it would help to perform the\nmultiplication in uint64_t, when you suspect that size_t is too\nsmall, if the final destination of the result of the multiplication\nis a function argument of type size_t?\n"},{"id":"416294","messageId":"3cd97f389f1f59566a85e4496321ee8486ac894c.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:50Z","receivedAt":"2021-02-05T21:44:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen calculating the sizes of certain chunks, we should use 64-bit\nmultiplication always. This allows us to properly predict the chunk\nsizes without risk of overflow.\n\nOther possible overflows were discovered by evaluating each\nmultiplication in midx.c and ensuring that at least one side of the\noperator was of type size_t or off_t.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 11 ++++++-----\n 1 file changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex aee9ed832d52..95648a1f368f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -246,7 +246,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n \n-\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n+\toffset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;\n \toffset32 = get_be32(offset_data + sizeof(uint32_t));\n \n \tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n@@ -262,7 +262,8 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \n static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n-\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n+\treturn get_be32(m->chunk_object_offsets +\n+\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n }\n \n static int nth_midxed_pack_entry(struct repository *r,\n@@ -914,15 +915,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n \t\t write_midx_oid_fanout);\n \tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n-\t\t ctx.entries_nr * the_hash_algo->rawsz,\n+\t\t (size_t)ctx.entries_nr * the_hash_algo->rawsz,\n \t\t write_midx_oid_lookup);\n \tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n-\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n+\t\t (size_t)ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n \t\t write_midx_object_offsets);\n \n \tif (ctx.large_offsets_needed)\n \t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n-\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n+\t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n-- \ngitgitgadget\n\n"},{"id":"416299","messageId":"16c37d2370cf4fd5fc309ac6dc8aa6443ffcf3d7.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:37Z","receivedAt":"2021-02-05T22:01:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of combining the logic from the commit-graph and\nmulti-pack-index file formats, create a new chunk-format API. Use a\n'struct chunkfile' pointer to keep track of data that has been\nregistered for writes. This struct is anonymous outside of\nchunk-format.c to ensure no user attempts to interfere with the data.\n\nThe next change will use this API in commit-graph.c, but the general\napproach is:\n\n 1. initialize the chunkfile with init_chunkfile(f).\n 2. add chunks in the intended writing order with add_chunk().\n 3. write any header information to the hashfile f.\n 4. write the chunkfile data using write_chunkfile().\n 5. free the chunkfile struct using free_chunkfile().\n\nHelped-by: Taylor Blau \nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n chunk-format.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 19 +++++++++++\n 3 files changed, 111 insertions(+)\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\ndiff --git a/Makefile b/Makefile\nindex 7b64106930a6..50a7663841e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -854,6 +854,7 @@ LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n+LIB_OBJS += chunk-format.o\n LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\ndiff --git a/chunk-format.c b/chunk-format.c\nnew file mode 100644\nindex 000000000000..6e0f1900213e\n--- /dev/null\n+++ b/chunk-format.c\n@@ -0,0 +1,91 @@\n+#include \"cache.h\"\n+#include \"chunk-format.h\"\n+#include \"csum-file.h\"\n+#define CHUNK_LOOKUP_WIDTH 12\n+\n+/*\n+ * When writing a chunk-based file format, collect the chunks in\n+ * an array of chunk_info structs. The size stores the _expected_\n+ * amount of data that will be written by write_fn.\n+ */\n+struct chunk_info {\n+\tuint32_t id;\n+\tuint64_t size;\n+\tchunk_write_fn write_fn;\n+};\n+\n+struct chunkfile {\n+\tstruct hashfile *f;\n+\n+\tstruct chunk_info *chunks;\n+\tsize_t chunks_nr;\n+\tsize_t chunks_alloc;\n+};\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f)\n+{\n+\tstruct chunkfile *cf = xcalloc(1, sizeof(*cf));\n+\tcf->f = f;\n+\treturn cf;\n+}\n+\n+void free_chunkfile(struct chunkfile *cf)\n+{\n+\tif (!cf)\n+\t\treturn;\n+\tfree(cf->chunks);\n+\tfree(cf);\n+}\n+\n+int get_num_chunks(struct chunkfile *cf)\n+{\n+\treturn cf->chunks_nr;\n+}\n+\n+void add_chunk(struct chunkfile *cf,\n+\t uint32_t id,\n+\t size_t size,\n+\t chunk_write_fn fn)\n+{\n+\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n+\n+\tcf->chunks[cf->chunks_nr].id = id;\n+\tcf->chunks[cf->chunks_nr].write_fn = fn;\n+\tcf->chunks[cf->chunks_nr].size = size;\n+\tcf->chunks_nr++;\n+}\n+\n+int write_chunkfile(struct chunkfile *cf, void *data)\n+{\n+\tint i;\n+\tuint64_t cur_offset = hashfile_total(cf->f);\n+\n+\t/* Add the table of contents to the current offset */\n+\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n+\t\thashwrite_be64(cf->f, cur_offset);\n+\n+\t\tcur_offset += cf->chunks[i].size;\n+\t}\n+\n+\t/* Trailing entry marks the end of the chunks */\n+\thashwrite_be32(cf->f, 0);\n+\thashwrite_be64(cf->f, cur_offset);\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\toff_t start_offset = hashfile_total(cf->f);\n+\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\n+\t\tif (hashfile_total(cf->f) - start_offset != cf->chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n+\t\t\t hashfile_total(cf->f) - start_offset);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nnew file mode 100644\nindex 000000000000..9a1d770accec\n--- /dev/null\n+++ b/chunk-format.h\n@@ -0,0 +1,19 @@\n+#ifndef CHUNK_FORMAT_H\n+#define CHUNK_FORMAT_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct hashfile;\n+struct chunkfile;\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f);\n+void free_chunkfile(struct chunkfile *cf);\n+int get_num_chunks(struct chunkfile *cf);\n+typedef int (*chunk_write_fn)(struct hashfile *f, void *data);\n+void add_chunk(struct chunkfile *cf,\n+\t uint32_t id,\n+\t size_t size,\n+\t chunk_write_fn fn);\n+int write_chunkfile(struct chunkfile *cf, void *data);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"416301","messageId":"e7064512ab7fe8abda200a98fee26346076bfb21.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 08/17] midx: add num_large_offsets to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:43Z","receivedAt":"2021-02-05T22:02:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t num_large_offsets\" into the context. With\nthis new data, write_midx_large_offsets() now matches the\nchunk_write_fn type.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 17 ++++++++++-------\n 1 file changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex cd994e333ecb..5be081f229ad 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -464,6 +464,7 @@ struct write_midx_context {\n \n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n+\tuint32_t num_large_offsets;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -772,11 +773,14 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n-\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects, *end = objects + nr_objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n \tsize_t written = 0;\n+\tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n \t\tstruct pack_midx_entry *obj;\n@@ -811,7 +815,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n@@ -861,7 +864,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n-\t\t\tnum_large_offsets++;\n+\t\t\tctx.num_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n@@ -961,7 +964,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tcur_chunk++;\n \t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n \t}\n \n \tchunk_ids[cur_chunk] = 0;\n@@ -1010,7 +1013,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n-- \ngitgitgadget\n\n"},{"id":"416303","messageId":"1d7484c0cffa1a316a1f6bdbcd986e98b0fe2b53.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 05/17] midx: use context in write_midx_pack_names()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:40Z","receivedAt":"2021-02-05T22:07:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align the write_midx_internal() to use the chunk-format\nAPI, start converting chunk writing methods to match chunk_write_fn. The\nfirst case is to convert write_midx_pack_names() to take \"void *data\".\nWe already have the necessary data in \"struct write_midx_context\", so\nthis conversion is rather mechanical.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 21 ++++++++++-----------\n 1 file changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 561f65a63a5b..88452b044337 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -643,27 +643,26 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t struct pack_info *info,\n-\t\t\t\t uint32_t num_packs)\n+static size_t write_midx_pack_names(struct hashfile *f, void *data)\n {\n+\tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n \tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < num_packs; i++) {\n+\tfor (i = 0; i < ctx->nr; i++) {\n \t\tsize_t writelen;\n \n-\t\tif (info[i].expired)\n+\t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n+\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t info[i - 1].pack_name,\n-\t\t\t info[i].pack_name);\n+\t\t\t ctx->info[i - 1].pack_name,\n+\t\t\t ctx->info[i].pack_name);\n \n-\t\twritelen = strlen(info[i].pack_name) + 1;\n-\t\thashwrite(f, info[i].pack_name, writelen);\n+\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n+\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -990,7 +989,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-- \ngitgitgadget\n\n"},{"id":"416305","messageId":"6bddd9e63b9b41d196db1bb74c60ad66566f66c7.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 14/17] midx: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:49Z","receivedAt":"2021-02-05T22:08:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). In particular, we\ncan use the return value of pair_chunk() to generate an error when a\nrequired chunk is missing.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 72 ++++++++++++++-----------------------\n t/t5319-multi-pack-index.sh | 6 ++--\n 2 files changed, 29 insertions(+), 49 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d9c7411b083b..aee9ed832d52 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -54,6 +54,19 @@ static char *get_midx_filename(const char *object_dir)\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+static int midx_read_oid_fanout(const unsigned char *chunk_start,\n+\t\t\t\tsize_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = data;\n+\tm->chunk_oid_fanout = (uint32_t *)chunk_start;\n+\n+\tif (chunk_size != 4 * 256) {\n+\t\terror(_(\"multi-pack-index OID fanout is of the wrong size\"));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\n@@ -65,6 +78,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n \tconst char *cur_pack_name;\n+\tstruct chunkfile *cf = NULL;\n \n \tfd = git_open(midx_name);\n \n@@ -114,58 +128,23 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n-\tfor (i = 0; i < m->num_chunks; i++) {\n-\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n-\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n-\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\n-\t\tif (chunk_offset >= m->data_len)\n-\t\t\tdie(_(\"invalid chunk offset (too large)\"));\n-\n-\t\tswitch (chunk_id) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase 0:\n-\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\t/*\n-\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n-\t\t\t\t * extensions to add optional chunks.\n-\t\t\t\t */\n-\t\t\t\tbreak;\n-\t\t}\n-\t}\n+\tcf = init_chunkfile(NULL);\n \n-\tif (!m->chunk_pack_names)\n+\tif (read_table_of_contents(cf, m->data, midx_size,\n+\t\t\t\t MIDX_HEADER_SIZE, m->num_chunks))\n+\t\tgoto cleanup_fail;\n+\n+\tif (pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, &m->chunk_pack_names) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n-\tif (!m->chunk_oid_fanout)\n+\tif (read_chunk(cf, MIDX_CHUNKID_OIDFANOUT, midx_read_oid_fanout, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n-\tif (!m->chunk_oid_lookup)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDLOOKUP, &m->chunk_oid_lookup) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n-\tif (!m->chunk_object_offsets)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS, &m->chunk_object_offsets) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n+\tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n@@ -191,6 +170,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n cleanup_fail:\n \tfree(m);\n \tfree(midx_name);\n+\tfree(cf);\n \tif (midx_map)\n \t\tmunmap(midx_map, midx_size);\n \tif (0 <= fd)\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a94f..ad4e878b65b8 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -314,12 +314,12 @@ test_expect_success 'verify bad OID version' '\n \n test_expect_success 'verify truncated chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\01\" $objdir \\\n-\t\t\"missing required\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'verify extended chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\07\" $objdir \\\n-\t\t\"terminating multi-pack-index chunk id appears earlier than expected\"\n+\t\t\"terminating chunk id appears earlier than expected\"\n '\n \n test_expect_success 'verify missing required chunk' '\n@@ -329,7 +329,7 @@ test_expect_success 'verify missing required chunk' '\n \n test_expect_success 'verify invalid chunk offset' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_OFFSET \"\\01\" $objdir \\\n-\t\t\"invalid chunk offset (too large)\"\n+\t\t\"improper chunk offset(s)\"\n '\n \n test_expect_success 'verify packnames out of order' '\n-- \ngitgitgadget\n\n"},{"id":"416308","messageId":"b283a38fb775614280cb1330029aafcd8ae8dfe8.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 07/17] midx: add pack_perm to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:42Z","receivedAt":"2021-02-05T22:12:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t *pack_perm\" and large_offsets_needed bit\ninto the context.\n\nUpdate write_midx_object_offsets() to match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 40 +++++++++++++++++++++-------------------\n 1 file changed, 21 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4520ef82b91b..cd994e333ecb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -461,6 +461,9 @@ struct write_midx_context {\n \n \tstruct pack_midx_entry *entries;\n \tuint32_t entries_nr;\n+\n+\tuint32_t *pack_perm;\n+\tunsigned large_offsets_needed:1;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -736,27 +739,27 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n-\t\t\t\t\tuint32_t *perm,\n-\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t\tvoid *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n \n-\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n-\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n \t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->offset);\n@@ -805,13 +808,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n-\tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -857,11 +858,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n+\tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n-\t\t\tlarge_offsets_needed = 1;\n+\t\t\tctx.large_offsets_needed = 1;\n \t}\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n@@ -900,13 +902,13 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n@@ -927,7 +929,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n-\tnum_chunks = large_offsets_needed ? 5 : 4;\n+\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -954,7 +956,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (large_offsets_needed) {\n+\tif (ctx.large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n \t\tcur_chunk++;\n@@ -1004,7 +1006,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -1039,7 +1041,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfree(ctx.info);\n \tfree(ctx.entries);\n-\tfree(pack_perm);\n+\tfree(ctx.pack_perm);\n \tfree(midx_name);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"416309","messageId":"ea0e7d40e537bb1de8a9d07b4974461e4641fd85.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 06/17] midx: add entries to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:41Z","receivedAt":"2021-02-05T22:12:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"struct pack_midx_entry *entries\" list and its count\ninto the context.\n\nUpdate write_midx_oid_fanout() and write_midx_oid_lookup() to take the\ncontext directly, as these are easy conversions with this new data.\n\nOnly the callers of write_midx_object_offsets() and\nwrite_midx_large_offsets() are updated here, since additional data in\nthe context before those methods can match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 49 ++++++++++++++++++++++++++-----------------------\n 1 file changed, 26 insertions(+), 23 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 88452b044337..4520ef82b91b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -458,6 +458,9 @@ struct write_midx_context {\n \tstruct multi_pack_index *m;\n \tstruct progress *progress;\n \tunsigned pack_paths_checked;\n+\n+\tstruct pack_midx_entry *entries;\n+\tuint32_t entries_nr;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -678,11 +681,11 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n }\n \n static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n-\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n \tuint32_t count = 0;\n \tuint32_t i;\n \n@@ -706,18 +709,19 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \treturn MIDX_CHUNK_FANOUT_SIZE;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+static size_t write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = data;\n+\tunsigned char hash_len = the_hash_algo->rawsz;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (i < nr_objects - 1) {\n+\t\tif (i < ctx->entries_nr - 1) {\n \t\t\tstruct pack_midx_entry *next = list;\n \t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n \t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n@@ -805,8 +809,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries, num_large_offsets = 0;\n-\tstruct pack_midx_entry *entries = NULL;\n+\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n@@ -852,12 +855,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n-\tfor (i = 0; i < nr_entries; i++) {\n-\t\tif (entries[i].offset > 0x7fffffff)\n+\tfor (i = 0; i < ctx.entries_nr; i++) {\n+\t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n-\t\tif (entries[i].offset > 0xffffffff)\n+\t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n@@ -947,10 +950,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * the_hash_algo->rawsz;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n \n \tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n \tif (large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n@@ -993,19 +996,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, the_hash_algo->rawsz, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1035,7 +1038,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tfree(ctx.info);\n-\tfree(entries);\n+\tfree(ctx.entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n \treturn result;\n-- \ngitgitgadget\n\n"},{"id":"416310","messageId":"b9a1bddf615fc1d725ffd1203669d0158296334d.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 16/17] chunk-format: restore duplicate chunk checks","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:51Z","receivedAt":"2021-02-05T22:12:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore refactoring into the chunk-format API, the commit-graph parsing\nlogic included checks for duplicate chunks. It is unlikely that we would\ndesire a chunk-based file format that allows duplicate chunk IDs in the\ntable of contents, so add duplicate checks into\nread_table_of_contents().\n\nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex bc9d4caf7276..e4889d9efcd1 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -98,6 +98,7 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\t uint64_t toc_offset,\n \t\t\t int toc_length)\n {\n+\tint i;\n \tuint32_t chunk_id;\n \tconst unsigned char *table_of_contents = mfile + toc_offset;\n \n@@ -124,6 +125,14 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\treturn -1;\n \t\t}\n \n+\t\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\t\tif (cf->chunks[i].id == chunk_id) {\n+\t\t\t\terror(_(\"duplicate chunk ID %\"PRIx32\" found\"),\n+\t\t\t\t\tchunk_id);\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t}\n+\n \t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n \t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n \t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n-- \ngitgitgadget\n\n"},{"id":"416311","messageId":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v2.git.1611759716.gitgitgadget@gmail.com","subject":"[PATCH v3 00/17] Refactor chunk-format into an API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:35Z","receivedAt":"2021-02-05T22:14:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a restart on the topic previously submitted [1] but dropped because\nak/corrected-commit-date was still in progress. This version is based on\nthat branch.\n\n[1]\nhttps://lore.kernel.org/git/pull.804.git.1607012215.gitgitgadget@gmail.com/\n\nThis version also changes the approach to use a more dynamic interaction\nwith a struct chunkfile pointer. This idea is credited to Taylor Blau [2],\nbut I started again from scratch. I also go further to make struct chunkfile\nanonymous to API consumers. It is defined only in chunk-format.c, which\nshould hopefully deter future users from interacting with that data\ndirectly.\n\n[2] https://lore.kernel.org/git/X8%2FI%2FRzXZksio+ri@nand.local/\n\nThis combined API is beneficial to reduce duplicated logic. Or rather, to\nensure that similar file formats have similar protections against bad data.\nThe multi-pack-index code did not have as many guards as the commit-graph\ncode did, but now they both share a common base that checks for things like\nduplicate chunks or offsets outside the size of the file.\n\nHere are some stats for the end-to-end change:\n\n * 570 insertions(+), 456 deletions(-).\n * commit-graph.c: 107 insertions(+), 192 deletions(-)\n * midx.c: 164 insertions(+), 260 deletions(-)\n\nWhile there is an overall increase to the code size, the consumers do get\nsmaller. Boilerplate things like abstracting method to match chunk_write_fn\nand chunk_read_fn make up a lot of these insertions. The \"interesting\" code\ngets a lot smaller and cleaner.\n\n\nUpdates in V3\n=============\n\n * API methods use better types and changed their order to match internal\n data more closely.\n\n * Use hashfile_total() instead of internal data values.\n\n * The implementation of pair_chunk() uses read_chunk().\n\n * init_chunkfile() has an in-code doc comment warning against using the\n same struct chunkfile for reads and writes.\n\n * More multiplications are correctly cast in midx.c.\n\n * The chunk-format technical docs are expanded.\n\n\nUpdates in V2\n=============\n\n * The method pair_chunk() now automatically sets a pointer while\n read_chunk() uses the callback. This greatly reduces the code size.\n\n * Pointer casts are now implicit instead of explicit.\n\n * Extra care is taken to not overflow when verifying chunk sizes on write.\n\nThanks, -Stolee\n\nDerrick Stolee (17):\n commit-graph: anonymize data in chunk_write_fn\n chunk-format: create chunk format write API\n commit-graph: use chunk-format write API\n midx: rename pack_info to write_midx_context\n midx: use context in write_midx_pack_names()\n midx: add entries to write_midx_context\n midx: add pack_perm to write_midx_context\n midx: add num_large_offsets to write_midx_context\n midx: return success/failure in chunk write methods\n midx: drop chunk progress during write\n midx: use chunk-format API in write_midx_internal()\n chunk-format: create read chunk API\n commit-graph: use chunk-format read API\n midx: use chunk-format read API\n midx: use 64-bit multiplication for chunk sizes\n chunk-format: restore duplicate chunk checks\n chunk-format: add technical docs\n\n Documentation/technical/chunk-format.txt | 116 +++++\n .../technical/commit-graph-format.txt | 3 +\n Documentation/technical/pack-format.txt | 3 +\n Makefile | 1 +\n chunk-format.c | 180 ++++++++\n chunk-format.h | 65 +++\n commit-graph.c | 299 +++++-------\n midx.c | 431 +++++++-----------\n t/t5318-commit-graph.sh | 2 +-\n t/t5319-multi-pack-index.sh | 6 +-\n 10 files changed, 648 insertions(+), 458 deletions(-)\n create mode 100644 Documentation/technical/chunk-format.txt\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\n\nbase-commit: 5a3b130cad0d5c770f766e3af6d32b41766374c0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-848%2Fderrickstolee%2Fchunk-format%2Frefactor-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-848/derrickstolee/chunk-format/refactor-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/848\n\nRange-diff vs v2:\n\n 1: 243dcec94368 = 1: 243dcec94368 commit-graph: anonymize data in chunk_write_fn\n 2: 814512f21671 ! 2: 16c37d2370cf chunk-format: create chunk format write API\n @@ Commit message\n 5. free the chunkfile struct using free_chunkfile().\n \n Helped-by: Taylor Blau \n + Helped-by: Junio C Hamano \n Signed-off-by: Derrick Stolee \n \n ## Makefile ##\n @@ chunk-format.c (new)\n +}\n +\n +void add_chunk(struct chunkfile *cf,\n -+\t uint64_t id,\n -+\t chunk_write_fn fn,\n -+\t size_t size)\n ++\t uint32_t id,\n ++\t size_t size,\n ++\t chunk_write_fn fn)\n +{\n +\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n +\n @@ chunk-format.c (new)\n +int write_chunkfile(struct chunkfile *cf, void *data)\n +{\n +\tint i;\n -+\tsize_t cur_offset = cf->f->offset + cf->f->total;\n ++\tuint64_t cur_offset = hashfile_total(cf->f);\n +\n +\t/* Add the table of contents to the current offset */\n +\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n @@ chunk-format.c (new)\n +\thashwrite_be64(cf->f, cur_offset);\n +\n +\tfor (i = 0; i < cf->chunks_nr; i++) {\n -+\t\tuint64_t start_offset = cf->f->total + cf->f->offset;\n ++\t\toff_t start_offset = hashfile_total(cf->f);\n +\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n +\n +\t\tif (result)\n +\t\t\treturn result;\n +\n -+\t\tif (cf->f->total + cf->f->offset - start_offset != cf->chunks[i].size)\n ++\t\tif (hashfile_total(cf->f) - start_offset != cf->chunks[i].size)\n +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n +\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n -+\t\t\t cf->f->total + cf->f->offset - start_offset);\n ++\t\t\t hashfile_total(cf->f) - start_offset);\n +\t}\n +\n +\treturn 0;\n @@ chunk-format.h (new)\n +struct chunkfile *init_chunkfile(struct hashfile *f);\n +void free_chunkfile(struct chunkfile *cf);\n +int get_num_chunks(struct chunkfile *cf);\n -+typedef int (*chunk_write_fn)(struct hashfile *f,\n -+\t\t\t void *data);\n ++typedef int (*chunk_write_fn)(struct hashfile *f, void *data);\n +void add_chunk(struct chunkfile *cf,\n -+\t uint64_t id,\n -+\t chunk_write_fn fn,\n -+\t size_t size);\n ++\t uint32_t id,\n ++\t size_t size,\n ++\t chunk_write_fn fn);\n +int write_chunkfile(struct chunkfile *cf, void *data);\n +\n +#endif\n 3: 70af6e3083f4 ! 3: e549e24d79af commit-graph: use chunk-format write API\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con\n -\tchunks[2].write_fn = write_graph_chunk_data;\n +\tcf = init_chunkfile(f);\n +\n -+\tadd_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n -+\t\t write_graph_chunk_fanout, GRAPH_FANOUT_SIZE);\n -+\tadd_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP,\n -+\t\t write_graph_chunk_oids, hashsz * ctx->commits.nr);\n -+\tadd_chunk(cf, GRAPH_CHUNKID_DATA,\n -+\t\t write_graph_chunk_data, (hashsz + 16) * ctx->commits.nr);\n ++\tadd_chunk(cf, GRAPH_CHUNKID_OIDFANOUT, GRAPH_FANOUT_SIZE,\n ++\t\t write_graph_chunk_fanout);\n ++\tadd_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, hashsz * ctx->commits.nr,\n ++\t\t write_graph_chunk_oids);\n ++\tadd_chunk(cf, GRAPH_CHUNKID_DATA, (hashsz + 16) * ctx->commits.nr,\n ++\t\t write_graph_chunk_data);\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_NO_GDAT, 0))\n \t\tctx->write_generation_data = 0;\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con\n -\t}\n +\tif (ctx->write_generation_data)\n +\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n -+\t\t\t write_graph_chunk_generation_data,\n -+\t\t\t sizeof(uint32_t) * ctx->commits.nr);\n ++\t\t\t sizeof(uint32_t) * ctx->commits.nr,\n ++\t\t\t write_graph_chunk_generation_data);\n +\tif (ctx->num_generation_data_overflows)\n +\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n -+\t\t\t write_graph_chunk_generation_data_overflow,\n -+\t\t\t sizeof(timestamp_t) * ctx->num_generation_data_overflows);\n ++\t\t\t sizeof(timestamp_t) * ctx->num_generation_data_overflows,\n ++\t\t\t write_graph_chunk_generation_data_overflow);\n +\tif (ctx->num_extra_edges)\n +\t\tadd_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES,\n -+\t\t\t write_graph_chunk_extra_edges,\n -+\t\t\t 4 * ctx->num_extra_edges);\n ++\t\t\t 4 * ctx->num_extra_edges,\n ++\t\t\t write_graph_chunk_extra_edges);\n \tif (ctx->changed_paths) {\n -\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n -\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con\n -\tchunks[num_chunks].id = 0;\n -\tchunks[num_chunks].size = 0;\n +\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n -+\t\t\t write_graph_chunk_bloom_indexes,\n -+\t\t\t sizeof(uint32_t) * ctx->commits.nr);\n ++\t\t\t sizeof(uint32_t) * ctx->commits.nr,\n ++\t\t\t write_graph_chunk_bloom_indexes);\n +\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n -+\t\t\t write_graph_chunk_bloom_data,\n +\t\t\t sizeof(uint32_t) * 3\n -+\t\t\t\t+ ctx->total_bloom_filter_data_size);\n ++\t\t\t\t+ ctx->total_bloom_filter_data_size,\n ++\t\t\t write_graph_chunk_bloom_data);\n +\t}\n +\tif (ctx->num_commit_graphs_after > 1)\n +\t\tadd_chunk(cf, GRAPH_CHUNKID_BASE,\n -+\t\t\t write_graph_chunk_base,\n -+\t\t\t hashsz * (ctx->num_commit_graphs_after - 1));\n ++\t\t\t hashsz * (ctx->num_commit_graphs_after - 1),\n ++\t\t\t write_graph_chunk_base);\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n 4: 0cac7890bed7 = 4: 66ff49ed9309 midx: rename pack_info to write_midx_context\n 5: 4a4e90b129ae = 5: 1d7484c0cffa midx: use context in write_midx_pack_names()\n 6: 30ad423997b7 = 6: ea0e7d40e537 midx: add entries to write_midx_context\n 7: 2f1c496f3ab5 = 7: b283a38fb775 midx: add pack_perm to write_midx_context\n 8: c4939548e51c = 8: e7064512ab7f midx: add num_large_offsets to write_midx_context\n 9: b3cc73c22567 ! 9: 7aa3242e15b7 midx: return success/failure in chunk write methods\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \tstop_progress(&progress);\n \n -\tif (written != chunk_offsets[num_chunks])\n -+\tif (f->total + f->offset != chunk_offsets[num_chunks])\n ++\tif (hashfile_total(f) != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n -\t\t written,\n -+\t\t f->total + f->offset,\n ++\t\t hashfile_total(f),\n \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n 10: 78744d3b7016 ! 10: 70f68c95e479 midx: drop chunk progress during write\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \t}\n -\tstop_progress(&progress);\n \n - \tif (f->total + f->offset != chunk_offsets[num_chunks])\n + \tif (hashfile_total(f) != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n 11: 07dc0cf8c683 ! 11: 787cd7f18d2e midx: use chunk-format API in write_midx_internal()\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n -\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n -\t\t\t\twrite_midx_pack_names(f, &ctx);\n -\t\t\t\tbreak;\n -+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES,\n -+\t\t write_midx_pack_names, pack_name_concat_len);\n -+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n -+\t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n ++\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n ++\t\t write_midx_pack_names);\n ++\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n ++\t\t write_midx_oid_fanout);\n +\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n -+\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n ++\t\t ctx.entries_nr * the_hash_algo->rawsz,\n ++\t\t write_midx_oid_lookup);\n +\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n -+\t\t write_midx_object_offsets,\n -+\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n ++\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n ++\t\t write_midx_object_offsets);\n \n -\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n -\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n -\t}\n +\tif (ctx.large_offsets_needed)\n +\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n -+\t\t\twrite_midx_large_offsets,\n -+\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n ++\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n ++\t\t\twrite_midx_large_offsets);\n \n --\tif (f->total + f->offset != chunk_offsets[num_chunks])\n +-\tif (hashfile_total(f) != chunk_offsets[num_chunks])\n -\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n --\t\t f->total + f->offset,\n +-\t\t hashfile_total(f),\n -\t\t chunk_offsets[num_chunks]);\n +\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n +\twrite_chunkfile(cf, &ctx);\n 12: d8d8e9e2aa3f ! 12: 366eb2afee83 chunk-format: create read chunk API\n @@ Commit message\n read. If the same struct instance was used for both reads and writes,\n then there would be failures.\n \n + Helped-by: Junio C Hamano \n Signed-off-by: Derrick Stolee \n \n ## chunk-format.c ##\n @@ chunk-format.c: int write_chunkfile(struct chunkfile *cf, void *data)\n +\treturn 0;\n +}\n +\n ++static int pair_chunk_fn(const unsigned char *chunk_start,\n ++\t\t\t size_t chunk_size,\n ++\t\t\t void *data)\n ++{\n ++\tconst unsigned char **p = data;\n ++\t*p = chunk_start;\n ++\treturn 0;\n ++}\n ++\n +int pair_chunk(struct chunkfile *cf,\n +\t uint32_t chunk_id,\n +\t const unsigned char **p)\n +{\n -+\tint i;\n -+\n -+\tfor (i = 0; i < cf->chunks_nr; i++) {\n -+\t\tif (cf->chunks[i].id == chunk_id) {\n -+\t\t\t*p = cf->chunks[i].start;\n -+\t\t\treturn 0;\n -+\t\t}\n -+\t}\n -+\n -+\treturn CHUNK_NOT_FOUND;\n ++\treturn read_chunk(cf, chunk_id, pair_chunk_fn, p);\n +}\n +\n +int read_chunk(struct chunkfile *cf,\n @@ chunk-format.c: int write_chunkfile(struct chunkfile *cf, void *data)\n +}\n \n ## chunk-format.h ##\n +@@\n + struct hashfile;\n + struct chunkfile;\n + \n ++/*\n ++ * Initialize a 'struct chunkfile' for writing _or_ reading a file\n ++ * with the chunk format.\n ++ *\n ++ * If writing a file, supply a non-NULL 'struct hashfile *' that will\n ++ * be used to write.\n ++ *\n ++ * If reading a file, then supply the memory-mapped data to the\n ++ * pair_chunk() or read_chunk() methods, as appropriate.\n ++ *\n ++ * DO NOT MIX THESE MODES. Use different 'struct chunkfile' instances\n ++ * for reading and writing.\n ++ */\n + struct chunkfile *init_chunkfile(struct hashfile *f);\n + void free_chunkfile(struct chunkfile *cf);\n + int get_num_chunks(struct chunkfile *cf);\n @@ chunk-format.h: void add_chunk(struct chunkfile *cf,\n - \t size_t size);\n + \t chunk_write_fn fn);\n int write_chunkfile(struct chunkfile *cf, void *data);\n \n +int read_table_of_contents(struct chunkfile *cf,\n 13: 8744d2785965 = 13: 7838ad32e2e0 commit-graph: use chunk-format read API\n 14: 750c03253c95 ! 14: 6bddd9e63b9b midx: use chunk-format read API\n @@ midx.c: struct multi_pack_index *load_multi_pack_index(const char *object_dir, i\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n +@@ midx.c: struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n + cleanup_fail:\n + \tfree(m);\n + \tfree(midx_name);\n ++\tfree(cf);\n + \tif (midx_map)\n + \t\tmunmap(midx_map, midx_size);\n + \tif (0 <= fd)\n \n ## t/t5319-multi-pack-index.sh ##\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'verify bad OID version' '\n 15: 83d292532a0f ! 15: 3cd97f389f1f midx: use 64-bit multiplication for chunk sizes\n @@ Commit message\n multiplication always. This allows us to properly predict the chunk\n sizes without risk of overflow.\n \n + Other possible overflows were discovered by evaluating each\n + multiplication in midx.c and ensuring that at least one side of the\n + operator was of type size_t or off_t.\n +\n Signed-off-by: Derrick Stolee \n \n ## midx.c ##\n +@@ midx.c: static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n + \tconst unsigned char *offset_data;\n + \tuint32_t offset32;\n + \n +-\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n ++\toffset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;\n + \toffset32 = get_be32(offset_data + sizeof(uint32_t));\n + \n + \tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n +@@ midx.c: static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n + \n + static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n + {\n +-\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n ++\treturn get_be32(m->chunk_object_offsets +\n ++\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n + }\n + \n + static int nth_midxed_pack_entry(struct repository *r,\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT,\n - \t\t write_midx_oid_fanout, MIDX_CHUNK_FANOUT_SIZE);\n + \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n + \t\t write_midx_oid_fanout);\n \tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n --\t\t write_midx_oid_lookup, ctx.entries_nr * the_hash_algo->rawsz);\n -+\t\t write_midx_oid_lookup, (uint64_t)ctx.entries_nr * the_hash_algo->rawsz);\n +-\t\t ctx.entries_nr * the_hash_algo->rawsz,\n ++\t\t (size_t)ctx.entries_nr * the_hash_algo->rawsz,\n + \t\t write_midx_oid_lookup);\n \tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n - \t\t write_midx_object_offsets,\n - \t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH);\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n +-\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n ++\t\t (size_t)ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n + \t\t write_midx_object_offsets);\n + \n \tif (ctx.large_offsets_needed)\n \t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n - \t\t\twrite_midx_large_offsets,\n --\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n -+\t\t\t(uint64_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH);\n +-\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n ++\t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n + \t\t\twrite_midx_large_offsets);\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n - \twrite_chunkfile(cf, &ctx);\n 16: 669eeec707ab ! 16: b9a1bddf615f chunk-format: restore duplicate chunk checks\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## chunk-format.c ##\n -@@ chunk-format.c: struct chunk_info {\n - \tchunk_write_fn write_fn;\n - \n - \tconst void *start;\n -+\tunsigned found:1;\n - };\n - \n - struct chunkfile {\n @@ chunk-format.c: int read_table_of_contents(struct chunkfile *cf,\n \t\t\t uint64_t toc_offset,\n \t\t\t int toc_length)\n 17: 8f3985ab5df3 ! 17: 4c7d751f1e39 chunk-format: add technical docs\n @@ Documentation/technical/chunk-format.txt (new)\n +\n +Functions for working with chunk-based file formats are declared in\n +`chunk-format.h`. Using these methods provide extra checks that assist\n -+developers when creating new file formats, including:\n ++developers when creating new file formats.\n +\n -+ 1. Writing and reading the table of contents.\n ++Writing chunk-based file formats\n ++--------------------------------\n +\n -+ 2. Verifying that the data written in a chunk matches the expected size\n -+ that was recorded in the table of contents.\n ++To write a chunk-based file format, create a `struct chunkfile` by\n ++calling `init_chunkfile()` and pass a `struct hashfile` pointer. The\n ++caller is responsible for opening the `hashfile` and writing header\n ++information so the file format is identifiable before the chunk-based\n ++format begins.\n +\n -+ 3. Checking that a table of contents describes offsets properly within\n -+ the file boundaries.\n ++Then, call `add_chunk()` for each chunk that is intended for write. This\n ++populates the `chunkfile` with information about the order and size of\n ++each chunk to write. Provide a `chunk_write_fn` function pointer to\n ++perform the write of the chunk data upon request.\n ++\n ++Call `write_chunkfile()` to write the table of contents to the `hashfile`\n ++followed by each of the chunks. This will verify that each chunk wrote\n ++the expected amount of data so the table of contents is correct.\n ++\n ++Finally, call `free_chunkfile()` to clear the `struct chunkfile` data. The\n ++caller is responsible for finalizing the `hashfile` by writing the trailing\n ++hash and closing the file.\n ++\n ++Reading chunk-based file formats\n ++--------------------------------\n ++\n ++To read a chunk-based file format, the file must be opened as a\n ++memory-mapped region. The chunk-format API expects that the entire file\n ++is mapped as a contiguous memory region.\n ++\n ++Initialize a `struct chunkfile` pointer with `init_chunkfile(NULL)`.\n ++\n ++After reading the header information from the beginning of the file,\n ++including the chunk count, call `read_table_of_contents()` to populate\n ++the `struct chunkfile` with the list of chunks, their offsets, and their\n ++sizes.\n ++\n ++Extract the data information for each chunk using `pair_chunk()` or\n ++`read_chunk()`:\n ++\n ++* `pair_chunk()` assigns a given pointer with the location inside the\n ++ memory-mapped file corresponding to that chunk's offset. If the chunk\n ++ does not exist, then the pointer is not modified.\n ++\n ++* `read_chunk()` takes a `chunk_read_fn` function pointer and calls it\n ++ with the appropriate initial pointer and size information. The function\n ++ is not called if the chunk does not exist. Use this method to read chunks\n ++ if you need to perform immediate parsing or if you need to execute logic\n ++ based on the size of the chunk.\n ++\n ++After calling these methods, call `free_chunkfile()` to clear the\n ++`struct chunkfile` data. This will not close the memory-mapped region.\n ++Callers are expected to own that data for the timeframe the pointers into\n ++the region are needed.\n ++\n ++Examples\n ++--------\n ++\n ++These file formats use the chunk-format API, and can be used as examples\n ++for future formats:\n ++\n ++* *commit-graph:* see `write_commit_graph_file()` and `parse_commit_graph()`\n ++ in `commit-graph.c` for how the chunk-format API is used to write and\n ++ parse the commit-graph file format documented in\n ++ link:technical/commit-graph-format.html[the commit-graph file format].\n ++\n ++* *multi-pack-index:* see `write_midx_internal()` and `load_multi_pack_index()`\n ++ in `midx.c` for how the chunk-format API is used to write and\n ++ parse the multi-pack-index file format documented in\n ++ link:technical/pack-format.html[the multi-pack-index file format].\n \n ## Documentation/technical/commit-graph-format.txt ##\n @@ Documentation/technical/commit-graph-format.txt: CHUNK LOOKUP:\n\n-- \ngitgitgadget\n"},{"id":"416313","messageId":"7838ad32e2e0b2e5941f5027d873f2453e08f7cb.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 13/17] commit-graph: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:48Z","receivedAt":"2021-02-05T22:16:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). While the current\nimplementation loses the duplicate-chunk detection, that will be added\nin a future change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 154 ++++++++++++++--------------------------\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 53 insertions(+), 103 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7c607d23b29f..32cf5091d2fb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -299,15 +299,43 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n+static int graph_read_oid_lookup(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = data;\n+\tg->chunk_oid_lookup = chunk_start;\n+\tg->num_commits = chunk_size / g->hash_len;\n+\treturn 0;\n+}\n+\n+static int graph_read_bloom_data(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = data;\n+\tuint32_t hash_version;\n+\tg->chunk_bloom_data = chunk_start;\n+\thash_version = get_be32(chunk_start);\n+\n+\tif (hash_version != 1)\n+\t\treturn 0;\n+\n+\tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n+\tg->bloom_filter_settings->hash_version = hash_version;\n+\tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n+\tg->bloom_filter_settings->bits_per_entry = get_be32(chunk_start + 8);\n+\tg->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n+\n+\treturn 0;\n+}\n+\n struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n-\tconst unsigned char *data, *chunk_lookup;\n-\tuint32_t i;\n+\tconst unsigned char *data;\n \tstruct commit_graph *graph;\n-\tuint64_t next_chunk_offset;\n \tuint32_t graph_signature;\n \tunsigned char graph_version, hash_version;\n+\tstruct chunkfile *cf = NULL;\n \n \tif (!graph_map)\n \t\treturn NULL;\n@@ -356,108 +384,28 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n-\tchunk_lookup = data + 8;\n-\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\tfor (i = 0; i < graph->num_chunks; i++) {\n-\t\tuint32_t chunk_id;\n-\t\tuint64_t chunk_offset = next_chunk_offset;\n-\t\tint chunk_repeated = 0;\n-\n-\t\tchunk_id = get_be32(chunk_lookup + 0);\n-\n-\t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n-\t\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\n-\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n-\t\t\terror(_(\"commit-graph improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n-\t\t\t (uint32_t)chunk_offset);\n-\t\t\tgoto free_and_return;\n-\t\t}\n-\n-\t\tswitch (chunk_id) {\n-\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n-\t\t\tif (graph->chunk_oid_fanout)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_oid_fanout = (uint32_t*)(data + chunk_offset);\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n-\t\t\tif (graph->chunk_oid_lookup)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n-\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n-\t\t\t\tgraph->num_commits = (next_chunk_offset - chunk_offset)\n-\t\t\t\t\t\t / graph->hash_len;\n-\t\t\t}\n-\t\t\tbreak;\n+\tcf = init_chunkfile(NULL);\n \n-\t\tcase GRAPH_CHUNKID_DATA:\n-\t\t\tif (graph->chunk_commit_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA:\n-\t\t\tif (graph->chunk_generation_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW:\n-\t\t\tif (graph->chunk_generation_data_overflow)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data_overflow = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_EXTRAEDGES:\n-\t\t\tif (graph->chunk_extra_edges)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BASE:\n-\t\t\tif (graph->chunk_base_graphs)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n-\t\t\tif (graph->chunk_bloom_indexes)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n-\t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMDATA:\n-\t\t\tif (graph->chunk_bloom_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n-\t\t\t\tuint32_t hash_version;\n-\t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n-\t\t\t\thash_version = get_be32(data + chunk_offset);\n-\n-\t\t\t\tif (hash_version != 1)\n-\t\t\t\t\tbreak;\n+\tif (read_table_of_contents(cf, graph->data, graph_size,\n+\t\t\t\t GRAPH_HEADER_SIZE, graph->num_chunks))\n+\t\tgoto free_and_return;\n \n-\t\t\t\tgraph->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n-\t\t\t\tgraph->bloom_filter_settings->hash_version = hash_version;\n-\t\t\t\tgraph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n-\t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n-\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n+\tpair_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n+\t\t (const unsigned char **)&graph->chunk_oid_fanout);\n+\tread_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, graph_read_oid_lookup, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_DATA, &graph->chunk_commit_data);\n+\tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, &graph->chunk_extra_edges);\n+\tpair_chunk(cf, GRAPH_CHUNKID_BASE, &graph->chunk_base_graphs);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t &graph->chunk_generation_data);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t &graph->chunk_generation_data_overflow);\n \n-\t\tif (chunk_repeated) {\n-\t\t\terror(_(\"commit-graph chunk id %08x appears multiple times\"), chunk_id);\n-\t\t\tgoto free_and_return;\n-\t\t}\n+\tif (r->settings.commit_graph_read_changed_paths) {\n+\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t &graph->chunk_bloom_indexes);\n+\t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -474,9 +422,11 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \tif (verify_commit_graph_lite(graph))\n \t\tgoto free_and_return;\n \n+\tfree_chunkfile(cf);\n \treturn graph;\n \n free_and_return:\n+\tfree_chunkfile(cf);\n \tfree(graph->bloom_filter_settings);\n \tfree(graph);\n \treturn NULL;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex fa27df579a57..c7da741284e5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -564,7 +564,7 @@ test_expect_success 'detect bad hash version' '\n \n test_expect_success 'detect low chunk count' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\01\" \\\n-\t\t\"missing the .* chunk\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'detect missing OID fanout chunk' '\n-- \ngitgitgadget\n\n"},{"id":"416314","messageId":"243dcec9436853ff8d1bf2580e76ab909b7cb324.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 01/17] commit-graph: anonymize data in chunk_write_fn","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:36Z","receivedAt":"2021-02-05T22:17:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for creating an API around file formats using chunks and\ntables of contents, prepare the commit-graph write code to use\nprototypes that will match this new API.\n\nSpecifically, convert chunk_write_fn to take a \"void *data\" parameter\ninstead of the commit-graph-specific \"struct write_commit_graph_context\"\npointer.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++----------\n 1 file changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f3bde2ad95a1..fae7d1b63931 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1040,8 +1040,9 @@ struct write_commit_graph_context {\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n \n@@ -1066,8 +1067,9 @@ static int write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static int write_graph_chunk_oids(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n@@ -1085,8 +1087,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static int write_graph_chunk_data(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n@@ -1187,8 +1190,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i, num_generation_data_overflows = 0;\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n@@ -1208,8 +1212,9 @@ static int write_graph_chunk_generation_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n-\t\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1226,8 +1231,9 @@ static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n }\n \n static int write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n@@ -1280,8 +1286,9 @@ static int write_graph_chunk_extra_edges(struct hashfile *f,\n }\n \n static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t cur_pos = 0;\n@@ -1315,8 +1322,9 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n }\n \n static int write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\tstruct write_commit_graph_context *ctx)\n+\t\t\t\t\tvoid *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \n@@ -1737,8 +1745,9 @@ static int write_graph_chunk_base_1(struct hashfile *f,\n }\n \n static int write_graph_chunk_base(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n \n \tif (num != ctx->num_commit_graphs_after - 1) {\n@@ -1750,7 +1759,7 @@ static int write_graph_chunk_base(struct hashfile *f,\n }\n \n typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t struct write_commit_graph_context *ctx);\n+\t\t\t void *data);\n \n struct chunk_info {\n \tuint32_t id;\n-- \ngitgitgadget\n\n"},{"id":"416315","messageId":"e549e24d79af924cc5970ec404037d29c3bc104c.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 03/17] commit-graph: use chunk-format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:38Z","receivedAt":"2021-02-05T22:18:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph write logic is ready to make use of the chunk-format\nwrite API. Each chunk write method is already in the correct prototype.\nWe only need to use the 'struct chunkfile' pointer and the correct API\ncalls.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 118 ++++++++++++++++---------------------------------\n 1 file changed, 37 insertions(+), 81 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fae7d1b63931..7c607d23b29f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -19,6 +19,7 @@\n #include \"shallow.h\"\n #include \"json-writer.h\"\n #include \"trace2.h\"\n+#include \"chunk-format.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -1758,27 +1759,17 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t void *data);\n-\n-struct chunk_info {\n-\tuint32_t id;\n-\tuint64_t size;\n-\tchunk_write_fn write_fn;\n-};\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n \tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tstruct chunk_info chunks[MAX_NUM_CHUNKS + 1];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n-\tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n+\tstruct chunkfile *cf;\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1824,76 +1815,50 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \t}\n \n-\tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunks[0].size = GRAPH_FANOUT_SIZE;\n-\tchunks[0].write_fn = write_graph_chunk_fanout;\n-\tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunks[1].size = hashsz * ctx->commits.nr;\n-\tchunks[1].write_fn = write_graph_chunk_oids;\n-\tchunks[2].id = GRAPH_CHUNKID_DATA;\n-\tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n-\tchunks[2].write_fn = write_graph_chunk_data;\n+\tcf = init_chunkfile(f);\n+\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDFANOUT, GRAPH_FANOUT_SIZE,\n+\t\t write_graph_chunk_fanout);\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, hashsz * ctx->commits.nr,\n+\t\t write_graph_chunk_oids);\n+\tadd_chunk(cf, GRAPH_CHUNKID_DATA, (hashsz + 16) * ctx->commits.nr,\n+\t\t write_graph_chunk_data);\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_NO_GDAT, 0))\n \t\tctx->write_generation_data = 0;\n-\tif (ctx->write_generation_data) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_generation_data_overflows) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW;\n-\t\tchunks[num_chunks].size = sizeof(timestamp_t) * ctx->num_generation_data_overflows;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data_overflow;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_extra_edges) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n-\t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n-\t\tnum_chunks++;\n-\t}\n+\tif (ctx->write_generation_data)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr,\n+\t\t\t write_graph_chunk_generation_data);\n+\tif (ctx->num_generation_data_overflows)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t\t sizeof(timestamp_t) * ctx->num_generation_data_overflows,\n+\t\t\t write_graph_chunk_generation_data_overflow);\n+\tif (ctx->num_extra_edges)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES,\n+\t\t\t 4 * ctx->num_extra_edges,\n+\t\t\t write_graph_chunk_extra_edges);\n \tif (ctx->changed_paths) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n-\t\tnum_chunks++;\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n-\t\t\t\t\t + ctx->total_bloom_filter_data_size;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n-\t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n-\t\tnum_chunks++;\n-\t}\n-\n-\tchunks[num_chunks].id = 0;\n-\tchunks[num_chunks].size = 0;\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr,\n+\t\t\t write_graph_chunk_bloom_indexes);\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t sizeof(uint32_t) * 3\n+\t\t\t\t+ ctx->total_bloom_filter_data_size,\n+\t\t\t write_graph_chunk_bloom_data);\n+\t}\n+\tif (ctx->num_commit_graphs_after > 1)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BASE,\n+\t\t\t hashsz * (ctx->num_commit_graphs_after - 1),\n+\t\t\t write_graph_chunk_base);\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n-\tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\n-\t\tchunk_offset += chunks[i].size;\n-\t}\n-\n \tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n@@ -1905,17 +1870,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tuint64_t start_offset = f->total + f->offset;\n-\n-\t\tif (chunks[i].write_fn(f, ctx))\n-\t\t\treturn -1;\n-\n-\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n-\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n-\t\t\t chunks[i].size, chunks[i].id,\n-\t\t\t f->total + f->offset - start_offset);\n-\t}\n+\twrite_chunkfile(cf, ctx);\n \n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n@@ -1932,6 +1887,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tclose_commit_graph(ctx->r->objects);\n \tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfree_chunkfile(cf);\n \n \tif (ctx->split) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n-- \ngitgitgadget\n\n"},{"id":"416316","messageId":"66ff49ed93091ea34cb21eff60b3114eba006ff3.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 04/17] midx: rename pack_info to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:39Z","receivedAt":"2021-02-05T22:18:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to streamline our chunk-based file formats, align some of\nthe code structure in write_midx_internal() to be similar to the\npatterns in write_commit_graph_file().\n\nSpecifically, let's create a \"struct write_midx_context\" that can be\nused as a data parameter to abstract function types.\n\nThis change only renames \"struct pack_info\" to \"struct\nwrite_midx_context\" and the names of instances from \"packs\" to \"ctx\". In\nfuture changes, we will expand the data inside \"struct\nwrite_midx_context\" and align our chunk-writing method with the\nchunk-format API.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 130 ++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 65 insertions(+), 65 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 79c282b070d2..561f65a63a5b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -451,7 +451,7 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n-struct pack_list {\n+struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n \tuint32_t alloc;\n@@ -463,37 +463,37 @@ struct pack_list {\n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t const char *file_name, void *data)\n {\n-\tstruct pack_list *packs = (struct pack_list *)data;\n+\tstruct write_midx_context *ctx = data;\n \n \tif (ends_with(file_name, \".idx\")) {\n-\t\tdisplay_progress(packs->progress, ++packs->pack_paths_checked);\n-\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n+\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n-\t\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t\t 0);\n+\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->info[packs->nr].p) {\n+\t\tif (!ctx->info[ctx->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n+\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->info[packs->nr].p);\n-\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n+\t\t\tclose_pack(ctx->info[ctx->nr].p);\n+\t\t\tFREE_AND_NULL(ctx->info[ctx->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n-\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n-\t\tpacks->info[packs->nr].expired = 0;\n-\t\tpacks->nr++;\n+\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n+\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n+\t\tctx->info[ctx->nr].expired = 0;\n+\t\tctx->nr++;\n \t}\n }\n \n@@ -801,7 +801,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n-\tstruct pack_list packs;\n+\tstruct write_midx_context ctx = { 0 };\n \tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n@@ -820,40 +820,40 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t midx_name);\n \n \tif (m)\n-\t\tpacks.m = m;\n+\t\tctx.m = m;\n \telse\n-\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n-\n-\tpacks.nr = 0;\n-\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n-\tpacks.info = NULL;\n-\tALLOC_ARRAY(packs.info, packs.alloc);\n-\n-\tif (packs.m) {\n-\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n-\n-\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n-\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.info[packs.nr].p = NULL;\n-\t\t\tpacks.info[packs.nr].expired = 0;\n-\t\t\tpacks.nr++;\n+\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\n+\tctx.nr = 0;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.info = NULL;\n+\tALLOC_ARRAY(ctx.info, ctx.alloc);\n+\n+\tif (ctx.m) {\n+\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n+\n+\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n+\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n+\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].expired = 0;\n+\t\t\tctx.nr++;\n \t\t}\n \t}\n \n-\tpacks.pack_paths_checked = 0;\n+\tctx.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n-\t\tpacks.progress = NULL;\n+\t\tctx.progress = NULL;\n \n-\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n-\tstop_progress(&packs.progress);\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n+\tstop_progress(&ctx.progress);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -862,19 +862,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n-\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n \tif (packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n-\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n \t\t\t\t\t packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n-\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n \t\t\t\t packs_to_drop->items[drop_index].string);\n@@ -882,7 +882,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n \t\t\t} else {\n-\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t\tctx.info[i].expired = 0;\n \t\t\t}\n \t\t}\n \n@@ -898,19 +898,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].expired) {\n+\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (!packs.info[i].expired)\n-\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (!ctx.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n@@ -921,19 +921,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\tif (packs.m)\n-\t\tclose_midx(packs.m);\n+\tif (ctx.m)\n+\t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\tif (packs.nr - dropped_packs == 0) {\n+\tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n+\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -990,7 +990,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -1027,15 +1027,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tcommit_lock_file(&lk);\n \n cleanup:\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].p) {\n-\t\t\tclose_pack(packs.info[i].p);\n-\t\t\tfree(packs.info[i].p);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].p) {\n+\t\t\tclose_pack(ctx.info[i].p);\n+\t\t\tfree(ctx.info[i].p);\n \t\t}\n-\t\tfree(packs.info[i].pack_name);\n+\t\tfree(ctx.info[i].pack_name);\n \t}\n \n-\tfree(packs.info);\n+\tfree(ctx.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \ngitgitgadget\n\n"},{"id":"416317","messageId":"787cd7f18d2e47d975e10bf0f56188bc35589124.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 11/17] midx: use chunk-format API in write_midx_internal()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:46Z","receivedAt":"2021-02-05T22:18:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-format API allows writing the table of contents and all chunks\nusing the anonymous 'struct chunkfile' type. We only need to convert our\nlocal chunk logic to this API for the multi-pack-index writes to share\nthat logic with the commit-graph file writes.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 105 +++++++++++----------------------------------------------\n 1 file changed, 20 insertions(+), 85 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4f4aa351e60e..d9c7411b083b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -11,6 +11,7 @@\n #include \"trace2.h\"\n #include \"run-command.h\"\n #include \"repository.h\"\n+#include \"chunk-format.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -799,18 +800,15 @@ static int write_midx_large_offsets(struct hashfile *f,\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop, unsigned flags)\n {\n-\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t header_size = 0;\n-\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n-\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tstruct chunkfile *cf;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name))\n@@ -923,98 +921,35 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n \n-\tcur_chunk = 0;\n-\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n-\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n-\n-\tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (ctx.large_offsets_needed) {\n-\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n-\n-\t\tcur_chunk++;\n-\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n-\t}\n-\n-\tchunk_ids[cur_chunk] = 0;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n-\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n-\t\t\t chunk_offsets[i - 1],\n-\t\t\t chunk_offsets[i]);\n-\n-\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n-\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n-\t\t\t chunk_offsets[i]);\n-\n-\t\thashwrite_be32(f, chunk_ids[i]);\n-\t\thashwrite_be64(f, chunk_offsets[i]);\n-\t}\n-\n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (f->total + f->offset != chunk_offsets[i])\n-\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n-\t\t\t chunk_offsets[i],\n-\t\t\t f->total + f->offset,\n-\t\t\t chunk_ids[i]);\n+\tcf = init_chunkfile(f);\n \n-\t\tswitch (chunk_ids[i]) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twrite_midx_pack_names(f, &ctx);\n-\t\t\t\tbreak;\n+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n+\t\t write_midx_pack_names);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n+\t\t write_midx_oid_fanout);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n+\t\t ctx.entries_nr * the_hash_algo->rawsz,\n+\t\t write_midx_oid_lookup);\n+\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n+\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n+\t\t write_midx_object_offsets);\n \n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twrite_midx_object_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twrite_midx_large_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n-\t\t\t\t chunk_ids[i]);\n-\t\t}\n-\t}\n+\tif (ctx.large_offsets_needed)\n+\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n+\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n+\t\t\twrite_midx_large_offsets);\n \n-\tif (hashfile_total(f) != chunk_offsets[num_chunks])\n-\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t hashfile_total(f),\n-\t\t chunk_offsets[num_chunks]);\n+\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n+\twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n cleanup:\n-- \ngitgitgadget\n\n"},{"id":"416318","messageId":"366eb2afee839feccdfd2244b231d2ad718c76d4.1612535453.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v3 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-05T14:30:47Z","receivedAt":"2021-02-05T22:24:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd the capability to read the table of contents, then pair the chunks\nwith necessary logic using read_chunk_fn pointers. Callers will be added\nin future changes, but the typical outline will be:\n\n 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n 2. call read_table_of_contents().\n 3. for each chunk to parse,\n a. call pair_chunk() to assign a pointer with the chunk position, or\n b. call read_chunk() to run a callback on the chunk start and size.\n 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n\nWe are re-using the anonymous 'struct chunkfile' data, as it is internal\nto the chunk-format API. This gives it essentially two modes: write and\nread. If the same struct instance was used for both reads and writes,\nthen there would be failures.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 46 +++++++++++++++++++++++++++++\n 2 files changed, 126 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 6e0f1900213e..bc9d4caf7276 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -12,6 +12,8 @@ struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n \tchunk_write_fn write_fn;\n+\n+\tconst void *start;\n };\n \n struct chunkfile {\n@@ -89,3 +91,81 @@ int write_chunkfile(struct chunkfile *cf, void *data)\n \n \treturn 0;\n }\n+\n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length)\n+{\n+\tuint32_t chunk_id;\n+\tconst unsigned char *table_of_contents = mfile + toc_offset;\n+\n+\tALLOC_GROW(cf->chunks, toc_length, cf->chunks_alloc);\n+\n+\twhile (toc_length--) {\n+\t\tuint64_t chunk_offset, next_chunk_offset;\n+\n+\t\tchunk_id = get_be32(table_of_contents);\n+\t\tchunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (!chunk_id) {\n+\t\t\terror(_(\"terminating chunk id appears earlier than expected\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\ttable_of_contents += CHUNK_LOOKUP_WIDTH;\n+\t\tnext_chunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (next_chunk_offset < chunk_offset ||\n+\t\t next_chunk_offset > mfile_size - the_hash_algo->rawsz) {\n+\t\t\terror(_(\"improper chunk offset(s) %\"PRIx64\" and %\"PRIx64\"\"),\n+\t\t\t chunk_offset, next_chunk_offset);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n+\t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n+\t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n+\t\tcf->chunks_nr++;\n+\t}\n+\n+\tchunk_id = get_be32(table_of_contents);\n+\tif (chunk_id) {\n+\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int pair_chunk_fn(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size,\n+\t\t\t void *data)\n+{\n+\tconst unsigned char **p = data;\n+\t*p = chunk_start;\n+\treturn 0;\n+}\n+\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t const unsigned char **p)\n+{\n+\treturn read_chunk(cf, chunk_id, pair_chunk_fn, p);\n+}\n+\n+int read_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tif (cf->chunks[i].id == chunk_id)\n+\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n+\t}\n+\n+\treturn CHUNK_NOT_FOUND;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nindex 9a1d770accec..0edcc57db4e7 100644\n--- a/chunk-format.h\n+++ b/chunk-format.h\n@@ -6,6 +6,19 @@\n struct hashfile;\n struct chunkfile;\n \n+/*\n+ * Initialize a 'struct chunkfile' for writing _or_ reading a file\n+ * with the chunk format.\n+ *\n+ * If writing a file, supply a non-NULL 'struct hashfile *' that will\n+ * be used to write.\n+ *\n+ * If reading a file, then supply the memory-mapped data to the\n+ * pair_chunk() or read_chunk() methods, as appropriate.\n+ *\n+ * DO NOT MIX THESE MODES. Use different 'struct chunkfile' instances\n+ * for reading and writing.\n+ */\n struct chunkfile *init_chunkfile(struct hashfile *f);\n void free_chunkfile(struct chunkfile *cf);\n int get_num_chunks(struct chunkfile *cf);\n@@ -16,4 +29,37 @@ void add_chunk(struct chunkfile *cf,\n \t chunk_write_fn fn);\n int write_chunkfile(struct chunkfile *cf, void *data);\n \n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length);\n+\n+#define CHUNK_NOT_FOUND (-2)\n+\n+/*\n+ * Find 'chunk_id' in the given chunkfile and assign the\n+ * given pointer to the position in the mmap'd file where\n+ * that chunk begins.\n+ *\n+ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n+ */\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t const unsigned char **p);\n+\n+typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size, void *data);\n+/*\n+ * Find 'chunk_id' in the given chunkfile and call the\n+ * given chunk_read_fn method with the information for\n+ * that chunk.\n+ *\n+ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n+ */\n+int read_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"416326","messageId":"b00734a3-0a45-463b-55b2-ef8cbb19d76b@gmail.com","threadId":"55053","inReplyTo":"xmqqlfc34eyd.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 09/17] midx: return success/failure in chunk write methods","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-05T11:42:24Z","receivedAt":"2021-02-05T23:02:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/4/2021 5:59 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> In this step I see the same \"f->total + f->offset\" sprinkled in some\n> places, but they will all go away in the \"midx: use chunk-format API\n> in write_midx_internal()\" step, so it is not such a huge deal.\n\nI'll move the ones that are introduced by this patch, but others\nalready exist that also get removed by a later patch. I won't\ntouch those until they are deleted.\n\nThanks,\n-Stolee\n"},{"id":"416327","messageId":"0b851243-e4c0-856e-05c3-eb670739cf6b@gmail.com","threadId":"55053","inReplyTo":"xmqq4kir4bw7.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 16/17] chunk-format: restore duplicate chunk checks","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-05T12:31:47Z","receivedAt":"2021-02-05T23:03:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/4/2021 7:05 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>> \tconst void *start;\n>> +\tunsigned found:1;\n> \n> This defines a .found member ...\n>> ... and no new code touches it.\n> \n> The way duplicate is found is by having a inner loop that checks the\n> IDs of chunks we've seen so far (quadratic, but presumably that\n> would not matter as long as we'd be dealing with just half a dozen\n> chunk types).\n> \n> Is the .found bit used for something else and needs to be added in a\n> different step?\n\nNope. It is just noise that I should have caught and deleted.\n\nThanks,\n-Stolee\n\n"},{"id":"416340","messageId":"CAPx1Gvfc9ryYSLJkfej+ryZdbtOtT==EDXpFeSZCBC6=RV6U4w@mail.gmail.com","threadId":"55053","inReplyTo":"xmqqwnvmz1pv.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2021-02-06T20:35:10Z","receivedAt":"2021-02-06T20:36:37Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Fri, Feb 5, 2021 at 12:41 PM Junio C Hamano wrote:\n> Chris Torek writes:\n> > There are (still) systems with 32-bit size_t (but 64-bit\n> > off_t / file sizes), so ... probably not. Is size_t ever more than\n> > 64 bits these days?\n>\n> Sorry, you lost me. I do not see how it would help to perform the\n> multiplication in uint64_t, when you suspect that size_t is too\n> small, if the final destination of the result of the multiplication\n> is a function argument of type size_t?\n\nNo, you and Derrick Stolee are right, I wasn't looking out far enough\nhere (to the actual function).\n\n(I was wondering though if there are systems where the valid range\nfor size_t could exceed that for off_t. Are there still systems\nusing 32-bit off_t? Sometimes I think there are too many abstracted\ntypes running around here -- how do we know which sizes are big\nenough? There is always uintmax_t, though, and for unsigned\ntypes, ((T)-1) gets you the maximum possible value.)\n\nChris\n"},{"id":"416379","messageId":"20210207195053.GA1016223@szeder.dev","threadId":"55053","inReplyTo":"xmqq8s834c4s.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-02-07T19:50:53Z","receivedAt":"2021-02-07T19:51:48Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Feb 04, 2021 at 04:00:19PM -0800, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n> > From: Derrick Stolee \n> >\n> > When calculating the sizes of certain chunks, we should use 64-bit\n> > multiplication always. This allows us to properly predict the chunk\n> > sizes without risk of overflow.\n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > midx.c | 4 ++--\n> > 1 file changed, 2 insertions(+), 2 deletions(-)\n> \n> This one I find somewhat questionable for multiple reasons.\n> \n> * the fourth parameter of add_chunk() is of size_t, not uint64_t;\n> shouldn't the multiplication be done in type size_t instead?\n> \n> * these mutiplications were introduced in \"midx: use chunk-format\n> API in write_midx_internal()\";\n\nNo, that patch also removes lines like: \n\n- chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n\n- chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n- ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n\nSo those potentially problematic multiplications were already there\nbefore this series, and in fact trace all the way back to the initial\nmidx patch series (commits 0d5b3a5ef7 (midx: write object ids in a \nchunk, 2018-07-12) and 662148c435 (midx: write object offsets,\n2018-07-12)).\n\n> that step should use the\n> arithmetic with cast (if necessary) from the start, no?\n\nAs it fixes a long-standing issue, it should rather be a bugfix patch\nat the beginning of the series.\n\n"},{"id":"416383","messageId":"20210207202042.GC1015009@szeder.dev","threadId":"55053","inReplyTo":"366eb2afee839feccdfd2244b231d2ad718c76d4.1612535453.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 12/17] chunk-format: create read chunk API","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-02-07T20:20:42Z","receivedAt":"2021-02-07T20:21:28Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Feb 05, 2021 at 02:30:47PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Add the capability to read the table of contents, then pair the chunks\n> with necessary logic using read_chunk_fn pointers. Callers will be added\n> in future changes, but the typical outline will be:\n> \n> 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n> 2. call read_table_of_contents().\n\nA reader should call read_table_of_contents(), noted.\n\n> 3. for each chunk to parse,\n> a. call pair_chunk() to assign a pointer with the chunk position, or\n> b. call read_chunk() to run a callback on the chunk start and size.\n> 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n\nHow could a user of this API learn about all chunks present in the\nchunkfile, including unrecognized chunks?\n\n> We are re-using the anonymous 'struct chunkfile' data, as it is internal\n> to the chunk-format API. This gives it essentially two modes: write and\n> read. If the same struct instance was used for both reads and writes,\n> then there would be failures.\n> \n> Helped-by: Junio C Hamano \n> Signed-off-by: Derrick Stolee \n\n> diff --git a/chunk-format.h b/chunk-format.h\n> index 9a1d770accec..0edcc57db4e7 100644\n> --- a/chunk-format.h\n> +++ b/chunk-format.h\n> @@ -6,6 +6,19 @@\n> struct hashfile;\n> struct chunkfile;\n> \n> +/*\n> + * Initialize a 'struct chunkfile' for writing _or_ reading a file\n> + * with the chunk format.\n> + *\n> + * If writing a file, supply a non-NULL 'struct hashfile *' that will\n> + * be used to write.\n> + *\n> + * If reading a file, then supply the memory-mapped data to the\n> + * pair_chunk() or read_chunk() methods, as appropriate.\n\nAnd call read_table_of_contents() in between.\n\n> + *\n> + * DO NOT MIX THESE MODES. Use different 'struct chunkfile' instances\n> + * for reading and writing.\n> + */\n> struct chunkfile *init_chunkfile(struct hashfile *f);\n> void free_chunkfile(struct chunkfile *cf);\n> int get_num_chunks(struct chunkfile *cf);\n> @@ -16,4 +29,37 @@ void add_chunk(struct chunkfile *cf,\n> \t chunk_write_fn fn);\n> int write_chunkfile(struct chunkfile *cf, void *data);\n> \n> +int read_table_of_contents(struct chunkfile *cf,\n> +\t\t\t const unsigned char *mfile,\n> +\t\t\t size_t mfile_size,\n> +\t\t\t uint64_t toc_offset,\n> +\t\t\t int toc_length);\n> +\n> +#define CHUNK_NOT_FOUND (-2)\n> +\n> +/*\n> + * Find 'chunk_id' in the given chunkfile and assign the\n> + * given pointer to the position in the mmap'd file where\n> + * that chunk begins.\n> + *\n> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n> + */\n> +int pair_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t const unsigned char **p);\n> +\n> +typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n> +\t\t\t size_t chunk_size, void *data);\n> +/*\n> + * Find 'chunk_id' in the given chunkfile and call the\n> + * given chunk_read_fn method with the information for\n> + * that chunk.\n> + *\n> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n> + */\n> +int read_chunk(struct chunkfile *cf,\n> +\t uint32_t chunk_id,\n> +\t chunk_read_fn fn,\n> +\t void *data);\n> +\n> #endif\n> -- \n> gitgitgadget\n> \n"},{"id":"416385","messageId":"20210207211359.GD1015009@szeder.dev","threadId":"55053","inReplyTo":"16c37d2370cf4fd5fc309ac6dc8aa6443ffcf3d7.1612535453.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/17] chunk-format: create chunk format write API","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-02-07T21:13:59Z","receivedAt":"2021-02-07T21:15:05Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> diff --git a/chunk-format.c b/chunk-format.c\n> new file mode 100644\n> index 000000000000..6e0f1900213e\n> --- /dev/null\n> +++ b/chunk-format.c\n> @@ -0,0 +1,91 @@\n> +#include \"cache.h\"\n> +#include \"chunk-format.h\"\n> +#include \"csum-file.h\"\n> +#define CHUNK_LOOKUP_WIDTH 12\n\nAs this macro is defined in 'chunk-format.c' it's not part of the\nchunkfile API. However, at the end of this patch series\n'commit-graph.c' still contains:\n\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n\nand uses it in a couple of safety checks (that didn't became part of\nthe common chunkfile module; why?), while 'midx.c' contains:\n\n #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n\nthough it's not used anymore.\n\nI think we should have only one such constant as part of the chunkfile\nAPI; and preferably use the definition from 'midx.c' as it is more\ninformative than yet another magic number.\n\nFurthermore, being called 'CHUNK_LOOKUP_WIDTH', I had to look up the\nplaces where this constant is used to make sure that it indeed means\nwhat I suspect it means. Perhaps CHUNK_TOC_ENTRY_SIZE would be a more\ndescriptive name for this constant.\n\n\nOn a somewhat related note: 'commit-graph.c' and 'midx.c' still\ncontains the constants MAX_NUM_CHUNKS and MIDX_MAX_CHUNKS,\nrespecticely, but neither of them is used anymore.\n\n"},{"id":"416395","messageId":"xmqqczxb6rq3.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"20210207195053.GA1016223@szeder.dev","subject":"Re: [PATCH v2 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-08T05:41:56Z","receivedAt":"2021-02-08T05:42:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> No, that patch also removes lines like: \n>\n> - chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n>\n> - chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n> - ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n\nOK. In other words, the above was replaced in the same patch with\n\n add_chunk(...., U32 * U32);\n\nwhere the called function expects the result of the multiplication\nas size_t in its function prototype. It is a bit sad that U32*U32\nto compute the argument that is to be passed as U64 must be casted\nas (uint64_t)U32*U32 by the caller X-<.\n\nThe original that the above replaced, shown in your quote, is:\n\n U64 = U64 + U32 * U32;\n\nI also wish that the fact that it is added to U64 is sufficient not\nto require the RHS to be written as U64 + (uint64_t) U32 * U32 (in\nother words, the original that was removed was OK without cast).\n\nSad.\n\n"},{"id":"416409","messageId":"f860ef94-567a-371d-7834-8f2221cdf7df@gmail.com","threadId":"55053","inReplyTo":"20210207202042.GC1015009@szeder.dev","subject":"Re: [PATCH v3 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-08T13:35:27Z","receivedAt":"2021-02-08T13:36:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/7/2021 3:20 PM, SZEDER Gábor wrote:\n> On Fri, Feb 05, 2021 at 02:30:47PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> Add the capability to read the table of contents, then pair the chunks\n>> with necessary logic using read_chunk_fn pointers. Callers will be added\n>> in future changes, but the typical outline will be:\n>>\n>> 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n>> 2. call read_table_of_contents().\n> \n> A reader should call read_table_of_contents(), noted.\n> \n>> 3. for each chunk to parse,\n>> a. call pair_chunk() to assign a pointer with the chunk position, or\n>> b. call read_chunk() to run a callback on the chunk start and size.\n>> 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n> \n> How could a user of this API learn about all chunks present in the\n> chunkfile, including unrecognized chunks?\n\nThat could certainly be added (when needed) without modifying the data\nstructures.\n\n>> +/*\n>> + * Initialize a 'struct chunkfile' for writing _or_ reading a file\n>> + * with the chunk format.\n>> + *\n>> + * If writing a file, supply a non-NULL 'struct hashfile *' that will\n>> + * be used to write.\n>> + *\n>> + * If reading a file, then supply the memory-mapped data to the\n>> + * pair_chunk() or read_chunk() methods, as appropriate.\n> \n> And call read_table_of_contents() in between.\n\nYes, you are right.\n \nThanks,\n-Stolee\n"},{"id":"416410","messageId":"96d4f87d-eb6e-faec-1d22-5f6ae307b624@gmail.com","threadId":"55053","inReplyTo":"20210207211359.GD1015009@szeder.dev","subject":"Re: [PATCH v3 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-08T13:44:06Z","receivedAt":"2021-02-08T13:47:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/7/2021 4:13 PM, SZEDER Gábor wrote:\n>> +#define CHUNK_LOOKUP_WIDTH 12\n> \n> As this macro is defined in 'chunk-format.c' it's not part of the\n> chunkfile API. However, at the end of this patch series\n> 'commit-graph.c' still contains:\n> \n> #define GRAPH_CHUNKLOOKUP_WIDTH 12\n> \n> and uses it in a couple of safety checks (that didn't became part of\n> the common chunkfile module; why?),\n\nChunk-based files don't have a minimum size unless we know the header\nsize and a minimum number of required chunks. I suppose that we could\nadd this in the future to further simplify consumers of the API.\n\n> while 'midx.c' contains:\n> \n> #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n> \n> though it's not used anymore.\n> \n> I think we should have only one such constant as part of the chunkfile\n> API; and preferably use the definition from 'midx.c' as it is more\n> informative than yet another magic number.\n> \n> Furthermore, being called 'CHUNK_LOOKUP_WIDTH', I had to look up the\n> places where this constant is used to make sure that it indeed means\n> what I suspect it means. Perhaps CHUNK_TOC_ENTRY_SIZE would be a more\n> descriptive name for this constant.\n\nMore descriptive, for sure.\n\n> On a somewhat related note: 'commit-graph.c' and 'midx.c' still\n> contains the constants MAX_NUM_CHUNKS and MIDX_MAX_CHUNKS,\n> respecticely, but neither of them is used anymore.\n\nThanks. The following patch can be added on top of this series\nto clean up these dangling macros.\n\nThanks,\n-Stolee\n\n--- >8 ---\n\nFrom 839b880ccee65eac63e8b77b12fab6531acc55b0 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Mon, 8 Feb 2021 08:38:47 -0500\nSubject: [PATCH] chunk-format: remove outdated macro constants\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nThe following macros were needed by midx.c and commit-graph.c to handle\ntheir independent implementations of the chunk-based file format, but\nnow the chunk-format API makes them obsolete:\n\n* MAX_NUM_CHUNKS\n* MIDX_MAX_CHUNKS\n* MIX_CHUNKLOOKUP_WIDTH\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 1 -\n midx.c | 2 --\n 2 files changed, 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 32cf5091d2f..3b5a8767269 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,7 +45,6 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 9\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \ndiff --git a/midx.c b/midx.c\nindex 95648a1f368..5c7f2ed2333 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -22,14 +22,12 @@\n #define MIDX_HEADER_SIZE 12\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n \n-#define MIDX_MAX_CHUNKS 5\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n-#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n-- \n2.30.0.vfs.0.0.exp\n\n"},{"id":"416450","messageId":"xmqqh7mm42ml.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"xmqqeehu1f1q.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-08T22:26:58Z","receivedAt":"2021-02-08T22:28:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> The users of pair_chunk() presumably are not ready to (or simply do\n> not want to) process the data immediately by using read_chunk() with\n> callback, but when they get ready to process the data, unlike\n> read_chunk callbacks, they do not get to learn how much they ought\n> to process---all they learn is the address of the beginning of the\n> chunk. I do not see a way to write pair_chunk() users safely to\n> guarantee that they do not overrun at the tail end of the chunk they\n> are processing.\n\nI've read through v3 and found it mostly done, but the above\nquestion still stands. I find it questionable why callers of\npair_chunk() only can learn where a chunk data begins, without\nbeing able to learn how big the region of memory is. IOW, why\ncan we get away without doing something like this? The users\nof pair_chunk() won't even know when they overrun the end of\nthe data the are given without something like this, no?\n\nThanks.\n\n+struct memory_region {\n+\tconst unsigned char *p;\n+\tsize_t sz;\n+};\n+\n static int pair_chunk_fn(const unsigned char *chunk_start,\n size_t chunk_size,\n void *data)\n {\n- const unsigned char **p = data;\n- *p = chunk_start;\n+ struct memory_region *x = data;\n+ x->p = chunk_start;\n+ x->sz = chunk_size;\n return 0;\n }\n\n\n\n int pair_chunk(struct chunkfile *cf,\n uint32_t chunk_id,\n- const unsigned char **p)\n+ const unsigned char **p,\n+ size_t *sz)\n {\n+ int ret;\n+ struct memory_region x;\n= return read_chunk(cf, chunk_id, pair_chunk_fn, &x);\n+ ret = read_chunk(cf, chunk_id, pair_chunk_fn, &x);\n+ *p = x.p;\n+ *sz = x.sz;\n+ return ret;\n }\n\n"},{"id":"416467","messageId":"abd61648-c351-e849-9cf1-ddabff514743@gmail.com","threadId":"55053","inReplyTo":"xmqqh7mm42ml.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T01:33:29Z","receivedAt":"2021-02-09T01:34:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/8/2021 5:26 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>> The users of pair_chunk() presumably are not ready to (or simply do\n>> not want to) process the data immediately by using read_chunk() with\n>> callback, but when they get ready to process the data, unlike\n>> read_chunk callbacks, they do not get to learn how much they ought\n>> to process---all they learn is the address of the beginning of the\n>> chunk. I do not see a way to write pair_chunk() users safely to\n>> guarantee that they do not overrun at the tail end of the chunk they\n>> are processing.\n> \n> I've read through v3 and found it mostly done, but the above\n> question still stands. I find it questionable why callers of\n> pair_chunk() only can learn where a chunk data begins, without\n> being able to learn how big the region of memory is. IOW, why\n> can we get away without doing something like this? The users\n> of pair_chunk() won't even know when they overrun the end of\n> the data the are given without something like this, no?\n\nI guess that the point is that if a caller wants to perform\nlogic on the size, then they should use read_chunk() instead.\nWe have some chunks that check the size is correct upon read,\nbut most chunks do not do this (currently).\n\nIn future series, additional protections could be added, and\nI would expect that to be done by converting callers of\npair_chunk() into callers of read_chunk() with appropriate\ncallback functions.\n\nThanks,\n-Stolee\n"},{"id":"416557","messageId":"xmqqlfbx0xzv.fsf@gitster.c.googlers.com","threadId":"55053","inReplyTo":"abd61648-c351-e849-9cf1-ddabff514743@gmail.com","subject":"Re: [PATCH v2 12/17] chunk-format: create read chunk API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-09T20:47:32Z","receivedAt":"2021-02-09T20:55:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> In future series, additional protections could be added, and\n> I would expect that to be done by converting callers of\n> pair_chunk() into callers of read_chunk() with appropriate\n> callback functions.\n\nI am perfectly OK with leaving the interface as-is, as apparently it\nis sufficient for the current callers. \n\nIt was just it looked insufficient for future callers that do not\nwant to use the callback interface, i.e. learn the pair\nwith a single API call and then handle the data in that region\nitself, instead of preparing a callback function and calling\nread_chunk() on it. As C does not have closures, it gets quickly\ncumbersome if such a caller wants to share pieces of information\nwith the callback function that is given to read_chunk(), but a\ncaller that learns from pair_chunk() and then works on\nthe region of memory itself do nto have to worry about that.\n\n"},{"id":"416759","messageId":"20210211194312.GI1015009@szeder.dev","threadId":"55053","inReplyTo":"96d4f87d-eb6e-faec-1d22-5f6ae307b624@gmail.com","subject":"Re: [PATCH v3 02/17] chunk-format: create chunk format write API","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-02-11T19:43:12Z","receivedAt":"2021-02-11T19:44:14Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Feb 08, 2021 at 08:44:06AM -0500, Derrick Stolee wrote:\n> On 2/7/2021 4:13 PM, SZEDER Gábor wrote:\n> >> +#define CHUNK_LOOKUP_WIDTH 12\n> > \n> > As this macro is defined in 'chunk-format.c' it's not part of the\n> > chunkfile API. However, at the end of this patch series\n> > 'commit-graph.c' still contains:\n> > \n> > #define GRAPH_CHUNKLOOKUP_WIDTH 12\n> > \n> > and uses it in a couple of safety checks (that didn't became part of\n> > the common chunkfile module; why?),\n> \n> Chunk-based files don't have a minimum size unless we know the header\n> size and a minimum number of required chunks. I suppose that we could\n> add this in the future to further simplify consumers of the API.\n> \n> > while 'midx.c' contains:\n> > \n> > #define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n> > \n> > though it's not used anymore.\n> > \n> > I think we should have only one such constant as part of the chunkfile\n> > API; and preferably use the definition from 'midx.c' as it is more\n> > informative than yet another magic number.\n> > \n> > Furthermore, being called 'CHUNK_LOOKUP_WIDTH', I had to look up the\n> > places where this constant is used to make sure that it indeed means\n> > what I suspect it means. Perhaps CHUNK_TOC_ENTRY_SIZE would be a more\n> > descriptive name for this constant.\n> \n> More descriptive, for sure.\n> \n> > On a somewhat related note: 'commit-graph.c' and 'midx.c' still\n> > contains the constants MAX_NUM_CHUNKS and MIDX_MAX_CHUNKS,\n> > respecticely, but neither of them is used anymore.\n> \n> Thanks. The following patch can be added on top of this series\n> to clean up these dangling macros.\n\nIt would be better to squash this into the patches that removed the\nlast uses of each of those constants.\n\nAnd it still leaves the magic number '12' duplicated in\n'commit-graph.c' and 'chunk-format.c'.\n\n> Thanks,\n> -Stolee\n> \n> --- >8 ---\n> \n> From 839b880ccee65eac63e8b77b12fab6531acc55b0 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Mon, 8 Feb 2021 08:38:47 -0500\n> Subject: [PATCH] chunk-format: remove outdated macro constants\n> MIME-Version: 1.0\n> Content-Type: text/plain; charset=UTF-8\n> Content-Transfer-Encoding: 8bit\n> \n> The following macros were needed by midx.c and commit-graph.c to handle\n> their independent implementations of the chunk-based file format, but\n> now the chunk-format API makes them obsolete:\n> \n> * MAX_NUM_CHUNKS\n> * MIDX_MAX_CHUNKS\n> * MIX_CHUNKLOOKUP_WIDTH\n> \n> Reported-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 1 -\n> midx.c | 2 --\n> 2 files changed, 3 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 32cf5091d2f..3b5a8767269 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -45,7 +45,6 @@ void git_test_write_commit_graph_or_die(void)\n> #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n> #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n> #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n> -#define MAX_NUM_CHUNKS 9\n> \n> #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n> \n> diff --git a/midx.c b/midx.c\n> index 95648a1f368..5c7f2ed2333 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -22,14 +22,12 @@\n> #define MIDX_HEADER_SIZE 12\n> #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n> \n> -#define MIDX_MAX_CHUNKS 5\n> #define MIDX_CHUNK_ALIGNMENT 4\n> #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n> #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n> #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n> #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n> #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n> -#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n> #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n> #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n> #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n> -- \n> 2.30.0.vfs.0.0.exp\n> \n"},{"id":"417309","messageId":"f5d44c012ad59c0b46fc6d33177074582e462d90.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 07/17] midx: add pack_perm to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:29Z","receivedAt":"2021-02-18T16:45:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t *pack_perm\" and large_offsets_needed bit\ninto the context.\n\nUpdate write_midx_object_offsets() to match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 40 +++++++++++++++++++++-------------------\n 1 file changed, 21 insertions(+), 19 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4520ef82b91b..cd994e333ecb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -461,6 +461,9 @@ struct write_midx_context {\n \n \tstruct pack_midx_entry *entries;\n \tuint32_t entries_nr;\n+\n+\tuint32_t *pack_perm;\n+\tunsigned large_offsets_needed:1;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -736,27 +739,27 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n-\t\t\t\t\tuint32_t *perm,\n-\t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t\tvoid *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n \n-\t\tif (large_offset_needed && obj->offset >> 31)\n+\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n-\t\telse if (!large_offset_needed && obj->offset >> 32)\n+\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n \t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->offset);\n@@ -805,13 +808,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n-\tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -857,11 +858,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n+\tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n-\t\t\tlarge_offsets_needed = 1;\n+\t\t\tctx.large_offsets_needed = 1;\n \t}\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n@@ -900,13 +902,13 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (i = 0; i < ctx.nr; i++) {\n \t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n@@ -927,7 +929,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n-\tnum_chunks = large_offsets_needed ? 5 : 4;\n+\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n \n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n@@ -954,7 +956,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (large_offsets_needed) {\n+\tif (ctx.large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n \t\tcur_chunk++;\n@@ -1004,7 +1006,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -1039,7 +1041,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfree(ctx.info);\n \tfree(ctx.entries);\n-\tfree(pack_perm);\n+\tfree(ctx.pack_perm);\n \tfree(midx_name);\n \treturn result;\n }\n-- \ngitgitgadget\n\n"},{"id":"417310","messageId":"c18f3738f5d4cb164dfba3a04c0f54bb1b44e013.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 05/17] midx: use context in write_midx_pack_names()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:27Z","receivedAt":"2021-02-18T16:45:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align the write_midx_internal() to use the chunk-format\nAPI, start converting chunk writing methods to match chunk_write_fn. The\nfirst case is to convert write_midx_pack_names() to take \"void *data\".\nWe already have the necessary data in \"struct write_midx_context\", so\nthis conversion is rather mechanical.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 21 ++++++++++-----------\n 1 file changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 561f65a63a5b..88452b044337 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -643,27 +643,26 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t struct pack_info *info,\n-\t\t\t\t uint32_t num_packs)\n+static size_t write_midx_pack_names(struct hashfile *f, void *data)\n {\n+\tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n \tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < num_packs; i++) {\n+\tfor (i = 0; i < ctx->nr; i++) {\n \t\tsize_t writelen;\n \n-\t\tif (info[i].expired)\n+\t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n+\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t info[i - 1].pack_name,\n-\t\t\t info[i].pack_name);\n+\t\t\t ctx->info[i - 1].pack_name,\n+\t\t\t ctx->info[i].pack_name);\n \n-\t\twritelen = strlen(info[i].pack_name) + 1;\n-\t\thashwrite(f, info[i].pack_name, writelen);\n+\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n+\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -990,7 +989,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-- \ngitgitgadget\n\n"},{"id":"417311","messageId":"ddebf2389880e14a332e354f7b1dbc86a3964985.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 03/17] commit-graph: use chunk-format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:25Z","receivedAt":"2021-02-18T16:45:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph write logic is ready to make use of the chunk-format\nwrite API. Each chunk write method is already in the correct prototype.\nWe only need to use the 'struct chunkfile' pointer and the correct API\ncalls.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 119 +++++++++++++++----------------------------------\n 1 file changed, 37 insertions(+), 82 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fae7d1b63931..a889130cc849 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -19,6 +19,7 @@\n #include \"shallow.h\"\n #include \"json-writer.h\"\n #include \"trace2.h\"\n+#include \"chunk-format.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -44,7 +45,6 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 9\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -1758,27 +1758,17 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t void *data);\n-\n-struct chunk_info {\n-\tuint32_t id;\n-\tuint64_t size;\n-\tchunk_write_fn write_fn;\n-};\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n \tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tstruct chunk_info chunks[MAX_NUM_CHUNKS + 1];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n-\tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n+\tstruct chunkfile *cf;\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1824,76 +1814,50 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \t}\n \n-\tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunks[0].size = GRAPH_FANOUT_SIZE;\n-\tchunks[0].write_fn = write_graph_chunk_fanout;\n-\tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunks[1].size = hashsz * ctx->commits.nr;\n-\tchunks[1].write_fn = write_graph_chunk_oids;\n-\tchunks[2].id = GRAPH_CHUNKID_DATA;\n-\tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n-\tchunks[2].write_fn = write_graph_chunk_data;\n+\tcf = init_chunkfile(f);\n+\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDFANOUT, GRAPH_FANOUT_SIZE,\n+\t\t write_graph_chunk_fanout);\n+\tadd_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, hashsz * ctx->commits.nr,\n+\t\t write_graph_chunk_oids);\n+\tadd_chunk(cf, GRAPH_CHUNKID_DATA, (hashsz + 16) * ctx->commits.nr,\n+\t\t write_graph_chunk_data);\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_NO_GDAT, 0))\n \t\tctx->write_generation_data = 0;\n-\tif (ctx->write_generation_data) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_generation_data_overflows) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW;\n-\t\tchunks[num_chunks].size = sizeof(timestamp_t) * ctx->num_generation_data_overflows;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_generation_data_overflow;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_extra_edges) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n-\t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n-\t\tnum_chunks++;\n-\t}\n+\tif (ctx->write_generation_data)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr,\n+\t\t\t write_graph_chunk_generation_data);\n+\tif (ctx->num_generation_data_overflows)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t\t sizeof(timestamp_t) * ctx->num_generation_data_overflows,\n+\t\t\t write_graph_chunk_generation_data_overflow);\n+\tif (ctx->num_extra_edges)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES,\n+\t\t\t 4 * ctx->num_extra_edges,\n+\t\t\t write_graph_chunk_extra_edges);\n \tif (ctx->changed_paths) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n-\t\tnum_chunks++;\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n-\t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n-\t\t\t\t\t + ctx->total_bloom_filter_data_size;\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n-\t\tnum_chunks++;\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1) {\n-\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n-\t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n-\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n-\t\tnum_chunks++;\n-\t}\n-\n-\tchunks[num_chunks].id = 0;\n-\tchunks[num_chunks].size = 0;\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t sizeof(uint32_t) * ctx->commits.nr,\n+\t\t\t write_graph_chunk_bloom_indexes);\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t sizeof(uint32_t) * 3\n+\t\t\t\t+ ctx->total_bloom_filter_data_size,\n+\t\t\t write_graph_chunk_bloom_data);\n+\t}\n+\tif (ctx->num_commit_graphs_after > 1)\n+\t\tadd_chunk(cf, GRAPH_CHUNKID_BASE,\n+\t\t\t hashsz * (ctx->num_commit_graphs_after - 1),\n+\t\t\t write_graph_chunk_base);\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n-\tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\n-\t\tchunk_offset += chunks[i].size;\n-\t}\n-\n \tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n@@ -1905,17 +1869,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tuint64_t start_offset = f->total + f->offset;\n-\n-\t\tif (chunks[i].write_fn(f, ctx))\n-\t\t\treturn -1;\n-\n-\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n-\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n-\t\t\t chunks[i].size, chunks[i].id,\n-\t\t\t f->total + f->offset - start_offset);\n-\t}\n+\twrite_chunkfile(cf, ctx);\n \n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n@@ -1932,6 +1886,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tclose_commit_graph(ctx->r->objects);\n \tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfree_chunkfile(cf);\n \n \tif (ctx->split) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n-- \ngitgitgadget\n\n"},{"id":"417312","messageId":"8fdaae017dab92249b4bd1230212a67c486cb622.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 04/17] midx: rename pack_info to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:26Z","receivedAt":"2021-02-18T16:45:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to streamline our chunk-based file formats, align some of\nthe code structure in write_midx_internal() to be similar to the\npatterns in write_commit_graph_file().\n\nSpecifically, let's create a \"struct write_midx_context\" that can be\nused as a data parameter to abstract function types.\n\nThis change only renames \"struct pack_info\" to \"struct\nwrite_midx_context\" and the names of instances from \"packs\" to \"ctx\". In\nfuture changes, we will expand the data inside \"struct\nwrite_midx_context\" and align our chunk-writing method with the\nchunk-format API.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 130 ++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 65 insertions(+), 65 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 79c282b070d2..561f65a63a5b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -451,7 +451,7 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n-struct pack_list {\n+struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n \tuint32_t alloc;\n@@ -463,37 +463,37 @@ struct pack_list {\n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t const char *file_name, void *data)\n {\n-\tstruct pack_list *packs = (struct pack_list *)data;\n+\tstruct write_midx_context *ctx = data;\n \n \tif (ends_with(file_name, \".idx\")) {\n-\t\tdisplay_progress(packs->progress, ++packs->pack_paths_checked);\n-\t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n+\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n+\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n-\t\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t\t 0);\n+\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->info[packs->nr].p) {\n+\t\tif (!ctx->info[ctx->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n+\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->info[packs->nr].p);\n-\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n+\t\t\tclose_pack(ctx->info[ctx->nr].p);\n+\t\t\tFREE_AND_NULL(ctx->info[ctx->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n-\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n-\t\tpacks->info[packs->nr].expired = 0;\n-\t\tpacks->nr++;\n+\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n+\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n+\t\tctx->info[ctx->nr].expired = 0;\n+\t\tctx->nr++;\n \t}\n }\n \n@@ -801,7 +801,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n-\tstruct pack_list packs;\n+\tstruct write_midx_context ctx = { 0 };\n \tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n@@ -820,40 +820,40 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t midx_name);\n \n \tif (m)\n-\t\tpacks.m = m;\n+\t\tctx.m = m;\n \telse\n-\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n-\n-\tpacks.nr = 0;\n-\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n-\tpacks.info = NULL;\n-\tALLOC_ARRAY(packs.info, packs.alloc);\n-\n-\tif (packs.m) {\n-\t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n-\n-\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n-\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.info[packs.nr].p = NULL;\n-\t\t\tpacks.info[packs.nr].expired = 0;\n-\t\t\tpacks.nr++;\n+\t\tctx.m = load_multi_pack_index(object_dir, 1);\n+\n+\tctx.nr = 0;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.info = NULL;\n+\tALLOC_ARRAY(ctx.info, ctx.alloc);\n+\n+\tif (ctx.m) {\n+\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n+\n+\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n+\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n+\t\t\tctx.info[ctx.nr].p = NULL;\n+\t\t\tctx.info[ctx.nr].expired = 0;\n+\t\t\tctx.nr++;\n \t\t}\n \t}\n \n-\tpacks.pack_paths_checked = 0;\n+\tctx.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n-\t\tpacks.progress = NULL;\n+\t\tctx.progress = NULL;\n \n-\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n-\tstop_progress(&packs.progress);\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n+\tstop_progress(&ctx.progress);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n+\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -862,19 +862,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n-\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n \tif (packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n-\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n \t\t\t\t\t packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n-\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n \t\t\t\t packs_to_drop->items[drop_index].string);\n@@ -882,7 +882,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n \t\t\t} else {\n-\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t\tctx.info[i].expired = 0;\n \t\t\t}\n \t\t}\n \n@@ -898,19 +898,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t *\n \t * pack_perm[old_id] = new_id\n \t */\n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].expired) {\n+\tALLOC_ARRAY(pack_perm, ctx.nr);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired) {\n \t\t\tdropped_packs++;\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tpack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n \t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (!packs.info[i].expired)\n-\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (!ctx.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n@@ -921,19 +921,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n \n-\tif (packs.m)\n-\t\tclose_midx(packs.m);\n+\tif (ctx.m)\n+\t\tclose_midx(ctx.m);\n \n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\tif (packs.nr - dropped_packs == 0) {\n+\tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n+\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -990,7 +990,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, ctx.info, ctx.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -1027,15 +1027,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tcommit_lock_file(&lk);\n \n cleanup:\n-\tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.info[i].p) {\n-\t\t\tclose_pack(packs.info[i].p);\n-\t\t\tfree(packs.info[i].p);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].p) {\n+\t\t\tclose_pack(ctx.info[i].p);\n+\t\t\tfree(ctx.info[i].p);\n \t\t}\n-\t\tfree(packs.info[i].pack_name);\n+\t\tfree(ctx.info[i].pack_name);\n \t}\n \n-\tfree(packs.info);\n+\tfree(ctx.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \ngitgitgadget\n\n"},{"id":"417313","messageId":"2bb2847aad6edba152f1fa249e8e7d0401b0a185.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 06/17] midx: add entries to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:28Z","receivedAt":"2021-02-18T16:46:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"struct pack_midx_entry *entries\" list and its count\ninto the context.\n\nUpdate write_midx_oid_fanout() and write_midx_oid_lookup() to take the\ncontext directly, as these are easy conversions with this new data.\n\nOnly the callers of write_midx_object_offsets() and\nwrite_midx_large_offsets() are updated here, since additional data in\nthe context before those methods can match chunk_write_fn.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 49 ++++++++++++++++++++++++++-----------------------\n 1 file changed, 26 insertions(+), 23 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 88452b044337..4520ef82b91b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -458,6 +458,9 @@ struct write_midx_context {\n \tstruct multi_pack_index *m;\n \tstruct progress *progress;\n \tunsigned pack_paths_checked;\n+\n+\tstruct pack_midx_entry *entries;\n+\tuint32_t entries_nr;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -678,11 +681,11 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n }\n \n static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n-\tstruct pack_midx_entry *last = objects + nr_objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n \tuint32_t count = 0;\n \tuint32_t i;\n \n@@ -706,18 +709,19 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \treturn MIDX_CHUNK_FANOUT_SIZE;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n-\t\t\t\t struct pack_midx_entry *objects,\n-\t\t\t\t uint32_t nr_objects)\n+static size_t write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects;\n+\tstruct write_midx_context *ctx = data;\n+\tunsigned char hash_len = the_hash_algo->rawsz;\n+\tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n \tsize_t written = 0;\n \n-\tfor (i = 0; i < nr_objects; i++) {\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (i < nr_objects - 1) {\n+\t\tif (i < ctx->entries_nr - 1) {\n \t\t\tstruct pack_midx_entry *next = list;\n \t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n \t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n@@ -805,8 +809,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t nr_entries, num_large_offsets = 0;\n-\tstruct pack_midx_entry *entries = NULL;\n+\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n@@ -852,12 +855,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &nr_entries);\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n \n-\tfor (i = 0; i < nr_entries; i++) {\n-\t\tif (entries[i].offset > 0x7fffffff)\n+\tfor (i = 0; i < ctx.entries_nr; i++) {\n+\t\tif (ctx.entries[i].offset > 0x7fffffff)\n \t\t\tnum_large_offsets++;\n-\t\tif (entries[i].offset > 0xffffffff)\n+\t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n@@ -947,10 +950,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * the_hash_algo->rawsz;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n \n \tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + nr_entries * MIDX_CHUNK_OFFSET_WIDTH;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n \tif (large_offsets_needed) {\n \t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n \n@@ -993,19 +996,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, the_hash_algo->rawsz, entries, nr_entries);\n+\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, entries, nr_entries);\n+\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1035,7 +1038,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tfree(ctx.info);\n-\tfree(entries);\n+\tfree(ctx.entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n \treturn result;\n-- \ngitgitgadget\n\n"},{"id":"417314","messageId":"da1fcc68357609cfb379a2487c139665efbba59f.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 02/17] chunk-format: create chunk format write API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:24Z","receivedAt":"2021-02-18T16:46:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of combining the logic from the commit-graph and\nmulti-pack-index file formats, create a new chunk-format API. Use a\n'struct chunkfile' pointer to keep track of data that has been\nregistered for writes. This struct is anonymous outside of\nchunk-format.c to ensure no user attempts to interfere with the data.\n\nThe next change will use this API in commit-graph.c, but the general\napproach is:\n\n 1. initialize the chunkfile with init_chunkfile(f).\n 2. add chunks in the intended writing order with add_chunk().\n 3. write any header information to the hashfile f.\n 4. write the chunkfile data using write_chunkfile().\n 5. free the chunkfile struct using free_chunkfile().\n\nHelped-by: Taylor Blau \nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n chunk-format.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 21 ++++++++++++\n 3 files changed, 112 insertions(+)\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\ndiff --git a/Makefile b/Makefile\nindex 7b64106930a6..50a7663841e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -854,6 +854,7 @@ LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += chdir-notify.o\n LIB_OBJS += checkout.o\n+LIB_OBJS += chunk-format.o\n LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\ndiff --git a/chunk-format.c b/chunk-format.c\nnew file mode 100644\nindex 000000000000..6c9b52b70c10\n--- /dev/null\n+++ b/chunk-format.c\n@@ -0,0 +1,90 @@\n+#include \"cache.h\"\n+#include \"chunk-format.h\"\n+#include \"csum-file.h\"\n+\n+/*\n+ * When writing a chunk-based file format, collect the chunks in\n+ * an array of chunk_info structs. The size stores the _expected_\n+ * amount of data that will be written by write_fn.\n+ */\n+struct chunk_info {\n+\tuint32_t id;\n+\tuint64_t size;\n+\tchunk_write_fn write_fn;\n+};\n+\n+struct chunkfile {\n+\tstruct hashfile *f;\n+\n+\tstruct chunk_info *chunks;\n+\tsize_t chunks_nr;\n+\tsize_t chunks_alloc;\n+};\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f)\n+{\n+\tstruct chunkfile *cf = xcalloc(1, sizeof(*cf));\n+\tcf->f = f;\n+\treturn cf;\n+}\n+\n+void free_chunkfile(struct chunkfile *cf)\n+{\n+\tif (!cf)\n+\t\treturn;\n+\tfree(cf->chunks);\n+\tfree(cf);\n+}\n+\n+int get_num_chunks(struct chunkfile *cf)\n+{\n+\treturn cf->chunks_nr;\n+}\n+\n+void add_chunk(struct chunkfile *cf,\n+\t uint32_t id,\n+\t size_t size,\n+\t chunk_write_fn fn)\n+{\n+\tALLOC_GROW(cf->chunks, cf->chunks_nr + 1, cf->chunks_alloc);\n+\n+\tcf->chunks[cf->chunks_nr].id = id;\n+\tcf->chunks[cf->chunks_nr].write_fn = fn;\n+\tcf->chunks[cf->chunks_nr].size = size;\n+\tcf->chunks_nr++;\n+}\n+\n+int write_chunkfile(struct chunkfile *cf, void *data)\n+{\n+\tint i;\n+\tuint64_t cur_offset = hashfile_total(cf->f);\n+\n+\t/* Add the table of contents to the current offset */\n+\tcur_offset += (cf->chunks_nr + 1) * CHUNK_TOC_ENTRY_SIZE;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n+\t\thashwrite_be64(cf->f, cur_offset);\n+\n+\t\tcur_offset += cf->chunks[i].size;\n+\t}\n+\n+\t/* Trailing entry marks the end of the chunks */\n+\thashwrite_be32(cf->f, 0);\n+\thashwrite_be64(cf->f, cur_offset);\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\toff_t start_offset = hashfile_total(cf->f);\n+\t\tint result = cf->chunks[i].write_fn(cf->f, data);\n+\n+\t\tif (result)\n+\t\t\treturn result;\n+\n+\t\tif (hashfile_total(cf->f) - start_offset != cf->chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t cf->chunks[i].size, cf->chunks[i].id,\n+\t\t\t hashfile_total(cf->f) - start_offset);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nnew file mode 100644\nindex 000000000000..ce598b66d9f8\n--- /dev/null\n+++ b/chunk-format.h\n@@ -0,0 +1,21 @@\n+#ifndef CHUNK_FORMAT_H\n+#define CHUNK_FORMAT_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct hashfile;\n+struct chunkfile;\n+\n+#define CHUNK_TOC_ENTRY_SIZE (sizeof(uint32_t) + sizeof(uint64_t))\n+\n+struct chunkfile *init_chunkfile(struct hashfile *f);\n+void free_chunkfile(struct chunkfile *cf);\n+int get_num_chunks(struct chunkfile *cf);\n+typedef int (*chunk_write_fn)(struct hashfile *f, void *data);\n+void add_chunk(struct chunkfile *cf,\n+\t uint32_t id,\n+\t size_t size,\n+\t chunk_write_fn fn);\n+int write_chunkfile(struct chunkfile *cf, void *data);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"417315","messageId":"243dcec9436853ff8d1bf2580e76ab909b7cb324.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 01/17] commit-graph: anonymize data in chunk_write_fn","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:23Z","receivedAt":"2021-02-18T16:48:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for creating an API around file formats using chunks and\ntables of contents, prepare the commit-graph write code to use\nprototypes that will match this new API.\n\nSpecifically, convert chunk_write_fn to take a \"void *data\" parameter\ninstead of the commit-graph-specific \"struct write_commit_graph_context\"\npointer.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++----------\n 1 file changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f3bde2ad95a1..fae7d1b63931 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1040,8 +1040,9 @@ struct write_commit_graph_context {\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n \n@@ -1066,8 +1067,9 @@ static int write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static int write_graph_chunk_oids(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n@@ -1085,8 +1087,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static int write_graph_chunk_data(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n@@ -1187,8 +1190,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i, num_generation_data_overflows = 0;\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n@@ -1208,8 +1212,9 @@ static int write_graph_chunk_generation_data(struct hashfile *f,\n }\n \n static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n-\t\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint i;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1226,8 +1231,9 @@ static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n }\n \n static int write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n@@ -1280,8 +1286,9 @@ static int write_graph_chunk_extra_edges(struct hashfile *f,\n }\n \n static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t cur_pos = 0;\n@@ -1315,8 +1322,9 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n }\n \n static int write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\tstruct write_commit_graph_context *ctx)\n+\t\t\t\t\tvoid *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \n@@ -1737,8 +1745,9 @@ static int write_graph_chunk_base_1(struct hashfile *f,\n }\n \n static int write_graph_chunk_base(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+\t\t\t\t void *data)\n {\n+\tstruct write_commit_graph_context *ctx = data;\n \tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n \n \tif (num != ctx->num_commit_graphs_after - 1) {\n@@ -1750,7 +1759,7 @@ static int write_graph_chunk_base(struct hashfile *f,\n }\n \n typedef int (*chunk_write_fn)(struct hashfile *f,\n-\t\t\t struct write_commit_graph_context *ctx);\n+\t\t\t void *data);\n \n struct chunk_info {\n \tuint32_t id;\n-- \ngitgitgadget\n\n"},{"id":"417316","messageId":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v3.git.1612535452.gitgitgadget@gmail.com","subject":"[PATCH v4 00/17] Refactor chunk-format into an API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:22Z","receivedAt":"2021-02-18T16:48:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a restart on the topic previously submitted [1] but dropped because\nak/corrected-commit-date was still in progress. This version is based on\nthat branch.\n\n[1]\nhttps://lore.kernel.org/git/pull.804.git.1607012215.gitgitgadget@gmail.com/\n\nThis version also changes the approach to use a more dynamic interaction\nwith a struct chunkfile pointer. This idea is credited to Taylor Blau [2],\nbut I started again from scratch. I also go further to make struct chunkfile\nanonymous to API consumers. It is defined only in chunk-format.c, which\nshould hopefully deter future users from interacting with that data\ndirectly.\n\n[2] https://lore.kernel.org/git/X8%2FI%2FRzXZksio+ri@nand.local/\n\nThis combined API is beneficial to reduce duplicated logic. Or rather, to\nensure that similar file formats have similar protections against bad data.\nThe multi-pack-index code did not have as many guards as the commit-graph\ncode did, but now they both share a common base that checks for things like\nduplicate chunks or offsets outside the size of the file.\n\nHere are some stats for the end-to-end change:\n\n * 570 insertions(+), 456 deletions(-).\n * commit-graph.c: 107 insertions(+), 192 deletions(-)\n * midx.c: 164 insertions(+), 260 deletions(-)\n\nWhile there is an overall increase to the code size, the consumers do get\nsmaller. Boilerplate things like abstracting method to match chunk_write_fn\nand chunk_read_fn make up a lot of these insertions. The \"interesting\" code\ngets a lot smaller and cleaner.\n\n\nUpdates in V4\n=============\n\n * Out-of-date macros in commit-graph.c and midx.c are removed in their\n appropriate patches.\n * Documentation around the read API is improved.\n\n\nUpdates in V3\n=============\n\n * API methods use better types and changed their order to match internal\n data more closely.\n\n * Use hashfile_total() instead of internal data values.\n\n * The implementation of pair_chunk() uses read_chunk().\n\n * init_chunkfile() has an in-code doc comment warning against using the\n same struct chunkfile for reads and writes.\n\n * More multiplications are correctly cast in midx.c.\n\n * The chunk-format technical docs are expanded.\n\n\nUpdates in V2\n=============\n\n * The method pair_chunk() now automatically sets a pointer while\n read_chunk() uses the callback. This greatly reduces the code size.\n\n * Pointer casts are now implicit instead of explicit.\n\n * Extra care is taken to not overflow when verifying chunk sizes on write.\n\nThanks, -Stolee\n\nDerrick Stolee (17):\n commit-graph: anonymize data in chunk_write_fn\n chunk-format: create chunk format write API\n commit-graph: use chunk-format write API\n midx: rename pack_info to write_midx_context\n midx: use context in write_midx_pack_names()\n midx: add entries to write_midx_context\n midx: add pack_perm to write_midx_context\n midx: add num_large_offsets to write_midx_context\n midx: return success/failure in chunk write methods\n midx: drop chunk progress during write\n midx: use chunk-format API in write_midx_internal()\n chunk-format: create read chunk API\n commit-graph: use chunk-format read API\n midx: use chunk-format read API\n midx: use 64-bit multiplication for chunk sizes\n chunk-format: restore duplicate chunk checks\n chunk-format: add technical docs\n\n Documentation/technical/chunk-format.txt | 116 +++++\n .../technical/commit-graph-format.txt | 3 +\n Documentation/technical/pack-format.txt | 3 +\n Makefile | 1 +\n chunk-format.c | 179 ++++++++\n chunk-format.h | 68 +++\n commit-graph.c | 305 +++++-------\n midx.c | 433 +++++++-----------\n t/t5318-commit-graph.sh | 2 +-\n t/t5319-multi-pack-index.sh | 6 +-\n 10 files changed, 652 insertions(+), 464 deletions(-)\n create mode 100644 Documentation/technical/chunk-format.txt\n create mode 100644 chunk-format.c\n create mode 100644 chunk-format.h\n\n\nbase-commit: 5a3b130cad0d5c770f766e3af6d32b41766374c0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-848%2Fderrickstolee%2Fchunk-format%2Frefactor-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-848/derrickstolee/chunk-format/refactor-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/848\n\nRange-diff vs v3:\n\n 1: 243dcec94368 = 1: 243dcec94368 commit-graph: anonymize data in chunk_write_fn\n 2: 16c37d2370cf ! 2: da1fcc683576 chunk-format: create chunk format write API\n @@ chunk-format.c (new)\n +#include \"cache.h\"\n +#include \"chunk-format.h\"\n +#include \"csum-file.h\"\n -+#define CHUNK_LOOKUP_WIDTH 12\n +\n +/*\n + * When writing a chunk-based file format, collect the chunks in\n @@ chunk-format.c (new)\n +\tuint64_t cur_offset = hashfile_total(cf->f);\n +\n +\t/* Add the table of contents to the current offset */\n -+\tcur_offset += (cf->chunks_nr + 1) * CHUNK_LOOKUP_WIDTH;\n ++\tcur_offset += (cf->chunks_nr + 1) * CHUNK_TOC_ENTRY_SIZE;\n +\n +\tfor (i = 0; i < cf->chunks_nr; i++) {\n +\t\thashwrite_be32(cf->f, cf->chunks[i].id);\n @@ chunk-format.h (new)\n +struct hashfile;\n +struct chunkfile;\n +\n ++#define CHUNK_TOC_ENTRY_SIZE (sizeof(uint32_t) + sizeof(uint64_t))\n ++\n +struct chunkfile *init_chunkfile(struct hashfile *f);\n +void free_chunkfile(struct chunkfile *cf);\n +int get_num_chunks(struct chunkfile *cf);\n 3: e549e24d79af ! 3: ddebf2389880 commit-graph: use chunk-format write API\n @@ commit-graph.c\n \n void git_test_write_commit_graph_or_die(void)\n {\n +@@ commit-graph.c: void git_test_write_commit_graph_or_die(void)\n + #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n + #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n + #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n +-#define MAX_NUM_CHUNKS 9\n + \n + #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n + \n @@ commit-graph.c: static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n 4: 66ff49ed9309 = 4: 8fdaae017dab midx: rename pack_info to write_midx_context\n 5: 1d7484c0cffa = 5: c18f3738f5d4 midx: use context in write_midx_pack_names()\n 6: ea0e7d40e537 = 6: 2bb2847aad6e midx: add entries to write_midx_context\n 7: b283a38fb775 = 7: f5d44c012ad5 midx: add pack_perm to write_midx_context\n 8: e7064512ab7f = 8: f5baadf276ff midx: add num_large_offsets to write_midx_context\n 9: 7aa3242e15b7 = 9: fd008449fc89 midx: return success/failure in chunk write methods\n 10: 70f68c95e479 = 10: 60e77e451800 midx: drop chunk progress during write\n 11: 787cd7f18d2e ! 11: 0b95aa7e152f midx: use chunk-format API in write_midx_internal()\n @@ midx.c\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n +@@\n + #define MIDX_HEADER_SIZE 12\n + #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n + \n +-#define MIDX_MAX_CHUNKS 5\n + #define MIDX_CHUNK_ALIGNMENT 4\n + #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n + #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n @@ midx.c: static int write_midx_large_offsets(struct hashfile *f,\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop, unsigned flags)\n 12: 366eb2afee83 ! 12: 3e0dbc45ce7f chunk-format: create read chunk API\n @@ chunk-format.c: int write_chunkfile(struct chunkfile *cf, void *data)\n +\t\t\treturn 1;\n +\t\t}\n +\n -+\t\ttable_of_contents += CHUNK_LOOKUP_WIDTH;\n ++\t\ttable_of_contents += CHUNK_TOC_ENTRY_SIZE;\n +\t\tnext_chunk_offset = get_be64(table_of_contents + 4);\n +\n +\t\tif (next_chunk_offset < chunk_offset ||\n @@ chunk-format.c: int write_chunkfile(struct chunkfile *cf, void *data)\n +}\n \n ## chunk-format.h ##\n -@@\n - struct hashfile;\n - struct chunkfile;\n +@@ chunk-format.h: struct chunkfile;\n + \n + #define CHUNK_TOC_ENTRY_SIZE (sizeof(uint32_t) + sizeof(uint64_t))\n \n +/*\n + * Initialize a 'struct chunkfile' for writing _or_ reading a file\n @@ chunk-format.h\n + * If writing a file, supply a non-NULL 'struct hashfile *' that will\n + * be used to write.\n + *\n -+ * If reading a file, then supply the memory-mapped data to the\n ++ * If reading a file, use a NULL 'struct hashfile *' and then call\n ++ * read_table_of_contents(). Supply the memory-mapped data to the\n + * pair_chunk() or read_chunk() methods, as appropriate.\n + *\n + * DO NOT MIX THESE MODES. Use different 'struct chunkfile' instances\n 13: 7838ad32e2e0 ! 13: 87f441fc5e15 commit-graph: use chunk-format read API\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## commit-graph.c ##\n +@@ commit-graph.c: void git_test_write_commit_graph_or_die(void)\n + \n + #define GRAPH_HEADER_SIZE 8\n + #define GRAPH_FANOUT_SIZE (4 * 256)\n +-#define GRAPH_CHUNKLOOKUP_WIDTH 12\n +-#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n ++#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * CHUNK_TOC_ENTRY_SIZE \\\n + \t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n + \n + #define CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW (1ULL << 31)\n @@ commit-graph.c: static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n @@ commit-graph.c: static int verify_commit_graph_lite(struct commit_graph *g)\n \n \tif (!graph_map)\n \t\treturn NULL;\n +@@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n + \tgraph->data_len = graph_size;\n + \n + \tif (graph_size < GRAPH_HEADER_SIZE +\n +-\t\t\t (graph->num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH +\n ++\t\t\t (graph->num_chunks + 1) * CHUNK_TOC_ENTRY_SIZE +\n + \t\t\t GRAPH_FANOUT_SIZE + the_hash_algo->rawsz) {\n + \t\terror(_(\"commit-graph file is too small to hold %u chunks\"),\n + \t\t graph->num_chunks);\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n 14: 6bddd9e63b9b ! 14: 64010031db0e midx: use chunk-format read API\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## midx.c ##\n +@@\n + #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n + #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n + #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n +-#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n + #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n + #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n + #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n @@ midx.c: static char *get_midx_filename(const char *object_dir)\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n 15: 3cd97f389f1f = 15: 16205d8f8c40 midx: use 64-bit multiplication for chunk sizes\n 16: b9a1bddf615f = 16: 0e58218be418 chunk-format: restore duplicate chunk checks\n 17: 4c7d751f1e39 = 17: 84bf6506dc12 chunk-format: add technical docs\n\n-- \ngitgitgadget\n"},{"id":"417317","messageId":"f5baadf276ff08549530f0cadc1658705fa46a51.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 08/17] midx: add num_large_offsets to write_midx_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:30Z","receivedAt":"2021-02-18T16:49:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an effort to align write_midx_internal() with the chunk-format API,\ncontinue to group necessary data into \"struct write_midx_context\". This\nchange collects the \"uint32_t num_large_offsets\" into the context. With\nthis new data, write_midx_large_offsets() now matches the\nchunk_write_fn type.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 17 ++++++++++-------\n 1 file changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex cd994e333ecb..5be081f229ad 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -464,6 +464,7 @@ struct write_midx_context {\n \n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n+\tuint32_t num_large_offsets;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -772,11 +773,14 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \treturn written;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_offset,\n-\t\t\t\t struct pack_midx_entry *objects, uint32_t nr_objects)\n+static size_t write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n-\tstruct pack_midx_entry *list = objects, *end = objects + nr_objects;\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n \tsize_t written = 0;\n+\tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n \t\tstruct pack_midx_entry *obj;\n@@ -811,7 +815,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tuint32_t num_large_offsets = 0;\n \tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n@@ -861,7 +864,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n \t\tif (ctx.entries[i].offset > 0x7fffffff)\n-\t\t\tnum_large_offsets++;\n+\t\t\tctx.num_large_offsets++;\n \t\tif (ctx.entries[i].offset > 0xffffffff)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n@@ -961,7 +964,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\tcur_chunk++;\n \t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n+\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n \t}\n \n \tchunk_ids[cur_chunk] = 0;\n@@ -1010,7 +1013,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, num_large_offsets, ctx.entries, ctx.entries_nr);\n+\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n-- \ngitgitgadget\n\n"},{"id":"417318","messageId":"fd008449fc89602411ddd3d607b4e5e10608b7e4.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 09/17] midx: return success/failure in chunk write methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:31Z","receivedAt":"2021-02-18T16:49:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHistorically, the chunk-writing methods in midx.c have returned the\namount of data written so the writer method could compare this with the\ntable of contents. This presents with some interesting issues:\n\n1. If a chunk writing method has a bug that miscalculates the written\n bytes, then we can satisfy the table of contents without actually\n writing the right amount of data to the hashfile. The commit-graph\n writing code checks the hashfile struct directly for a more robust\n verification.\n\n2. There is no way for a chunk writing method to gracefully fail.\n Returning an int presents an opportunity to fail without a die().\n\n3. The current pattern doesn't match chunk_write_fn type exactly, so we\n cannot share code with commit-graph.c\n\nFor these reasons, convert the midx chunk writer methods to return an\n'int'. Since none of them fail at the moment, they all return 0.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 63 +++++++++++++++++++++++++---------------------------------\n 1 file changed, 27 insertions(+), 36 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 5be081f229ad..c92a6c47be01 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -650,7 +650,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \treturn deduplicated_entries;\n }\n \n-static size_t write_midx_pack_names(struct hashfile *f, void *data)\n+static int write_midx_pack_names(struct hashfile *f, void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tuint32_t i;\n@@ -678,14 +678,13 @@ static size_t write_midx_pack_names(struct hashfile *f, void *data)\n \tif (i < MIDX_CHUNK_ALIGNMENT) {\n \t\tmemset(padding, 0, sizeof(padding));\n \t\thashwrite(f, padding, i);\n-\t\twritten += i;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n@@ -710,17 +709,16 @@ static size_t write_midx_oid_fanout(struct hashfile *f,\n \t\tlist = next;\n \t}\n \n-\treturn MIDX_CHUNK_FANOUT_SIZE;\n+\treturn 0;\n }\n \n-static size_t write_midx_oid_lookup(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tunsigned char hash_len = the_hash_algo->rawsz;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -734,19 +732,17 @@ static size_t write_midx_oid_lookup(struct hashfile *f,\n \t\t}\n \n \t\thashwrite(f, obj->oid.hash, (int)hash_len);\n-\t\twritten += hash_len;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_object_offsets(struct hashfile *f,\n-\t\t\t\t\tvoid *data)\n+static int write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tuint32_t i, nr_large_offset = 0;\n-\tsize_t written = 0;\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n@@ -766,20 +762,17 @@ static size_t write_midx_object_offsets(struct hashfile *f,\n \t\t\t obj->offset);\n \t\telse\n \t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n-\n-\t\twritten += MIDX_CHUNK_OFFSET_WIDTH;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n-static size_t write_midx_large_offsets(struct hashfile *f,\n-\t\t\t\t void *data)\n+static int write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t void *data)\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct pack_midx_entry *list = ctx->entries;\n \tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n-\tsize_t written = 0;\n \tuint32_t nr_large_offset = ctx->num_large_offsets;\n \n \twhile (nr_large_offset) {\n@@ -795,12 +788,12 @@ static size_t write_midx_large_offsets(struct hashfile *f,\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\twritten += hashwrite_be64(f, offset);\n+\t\thashwrite_be64(f, offset);\n \n \t\tnr_large_offset--;\n \t}\n \n-\treturn written;\n+\treturn 0;\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n@@ -812,7 +805,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t written = 0;\n+\tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tstruct progress *progress = NULL;\n@@ -940,10 +933,10 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\tgoto cleanup;\n \t}\n \n-\twritten = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n+\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n@@ -981,39 +974,37 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \t\thashwrite_be32(f, chunk_ids[i]);\n \t\thashwrite_be64(f, chunk_offsets[i]);\n-\n-\t\twritten += MIDX_CHUNKLOOKUP_WIDTH;\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (written != chunk_offsets[i])\n+\t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n \t\t\t chunk_offsets[i],\n-\t\t\t written,\n+\t\t\t f->total + f->offset,\n \t\t\t chunk_ids[i]);\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, &ctx);\n+\t\t\t\twrite_midx_pack_names(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twritten += write_midx_oid_fanout(f, &ctx);\n+\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twritten += write_midx_oid_lookup(f, &ctx);\n+\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, &ctx);\n+\t\t\t\twrite_midx_object_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twritten += write_midx_large_offsets(f, &ctx);\n+\t\t\t\twrite_midx_large_offsets(f, &ctx);\n \t\t\t\tbreak;\n \n \t\t\tdefault:\n@@ -1025,9 +1016,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \tstop_progress(&progress);\n \n-\tif (written != chunk_offsets[num_chunks])\n+\tif (hashfile_total(f) != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t written,\n+\t\t hashfile_total(f),\n \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n-- \ngitgitgadget\n\n"},{"id":"417319","messageId":"60e77e45180052cd859893cadc9dbe39ada5d06b.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 10/17] midx: drop chunk progress during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:32Z","receivedAt":"2021-02-18T16:51:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMost expensive operations in write_midx_internal() use the context\nstruct's progress member, and these indicate the process of the\nexpensive operations within the chunk writing methods. However, there is\na competing progress struct that counts the progress over all chunks.\nThis is not very helpful compared to the others, so drop it.\n\nThis also reduces our barriers to combining the chunk writing code with\nchunk-format.c.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex c92a6c47be01..4f4aa351e60e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -808,7 +808,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tuint64_t header_size = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n-\tstruct progress *progress = NULL;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -976,9 +975,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\thashwrite_be64(f, chunk_offsets[i]);\n \t}\n \n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n-\t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (f->total + f->offset != chunk_offsets[i])\n \t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n@@ -1011,10 +1007,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n \t\t\t\t chunk_ids[i]);\n \t\t}\n-\n-\t\tdisplay_progress(progress, i + 1);\n \t}\n-\tstop_progress(&progress);\n \n \tif (hashfile_total(f) != chunk_offsets[num_chunks])\n \t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-- \ngitgitgadget\n\n"},{"id":"417320","messageId":"0b95aa7e152f01b4c33c79858d8979ffcc4eb420.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 11/17] midx: use chunk-format API in write_midx_internal()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:33Z","receivedAt":"2021-02-18T16:52:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-format API allows writing the table of contents and all chunks\nusing the anonymous 'struct chunkfile' type. We only need to convert our\nlocal chunk logic to this API for the multi-pack-index writes to share\nthat logic with the commit-graph file writes.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 106 +++++++++++----------------------------------------------\n 1 file changed, 20 insertions(+), 86 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4f4aa351e60e..d2fd9c10feea 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -11,6 +11,7 @@\n #include \"trace2.h\"\n #include \"run-command.h\"\n #include \"repository.h\"\n+#include \"chunk-format.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -21,7 +22,6 @@\n #define MIDX_HEADER_SIZE 12\n #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n \n-#define MIDX_MAX_CHUNKS 5\n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -799,18 +799,15 @@ static int write_midx_large_offsets(struct hashfile *f,\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop, unsigned flags)\n {\n-\tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n-\tuint64_t header_size = 0;\n-\tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n-\tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tstruct chunkfile *cf;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name))\n@@ -923,98 +920,35 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n \n-\tcur_chunk = 0;\n-\tnum_chunks = ctx.large_offsets_needed ? 5 : 4;\n-\n \tif (ctx.nr - dropped_packs == 0) {\n \t\terror(_(\"no pack files to index.\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n-\theader_size = write_midx_header(f, num_chunks, ctx.nr - dropped_packs);\n-\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n-\tchunk_offsets[cur_chunk] = header_size + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + MIDX_CHUNK_FANOUT_SIZE;\n-\n-\tcur_chunk++;\n-\tchunk_ids[cur_chunk] = MIDX_CHUNKID_OBJECTOFFSETS;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * the_hash_algo->rawsz;\n-\n-\tcur_chunk++;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH;\n-\tif (ctx.large_offsets_needed) {\n-\t\tchunk_ids[cur_chunk] = MIDX_CHUNKID_LARGEOFFSETS;\n-\n-\t\tcur_chunk++;\n-\t\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] +\n-\t\t\t\t\t ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH;\n-\t}\n-\n-\tchunk_ids[cur_chunk] = 0;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tif (i && chunk_offsets[i] < chunk_offsets[i - 1])\n-\t\t\tBUG(\"incorrect chunk offsets: %\"PRIu64\" before %\"PRIu64,\n-\t\t\t chunk_offsets[i - 1],\n-\t\t\t chunk_offsets[i]);\n-\n-\t\tif (chunk_offsets[i] % MIDX_CHUNK_ALIGNMENT)\n-\t\t\tBUG(\"chunk offset %\"PRIu64\" is not properly aligned\",\n-\t\t\t chunk_offsets[i]);\n-\n-\t\thashwrite_be32(f, chunk_ids[i]);\n-\t\thashwrite_be64(f, chunk_offsets[i]);\n-\t}\n-\n-\tfor (i = 0; i < num_chunks; i++) {\n-\t\tif (f->total + f->offset != chunk_offsets[i])\n-\t\t\tBUG(\"incorrect chunk offset (%\"PRIu64\" != %\"PRIu64\") for chunk id %\"PRIx32,\n-\t\t\t chunk_offsets[i],\n-\t\t\t f->total + f->offset,\n-\t\t\t chunk_ids[i]);\n+\tcf = init_chunkfile(f);\n \n-\t\tswitch (chunk_ids[i]) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twrite_midx_pack_names(f, &ctx);\n-\t\t\t\tbreak;\n+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n+\t\t write_midx_pack_names);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n+\t\t write_midx_oid_fanout);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n+\t\t ctx.entries_nr * the_hash_algo->rawsz,\n+\t\t write_midx_oid_lookup);\n+\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n+\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n+\t\t write_midx_object_offsets);\n \n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\twrite_midx_oid_fanout(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\twrite_midx_oid_lookup(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twrite_midx_object_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\twrite_midx_large_offsets(f, &ctx);\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\tBUG(\"trying to write unknown chunk id %\"PRIx32,\n-\t\t\t\t chunk_ids[i]);\n-\t\t}\n-\t}\n+\tif (ctx.large_offsets_needed)\n+\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n+\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n+\t\t\twrite_midx_large_offsets);\n \n-\tif (hashfile_total(f) != chunk_offsets[num_chunks])\n-\t\tBUG(\"incorrect final offset %\"PRIu64\" != %\"PRIu64,\n-\t\t hashfile_total(f),\n-\t\t chunk_offsets[num_chunks]);\n+\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n+\twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n cleanup:\n-- \ngitgitgadget\n\n"},{"id":"417321","messageId":"3e0dbc45ce7ff5f1bda63ae8b8b45343b790417b.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 12/17] chunk-format: create read chunk API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:34Z","receivedAt":"2021-02-18T16:52:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd the capability to read the table of contents, then pair the chunks\nwith necessary logic using read_chunk_fn pointers. Callers will be added\nin future changes, but the typical outline will be:\n\n 1. initialize a 'struct chunkfile' with init_chunkfile(NULL).\n 2. call read_table_of_contents().\n 3. for each chunk to parse,\n a. call pair_chunk() to assign a pointer with the chunk position, or\n b. call read_chunk() to run a callback on the chunk start and size.\n 4. call free_chunkfile() to clear the 'struct chunkfile' data.\n\nWe are re-using the anonymous 'struct chunkfile' data, as it is internal\nto the chunk-format API. This gives it essentially two modes: write and\nread. If the same struct instance was used for both reads and writes,\nthen there would be failures.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++\n chunk-format.h | 47 +++++++++++++++++++++++++++++\n 2 files changed, 127 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 6c9b52b70c10..2c1fecf1c3e5 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -11,6 +11,8 @@ struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n \tchunk_write_fn write_fn;\n+\n+\tconst void *start;\n };\n \n struct chunkfile {\n@@ -88,3 +90,81 @@ int write_chunkfile(struct chunkfile *cf, void *data)\n \n \treturn 0;\n }\n+\n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length)\n+{\n+\tuint32_t chunk_id;\n+\tconst unsigned char *table_of_contents = mfile + toc_offset;\n+\n+\tALLOC_GROW(cf->chunks, toc_length, cf->chunks_alloc);\n+\n+\twhile (toc_length--) {\n+\t\tuint64_t chunk_offset, next_chunk_offset;\n+\n+\t\tchunk_id = get_be32(table_of_contents);\n+\t\tchunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (!chunk_id) {\n+\t\t\terror(_(\"terminating chunk id appears earlier than expected\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\ttable_of_contents += CHUNK_TOC_ENTRY_SIZE;\n+\t\tnext_chunk_offset = get_be64(table_of_contents + 4);\n+\n+\t\tif (next_chunk_offset < chunk_offset ||\n+\t\t next_chunk_offset > mfile_size - the_hash_algo->rawsz) {\n+\t\t\terror(_(\"improper chunk offset(s) %\"PRIx64\" and %\"PRIx64\"\"),\n+\t\t\t chunk_offset, next_chunk_offset);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n+\t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n+\t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n+\t\tcf->chunks_nr++;\n+\t}\n+\n+\tchunk_id = get_be32(table_of_contents);\n+\tif (chunk_id) {\n+\t\terror(_(\"final chunk has non-zero id %\"PRIx32\"\"), chunk_id);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int pair_chunk_fn(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size,\n+\t\t\t void *data)\n+{\n+\tconst unsigned char **p = data;\n+\t*p = chunk_start;\n+\treturn 0;\n+}\n+\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t const unsigned char **p)\n+{\n+\treturn read_chunk(cf, chunk_id, pair_chunk_fn, p);\n+}\n+\n+int read_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\tif (cf->chunks[i].id == chunk_id)\n+\t\t\treturn fn(cf->chunks[i].start, cf->chunks[i].size, data);\n+\t}\n+\n+\treturn CHUNK_NOT_FOUND;\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nindex ce598b66d9f8..9ccbe0037792 100644\n--- a/chunk-format.h\n+++ b/chunk-format.h\n@@ -8,6 +8,20 @@ struct chunkfile;\n \n #define CHUNK_TOC_ENTRY_SIZE (sizeof(uint32_t) + sizeof(uint64_t))\n \n+/*\n+ * Initialize a 'struct chunkfile' for writing _or_ reading a file\n+ * with the chunk format.\n+ *\n+ * If writing a file, supply a non-NULL 'struct hashfile *' that will\n+ * be used to write.\n+ *\n+ * If reading a file, use a NULL 'struct hashfile *' and then call\n+ * read_table_of_contents(). Supply the memory-mapped data to the\n+ * pair_chunk() or read_chunk() methods, as appropriate.\n+ *\n+ * DO NOT MIX THESE MODES. Use different 'struct chunkfile' instances\n+ * for reading and writing.\n+ */\n struct chunkfile *init_chunkfile(struct hashfile *f);\n void free_chunkfile(struct chunkfile *cf);\n int get_num_chunks(struct chunkfile *cf);\n@@ -18,4 +32,37 @@ void add_chunk(struct chunkfile *cf,\n \t chunk_write_fn fn);\n int write_chunkfile(struct chunkfile *cf, void *data);\n \n+int read_table_of_contents(struct chunkfile *cf,\n+\t\t\t const unsigned char *mfile,\n+\t\t\t size_t mfile_size,\n+\t\t\t uint64_t toc_offset,\n+\t\t\t int toc_length);\n+\n+#define CHUNK_NOT_FOUND (-2)\n+\n+/*\n+ * Find 'chunk_id' in the given chunkfile and assign the\n+ * given pointer to the position in the mmap'd file where\n+ * that chunk begins.\n+ *\n+ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n+ */\n+int pair_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t const unsigned char **p);\n+\n+typedef int (*chunk_read_fn)(const unsigned char *chunk_start,\n+\t\t\t size_t chunk_size, void *data);\n+/*\n+ * Find 'chunk_id' in the given chunkfile and call the\n+ * given chunk_read_fn method with the information for\n+ * that chunk.\n+ *\n+ * Returns CHUNK_NOT_FOUND if the chunk does not exist.\n+ */\n+int read_chunk(struct chunkfile *cf,\n+\t uint32_t chunk_id,\n+\t chunk_read_fn fn,\n+\t void *data);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"417322","messageId":"84bf6506dc12163b37f46192b3742c8fb234322f.1613657260.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 17/17] chunk-format: add technical docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:39Z","receivedAt":"2021-02-18T16:54:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe chunk-based file format is now an API in the code, but we should\nalso take time to document it as a file format. Specifically, it matches\nthe CHUNK LOOKUP sections of the commit-graph and multi-pack-index\nfiles, but there are some commonalities that should be grouped in this\ndocument.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/chunk-format.txt | 116 ++++++++++++++++++\n .../technical/commit-graph-format.txt | 3 +\n Documentation/technical/pack-format.txt | 3 +\n 3 files changed, 122 insertions(+)\n create mode 100644 Documentation/technical/chunk-format.txt\n\ndiff --git a/Documentation/technical/chunk-format.txt b/Documentation/technical/chunk-format.txt\nnew file mode 100644\nindex 000000000000..593614fcedab\n--- /dev/null\n+++ b/Documentation/technical/chunk-format.txt\n@@ -0,0 +1,116 @@\n+Chunk-based file formats\n+========================\n+\n+Some file formats in Git use a common concept of \"chunks\" to describe\n+sections of the file. This allows structured access to a large file by\n+scanning a small \"table of contents\" for the remaining data. This common\n+format is used by the `commit-graph` and `multi-pack-index` files. See\n+link:technical/pack-format.html[the `multi-pack-index` format] and\n+link:technical/commit-graph-format.html[the `commit-graph` format] for\n+how they use the chunks to describe structured data.\n+\n+A chunk-based file format begins with some header information custom to\n+that format. That header should include enough information to identify\n+the file type, format version, and number of chunks in the file. From this\n+information, that file can determine the start of the chunk-based region.\n+\n+The chunk-based region starts with a table of contents describing where\n+each chunk starts and ends. This consists of (C+1) rows of 12 bytes each,\n+where C is the number of chunks. Consider the following table:\n+\n+ | Chunk ID (4 bytes) | Chunk Offset (8 bytes) |\n+ |--------------------|------------------------|\n+ | ID[0] | OFFSET[0] |\n+ | ... | ... |\n+ | ID[C] | OFFSET[C] |\n+ | 0x0000 | OFFSET[C+1] |\n+\n+Each row consists of a 4-byte chunk identifier (ID) and an 8-byte offset.\n+Each integer is stored in network-byte order.\n+\n+The chunk identifier `ID[i]` is a label for the data stored within this\n+fill from `OFFSET[i]` (inclusive) to `OFFSET[i+1]` (exclusive). Thus, the\n+size of the `i`th chunk is equal to the difference between `OFFSET[i+1]`\n+and `OFFSET[i]`. This requires that the chunk data appears contiguously\n+in the same order as the table of contents.\n+\n+The final entry in the table of contents must be four zero bytes. This\n+confirms that the table of contents is ending and provides the offset for\n+the end of the chunk-based data.\n+\n+Note: The chunk-based format expects that the file contains _at least_ a\n+trailing hash after `OFFSET[C+1]`.\n+\n+Functions for working with chunk-based file formats are declared in\n+`chunk-format.h`. Using these methods provide extra checks that assist\n+developers when creating new file formats.\n+\n+Writing chunk-based file formats\n+--------------------------------\n+\n+To write a chunk-based file format, create a `struct chunkfile` by\n+calling `init_chunkfile()` and pass a `struct hashfile` pointer. The\n+caller is responsible for opening the `hashfile` and writing header\n+information so the file format is identifiable before the chunk-based\n+format begins.\n+\n+Then, call `add_chunk()` for each chunk that is intended for write. This\n+populates the `chunkfile` with information about the order and size of\n+each chunk to write. Provide a `chunk_write_fn` function pointer to\n+perform the write of the chunk data upon request.\n+\n+Call `write_chunkfile()` to write the table of contents to the `hashfile`\n+followed by each of the chunks. This will verify that each chunk wrote\n+the expected amount of data so the table of contents is correct.\n+\n+Finally, call `free_chunkfile()` to clear the `struct chunkfile` data. The\n+caller is responsible for finalizing the `hashfile` by writing the trailing\n+hash and closing the file.\n+\n+Reading chunk-based file formats\n+--------------------------------\n+\n+To read a chunk-based file format, the file must be opened as a\n+memory-mapped region. The chunk-format API expects that the entire file\n+is mapped as a contiguous memory region.\n+\n+Initialize a `struct chunkfile` pointer with `init_chunkfile(NULL)`.\n+\n+After reading the header information from the beginning of the file,\n+including the chunk count, call `read_table_of_contents()` to populate\n+the `struct chunkfile` with the list of chunks, their offsets, and their\n+sizes.\n+\n+Extract the data information for each chunk using `pair_chunk()` or\n+`read_chunk()`:\n+\n+* `pair_chunk()` assigns a given pointer with the location inside the\n+ memory-mapped file corresponding to that chunk's offset. If the chunk\n+ does not exist, then the pointer is not modified.\n+\n+* `read_chunk()` takes a `chunk_read_fn` function pointer and calls it\n+ with the appropriate initial pointer and size information. The function\n+ is not called if the chunk does not exist. Use this method to read chunks\n+ if you need to perform immediate parsing or if you need to execute logic\n+ based on the size of the chunk.\n+\n+After calling these methods, call `free_chunkfile()` to clear the\n+`struct chunkfile` data. This will not close the memory-mapped region.\n+Callers are expected to own that data for the timeframe the pointers into\n+the region are needed.\n+\n+Examples\n+--------\n+\n+These file formats use the chunk-format API, and can be used as examples\n+for future formats:\n+\n+* *commit-graph:* see `write_commit_graph_file()` and `parse_commit_graph()`\n+ in `commit-graph.c` for how the chunk-format API is used to write and\n+ parse the commit-graph file format documented in\n+ link:technical/commit-graph-format.html[the commit-graph file format].\n+\n+* *multi-pack-index:* see `write_midx_internal()` and `load_multi_pack_index()`\n+ in `midx.c` for how the chunk-format API is used to write and\n+ parse the multi-pack-index file format documented in\n+ link:technical/pack-format.html[the multi-pack-index file format].\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex b6658eff1882..87971c27dd73 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -61,6 +61,9 @@ CHUNK LOOKUP:\n the length using the next chunk position if necessary.) Each chunk\n ID appears at most once.\n \n+ The CHUNK LOOKUP matches the table of contents from\n+ link:technical/chunk-format.html[the chunk-based file format].\n+\n The remaining data in the body is described one chunk at a time, and\n these chunks may be given in any order. Chunks are required unless\n otherwise specified.\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex f96b2e605f34..2fb1e60d29ec 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -301,6 +301,9 @@ CHUNK LOOKUP:\n \t (Chunks are provided in file-order, so you can infer the length\n \t using the next chunk position if necessary.)\n \n+\tThe CHUNK LOOKUP matches the table of contents from\n+\tlink:technical/chunk-format.html[the chunk-based file format].\n+\n \tThe remaining data in the body is described one chunk at a time, and\n \tthese chunks may be given in any order. Chunks are required unless\n \totherwise specified.\n-- \ngitgitgadget\n"},{"id":"417323","messageId":"16205d8f8c408ffe17da366812483e1575700d7d.1613657260.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 15/17] midx: use 64-bit multiplication for chunk sizes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:37Z","receivedAt":"2021-02-18T16:54:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen calculating the sizes of certain chunks, we should use 64-bit\nmultiplication always. This allows us to properly predict the chunk\nsizes without risk of overflow.\n\nOther possible overflows were discovered by evaluating each\nmultiplication in midx.c and ensuring that at least one side of the\noperator was of type size_t or off_t.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 11 ++++++-----\n 1 file changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d7ea0d1375fa..5c7f2ed23330 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -244,7 +244,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n \n-\toffset_data = m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH;\n+\toffset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;\n \toffset32 = get_be32(offset_data + sizeof(uint32_t));\n \n \tif (m->chunk_large_offsets && offset32 & MIDX_LARGE_OFFSET_NEEDED) {\n@@ -260,7 +260,8 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \n static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n-\treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n+\treturn get_be32(m->chunk_object_offsets +\n+\t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n }\n \n static int nth_midxed_pack_entry(struct repository *r,\n@@ -912,15 +913,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n \t\t write_midx_oid_fanout);\n \tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n-\t\t ctx.entries_nr * the_hash_algo->rawsz,\n+\t\t (size_t)ctx.entries_nr * the_hash_algo->rawsz,\n \t\t write_midx_oid_lookup);\n \tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n-\t\t ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n+\t\t (size_t)ctx.entries_nr * MIDX_CHUNK_OFFSET_WIDTH,\n \t\t write_midx_object_offsets);\n \n \tif (ctx.large_offsets_needed)\n \t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n-\t\t\tctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n+\t\t\t(size_t)ctx.num_large_offsets * MIDX_CHUNK_LARGE_OFFSET_WIDTH,\n \t\t\twrite_midx_large_offsets);\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n-- \ngitgitgadget\n\n"},{"id":"417324","messageId":"64010031db0e44c8c040fa1a53f568574dd5a940.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 14/17] midx: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:36Z","receivedAt":"2021-02-18T16:54:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). In particular, we\ncan use the return value of pair_chunk() to generate an error when a\nrequired chunk is missing.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 73 +++++++++++++------------------------\n t/t5319-multi-pack-index.sh | 6 +--\n 2 files changed, 29 insertions(+), 50 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d2fd9c10feea..d7ea0d1375fa 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -28,7 +28,6 @@\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n-#define MIDX_CHUNKLOOKUP_WIDTH (sizeof(uint32_t) + sizeof(uint64_t))\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n@@ -53,6 +52,19 @@ static char *get_midx_filename(const char *object_dir)\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+static int midx_read_oid_fanout(const unsigned char *chunk_start,\n+\t\t\t\tsize_t chunk_size, void *data)\n+{\n+\tstruct multi_pack_index *m = data;\n+\tm->chunk_oid_fanout = (uint32_t *)chunk_start;\n+\n+\tif (chunk_size != 4 * 256) {\n+\t\terror(_(\"multi-pack-index OID fanout is of the wrong size\"));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\n@@ -64,6 +76,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \tchar *midx_name = get_midx_filename(object_dir);\n \tuint32_t i;\n \tconst char *cur_pack_name;\n+\tstruct chunkfile *cf = NULL;\n \n \tfd = git_open(midx_name);\n \n@@ -113,58 +126,23 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n-\tfor (i = 0; i < m->num_chunks; i++) {\n-\t\tuint32_t chunk_id = get_be32(m->data + MIDX_HEADER_SIZE +\n-\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\t\tuint64_t chunk_offset = get_be64(m->data + MIDX_HEADER_SIZE + 4 +\n-\t\t\t\t\t\t MIDX_CHUNKLOOKUP_WIDTH * i);\n-\n-\t\tif (chunk_offset >= m->data_len)\n-\t\t\tdie(_(\"invalid chunk offset (too large)\"));\n-\n-\t\tswitch (chunk_id) {\n-\t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\tm->chunk_pack_names = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n-\t\t\t\tm->chunk_oid_fanout = (uint32_t *)(m->data + chunk_offset);\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OIDLOOKUP:\n-\t\t\t\tm->chunk_oid_lookup = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\tm->chunk_object_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n-\t\t\t\tm->chunk_large_offsets = m->data + chunk_offset;\n-\t\t\t\tbreak;\n-\n-\t\t\tcase 0:\n-\t\t\t\tdie(_(\"terminating multi-pack-index chunk id appears earlier than expected\"));\n-\t\t\t\tbreak;\n-\n-\t\t\tdefault:\n-\t\t\t\t/*\n-\t\t\t\t * Do nothing on unrecognized chunks, allowing future\n-\t\t\t\t * extensions to add optional chunks.\n-\t\t\t\t */\n-\t\t\t\tbreak;\n-\t\t}\n-\t}\n+\tcf = init_chunkfile(NULL);\n \n-\tif (!m->chunk_pack_names)\n+\tif (read_table_of_contents(cf, m->data, midx_size,\n+\t\t\t\t MIDX_HEADER_SIZE, m->num_chunks))\n+\t\tgoto cleanup_fail;\n+\n+\tif (pair_chunk(cf, MIDX_CHUNKID_PACKNAMES, &m->chunk_pack_names) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required pack-name chunk\"));\n-\tif (!m->chunk_oid_fanout)\n+\tif (read_chunk(cf, MIDX_CHUNKID_OIDFANOUT, midx_read_oid_fanout, m) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID fanout chunk\"));\n-\tif (!m->chunk_oid_lookup)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OIDLOOKUP, &m->chunk_oid_lookup) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required OID lookup chunk\"));\n-\tif (!m->chunk_object_offsets)\n+\tif (pair_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS, &m->chunk_object_offsets) == CHUNK_NOT_FOUND)\n \t\tdie(_(\"multi-pack-index missing required object offsets chunk\"));\n \n+\tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets);\n+\n \tm->num_objects = ntohl(m->chunk_oid_fanout[255]);\n \n \tm->pack_names = xcalloc(m->num_packs, sizeof(*m->pack_names));\n@@ -190,6 +168,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n cleanup_fail:\n \tfree(m);\n \tfree(midx_name);\n+\tfree(cf);\n \tif (midx_map)\n \t\tmunmap(midx_map, midx_size);\n \tif (0 <= fd)\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a94f..ad4e878b65b8 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -314,12 +314,12 @@ test_expect_success 'verify bad OID version' '\n \n test_expect_success 'verify truncated chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\01\" $objdir \\\n-\t\t\"missing required\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'verify extended chunk count' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_COUNT \"\\07\" $objdir \\\n-\t\t\"terminating multi-pack-index chunk id appears earlier than expected\"\n+\t\t\"terminating chunk id appears earlier than expected\"\n '\n \n test_expect_success 'verify missing required chunk' '\n@@ -329,7 +329,7 @@ test_expect_success 'verify missing required chunk' '\n \n test_expect_success 'verify invalid chunk offset' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_CHUNK_OFFSET \"\\01\" $objdir \\\n-\t\t\"invalid chunk offset (too large)\"\n+\t\t\"improper chunk offset(s)\"\n '\n \n test_expect_success 'verify packnames out of order' '\n-- \ngitgitgadget\n\n"},{"id":"417325","messageId":"87f441fc5e154c95494f3f1219fb0e0c7d9b3244.1613657259.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 13/17] commit-graph: use chunk-format read API","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:35Z","receivedAt":"2021-02-18T16:54:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of parsing the table of contents directly, use the chunk-format\nAPI methods read_table_of_contents() and pair_chunk(). While the current\nimplementation loses the duplicate-chunk detection, that will be added\nin a future change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 159 ++++++++++++++--------------------------\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 55 insertions(+), 106 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a889130cc849..76514a879e88 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -59,8 +59,7 @@ void git_test_write_commit_graph_or_die(void)\n \n #define GRAPH_HEADER_SIZE 8\n #define GRAPH_FANOUT_SIZE (4 * 256)\n-#define GRAPH_CHUNKLOOKUP_WIDTH 12\n-#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n+#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * CHUNK_TOC_ENTRY_SIZE \\\n \t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n #define CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW (1ULL << 31)\n@@ -298,15 +297,43 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n+static int graph_read_oid_lookup(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = data;\n+\tg->chunk_oid_lookup = chunk_start;\n+\tg->num_commits = chunk_size / g->hash_len;\n+\treturn 0;\n+}\n+\n+static int graph_read_bloom_data(const unsigned char *chunk_start,\n+\t\t\t\t size_t chunk_size, void *data)\n+{\n+\tstruct commit_graph *g = data;\n+\tuint32_t hash_version;\n+\tg->chunk_bloom_data = chunk_start;\n+\thash_version = get_be32(chunk_start);\n+\n+\tif (hash_version != 1)\n+\t\treturn 0;\n+\n+\tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n+\tg->bloom_filter_settings->hash_version = hash_version;\n+\tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n+\tg->bloom_filter_settings->bits_per_entry = get_be32(chunk_start + 8);\n+\tg->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n+\n+\treturn 0;\n+}\n+\n struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n-\tconst unsigned char *data, *chunk_lookup;\n-\tuint32_t i;\n+\tconst unsigned char *data;\n \tstruct commit_graph *graph;\n-\tuint64_t next_chunk_offset;\n \tuint32_t graph_signature;\n \tunsigned char graph_version, hash_version;\n+\tstruct chunkfile *cf = NULL;\n \n \tif (!graph_map)\n \t\treturn NULL;\n@@ -347,7 +374,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \tgraph->data_len = graph_size;\n \n \tif (graph_size < GRAPH_HEADER_SIZE +\n-\t\t\t (graph->num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH +\n+\t\t\t (graph->num_chunks + 1) * CHUNK_TOC_ENTRY_SIZE +\n \t\t\t GRAPH_FANOUT_SIZE + the_hash_algo->rawsz) {\n \t\terror(_(\"commit-graph file is too small to hold %u chunks\"),\n \t\t graph->num_chunks);\n@@ -355,108 +382,28 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n-\tchunk_lookup = data + 8;\n-\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\tfor (i = 0; i < graph->num_chunks; i++) {\n-\t\tuint32_t chunk_id;\n-\t\tuint64_t chunk_offset = next_chunk_offset;\n-\t\tint chunk_repeated = 0;\n-\n-\t\tchunk_id = get_be32(chunk_lookup + 0);\n-\n-\t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n-\t\tnext_chunk_offset = get_be64(chunk_lookup + 4);\n-\n-\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n-\t\t\terror(_(\"commit-graph improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n-\t\t\t (uint32_t)chunk_offset);\n-\t\t\tgoto free_and_return;\n-\t\t}\n-\n-\t\tswitch (chunk_id) {\n-\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n-\t\t\tif (graph->chunk_oid_fanout)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_oid_fanout = (uint32_t*)(data + chunk_offset);\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n-\t\t\tif (graph->chunk_oid_lookup)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n-\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n-\t\t\t\tgraph->num_commits = (next_chunk_offset - chunk_offset)\n-\t\t\t\t\t\t / graph->hash_len;\n-\t\t\t}\n-\t\t\tbreak;\n+\tcf = init_chunkfile(NULL);\n \n-\t\tcase GRAPH_CHUNKID_DATA:\n-\t\t\tif (graph->chunk_commit_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA:\n-\t\t\tif (graph->chunk_generation_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW:\n-\t\t\tif (graph->chunk_generation_data_overflow)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_generation_data_overflow = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_EXTRAEDGES:\n-\t\t\tif (graph->chunk_extra_edges)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BASE:\n-\t\t\tif (graph->chunk_base_graphs)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n-\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n-\t\t\tif (graph->chunk_bloom_indexes)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n-\t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n-\t\t\tbreak;\n-\n-\t\tcase GRAPH_CHUNKID_BLOOMDATA:\n-\t\t\tif (graph->chunk_bloom_data)\n-\t\t\t\tchunk_repeated = 1;\n-\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n-\t\t\t\tuint32_t hash_version;\n-\t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n-\t\t\t\thash_version = get_be32(data + chunk_offset);\n-\n-\t\t\t\tif (hash_version != 1)\n-\t\t\t\t\tbreak;\n+\tif (read_table_of_contents(cf, graph->data, graph_size,\n+\t\t\t\t GRAPH_HEADER_SIZE, graph->num_chunks))\n+\t\tgoto free_and_return;\n \n-\t\t\t\tgraph->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n-\t\t\t\tgraph->bloom_filter_settings->hash_version = hash_version;\n-\t\t\t\tgraph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n-\t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n-\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n-\t\t\t}\n-\t\t\tbreak;\n-\t\t}\n+\tpair_chunk(cf, GRAPH_CHUNKID_OIDFANOUT,\n+\t\t (const unsigned char **)&graph->chunk_oid_fanout);\n+\tread_chunk(cf, GRAPH_CHUNKID_OIDLOOKUP, graph_read_oid_lookup, graph);\n+\tpair_chunk(cf, GRAPH_CHUNKID_DATA, &graph->chunk_commit_data);\n+\tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, &graph->chunk_extra_edges);\n+\tpair_chunk(cf, GRAPH_CHUNKID_BASE, &graph->chunk_base_graphs);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t &graph->chunk_generation_data);\n+\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n+\t\t &graph->chunk_generation_data_overflow);\n \n-\t\tif (chunk_repeated) {\n-\t\t\terror(_(\"commit-graph chunk id %08x appears multiple times\"), chunk_id);\n-\t\t\tgoto free_and_return;\n-\t\t}\n+\tif (r->settings.commit_graph_read_changed_paths) {\n+\t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n+\t\t\t &graph->chunk_bloom_indexes);\n+\t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -473,9 +420,11 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \tif (verify_commit_graph_lite(graph))\n \t\tgoto free_and_return;\n \n+\tfree_chunkfile(cf);\n \treturn graph;\n \n free_and_return:\n+\tfree_chunkfile(cf);\n \tfree(graph->bloom_filter_settings);\n \tfree(graph);\n \treturn NULL;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex fa27df579a57..c7da741284e5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -564,7 +564,7 @@ test_expect_success 'detect bad hash version' '\n \n test_expect_success 'detect low chunk count' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\01\" \\\n-\t\t\"missing the .* chunk\"\n+\t\t\"final chunk has non-zero id\"\n '\n \n test_expect_success 'detect missing OID fanout chunk' '\n-- \ngitgitgadget\n\n"},{"id":"417326","messageId":"0e58218be4185b428aec48a9169134517330bcde.1613657260.git.gitgitgadget@gmail.com","threadId":"55053","inReplyTo":"pull.848.v4.git.1613657259.gitgitgadget@gmail.com","subject":"[PATCH v4 16/17] chunk-format: restore duplicate chunk checks","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-18T14:07:38Z","receivedAt":"2021-02-18T16:54:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore refactoring into the chunk-format API, the commit-graph parsing\nlogic included checks for duplicate chunks. It is unlikely that we would\ndesire a chunk-based file format that allows duplicate chunk IDs in the\ntable of contents, so add duplicate checks into\nread_table_of_contents().\n\nSigned-off-by: Derrick Stolee \n---\n chunk-format.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 2c1fecf1c3e5..da191e59a29d 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -97,6 +97,7 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\t uint64_t toc_offset,\n \t\t\t int toc_length)\n {\n+\tint i;\n \tuint32_t chunk_id;\n \tconst unsigned char *table_of_contents = mfile + toc_offset;\n \n@@ -123,6 +124,14 @@ int read_table_of_contents(struct chunkfile *cf,\n \t\t\treturn -1;\n \t\t}\n \n+\t\tfor (i = 0; i < cf->chunks_nr; i++) {\n+\t\t\tif (cf->chunks[i].id == chunk_id) {\n+\t\t\t\terror(_(\"duplicate chunk ID %\"PRIx32\" found\"),\n+\t\t\t\t\tchunk_id);\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t}\n+\n \t\tcf->chunks[cf->chunks_nr].id = chunk_id;\n \t\tcf->chunks[cf->chunks_nr].start = mfile + chunk_offset;\n \t\tcf->chunks[cf->chunks_nr].size = next_chunk_offset - chunk_offset;\n-- \ngitgitgadget\n\n"},{"id":"417359","messageId":"xmqqft1tdp4o.fsf@gitster.g","threadId":"55053","inReplyTo":"84bf6506dc12163b37f46192b3742c8fb234322f.1613657260.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 17/17] chunk-format: add technical docs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-18T21:47:51Z","receivedAt":"2021-02-18T21:48:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +Chunk-based file formats\n> +========================\n> +\n> +Some file formats in Git use a common concept of \"chunks\" to describe\n> +sections of the file. This allows structured access to a large file by\n> +scanning a small \"table of contents\" for the remaining data. This common\n> +format is used by the `commit-graph` and `multi-pack-index` files. See\n> +link:technical/pack-format.html[the `multi-pack-index` format] and\n> +link:technical/commit-graph-format.html[the `commit-graph` format] for\n> +how they use the chunks to describe structured data.\n\nI've read the doc added here to the end; well written and easy to\nunderstand.\n\nI wonder how/if well reftable files fit in the scheme, or if it\ndoesn't, should the chunk file format API be updated to accomodate\nit (or the other way around)?\n\n> +Extract the data information for each chunk using `pair_chunk()` or\n> +`read_chunk()`:\n> +\n> +* `pair_chunk()` assigns a given pointer with the location inside the\n> + memory-mapped file corresponding to that chunk's offset. If the chunk\n> + does not exist, then the pointer is not modified.\n\nI think it is worth adding:\n\n The caller is expected to know where the returned chunk ends by\n some out-of-band means, as this function only gives the offset\n but not the size, unlike the read_chunk() function.\n\n> +* `read_chunk()` takes a `chunk_read_fn` function pointer and calls it\n> + with the appropriate initial pointer and size information. The function\n> + is not called if the chunk does not exist. Use this method to read chunks\n> + if you need to perform immediate parsing or if you need to execute logic\n> + based on the size of the chunk.\n> +\n> +After calling these methods, call `free_chunkfile()` to clear the\n> +`struct chunkfile` data. This will not close the memory-mapped region.\n> +Callers are expected to own that data for the timeframe the pointers into\n> +the region are needed.\n\nThanks.\n"},{"id":"417402","messageId":"e16a548f-8070-e4be-b0b1-f36a259d0df1@gmail.com","threadId":"55053","inReplyTo":"xmqqft1tdp4o.fsf@gitster.g","subject":"Re: [PATCH v4 17/17] chunk-format: add technical docs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-19T12:42:07Z","receivedAt":"2021-02-19T12:42:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/18/2021 4:47 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +Chunk-based file formats\n>> +========================\n>> +\n>> +Some file formats in Git use a common concept of \"chunks\" to describe\n>> +sections of the file. This allows structured access to a large file by\n>> +scanning a small \"table of contents\" for the remaining data. This common\n>> +format is used by the `commit-graph` and `multi-pack-index` files. See\n>> +link:technical/pack-format.html[the `multi-pack-index` format] and\n>> +link:technical/commit-graph-format.html[the `commit-graph` format] for\n>> +how they use the chunks to describe structured data.\n> \n> I've read the doc added here to the end; well written and easy to\n> understand.\n> \n> I wonder how/if well reftable files fit in the scheme, or if it\n> doesn't, should the chunk file format API be updated to accomodate\n> it (or the other way around)?\n\nI'm not sure that reftable can work with this format, especially with\nits design to do most updates as append-only (IIUC). And to change the\nformat to work with the chunk format would violate the compatibility\nwith the JGit version. I would be interested if something like the\npacked-refs file could use a minor update, but only if there is a\nrealistic benefit to using chunks over the current format.\n\nThe files that are on my radar for adopting a new file format using the\nchunk-format API are:\n\n* reachability bitmaps: using a similar approach to the commit-graph,\n we could avoid parsing the entire file before checking if a specific\n commit has a bitmap. (Requires a commit lookup chunk, a bitmap data\n chunk, and an offset chunk to connect them.)\n\n* index v5: I'm trying to collect a bunch of information about how to\n update the index for better compression, and the chunk-based approach\n can provide some fixed-width columns that can vary in length depending\n on the required data (presenting the interesting behavior from v2 and v3,\n along with possible approaches previously presented as a potential v5).\n The paths could be presented as a chunk, giving the interesting options\n between v2/3 and v4 (prefix compression). I haven't even started the\n actual work here, but I've been thinking about it a lot. I'll have time\n next month to start prototyping.\n\nAre there other interesting files that could use a new version here?\nWhat other pain points are known to experts in the area?\n\n>> +Extract the data information for each chunk using `pair_chunk()` or\n>> +`read_chunk()`:\n>> +\n>> +* `pair_chunk()` assigns a given pointer with the location inside the\n>> + memory-mapped file corresponding to that chunk's offset. If the chunk\n>> + does not exist, then the pointer is not modified.\n> \n> I think it is worth adding:\n> \n> The caller is expected to know where the returned chunk ends by\n> some out-of-band means, as this function only gives the offset\n> but not the size, unlike the read_chunk() function.\n\nTrue. I suppose that could be more explicit, although it can be gleaned\nfrom the omission of any size information.\n\nThanks,\n-Stolee\n"},{"id":"417681","messageId":"20210224165242.GB3590451@szeder.dev","threadId":"55053","inReplyTo":"ddebf2389880e14a332e354f7b1dbc86a3964985.1613657259.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 03/17] commit-graph: use chunk-format write API","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-02-24T16:52:42Z","receivedAt":"2021-02-24T16:54:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Feb 18, 2021 at 02:07:25PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The commit-graph write logic is ready to make use of the chunk-format\n> write API. Each chunk write method is already in the correct prototype.\n> We only need to use the 'struct chunkfile' pointer and the correct API\n> calls.\n\nThis patch series messes up the \"Writing out commit graph\" progress\ndisplay, and starting at this commit I get:\n\n $ git commit-graph write --reachable\n Expanding reachable commits in commit graph: 837569, done.\n Writing out commit graph in 3 passes: 166% (4187845/2512707), done.\n\nNote that 166%.\n\nBefore this commit I got:\n\n Expanding reachable commits in commit graph: 837569, done.\n Writing out commit graph in 5 passes: 100% (4187845/4187845), done.\n\nNote the different number of passes.\n\n"},{"id":"417683","messageId":"YDaI9pVzW5wkK1aT@nand.local","threadId":"55053","inReplyTo":"20210224165242.GB3590451@szeder.dev","subject":"Re: [PATCH v4 03/17] commit-graph: use chunk-format write API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T17:12:22Z","receivedAt":"2021-02-24T17:13:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 24, 2021 at 05:52:42PM +0100, SZEDER Gábor wrote:\n> On Thu, Feb 18, 2021 at 02:07:25PM +0000, Derrick Stolee via GitGitGadget wrote:\n> > From: Derrick Stolee \n> >\n> > The commit-graph write logic is ready to make use of the chunk-format\n> > write API. Each chunk write method is already in the correct prototype.\n> > We only need to use the 'struct chunkfile' pointer and the correct API\n> > calls.\n>\n> This patch series messes up the \"Writing out commit graph\" progress\n> display, and starting at this commit I get:\n\nI can confirm. It looks like we never dropped the 'num_chunks' variable,\nwhich should have happened in this patch.\n\nHere's something to apply on top which fixes the issue. Thanks for\nreporting.\n\n--- >8 ---\n\nSubject: [PATCH] commit-graph.c: display correct number of chunks when writing\n\nWhen writing a commit-graph, a progress meter is shown which indicates\nthe number of pieces of data to write (one per commit in each chunk).\n\nIn 47410aa837 (commit-graph: use chunk-format write API, 2021-02-18),\nthe number of chunks became tracked by the new chunk-format API. But a\nstray local variable was left behind from when write_commit_graph_file()\nused to keep track of the same.\n\nSince this was no longer updated after 47410aa837, the progress meter\nappeared broken:\n\n $ git commit-graph write --reachable\n Expanding reachable commits in commit graph: 837569, done.\n Writing out commit graph in 3 passes: 166% (4187845/2512707), done.\n\nDrop the local variable and rely instead on the chunk-format API to tell\nus the correct number of chunks.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 78b993c367..6aa0c488f5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1791,7 +1791,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct lock_file lk = LOCK_INIT;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tint num_chunks = 3;\n \tstruct object_id file_hash;\n \tstruct chunkfile *cf;\n\n@@ -1887,11 +1886,11 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n \t\t\t \"Writing out commit graph in %d passes\",\n-\t\t\t num_chunks),\n-\t\t\t num_chunks);\n+\t\t\t get_num_chunks(cf)),\n+\t\t\t get_num_chunks(cf));\n \t\tctx->progress = start_delayed_progress(\n \t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * ctx->commits.nr);\n+\t\t\tget_num_chunks(cf) * ctx->commits.nr);\n \t}\n\n \twrite_chunkfile(cf, ctx);\n--\n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417688","messageId":"bb2ca73b-f60b-0030-2b28-753da48bf7a5@gmail.com","threadId":"55053","inReplyTo":"YDaI9pVzW5wkK1aT@nand.local","subject":"Re: [PATCH v4 03/17] commit-graph: use chunk-format write API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-24T17:52:16Z","receivedAt":"2021-02-24T17:53:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2021 12:12 PM, Taylor Blau wrote:\n> On Wed, Feb 24, 2021 at 05:52:42PM +0100, SZEDER Gábor wrote:\n>> On Thu, Feb 18, 2021 at 02:07:25PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>> From: Derrick Stolee \n>>>\n>>> The commit-graph write logic is ready to make use of the chunk-format\n>>> write API. Each chunk write method is already in the correct prototype.\n>>> We only need to use the 'struct chunkfile' pointer and the correct API\n>>> calls.\n>>\n>> This patch series messes up the \"Writing out commit graph\" progress\n>> display, and starting at this commit I get:\n\nThanks for the report and identifying the exact place that caused the\nmistake.\n \n> I can confirm. It looks like we never dropped the 'num_chunks' variable,\n> which should have happened in this patch.\n\nYes, makes sense. Hard to see that 'num_chunks' wasn't used because it\n_was_ being used, just not as intended.\n\n> @@ -1887,11 +1886,11 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \t\tstrbuf_addf(&progress_title,\n> \t\t\t Q_(\"Writing out commit graph in %d pass\",\n> \t\t\t \"Writing out commit graph in %d passes\",\n> -\t\t\t num_chunks),\n> -\t\t\t num_chunks);\n> +\t\t\t get_num_chunks(cf)),\n> +\t\t\t get_num_chunks(cf));\n> \t\tctx->progress = start_delayed_progress(\n> \t\t\tprogress_title.buf,\n> -\t\t\tnum_chunks * ctx->commits.nr);\n> +\t\t\tget_num_chunks(cf) * ctx->commits.nr);\n\nThis is obviously correct. Thanks for the quick patch!\n\n-Stolee\n"},{"id":"417715","messageId":"xmqqv9ahz1x2.fsf@gitster.g","threadId":"55053","inReplyTo":"bb2ca73b-f60b-0030-2b28-753da48bf7a5@gmail.com","subject":"Re: [PATCH v4 03/17] commit-graph: use chunk-format write API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-24T19:44:09Z","receivedAt":"2021-02-24T19:44:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>> This patch series messes up the \"Writing out commit graph\" progress\n>>> display, and starting at this commit I get:\n>\n> Thanks for the report and identifying the exact place that caused the\n> mistake.\n> \n>> I can confirm. It looks like we never dropped the 'num_chunks' variable,\n>> which should have happened in this patch.\n>\n> Yes, makes sense. Hard to see that 'num_chunks' wasn't used because it\n> _was_ being used, just not as intended.\n> ...\n>\n> This is obviously correct. Thanks for the quick patch!\n\nThanks all for noticing and fixing before the series hit the master\nbranch.\n\n"}]}