{"thread":{"id":"49993","subject":"[PATCH 0/5] Create 'expire' and 'repack' verbs for git-multi-pack-index","startedAt":"2018-12-10T18:06:23Z","lastAt":"2019-06-10T17:58:03Z","messageCount":92,"participants":["Derrick Stolee via GitGitGadget","Stefan Beller","SZEDER Gábor","Derrick Stolee","Junio C Hamano","Jonathan Tan","Josh Steadmon"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"364947","messageId":"pull.92.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":null,"subject":"[PATCH 0/5] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T18:06:18Z","receivedAt":"2018-12-10T18:06:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index provides a fast way to find an object among a large\nlist of pack-files. It stores a single pack-reference for each object id, so\nduplicate objects are ignored. Among a list of pack-files storing the same\nobject, the most-recently modified one is used.\n\nCreate new verbs for the multi-pack-index builtin.\n\n * 'git multi-pack-index expire': If we have a pack-file indexed by the\n multi-pack-index, but all objects in that pack are duplicated in\n more-recently modified packs, then delete that pack (and any others like\n it). Delete the reference to that pack in the multi-pack-index.\n \n \n * 'git multi-pack-index repack --batch-size=': Starting from the oldest\n pack-files covered by the multi-pack-index, find those whose on-disk size\n is below the batch size until we have a collection of packs whose sizes\n add up to the batch size. Create a new pack containing all objects that\n the multi-pack-index references to those packs.\n \n \n\nThis allows us to create a new pattern for repacking objects: run 'repack'.\nAfter enough time has passed that all Git commands that started before the\nlast 'repack' are finished, run 'expire' again. This approach has some\nadvantages over the existing \"repack everything\" model:\n\n 1. Incremental. We can repack a small batch of objects at a time, instead\n of repacking all reachable objects. We can also limit ourselves to the\n objects that do not appear in newer pack-files.\n \n \n 2. Highly Available. By adding a new pack-file (and not deleting the old\n pack-files) we do not interrupt concurrent Git commands, and do not\n suffer performance degradation. By expiring only pack-files that have no\n referenced objects, we know that Git commands that are doing normal\n object lookups* will not be interrupted.\n \n \n 3. Note: if someone concurrently runs a Git command that uses\n get_all_packs(), then that command could try to read the pack-files and\n pack-indexes that we are deleting during an expire command. Such\n commands are usually related to object maintenance (i.e. fsck, gc,\n pack-objects) or are related to less-often-used features (i.e.\n fast-import, http-backend, server-info).\n \n \n\nWe plan to use this approach in VFS for Git to do background maintenance of\nthe \"shared object cache\" which is a Git alternate directory filled with\npackfiles containing commits and trees. We currently download pack-files on\nan hourly basis to keep up-to-date with the central server. The cache\nservers supply packs on an hourly and daily basis, so most of the hourly\npacks become useless after a new daily pack is downloaded. The 'expire'\ncommand would clear out most of those packs, but many will still remain with\nfewer than 100 objects remaining. The 'repack' command (with a batch size of\n1-3gb, probably) can condense the remaining packs in commands that run for\n1-3 min at a time. Since the daily packs range from 100-250mb, we will also\ncombine and condense those packs.\n\nThanks, -Stolee\n\nDerrick Stolee (5):\n multi-pack-index: prepare for 'expire' verb\n midx: refactor permutation logic\n multi-pack-index: implement 'expire' verb\n multi-pack-index: prepare 'repack' verb\n midx: implement midx_repack()\n\n Documentation/git-multi-pack-index.txt | 20 +++\n builtin/multi-pack-index.c | 12 +-\n midx.c | 222 +++++++++++++++++++++++--\n midx.h | 2 +\n t/t5319-multi-pack-index.sh | 98 +++++++++++\n 5 files changed, 343 insertions(+), 11 deletions(-)\n\n\nbase-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-92%2Fderrickstolee%2Fmidx-expire%2Fupstream-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-92/derrickstolee/midx-expire/upstream-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/92\n-- \ngitgitgadget\n"},{"id":"364948","messageId":"1e34b48a2051089052c0b53b0059576b5ace45de.1544465177.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.git.gitgitgadget@gmail.com","subject":"[PATCH 1/5] multi-pack-index: prepare for 'expire' verb","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T18:06:20Z","receivedAt":"2018-12-10T18:06:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe multi-pack-index tracks objects in a collection of pack-files.\nOnly one copy of each object is indexed, using the modified time\nof the pack-files to determine tie-breakers. It is possible to\nhave a pack-file with no referenced objects because all objects\nhave a duplicate in a newer pack-file.\n\nIntroduce a new 'expire' verb to the multi-pack-index builtin.\nThis verb will delete these unused pack-files and rewrite the\nmulti-pack-index to no longer refer to those files. More details\nabout the specifics will follow as the method is implemented.\n\nAdd a test that verifies the 'expire' verb is correctly wired,\nbut will still be valid when the verb is implemented. Specifically,\ncreate a set of packs that should all have referenced objects and\nshould not be removed during an 'expire' operation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 8 +++++\n builtin/multi-pack-index.c | 4 ++-\n midx.c | 5 +++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 47 ++++++++++++++++++++++++++\n 5 files changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex f7778a2c85..822d83c845 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -31,6 +31,14 @@ verify::\n \tWhen given as the verb, verify the contents of the MIDX file\n \tat `/packs/multi-pack-index`.\n \n+expire::\n+\tWhen given as the verb, delete the pack-files that are tracked\n+\tby the MIDX file at `/packs/multi-pack-index` but have\n+\tno objects referenced by the MIDX. All objects in these pack-\n+\tfiles have another copy in a more-recently modified pack-file.\n+\tRewrite the MIDX file afterward to remove all references to\n+\tthese pack-files.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex fca70f8e4f..145de3a46c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,7 +5,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n \tNULL\n };\n \n@@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(opts.object_dir);\n+\tif (!strcmp(argv[0], \"expire\"))\n+\t\treturn expire_midx_packs(opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 730ff84dff..bb825ef816 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir)\n \n \treturn verify_midx_error;\n }\n+\n+int expire_midx_packs(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 774f652530..e3a2b740b5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n+int expire_midx_packs(const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 70926b5bc0..948effc1ee 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -348,4 +348,51 @@ test_expect_success 'verify incorrect 64-bit offset' '\n \t\t\"incorrect object offset\"\n '\n \n+test_expect_success 'setup expire tests' '\n+\tmkdir dup &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit init &&\n+\t\tfor i in $(test_seq 1 20)\n+\t\tdo\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit branch A HEAD &&\n+\t\tgit branch B HEAD~8 &&\n+\t\tgit branch C HEAD~13 &&\n+\t\tgit branch D HEAD~16 &&\n+\t\tgit branch E HEAD~18 &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n+test_expect_success 'expire does not remove any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"364949","messageId":"244bdf2a6fc300f2b535ac8edfc2fbdaf5260266.1544465177.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.git.gitgitgadget@gmail.com","subject":"[PATCH 3/5] multi-pack-index: implement 'expire' verb","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T18:06:22Z","receivedAt":"2018-12-10T18:06:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git multi-pack-index expire' command looks at the existing\nmult-pack-index, counts the number of objects referenced in each\npack-file, deletes the pack-fils with no referenced objects, and\nrewrites the multi-pack-index to no longer reference those packs.\n\nRefactor the write_midx_file() method to call write_midx_internal()\nwhich now takes an existing 'struct multi_pack_index' and a list\nof pack-files to drop (as specified by the names of their pack-\nindexes). As we write the new multi-pack-index, we drop those\nfile names from the list of known pack-files.\n\nThe expire_midx_packs() method removes the unreferenced pack-files\nafter carefully closing the packs to avoid open handles.\n\nTest that a new pack-file that covers the contents of two other\npack-files leads to those pack-files being deleted during the\nexpire command. Be sure to read the multi-pack-index to ensure\nit no longer references those packs.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 87 +++++++++++++++++++++++++++++++++++--\n t/t5319-multi-pack-index.sh | 15 +++++++\n 2 files changed, 98 insertions(+), 4 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3bd7183a53..50e4cd7270 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -751,7 +751,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n-int write_midx_file(const char *object_dir)\n+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+\t\t\t struct string_list *packs_to_drop)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -765,6 +766,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n+\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -773,7 +775,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n-\tpacks.m = load_multi_pack_index(object_dir, 1);\n+\tif (m)\n+\t\tpacks.m = m;\n+\telse\n+\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n \tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n@@ -787,7 +792,24 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(packs.perm, packs.alloc_perm);\n \n \tif (packs.m) {\n+\t\tint drop_index = 0, missing_drops = 0;\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n+\t\t\tif (packs_to_drop && drop_index < packs_to_drop->nr) {\n+\t\t\t\tint cmp = strcmp(packs.m->pack_names[i],\n+\t\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\t\tif (!cmp) {\n+\t\t\t\t\tdrop_index++;\n+\t\t\t\t\tcontinue;\n+\t\t\t\t} else if (cmp > 0) {\n+\t\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t\tdrop_index++;\n+\t\t\t\t\ti--;\n+\t\t\t\t\tmissing_drops++;\n+\t\t\t\t}\n+\t\t\t}\n+\n \t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n \t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n \t\t\tALLOC_GROW(packs.perm, packs.nr + 1, packs.alloc_perm);\n@@ -798,6 +820,12 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n \t\t\tpacks.nr++;\n \t\t}\n+\n+\t\tif (packs_to_drop && (drop_index < packs_to_drop->nr || missing_drops)) {\n+\t\t\terror(_(\"did not see all pack-files to drop\"));\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n \t}\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n@@ -932,7 +960,12 @@ cleanup:\n \tfree(packs.perm);\n \tfree(entries);\n \tfree(midx_name);\n-\treturn 0;\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1034,5 +1067,51 @@ int verify_midx_file(const char *object_dir)\n \n int expire_midx_packs(const char *object_dir)\n {\n-\treturn 0;\n+\tuint32_t i, *count, result = 0;\n+\tsize_t dirlen;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t}\n+\n+\tstrbuf_addstr(&buf, object_dir);\n+\tstrbuf_addstr(&buf, \"/pack/\");\n+\tdirlen = buf.len;\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]) {\n+\t\t\tclose_pack(m->packs[i]);\n+\t\t\tm->packs[i] = NULL;\n+\t\t}\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\n+\t\tstrbuf_setlen(&buf, dirlen);\n+\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n+\t\tunlink(buf.buf);\n+\n+\t\tstrip_suffix_mem(buf.buf, &buf.len, \"idx\");\n+\t\tstrbuf_addstr(&buf, \"pack\");\n+\t\tunlink(buf.buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 948effc1ee..210279a3cf 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -395,4 +395,19 @@ test_expect_success 'expire does not remove any packs' '\n \t)\n '\n \n+test_expect_success 'expire removes unreferenced packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"364950","messageId":"8f496ccb461aae2f169159d9dce957c1d16a31dc.1544465177.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.git.gitgitgadget@gmail.com","subject":"[PATCH 2/5] midx: refactor permutation logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T18:06:21Z","receivedAt":"2018-12-10T18:06:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen writing a multi-pack-index, we keep track of an integer\npermutation, tracking the list of pack-files that we know about\n(both from the existing multi-pack-index and the new pack-files\nbeing introduced) and converting them into a sorted order for\nthe new multi-pack-index.\n\nIn anticipation of dropping pack-files from the existing multi-\npack-index, refactor the logic around how we track this permutation.\n\nFirst, insert the permutation into the pack_list structure. This\nallows us to grow the permutation dynamically as we add packs.\n\nSecond, fill the permutation with values corresponding to their\nposition in the list of pack-files, sorted as follows:\n\n 1. The pack-files in the existing multi-pack-index,\n sorted lexicographically.\n\n 2. The pack-files not in the existing multi-pack-index,\n sorted as discovered from the filesystem.\n\nThere is a subtle thing in how we initialize this permutation,\nspecifically how we use 'i' for the initial value. This will\nmatter more when we implement the logic for dropping existing\npacks, as we will create holes in the ordering.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 20 +++++++++++++-------\n 1 file changed, 13 insertions(+), 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bb825ef816..3bd7183a53 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -380,9 +380,11 @@ static size_t write_midx_header(struct hashfile *f,\n struct pack_list {\n \tstruct packed_git **list;\n \tchar **names;\n+\tuint32_t *perm;\n \tuint32_t nr;\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n+\tuint32_t alloc_perm;\n \tsize_t pack_name_concat_len;\n \tstruct multi_pack_index *m;\n };\n@@ -398,6 +400,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n+\t\tALLOC_GROW(packs->perm, packs->nr + 1, packs->alloc_perm);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n \t\t\t\t\t\t\tfull_path_len,\n@@ -417,6 +420,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\treturn;\n \t\t}\n \n+\t\tpacks->perm[packs->nr] = packs->nr;\n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n \t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n@@ -443,7 +447,7 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \tALLOC_ARRAY(pairs, nr_packs);\n \n \tfor (i = 0; i < nr_packs; i++) {\n-\t\tpairs[i].pack_int_id = i;\n+\t\tpairs[i].pack_int_id = perm[i];\n \t\tpairs[i].pack_name = pack_names[i];\n \t}\n \n@@ -755,7 +759,6 @@ int write_midx_file(const char *object_dir)\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n-\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n@@ -774,18 +777,22 @@ int write_midx_file(const char *object_dir)\n \n \tpacks.nr = 0;\n \tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n-\tpacks.alloc_names = packs.alloc_list;\n+\tpacks.alloc_perm = packs.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.names = NULL;\n+\tpacks.perm = NULL;\n \tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n+\tALLOC_ARRAY(packs.perm, packs.alloc_perm);\n \n \tif (packs.m) {\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n \t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n \t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\t\t\tALLOC_GROW(packs.perm, packs.nr + 1, packs.alloc_perm);\n \n+\t\t\tpacks.perm[packs.nr] = i;\n \t\t\tpacks.list[packs.nr] = NULL;\n \t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n \t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n@@ -802,10 +809,9 @@ int write_midx_file(const char *object_dir)\n \t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n+\tsort_packs_by_name(packs.names, packs.nr, packs.perm);\n \n-\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.list, packs.perm, packs.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -923,8 +929,8 @@ cleanup:\n \n \tfree(packs.list);\n \tfree(packs.names);\n+\tfree(packs.perm);\n \tfree(entries);\n-\tfree(pack_perm);\n \tfree(midx_name);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"364951","messageId":"72b213959171af3bfe4d849b925920ddbfb3d4b7.1544465177.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.git.gitgitgadget@gmail.com","subject":"[PATCH 4/5] multi-pack-index: prepare 'repack' verb","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T18:06:24Z","receivedAt":"2018-12-10T18:06:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment where the multi-pack-index is useful, it is due\nto many pack-files and an inability to repack the object store\ninto a single pack-file. However, it is likely that many of these\npack-files are rather small, and could be repacked into a slightly\nlarger pack-file without too much effort. It may also be important\nto ensure the object store is highly available and the repack\noperation does not interrupt concurrent git commands.\n\nIntroduce a 'repack' verb to 'git multi-pack-index' that takes a\n'--batch-size' option. The verb will inspect the multi-pack-index\nfor referenced pack-files whose size is smaller than the batch\nsize, until collecting a list of pack-files whose sizes sum to\nlarger than the batch size. Then, a new pack-file will be created\ncontaining the objects from those pack-files that are referenced\nby the multi-pack-index. The resulting pack is likely to actually\nbe smaller than the batch size due to compression and the fact\nthat there may be objects in the pack-files that have duplicate\ncopies in other pack-files.\n\nThe current change introduces the command-line arguments, and we\nadd a test that ensures we parse these options properly. Since\nwe specify a small batch size, we will guarantee that future\nimplementations do not change the list of pack-files.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 12 ++++++++++++\n builtin/multi-pack-index.c | 10 +++++++++-\n midx.c | 5 +++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 11 +++++++++++\n 5 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 822d83c845..e43e7da71e 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -39,6 +39,18 @@ expire::\n \tRewrite the MIDX file afterward to remove all references to\n \tthese pack-files.\n \n+repack::\n+\tWhen given as the verb, collect a batch of pack-files whose\n+\tsize are all at most the size given by --batch-size, but\n+\twhose sizes sum to larger than --batch-size. The batch is\n+\tselected by greedily adding small pack-files starting with\n+\tthe oldest pack-files that fit the size. Create a new pack-\n+\tfile containing the objects the multi-pack-index indexes\n+\tinto thos pack-files, and rewrite the multi-pack-index to\n+\tcontain that pack-file. A later run of 'git multi-pack-index\n+\texpire' will delete the pack-files that were part of this\n+\tbatch.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 145de3a46c..d87a2235e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,12 +5,13 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n \tNULL\n };\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tunsigned long batch_size;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n@@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n+\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n@@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn 1;\n \t}\n \n+\tif (!strcmp(argv[0], \"repack\"))\n+\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n+\tif (opts.batch_size)\n+\t\tdie(_(\"--batch-size option is only for 'repack' verb\"));\n+\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\ndiff --git a/midx.c b/midx.c\nindex 50e4cd7270..4caf148464 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1115,3 +1115,8 @@ int expire_midx_packs(const char *object_dir)\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex e3a2b740b5..394a21ee96 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n int expire_midx_packs(const char *object_dir);\n+int midx_repack(const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 210279a3cf..c23e930a5d 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -410,4 +410,15 @@ test_expect_success 'expire removes unreferenced packs' '\n \t)\n '\n \n+test_expect_success 'repack does not create any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n+\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"364952","messageId":"41ef671ec8361a9635dc78c078d2d84e9d985236.1544465177.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.git.gitgitgadget@gmail.com","subject":"[PATCH 5/5] midx: implement midx_repack()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T18:06:25Z","receivedAt":"2018-12-10T18:06:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo repack using a multi-pack-index, first sort all pack-files by\ntheir modified time. Second, walk those pack-files from oldest\nto newest, adding the packs to a list if they are smaller than the\ngiven pack-size. Finally, collect the objects from the multi-pack-\nindex that are in those packs and send them to 'git pack-objects'.\n\nWhile first designing a 'git multi-pack-index repack' operation, I\nstarted by collecting the batches based on the size of the objects\ninstead of the size of the pack-files. This allows repacking a\nlarge pack-file that has very few referencd objects. However, this\ncame at a significant cost of parsing pack-files instead of simply\nreading the multi-pack-index and getting the file information for\nthe pack-files. This object-size idea could be a direction for\nfuture expansion in this area.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 109 +++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 25 +++++++++\n 2 files changed, 133 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4caf148464..3718e78132 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -8,6 +8,7 @@\n #include \"sha1-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n+#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -1116,7 +1117,113 @@ int expire_midx_packs(const char *object_dir)\n \treturn result;\n }\n \n-int midx_repack(const char *object_dir, size_t batch_size)\n+struct time_and_id {\n+\ttimestamp_t mtime;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n {\n+\tconst struct time_and_id *a, *b;\n+\n+\ta = (const struct time_and_id *)a_;\n+\tb = (const struct time_and_id *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n \treturn 0;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\tint result = 0;\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct time_and_id *pack_ti;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n+\tpack_ti = xcalloc(m->num_packs, sizeof(struct time_and_id));\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_ti[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_ti[i].mtime = m->packs[i]->mtime;\n+\t}\n+\tQSORT(pack_ti, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_ti[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (p->pack_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += p->pack_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tif (total_size < batch_size || packs_to_repack < 2)\n+\t\tgoto cleanup;\n+\n+\targv_array_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\targv_array_push(&cmd.args, base_name.buf);\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\txwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);\n+\t\txwrite(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, m, NULL);\n+\tm = NULL;\n+\n+cleanup:\n+\tif (m)\n+\t\tclose_midx(m);\n+\tfree(include_pack);\n+\tfree(pack_ti);\n+\treturn result;\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex c23e930a5d..3cc9c918d5 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -421,4 +421,29 @@ test_expect_success 'repack does not create any packs' '\n \t)\n '\n \n+test_expect_success 'repack creates a new pack' '\n+\t(\n+\t\tcd dup &&\n+\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n+\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 5 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 5 midx-list\n+\t)\n+'\n+\n+test_expect_success 'expire removes repacked packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls -S .git/objects/pack/*pack | head -n 3 >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/*pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 3 midx-list\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"365003","messageId":"CAGZ79kYeHgCdzNNXUnp1NaokT=Vpj4mUVt1bZb5c+EkgO2z-4w@mail.gmail.com","threadId":"49993","inReplyTo":"1e34b48a2051089052c0b53b0059576b5ace45de.1544465177.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/5] multi-pack-index: prepare for 'expire' verb","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-12-11T01:35:28Z","receivedAt":"2018-12-11T01:35:43Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The multi-pack-index tracks objects in a collection of pack-files.\n> Only one copy of each object is indexed, using the modified time\n> of the pack-files to determine tie-breakers. It is possible to\n> have a pack-file with no referenced objects because all objects\n> have a duplicate in a newer pack-file.\n>\n> Introduce a new 'expire' verb to the multi-pack-index builtin.\n> This verb will delete these unused pack-files and rewrite the\n> multi-pack-index to no longer refer to those files. More details\n> about the specifics will follow as the method is implemented.\n>\n> Add a test that verifies the 'expire' verb is correctly wired,\n> but will still be valid when the verb is implemented. Specifically,\n> create a set of packs that should all have referenced objects and\n> should not be removed during an 'expire' operation.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-multi-pack-index.txt | 8 +++++\n> builtin/multi-pack-index.c | 4 ++-\n> midx.c | 5 +++\n> midx.h | 1 +\n> t/t5319-multi-pack-index.sh | 47 ++++++++++++++++++++++++++\n> 5 files changed, 64 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index f7778a2c85..822d83c845 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -31,6 +31,14 @@ verify::\n> When given as the verb, verify the contents of the MIDX file\n> at `/packs/multi-pack-index`.\n>\n> +expire::\n> + When given as the verb,\n\nCan it be given in another way? Or rather \"if the verb is expire\",\nthen ...\n(I just checked the current man page, and both write and verify use\nthis pattern as well. I find it strange as this first part of the sentence\nconveys little information, but is repeated 3 times now (once for\neach verb)).\n\nMaybe we can restructure the man page to have it more like\n\n The following verbs are available:\n +write::\n + create a new MIDX file, writing to /packs/multi-pack-index.\n +\n +verify::\n + verify the contents ...\n\n> delete the pack-files that are tracked\n> + by the MIDX file at `/packs/multi-pack-index`\n\nWe're mentioning the location a lot. Could we keep a more detailed\nnote in --object-dir and not go into such detail in the verbs section?\n(Then the paragraph here would be more concise. That makes it\neasier to understand)\n\n> but have\n> + no objects referenced by the MIDX. All objects in these pack-\n> + files have another copy in a more-recently modified pack-file.\n\nThe second sentence reads like a reason on why the first is a good\nthing to have, so maybe use some subordinating conjunction adverb\n(\"because\") to make tell the reader\n\n> + Rewrite the MIDX file afterward to remove all references to\n> + these pack-files.\n\nMakes sense.\n\n\n>\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index 70926b5bc0..948effc1ee 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -348,4 +348,51 @@ test_expect_success 'verify incorrect 64-bit offset' '\n> \"incorrect object offset\"\n> '\n>\n> +test_expect_success 'setup expire tests' '\n> + mkdir dup &&\n> + (\n> + cd dup &&\n> + git init &&\n> + for i in $(test_seq 1 20)\n> + do\n> + test_commit $i\n> + done &&\n> + git branch A HEAD &&\n> + git branch B HEAD~8 &&\n> + git branch C HEAD~13 &&\n> + git branch D HEAD~16 &&\n> + git branch E HEAD~18 &&\n> + git pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n> + refs/heads/E\n> + EOF\n> + git pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n> + refs/heads/D\n> + ^refs/heads/E\n> + EOF\n> + git pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n> + refs/heads/C\n> + ^refs/heads/D\n> + EOF\n> + git pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n> + refs/heads/B\n> + ^refs/heads/C\n> + EOF\n> + git pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n> + refs/heads/A\n> + ^refs/heads/B\n> + EOF\n> + git multi-pack-index write\n> + )\n> +'\n> +\n> +test_expect_success 'expire does not remove any packs' '\n\nWith the clever setup, this test is already correctly testing\nwhat the docs claims it should do, despite having\nno implementation. Nice.\nAlthough the core issue is that the packs are disjunct sets\nof objects, so maybe /s/any packs/required packs/ or such?\n"},{"id":"365008","messageId":"CAGZ79kaNJ8=iCunkeeHnMcOwFik9D-V4g9o7SuvLnnM0vvfBnA@mail.gmail.com","threadId":"49993","inReplyTo":"72b213959171af3bfe4d849b925920ddbfb3d4b7.1544465177.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/5] multi-pack-index: prepare 'repack' verb","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-12-11T01:54:15Z","receivedAt":"2018-12-11T01:54:30Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> In an environment where the multi-pack-index is useful, it is due\n> to many pack-files and an inability to repack the object store\n> into a single pack-file. However, it is likely that many of these\n> pack-files are rather small, and could be repacked into a slightly\n> larger pack-file without too much effort. It may also be important\n> to ensure the object store is highly available and the repack\n> operation does not interrupt concurrent git commands.\n>\n> Introduce a 'repack' verb to 'git multi-pack-index' that takes a\n> '--batch-size' option. The verb will inspect the multi-pack-index\n> for referenced pack-files whose size is smaller than the batch\n> size, until collecting a list of pack-files whose sizes sum to\n> larger than the batch size. Then, a new pack-file will be created\n> containing the objects from those pack-files that are referenced\n> by the multi-pack-index. The resulting pack is likely to actually\n> be smaller than the batch size due to compression and the fact\n> that there may be objects in the pack-files that have duplicate\n> copies in other pack-files.\n\nThis highlights an optimization problem: How do we pick the\nbatches optimally?\nIdeally we'd pick packs that have an overlap of many\nsame objects to dedup them completely, next best would\nbe to have objects that are very similar, such that they delta\nvery well.\n(Assuming that the sum of the resulting pack sizes is a metric\nwe'd want to optimize for eventually)\n\nFor now it seems we just take a random cut of \"small\" packs.\n\n>\n> The current change introduces the command-line arguments, and we\n> add a test that ensures we parse these options properly. Since\n> we specify a small batch size, we will guarantee that future\n> implementations do not change the list of pack-files.\n\nThis is another clever trick that makes the test correct\ndespite no implementation yet. :-)\n\n> +repack::\n> + When given as the verb, collect a batch of pack-files whose\n> + size are all at most the size given by --batch-size,\n\nokay.\n\n> but\n> + whose sizes sum to larger than --batch-size.\n\nor less if there are not enough packs.\n\nNow it would be interesting if we can specify an upper bound\n(e.g. my laptop has 8GB of ram, can I use this incremental\nrepacking optimally by telling git to make batches of at most\n7.5G), the answer seems to follow...\n\n> The batch is\n> + selected by greedily adding small pack-files starting with\n> + the oldest pack-files that fit the size. Create a new pack-\n> + file containing the objects the multi-pack-index indexes\n> + into thos pack-files, and rewrite the multi-pack-index to\n\nthose\n\n> + contain that pack-file. A later run of 'git multi-pack-index\n> + expire' will delete the pack-files that were part of this\n> + batch.\n\n... but the optimization seems to be rather about getting rid\nof the oldest packs first instead of getting as close to the batch\nsize. (e.g. another way to look at this is to \"find the permutation\nof all packs that (each are smaller than batch size), but in sum\nare the smallest threshold above the batch size).\n\nI guess that the strategy of picking the oldest is just easiest\nto implement and should be sufficient for now, but memory\nbounds might be interesting to keep in mind, just as the\noptimal packing from above.\n"},{"id":"365009","messageId":"20181211015957.GR30222@szeder.dev","threadId":"49993","inReplyTo":"CAGZ79kYeHgCdzNNXUnp1NaokT=Vpj4mUVt1bZb5c+EkgO2z-4w@mail.gmail.com","subject":"Re: [PATCH 1/5] multi-pack-index: prepare for 'expire' verb","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-11T01:59:57Z","receivedAt":"2018-12-11T02:00:05Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Dec 10, 2018 at 05:35:28PM -0800, Stefan Beller wrote:\n> On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > From: Derrick Stolee \n> >\n> > The multi-pack-index tracks objects in a collection of pack-files.\n> > Only one copy of each object is indexed, using the modified time\n> > of the pack-files to determine tie-breakers. It is possible to\n> > have a pack-file with no referenced objects because all objects\n> > have a duplicate in a newer pack-file.\n> >\n> > Introduce a new 'expire' verb to the multi-pack-index builtin.\n> > This verb will delete these unused pack-files and rewrite the\n> > multi-pack-index to no longer refer to those files. More details\n> > about the specifics will follow as the method is implemented.\n> >\n> > Add a test that verifies the 'expire' verb is correctly wired,\n> > but will still be valid when the verb is implemented. Specifically,\n> > create a set of packs that should all have referenced objects and\n> > should not be removed during an 'expire' operation.\n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > Documentation/git-multi-pack-index.txt | 8 +++++\n> > builtin/multi-pack-index.c | 4 ++-\n> > midx.c | 5 +++\n> > midx.h | 1 +\n> > t/t5319-multi-pack-index.sh | 47 ++++++++++++++++++++++++++\n> > 5 files changed, 64 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> > index f7778a2c85..822d83c845 100644\n> > --- a/Documentation/git-multi-pack-index.txt\n> > +++ b/Documentation/git-multi-pack-index.txt\n> > @@ -31,6 +31,14 @@ verify::\n> > When given as the verb, verify the contents of the MIDX file\n> > at `/packs/multi-pack-index`.\n> >\n> > +expire::\n> > + When given as the verb,\n> \n> Can it be given in another way? Or rather \"if the verb is expire\",\n> then ...\n> (I just checked the current man page, and both write and verify use\n> this pattern as well. I find it strange as this first part of the sentence\n> conveys little information, but is repeated 3 times now (once for\n> each verb)).\n> \n> Maybe we can restructure the man page to have it more like\n> \n> The following verbs are available:\n> +write::\n> + create a new MIDX file, writing to /packs/multi-pack-index.\n> +\n> +verify::\n> + verify the contents ...\n\nI think a s/verb/subcommand/ would help a lot, too, because that's\nwhat we call it everywhere else.\n\n"},{"id":"365015","messageId":"CAGZ79kbPcy2U9XJA+Je0zRxFsQJGA9u8nfYZe_s75V8c97+dNw@mail.gmail.com","threadId":"49993","inReplyTo":"41ef671ec8361a9635dc78c078d2d84e9d985236.1544465177.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] midx: implement midx_repack()","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-12-11T02:32:23Z","receivedAt":"2018-12-11T02:32:38Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> To repack using a multi-pack-index, first sort all pack-files by\n> their modified time. Second, walk those pack-files from oldest\n> to newest, adding the packs to a list if they are smaller than the\n> given pack-size. Finally, collect the objects from the multi-pack-\n> index that are in those packs and send them to 'git pack-objects'.\n\nMakes sense.\n\nWith this operation we only coalesce some packfiles into a new\npack file. So to perform the \"complete\" repack this command\nhas to be run repeatedly until there is at most one packfile\nleft that is smaller than batch size.\n\nImagine the following scenario:\n\n There are 5 packfiles A, B, C, D, E,\n created last Monday thru Friday (A is oldest, E youngest).\n The sizes are [A=4, B=6, C=5, D=5, E=4]\n\n You'd issue a repack with batch size=10, such that\n A and B would be repacked into F, which is\n created today, size is less or equal than 10.\n\n You issue another repack tomorrow, which then would\n coalesce C and D to G, which is\n dated tomorrow, size is less or equal to 10 as well.\n\n You issue a third repack, which then takes E\n (as it is the oldest) and would probably find F as the\n next oldest (assuming it is less than 10), to repack\n into H.\n\n H is then compromised of A, B and E, and G is C+D.\n\nIn a way these repacks, always picking up the oldest,\nsound like you \"roll forward\" objects into new packs.\nAs the new packs are newest (we have no packs from\nthe future), we'd cycle through different packs to look at\nfor packing on each repacking.\n\nIt is however more likely that content is more similar\non a temporal basis. (e.g. I am boldly claiming that\n[ABC, DE] would take less space than [ABE, CD]\nas produced above).\n\n(The obvious solution to this hypothetical would be\nto backdate the resulting pack to the youngest pack\nthat is input to the new pack, but I dislike fudging with\nthe time a file is created/touched, so let's not go there)\n\nWould the object count make sense as input instead of\nthe pack date?\n\n\n> While first designing a 'git multi-pack-index repack' operation, I\n> started by collecting the batches based on the size of the objects\n> instead of the size of the pack-files. This allows repacking a\n> large pack-file that has very few referencd objects. However, this\n\nreferenced\n\n> came at a significant cost of parsing pack-files instead of simply\n> reading the multi-pack-index and getting the file information for\n> the pack-files. This object-size idea could be a direction for\n> future expansion in this area.\n\nAh, that also explains why the above idea is toast.\n\nWould it make sense to extend or annotate the midx file\nto give hints at which packs are easy to combine?\n\nI guess such an \"annotation worker\" could run in a separate\nthread / pool with the lowest priority as this seems like a\ndecent fallback for the lack of any better information how\nto pick the packfiles.\n\nThanks,\nStefan\n"},{"id":"365069","messageId":"0f0888e6-2df5-ec5c-5c9a-7e6956cb9612@gmail.com","threadId":"49993","inReplyTo":"20181211015957.GR30222@szeder.dev","subject":"Re: [PATCH 1/5] multi-pack-index: prepare for 'expire' verb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-12-11T12:32:35Z","receivedAt":"2018-12-11T12:32:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/10/2018 8:59 PM, SZEDER Gábor wrote:\n> On Mon, Dec 10, 2018 at 05:35:28PM -0800, Stefan Beller wrote:\n>> On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n>> wrote:\n>>> +expire::\n>>> + When given as the verb,\n>> Can it be given in another way? Or rather \"if the verb is expire\",\n>> then ...\n>> (I just checked the current man page, and both write and verify use\n>> this pattern as well. I find it strange as this first part of the sentence\n>> conveys little information, but is repeated 3 times now (once for\n>> each verb)).\n>>\n>> Maybe we can restructure the man page to have it more like\n>>\n>> The following verbs are available:\n>> +write::\n>> + create a new MIDX file, writing to /packs/multi-pack-index.\n>> +\n>> +verify::\n>> + verify the contents ...\n> I think a s/verb/subcommand/ would help a lot, too, because that's\n> what we call it everywhere else.\n\nThanks, both. V2 will include a new patch that reformats the doc to use \nthese suggestions, then extend it for the new subcommand.\n\n-Stolee\n\n"},{"id":"365073","messageId":"931d481e-8b9f-0f1f-e455-10111ec7b0dc@gmail.com","threadId":"49993","inReplyTo":"CAGZ79kaNJ8=iCunkeeHnMcOwFik9D-V4g9o7SuvLnnM0vvfBnA@mail.gmail.com","subject":"Re: [PATCH 4/5] multi-pack-index: prepare 'repack' verb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-12-11T12:45:01Z","receivedAt":"2018-12-11T12:45:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/10/2018 8:54 PM, Stefan Beller wrote:\n> On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> From: Derrick Stolee \n>>\n>> In an environment where the multi-pack-index is useful, it is due\n>> to many pack-files and an inability to repack the object store\n>> into a single pack-file. However, it is likely that many of these\n>> pack-files are rather small, and could be repacked into a slightly\n>> larger pack-file without too much effort. It may also be important\n>> to ensure the object store is highly available and the repack\n>> operation does not interrupt concurrent git commands.\n>>\n>> Introduce a 'repack' verb to 'git multi-pack-index' that takes a\n>> '--batch-size' option. The verb will inspect the multi-pack-index\n>> for referenced pack-files whose size is smaller than the batch\n>> size, until collecting a list of pack-files whose sizes sum to\n>> larger than the batch size. Then, a new pack-file will be created\n>> containing the objects from those pack-files that are referenced\n>> by the multi-pack-index. The resulting pack is likely to actually\n>> be smaller than the batch size due to compression and the fact\n>> that there may be objects in the pack-files that have duplicate\n>> copies in other pack-files.\n> This highlights an optimization problem: How do we pick the\n> batches optimally?\n> Ideally we'd pick packs that have an overlap of many\n> same objects to dedup them completely, next best would\n> be to have objects that are very similar, such that they delta\n> very well.\n> (Assuming that the sum of the resulting pack sizes is a metric\n> we'd want to optimize for eventually)\n>\n> For now it seems we just take a random cut of \"small\" packs.\n>\n>> The current change introduces the command-line arguments, and we\n>> add a test that ensures we parse these options properly. Since\n>> we specify a small batch size, we will guarantee that future\n>> implementations do not change the list of pack-files.\n> This is another clever trick that makes the test correct\n> despite no implementation yet. :-)\n>\n>> +repack::\n>> + When given as the verb, collect a batch of pack-files whose\n>> + size are all at most the size given by --batch-size,\n> okay.\n>\n>> but\n>> + whose sizes sum to larger than --batch-size.\n> or less if there are not enough packs.\n\nIf there are not enough packs to reach the batch size, then we do nothing.\n\n> Now it would be interesting if we can specify an upper bound\n> (e.g. my laptop has 8GB of ram, can I use this incremental\n> repacking optimally by telling git to make batches of at most\n> 7.5G), the answer seems to follow...\n\nWell, this gets us to the \"unsigned long\" problem and how it pervades \nthe OPT_MAGNITUDE() code and things that use that kind of parameter. \nThis means that Windows users can specify a maximum of (1 << 32) - 1. \nThis size is intended to be large enough to make a reasonable change in \nthe pack organization without rewriting the entire repo (e.g. 1GB). If \nthere is another word than \"repack\" that means \"collect objects from a \nsubset of pack-files and place them into a new pack-file, doing a small \namount of redeltification in the process\" then I am open to suggestions.\n\nI tried (briefly) to fix this dependence, but it got too large for me to \nhandle at the same time as this change. I'll consider revisiting it \nagain later.\n\n>> The batch is\n>> + selected by greedily adding small pack-files starting with\n>> + the oldest pack-files that fit the size. Create a new pack-\n>> + file containing the objects the multi-pack-index indexes\n>> + into thos pack-files, and rewrite the multi-pack-index to\n> those\n>\n>> + contain that pack-file. A later run of 'git multi-pack-index\n>> + expire' will delete the pack-files that were part of this\n>> + batch.\n> ... but the optimization seems to be rather about getting rid\n> of the oldest packs first instead of getting as close to the batch\n> size. (e.g. another way to look at this is to \"find the permutation\n> of all packs that (each are smaller than batch size), but in sum\n> are the smallest threshold above the batch size).\n\nYou are describing the subset-sum problem, with an additional \noptimization component. While there are dynamic programming approaches \nthat are usually effective (if the sum is small), this problem is \nNP-complete, and hence could lead to complications.\n\n> I guess that the strategy of picking the oldest is just easiest\n> to implement and should be sufficient for now, but memory\n> bounds might be interesting to keep in mind, just as the\n> optimal packing from above.\n\nIt is easy to implement, and fast. Further, we do have a heuristic that \nthe pack modified time correlates with the time the objects were \nintroduced to the repository and hence may compress well when placed \ntogether.\n\nThanks,\n\n-Stolee\n\n"},{"id":"365075","messageId":"113b5ed8-9b25-650b-7f35-6f35bc69075b@gmail.com","threadId":"49993","inReplyTo":"CAGZ79kbPcy2U9XJA+Je0zRxFsQJGA9u8nfYZe_s75V8c97+dNw@mail.gmail.com","subject":"Re: [PATCH 5/5] midx: implement midx_repack()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-12-11T13:00:15Z","receivedAt":"2018-12-11T13:00:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/10/2018 9:32 PM, Stefan Beller wrote:\n> On Mon, Dec 10, 2018 at 10:06 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> From: Derrick Stolee \n>>\n>> To repack using a multi-pack-index, first sort all pack-files by\n>> their modified time. Second, walk those pack-files from oldest\n>> to newest, adding the packs to a list if they are smaller than the\n>> given pack-size. Finally, collect the objects from the multi-pack-\n>> index that are in those packs and send them to 'git pack-objects'.\n> Makes sense.\n>\n> With this operation we only coalesce some packfiles into a new\n> pack file. So to perform the \"complete\" repack this command\n> has to be run repeatedly until there is at most one packfile\n> left that is smaller than batch size.\n\nWell, the batch size essentially means \"If a pack-file is larger than \n, then leave it be. I'm happy with packs that large.\" This assumes \nthat the reason the pack is that large is because it was already \ncombined with other packs or contains a lot of objects.\n\n>\n> Imagine the following scenario:\n>\n> There are 5 packfiles A, B, C, D, E,\n> created last Monday thru Friday (A is oldest, E youngest).\n> The sizes are [A=4, B=6, C=5, D=5, E=4]\n>\n> You'd issue a repack with batch size=10, such that\n> A and B would be repacked into F, which is\n> created today, size is less or equal than 10.\n>\n> You issue another repack tomorrow, which then would\n> coalesce C and D to G, which is\n> dated tomorrow, size is less or equal to 10 as well.\n>\n> You issue a third repack, which then takes E\n> (as it is the oldest) and would probably find F as the\n> next oldest (assuming it is less than 10), to repack\n> into H.\n>\n> H is then compromised of A, B and E, and G is C+D.\n>\n> In a way these repacks, always picking up the oldest,\n> sound like you \"roll forward\" objects into new packs.\n> As the new packs are newest (we have no packs from\n> the future), we'd cycle through different packs to look at\n> for packing on each repacking.\n>\n> It is however more likely that content is more similar\n> on a temporal basis. (e.g. I am boldly claiming that\n> [ABC, DE] would take less space than [ABE, CD]\n> as produced above).\n>\n> (The obvious solution to this hypothetical would be\n> to backdate the resulting pack to the youngest pack\n> that is input to the new pack, but I dislike fudging with\n> the time a file is created/touched, so let's not go there)\n\nThis raises a good point about what happens when we \"roll over\" into the \n\"repacked\" packs.\n\nI'm not claiming that this is an optimal way to save space, but is a way \nto incrementally collect small packs into slightly larger packs, all \nwhile not interrupting concurrent Git commands. Reducing pack count \nimproves data locality, which is my goal here. In our environment, we do \nsee reduced space as a benefit, even if it is not optimal.\n\n>\n> Would the object count make sense as input instead of\n> the pack date?\n>\n>\n>> While first designing a 'git multi-pack-index repack' operation, I\n>> started by collecting the batches based on the size of the objects\n>> instead of the size of the pack-files. This allows repacking a\n>> large pack-file that has very few referencd objects. However, this\n> referenced\n>\n>> came at a significant cost of parsing pack-files instead of simply\n>> reading the multi-pack-index and getting the file information for\n>> the pack-files. This object-size idea could be a direction for\n>> future expansion in this area.\n> Ah, that also explains why the above idea is toast.\n>\n> Would it make sense to extend or annotate the midx file\n> to give hints at which packs are easy to combine?\n>\n> I guess such an \"annotation worker\" could run in a separate\n> thread / pool with the lowest priority as this seems like a\n> decent fallback for the lack of any better information how\n> to pick the packfiles.\n\nOne idea I had earlier (and is in \nDocumentation/technical/multi-pack-index.txt) is to have the midx track \nmetadata about pack-files. We could avoid this \"rollover\" problem by \ntracking which packs were repacked using this mechanism. This could \ncreate a \"pack generation\" value, and we could collect a batch of packs \nthat have the same generation. This does seem a bit overcomplicated for \nthe potential benefit, and could waste better use of that metadata \nconcept. For instance, we could use the metadata to track the \ninformation given by \".keep\" and \".promisor\" files.\n\nThanks,\n\n-Stolee\n\n"},{"id":"365158","messageId":"xmqqbm5rjipo.fsf@gitster-ct.c.googlers.com","threadId":"49993","inReplyTo":"41ef671ec8361a9635dc78c078d2d84e9d985236.1544465177.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] midx: implement midx_repack()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-12T07:40:51Z","receivedAt":"2018-12-12T07:40:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n\nawk is capable of remembering $5 from each line of input, sorting\nthem and picking the second smallest element from it, isn't it?\n\n> +\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n\n... or incrementing the number by one, before reporting, for that\nmatter.\n\n"},{"id":"365225","messageId":"xmqqk1kef42h.fsf@gitster-ct.c.googlers.com","threadId":"49993","inReplyTo":"xmqqbm5rjipo.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 5/5] midx: implement midx_repack()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-13T04:23:02Z","receivedAt":"2018-12-13T04:23:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n>> +\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n>\n> awk is capable of remembering $5 from each line of input, sorting\n> them and picking the second smallest element from it, isn't it?\n>\n>> +\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n>\n> ... or incrementing the number by one, before reporting, for that\n> matter.\n\n\nOops, please disregard. My awk is rusty. Unless we are willing to\nrely on gawk, which we are not, it is not practical to sort inside\nawk.\n\n"},{"id":"365742","messageId":"a697df120c9a4b5bc39fb9a028d4d19ff0559cf3.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] repack: refactor pack deletion for future use","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:37Z","receivedAt":"2018-12-21T16:28:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe repack builtin deletes redundant pack-files and their\nassociated .idx, .promisor, .bitmap, and .keep files. We will want\nto re-use this logic in the future for other types of repack, so\npull the logic into 'unlink_pack_path()' in packfile.c.\n\nThe 'ignore_keep' parameter is enabled for the use in repack, but\nwill be important for a future caller.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 14 ++------------\n packfile.c | 28 ++++++++++++++++++++++++++++\n packfile.h | 7 +++++++\n 3 files changed, 37 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 45583683ee..3d445b34b4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n-\tconst char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n-\tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t plen;\n-\n-\tstrbuf_addf(&buf, \"%s/%s\", dir_name, base_name);\n-\tplen = buf.len;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tstrbuf_setlen(&buf, plen);\n-\t\tstrbuf_addstr(&buf, exts[i]);\n-\t\tunlink(buf.buf);\n-\t}\n+\tstrbuf_addf(&buf, \"%s/%s.pack\", dir_name, base_name);\n+\tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex d1e6683ffe..bacecb4d0d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o)\n \t}\n }\n \n+void unlink_pack_path(const char *pack_name, int force_delete)\n+{\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tint i;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t plen;\n+\n+\tstrbuf_addstr(&buf, pack_name);\n+\tstrip_suffix_mem(buf.buf, &buf.len, \".pack\");\n+\tplen = buf.len;\n+\n+\tif (!force_delete) {\n+\t\tstrbuf_addstr(&buf, \".keep\");\n+\t\tif (!access(buf.buf, F_OK)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tstrbuf_setlen(&buf, plen);\n+\t\tstrbuf_addstr(&buf, exts[i]);\n+\t\tunlink(buf.buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\ndiff --git a/packfile.h b/packfile.h\nindex 6c4037605d..5b7bcdb1dd 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -86,6 +86,13 @@ extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Unlink the .pack and associated extension files.\n+ * Does not unlink if 'force_delete' is false and the pack-file is\n+ * marked as \".keep\".\n+ */\n+extern void unlink_pack_path(const char *pack_name, int force_delete);\n+\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n-- \ngitgitgadget\n\n"},{"id":"365743","messageId":"pull.92.v2.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.git.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:36Z","receivedAt":"2018-12-21T16:28:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index provides a fast way to find an object among a large\nlist of pack-files. It stores a single pack-reference for each object id, so\nduplicate objects are ignored. Among a list of pack-files storing the same\nobject, the most-recently modified one is used.\n\nCreate new subcommands for the multi-pack-index builtin.\n\n * 'git multi-pack-index expire': If we have a pack-file indexed by the\n multi-pack-index, but all objects in that pack are duplicated in\n more-recently modified packs, then delete that pack (and any others like\n it). Delete the reference to that pack in the multi-pack-index.\n \n \n * 'git multi-pack-index repack --batch-size=': Starting from the oldest\n pack-files covered by the multi-pack-index, find those whose on-disk size\n is below the batch size until we have a collection of packs whose sizes\n add up to the batch size. Create a new pack containing all objects that\n the multi-pack-index references to those packs.\n \n \n\nThis allows us to create a new pattern for repacking objects: run 'repack'.\nAfter enough time has passed that all Git commands that started before the\nlast 'repack' are finished, run 'expire' again. This approach has some\nadvantages over the existing \"repack everything\" model:\n\n 1. Incremental. We can repack a small batch of objects at a time, instead\n of repacking all reachable objects. We can also limit ourselves to the\n objects that do not appear in newer pack-files.\n \n \n 2. Highly Available. By adding a new pack-file (and not deleting the old\n pack-files) we do not interrupt concurrent Git commands, and do not\n suffer performance degradation. By expiring only pack-files that have no\n referenced objects, we know that Git commands that are doing normal\n object lookups* will not be interrupted.\n \n \n 3. Note: if someone concurrently runs a Git command that uses\n get_all_packs(), then that command could try to read the pack-files and\n pack-indexes that we are deleting during an expire command. Such\n commands are usually related to object maintenance (i.e. fsck, gc,\n pack-objects) or are related to less-often-used features (i.e.\n fast-import, http-backend, server-info).\n \n \n\nWe plan to use this approach in VFS for Git to do background maintenance of\nthe \"shared object cache\" which is a Git alternate directory filled with\npackfiles containing commits and trees. We currently download pack-files on\nan hourly basis to keep up-to-date with the central server. The cache\nservers supply packs on an hourly and daily basis, so most of the hourly\npacks become useless after a new daily pack is downloaded. The 'expire'\ncommand would clear out most of those packs, but many will still remain with\nfewer than 100 objects remaining. The 'repack' command (with a batch size of\n1-3gb, probably) can condense the remaining packs in commands that run for\n1-3 min at a time. Since the daily packs range from 100-250mb, we will also\ncombine and condense those packs.\n\nUpdates in V2:\n\n * Added a method, unlink_pack_path() to remove packfiles, but with the\n additional check for a .keep file. This borrows logic from \n builtin/repack.c.\n \n \n * Modified documentation and commit messages to replace 'verb' with\n 'subcommand'. Simplified the documentation. (I left 'verbs' in the title\n of the cover letter for consistency.)\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (7):\n repack: refactor pack deletion for future use\n Docs: rearrange subcommands for multi-pack-index\n multi-pack-index: prepare for 'expire' subcommand\n midx: refactor permutation logic\n multi-pack-index: implement 'expire' verb\n multi-pack-index: prepare 'repack' subcommand\n midx: implement midx_repack()\n\n Documentation/git-multi-pack-index.txt | 26 ++-\n builtin/multi-pack-index.c | 12 +-\n builtin/repack.c | 14 +-\n midx.c | 217 +++++++++++++++++++++++--\n midx.h | 2 +\n packfile.c | 28 ++++\n packfile.h | 7 +\n t/t5319-multi-pack-index.sh | 98 +++++++++++\n 8 files changed, 376 insertions(+), 28 deletions(-)\n\n\nbase-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-92%2Fderrickstolee%2Fmidx-expire%2Fupstream-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-92/derrickstolee/midx-expire/upstream-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/92\n\nRange-diff vs v1:\n\n -: ---------- > 1: a697df120c repack: refactor pack deletion for future use\n -: ---------- > 2: 55df6b20ff Docs: rearrange subcommands for multi-pack-index\n 1: 1e34b48a20 ! 3: 2529afe89e multi-pack-index: prepare for 'expire' verb\n @@ -1,6 +1,6 @@\n Author: Derrick Stolee \n \n - multi-pack-index: prepare for 'expire' verb\n + multi-pack-index: prepare for 'expire' subcommand\n \n The multi-pack-index tracks objects in a collection of pack-files.\n Only one copy of each object is indexed, using the modified time\n @@ -8,12 +8,12 @@\n have a pack-file with no referenced objects because all objects\n have a duplicate in a newer pack-file.\n \n - Introduce a new 'expire' verb to the multi-pack-index builtin.\n - This verb will delete these unused pack-files and rewrite the\n + Introduce a new 'expire' subcommand to the multi-pack-index builtin.\n + This subcommand will delete these unused pack-files and rewrite the\n multi-pack-index to no longer refer to those files. More details\n about the specifics will follow as the method is implemented.\n \n - Add a test that verifies the 'expire' verb is correctly wired,\n + Add a test that verifies the 'expire' subcommand is correctly wired,\n but will still be valid when the verb is implemented. Specifically,\n create a set of packs that should all have referenced objects and\n should not be removed during an 'expire' operation.\n @@ -24,16 +24,13 @@\n --- a/Documentation/git-multi-pack-index.txt\n +++ b/Documentation/git-multi-pack-index.txt\n @@\n - \tWhen given as the verb, verify the contents of the MIDX file\n - \tat `/packs/multi-pack-index`.\n + verify::\n + \tVerify the contents of the MIDX file.\n \n +expire::\n -+\tWhen given as the verb, delete the pack-files that are tracked\n -+\tby the MIDX file at `/packs/multi-pack-index` but have\n -+\tno objects referenced by the MIDX. All objects in these pack-\n -+\tfiles have another copy in a more-recently modified pack-file.\n -+\tRewrite the MIDX file afterward to remove all references to\n -+\tthese pack-files.\n ++\tDelete the pack-files that are tracked \tby the MIDX file, but\n ++\thave no objects referenced by the MIDX. Rewrite the MIDX file\n ++\tafterward to remove all references to these pack-files.\n +\n \n EXAMPLES\n 2: 8f496ccb46 = 4: 0c29a242fe midx: refactor permutation logic\n 3: 244bdf2a6f ! 5: 1c4af93f5e multi-pack-index: implement 'expire' verb\n @@ -75,6 +75,7 @@\n +\t\t\t\t\tdrop_index++;\n +\t\t\t\t\ti--;\n +\t\t\t\t\tmissing_drops++;\n ++\t\t\t\t\tcontinue;\n +\t\t\t\t}\n +\t\t\t}\n +\n @@ -114,8 +115,6 @@\n {\n -\treturn 0;\n +\tuint32_t i, *count, result = 0;\n -+\tsize_t dirlen;\n -+\tstruct strbuf buf = STRBUF_INIT;\n +\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n +\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n +\n @@ -128,31 +127,27 @@\n +\t\tcount[pack_int_id]++;\n +\t}\n +\n -+\tstrbuf_addstr(&buf, object_dir);\n -+\tstrbuf_addstr(&buf, \"/pack/\");\n -+\tdirlen = buf.len;\n -+\n +\tfor (i = 0; i < m->num_packs; i++) {\n ++\t\tchar *pack_name;\n ++\n +\t\tif (count[i])\n +\t\t\tcontinue;\n +\n -+\t\tif (m->packs[i]) {\n -+\t\t\tclose_pack(m->packs[i]);\n -+\t\t\tm->packs[i] = NULL;\n -+\t\t}\n ++\t\tif (prepare_midx_pack(m, i))\n ++\t\t\tcontinue;\n +\n -+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n ++\t\tif (m->packs[i]->pack_keep)\n ++\t\t\tcontinue;\n +\n -+\t\tstrbuf_setlen(&buf, dirlen);\n -+\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n -+\t\tunlink(buf.buf);\n ++\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n ++\t\tclose_pack(m->packs[i]);\n ++\t\tFREE_AND_NULL(m->packs[i]);\n +\n -+\t\tstrip_suffix_mem(buf.buf, &buf.len, \"idx\");\n -+\t\tstrbuf_addstr(&buf, \"pack\");\n -+\t\tunlink(buf.buf);\n ++\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n ++\t\tunlink_pack_path(pack_name, 0);\n ++\t\tfree(pack_name);\n +\t}\n +\n -+\tstrbuf_release(&buf);\n +\tfree(count);\n +\n +\tif (packs_to_drop.nr)\n 4: 72b2139591 ! 6: af08e21c97 multi-pack-index: prepare 'repack' verb\n @@ -1,6 +1,6 @@\n Author: Derrick Stolee \n \n - multi-pack-index: prepare 'repack' verb\n + multi-pack-index: prepare 'repack' subcommand\n \n In an environment where the multi-pack-index is useful, it is due\n to many pack-files and an inability to repack the object store\n @@ -10,16 +10,16 @@\n to ensure the object store is highly available and the repack\n operation does not interrupt concurrent git commands.\n \n - Introduce a 'repack' verb to 'git multi-pack-index' that takes a\n - '--batch-size' option. The verb will inspect the multi-pack-index\n - for referenced pack-files whose size is smaller than the batch\n - size, until collecting a list of pack-files whose sizes sum to\n - larger than the batch size. Then, a new pack-file will be created\n - containing the objects from those pack-files that are referenced\n - by the multi-pack-index. The resulting pack is likely to actually\n - be smaller than the batch size due to compression and the fact\n - that there may be objects in the pack-files that have duplicate\n - copies in other pack-files.\n + Introduce a 'repack' subcommand to 'git multi-pack-index' that\n + takes a '--batch-size' option. The verb will inspect the\n + multi-pack-index for referenced pack-files whose size is smaller\n + than the batch size, until collecting a list of pack-files whose\n + sizes sum to larger than the batch size. Then, a new pack-file\n + will be created containing the objects from those pack-files that\n + are referenced by the multi-pack-index. The resulting pack is\n + likely to actually be smaller than the batch size due to\n + compression and the fact that there may be objects in the pack-\n + files that have duplicate copies in other pack-files.\n \n The current change introduces the command-line arguments, and we\n add a test that ensures we parse these options properly. Since\n @@ -32,20 +32,19 @@\n --- a/Documentation/git-multi-pack-index.txt\n +++ b/Documentation/git-multi-pack-index.txt\n @@\n - \tRewrite the MIDX file afterward to remove all references to\n - \tthese pack-files.\n + \thave no objects referenced by the MIDX. Rewrite the MIDX file\n + \tafterward to remove all references to these pack-files.\n \n +repack::\n -+\tWhen given as the verb, collect a batch of pack-files whose\n -+\tsize are all at most the size given by --batch-size, but\n -+\twhose sizes sum to larger than --batch-size. The batch is\n -+\tselected by greedily adding small pack-files starting with\n -+\tthe oldest pack-files that fit the size. Create a new pack-\n -+\tfile containing the objects the multi-pack-index indexes\n -+\tinto thos pack-files, and rewrite the multi-pack-index to\n -+\tcontain that pack-file. A later run of 'git multi-pack-index\n -+\texpire' will delete the pack-files that were part of this\n -+\tbatch.\n ++\tCollect a batch of pack-files whose size are all at most the\n ++\tsize given by --batch-size, but whose sizes sum to larger\n ++\tthan --batch-size. The batch is selected by greedily adding\n ++\tsmall pack-files starting with the oldest pack-files that fit\n ++\tthe size. Create a new pack-file containing the objects the\n ++\tmulti-pack-index indexes into those pack-files, and rewrite\n ++\tthe multi-pack-index to contain that pack-file. A later run\n ++\tof 'git multi-pack-index expire' will delete the pack-files\n ++\tthat were part of this batch.\n +\n \n EXAMPLES\n @@ -123,7 +122,7 @@\n \t)\n '\n \n -+test_expect_success 'repack does not create any packs' '\n ++test_expect_success 'repack with minimum size does not alter existing packs' '\n +\t(\n +\t\tcd dup &&\n +\t\tls .git/objects/pack >expect &&\n 5: 41ef671ec8 = 7: bef7aa007c midx: implement midx_repack()\n\n-- \ngitgitgadget\n"},{"id":"365744","messageId":"55df6b20ffdad67f9cb8020f264c1420d381bb9a.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] Docs: rearrange subcommands for multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:38Z","receivedAt":"2018-12-21T16:28:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe will add new subcommands to the multi-pack-index, and that will\nmake the documentation a bit messier. Clean up the 'verb'\ndescriptions by renaming the concept to 'subcommand' and removing\nthe reference to the object directory.\n\nHelped-by: Stefan Beller \nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex f7778a2c85..1af406aca2 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] \n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -23,13 +23,13 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+The following subcommands are available:\n+\n write::\n-\tWhen given as the verb, write a new MIDX file to\n-\t`/packs/multi-pack-index`.\n+\tWrite a new MIDX file.\n \n verify::\n-\tWhen given as the verb, verify the contents of the MIDX file\n-\tat `/packs/multi-pack-index`.\n+\tVerify the contents of the MIDX file.\n \n \n EXAMPLES\n-- \ngitgitgadget\n\n"},{"id":"365745","messageId":"2529afe89e7917b54b1d0f51a1a009f64fabaad7.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] multi-pack-index: prepare for 'expire' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:39Z","receivedAt":"2018-12-21T16:28:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe multi-pack-index tracks objects in a collection of pack-files.\nOnly one copy of each object is indexed, using the modified time\nof the pack-files to determine tie-breakers. It is possible to\nhave a pack-file with no referenced objects because all objects\nhave a duplicate in a newer pack-file.\n\nIntroduce a new 'expire' subcommand to the multi-pack-index builtin.\nThis subcommand will delete these unused pack-files and rewrite the\nmulti-pack-index to no longer refer to those files. More details\nabout the specifics will follow as the method is implemented.\n\nAdd a test that verifies the 'expire' subcommand is correctly wired,\nbut will still be valid when the verb is implemented. Specifically,\ncreate a set of packs that should all have referenced objects and\nshould not be removed during an 'expire' operation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 5 +++\n builtin/multi-pack-index.c | 4 ++-\n midx.c | 5 +++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 47 ++++++++++++++++++++++++++\n 5 files changed, 61 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 1af406aca2..6186c4c936 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -31,6 +31,11 @@ write::\n verify::\n \tVerify the contents of the MIDX file.\n \n+expire::\n+\tDelete the pack-files that are tracked \tby the MIDX file, but\n+\thave no objects referenced by the MIDX. Rewrite the MIDX file\n+\tafterward to remove all references to these pack-files.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex fca70f8e4f..145de3a46c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,7 +5,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n \tNULL\n };\n \n@@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(opts.object_dir);\n+\tif (!strcmp(argv[0], \"expire\"))\n+\t\treturn expire_midx_packs(opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 730ff84dff..bb825ef816 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir)\n \n \treturn verify_midx_error;\n }\n+\n+int expire_midx_packs(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 774f652530..e3a2b740b5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n+int expire_midx_packs(const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 70926b5bc0..948effc1ee 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -348,4 +348,51 @@ test_expect_success 'verify incorrect 64-bit offset' '\n \t\t\"incorrect object offset\"\n '\n \n+test_expect_success 'setup expire tests' '\n+\tmkdir dup &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit init &&\n+\t\tfor i in $(test_seq 1 20)\n+\t\tdo\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit branch A HEAD &&\n+\t\tgit branch B HEAD~8 &&\n+\t\tgit branch C HEAD~13 &&\n+\t\tgit branch D HEAD~16 &&\n+\t\tgit branch E HEAD~18 &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n+test_expect_success 'expire does not remove any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"365746","messageId":"af08e21c974f72c96f4892571e9e617137a94c2a.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] multi-pack-index: prepare 'repack' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:42Z","receivedAt":"2018-12-21T16:28:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment where the multi-pack-index is useful, it is due\nto many pack-files and an inability to repack the object store\ninto a single pack-file. However, it is likely that many of these\npack-files are rather small, and could be repacked into a slightly\nlarger pack-file without too much effort. It may also be important\nto ensure the object store is highly available and the repack\noperation does not interrupt concurrent git commands.\n\nIntroduce a 'repack' subcommand to 'git multi-pack-index' that\ntakes a '--batch-size' option. The verb will inspect the\nmulti-pack-index for referenced pack-files whose size is smaller\nthan the batch size, until collecting a list of pack-files whose\nsizes sum to larger than the batch size. Then, a new pack-file\nwill be created containing the objects from those pack-files that\nare referenced by the multi-pack-index. The resulting pack is\nlikely to actually be smaller than the batch size due to\ncompression and the fact that there may be objects in the pack-\nfiles that have duplicate copies in other pack-files.\n\nThe current change introduces the command-line arguments, and we\nadd a test that ensures we parse these options properly. Since\nwe specify a small batch size, we will guarantee that future\nimplementations do not change the list of pack-files.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 11 +++++++++++\n builtin/multi-pack-index.c | 10 +++++++++-\n midx.c | 5 +++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 11 +++++++++++\n 5 files changed, 37 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 6186c4c936..cc63531cc0 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -36,6 +36,17 @@ expire::\n \thave no objects referenced by the MIDX. Rewrite the MIDX file\n \tafterward to remove all references to these pack-files.\n \n+repack::\n+\tCollect a batch of pack-files whose size are all at most the\n+\tsize given by --batch-size, but whose sizes sum to larger\n+\tthan --batch-size. The batch is selected by greedily adding\n+\tsmall pack-files starting with the oldest pack-files that fit\n+\tthe size. Create a new pack-file containing the objects the\n+\tmulti-pack-index indexes into those pack-files, and rewrite\n+\tthe multi-pack-index to contain that pack-file. A later run\n+\tof 'git multi-pack-index expire' will delete the pack-files\n+\tthat were part of this batch.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 145de3a46c..d87a2235e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,12 +5,13 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n \tNULL\n };\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tunsigned long batch_size;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n@@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n+\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n@@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn 1;\n \t}\n \n+\tif (!strcmp(argv[0], \"repack\"))\n+\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n+\tif (opts.batch_size)\n+\t\tdie(_(\"--batch-size option is only for 'repack' verb\"));\n+\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\ndiff --git a/midx.c b/midx.c\nindex 043cd1fd97..127c43f7b0 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1110,3 +1110,8 @@ int expire_midx_packs(const char *object_dir)\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex e3a2b740b5..394a21ee96 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n int expire_midx_packs(const char *object_dir);\n+int midx_repack(const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 210279a3cf..f675621080 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -410,4 +410,15 @@ test_expect_success 'expire removes unreferenced packs' '\n \t)\n '\n \n+test_expect_success 'repack with minimum size does not alter existing packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n+\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"365747","messageId":"1c4af93f5e9833d74a7cf7fc6d38df7fd93c1acc.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] multi-pack-index: implement 'expire' verb","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:41Z","receivedAt":"2018-12-21T16:28:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git multi-pack-index expire' command looks at the existing\nmult-pack-index, counts the number of objects referenced in each\npack-file, deletes the pack-fils with no referenced objects, and\nrewrites the multi-pack-index to no longer reference those packs.\n\nRefactor the write_midx_file() method to call write_midx_internal()\nwhich now takes an existing 'struct multi_pack_index' and a list\nof pack-files to drop (as specified by the names of their pack-\nindexes). As we write the new multi-pack-index, we drop those\nfile names from the list of known pack-files.\n\nThe expire_midx_packs() method removes the unreferenced pack-files\nafter carefully closing the packs to avoid open handles.\n\nTest that a new pack-file that covers the contents of two other\npack-files leads to those pack-files being deleted during the\nexpire command. Be sure to read the multi-pack-index to ensure\nit no longer references those packs.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 82 +++++++++++++++++++++++++++++++++++--\n t/t5319-multi-pack-index.sh | 15 +++++++\n 2 files changed, 93 insertions(+), 4 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3bd7183a53..043cd1fd97 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -751,7 +751,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n-int write_midx_file(const char *object_dir)\n+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+\t\t\t struct string_list *packs_to_drop)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -765,6 +766,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n+\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -773,7 +775,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n-\tpacks.m = load_multi_pack_index(object_dir, 1);\n+\tif (m)\n+\t\tpacks.m = m;\n+\telse\n+\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n \tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n@@ -787,7 +792,25 @@ int write_midx_file(const char *object_dir)\n \tALLOC_ARRAY(packs.perm, packs.alloc_perm);\n \n \tif (packs.m) {\n+\t\tint drop_index = 0, missing_drops = 0;\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n+\t\t\tif (packs_to_drop && drop_index < packs_to_drop->nr) {\n+\t\t\t\tint cmp = strcmp(packs.m->pack_names[i],\n+\t\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\t\tif (!cmp) {\n+\t\t\t\t\tdrop_index++;\n+\t\t\t\t\tcontinue;\n+\t\t\t\t} else if (cmp > 0) {\n+\t\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t\tdrop_index++;\n+\t\t\t\t\ti--;\n+\t\t\t\t\tmissing_drops++;\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t}\n+\n \t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n \t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n \t\t\tALLOC_GROW(packs.perm, packs.nr + 1, packs.alloc_perm);\n@@ -798,6 +821,12 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n \t\t\tpacks.nr++;\n \t\t}\n+\n+\t\tif (packs_to_drop && (drop_index < packs_to_drop->nr || missing_drops)) {\n+\t\t\terror(_(\"did not see all pack-files to drop\"));\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n \t}\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n@@ -932,7 +961,12 @@ int write_midx_file(const char *object_dir)\n \tfree(packs.perm);\n \tfree(entries);\n \tfree(midx_name);\n-\treturn 0;\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1034,5 +1068,45 @@ int verify_midx_file(const char *object_dir)\n \n int expire_midx_packs(const char *object_dir)\n {\n-\treturn 0;\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t}\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\t\tFREE_AND_NULL(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 948effc1ee..210279a3cf 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -395,4 +395,19 @@ test_expect_success 'expire does not remove any packs' '\n \t)\n '\n \n+test_expect_success 'expire removes unreferenced packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"365748","messageId":"bef7aa007c572a46c0eeb8790e1814a21ae10024.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] midx: implement midx_repack()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:42Z","receivedAt":"2018-12-21T16:28:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo repack using a multi-pack-index, first sort all pack-files by\ntheir modified time. Second, walk those pack-files from oldest\nto newest, adding the packs to a list if they are smaller than the\ngiven pack-size. Finally, collect the objects from the multi-pack-\nindex that are in those packs and send them to 'git pack-objects'.\n\nWhile first designing a 'git multi-pack-index repack' operation, I\nstarted by collecting the batches based on the size of the objects\ninstead of the size of the pack-files. This allows repacking a\nlarge pack-file that has very few referencd objects. However, this\ncame at a significant cost of parsing pack-files instead of simply\nreading the multi-pack-index and getting the file information for\nthe pack-files. This object-size idea could be a direction for\nfuture expansion in this area.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 109 +++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 25 +++++++++\n 2 files changed, 133 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 127c43f7b0..f6bc111438 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -8,6 +8,7 @@\n #include \"sha1-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n+#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -1111,7 +1112,113 @@ int expire_midx_packs(const char *object_dir)\n \treturn result;\n }\n \n-int midx_repack(const char *object_dir, size_t batch_size)\n+struct time_and_id {\n+\ttimestamp_t mtime;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n {\n+\tconst struct time_and_id *a, *b;\n+\n+\ta = (const struct time_and_id *)a_;\n+\tb = (const struct time_and_id *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n \treturn 0;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\tint result = 0;\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct time_and_id *pack_ti;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n+\tpack_ti = xcalloc(m->num_packs, sizeof(struct time_and_id));\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_ti[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_ti[i].mtime = m->packs[i]->mtime;\n+\t}\n+\tQSORT(pack_ti, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_ti[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (p->pack_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += p->pack_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tif (total_size < batch_size || packs_to_repack < 2)\n+\t\tgoto cleanup;\n+\n+\targv_array_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\targv_array_push(&cmd.args, base_name.buf);\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\txwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);\n+\t\txwrite(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, m, NULL);\n+\tm = NULL;\n+\n+cleanup:\n+\tif (m)\n+\t\tclose_midx(m);\n+\tfree(include_pack);\n+\tfree(pack_ti);\n+\treturn result;\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex f675621080..3f5e9ea653 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -421,4 +421,29 @@ test_expect_success 'repack with minimum size does not alter existing packs' '\n \t)\n '\n \n+test_expect_success 'repack creates a new pack' '\n+\t(\n+\t\tcd dup &&\n+\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n+\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 5 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 5 midx-list\n+\t)\n+'\n+\n+test_expect_success 'expire removes repacked packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls -S .git/objects/pack/*pack | head -n 3 >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/*pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 3 midx-list\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"365749","messageId":"0c29a242fec32b0077291cb36a2831e6202b3453.1545409716.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] midx: refactor permutation logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-21T16:28:40Z","receivedAt":"2018-12-21T16:28:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen writing a multi-pack-index, we keep track of an integer\npermutation, tracking the list of pack-files that we know about\n(both from the existing multi-pack-index and the new pack-files\nbeing introduced) and converting them into a sorted order for\nthe new multi-pack-index.\n\nIn anticipation of dropping pack-files from the existing multi-\npack-index, refactor the logic around how we track this permutation.\n\nFirst, insert the permutation into the pack_list structure. This\nallows us to grow the permutation dynamically as we add packs.\n\nSecond, fill the permutation with values corresponding to their\nposition in the list of pack-files, sorted as follows:\n\n 1. The pack-files in the existing multi-pack-index,\n sorted lexicographically.\n\n 2. The pack-files not in the existing multi-pack-index,\n sorted as discovered from the filesystem.\n\nThere is a subtle thing in how we initialize this permutation,\nspecifically how we use 'i' for the initial value. This will\nmatter more when we implement the logic for dropping existing\npacks, as we will create holes in the ordering.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 20 +++++++++++++-------\n 1 file changed, 13 insertions(+), 7 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bb825ef816..3bd7183a53 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -380,9 +380,11 @@ static size_t write_midx_header(struct hashfile *f,\n struct pack_list {\n \tstruct packed_git **list;\n \tchar **names;\n+\tuint32_t *perm;\n \tuint32_t nr;\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n+\tuint32_t alloc_perm;\n \tsize_t pack_name_concat_len;\n \tstruct multi_pack_index *m;\n };\n@@ -398,6 +400,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n \t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n+\t\tALLOC_GROW(packs->perm, packs->nr + 1, packs->alloc_perm);\n \n \t\tpacks->list[packs->nr] = add_packed_git(full_path,\n \t\t\t\t\t\t\tfull_path_len,\n@@ -417,6 +420,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\treturn;\n \t\t}\n \n+\t\tpacks->perm[packs->nr] = packs->nr;\n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n \t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n@@ -443,7 +447,7 @@ static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *p\n \tALLOC_ARRAY(pairs, nr_packs);\n \n \tfor (i = 0; i < nr_packs; i++) {\n-\t\tpairs[i].pack_int_id = i;\n+\t\tpairs[i].pack_int_id = perm[i];\n \t\tpairs[i].pack_name = pack_names[i];\n \t}\n \n@@ -755,7 +759,6 @@ int write_midx_file(const char *object_dir)\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n-\tuint32_t *pack_perm = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n@@ -774,18 +777,22 @@ int write_midx_file(const char *object_dir)\n \n \tpacks.nr = 0;\n \tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n-\tpacks.alloc_names = packs.alloc_list;\n+\tpacks.alloc_perm = packs.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.names = NULL;\n+\tpacks.perm = NULL;\n \tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n+\tALLOC_ARRAY(packs.perm, packs.alloc_perm);\n \n \tif (packs.m) {\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n \t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n \t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\t\t\tALLOC_GROW(packs.perm, packs.nr + 1, packs.alloc_perm);\n \n+\t\t\tpacks.perm[packs.nr] = i;\n \t\t\tpacks.list[packs.nr] = NULL;\n \t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n \t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n@@ -802,10 +809,9 @@ int write_midx_file(const char *object_dir)\n \t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n+\tsort_packs_by_name(packs.names, packs.nr, packs.perm);\n \n-\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.list, packs.perm, packs.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -923,8 +929,8 @@ int write_midx_file(const char *object_dir)\n \n \tfree(packs.list);\n \tfree(packs.names);\n+\tfree(packs.perm);\n \tfree(entries);\n-\tfree(pack_perm);\n \tfree(midx_name);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"366415","messageId":"pull.92.v3.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 0/9] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:10Z","receivedAt":"2019-01-09T15:21:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index provides a fast way to find an object among a large\nlist of pack-files. It stores a single pack-reference for each object id, so\nduplicate objects are ignored. Among a list of pack-files storing the same\nobject, the most-recently modified one is used.\n\nCreate new subcommands for the multi-pack-index builtin.\n\n * 'git multi-pack-index expire': If we have a pack-file indexed by the\n multi-pack-index, but all objects in that pack are duplicated in\n more-recently modified packs, then delete that pack (and any others like\n it). Delete the reference to that pack in the multi-pack-index.\n \n \n * 'git multi-pack-index repack --batch-size=': Starting from the oldest\n pack-files covered by the multi-pack-index, find those whose on-disk size\n is below the batch size until we have a collection of packs whose sizes\n add up to the batch size. Create a new pack containing all objects that\n the multi-pack-index references to those packs.\n \n \n\nThis allows us to create a new pattern for repacking objects: run 'repack'.\nAfter enough time has passed that all Git commands that started before the\nlast 'repack' are finished, run 'expire' again. This approach has some\nadvantages over the existing \"repack everything\" model:\n\n 1. Incremental. We can repack a small batch of objects at a time, instead\n of repacking all reachable objects. We can also limit ourselves to the\n objects that do not appear in newer pack-files.\n \n \n 2. Highly Available. By adding a new pack-file (and not deleting the old\n pack-files) we do not interrupt concurrent Git commands, and do not\n suffer performance degradation. By expiring only pack-files that have no\n referenced objects, we know that Git commands that are doing normal\n object lookups* will not be interrupted.\n \n \n 3. Note: if someone concurrently runs a Git command that uses\n get_all_packs(), then that command could try to read the pack-files and\n pack-indexes that we are deleting during an expire command. Such\n commands are usually related to object maintenance (i.e. fsck, gc,\n pack-objects) or are related to less-often-used features (i.e.\n fast-import, http-backend, server-info).\n \n \n\nWe plan to use this approach in VFS for Git to do background maintenance of\nthe \"shared object cache\" which is a Git alternate directory filled with\npackfiles containing commits and trees. We currently download pack-files on\nan hourly basis to keep up-to-date with the central server. The cache\nservers supply packs on an hourly and daily basis, so most of the hourly\npacks become useless after a new daily pack is downloaded. The 'expire'\ncommand would clear out most of those packs, but many will still remain with\nfewer than 100 objects remaining. The 'repack' command (with a batch size of\n1-3gb, probably) can condense the remaining packs in commands that run for\n1-3 min at a time. Since the daily packs range from 100-250mb, we will also\ncombine and condense those packs.\n\nUpdates in V2:\n\n * Added a method, unlink_pack_path() to remove packfiles, but with the\n additional check for a .keep file. This borrows logic from \n builtin/repack.c.\n \n \n * Modified documentation and commit messages to replace 'verb' with\n 'subcommand'. Simplified the documentation. (I left 'verbs' in the title\n of the cover letter for consistency.)\n \n \n\nUpdates in V3:\n\n * There was a bug in the expire logic when simultaneously removing packs\n and adding uncovered packs, specifically around the pack permutation.\n This was hard to see during review because I was using the 'pack_perm'\n array for multiple purposes. First, I was reducing its length, and then I\n was adding to it and resorting. In V3, I significantly overhauled the\n logic here, which required some extra commits before implementing\n 'expire'. The final commit includes a test that would cover this case.\n\nThanks, -Stolee\n\nDerrick Stolee (9):\n repack: refactor pack deletion for future use\n Docs: rearrange subcommands for multi-pack-index\n multi-pack-index: prepare for 'expire' subcommand\n midx: simplify computation of pack name lengths\n midx: refactor permutation logic and pack sorting\n multi-pack-index: implement 'expire' verb\n multi-pack-index: prepare 'repack' subcommand\n midx: implement midx_repack()\n multi-pack-index: test expire while adding packs\n\n Documentation/git-multi-pack-index.txt | 26 +-\n builtin/multi-pack-index.c | 12 +-\n builtin/repack.c | 14 +-\n midx.c | 393 ++++++++++++++++++-------\n midx.h | 2 +\n packfile.c | 28 ++\n packfile.h | 7 +\n t/t5319-multi-pack-index.sh | 133 +++++++++\n 8 files changed, 497 insertions(+), 118 deletions(-)\n\n\nbase-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-92%2Fderrickstolee%2Fmidx-expire%2Fupstream-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-92/derrickstolee/midx-expire/upstream-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/92\n\nRange-diff vs v2:\n\n 1: a697df120c = 1: 62b393b816 repack: refactor pack deletion for future use\n 2: 55df6b20ff = 2: 7886785904 Docs: rearrange subcommands for multi-pack-index\n 3: 2529afe89e = 3: f06382b4ae multi-pack-index: prepare for 'expire' subcommand\n 4: 0c29a242fe < -: ---------- midx: refactor permutation logic\n -: ---------- > 4: 2a763990ae midx: simplify computation of pack name lengths\n -: ---------- > 5: a0d4cc6cb3 midx: refactor permutation logic and pack sorting\n 5: 1c4af93f5e ! 6: 4dbff40e7a multi-pack-index: implement 'expire' verb\n @@ -26,6 +26,62 @@\n diff --git a/midx.c b/midx.c\n --- a/midx.c\n +++ b/midx.c\n +@@\n + #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n + #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n + \n ++#define PACK_EXPIRED UINT_MAX\n ++\n + static char *get_midx_filename(const char *object_dir)\n + {\n + \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n +@@\n + \tuint32_t orig_pack_int_id;\n + \tchar *pack_name;\n + \tstruct packed_git *p;\n ++\tunsigned expired : 1;\n + };\n + \n + static int pack_info_compare(const void *_a, const void *_b)\n +@@\n + \n + \t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n + \t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n ++\t\tpacks->info[packs->nr].expired = 0;\n + \t\tpacks->nr++;\n + \t}\n + }\n +@@\n + \tsize_t written = 0;\n + \n + \tfor (i = 0; i < num_packs; i++) {\n +-\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n ++\t\tsize_t writelen;\n ++\n ++\t\tif (info[i].expired)\n ++\t\t\tcontinue;\n + \n + \t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n + \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n + \t\t\t info[i - 1].pack_name,\n + \t\t\t info[i].pack_name);\n + \n ++\t\twritelen = strlen(info[i].pack_name) + 1;\n + \t\thashwrite(f, info[i].pack_name, writelen);\n + \t\twritten += writelen;\n + \t}\n +@@\n + \tfor (i = 0; i < nr_objects; i++) {\n + \t\tstruct pack_midx_entry *obj = list++;\n + \n ++\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n ++\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n ++\t\t\t oid_to_hex(&obj->oid),\n ++\t\t\t obj->pack_int_id);\n ++\n + \t\thashwrite_be32(f, perm[obj->pack_int_id]);\n + \n + \t\tif (large_offset_needed && obj->offset >> 31)\n @@\n \treturn written;\n }\n @@ -37,9 +93,10 @@\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n @@\n - \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n + \tint pack_name_concat_len = 0;\n ++\tint dropped_packs = 0;\n +\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n @@ -55,49 +112,87 @@\n +\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n - \tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n + \tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n @@\n - \tALLOC_ARRAY(packs.perm, packs.alloc_perm);\n - \n - \tif (packs.m) {\n -+\t\tint drop_index = 0, missing_drops = 0;\n - \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n -+\t\t\tif (packs_to_drop && drop_index < packs_to_drop->nr) {\n -+\t\t\t\tint cmp = strcmp(packs.m->pack_names[i],\n -+\t\t\t\t\t\t packs_to_drop->items[drop_index].string);\n -+\n -+\t\t\t\tif (!cmp) {\n -+\t\t\t\t\tdrop_index++;\n -+\t\t\t\t\tcontinue;\n -+\t\t\t\t} else if (cmp > 0) {\n -+\t\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n -+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n -+\t\t\t\t\tdrop_index++;\n -+\t\t\t\t\ti--;\n -+\t\t\t\t\tmissing_drops++;\n -+\t\t\t\t\tcontinue;\n -+\t\t\t\t}\n -+\t\t\t}\n -+\n - \t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n - \t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n - \t\t\tALLOC_GROW(packs.perm, packs.nr + 1, packs.alloc_perm);\n -@@\n - \t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n + \t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n + \t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n + \t\t\tpacks.info[packs.nr].p = NULL;\n ++\t\t\tpacks.info[packs.nr].expired = 0;\n \t\t\tpacks.nr++;\n \t\t}\n + \t}\n + \n + \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n + \n +-\tif (packs.m && packs.nr == packs.m->num_packs)\n ++\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n + \t\tgoto cleanup;\n + \n + \tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n +@@\n + \n + \tQSORT(packs.info, packs.nr, pack_info_compare);\n + \n ++\tif (packs_to_drop && packs_to_drop->nr) {\n ++\t\tint drop_index = 0;\n ++\t\tint missing_drops = 0;\n ++\n ++\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n ++\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n ++\t\t\t\t\t packs_to_drop->items[drop_index].string);\n ++\n ++\t\t\tif (!cmp) {\n ++\t\t\t\tdrop_index++;\n ++\t\t\t\tpacks.info[i].expired = 1;\n ++\t\t\t} else if (cmp > 0) {\n ++\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n ++\t\t\t\t packs_to_drop->items[drop_index].string);\n ++\t\t\t\tdrop_index++;\n ++\t\t\t\tmissing_drops++;\n ++\t\t\t\ti--;\n ++\t\t\t} else {\n ++\t\t\t\tpacks.info[i].expired = 0;\n ++\t\t\t}\n ++\t\t}\n +\n -+\t\tif (packs_to_drop && (drop_index < packs_to_drop->nr || missing_drops)) {\n -+\t\t\terror(_(\"did not see all pack-files to drop\"));\n ++\t\tif (missing_drops) {\n +\t\t\tresult = 1;\n +\t\t\tgoto cleanup;\n ++\t\t}\n ++\t}\n ++\n + \tALLOC_ARRAY(pack_perm, packs.nr);\n + \tfor (i = 0; i < packs.nr; i++) {\n +-\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n ++\t\tif (packs.info[i].expired) {\n ++\t\t\tdropped_packs++;\n ++\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n ++\t\t} else {\n ++\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n +\t\t}\n \t}\n \n - \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n +-\tfor (i = 0; i < packs.nr; i++)\n +-\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n ++\tfor (i = 0; i < packs.nr; i++) {\n ++\t\tif (!packs.info[i].expired)\n ++\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n ++\t}\n + \n + \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n + \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n +@@\n + \tcur_chunk = 0;\n + \tnum_chunks = large_offsets_needed ? 5 : 4;\n + \n +-\twritten = write_midx_header(f, num_chunks, packs.nr);\n ++\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n + \n + \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n + \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n @@\n - \tfree(packs.perm);\n \tfree(entries);\n + \tfree(pack_perm);\n \tfree(midx_name);\n -\treturn 0;\n +\treturn result;\n @@ -175,7 +270,10 @@\n +\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n +\t\tgit multi-pack-index expire &&\n +\t\tls .git/objects/pack >actual &&\n -+\t\ttest_cmp expect actual\n ++\t\ttest_cmp expect actual &&\n ++\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n ++\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n ++\t\ttest_cmp expect-idx actual-midx\n +\t)\n +'\n +\n 6: af08e21c97 = 7: b39f90ad09 multi-pack-index: prepare 'repack' subcommand\n 7: bef7aa007c = 8: a4c2d5a8e1 midx: implement midx_repack()\n -: ---------- > 9: b97fb35ba9 multi-pack-index: test expire while adding packs\n\n-- \ngitgitgadget\n"},{"id":"366416","messageId":"62b393b816d65a578c7a0ff9b40f01e0637e4d67.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 1/9] repack: refactor pack deletion for future use","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:12Z","receivedAt":"2019-01-09T15:21:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe repack builtin deletes redundant pack-files and their\nassociated .idx, .promisor, .bitmap, and .keep files. We will want\nto re-use this logic in the future for other types of repack, so\npull the logic into 'unlink_pack_path()' in packfile.c.\n\nThe 'ignore_keep' parameter is enabled for the use in repack, but\nwill be important for a future caller.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 14 ++------------\n packfile.c | 28 ++++++++++++++++++++++++++++\n packfile.h | 7 +++++++\n 3 files changed, 37 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 45583683ee..3d445b34b4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n-\tconst char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n-\tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t plen;\n-\n-\tstrbuf_addf(&buf, \"%s/%s\", dir_name, base_name);\n-\tplen = buf.len;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tstrbuf_setlen(&buf, plen);\n-\t\tstrbuf_addstr(&buf, exts[i]);\n-\t\tunlink(buf.buf);\n-\t}\n+\tstrbuf_addf(&buf, \"%s/%s.pack\", dir_name, base_name);\n+\tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex d1e6683ffe..bacecb4d0d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o)\n \t}\n }\n \n+void unlink_pack_path(const char *pack_name, int force_delete)\n+{\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tint i;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t plen;\n+\n+\tstrbuf_addstr(&buf, pack_name);\n+\tstrip_suffix_mem(buf.buf, &buf.len, \".pack\");\n+\tplen = buf.len;\n+\n+\tif (!force_delete) {\n+\t\tstrbuf_addstr(&buf, \".keep\");\n+\t\tif (!access(buf.buf, F_OK)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tstrbuf_setlen(&buf, plen);\n+\t\tstrbuf_addstr(&buf, exts[i]);\n+\t\tunlink(buf.buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\ndiff --git a/packfile.h b/packfile.h\nindex 6c4037605d..5b7bcdb1dd 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -86,6 +86,13 @@ extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Unlink the .pack and associated extension files.\n+ * Does not unlink if 'force_delete' is false and the pack-file is\n+ * marked as \".keep\".\n+ */\n+extern void unlink_pack_path(const char *pack_name, int force_delete);\n+\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n-- \ngitgitgadget\n\n"},{"id":"366417","messageId":"f06382b4aece30a3fdcd0823e870ee1a49ca34f6.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 3/9] multi-pack-index: prepare for 'expire' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:14Z","receivedAt":"2019-01-09T15:21:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe multi-pack-index tracks objects in a collection of pack-files.\nOnly one copy of each object is indexed, using the modified time\nof the pack-files to determine tie-breakers. It is possible to\nhave a pack-file with no referenced objects because all objects\nhave a duplicate in a newer pack-file.\n\nIntroduce a new 'expire' subcommand to the multi-pack-index builtin.\nThis subcommand will delete these unused pack-files and rewrite the\nmulti-pack-index to no longer refer to those files. More details\nabout the specifics will follow as the method is implemented.\n\nAdd a test that verifies the 'expire' subcommand is correctly wired,\nbut will still be valid when the verb is implemented. Specifically,\ncreate a set of packs that should all have referenced objects and\nshould not be removed during an 'expire' operation.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 5 +++\n builtin/multi-pack-index.c | 4 ++-\n midx.c | 5 +++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 47 ++++++++++++++++++++++++++\n 5 files changed, 61 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 1af406aca2..6186c4c936 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -31,6 +31,11 @@ write::\n verify::\n \tVerify the contents of the MIDX file.\n \n+expire::\n+\tDelete the pack-files that are tracked \tby the MIDX file, but\n+\thave no objects referenced by the MIDX. Rewrite the MIDX file\n+\tafterward to remove all references to these pack-files.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex fca70f8e4f..145de3a46c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,7 +5,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n \tNULL\n };\n \n@@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(opts.object_dir);\n+\tif (!strcmp(argv[0], \"expire\"))\n+\t\treturn expire_midx_packs(opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 730ff84dff..bb825ef816 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir)\n \n \treturn verify_midx_error;\n }\n+\n+int expire_midx_packs(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 774f652530..e3a2b740b5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n+int expire_midx_packs(const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 70926b5bc0..948effc1ee 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -348,4 +348,51 @@ test_expect_success 'verify incorrect 64-bit offset' '\n \t\t\"incorrect object offset\"\n '\n \n+test_expect_success 'setup expire tests' '\n+\tmkdir dup &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit init &&\n+\t\tfor i in $(test_seq 1 20)\n+\t\tdo\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit branch A HEAD &&\n+\t\tgit branch B HEAD~8 &&\n+\t\tgit branch C HEAD~13 &&\n+\t\tgit branch D HEAD~16 &&\n+\t\tgit branch E HEAD~18 &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n+test_expect_success 'expire does not remove any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"366418","messageId":"2a763990ae3842582e9dd20000f6df992d9e8e2d.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 4/9] midx: simplify computation of pack name lengths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:14Z","receivedAt":"2019-01-09T15:21:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore writing the multi-pack-index, we compute the length of the\npack-index names concatenated together. This forms the data in the\npack name chunk, and we precompute it to compute chunk offsets.\nThe value is also modified to fit alignment needs.\n\nPreviously, this computation was coupled with adding packs from\nthe existing multi-pack-index and the remaining packs in the object\ndir not already covered by the multi-pack-index.\n\nIn anticipation of this becoming more complicated with the 'expire'\ncommand, simplify the computation by centralizing it to a single\nloop before writing the file.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bb825ef816..f087bbbe82 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -383,7 +383,6 @@ struct pack_list {\n \tuint32_t nr;\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n-\tsize_t pack_name_concat_len;\n \tstruct multi_pack_index *m;\n };\n \n@@ -418,7 +417,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t}\n \n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n-\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n \t}\n }\n@@ -762,6 +760,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n+\tint pack_name_concat_len = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -777,7 +776,6 @@ int write_midx_file(const char *object_dir)\n \tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.names = NULL;\n-\tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n@@ -788,7 +786,6 @@ int write_midx_file(const char *object_dir)\n \n \t\t\tpacks.list[packs.nr] = NULL;\n \t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -798,10 +795,6 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n-\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n-\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n@@ -814,6 +807,13 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tfor (i = 0; i < packs.nr; i++)\n+\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -831,7 +831,7 @@ int write_midx_file(const char *object_dir)\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-- \ngitgitgadget\n\n"},{"id":"366419","messageId":"4dbff40e7ad653d9c5c3b19dd63a345a21a73dbd.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 6/9] multi-pack-index: implement 'expire' verb","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:16Z","receivedAt":"2019-01-09T15:21:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git multi-pack-index expire' command looks at the existing\nmult-pack-index, counts the number of objects referenced in each\npack-file, deletes the pack-fils with no referenced objects, and\nrewrites the multi-pack-index to no longer reference those packs.\n\nRefactor the write_midx_file() method to call write_midx_internal()\nwhich now takes an existing 'struct multi_pack_index' and a list\nof pack-files to drop (as specified by the names of their pack-\nindexes). As we write the new multi-pack-index, we drop those\nfile names from the list of known pack-files.\n\nThe expire_midx_packs() method removes the unreferenced pack-files\nafter carefully closing the packs to avoid open handles.\n\nTest that a new pack-file that covers the contents of two other\npack-files leads to those pack-files being deleted during the\nexpire command. Be sure to read the multi-pack-index to ensure\nit no longer references those packs.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 120 +++++++++++++++++++++++++++++++++---\n t/t5319-multi-pack-index.sh | 18 ++++++\n 2 files changed, 128 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 7ae5275c25..6ccbec3f19 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,8 @@\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n+#define PACK_EXPIRED UINT_MAX\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -381,6 +383,7 @@ struct pack_info {\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n+\tunsigned expired : 1;\n };\n \n static int pack_info_compare(const void *_a, const void *_b)\n@@ -428,6 +431,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n \t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n+\t\tpacks->info[packs->nr].expired = 0;\n \t\tpacks->nr++;\n \t}\n }\n@@ -587,13 +591,17 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n+\t\tsize_t writelen;\n+\n+\t\tif (info[i].expired)\n+\t\t\tcontinue;\n \n \t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n \t\t\t info[i - 1].pack_name,\n \t\t\t info[i].pack_name);\n \n+\t\twritelen = strlen(info[i].pack_name) + 1;\n \t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n@@ -675,6 +683,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n+\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->pack_int_id);\n+\n \t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n@@ -721,7 +734,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n-int write_midx_file(const char *object_dir)\n+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+\t\t\t struct string_list *packs_to_drop)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -737,6 +751,8 @@ int write_midx_file(const char *object_dir)\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n+\tint dropped_packs = 0;\n+\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -745,7 +761,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n-\tpacks.m = load_multi_pack_index(object_dir, 1);\n+\tif (m)\n+\t\tpacks.m = m;\n+\telse\n+\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n \tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n@@ -759,13 +778,14 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n \t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n \t\t\tpacks.info[packs.nr].p = NULL;\n+\t\t\tpacks.info[packs.nr].expired = 0;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs)\n+\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n \tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n@@ -779,13 +799,48 @@ int write_midx_file(const char *object_dir)\n \n \tQSORT(packs.info, packs.nr, pack_info_compare);\n \n+\tif (packs_to_drop && packs_to_drop->nr) {\n+\t\tint drop_index = 0;\n+\t\tint missing_drops = 0;\n+\n+\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\tif (!cmp) {\n+\t\t\t\tdrop_index++;\n+\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t} else if (cmp > 0) {\n+\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\tdrop_index++;\n+\t\t\t\tmissing_drops++;\n+\t\t\t\ti--;\n+\t\t\t} else {\n+\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (missing_drops) {\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t\tif (packs.info[i].expired) {\n+\t\t\tdropped_packs++;\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t} else {\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tif (!packs.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -801,7 +856,7 @@ int write_midx_file(const char *object_dir)\n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -902,7 +957,12 @@ int write_midx_file(const char *object_dir)\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-\treturn 0;\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1004,5 +1064,45 @@ int verify_midx_file(const char *object_dir)\n \n int expire_midx_packs(const char *object_dir)\n {\n-\treturn 0;\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t}\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\t\tFREE_AND_NULL(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 948effc1ee..f55a60a89c 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -395,4 +395,22 @@ test_expect_success 'expire does not remove any packs' '\n \t)\n '\n \n+test_expect_success 'expire removes unreferenced packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n+\t\ttest_cmp expect-idx actual-midx\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"366420","messageId":"a0d4cc6cb3fbca0de732dfd34cbec4d765b55228.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 5/9] midx: refactor permutation logic and pack sorting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:15Z","receivedAt":"2019-01-09T15:21:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of the expire subcommand, refactor the way we sort\nthe packfiles by name. This will greatly simplify our approach to\ndropping expired packs from the list.\n\nFirst, create 'struct pack_info' to replace 'struct pack_pair'.\nThis struct contains the necessary information about a pack,\nincluding its name, a pointer to its packfile struct (if not\nalready in the multi-pack-index), and the original pack-int-id.\n\nSecond, track the pack information using an array of pack_info\nstructs in the pack_list struct. This simplifies the logic around\nthe multiple arrays we were tracking in that struct.\n\nFinally, update get_sorted_entries() to not permute the pack-int-id\nand instead supply the permutation to write_midx_object_offsets().\nThis requires sorting the packs after get_sorted_entries().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 150 ++++++++++++++++++++++++---------------------------------\n 1 file changed, 63 insertions(+), 87 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f087bbbe82..7ae5275c25 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -377,12 +377,23 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_info {\n+\tuint32_t orig_pack_int_id;\n+\tchar *pack_name;\n+\tstruct packed_git *p;\n+};\n+\n+static int pack_info_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_info *a = (struct pack_info *)_a;\n+\tstruct pack_info *b = (struct pack_info *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n struct pack_list {\n-\tstruct packed_git **list;\n-\tchar **names;\n+\tstruct pack_info *info;\n \tuint32_t nr;\n-\tuint32_t alloc_list;\n-\tuint32_t alloc_names;\n+\tuint32_t alloc;\n \tstruct multi_pack_index *m;\n };\n \n@@ -395,66 +406,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n-\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n+\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n \n-\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n-\t\t\t\t\t\t\tfull_path_len,\n-\t\t\t\t\t\t\t0);\n+\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->list[packs->nr]) {\n+\t\tif (!packs->info[packs->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->list[packs->nr]);\n-\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t\tclose_pack(packs->info[packs->nr].p);\n+\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n \t\tpacks->nr++;\n \t}\n }\n \n-struct pack_pair {\n-\tuint32_t pack_int_id;\n-\tchar *pack_name;\n-};\n-\n-static int pack_pair_compare(const void *_a, const void *_b)\n-{\n-\tstruct pack_pair *a = (struct pack_pair *)_a;\n-\tstruct pack_pair *b = (struct pack_pair *)_b;\n-\treturn strcmp(a->pack_name, b->pack_name);\n-}\n-\n-static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n-{\n-\tuint32_t i;\n-\tstruct pack_pair *pairs;\n-\n-\tALLOC_ARRAY(pairs, nr_packs);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpairs[i].pack_int_id = i;\n-\t\tpairs[i].pack_name = pack_names[i];\n-\t}\n-\n-\tQSORT(pairs, nr_packs, pack_pair_compare);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpack_names[i] = pairs[i].pack_name;\n-\t\tperm[pairs[i].pack_int_id] = i;\n-\t}\n-\n-\tfree(pairs);\n-}\n-\n struct pack_midx_entry {\n \tstruct object_id oid;\n \tuint32_t pack_int_id;\n@@ -480,7 +457,6 @@ static int midx_oid_compare(const void *_a, const void *_b)\n }\n \n static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n-\t\t\t\t uint32_t *pack_perm,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n@@ -488,7 +464,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n-\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n \te->offset = nth_midxed_offset(m, pos);\n \n \t/* consider objects in midx to be from \"old\" packs */\n@@ -522,8 +498,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * of a packfile containing the object).\n */\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n-\t\t\t\t\t\t struct packed_git **p,\n-\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n {\n@@ -534,7 +509,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \tuint32_t start_pack = m ? m->num_packs : 0;\n \n \tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n-\t\ttotal_objects += p[cur_pack]->num_objects;\n+\t\ttotal_objects += info[cur_pack].p->num_objects;\n \n \t/*\n \t * As we de-duplicate by fanout value, we expect the fanout\n@@ -559,7 +534,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n \t\t\t\tnr_fanout++;\n@@ -570,12 +545,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n-\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n-\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(info[cur_pack].p, cur_fanout);\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -604,7 +579,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n }\n \n static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t char **pack_names,\n+\t\t\t\t struct pack_info *info,\n \t\t\t\t uint32_t num_packs)\n {\n \tuint32_t i;\n@@ -612,14 +587,14 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n \n-\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t pack_names[i - 1],\n-\t\t\t pack_names[i]);\n+\t\t\t info[i - 1].pack_name,\n+\t\t\t info[i].pack_name);\n \n-\t\thashwrite(f, pack_names[i], writelen);\n+\t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -690,6 +665,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n }\n \n static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tuint32_t *perm,\n \t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n {\n \tstruct pack_midx_entry *list = objects;\n@@ -699,7 +675,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\thashwrite_be32(f, obj->pack_int_id);\n+\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -772,20 +748,17 @@ int write_midx_file(const char *object_dir)\n \tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n-\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n-\tpacks.alloc_names = packs.alloc_list;\n-\tpacks.list = NULL;\n-\tpacks.names = NULL;\n-\tALLOC_ARRAY(packs.list, packs.alloc_list);\n-\tALLOC_ARRAY(packs.names, packs.alloc_names);\n+\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n+\tpacks.info = NULL;\n+\tALLOC_ARRAY(packs.info, packs.alloc);\n \n \tif (packs.m) {\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n-\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n \n-\t\t\tpacks.list[packs.nr] = NULL;\n-\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n+\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].p = NULL;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -795,10 +768,7 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n-\n-\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -807,8 +777,15 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t}\n+\n \tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -879,7 +856,7 @@ int write_midx_file(const char *object_dir)\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -891,7 +868,7 @@ int write_midx_file(const char *object_dir)\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -914,15 +891,14 @@ int write_midx_file(const char *object_dir)\n \n cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.list[i]) {\n-\t\t\tclose_pack(packs.list[i]);\n-\t\t\tfree(packs.list[i]);\n+\t\tif (packs.info[i].p) {\n+\t\t\tclose_pack(packs.info[i].p);\n+\t\t\tfree(packs.info[i].p);\n \t\t}\n-\t\tfree(packs.names[i]);\n+\t\tfree(packs.info[i].pack_name);\n \t}\n \n-\tfree(packs.list);\n-\tfree(packs.names);\n+\tfree(packs.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \ngitgitgadget\n\n"},{"id":"366421","messageId":"a4c2d5a8e16f888f9a3d1090fa92d986919f0162.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 8/9] midx: implement midx_repack()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:18Z","receivedAt":"2019-01-09T15:21:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo repack using a multi-pack-index, first sort all pack-files by\ntheir modified time. Second, walk those pack-files from oldest\nto newest, adding the packs to a list if they are smaller than the\ngiven pack-size. Finally, collect the objects from the multi-pack-\nindex that are in those packs and send them to 'git pack-objects'.\n\nWhile first designing a 'git multi-pack-index repack' operation, I\nstarted by collecting the batches based on the size of the objects\ninstead of the size of the pack-files. This allows repacking a\nlarge pack-file that has very few referencd objects. However, this\ncame at a significant cost of parsing pack-files instead of simply\nreading the multi-pack-index and getting the file information for\nthe pack-files. This object-size idea could be a direction for\nfuture expansion in this area.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 109 +++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 25 +++++++++\n 2 files changed, 133 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 30ff4430ab..f0f16690e6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -8,6 +8,7 @@\n #include \"sha1-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n+#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -1107,7 +1108,113 @@ int expire_midx_packs(const char *object_dir)\n \treturn result;\n }\n \n-int midx_repack(const char *object_dir, size_t batch_size)\n+struct time_and_id {\n+\ttimestamp_t mtime;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n {\n+\tconst struct time_and_id *a, *b;\n+\n+\ta = (const struct time_and_id *)a_;\n+\tb = (const struct time_and_id *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n \treturn 0;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\tint result = 0;\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct time_and_id *pack_ti;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n+\tpack_ti = xcalloc(m->num_packs, sizeof(struct time_and_id));\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_ti[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_ti[i].mtime = m->packs[i]->mtime;\n+\t}\n+\tQSORT(pack_ti, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_ti[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (p->pack_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += p->pack_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tif (total_size < batch_size || packs_to_repack < 2)\n+\t\tgoto cleanup;\n+\n+\targv_array_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\targv_array_push(&cmd.args, base_name.buf);\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\txwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);\n+\t\txwrite(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, m, NULL);\n+\tm = NULL;\n+\n+cleanup:\n+\tif (m)\n+\t\tclose_midx(m);\n+\tfree(include_pack);\n+\tfree(pack_ti);\n+\treturn result;\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex d64767600a..99afb5ec51 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -424,4 +424,29 @@ test_expect_success 'repack with minimum size does not alter existing packs' '\n \t)\n '\n \n+test_expect_success 'repack creates a new pack' '\n+\t(\n+\t\tcd dup &&\n+\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n+\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 5 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 5 midx-list\n+\t)\n+'\n+\n+test_expect_success 'expire removes repacked packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls -S .git/objects/pack/*pack | head -n 3 >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/*pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 3 midx-list\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"366422","messageId":"b97fb35ba995a3688cbac752ded2bd785188edfb.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 9/9] multi-pack-index: test expire while adding packs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:19Z","receivedAt":"2019-01-09T15:21:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring development of the multi-pack-index expire subcommand, a\nversion went out that improperly computed the pack order if a new\npack was introduced while other packs were being removed. Part of\nthe subtlety of the bug involved the new pack being placed before\nother packs that already existed in the multi-pack-index.\n\nAdd a test to t5319-multi-pack-index.sh that catches this issue.\nThe test adds new packs that cause another pack to be expired, and\ncreates new packs that are lexicographically sorted before and\nafter the existing packs.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 99afb5ec51..1213549d25 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -449,4 +449,36 @@ test_expect_success 'expire removes repacked packs' '\n \t)\n '\n \n+test_expect_success 'expire works when adding new packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/a-pack <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/z-pack <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack/ | grep idx >expect &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"366423","messageId":"b39f90ad09265614efd466fee5089354a193ae8c.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:17Z","receivedAt":"2019-01-09T15:21:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment where the multi-pack-index is useful, it is due\nto many pack-files and an inability to repack the object store\ninto a single pack-file. However, it is likely that many of these\npack-files are rather small, and could be repacked into a slightly\nlarger pack-file without too much effort. It may also be important\nto ensure the object store is highly available and the repack\noperation does not interrupt concurrent git commands.\n\nIntroduce a 'repack' subcommand to 'git multi-pack-index' that\ntakes a '--batch-size' option. The verb will inspect the\nmulti-pack-index for referenced pack-files whose size is smaller\nthan the batch size, until collecting a list of pack-files whose\nsizes sum to larger than the batch size. Then, a new pack-file\nwill be created containing the objects from those pack-files that\nare referenced by the multi-pack-index. The resulting pack is\nlikely to actually be smaller than the batch size due to\ncompression and the fact that there may be objects in the pack-\nfiles that have duplicate copies in other pack-files.\n\nThe current change introduces the command-line arguments, and we\nadd a test that ensures we parse these options properly. Since\nwe specify a small batch size, we will guarantee that future\nimplementations do not change the list of pack-files.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 11 +++++++++++\n builtin/multi-pack-index.c | 10 +++++++++-\n midx.c | 5 +++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 11 +++++++++++\n 5 files changed, 37 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 6186c4c936..cc63531cc0 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -36,6 +36,17 @@ expire::\n \thave no objects referenced by the MIDX. Rewrite the MIDX file\n \tafterward to remove all references to these pack-files.\n \n+repack::\n+\tCollect a batch of pack-files whose size are all at most the\n+\tsize given by --batch-size, but whose sizes sum to larger\n+\tthan --batch-size. The batch is selected by greedily adding\n+\tsmall pack-files starting with the oldest pack-files that fit\n+\tthe size. Create a new pack-file containing the objects the\n+\tmulti-pack-index indexes into those pack-files, and rewrite\n+\tthe multi-pack-index to contain that pack-file. A later run\n+\tof 'git multi-pack-index expire' will delete the pack-files\n+\tthat were part of this batch.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 145de3a46c..d87a2235e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,12 +5,13 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n \tNULL\n };\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tunsigned long batch_size;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n@@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n+\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n@@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn 1;\n \t}\n \n+\tif (!strcmp(argv[0], \"repack\"))\n+\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n+\tif (opts.batch_size)\n+\t\tdie(_(\"--batch-size option is only for 'repack' verb\"));\n+\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\ndiff --git a/midx.c b/midx.c\nindex 6ccbec3f19..30ff4430ab 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1106,3 +1106,8 @@ int expire_midx_packs(const char *object_dir)\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex e3a2b740b5..394a21ee96 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n int expire_midx_packs(const char *object_dir);\n+int midx_repack(const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex f55a60a89c..d64767600a 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -413,4 +413,15 @@ test_expect_success 'expire removes unreferenced packs' '\n \t)\n '\n \n+test_expect_success 'repack with minimum size does not alter existing packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n+\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"366424","messageId":"78867859042eea6318da65e236f903d6102ed93b.1547047269.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 2/9] Docs: rearrange subcommands for multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-09T15:21:13Z","receivedAt":"2019-01-09T15:21:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe will add new subcommands to the multi-pack-index, and that will\nmake the documentation a bit messier. Clean up the 'verb'\ndescriptions by renaming the concept to 'subcommand' and removing\nthe reference to the object directory.\n\nHelped-by: Stefan Beller \nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex f7778a2c85..1af406aca2 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] \n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -23,13 +23,13 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+The following subcommands are available:\n+\n write::\n-\tWhen given as the verb, write a new MIDX file to\n-\t`/packs/multi-pack-index`.\n+\tWrite a new MIDX file.\n \n verify::\n-\tWhen given as the verb, verify the contents of the MIDX file\n-\tat `/packs/multi-pack-index`.\n+\tVerify the contents of the MIDX file.\n \n \n EXAMPLES\n-- \ngitgitgadget\n\n"},{"id":"366425","messageId":"20190109155441.GH4673@szeder.dev","threadId":"49993","inReplyTo":"4dbff40e7ad653d9c5c3b19dd63a345a21a73dbd.1547047269.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/9] multi-pack-index: implement 'expire' verb","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-09T15:54:41Z","receivedAt":"2019-01-09T15:54:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 09, 2019 at 07:21:16AM -0800, Derrick Stolee via GitGitGadget wrote:\n> The 'git multi-pack-index expire' command ...\n\nThe subject line could use a s/verb/subcommand/.\n\n"},{"id":"366426","messageId":"20190109155606.GI4673@szeder.dev","threadId":"49993","inReplyTo":"b39f90ad09265614efd466fee5089354a193ae8c.1547047269.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-09T15:56:06Z","receivedAt":"2019-01-09T15:56:12Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 09, 2019 at 07:21:17AM -0800, Derrick Stolee via GitGitGadget wrote:\n> Introduce a 'repack' subcommand to 'git multi-pack-index' that\n> takes a '--batch-size' option. The verb will inspect the\n\ns/verb/subcommand/\n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 145de3a46c..d87a2235e3 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n\n> @@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> \t\treturn 1;\n> \t}\n> \n> +\tif (!strcmp(argv[0], \"repack\"))\n> +\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n> +\tif (opts.batch_size)\n> +\t\tdie(_(\"--batch-size option is only for 'repack' verb\"));\n\nLikewise.\n\n"},{"id":"366511","messageId":"xmqqtvig2xru.fsf@gitster-ct.c.googlers.com","threadId":"49993","inReplyTo":"20190109155441.GH4673@szeder.dev","subject":"Re: [PATCH v3 6/9] multi-pack-index: implement 'expire' verb","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-10T18:05:57Z","receivedAt":"2019-01-10T18:06:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> On Wed, Jan 09, 2019 at 07:21:16AM -0800, Derrick Stolee via GitGitGadget wrote:\n>> The 'git multi-pack-index expire' command ...\n>\n> The subject line could use a s/verb/subcommand/.\n\nYeah, that probably is more in line with the existing terminology\nfor other Git commands.\n"},{"id":"366997","messageId":"3aa0a7ea-6c30-2c61-0815-2b9ab8304564@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/9] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-17T15:27:01Z","receivedAt":"2019-01-17T15:27:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"I've seen comments about some leftover uses of the word 'verb' instead \nof 'subcommand'. I can definitely clean those up, but I'm hoping that \nmore comments could be made on the technical aspects of this series.\n\nThanks,\n-Stolee\n"},{"id":"367480","messageId":"20190123210054.118647-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"a0d4cc6cb3fbca0de732dfd34cbec4d765b55228.1547047269.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/9] midx: refactor permutation logic and pack sorting","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T21:00:54Z","receivedAt":"2019-01-23T21:01:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Following Stolee's wishes [1], I'll stick to the technical aspects here.\nPatches 1-4 look correct technically to me, so let me start here.\n\n[1] https://public-inbox.org/git/3aa0a7ea-6c30-2c61-0815-2b9ab8304564@gmail.com/\n\n> +struct pack_info {\n> +\tuint32_t orig_pack_int_id;\n> +\tchar *pack_name;\n> +\tstruct packed_git *p;\n> +};\n> +\n> +static int pack_info_compare(const void *_a, const void *_b)\n> +{\n> +\tstruct pack_info *a = (struct pack_info *)_a;\n> +\tstruct pack_info *b = (struct pack_info *)_b;\n> +\treturn strcmp(a->pack_name, b->pack_name);\n> +}\n> +\n> struct pack_list {\n> -\tstruct packed_git **list;\n> -\tchar **names;\n> +\tstruct pack_info *info;\n> \tuint32_t nr;\n> -\tuint32_t alloc_list;\n> -\tuint32_t alloc_names;\n> +\tuint32_t alloc;\n> \tstruct multi_pack_index *m;\n> };\n\n\"list\" and \"names\" look like a parallel array, but as far as I can tell,\nthey are actually not, because \"names\" is eventually sorted but not\n\"list\". So combining these two arrays is great, and removes a lot of\nconfusion.\n\nNow the packed_git list will be sorted too when the names are sorted; I\nwas surprised that code did not need to be changed to take this into\naccount, but I see now that it's because the struct packed_gits don't\nneed to be referenced anymore after get_sorted_entries().\n\n> -struct pack_pair {\n> -\tuint32_t pack_int_id;\n> -\tchar *pack_name;\n> -};\n> -\n> -static int pack_pair_compare(const void *_a, const void *_b)\n> -{\n> -\tstruct pack_pair *a = (struct pack_pair *)_a;\n> -\tstruct pack_pair *b = (struct pack_pair *)_b;\n> -\treturn strcmp(a->pack_name, b->pack_name);\n> -}\n> -\n> -static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n> -{\n> -\tuint32_t i;\n> -\tstruct pack_pair *pairs;\n> -\n> -\tALLOC_ARRAY(pairs, nr_packs);\n> -\n> -\tfor (i = 0; i < nr_packs; i++) {\n> -\t\tpairs[i].pack_int_id = i;\n> -\t\tpairs[i].pack_name = pack_names[i];\n> -\t}\n> -\n> -\tQSORT(pairs, nr_packs, pack_pair_compare);\n> -\n> -\tfor (i = 0; i < nr_packs; i++) {\n> -\t\tpack_names[i] = pairs[i].pack_name;\n> -\t\tperm[pairs[i].pack_int_id] = i;\n> -\t}\n> -\n> -\tfree(pairs);\n> -}\n\nIt's nice that this gets removed.\n\n> static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n> -\t\t\t\t uint32_t *pack_perm,\n> \t\t\t\t struct pack_midx_entry *e,\n> \t\t\t\t uint32_t pos)\n> {\n> @@ -488,7 +464,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n> \t\treturn 1;\n> \n> \tnth_midxed_object_oid(&e->oid, m, pos);\n> -\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n> +\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n> \te->offset = nth_midxed_offset(m, pos);\n\nnth_midxed_pack_midx_entry() is only called from get_sorted_entries(),\nwhich is now called *before* any sorting of pack_info, so there is no\nlonger any need for considering pack_perm. As to why\nget_sorted_entries() needs to be called before sorting of pack_info and\nnot after, that question will be answered later.\n\n> @@ -522,8 +498,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n> * of a packfile containing the object).\n> */\n> static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> -\t\t\t\t\t\t struct packed_git **p,\n> -\t\t\t\t\t\t uint32_t *perm,\n> +\t\t\t\t\t\t struct pack_info *info,\n> \t\t\t\t\t\t uint32_t nr_packs,\n> \t\t\t\t\t\t uint32_t *nr_objects)\n\n[snip rest of get_sorted_entries]\n\nAs stated above, get_sorted_entries() is now called before any sorting\nof pack_info, so there is no need for perm.\n\n> static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n> +\t\t\t\t\tuint32_t *perm,\n> \t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n> {\n> \tstruct pack_midx_entry *list = objects;\n> @@ -699,7 +675,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n> \tfor (i = 0; i < nr_objects; i++) {\n> \t\tstruct pack_midx_entry *obj = list++;\n> \n> -\t\thashwrite_be32(f, obj->pack_int_id);\n> +\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n> \n> \t\tif (large_offset_needed && obj->offset >> 31)\n> \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n\nThis was unexpected to me - why is perm introduced here when it wasn't\nneeded before? This is because get_sorted_entries() is now called before\nany sorting of pack_info. get_sorted_entries() generates entries using\nthe old pack_int_id, but the resulting packfile is written using new\npack numbers, so when writing the entries to the file, they must be\nwritten using the new numbers.\n\nThe mapping of perm is: perm[old number] = new number.\n\n> @@ -795,10 +768,7 @@ int write_midx_file(const char *object_dir)\n> \tif (packs.m && packs.nr == packs.m->num_packs)\n> \t\tgoto cleanup;\n> \n> -\tALLOC_ARRAY(pack_perm, packs.nr);\n> -\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n> -\n> -\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n> +\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n> \n> \tfor (i = 0; i < nr_entries; i++) {\n> \t\tif (entries[i].offset > 0x7fffffff)\n> @@ -807,8 +777,15 @@ int write_midx_file(const char *object_dir)\n> \t\t\tlarge_offsets_needed = 1;\n> \t}\n> \n> +\tQSORT(packs.info, packs.nr, pack_info_compare);\n> +\n> +\tALLOC_ARRAY(pack_perm, packs.nr);\n> +\tfor (i = 0; i < packs.nr; i++) {\n> +\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n> +\t}\n> +\n> \tfor (i = 0; i < packs.nr; i++)\n> -\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n> +\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n> \n> \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n> \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n\nIndeed, the sorting of pack_info is moved to after get_sorted_entries().\nAlso, pack_perm[old number] = new number, as expected.\n\nI think a comment explaining why the perm is needed would be helpful -\nsomething explaining that the entries were generated using the old pack\nnumbers, so we need this mapping to be able to write them using the new\nnumbers.\n\nAs part of this review, I did attempt to make everything use the sorted\npack_info order, but failed. I got as far as making get_sorted_entries()\nalways use start_pack=0 and skip the pack_infos that didn't have the p\npointer (to match the existing behavior of get_sorted_entries() that\nonly operates on new pack_infos being added by add_pack_to_midx), but it\nstill didn't work, and I didn't investigate further.\n"},{"id":"367491","messageId":"20190123221354.136961-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"4dbff40e7ad653d9c5c3b19dd63a345a21a73dbd.1547047269.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/9] multi-pack-index: implement 'expire' verb","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T22:13:54Z","receivedAt":"2019-01-23T22:14:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> The 'git multi-pack-index expire' command looks at the existing\n> mult-pack-index, counts the number of objects referenced in each\n> pack-file, deletes the pack-fils with no referenced objects, and\n> rewrites the multi-pack-index to no longer reference those packs.\n\nThanks - this was quite straightforwardly written.\n\n> @@ -745,7 +761,10 @@ int write_midx_file(const char *object_dir)\n> \t\t\t midx_name);\n> \t}\n> \n> -\tpacks.m = load_multi_pack_index(object_dir, 1);\n> +\tif (m)\n> +\t\tpacks.m = m;\n> +\telse\n> +\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n\nIf we already loaded the m, we can just pass it in - OK.\n\n> +\tif (packs_to_drop && packs_to_drop->nr) {\n> +\t\tint drop_index = 0;\n> +\t\tint missing_drops = 0;\n> +\n> +\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n> +\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n> +\t\t\t\t\t packs_to_drop->items[drop_index].string);\n> +\n> +\t\t\tif (!cmp) {\n> +\t\t\t\tdrop_index++;\n> +\t\t\t\tpacks.info[i].expired = 1;\n> +\t\t\t} else if (cmp > 0) {\n> +\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n> +\t\t\t\t packs_to_drop->items[drop_index].string);\n> +\t\t\t\tdrop_index++;\n> +\t\t\t\tmissing_drops++;\n> +\t\t\t\ti--;\n> +\t\t\t} else {\n> +\t\t\t\tpacks.info[i].expired = 0;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (missing_drops) {\n> +\t\t\tresult = 1;\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\t}\n\nThis takes into account that packfiles can shift while we run this\ncommand, I see. Other than that, this is a common pattern - how we\niterate through 2 sorted arrays, one a subsequence of each other.\n\nAnd indeed packs_to_drop is a sorted list, because we use\nstring_list_insert() below.\n\n> \tALLOC_ARRAY(pack_perm, packs.nr);\n> \tfor (i = 0; i < packs.nr; i++) {\n> -\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n> +\t\tif (packs.info[i].expired) {\n> +\t\t\tdropped_packs++;\n> +\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n> +\t\t} else {\n> +\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n> +\t\t}\n\nHere...\n\n> \t}\n> \n> -\tfor (i = 0; i < packs.nr; i++)\n> -\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n> +\tfor (i = 0; i < packs.nr; i++) {\n> +\t\tif (!packs.info[i].expired)\n> +\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n> +\t}\n\n...and here and elsewhere, we have to contend with the fact that\npacks.info has pack_info that we don't want to write. I think it would\nbe slightly better to filter out the expired ones from packs.info, and\nthen when generating pack_perm, first memset it to 0xff. This way, we\nwouldn't have to check expiry everywhere. But I don't feel too strongly\nabout this.\n\n> int expire_midx_packs(const char *object_dir)\n> {\n> -\treturn 0;\n> +\tuint32_t i, *count, result = 0;\n> +\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n> +\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n> +\n> +\tif (!m)\n> +\t\treturn 0;\n> +\n> +\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n> +\tfor (i = 0; i < m->num_objects; i++) {\n> +\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n> +\t\tcount[pack_int_id]++;\n> +\t}\n> +\n> +\tfor (i = 0; i < m->num_packs; i++) {\n> +\t\tchar *pack_name;\n> +\n> +\t\tif (count[i])\n> +\t\t\tcontinue;\n> +\n> +\t\tif (prepare_midx_pack(m, i))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (m->packs[i]->pack_keep)\n> +\t\t\tcontinue;\n> +\n> +\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n> +\t\tclose_pack(m->packs[i]);\n> +\t\tFREE_AND_NULL(m->packs[i]);\n> +\n> +\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n> +\t\tunlink_pack_path(pack_name, 0);\n> +\t\tfree(pack_name);\n> +\t}\n> +\n> +\tfree(count);\n> +\n> +\tif (packs_to_drop.nr)\n> +\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n> +\n> +\tstring_list_clear(&packs_to_drop, 0);\n> +\treturn result;\n> }\n\nThis is as I expected - unlink all the files we don't want, and even\nthough much of the midx hasn't changed, we still need to write it\nbecause it has a new list of packfiles.\n\n> +test_expect_success 'expire removes unreferenced packs' '\n> +\t(\n> +\t\tcd dup &&\n> +\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n> +\t\trefs/heads/A\n> +\t\t^refs/heads/C\n> +\t\tEOF\n> +\t\tgit multi-pack-index write &&\n> +\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n> +\t\tgit multi-pack-index expire &&\n> +\t\tls .git/objects/pack >actual &&\n> +\t\ttest_cmp expect actual &&\n> +\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n> +\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n> +\t\ttest_cmp expect-idx actual-midx\n> +\t)\n> +'\n\nMaybe add a fsck at the end for sanity's sake. Also, I think that\npreservation of .keep packfiles is an important feature, and maybe worth\na test.\n"},{"id":"367492","messageId":"20190123223318.146120-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"a4c2d5a8e16f888f9a3d1090fa92d986919f0162.1547047269.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 8/9] midx: implement midx_repack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T22:33:18Z","receivedAt":"2019-01-23T22:33:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> To repack using a multi-pack-index, first sort all pack-files by\n> their modified time. Second, walk those pack-files from oldest\n> to newest, adding the packs to a list if they are smaller than the\n> given pack-size. Finally, collect the objects from the multi-pack-\n> index that are in those packs and send them to 'git pack-objects'.\n\nAlso mention that we stop once the total is at least the batch size.\n\n> +test_expect_success 'repack creates a new pack' '\n> +\t(\n> +\t\tcd dup &&\n> +\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n> +\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n> +\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n> +\t\tls .git/objects/pack/*idx >idx-list &&\n> +\t\ttest_line_count = 5 idx-list &&\n> +\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n> +\t\ttest_line_count = 5 midx-list\n> +\t)\n> +'\n\nCan there be a test_line_count of the output of ls at the beginning, so\nthat we do not have to look at the previous test to indeed see that the\n5 is greater than before?\n\nAlso add a test to check what happens when we have 3 packs below the\nbatch size, but taking 2 of them together is sufficient to exceed the\nbatch size.\n\n> +test_expect_success 'expire removes repacked packs' '\n> +\t(\n> +\t\tcd dup &&\n> +\t\tls -S .git/objects/pack/*pack | head -n 3 >expect &&\n> +\t\tgit multi-pack-index expire &&\n> +\t\tls -S .git/objects/pack/*pack >actual &&\n> +\t\ttest_cmp expect actual &&\n> +\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n> +\t\ttest_line_count = 3 midx-list\n> +\t)\n> +'\n\nSame comment as above.\n"},{"id":"367493","messageId":"20190123223845.147378-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"b39f90ad09265614efd466fee5089354a193ae8c.1547047269.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T22:38:45Z","receivedAt":"2019-01-23T22:38:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index 6186c4c936..cc63531cc0 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -36,6 +36,17 @@ expire::\n> \thave no objects referenced by the MIDX. Rewrite the MIDX file\n> \tafterward to remove all references to these pack-files.\n> \n> +repack::\n> +\tCollect a batch of pack-files whose size are all at most the\n> +\tsize given by --batch-size, but whose sizes sum to larger\n> +\tthan --batch-size. The batch is selected by greedily adding\n> +\tsmall pack-files starting with the oldest pack-files that fit\n> +\tthe size. Create a new pack-file containing the objects the\n> +\tmulti-pack-index indexes into those pack-files, and rewrite\n> +\tthe multi-pack-index to contain that pack-file. A later run\n> +\tof 'git multi-pack-index expire' will delete the pack-files\n> +\tthat were part of this batch.\n\nI see in the subsequent patch that you stop once the batch size is\nmatched or exceeded - I see that you mention \"whose sizes sum to larger\nthan --batch-size\", but this leads me to think that if the total so\nhappens to not exceed the batch size, don't do anything, but otherwise\nrepack *all* the small packs together.\n\nI would write this as:\n\n Create a new packfile containing the objects in the N least-sized\n packfiles referenced by the multi-pack-index, where N is the smallest\n number such that the total size of the packfiles equals or exceeds the\n given batch size. Rewrite the multi-pack-index to reference the new\n packfile instead of the N packfiles. A later run of ...\n"},{"id":"367494","messageId":"20190123224411.149354-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/9] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T22:44:11Z","receivedAt":"2019-01-23T22:44:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> The multi-pack-index provides a fast way to find an object among a large\n> list of pack-files. It stores a single pack-reference for each object id, so\n> duplicate objects are ignored. Among a list of pack-files storing the same\n> object, the most-recently modified one is used.\n> \n> Create new subcommands for the multi-pack-index builtin.\n> \n> * 'git multi-pack-index expire': If we have a pack-file indexed by the\n> multi-pack-index, but all objects in that pack are duplicated in\n> more-recently modified packs, then delete that pack (and any others like\n> it). Delete the reference to that pack in the multi-pack-index.\n> \n> \n> * 'git multi-pack-index repack --batch-size=': Starting from the oldest\n> pack-files covered by the multi-pack-index, find those whose on-disk size\n> is below the batch size until we have a collection of packs whose sizes\n> add up to the batch size. Create a new pack containing all objects that\n> the multi-pack-index references to those packs.\n\n[snip]\n\nThanks - as you further explain in the snipped part, this is very useful\nfor users of repositories that use MIDX.\n\nI only have minor comments (that I have written in individual replies)\nand the series overall looks good to me.\n\nI personally would have squashed patches 3 and 7 (the \"prepare for\"\npatches) into the patches that implement the respective commands,\nbecause I'd rather not have points where the commands don't work. Having\nsaid that, rebase is probably not going to be affected, so I don't feel\nstrongly about this.\n"},{"id":"367565","messageId":"14fe05d0-75e5-0975-d490-550539dbb34a@gmail.com","threadId":"49993","inReplyTo":"20190123210054.118647-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 5/9] midx: refactor permutation logic and pack sorting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-24T17:34:23Z","receivedAt":"2019-01-24T17:34:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/23/2019 4:00 PM, Jonathan Tan wrote:\n> Indeed, the sorting of pack_info is moved to after get_sorted_entries().\n> Also, pack_perm[old number] = new number, as expected.\n\nThanks for chiming in with all the detail on the use of 'perm'. This is \nthe most confusing part of this code path.\n\n> I think a comment explaining why the perm is needed would be helpful -\n> something explaining that the entries were generated using the old pack\n> numbers, so we need this mapping to be able to write them using the new\n> numbers.\n\nI can put this comment in the struct definition. Is that the right place \nfor it?\n\n\n> As part of this review, I did attempt to make everything use the sorted\n> pack_info order, but failed. I got as far as making get_sorted_entries()\n> always use start_pack=0 and skip the pack_infos that didn't have the p\n> pointer (to match the existing behavior of get_sorted_entries() that\n> only operates on new pack_infos being added by add_pack_to_midx), but it\n> still didn't work, and I didn't investigate further.\n\nIn the previous version, I tried to use 'perm' and the arrays as they \nexisted previously, but that led to the bug in that version. Hopefully \nthe new code is easier to read and understand. The delta may actually be \nhard to clearly see that it does the same work, but reading the code \nafter it is applied looks clearer (to me).\n\nThanks,\n-Stolee\n"},{"id":"367566","messageId":"26af4406-1ff8-bb4f-f6fb-5505e2d66df7@gmail.com","threadId":"49993","inReplyTo":"20190123221354.136961-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 6/9] multi-pack-index: implement 'expire' verb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-24T17:36:11Z","receivedAt":"2019-01-24T17:36:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/23/2019 5:13 PM, Jonathan Tan wrote:\n> Maybe add a fsck at the end for sanity's sake. Also, I think that\n> preservation of .keep packfiles is an important feature, and maybe worth\n> a test.\nGood points! I forgot to test the .keep stuff directly here because I \nhave an equivalent test in VFS for Git, so my wires got crossed.\n\nThanks,\n-Stolee\n"},{"id":"367581","messageId":"295eb5fb-5497-0517-a5ed-af96aafb3133@gmail.com","threadId":"49993","inReplyTo":"14fe05d0-75e5-0975-d490-550539dbb34a@gmail.com","subject":"Re: [PATCH v3 5/9] midx: refactor permutation logic and pack sorting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-24T19:17:38Z","receivedAt":"2019-01-24T19:17:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2019 12:34 PM, Derrick Stolee wrote:\n> On 1/23/2019 4:00 PM, Jonathan Tan wrote:\n>> Indeed, the sorting of pack_info is moved to after get_sorted_entries().\n>> Also, pack_perm[old number] = new number, as expected.\n>\n> Thanks for chiming in with all the detail on the use of 'perm'. This \n> is the most confusing part of this code path.\n>\n>> I think a comment explaining why the perm is needed would be helpful -\n>> something explaining that the entries were generated using the old pack\n>> numbers, so we need this mapping to be able to write them using the new\n>> numbers.\n>\n> I can put this comment in the struct definition. Is that the right \n> place for it?\n\nI mistakenly thought the pack_perm array was placed into the pack_list \nstruct. I'll put the comment right before we populate the contents of \nthe array.\n\nThanks,\n-Stolee\n"},{"id":"367587","messageId":"d3194048-3907-71d0-1e81-490707f39ff2@gmail.com","threadId":"49993","inReplyTo":"20190123223845.147378-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-24T19:36:57Z","receivedAt":"2019-01-24T19:55:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/23/2019 5:38 PM, Jonathan Tan wrote:\n>> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n>> index 6186c4c936..cc63531cc0 100644\n>> --- a/Documentation/git-multi-pack-index.txt\n>> +++ b/Documentation/git-multi-pack-index.txt\n>> @@ -36,6 +36,17 @@ expire::\n>> \thave no objects referenced by the MIDX. Rewrite the MIDX file\n>> \tafterward to remove all references to these pack-files.\n>> \n>> +repack::\n>> +\tCollect a batch of pack-files whose size are all at most the\n>> +\tsize given by --batch-size, but whose sizes sum to larger\n>> +\tthan --batch-size. The batch is selected by greedily adding\n>> +\tsmall pack-files starting with the oldest pack-files that fit\n>> +\tthe size. Create a new pack-file containing the objects the\n>> +\tmulti-pack-index indexes into those pack-files, and rewrite\n>> +\tthe multi-pack-index to contain that pack-file. A later run\n>> +\tof 'git multi-pack-index expire' will delete the pack-files\n>> +\tthat were part of this batch.\n> I see in the subsequent patch that you stop once the batch size is\n> matched or exceeded - I see that you mention \"whose sizes sum to larger\n> than --batch-size\", but this leads me to think that if the total so\n> happens to not exceed the batch size, don't do anything, but otherwise\n> repack *all* the small packs together.\n>\n> I would write this as:\n>\n> Create a new packfile containing the objects in the N least-sized\n> packfiles referenced by the multi-pack-index, where N is the smallest\n> number such that the total size of the packfiles equals or exceeds the\n> given batch size. Rewrite the multi-pack-index to reference the new\n> packfile instead of the N packfiles. A later run of ...\n\nThanks for the suggestion.\n\nIt is slightly wrong, in that we don't sort by size. Instead we sort by \nmodified time. That makes is a little complicated, but I'll give it \nanother shot using your framing:\n\n         Create a new pack-file containing objects in small pack-files\n         referenced by the multi-pack-index. Select the pack-files by\n         examining packs from oldest-to-newest, adding a pack if its\n         size is below the batch size. Stop adding packs when the sum\n         of sizes of the added packs is above the batch size. If the\n         total size does not reach the batch size, then do nothing.\n         Rewrite the multi-pack-index to reference the new pack-file.\n         A later run of 'git multi-pack-index expire' will delete the\n         pack-files that were part of this batch.\n\n-Stolee\n"},{"id":"367607","messageId":"20190124213806.179907-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"d3194048-3907-71d0-1e81-490707f39ff2@gmail.com","subject":"Re: [PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-24T21:38:06Z","receivedAt":"2019-01-24T21:38:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Thanks for the suggestion.\n> \n> It is slightly wrong, in that we don't sort by size. Instead we sort by \n> modified time. That makes is a little complicated, but I'll give it \n> another shot using your framing:\n> \n>         Create a new pack-file containing objects in small pack-files\n>         referenced by the multi-pack-index. Select the pack-files by\n>         examining packs from oldest-to-newest, adding a pack if its\n>         size is below the batch size. Stop adding packs when the sum\n>         of sizes of the added packs is above the batch size. If the\n>         total size does not reach the batch size, then do nothing.\n>         Rewrite the multi-pack-index to reference the new pack-file.\n>         A later run of 'git multi-pack-index expire' will delete the\n>         pack-files that were part of this batch.\n\nThanks, this looks good.\n"},{"id":"367610","messageId":"62b393b816d65a578c7a0ff9b40f01e0637e4d67.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 01/10] repack: refactor pack deletion for future use","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:54Z","receivedAt":"2019-01-24T21:51:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe repack builtin deletes redundant pack-files and their\nassociated .idx, .promisor, .bitmap, and .keep files. We will want\nto re-use this logic in the future for other types of repack, so\npull the logic into 'unlink_pack_path()' in packfile.c.\n\nThe 'ignore_keep' parameter is enabled for the use in repack, but\nwill be important for a future caller.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 14 ++------------\n packfile.c | 28 ++++++++++++++++++++++++++++\n packfile.h | 7 +++++++\n 3 files changed, 37 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 45583683ee..3d445b34b4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n-\tconst char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n-\tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t plen;\n-\n-\tstrbuf_addf(&buf, \"%s/%s\", dir_name, base_name);\n-\tplen = buf.len;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tstrbuf_setlen(&buf, plen);\n-\t\tstrbuf_addstr(&buf, exts[i]);\n-\t\tunlink(buf.buf);\n-\t}\n+\tstrbuf_addf(&buf, \"%s/%s.pack\", dir_name, base_name);\n+\tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex d1e6683ffe..bacecb4d0d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o)\n \t}\n }\n \n+void unlink_pack_path(const char *pack_name, int force_delete)\n+{\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tint i;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t plen;\n+\n+\tstrbuf_addstr(&buf, pack_name);\n+\tstrip_suffix_mem(buf.buf, &buf.len, \".pack\");\n+\tplen = buf.len;\n+\n+\tif (!force_delete) {\n+\t\tstrbuf_addstr(&buf, \".keep\");\n+\t\tif (!access(buf.buf, F_OK)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tstrbuf_setlen(&buf, plen);\n+\t\tstrbuf_addstr(&buf, exts[i]);\n+\t\tunlink(buf.buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\ndiff --git a/packfile.h b/packfile.h\nindex 6c4037605d..5b7bcdb1dd 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -86,6 +86,13 @@ extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Unlink the .pack and associated extension files.\n+ * Does not unlink if 'force_delete' is false and the pack-file is\n+ * marked as \".keep\".\n+ */\n+extern void unlink_pack_path(const char *pack_name, int force_delete);\n+\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n-- \ngitgitgadget\n\n"},{"id":"367612","messageId":"78867859042eea6318da65e236f903d6102ed93b.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 02/10] Docs: rearrange subcommands for multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:55Z","receivedAt":"2019-01-24T21:51:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe will add new subcommands to the multi-pack-index, and that will\nmake the documentation a bit messier. Clean up the 'verb'\ndescriptions by renaming the concept to 'subcommand' and removing\nthe reference to the object directory.\n\nHelped-by: Stefan Beller \nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex f7778a2c85..1af406aca2 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] \n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -23,13 +23,13 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+The following subcommands are available:\n+\n write::\n-\tWhen given as the verb, write a new MIDX file to\n-\t`/packs/multi-pack-index`.\n+\tWrite a new MIDX file.\n \n verify::\n-\tWhen given as the verb, verify the contents of the MIDX file\n-\tat `/packs/multi-pack-index`.\n+\tVerify the contents of the MIDX file.\n \n \n EXAMPLES\n-- \ngitgitgadget\n\n"},{"id":"367611","messageId":"pull.92.v4.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 00/10] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:54Z","receivedAt":"2019-01-24T21:52:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index provides a fast way to find an object among a large\nlist of pack-files. It stores a single pack-reference for each object id, so\nduplicate objects are ignored. Among a list of pack-files storing the same\nobject, the most-recently modified one is used.\n\nCreate new subcommands for the multi-pack-index builtin.\n\n * 'git multi-pack-index expire': If we have a pack-file indexed by the\n multi-pack-index, but all objects in that pack are duplicated in\n more-recently modified packs, then delete that pack (and any others like\n it). Delete the reference to that pack in the multi-pack-index.\n \n \n * 'git multi-pack-index repack --batch-size=': Starting from the oldest\n pack-files covered by the multi-pack-index, find those whose on-disk size\n is below the batch size until we have a collection of packs whose sizes\n add up to the batch size. Create a new pack containing all objects that\n the multi-pack-index references to those packs.\n \n \n\nThis allows us to create a new pattern for repacking objects: run 'repack'.\nAfter enough time has passed that all Git commands that started before the\nlast 'repack' are finished, run 'expire' again. This approach has some\nadvantages over the existing \"repack everything\" model:\n\n 1. Incremental. We can repack a small batch of objects at a time, instead\n of repacking all reachable objects. We can also limit ourselves to the\n objects that do not appear in newer pack-files.\n \n \n 2. Highly Available. By adding a new pack-file (and not deleting the old\n pack-files) we do not interrupt concurrent Git commands, and do not\n suffer performance degradation. By expiring only pack-files that have no\n referenced objects, we know that Git commands that are doing normal\n object lookups* will not be interrupted.\n \n \n 3. Note: if someone concurrently runs a Git command that uses\n get_all_packs(), then that command could try to read the pack-files and\n pack-indexes that we are deleting during an expire command. Such\n commands are usually related to object maintenance (i.e. fsck, gc,\n pack-objects) or are related to less-often-used features (i.e.\n fast-import, http-backend, server-info).\n \n \n\nWe plan to use this approach in VFS for Git to do background maintenance of\nthe \"shared object cache\" which is a Git alternate directory filled with\npackfiles containing commits and trees. We currently download pack-files on\nan hourly basis to keep up-to-date with the central server. The cache\nservers supply packs on an hourly and daily basis, so most of the hourly\npacks become useless after a new daily pack is downloaded. The 'expire'\ncommand would clear out most of those packs, but many will still remain with\nfewer than 100 objects remaining. The 'repack' command (with a batch size of\n1-3gb, probably) can condense the remaining packs in commands that run for\n1-3 min at a time. Since the daily packs range from 100-250mb, we will also\ncombine and condense those packs.\n\nUpdates in V2:\n\n * Added a method, unlink_pack_path() to remove packfiles, but with the\n additional check for a .keep file. This borrows logic from \n builtin/repack.c.\n \n \n * Modified documentation and commit messages to replace 'verb' with\n 'subcommand'. Simplified the documentation. (I left 'verbs' in the title\n of the cover letter for consistency.)\n \n \n\nUpdates in V3:\n\n * There was a bug in the expire logic when simultaneously removing packs\n and adding uncovered packs, specifically around the pack permutation.\n This was hard to see during review because I was using the 'pack_perm'\n array for multiple purposes. First, I was reducing its length, and then I\n was adding to it and resorting. In V3, I significantly overhauled the\n logic here, which required some extra commits before implementing\n 'expire'. The final commit includes a test that would cover this case.\n\nUpdates in V4:\n\n * More 'verb' and 'command' instances replaced with 'subcommand'. I grepped\n the patch to check these should be fixed everywhere.\n \n \n * Update the tests to check .keep files (in last patch).\n \n \n * Modify the tests to show the terminating condition of --batch-size when\n there are three packs that fit under the size, but the first two are\n large enough to stop adding packs. This required rearranging the packs\n slightly to get different sizes than we had before. Also, I added 'touch\n -t' to set the modified times so we can fix the order in which the packs\n are selected.\n \n \n * Added a comment about the purpose of pack_perm.\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (10):\n repack: refactor pack deletion for future use\n Docs: rearrange subcommands for multi-pack-index\n multi-pack-index: prepare for 'expire' subcommand\n midx: simplify computation of pack name lengths\n midx: refactor permutation logic and pack sorting\n multi-pack-index: implement 'expire' subcommand\n multi-pack-index: prepare 'repack' subcommand\n midx: implement midx_repack()\n multi-pack-index: test expire while adding packs\n midx: add test that 'expire' respects .keep files\n\n Documentation/git-multi-pack-index.txt | 26 +-\n builtin/multi-pack-index.c | 14 +-\n builtin/repack.c | 14 +-\n midx.c | 399 ++++++++++++++++++-------\n midx.h | 2 +\n packfile.c | 28 ++\n packfile.h | 7 +\n t/t5319-multi-pack-index.sh | 165 ++++++++++\n 8 files changed, 536 insertions(+), 119 deletions(-)\n\n\nbase-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-92%2Fderrickstolee%2Fmidx-expire%2Fupstream-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-92/derrickstolee/midx-expire/upstream-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/92\n\nRange-diff vs v3:\n\n 1: 62b393b816 = 1: 62b393b816 repack: refactor pack deletion for future use\n 2: 7886785904 = 2: 7886785904 Docs: rearrange subcommands for multi-pack-index\n 3: f06382b4ae ! 3: 628ca46036 multi-pack-index: prepare for 'expire' subcommand\n @@ -16,7 +16,9 @@\n Add a test that verifies the 'expire' subcommand is correctly wired,\n but will still be valid when the verb is implemented. Specifically,\n create a set of packs that should all have referenced objects and\n - should not be removed during an 'expire' operation.\n + should not be removed during an 'expire' operation. The packs are\n + created carefully to ensure they have a specific order when sorted\n + by size. This will be important in a later test.\n \n Signed-off-by: Derrick Stolee \n \n @@ -95,6 +97,8 @@\n +\t(\n +\t\tcd dup &&\n +\t\tgit init &&\n ++\t\ttest-tool genrandom \"data\" 4096 >large_file.txt &&\n ++\t\tgit update-index --add large_file.txt &&\n +\t\tfor i in $(test_seq 1 20)\n +\t\tdo\n +\t\t\ttest_commit $i\n @@ -104,24 +108,24 @@\n +\t\tgit branch C HEAD~13 &&\n +\t\tgit branch D HEAD~16 &&\n +\t\tgit branch E HEAD~18 &&\n -+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n -+\t\trefs/heads/E\n ++\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n ++\t\trefs/heads/A\n ++\t\t^refs/heads/B\n +\t\tEOF\n -+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n -+\t\trefs/heads/D\n -+\t\t^refs/heads/E\n ++\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n ++\t\trefs/heads/B\n ++\t\t^refs/heads/C\n +\t\tEOF\n +\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n +\t\trefs/heads/C\n +\t\t^refs/heads/D\n +\t\tEOF\n -+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n -+\t\trefs/heads/B\n -+\t\t^refs/heads/C\n ++\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n ++\t\trefs/heads/D\n ++\t\t^refs/heads/E\n +\t\tEOF\n -+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n -+\t\trefs/heads/A\n -+\t\t^refs/heads/B\n ++\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n ++\t\trefs/heads/E\n +\t\tEOF\n +\t\tgit multi-pack-index write\n +\t)\n 4: 2a763990ae ! 4: d55c1d7ee7 midx: simplify computation of pack name lengths\n @@ -12,7 +12,7 @@\n dir not already covered by the multi-pack-index.\n \n In anticipation of this becoming more complicated with the 'expire'\n - command, simplify the computation by centralizing it to a single\n + subcommand, simplify the computation by centralizing it to a single\n loop before writing the file.\n \n Signed-off-by: Derrick Stolee \n 5: a0d4cc6cb3 ! 5: 3950743b96 midx: refactor permutation logic and pack sorting\n @@ -282,6 +282,12 @@\n \n +\tQSORT(packs.info, packs.nr, pack_info_compare);\n +\n ++\t/*\n ++\t * pack_perm stores a permutation between pack-int-ids from the\n ++\t * previous multi-pack-index to the new one we are writing:\n ++\t *\n ++\t * pack_perm[old_id] = new_id\n ++\t */\n +\tALLOC_ARRAY(pack_perm, packs.nr);\n +\tfor (i = 0; i < packs.nr; i++) {\n +\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n 6: 4dbff40e7a ! 6: 6691d97902 multi-pack-index: implement 'expire' verb\n @@ -1,8 +1,8 @@\n Author: Derrick Stolee \n \n - multi-pack-index: implement 'expire' verb\n + multi-pack-index: implement 'expire' subcommand\n \n - The 'git multi-pack-index expire' command looks at the existing\n + The 'git multi-pack-index expire' subcommand looks at the existing\n mult-pack-index, counts the number of objects referenced in each\n pack-file, deletes the pack-fils with no referenced objects, and\n rewrites the multi-pack-index to no longer reference those packs.\n @@ -18,7 +18,7 @@\n \n Test that a new pack-file that covers the contents of two other\n pack-files leads to those pack-files being deleted during the\n - expire command. Be sure to read the multi-pack-index to ensure\n + expire subcommand. Be sure to read the multi-pack-index to ensure\n it no longer references those packs.\n \n Signed-off-by: Derrick Stolee \n @@ -161,6 +161,11 @@\n +\t\t}\n +\t}\n +\n + \t/*\n + \t * pack_perm stores a permutation between pack-int-ids from the\n + \t * previous multi-pack-index to the new one we are writing:\n +@@\n + \t */\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tfor (i = 0; i < packs.nr; i++) {\n -\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n @@ -273,7 +278,9 @@\n +\t\ttest_cmp expect actual &&\n +\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n +\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n -+\t\ttest_cmp expect-idx actual-midx\n ++\t\ttest_cmp expect-idx actual-midx &&\n ++\t\tgit multi-pack-index verify &&\n ++\t\tgit fsck\n +\t)\n +'\n +\n 7: b39f90ad09 ! 7: f5a8ff21dd multi-pack-index: prepare 'repack' subcommand\n @@ -11,7 +11,7 @@\n operation does not interrupt concurrent git commands.\n \n Introduce a 'repack' subcommand to 'git multi-pack-index' that\n - takes a '--batch-size' option. The verb will inspect the\n + takes a '--batch-size' option. The subcommand will inspect the\n multi-pack-index for referenced pack-files whose size is smaller\n than the batch size, until collecting a list of pack-files whose\n sizes sum to larger than the batch size. Then, a new pack-file\n @@ -26,6 +26,11 @@\n we specify a small batch size, we will guarantee that future\n implementations do not change the list of pack-files.\n \n + In addition, we hard-code the modified times of the packs in\n + the pack directory to ensure the list of packs sorted by modified\n + time matches the order if sorted by size (ascending). This will\n + be important in a future test.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n @@ -36,15 +41,15 @@\n \tafterward to remove all references to these pack-files.\n \n +repack::\n -+\tCollect a batch of pack-files whose size are all at most the\n -+\tsize given by --batch-size, but whose sizes sum to larger\n -+\tthan --batch-size. The batch is selected by greedily adding\n -+\tsmall pack-files starting with the oldest pack-files that fit\n -+\tthe size. Create a new pack-file containing the objects the\n -+\tmulti-pack-index indexes into those pack-files, and rewrite\n -+\tthe multi-pack-index to contain that pack-file. A later run\n -+\tof 'git multi-pack-index expire' will delete the pack-files\n -+\tthat were part of this batch.\n ++\tCreate a new pack-file containing objects in small pack-files\n ++\treferenced by the multi-pack-index. Select the pack-files by\n ++\texamining packs from oldest-to-newest, adding a pack if its\n ++\tsize is below the batch size. Stop adding packs when the sum\n ++\tof sizes of the added packs is above the batch size. If the\n ++\ttotal size does not reach the batch size, then do nothing.\n ++\tRewrite the multi-pack-index to reference the new pack-file.\n ++\tA later run of 'git multi-pack-index expire' will delete the\n ++\tpack-files that were part of this batch.\n +\n \n EXAMPLES\n @@ -84,11 +89,18 @@\n +\tif (!strcmp(argv[0], \"repack\"))\n +\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n +\tif (opts.batch_size)\n -+\t\tdie(_(\"--batch-size option is only for 'repack' verb\"));\n ++\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n +\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n +@@\n + \tif (!strcmp(argv[0], \"expire\"))\n + \t\treturn expire_midx_packs(opts.object_dir);\n + \n +-\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n ++\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n + }\n \n diff --git a/midx.c b/midx.c\n --- a/midx.c\n @@ -125,6 +137,12 @@\n +test_expect_success 'repack with minimum size does not alter existing packs' '\n +\t(\n +\t\tcd dup &&\n ++\t\trm -rf .git/objects/pack &&\n ++\t\tmv .git/objects/pack-backup .git/objects/pack &&\n ++\t\ttouch -m -t 201901010000 .git/objects/pack/pack-D* &&\n ++\t\ttouch -m -t 201901010001 .git/objects/pack/pack-C* &&\n ++\t\ttouch -m -t 201901010002 .git/objects/pack/pack-B* &&\n ++\t\ttouch -m -t 201901010003 .git/objects/pack/pack-A* &&\n +\t\tls .git/objects/pack >expect &&\n +\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n +\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n 8: a4c2d5a8e1 ! 8: ba1a1c7bbb midx: implement midx_repack()\n @@ -149,6 +149,16 @@\n diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n --- a/t/t5319-multi-pack-index.sh\n +++ b/t/t5319-multi-pack-index.sh\n +@@\n + \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n + \t\trefs/heads/E\n + \t\tEOF\n +-\t\tgit multi-pack-index write\n ++\t\tgit multi-pack-index write &&\n ++\t\tcp -r .git/objects/pack .git/objects/pack-backup\n + \t)\n + '\n + \n @@\n \t)\n '\n @@ -156,25 +166,28 @@\n +test_expect_success 'repack creates a new pack' '\n +\t(\n +\t\tcd dup &&\n -+\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n -+\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n -+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n +\t\tls .git/objects/pack/*idx >idx-list &&\n +\t\ttest_line_count = 5 idx-list &&\n ++\t\tTHIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 3 | tail -n 1) &&\n ++\t\tBATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) &&\n ++\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n ++\t\tls .git/objects/pack/*idx >idx-list &&\n ++\t\ttest_line_count = 6 idx-list &&\n +\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n -+\t\ttest_line_count = 5 midx-list\n ++\t\ttest_line_count = 6 midx-list\n +\t)\n +'\n +\n +test_expect_success 'expire removes repacked packs' '\n +\t(\n +\t\tcd dup &&\n -+\t\tls -S .git/objects/pack/*pack | head -n 3 >expect &&\n ++\t\tls -al .git/objects/pack/*pack &&\n ++\t\tls -S .git/objects/pack/*pack | head -n 4 >expect &&\n +\t\tgit multi-pack-index expire &&\n +\t\tls -S .git/objects/pack/*pack >actual &&\n +\t\ttest_cmp expect actual &&\n +\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n -+\t\ttest_line_count = 3 midx-list\n ++\t\ttest_line_count = 4 midx-list\n +\t)\n +'\n +\n 9: b97fb35ba9 = 9: b1c6892417 multi-pack-index: test expire while adding packs\n -: ---------- > 10: 481b08890f midx: add test that 'expire' respects .keep files\n\n-- \ngitgitgadget\n"},{"id":"367613","messageId":"d55c1d7ee7b3f3089540cb9554013b6909c70c0f.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 04/10] midx: simplify computation of pack name lengths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:57Z","receivedAt":"2019-01-24T21:52:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore writing the multi-pack-index, we compute the length of the\npack-index names concatenated together. This forms the data in the\npack name chunk, and we precompute it to compute chunk offsets.\nThe value is also modified to fit alignment needs.\n\nPreviously, this computation was coupled with adding packs from\nthe existing multi-pack-index and the remaining packs in the object\ndir not already covered by the multi-pack-index.\n\nIn anticipation of this becoming more complicated with the 'expire'\nsubcommand, simplify the computation by centralizing it to a single\nloop before writing the file.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bb825ef816..f087bbbe82 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -383,7 +383,6 @@ struct pack_list {\n \tuint32_t nr;\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n-\tsize_t pack_name_concat_len;\n \tstruct multi_pack_index *m;\n };\n \n@@ -418,7 +417,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t}\n \n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n-\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n \t}\n }\n@@ -762,6 +760,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n+\tint pack_name_concat_len = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -777,7 +776,6 @@ int write_midx_file(const char *object_dir)\n \tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.names = NULL;\n-\tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n@@ -788,7 +786,6 @@ int write_midx_file(const char *object_dir)\n \n \t\t\tpacks.list[packs.nr] = NULL;\n \t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -798,10 +795,6 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n-\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n-\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n@@ -814,6 +807,13 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tfor (i = 0; i < packs.nr; i++)\n+\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -831,7 +831,7 @@ int write_midx_file(const char *object_dir)\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-- \ngitgitgadget\n\n"},{"id":"367614","messageId":"3950743b963049f502f1e5caefcfe477ebbcdd8b.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 05/10] midx: refactor permutation logic and pack sorting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:57Z","receivedAt":"2019-01-24T21:52:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of the expire subcommand, refactor the way we sort\nthe packfiles by name. This will greatly simplify our approach to\ndropping expired packs from the list.\n\nFirst, create 'struct pack_info' to replace 'struct pack_pair'.\nThis struct contains the necessary information about a pack,\nincluding its name, a pointer to its packfile struct (if not\nalready in the multi-pack-index), and the original pack-int-id.\n\nSecond, track the pack information using an array of pack_info\nstructs in the pack_list struct. This simplifies the logic around\nthe multiple arrays we were tracking in that struct.\n\nFinally, update get_sorted_entries() to not permute the pack-int-id\nand instead supply the permutation to write_midx_object_offsets().\nThis requires sorting the packs after get_sorted_entries().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 156 +++++++++++++++++++++++++--------------------------------\n 1 file changed, 69 insertions(+), 87 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f087bbbe82..95c39106b2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -377,12 +377,23 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_info {\n+\tuint32_t orig_pack_int_id;\n+\tchar *pack_name;\n+\tstruct packed_git *p;\n+};\n+\n+static int pack_info_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_info *a = (struct pack_info *)_a;\n+\tstruct pack_info *b = (struct pack_info *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n struct pack_list {\n-\tstruct packed_git **list;\n-\tchar **names;\n+\tstruct pack_info *info;\n \tuint32_t nr;\n-\tuint32_t alloc_list;\n-\tuint32_t alloc_names;\n+\tuint32_t alloc;\n \tstruct multi_pack_index *m;\n };\n \n@@ -395,66 +406,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n-\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n+\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n \n-\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n-\t\t\t\t\t\t\tfull_path_len,\n-\t\t\t\t\t\t\t0);\n+\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->list[packs->nr]) {\n+\t\tif (!packs->info[packs->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->list[packs->nr]);\n-\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t\tclose_pack(packs->info[packs->nr].p);\n+\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n \t\tpacks->nr++;\n \t}\n }\n \n-struct pack_pair {\n-\tuint32_t pack_int_id;\n-\tchar *pack_name;\n-};\n-\n-static int pack_pair_compare(const void *_a, const void *_b)\n-{\n-\tstruct pack_pair *a = (struct pack_pair *)_a;\n-\tstruct pack_pair *b = (struct pack_pair *)_b;\n-\treturn strcmp(a->pack_name, b->pack_name);\n-}\n-\n-static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n-{\n-\tuint32_t i;\n-\tstruct pack_pair *pairs;\n-\n-\tALLOC_ARRAY(pairs, nr_packs);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpairs[i].pack_int_id = i;\n-\t\tpairs[i].pack_name = pack_names[i];\n-\t}\n-\n-\tQSORT(pairs, nr_packs, pack_pair_compare);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpack_names[i] = pairs[i].pack_name;\n-\t\tperm[pairs[i].pack_int_id] = i;\n-\t}\n-\n-\tfree(pairs);\n-}\n-\n struct pack_midx_entry {\n \tstruct object_id oid;\n \tuint32_t pack_int_id;\n@@ -480,7 +457,6 @@ static int midx_oid_compare(const void *_a, const void *_b)\n }\n \n static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n-\t\t\t\t uint32_t *pack_perm,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n@@ -488,7 +464,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n-\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n \te->offset = nth_midxed_offset(m, pos);\n \n \t/* consider objects in midx to be from \"old\" packs */\n@@ -522,8 +498,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * of a packfile containing the object).\n */\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n-\t\t\t\t\t\t struct packed_git **p,\n-\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n {\n@@ -534,7 +509,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \tuint32_t start_pack = m ? m->num_packs : 0;\n \n \tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n-\t\ttotal_objects += p[cur_pack]->num_objects;\n+\t\ttotal_objects += info[cur_pack].p->num_objects;\n \n \t/*\n \t * As we de-duplicate by fanout value, we expect the fanout\n@@ -559,7 +534,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n \t\t\t\tnr_fanout++;\n@@ -570,12 +545,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n-\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n-\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(info[cur_pack].p, cur_fanout);\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -604,7 +579,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n }\n \n static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t char **pack_names,\n+\t\t\t\t struct pack_info *info,\n \t\t\t\t uint32_t num_packs)\n {\n \tuint32_t i;\n@@ -612,14 +587,14 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n \n-\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t pack_names[i - 1],\n-\t\t\t pack_names[i]);\n+\t\t\t info[i - 1].pack_name,\n+\t\t\t info[i].pack_name);\n \n-\t\thashwrite(f, pack_names[i], writelen);\n+\t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -690,6 +665,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n }\n \n static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tuint32_t *perm,\n \t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n {\n \tstruct pack_midx_entry *list = objects;\n@@ -699,7 +675,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\thashwrite_be32(f, obj->pack_int_id);\n+\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -772,20 +748,17 @@ int write_midx_file(const char *object_dir)\n \tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n-\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n-\tpacks.alloc_names = packs.alloc_list;\n-\tpacks.list = NULL;\n-\tpacks.names = NULL;\n-\tALLOC_ARRAY(packs.list, packs.alloc_list);\n-\tALLOC_ARRAY(packs.names, packs.alloc_names);\n+\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n+\tpacks.info = NULL;\n+\tALLOC_ARRAY(packs.info, packs.alloc);\n \n \tif (packs.m) {\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n-\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n \n-\t\t\tpacks.list[packs.nr] = NULL;\n-\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n+\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].p = NULL;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -795,10 +768,7 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n-\n-\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -807,8 +777,21 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\n+\t/*\n+\t * pack_perm stores a permutation between pack-int-ids from the\n+\t * previous multi-pack-index to the new one we are writing:\n+\t *\n+\t * pack_perm[old_id] = new_id\n+\t */\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t}\n+\n \tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -879,7 +862,7 @@ int write_midx_file(const char *object_dir)\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -891,7 +874,7 @@ int write_midx_file(const char *object_dir)\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -914,15 +897,14 @@ int write_midx_file(const char *object_dir)\n \n cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.list[i]) {\n-\t\t\tclose_pack(packs.list[i]);\n-\t\t\tfree(packs.list[i]);\n+\t\tif (packs.info[i].p) {\n+\t\t\tclose_pack(packs.info[i].p);\n+\t\t\tfree(packs.info[i].p);\n \t\t}\n-\t\tfree(packs.names[i]);\n+\t\tfree(packs.info[i].pack_name);\n \t}\n \n-\tfree(packs.list);\n-\tfree(packs.names);\n+\tfree(packs.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \ngitgitgadget\n\n"},{"id":"367615","messageId":"6691d9790229d308bc4071bfe1d116da13e3a868.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 06/10] multi-pack-index: implement 'expire' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:59Z","receivedAt":"2019-01-24T21:52:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git multi-pack-index expire' subcommand looks at the existing\nmult-pack-index, counts the number of objects referenced in each\npack-file, deletes the pack-fils with no referenced objects, and\nrewrites the multi-pack-index to no longer reference those packs.\n\nRefactor the write_midx_file() method to call write_midx_internal()\nwhich now takes an existing 'struct multi_pack_index' and a list\nof pack-files to drop (as specified by the names of their pack-\nindexes). As we write the new multi-pack-index, we drop those\nfile names from the list of known pack-files.\n\nThe expire_midx_packs() method removes the unreferenced pack-files\nafter carefully closing the packs to avoid open handles.\n\nTest that a new pack-file that covers the contents of two other\npack-files leads to those pack-files being deleted during the\nexpire subcommand. Be sure to read the multi-pack-index to ensure\nit no longer references those packs.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 120 +++++++++++++++++++++++++++++++++---\n t/t5319-multi-pack-index.sh | 20 ++++++\n 2 files changed, 130 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 95c39106b2..299e9b2e8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,8 @@\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n+#define PACK_EXPIRED UINT_MAX\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -381,6 +383,7 @@ struct pack_info {\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n+\tunsigned expired : 1;\n };\n \n static int pack_info_compare(const void *_a, const void *_b)\n@@ -428,6 +431,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n \t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n+\t\tpacks->info[packs->nr].expired = 0;\n \t\tpacks->nr++;\n \t}\n }\n@@ -587,13 +591,17 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n+\t\tsize_t writelen;\n+\n+\t\tif (info[i].expired)\n+\t\t\tcontinue;\n \n \t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n \t\t\t info[i - 1].pack_name,\n \t\t\t info[i].pack_name);\n \n+\t\twritelen = strlen(info[i].pack_name) + 1;\n \t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n@@ -675,6 +683,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n+\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->pack_int_id);\n+\n \t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n@@ -721,7 +734,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n-int write_midx_file(const char *object_dir)\n+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+\t\t\t struct string_list *packs_to_drop)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -737,6 +751,8 @@ int write_midx_file(const char *object_dir)\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n+\tint dropped_packs = 0;\n+\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -745,7 +761,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n-\tpacks.m = load_multi_pack_index(object_dir, 1);\n+\tif (m)\n+\t\tpacks.m = m;\n+\telse\n+\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n \tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n@@ -759,13 +778,14 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n \t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n \t\t\tpacks.info[packs.nr].p = NULL;\n+\t\t\tpacks.info[packs.nr].expired = 0;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs)\n+\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n \tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n@@ -779,6 +799,34 @@ int write_midx_file(const char *object_dir)\n \n \tQSORT(packs.info, packs.nr, pack_info_compare);\n \n+\tif (packs_to_drop && packs_to_drop->nr) {\n+\t\tint drop_index = 0;\n+\t\tint missing_drops = 0;\n+\n+\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\tif (!cmp) {\n+\t\t\t\tdrop_index++;\n+\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t} else if (cmp > 0) {\n+\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\tdrop_index++;\n+\t\t\t\tmissing_drops++;\n+\t\t\t\ti--;\n+\t\t\t} else {\n+\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (missing_drops) {\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \t/*\n \t * pack_perm stores a permutation between pack-int-ids from the\n \t * previous multi-pack-index to the new one we are writing:\n@@ -787,11 +835,18 @@ int write_midx_file(const char *object_dir)\n \t */\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t\tif (packs.info[i].expired) {\n+\t\t\tdropped_packs++;\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t} else {\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tif (!packs.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -807,7 +862,7 @@ int write_midx_file(const char *object_dir)\n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -908,7 +963,12 @@ int write_midx_file(const char *object_dir)\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-\treturn 0;\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1010,5 +1070,45 @@ int verify_midx_file(const char *object_dir)\n \n int expire_midx_packs(const char *object_dir)\n {\n-\treturn 0;\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t}\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\t\tFREE_AND_NULL(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex a8528f7da0..65e85debec 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -397,4 +397,24 @@ test_expect_success 'expire does not remove any packs' '\n \t)\n '\n \n+test_expect_success 'expire removes unreferenced packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n+\t\ttest_cmp expect-idx actual-midx &&\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"367616","messageId":"628ca4603690b1239d722df8560dcd0b3790738d.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 03/10] multi-pack-index: prepare for 'expire' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:56Z","receivedAt":"2019-01-24T21:52:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe multi-pack-index tracks objects in a collection of pack-files.\nOnly one copy of each object is indexed, using the modified time\nof the pack-files to determine tie-breakers. It is possible to\nhave a pack-file with no referenced objects because all objects\nhave a duplicate in a newer pack-file.\n\nIntroduce a new 'expire' subcommand to the multi-pack-index builtin.\nThis subcommand will delete these unused pack-files and rewrite the\nmulti-pack-index to no longer refer to those files. More details\nabout the specifics will follow as the method is implemented.\n\nAdd a test that verifies the 'expire' subcommand is correctly wired,\nbut will still be valid when the verb is implemented. Specifically,\ncreate a set of packs that should all have referenced objects and\nshould not be removed during an 'expire' operation. The packs are\ncreated carefully to ensure they have a specific order when sorted\nby size. This will be important in a later test.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 5 +++\n builtin/multi-pack-index.c | 4 ++-\n midx.c | 5 +++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 49 ++++++++++++++++++++++++++\n 5 files changed, 63 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 1af406aca2..6186c4c936 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -31,6 +31,11 @@ write::\n verify::\n \tVerify the contents of the MIDX file.\n \n+expire::\n+\tDelete the pack-files that are tracked \tby the MIDX file, but\n+\thave no objects referenced by the MIDX. Rewrite the MIDX file\n+\tafterward to remove all references to these pack-files.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex fca70f8e4f..145de3a46c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,7 +5,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n \tNULL\n };\n \n@@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(opts.object_dir);\n+\tif (!strcmp(argv[0], \"expire\"))\n+\t\treturn expire_midx_packs(opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 730ff84dff..bb825ef816 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir)\n \n \treturn verify_midx_error;\n }\n+\n+int expire_midx_packs(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 774f652530..e3a2b740b5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n+int expire_midx_packs(const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 70926b5bc0..a8528f7da0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -348,4 +348,53 @@ test_expect_success 'verify incorrect 64-bit offset' '\n \t\t\"incorrect object offset\"\n '\n \n+test_expect_success 'setup expire tests' '\n+\tmkdir dup &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit init &&\n+\t\ttest-tool genrandom \"data\" 4096 >large_file.txt &&\n+\t\tgit update-index --add large_file.txt &&\n+\t\tfor i in $(test_seq 1 20)\n+\t\tdo\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit branch A HEAD &&\n+\t\tgit branch B HEAD~8 &&\n+\t\tgit branch C HEAD~13 &&\n+\t\tgit branch D HEAD~16 &&\n+\t\tgit branch E HEAD~18 &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n+test_expect_success 'expire does not remove any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"367617","messageId":"ba1a1c7bbbdbd8f3b82187763df4ec21b7032fb5.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 08/10] midx: implement midx_repack()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:52:00Z","receivedAt":"2019-01-24T21:52:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo repack using a multi-pack-index, first sort all pack-files by\ntheir modified time. Second, walk those pack-files from oldest\nto newest, adding the packs to a list if they are smaller than the\ngiven pack-size. Finally, collect the objects from the multi-pack-\nindex that are in those packs and send them to 'git pack-objects'.\n\nWhile first designing a 'git multi-pack-index repack' operation, I\nstarted by collecting the batches based on the size of the objects\ninstead of the size of the pack-files. This allows repacking a\nlarge pack-file that has very few referencd objects. However, this\ncame at a significant cost of parsing pack-files instead of simply\nreading the multi-pack-index and getting the file information for\nthe pack-files. This object-size idea could be a direction for\nfuture expansion in this area.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 109 +++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 31 +++++++++-\n 2 files changed, 138 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 768a7dff73..7d81bf943e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -8,6 +8,7 @@\n #include \"sha1-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n+#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -1113,7 +1114,113 @@ int expire_midx_packs(const char *object_dir)\n \treturn result;\n }\n \n-int midx_repack(const char *object_dir, size_t batch_size)\n+struct time_and_id {\n+\ttimestamp_t mtime;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n {\n+\tconst struct time_and_id *a, *b;\n+\n+\ta = (const struct time_and_id *)a_;\n+\tb = (const struct time_and_id *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n \treturn 0;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\tint result = 0;\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct time_and_id *pack_ti;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n+\tpack_ti = xcalloc(m->num_packs, sizeof(struct time_and_id));\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_ti[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_ti[i].mtime = m->packs[i]->mtime;\n+\t}\n+\tQSORT(pack_ti, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_ti[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (p->pack_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += p->pack_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tif (total_size < batch_size || packs_to_repack < 2)\n+\t\tgoto cleanup;\n+\n+\targv_array_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\targv_array_push(&cmd.args, base_name.buf);\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\txwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);\n+\t\txwrite(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, m, NULL);\n+\tm = NULL;\n+\n+cleanup:\n+\tif (m)\n+\t\tclose_midx(m);\n+\tfree(include_pack);\n+\tfree(pack_ti);\n+\treturn result;\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex acc5e65ecc..d6c1353514 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -383,7 +383,8 @@ test_expect_success 'setup expire tests' '\n \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n \t\trefs/heads/E\n \t\tEOF\n-\t\tgit multi-pack-index write\n+\t\tgit multi-pack-index write &&\n+\t\tcp -r .git/objects/pack .git/objects/pack-backup\n \t)\n '\n \n@@ -434,4 +435,32 @@ test_expect_success 'repack with minimum size does not alter existing packs' '\n \t)\n '\n \n+test_expect_success 'repack creates a new pack' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 5 idx-list &&\n+\t\tTHIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 3 | tail -n 1) &&\n+\t\tBATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) &&\n+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 6 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 6 midx-list\n+\t)\n+'\n+\n+test_expect_success 'expire removes repacked packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls -al .git/objects/pack/*pack &&\n+\t\tls -S .git/objects/pack/*pack | head -n 4 >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/*pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 4 midx-list\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"367618","messageId":"b1c6892417d60b0052535f9332a001868447fa9b.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 09/10] multi-pack-index: test expire while adding packs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:52:01Z","receivedAt":"2019-01-24T21:52:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring development of the multi-pack-index expire subcommand, a\nversion went out that improperly computed the pack order if a new\npack was introduced while other packs were being removed. Part of\nthe subtlety of the bug involved the new pack being placed before\nother packs that already existed in the multi-pack-index.\n\nAdd a test to t5319-multi-pack-index.sh that catches this issue.\nThe test adds new packs that cause another pack to be expired, and\ncreates new packs that are lexicographically sorted before and\nafter the existing packs.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex d6c1353514..19b769eea0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -463,4 +463,36 @@ test_expect_success 'expire removes repacked packs' '\n \t)\n '\n \n+test_expect_success 'expire works when adding new packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/a-pack <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/z-pack <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack/ | grep idx >expect &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"367619","messageId":"481b08890fa633a61995300b7dd47979a0db53dd.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 10/10] midx: add test that 'expire' respects .keep files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:52:02Z","receivedAt":"2019-01-24T21:52:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git multi-pack-index expire' subcommand may delete packs that\nare not needed from the perspective of the multi-pack-index. If\na pack has a .keep file, then we should not delete that pack. Add\na test that ensures we preserve a pack that would otherwise be\nexpired. First, create a new pack that contains every object in\nthe repo, then add it to the multi-pack-index. Then create a .keep\nfile for a pack starting with \"a-pack\" that was added in the\nprevious test. Finally, expire and verify that the pack remains\nand the other packs were expired.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 18 ++++++++++++++++++\n 1 file changed, 18 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 19b769eea0..bcfa520401 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -495,4 +495,22 @@ test_expect_success 'expire works when adding new packs' '\n \t)\n '\n \n+test_expect_success 'expire respects .keep files' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-all <<-EOF &&\n+\t\trefs/heads/A\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n+\t\ttouch $PACKA.keep &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n+\t\ttest_line_count = 3 a-pack-files &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 2 midx-list\n+\t)\n+'\n+\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"367620","messageId":"f5a8ff21dda5b29d8fe75b3b52be873bdb4b3031.1548366713.git.gitgitgadget@gmail.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 07/10] multi-pack-index: prepare 'repack' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-24T21:51:59Z","receivedAt":"2019-01-24T21:52:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment where the multi-pack-index is useful, it is due\nto many pack-files and an inability to repack the object store\ninto a single pack-file. However, it is likely that many of these\npack-files are rather small, and could be repacked into a slightly\nlarger pack-file without too much effort. It may also be important\nto ensure the object store is highly available and the repack\noperation does not interrupt concurrent git commands.\n\nIntroduce a 'repack' subcommand to 'git multi-pack-index' that\ntakes a '--batch-size' option. The subcommand will inspect the\nmulti-pack-index for referenced pack-files whose size is smaller\nthan the batch size, until collecting a list of pack-files whose\nsizes sum to larger than the batch size. Then, a new pack-file\nwill be created containing the objects from those pack-files that\nare referenced by the multi-pack-index. The resulting pack is\nlikely to actually be smaller than the batch size due to\ncompression and the fact that there may be objects in the pack-\nfiles that have duplicate copies in other pack-files.\n\nThe current change introduces the command-line arguments, and we\nadd a test that ensures we parse these options properly. Since\nwe specify a small batch size, we will guarantee that future\nimplementations do not change the list of pack-files.\n\nIn addition, we hard-code the modified times of the packs in\nthe pack directory to ensure the list of packs sorted by modified\ntime matches the order if sorted by size (ascending). This will\nbe important in a future test.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 11 +++++++++++\n builtin/multi-pack-index.c | 12 ++++++++++--\n midx.c | 5 +++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 17 +++++++++++++++++\n 5 files changed, 44 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 6186c4c936..de345c2400 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -36,6 +36,17 @@ expire::\n \thave no objects referenced by the MIDX. Rewrite the MIDX file\n \tafterward to remove all references to these pack-files.\n \n+repack::\n+\tCreate a new pack-file containing objects in small pack-files\n+\treferenced by the multi-pack-index. Select the pack-files by\n+\texamining packs from oldest-to-newest, adding a pack if its\n+\tsize is below the batch size. Stop adding packs when the sum\n+\tof sizes of the added packs is above the batch size. If the\n+\ttotal size does not reach the batch size, then do nothing.\n+\tRewrite the multi-pack-index to reference the new pack-file.\n+\tA later run of 'git multi-pack-index expire' will delete the\n+\tpack-files that were part of this batch.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 145de3a46c..c66239de33 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,12 +5,13 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n \tNULL\n };\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tunsigned long batch_size;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n@@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n+\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n@@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn 1;\n \t}\n \n+\tif (!strcmp(argv[0], \"repack\"))\n+\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n+\tif (opts.batch_size)\n+\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n+\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n@@ -47,5 +55,5 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!strcmp(argv[0], \"expire\"))\n \t\treturn expire_midx_packs(opts.object_dir);\n \n-\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n+\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 299e9b2e8f..768a7dff73 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1112,3 +1112,8 @@ int expire_midx_packs(const char *object_dir)\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex e3a2b740b5..394a21ee96 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n int expire_midx_packs(const char *object_dir);\n+int midx_repack(const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 65e85debec..acc5e65ecc 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -417,4 +417,21 @@ test_expect_success 'expire removes unreferenced packs' '\n \t)\n '\n \n+test_expect_success 'repack with minimum size does not alter existing packs' '\n+\t(\n+\t\tcd dup &&\n+\t\trm -rf .git/objects/pack &&\n+\t\tmv .git/objects/pack-backup .git/objects/pack &&\n+\t\ttouch -m -t 201901010000 .git/objects/pack/pack-D* &&\n+\t\ttouch -m -t 201901010001 .git/objects/pack/pack-C* &&\n+\t\ttouch -m -t 201901010002 .git/objects/pack/pack-B* &&\n+\t\ttouch -m -t 201901010003 .git/objects/pack/pack-A* &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n+\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"367624","messageId":"20190124221457.11062-1-jonathantanmy@google.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/10] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-24T22:14:57Z","receivedAt":"2019-01-24T22:15:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Updates in V4:\n> \n> * More 'verb' and 'command' instances replaced with 'subcommand'. I grepped\n> the patch to check these should be fixed everywhere.\n> \n> \n> * Update the tests to check .keep files (in last patch).\n> \n> \n> * Modify the tests to show the terminating condition of --batch-size when\n> there are three packs that fit under the size, but the first two are\n> large enough to stop adding packs. This required rearranging the packs\n> slightly to get different sizes than we had before. Also, I added 'touch\n> -t' to set the modified times so we can fix the order in which the packs\n> are selected.\n> \n> \n> * Added a comment about the purpose of pack_perm.\n\nThanks, the interdiff and patch 10 look good to me (I already reviewed\nV3). I also verified that in the last test, if there is no .keep file,\nthe test fails as expected.\n"},{"id":"367695","messageId":"20190125232422.GC72177@google.com","threadId":"49993","inReplyTo":"f5a8ff21dda5b29d8fe75b3b52be873bdb4b3031.1548366713.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 07/10] multi-pack-index: prepare 'repack' subcommand","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2019-01-25T23:24:22Z","receivedAt":"2019-01-25T23:24:32Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2019.01.24 13:51, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> In an environment where the multi-pack-index is useful, it is due\n> to many pack-files and an inability to repack the object store\n> into a single pack-file. However, it is likely that many of these\n> pack-files are rather small, and could be repacked into a slightly\n> larger pack-file without too much effort. It may also be important\n> to ensure the object store is highly available and the repack\n> operation does not interrupt concurrent git commands.\n> \n> Introduce a 'repack' subcommand to 'git multi-pack-index' that\n> takes a '--batch-size' option. The subcommand will inspect the\n> multi-pack-index for referenced pack-files whose size is smaller\n> than the batch size, until collecting a list of pack-files whose\n> sizes sum to larger than the batch size. Then, a new pack-file\n> will be created containing the objects from those pack-files that\n> are referenced by the multi-pack-index. The resulting pack is\n> likely to actually be smaller than the batch size due to\n> compression and the fact that there may be objects in the pack-\n> files that have duplicate copies in other pack-files.\n> \n> The current change introduces the command-line arguments, and we\n> add a test that ensures we parse these options properly. Since\n> we specify a small batch size, we will guarantee that future\n> implementations do not change the list of pack-files.\n> \n> In addition, we hard-code the modified times of the packs in\n> the pack directory to ensure the list of packs sorted by modified\n> time matches the order if sorted by size (ascending). This will\n> be important in a future test.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-multi-pack-index.txt | 11 +++++++++++\n> builtin/multi-pack-index.c | 12 ++++++++++--\n> midx.c | 5 +++++\n> midx.h | 1 +\n> t/t5319-multi-pack-index.sh | 17 +++++++++++++++++\n> 5 files changed, 44 insertions(+), 2 deletions(-)\n> \n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index 6186c4c936..de345c2400 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -36,6 +36,17 @@ expire::\n> \thave no objects referenced by the MIDX. Rewrite the MIDX file\n> \tafterward to remove all references to these pack-files.\n> \n> +repack::\n> +\tCreate a new pack-file containing objects in small pack-files\n> +\treferenced by the multi-pack-index. Select the pack-files by\n> +\texamining packs from oldest-to-newest, adding a pack if its\n> +\tsize is below the batch size. Stop adding packs when the sum\n> +\tof sizes of the added packs is above the batch size. If the\n> +\ttotal size does not reach the batch size, then do nothing.\n> +\tRewrite the multi-pack-index to reference the new pack-file.\n> +\tA later run of 'git multi-pack-index expire' will delete the\n> +\tpack-files that were part of this batch.\n> +\n> \n> EXAMPLES\n> --------\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 145de3a46c..c66239de33 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -5,12 +5,13 @@\n> #include \"midx.h\"\n> \n> static char const * const builtin_multi_pack_index_usage[] = {\n> -\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n> +\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n> \tNULL\n> };\n> \n> static struct opts_multi_pack_index {\n> \tconst char *object_dir;\n> +\tunsigned long batch_size;\n> } opts;\n> \n> int cmd_multi_pack_index(int argc, const char **argv,\n> @@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> \tstatic struct option builtin_multi_pack_index_options[] = {\n> \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n> \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n> +\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n> +\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> \t\treturn 1;\n> \t}\n> \n> +\tif (!strcmp(argv[0], \"repack\"))\n> +\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n> +\tif (opts.batch_size)\n> +\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n> +\n> \tif (!strcmp(argv[0], \"write\"))\n> \t\treturn write_midx_file(opts.object_dir);\n> \tif (!strcmp(argv[0], \"verify\"))\n> @@ -47,5 +55,5 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> \tif (!strcmp(argv[0], \"expire\"))\n> \t\treturn expire_midx_packs(opts.object_dir);\n> \n> -\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n> +\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n> }\n> diff --git a/midx.c b/midx.c\n> index 299e9b2e8f..768a7dff73 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1112,3 +1112,8 @@ int expire_midx_packs(const char *object_dir)\n> \tstring_list_clear(&packs_to_drop, 0);\n> \treturn result;\n> }\n> +\n> +int midx_repack(const char *object_dir, size_t batch_size)\n> +{\n> +\treturn 0;\n> +}\n> diff --git a/midx.h b/midx.h\n> index e3a2b740b5..394a21ee96 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);\n> void clear_midx_file(struct repository *r);\n> int verify_midx_file(const char *object_dir);\n> int expire_midx_packs(const char *object_dir);\n> +int midx_repack(const char *object_dir, size_t batch_size);\n> \n> void close_midx(struct multi_pack_index *m);\n> \n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index 65e85debec..acc5e65ecc 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -417,4 +417,21 @@ test_expect_success 'expire removes unreferenced packs' '\n> \t)\n> '\n> \n> +test_expect_success 'repack with minimum size does not alter existing packs' '\n> +\t(\n> +\t\tcd dup &&\n> +\t\trm -rf .git/objects/pack &&\n> +\t\tmv .git/objects/pack-backup .git/objects/pack &&\n> +\t\ttouch -m -t 201901010000 .git/objects/pack/pack-D* &&\n> +\t\ttouch -m -t 201901010001 .git/objects/pack/pack-C* &&\n> +\t\ttouch -m -t 201901010002 .git/objects/pack/pack-B* &&\n> +\t\ttouch -m -t 201901010003 .git/objects/pack/pack-A* &&\n> +\t\tls .git/objects/pack >expect &&\n> +\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n> +\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n> +\t\tls .git/objects/pack >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> test_done\n\nThis test failes for me, with the following error:\nmv: cannot stat '.git/objects/pack-backup': No such file or directory\n\n\n> -- \n> gitgitgadget\n> \n"},{"id":"367696","messageId":"20190125234926.GD72177@google.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/10] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2019-01-25T23:49:26Z","receivedAt":"2019-01-25T23:50:09Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2019.01.24 13:51, Derrick Stolee via GitGitGadget wrote:\n> The multi-pack-index provides a fast way to find an object among a large\n> list of pack-files. It stores a single pack-reference for each object id, so\n> duplicate objects are ignored. Among a list of pack-files storing the same\n> object, the most-recently modified one is used.\n> \n> Create new subcommands for the multi-pack-index builtin.\n> \n> * 'git multi-pack-index expire': If we have a pack-file indexed by the\n> multi-pack-index, but all objects in that pack are duplicated in\n> more-recently modified packs, then delete that pack (and any others like\n> it). Delete the reference to that pack in the multi-pack-index.\n> \n> \n> * 'git multi-pack-index repack --batch-size=': Starting from the oldest\n> pack-files covered by the multi-pack-index, find those whose on-disk size\n> is below the batch size until we have a collection of packs whose sizes\n> add up to the batch size. Create a new pack containing all objects that\n> the multi-pack-index references to those packs.\n> \n> \n> \n> This allows us to create a new pattern for repacking objects: run 'repack'.\n> After enough time has passed that all Git commands that started before the\n> last 'repack' are finished, run 'expire' again. This approach has some\n> advantages over the existing \"repack everything\" model:\n> \n> 1. Incremental. We can repack a small batch of objects at a time, instead\n> of repacking all reachable objects. We can also limit ourselves to the\n> objects that do not appear in newer pack-files.\n> \n> \n> 2. Highly Available. By adding a new pack-file (and not deleting the old\n> pack-files) we do not interrupt concurrent Git commands, and do not\n> suffer performance degradation. By expiring only pack-files that have no\n> referenced objects, we know that Git commands that are doing normal\n> object lookups* will not be interrupted.\n> \n> \n> 3. Note: if someone concurrently runs a Git command that uses\n> get_all_packs(), then that command could try to read the pack-files and\n> pack-indexes that we are deleting during an expire command. Such\n> commands are usually related to object maintenance (i.e. fsck, gc,\n> pack-objects) or are related to less-often-used features (i.e.\n> fast-import, http-backend, server-info).\n> \n> \n> \n> We plan to use this approach in VFS for Git to do background maintenance of\n> the \"shared object cache\" which is a Git alternate directory filled with\n> packfiles containing commits and trees. We currently download pack-files on\n> an hourly basis to keep up-to-date with the central server. The cache\n> servers supply packs on an hourly and daily basis, so most of the hourly\n> packs become useless after a new daily pack is downloaded. The 'expire'\n> command would clear out most of those packs, but many will still remain with\n> fewer than 100 objects remaining. The 'repack' command (with a batch size of\n> 1-3gb, probably) can condense the remaining packs in commands that run for\n> 1-3 min at a time. Since the daily packs range from 100-250mb, we will also\n> combine and condense those packs.\n> \n> Updates in V2:\n> \n> * Added a method, unlink_pack_path() to remove packfiles, but with the\n> additional check for a .keep file. This borrows logic from \n> builtin/repack.c.\n> \n> \n> * Modified documentation and commit messages to replace 'verb' with\n> 'subcommand'. Simplified the documentation. (I left 'verbs' in the title\n> of the cover letter for consistency.)\n> \n> \n> \n> Updates in V3:\n> \n> * There was a bug in the expire logic when simultaneously removing packs\n> and adding uncovered packs, specifically around the pack permutation.\n> This was hard to see during review because I was using the 'pack_perm'\n> array for multiple purposes. First, I was reducing its length, and then I\n> was adding to it and resorting. In V3, I significantly overhauled the\n> logic here, which required some extra commits before implementing\n> 'expire'. The final commit includes a test that would cover this case.\n> \n> Updates in V4:\n> \n> * More 'verb' and 'command' instances replaced with 'subcommand'. I grepped\n> the patch to check these should be fixed everywhere.\n> \n> \n> * Update the tests to check .keep files (in last patch).\n> \n> \n> * Modify the tests to show the terminating condition of --batch-size when\n> there are three packs that fit under the size, but the first two are\n> large enough to stop adding packs. This required rearranging the packs\n> slightly to get different sizes than we had before. Also, I added 'touch\n> -t' to set the modified times so we can fix the order in which the packs\n> are selected.\n> \n> \n> * Added a comment about the purpose of pack_perm.\n> \n> \n> \n> Thanks, -Stolee\n> \n> Derrick Stolee (10):\n> repack: refactor pack deletion for future use\n> Docs: rearrange subcommands for multi-pack-index\n> multi-pack-index: prepare for 'expire' subcommand\n> midx: simplify computation of pack name lengths\n> midx: refactor permutation logic and pack sorting\n> multi-pack-index: implement 'expire' subcommand\n> multi-pack-index: prepare 'repack' subcommand\n> midx: implement midx_repack()\n> multi-pack-index: test expire while adding packs\n> midx: add test that 'expire' respects .keep files\n> \n> Documentation/git-multi-pack-index.txt | 26 +-\n> builtin/multi-pack-index.c | 14 +-\n> builtin/repack.c | 14 +-\n> midx.c | 399 ++++++++++++++++++-------\n> midx.h | 2 +\n> packfile.c | 28 ++\n> packfile.h | 7 +\n> t/t5319-multi-pack-index.sh | 165 ++++++++++\n> 8 files changed, 536 insertions(+), 119 deletions(-)\n> \n> \n> base-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-92%2Fderrickstolee%2Fmidx-expire%2Fupstream-v4\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-92/derrickstolee/midx-expire/upstream-v4\n> Pull-Request: https://github.com/gitgitgadget/git/pull/92\n> \n> Range-diff vs v3:\n> \n> 1: 62b393b816 = 1: 62b393b816 repack: refactor pack deletion for future use\n> 2: 7886785904 = 2: 7886785904 Docs: rearrange subcommands for multi-pack-index\n> 3: f06382b4ae ! 3: 628ca46036 multi-pack-index: prepare for 'expire' subcommand\n> @@ -16,7 +16,9 @@\n> Add a test that verifies the 'expire' subcommand is correctly wired,\n> but will still be valid when the verb is implemented. Specifically,\n> create a set of packs that should all have referenced objects and\n> - should not be removed during an 'expire' operation.\n> + should not be removed during an 'expire' operation. The packs are\n> + created carefully to ensure they have a specific order when sorted\n> + by size. This will be important in a later test.\n> \n> Signed-off-by: Derrick Stolee \n> \n> @@ -95,6 +97,8 @@\n> +\t(\n> +\t\tcd dup &&\n> +\t\tgit init &&\n> ++\t\ttest-tool genrandom \"data\" 4096 >large_file.txt &&\n> ++\t\tgit update-index --add large_file.txt &&\n> +\t\tfor i in $(test_seq 1 20)\n> +\t\tdo\n> +\t\t\ttest_commit $i\n> @@ -104,24 +108,24 @@\n> +\t\tgit branch C HEAD~13 &&\n> +\t\tgit branch D HEAD~16 &&\n> +\t\tgit branch E HEAD~18 &&\n> -+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n> -+\t\trefs/heads/E\n> ++\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n> ++\t\trefs/heads/A\n> ++\t\t^refs/heads/B\n> +\t\tEOF\n> -+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n> -+\t\trefs/heads/D\n> -+\t\t^refs/heads/E\n> ++\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n> ++\t\trefs/heads/B\n> ++\t\t^refs/heads/C\n> +\t\tEOF\n> +\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n> +\t\trefs/heads/C\n> +\t\t^refs/heads/D\n> +\t\tEOF\n> -+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n> -+\t\trefs/heads/B\n> -+\t\t^refs/heads/C\n> ++\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n> ++\t\trefs/heads/D\n> ++\t\t^refs/heads/E\n> +\t\tEOF\n> -+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n> -+\t\trefs/heads/A\n> -+\t\t^refs/heads/B\n> ++\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n> ++\t\trefs/heads/E\n> +\t\tEOF\n> +\t\tgit multi-pack-index write\n> +\t)\n> 4: 2a763990ae ! 4: d55c1d7ee7 midx: simplify computation of pack name lengths\n> @@ -12,7 +12,7 @@\n> dir not already covered by the multi-pack-index.\n> \n> In anticipation of this becoming more complicated with the 'expire'\n> - command, simplify the computation by centralizing it to a single\n> + subcommand, simplify the computation by centralizing it to a single\n> loop before writing the file.\n> \n> Signed-off-by: Derrick Stolee \n> 5: a0d4cc6cb3 ! 5: 3950743b96 midx: refactor permutation logic and pack sorting\n> @@ -282,6 +282,12 @@\n> \n> +\tQSORT(packs.info, packs.nr, pack_info_compare);\n> +\n> ++\t/*\n> ++\t * pack_perm stores a permutation between pack-int-ids from the\n> ++\t * previous multi-pack-index to the new one we are writing:\n> ++\t *\n> ++\t * pack_perm[old_id] = new_id\n> ++\t */\n> +\tALLOC_ARRAY(pack_perm, packs.nr);\n> +\tfor (i = 0; i < packs.nr; i++) {\n> +\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n> 6: 4dbff40e7a ! 6: 6691d97902 multi-pack-index: implement 'expire' verb\n> @@ -1,8 +1,8 @@\n> Author: Derrick Stolee \n> \n> - multi-pack-index: implement 'expire' verb\n> + multi-pack-index: implement 'expire' subcommand\n> \n> - The 'git multi-pack-index expire' command looks at the existing\n> + The 'git multi-pack-index expire' subcommand looks at the existing\n> mult-pack-index, counts the number of objects referenced in each\n> pack-file, deletes the pack-fils with no referenced objects, and\n> rewrites the multi-pack-index to no longer reference those packs.\n> @@ -18,7 +18,7 @@\n> \n> Test that a new pack-file that covers the contents of two other\n> pack-files leads to those pack-files being deleted during the\n> - expire command. Be sure to read the multi-pack-index to ensure\n> + expire subcommand. Be sure to read the multi-pack-index to ensure\n> it no longer references those packs.\n> \n> Signed-off-by: Derrick Stolee \n> @@ -161,6 +161,11 @@\n> +\t\t}\n> +\t}\n> +\n> + \t/*\n> + \t * pack_perm stores a permutation between pack-int-ids from the\n> + \t * previous multi-pack-index to the new one we are writing:\n> +@@\n> + \t */\n> \tALLOC_ARRAY(pack_perm, packs.nr);\n> \tfor (i = 0; i < packs.nr; i++) {\n> -\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n> @@ -273,7 +278,9 @@\n> +\t\ttest_cmp expect actual &&\n> +\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n> +\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n> -+\t\ttest_cmp expect-idx actual-midx\n> ++\t\ttest_cmp expect-idx actual-midx &&\n> ++\t\tgit multi-pack-index verify &&\n> ++\t\tgit fsck\n> +\t)\n> +'\n> +\n> 7: b39f90ad09 ! 7: f5a8ff21dd multi-pack-index: prepare 'repack' subcommand\n> @@ -11,7 +11,7 @@\n> operation does not interrupt concurrent git commands.\n> \n> Introduce a 'repack' subcommand to 'git multi-pack-index' that\n> - takes a '--batch-size' option. The verb will inspect the\n> + takes a '--batch-size' option. The subcommand will inspect the\n> multi-pack-index for referenced pack-files whose size is smaller\n> than the batch size, until collecting a list of pack-files whose\n> sizes sum to larger than the batch size. Then, a new pack-file\n> @@ -26,6 +26,11 @@\n> we specify a small batch size, we will guarantee that future\n> implementations do not change the list of pack-files.\n> \n> + In addition, we hard-code the modified times of the packs in\n> + the pack directory to ensure the list of packs sorted by modified\n> + time matches the order if sorted by size (ascending). This will\n> + be important in a future test.\n> +\n> Signed-off-by: Derrick Stolee \n> \n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> @@ -36,15 +41,15 @@\n> \tafterward to remove all references to these pack-files.\n> \n> +repack::\n> -+\tCollect a batch of pack-files whose size are all at most the\n> -+\tsize given by --batch-size, but whose sizes sum to larger\n> -+\tthan --batch-size. The batch is selected by greedily adding\n> -+\tsmall pack-files starting with the oldest pack-files that fit\n> -+\tthe size. Create a new pack-file containing the objects the\n> -+\tmulti-pack-index indexes into those pack-files, and rewrite\n> -+\tthe multi-pack-index to contain that pack-file. A later run\n> -+\tof 'git multi-pack-index expire' will delete the pack-files\n> -+\tthat were part of this batch.\n> ++\tCreate a new pack-file containing objects in small pack-files\n> ++\treferenced by the multi-pack-index. Select the pack-files by\n> ++\texamining packs from oldest-to-newest, adding a pack if its\n> ++\tsize is below the batch size. Stop adding packs when the sum\n> ++\tof sizes of the added packs is above the batch size. If the\n> ++\ttotal size does not reach the batch size, then do nothing.\n> ++\tRewrite the multi-pack-index to reference the new pack-file.\n> ++\tA later run of 'git multi-pack-index expire' will delete the\n> ++\tpack-files that were part of this batch.\n> +\n> \n> EXAMPLES\n> @@ -84,11 +89,18 @@\n> +\tif (!strcmp(argv[0], \"repack\"))\n> +\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n> +\tif (opts.batch_size)\n> -+\t\tdie(_(\"--batch-size option is only for 'repack' verb\"));\n> ++\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n> +\n> \tif (!strcmp(argv[0], \"write\"))\n> \t\treturn write_midx_file(opts.object_dir);\n> \tif (!strcmp(argv[0], \"verify\"))\n> +@@\n> + \tif (!strcmp(argv[0], \"expire\"))\n> + \t\treturn expire_midx_packs(opts.object_dir);\n> + \n> +-\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n> ++\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n> + }\n> \n> diff --git a/midx.c b/midx.c\n> --- a/midx.c\n> @@ -125,6 +137,12 @@\n> +test_expect_success 'repack with minimum size does not alter existing packs' '\n> +\t(\n> +\t\tcd dup &&\n> ++\t\trm -rf .git/objects/pack &&\n> ++\t\tmv .git/objects/pack-backup .git/objects/pack &&\n> ++\t\ttouch -m -t 201901010000 .git/objects/pack/pack-D* &&\n> ++\t\ttouch -m -t 201901010001 .git/objects/pack/pack-C* &&\n> ++\t\ttouch -m -t 201901010002 .git/objects/pack/pack-B* &&\n> ++\t\ttouch -m -t 201901010003 .git/objects/pack/pack-A* &&\n> +\t\tls .git/objects/pack >expect &&\n> +\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n> +\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n> 8: a4c2d5a8e1 ! 8: ba1a1c7bbb midx: implement midx_repack()\n> @@ -149,6 +149,16 @@\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> +@@\n> + \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n> + \t\trefs/heads/E\n> + \t\tEOF\n> +-\t\tgit multi-pack-index write\n> ++\t\tgit multi-pack-index write &&\n> ++\t\tcp -r .git/objects/pack .git/objects/pack-backup\n> + \t)\n> + '\n> + \n> @@\n> \t)\n> '\n> @@ -156,25 +166,28 @@\n> +test_expect_success 'repack creates a new pack' '\n> +\t(\n> +\t\tcd dup &&\n> -+\t\tSECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 2 | tail -n 1) &&\n> -+\t\tBATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&\n> -+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n> +\t\tls .git/objects/pack/*idx >idx-list &&\n> +\t\ttest_line_count = 5 idx-list &&\n> ++\t\tTHIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 3 | tail -n 1) &&\n> ++\t\tBATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) &&\n> ++\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n> ++\t\tls .git/objects/pack/*idx >idx-list &&\n> ++\t\ttest_line_count = 6 idx-list &&\n> +\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n> -+\t\ttest_line_count = 5 midx-list\n> ++\t\ttest_line_count = 6 midx-list\n> +\t)\n> +'\n> +\n> +test_expect_success 'expire removes repacked packs' '\n> +\t(\n> +\t\tcd dup &&\n> -+\t\tls -S .git/objects/pack/*pack | head -n 3 >expect &&\n> ++\t\tls -al .git/objects/pack/*pack &&\n> ++\t\tls -S .git/objects/pack/*pack | head -n 4 >expect &&\n> +\t\tgit multi-pack-index expire &&\n> +\t\tls -S .git/objects/pack/*pack >actual &&\n> +\t\ttest_cmp expect actual &&\n> +\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n> -+\t\ttest_line_count = 3 midx-list\n> ++\t\ttest_line_count = 4 midx-list\n> +\t)\n> +'\n> +\n> 9: b97fb35ba9 = 9: b1c6892417 multi-pack-index: test expire while adding packs\n> -: ---------- > 10: 481b08890f midx: add test that 'expire' respects .keep files\n> \n> -- \n> gitgitgadget\n\nWith the exception of the broken test in patch 7, and some minor style\ndiffs, this all looks good to me.\n"},{"id":"367710","messageId":"2970aa83-40cd-e721-af4c-a27798995eab@gmail.com","threadId":"49993","inReplyTo":"ba1a1c7bbbdbd8f3b82187763df4ec21b7032fb5.1548366713.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 08/10] midx: implement midx_repack()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-26T17:10:01Z","receivedAt":"2019-01-26T17:10:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2019 4:52 PM, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index acc5e65ecc..d6c1353514 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -383,7 +383,8 @@ test_expect_success 'setup expire tests' '\n> \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n> \t\trefs/heads/E\n> \t\tEOF\n> -\t\tgit multi-pack-index write\n> +\t\tgit multi-pack-index write &&\n> +\t\tcp -r .git/objects/pack .git/objects/pack-backup\n> \t)\n> '\n\nJosh: Thanks for catching the failure in PATCH 7. It's due to this line \nthat should be part of that commit, not this one.\n\nThanks,\n\n-Stolee\n\n"},{"id":"367780","messageId":"xmqqwompemv4.fsf@gitster-ct.c.googlers.com","threadId":"49993","inReplyTo":"2970aa83-40cd-e721-af4c-a27798995eab@gmail.com","subject":"Re: [PATCH v4 08/10] midx: implement midx_repack()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-27T22:50:39Z","receivedAt":"2019-01-27T22:50:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 1/24/2019 4:52 PM, Derrick Stolee via GitGitGadget wrote:\n>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n>> index acc5e65ecc..d6c1353514 100755\n>> --- a/t/t5319-multi-pack-index.sh\n>> +++ b/t/t5319-multi-pack-index.sh\n>> @@ -383,7 +383,8 @@ test_expect_success 'setup expire tests' '\n>> \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n>> \t\trefs/heads/E\n>> \t\tEOF\n>> -\t\tgit multi-pack-index write\n>> +\t\tgit multi-pack-index write &&\n>> +\t\tcp -r .git/objects/pack .git/objects/pack-backup\n>> \t)\n>> '\n>\n> Josh: Thanks for catching the failure in PATCH 7. It's due to this\n> line that should be part of that commit, not this one.\n\nWill move the hunk while queuing. Thanks, both.\n"},{"id":"374395","messageId":"20190424151428.170316-1-dstolee@microsoft.com","threadId":"49993","inReplyTo":"pull.92.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v5 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:17Z","receivedAt":"2019-04-24T15:14:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index provides a fast way to find an object among a large list\nof pack-files. It stores a single pack-reference for each object id, so\nduplicate objects are ignored. Among a list of pack-files storing the same\nobject, the most-recently modified one is used.\n\nCreate new subcommands for the multi-pack-index builtin.\n\n* 'git multi-pack-index expire': If we have a pack-file indexed by the\n multi-pack-index, but all objects in that pack are duplicated in\n more-recently modified packs, then delete that pack (and any others like it).\n Delete the reference to that pack in the multi-pack-index.\n\n* 'git multi-pack-index repack --batch-size=': Starting from the oldest\n pack-files covered by the multi-pack-index, find those whose \"expected size\"\n is below the batch size until we have a collection of packs whose expected\n sizes add up to the batch size. We compute the expected size by multiplying\n the number of referenced objects by the pack-size and dividing by the total\n number of objects in the pack. If the batch-size is zero, then select all\n packs. Create a new pack containing all objects that the multi-pack-index\n references to those packs.\n\nThis allows us to create a new pattern for repacking objects: run 'repack'.\nAfter enough time has passed that all Git commands that started before the last\n'repack' are finished, run 'expire' again. This approach has some advantages\nover the existing \"repack everything\" model:\n\n1. Incremental. We can repack a small batch of objects at a time, instead of\nrepacking all reachable objects. We can also limit ourselves to the objects\nthat do not appear in newer pack-files.\n\n2. Highly Available. By adding a new pack-file (and not deleting the old\npack-files) we do not interrupt concurrent Git commands, and do not suffer\nperformance degradation. By expiring only pack-files that have no referenced\nobjects, we know that Git commands that are doing normal object lookups* will\nnot be interrupted.\n\n* Note: if someone concurrently runs a Git command that uses get_all_packs(),\n* then that command could try to read the pack-files and pack-indexes that we\n* are deleting during an expire command. Such commands are usually related to\n* object maintenance (i.e. fsck, gc, pack-objects) or are related to\n* less-often-used features (i.e. fast-import, http-backend, server-info).\n\nWe **are using** this approach in VFS for Git to do background maintenance of\nthe \"shared object cache\" which is a Git alternate directory filled with\npackfiles containing commits and trees. We currently download pack-files on an\nhourly basis to keep up-to-date with the central server. The cache servers\nsupply packs on an hourly and daily basis, so most of the hourly packs become\nuseless after a new daily pack is downloaded. The 'expire' command would clear\nout most of those packs, but many will still remain with fewer than 100 objects\nremaining. The 'repack' command (with a batch size of 1-3gb, probably) can\ncondense the remaining packs in commands that run for 1-3 min at a time. Since\nthe daily packs range from 100-250mb, we will also combine and condense those\npacks.\n\nUpdates in V5:\n\n* Fixed the error in PATCH 7 due to a missing line that existed in PATCH 8. Thanks, Josh Steadmon!\n\n* The 'repack' subcommand now computes the \"expected size\" of a pack instead of\n relying on the total size of the pack. This is actually really important to\n the way VFS for Git uses prefetch packs, and some packs are not being\n repacked because the pack size is larger than the batch size, but really\n there are only a few referenced objects.\n\n* The 'repack' subcommand now allows a batch size of zero to mean \"create one\n pack containing all objects in the multi-pack-index\". A new commit adds a\n test that hits the boundary cases here, but follows the 'expire' subcommand\n so we can show that cycle of repack-then-expire to safely replace the packs.\n\nJunio: It appears that there are some conflicts with the trace2 changes in\nmaster. These are not new to the updates in this version. I saw how you\nresolved these conflicts and replaying that resolution should work for you.\n\nThanks,\n-Stolee\n\nDerrick Stolee (11):\n repack: refactor pack deletion for future use\n Docs: rearrange subcommands for multi-pack-index\n multi-pack-index: prepare for 'expire' subcommand\n midx: simplify computation of pack name lengths\n midx: refactor permutation logic and pack sorting\n multi-pack-index: implement 'expire' subcommand\n multi-pack-index: prepare 'repack' subcommand\n midx: implement midx_repack()\n multi-pack-index: test expire while adding packs\n midx: add test that 'expire' respects .keep files\n t5319-multi-pack-index.sh: test batch size zero\n\n Documentation/git-multi-pack-index.txt | 32 +-\n builtin/multi-pack-index.c | 14 +-\n builtin/repack.c | 14 +-\n midx.c | 440 +++++++++++++++++++------\n midx.h | 2 +\n packfile.c | 28 ++\n packfile.h | 7 +\n t/t5319-multi-pack-index.sh | 184 +++++++++++\n 8 files changed, 602 insertions(+), 119 deletions(-)\n\n\nbase-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595\n-- \n2.21.0.1096.g1c91fdc207\n\nRange diff against v4:\n\n 1: 62b393b816 = 1: 62b393b816 repack: refactor pack deletion for future use\n 2: 7886785904 = 2: 7886785904 Docs: rearrange subcommands for multi-pack-index\n 3: 628ca46036 = 3: 628ca46036 multi-pack-index: prepare for 'expire' subcommand\n 4: d55c1d7ee7 = 4: d55c1d7ee7 midx: simplify computation of pack name lengths\n 5: 3950743b96 = 5: 3950743b96 midx: refactor permutation logic and pack sorting\n 6: 6691d97902 = 6: 6691d97902 multi-pack-index: implement 'expire' subcommand\n 7: f5a8ff21dd ! 7: e66e383231 multi-pack-index: prepare 'repack' subcommand\n @@ -42,14 +42,20 @@\n \n +repack::\n +\tCreate a new pack-file containing objects in small pack-files\n -+\treferenced by the multi-pack-index. Select the pack-files by\n -+\texamining packs from oldest-to-newest, adding a pack if its\n -+\tsize is below the batch size. Stop adding packs when the sum\n -+\tof sizes of the added packs is above the batch size. If the\n -+\ttotal size does not reach the batch size, then do nothing.\n -+\tRewrite the multi-pack-index to reference the new pack-file.\n -+\tA later run of 'git multi-pack-index expire' will delete the\n -+\tpack-files that were part of this batch.\n ++\treferenced by the multi-pack-index. If the size given by the\n ++\t`--batch-size=` argument is zero, then create a pack\n ++\tcontaining all objects referenced by the multi-pack-index. For\n ++\ta non-zero batch size, Select the pack-files by examining packs\n ++\tfrom oldest-to-newest, computing the \"expected size\" by counting\n ++\tthe number of objects in the pack referenced by the\n ++\tmulti-pack-index, then divide by the total number of objects in\n ++\tthe pack and multiply by the pack size. We select packs with\n ++\texpected size below the batch size until the set of packs have\n ++\ttotal expected size at least the batch size. If the total size\n ++\tdoes not reach the batch size, then do nothing. If a new pack-\n ++\tfile is created, rewrite the multi-pack-index to reference the\n ++\tnew pack-file. A later run of 'git multi-pack-index expire' will\n ++\tdelete the pack-files that were part of this batch.\n +\n \n EXAMPLES\n @@ -130,6 +136,16 @@\n diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n --- a/t/t5319-multi-pack-index.sh\n +++ b/t/t5319-multi-pack-index.sh\n +@@\n + \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n + \t\trefs/heads/E\n + \t\tEOF\n +-\t\tgit multi-pack-index write\n ++\t\tgit multi-pack-index write &&\n ++\t\tcp -r .git/objects/pack .git/objects/pack-backup\n + \t)\n + '\n + \n @@\n \t)\n '\n 8: ba1a1c7bbb ! 8: 8e2a1140a5 midx: implement midx_repack()\n @@ -2,20 +2,35 @@\n \n midx: implement midx_repack()\n \n - To repack using a multi-pack-index, first sort all pack-files by\n + To repack with a non-zero batch-size, first sort all pack-files by\n their modified time. Second, walk those pack-files from oldest\n - to newest, adding the packs to a list if they are smaller than the\n - given pack-size. Finally, collect the objects from the multi-pack-\n - index that are in those packs and send them to 'git pack-objects'.\n + to newest, compute their expected size, and add the packs to a list\n + if they are smaller than the given batch-size. Stop when the total\n + expected size is at least the batch size.\n +\n + If the batch size is zero, select all packs in the multi-pack-index.\n +\n + Finally, collect the objects from the multi-pack-index that are in\n + the selected packs and send them to 'git pack-objects'. Write a new\n + multi-pack-index that includes the new pack.\n +\n + Using a batch size of zero is very similar to a standard 'git repack'\n + command, except that we do not delete the old packs and instead rely\n + on the new multi-pack-index to prevent new processes from reading the\n + old packs. This does not disrupt other Git processes that are currently\n + reading the old packs based on the old multi-pack-index.\n \n While first designing a 'git multi-pack-index repack' operation, I\n - started by collecting the batches based on the size of the objects\n - instead of the size of the pack-files. This allows repacking a\n - large pack-file that has very few referencd objects. However, this\n + started by collecting the batches based on the actual size of the\n + objects instead of the size of the pack-files. This allows repacking\n + a large pack-file that has very few referencd objects. However, this\n came at a significant cost of parsing pack-files instead of simply\n reading the multi-pack-index and getting the file information for\n - the pack-files. This object-size idea could be a direction for\n - future expansion in this area.\n + the pack-files. The \"expected size\" version provides similar\n + behavior, but could skip a pack-file if the average object size is\n + much larger than the actual size of the referenced objects, or\n + can create a large pack if the actual size of the referenced objects\n + is larger than the expected size.\n \n Signed-off-by: Derrick Stolee \n \n @@ -35,69 +50,111 @@\n }\n \n -int midx_repack(const char *object_dir, size_t batch_size)\n -+struct time_and_id {\n ++struct repack_info {\n +\ttimestamp_t mtime;\n ++\tuint32_t referenced_objects;\n +\tuint32_t pack_int_id;\n +};\n +\n +static int compare_by_mtime(const void *a_, const void *b_)\n {\n -+\tconst struct time_and_id *a, *b;\n ++\tconst struct repack_info *a, *b;\n +\n -+\ta = (const struct time_and_id *)a_;\n -+\tb = (const struct time_and_id *)b_;\n ++\ta = (const struct repack_info *)a_;\n ++\tb = (const struct repack_info *)b_;\n +\n +\tif (a->mtime < b->mtime)\n +\t\treturn -1;\n +\tif (a->mtime > b->mtime)\n +\t\treturn 1;\n - \treturn 0;\n - }\n ++\treturn 0;\n ++}\n +\n -+int midx_repack(const char *object_dir, size_t batch_size)\n ++static int fill_included_packs_all(struct multi_pack_index *m,\n ++\t\t\t\t unsigned char *include_pack)\n +{\n -+\tint result = 0;\n -+\tuint32_t i, packs_to_repack;\n -+\tsize_t total_size;\n -+\tstruct time_and_id *pack_ti;\n -+\tunsigned char *include_pack;\n -+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n -+\tstruct strbuf base_name = STRBUF_INIT;\n -+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n ++\tuint32_t i;\n +\n -+\tif (!m)\n -+\t\treturn 0;\n ++\tfor (i = 0; i < m->num_packs; i++)\n ++\t\tinclude_pack[i] = 1;\n +\n -+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n -+\tpack_ti = xcalloc(m->num_packs, sizeof(struct time_and_id));\n ++\treturn m->num_packs < 2;\n ++}\n ++\n ++static int fill_included_packs_batch(struct multi_pack_index *m,\n ++\t\t\t\t unsigned char *include_pack,\n ++\t\t\t\t size_t batch_size)\n ++{\n ++\tuint32_t i, packs_to_repack;\n ++\tsize_t total_size;\n ++\tstruct repack_info *pack_info = xcalloc(m->num_packs, sizeof(struct repack_info));\n +\n +\tfor (i = 0; i < m->num_packs; i++) {\n -+\t\tpack_ti[i].pack_int_id = i;\n ++\t\tpack_info[i].pack_int_id = i;\n +\n +\t\tif (prepare_midx_pack(m, i))\n +\t\t\tcontinue;\n +\n -+\t\tpack_ti[i].mtime = m->packs[i]->mtime;\n ++\t\tpack_info[i].mtime = m->packs[i]->mtime;\n ++\t}\n ++\n ++\tfor (i = 0; batch_size && i < m->num_objects; i++) {\n ++\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n ++\t\tpack_info[pack_int_id].referenced_objects++;\n +\t}\n -+\tQSORT(pack_ti, m->num_packs, compare_by_mtime);\n ++\n ++\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n +\n +\ttotal_size = 0;\n +\tpacks_to_repack = 0;\n +\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n -+\t\tint pack_int_id = pack_ti[i].pack_int_id;\n ++\t\tint pack_int_id = pack_info[i].pack_int_id;\n +\t\tstruct packed_git *p = m->packs[pack_int_id];\n ++\t\tsize_t expected_size;\n +\n +\t\tif (!p)\n +\t\t\tcontinue;\n -+\t\tif (p->pack_size >= batch_size)\n ++\t\tif (open_pack_index(p) || !p->num_objects)\n ++\t\t\tcontinue;\n ++\n ++\t\texpected_size = (size_t)(p->pack_size\n ++\t\t\t\t\t * pack_info[i].referenced_objects);\n ++\t\texpected_size /= p->num_objects;\n ++\n ++\t\tif (expected_size >= batch_size)\n +\t\t\tcontinue;\n +\n +\t\tpacks_to_repack++;\n -+\t\ttotal_size += p->pack_size;\n ++\t\ttotal_size += expected_size;\n +\t\tinclude_pack[pack_int_id] = 1;\n +\t}\n +\n ++\tfree(pack_info);\n ++\n +\tif (total_size < batch_size || packs_to_repack < 2)\n ++\t\treturn 1;\n ++\n + \treturn 0;\n ++}\t\n ++\n ++int midx_repack(const char *object_dir, size_t batch_size)\n ++{\n ++\tint result = 0;\n ++\tuint32_t i;\n ++\tunsigned char *include_pack;\n ++\tstruct child_process cmd = CHILD_PROCESS_INIT;\n ++\tstruct strbuf base_name = STRBUF_INIT;\n ++\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n ++\n ++\tif (!m)\n ++\t\treturn 0;\n ++\n ++\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n ++\n ++\tif (batch_size) {\n ++\t\tif (fill_included_packs_batch(m, include_pack, batch_size))\n ++\t\t\tgoto cleanup;\n ++\t} else if (fill_included_packs_all(m, include_pack))\n +\t\tgoto cleanup;\n +\n +\targv_array_push(&cmd.args, \"pack-objects\");\n @@ -142,23 +199,12 @@\n +\tif (m)\n +\t\tclose_midx(m);\n +\tfree(include_pack);\n -+\tfree(pack_ti);\n +\treturn result;\n -+}\n + }\n \n diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n --- a/t/t5319-multi-pack-index.sh\n +++ b/t/t5319-multi-pack-index.sh\n -@@\n - \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n - \t\trefs/heads/E\n - \t\tEOF\n --\t\tgit multi-pack-index write\n -+\t\tgit multi-pack-index write &&\n -+\t\tcp -r .git/objects/pack .git/objects/pack-backup\n - \t)\n - '\n - \n @@\n \t)\n '\n 9: b1c6892417 = 9: f1e4669bf6 multi-pack-index: test expire while adding packs\n10: 481b08890f = 10: b1e9df9980 midx: add test that 'expire' respects .keep files\n -: ---------- > 11: f6217e9fae t5319-multi-pack-index.sh: test batch size zero\n"},{"id":"374396","messageId":"20190424151428.170316-2-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 01/11] repack: refactor pack deletion for future use","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:18Z","receivedAt":"2019-04-24T15:14:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The repack builtin deletes redundant pack-files and their\nassociated .idx, .promisor, .bitmap, and .keep files. We will want\nto re-use this logic in the future for other types of repack, so\npull the logic into 'unlink_pack_path()' in packfile.c.\n\nThe 'ignore_keep' parameter is enabled for the use in repack, but\nwill be important for a future caller.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 14 ++------------\n packfile.c | 28 ++++++++++++++++++++++++++++\n packfile.h | 7 +++++++\n 3 files changed, 37 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 45583683ee..3d445b34b4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n-\tconst char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n-\tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t plen;\n-\n-\tstrbuf_addf(&buf, \"%s/%s\", dir_name, base_name);\n-\tplen = buf.len;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tstrbuf_setlen(&buf, plen);\n-\t\tstrbuf_addstr(&buf, exts[i]);\n-\t\tunlink(buf.buf);\n-\t}\n+\tstrbuf_addf(&buf, \"%s/%s.pack\", dir_name, base_name);\n+\tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex d1e6683ffe..bacecb4d0d 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o)\n \t}\n }\n \n+void unlink_pack_path(const char *pack_name, int force_delete)\n+{\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tint i;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t plen;\n+\n+\tstrbuf_addstr(&buf, pack_name);\n+\tstrip_suffix_mem(buf.buf, &buf.len, \".pack\");\n+\tplen = buf.len;\n+\n+\tif (!force_delete) {\n+\t\tstrbuf_addstr(&buf, \".keep\");\n+\t\tif (!access(buf.buf, F_OK)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tstrbuf_setlen(&buf, plen);\n+\t\tstrbuf_addstr(&buf, exts[i]);\n+\t\tunlink(buf.buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\ndiff --git a/packfile.h b/packfile.h\nindex 6c4037605d..5b7bcdb1dd 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -86,6 +86,13 @@ extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Unlink the .pack and associated extension files.\n+ * Does not unlink if 'force_delete' is false and the pack-file is\n+ * marked as \".keep\".\n+ */\n+extern void unlink_pack_path(const char *pack_name, int force_delete);\n+\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374397","messageId":"20190424151428.170316-4-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 03/11] multi-pack-index: prepare for 'expire' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:20Z","receivedAt":"2019-04-24T15:14:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index tracks objects in a collection of pack-files.\nOnly one copy of each object is indexed, using the modified time\nof the pack-files to determine tie-breakers. It is possible to\nhave a pack-file with no referenced objects because all objects\nhave a duplicate in a newer pack-file.\n\nIntroduce a new 'expire' subcommand to the multi-pack-index builtin.\nThis subcommand will delete these unused pack-files and rewrite the\nmulti-pack-index to no longer refer to those files. More details\nabout the specifics will follow as the method is implemented.\n\nAdd a test that verifies the 'expire' subcommand is correctly wired,\nbut will still be valid when the verb is implemented. Specifically,\ncreate a set of packs that should all have referenced objects and\nshould not be removed during an 'expire' operation. The packs are\ncreated carefully to ensure they have a specific order when sorted\nby size. This will be important in a later test.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 5 +++\n builtin/multi-pack-index.c | 4 ++-\n midx.c | 5 +++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 49 ++++++++++++++++++++++++++\n 5 files changed, 63 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 1af406aca2..6186c4c936 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -31,6 +31,11 @@ write::\n verify::\n \tVerify the contents of the MIDX file.\n \n+expire::\n+\tDelete the pack-files that are tracked \tby the MIDX file, but\n+\thave no objects referenced by the MIDX. Rewrite the MIDX file\n+\tafterward to remove all references to these pack-files.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex fca70f8e4f..145de3a46c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,7 +5,7 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n \tNULL\n };\n \n@@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(opts.object_dir);\n+\tif (!strcmp(argv[0], \"expire\"))\n+\t\treturn expire_midx_packs(opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 730ff84dff..bb825ef816 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir)\n \n \treturn verify_midx_error;\n }\n+\n+int expire_midx_packs(const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 774f652530..e3a2b740b5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n+int expire_midx_packs(const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 70926b5bc0..a8528f7da0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -348,4 +348,53 @@ test_expect_success 'verify incorrect 64-bit offset' '\n \t\t\"incorrect object offset\"\n '\n \n+test_expect_success 'setup expire tests' '\n+\tmkdir dup &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit init &&\n+\t\ttest-tool genrandom \"data\" 4096 >large_file.txt &&\n+\t\tgit update-index --add large_file.txt &&\n+\t\tfor i in $(test_seq 1 20)\n+\t\tdo\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit branch A HEAD &&\n+\t\tgit branch B HEAD~8 &&\n+\t\tgit branch C HEAD~13 &&\n+\t\tgit branch D HEAD~16 &&\n+\t\tgit branch E HEAD~18 &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n+test_expect_success 'expire does not remove any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374398","messageId":"20190424151428.170316-3-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 02/11] Docs: rearrange subcommands for multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:19Z","receivedAt":"2019-04-24T15:14:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We will add new subcommands to the multi-pack-index, and that will\nmake the documentation a bit messier. Clean up the 'verb'\ndescriptions by renaming the concept to 'subcommand' and removing\nthe reference to the object directory.\n\nHelped-by: Stefan Beller \nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex f7778a2c85..1af406aca2 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] \n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -23,13 +23,13 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+The following subcommands are available:\n+\n write::\n-\tWhen given as the verb, write a new MIDX file to\n-\t`/packs/multi-pack-index`.\n+\tWrite a new MIDX file.\n \n verify::\n-\tWhen given as the verb, verify the contents of the MIDX file\n-\tat `/packs/multi-pack-index`.\n+\tVerify the contents of the MIDX file.\n \n \n EXAMPLES\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374399","messageId":"20190424151428.170316-5-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 04/11] midx: simplify computation of pack name lengths","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:21Z","receivedAt":"2019-04-24T15:14:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before writing the multi-pack-index, we compute the length of the\npack-index names concatenated together. This forms the data in the\npack name chunk, and we precompute it to compute chunk offsets.\nThe value is also modified to fit alignment needs.\n\nPreviously, this computation was coupled with adding packs from\nthe existing multi-pack-index and the remaining packs in the object\ndir not already covered by the multi-pack-index.\n\nIn anticipation of this becoming more complicated with the 'expire'\nsubcommand, simplify the computation by centralizing it to a single\nloop before writing the file.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bb825ef816..f087bbbe82 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -383,7 +383,6 @@ struct pack_list {\n \tuint32_t nr;\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n-\tsize_t pack_name_concat_len;\n \tstruct multi_pack_index *m;\n };\n \n@@ -418,7 +417,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t}\n \n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n-\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n \t}\n }\n@@ -762,6 +760,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n+\tint pack_name_concat_len = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -777,7 +776,6 @@ int write_midx_file(const char *object_dir)\n \tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.names = NULL;\n-\tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n@@ -788,7 +786,6 @@ int write_midx_file(const char *object_dir)\n \n \t\t\tpacks.list[packs.nr] = NULL;\n \t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -798,10 +795,6 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n-\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n-\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n@@ -814,6 +807,13 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tfor (i = 0; i < packs.nr; i++)\n+\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -831,7 +831,7 @@ int write_midx_file(const char *object_dir)\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374400","messageId":"20190424151428.170316-8-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 07/11] multi-pack-index: prepare 'repack' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:24Z","receivedAt":"2019-04-24T15:14:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In an environment where the multi-pack-index is useful, it is due\nto many pack-files and an inability to repack the object store\ninto a single pack-file. However, it is likely that many of these\npack-files are rather small, and could be repacked into a slightly\nlarger pack-file without too much effort. It may also be important\nto ensure the object store is highly available and the repack\noperation does not interrupt concurrent git commands.\n\nIntroduce a 'repack' subcommand to 'git multi-pack-index' that\ntakes a '--batch-size' option. The subcommand will inspect the\nmulti-pack-index for referenced pack-files whose size is smaller\nthan the batch size, until collecting a list of pack-files whose\nsizes sum to larger than the batch size. Then, a new pack-file\nwill be created containing the objects from those pack-files that\nare referenced by the multi-pack-index. The resulting pack is\nlikely to actually be smaller than the batch size due to\ncompression and the fact that there may be objects in the pack-\nfiles that have duplicate copies in other pack-files.\n\nThe current change introduces the command-line arguments, and we\nadd a test that ensures we parse these options properly. Since\nwe specify a small batch size, we will guarantee that future\nimplementations do not change the list of pack-files.\n\nIn addition, we hard-code the modified times of the packs in\nthe pack directory to ensure the list of packs sorted by modified\ntime matches the order if sorted by size (ascending). This will\nbe important in a future test.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 17 +++++++++++++++++\n builtin/multi-pack-index.c | 12 ++++++++++--\n midx.c | 5 +++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 20 +++++++++++++++++++-\n 5 files changed, 52 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 6186c4c936..233b2b7862 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -36,6 +36,23 @@ expire::\n \thave no objects referenced by the MIDX. Rewrite the MIDX file\n \tafterward to remove all references to these pack-files.\n \n+repack::\n+\tCreate a new pack-file containing objects in small pack-files\n+\treferenced by the multi-pack-index. If the size given by the\n+\t`--batch-size=` argument is zero, then create a pack\n+\tcontaining all objects referenced by the multi-pack-index. For\n+\ta non-zero batch size, Select the pack-files by examining packs\n+\tfrom oldest-to-newest, computing the \"expected size\" by counting\n+\tthe number of objects in the pack referenced by the\n+\tmulti-pack-index, then divide by the total number of objects in\n+\tthe pack and multiply by the pack size. We select packs with\n+\texpected size below the batch size until the set of packs have\n+\ttotal expected size at least the batch size. If the total size\n+\tdoes not reach the batch size, then do nothing. If a new pack-\n+\tfile is created, rewrite the multi-pack-index to reference the\n+\tnew pack-file. A later run of 'git multi-pack-index expire' will\n+\tdelete the pack-files that were part of this batch.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 145de3a46c..c66239de33 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,12 +5,13 @@\n #include \"midx.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n \tNULL\n };\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tunsigned long batch_size;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n@@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n+\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n@@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn 1;\n \t}\n \n+\tif (!strcmp(argv[0], \"repack\"))\n+\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n+\tif (opts.batch_size)\n+\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n+\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n@@ -47,5 +55,5 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!strcmp(argv[0], \"expire\"))\n \t\treturn expire_midx_packs(opts.object_dir);\n \n-\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n+\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 299e9b2e8f..768a7dff73 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1112,3 +1112,8 @@ int expire_midx_packs(const char *object_dir)\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n }\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex e3a2b740b5..394a21ee96 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(const char *object_dir);\n int expire_midx_packs(const char *object_dir);\n+int midx_repack(const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 65e85debec..26ae8b3f62 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -383,7 +383,8 @@ test_expect_success 'setup expire tests' '\n \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n \t\trefs/heads/E\n \t\tEOF\n-\t\tgit multi-pack-index write\n+\t\tgit multi-pack-index write &&\n+\t\tcp -r .git/objects/pack .git/objects/pack-backup\n \t)\n '\n \n@@ -417,4 +418,21 @@ test_expect_success 'expire removes unreferenced packs' '\n \t)\n '\n \n+test_expect_success 'repack with minimum size does not alter existing packs' '\n+\t(\n+\t\tcd dup &&\n+\t\trm -rf .git/objects/pack &&\n+\t\tmv .git/objects/pack-backup .git/objects/pack &&\n+\t\ttouch -m -t 201901010000 .git/objects/pack/pack-D* &&\n+\t\ttouch -m -t 201901010001 .git/objects/pack/pack-C* &&\n+\t\ttouch -m -t 201901010002 .git/objects/pack/pack-B* &&\n+\t\ttouch -m -t 201901010003 .git/objects/pack/pack-A* &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n+\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374401","messageId":"20190424151428.170316-9-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 08/11] midx: implement midx_repack()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:25Z","receivedAt":"2019-04-24T15:14:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"To repack with a non-zero batch-size, first sort all pack-files by\ntheir modified time. Second, walk those pack-files from oldest\nto newest, compute their expected size, and add the packs to a list\nif they are smaller than the given batch-size. Stop when the total\nexpected size is at least the batch size.\n\nIf the batch size is zero, select all packs in the multi-pack-index.\n\nFinally, collect the objects from the multi-pack-index that are in\nthe selected packs and send them to 'git pack-objects'. Write a new\nmulti-pack-index that includes the new pack.\n\nUsing a batch size of zero is very similar to a standard 'git repack'\ncommand, except that we do not delete the old packs and instead rely\non the new multi-pack-index to prevent new processes from reading the\nold packs. This does not disrupt other Git processes that are currently\nreading the old packs based on the old multi-pack-index.\n\nWhile first designing a 'git multi-pack-index repack' operation, I\nstarted by collecting the batches based on the actual size of the\nobjects instead of the size of the pack-files. This allows repacking\na large pack-file that has very few referencd objects. However, this\ncame at a significant cost of parsing pack-files instead of simply\nreading the multi-pack-index and getting the file information for\nthe pack-files. The \"expected size\" version provides similar\nbehavior, but could skip a pack-file if the average object size is\nmuch larger than the actual size of the referenced objects, or\ncan create a large pack if the actual size of the referenced objects\nis larger than the expected size.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 150 +++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 28 +++++++\n 2 files changed, 177 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 768a7dff73..01c6a05732 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -8,6 +8,7 @@\n #include \"sha1-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n+#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -1113,7 +1114,154 @@ int expire_midx_packs(const char *object_dir)\n \treturn result;\n }\n \n-int midx_repack(const char *object_dir, size_t batch_size)\n+struct repack_info {\n+\ttimestamp_t mtime;\n+\tuint32_t referenced_objects;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n {\n+\tconst struct repack_info *a, *b;\n+\n+\ta = (const struct repack_info *)a_;\n+\tb = (const struct repack_info *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fill_included_packs_all(struct multi_pack_index *m,\n+\t\t\t\t unsigned char *include_pack)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < m->num_packs; i++)\n+\t\tinclude_pack[i] = 1;\n+\n+\treturn m->num_packs < 2;\n+}\n+\n+static int fill_included_packs_batch(struct multi_pack_index *m,\n+\t\t\t\t unsigned char *include_pack,\n+\t\t\t\t size_t batch_size)\n+{\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct repack_info *pack_info = xcalloc(m->num_packs, sizeof(struct repack_info));\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_info[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_info[i].mtime = m->packs[i]->mtime;\n+\t}\n+\n+\tfor (i = 0; batch_size && i < m->num_objects; i++) {\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tpack_info[pack_int_id].referenced_objects++;\n+\t}\n+\n+\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_info[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\t\tsize_t expected_size;\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\t\tcontinue;\n+\n+\t\texpected_size = (size_t)(p->pack_size\n+\t\t\t\t\t * pack_info[i].referenced_objects);\n+\t\texpected_size /= p->num_objects;\n+\n+\t\tif (expected_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += expected_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tfree(pack_info);\n+\n+\tif (total_size < batch_size || packs_to_repack < 2)\n+\t\treturn 1;\n+\n \treturn 0;\n+}\t\n+\n+int midx_repack(const char *object_dir, size_t batch_size)\n+{\n+\tint result = 0;\n+\tuint32_t i;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n+\n+\tif (batch_size) {\n+\t\tif (fill_included_packs_batch(m, include_pack, batch_size))\n+\t\t\tgoto cleanup;\n+\t} else if (fill_included_packs_all(m, include_pack))\n+\t\tgoto cleanup;\n+\n+\targv_array_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\targv_array_push(&cmd.args, base_name.buf);\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\txwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);\n+\t\txwrite(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, m, NULL);\n+\tm = NULL;\n+\n+cleanup:\n+\tif (m)\n+\t\tclose_midx(m);\n+\tfree(include_pack);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 26ae8b3f62..d6c1353514 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -435,4 +435,32 @@ test_expect_success 'repack with minimum size does not alter existing packs' '\n \t)\n '\n \n+test_expect_success 'repack creates a new pack' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 5 idx-list &&\n+\t\tTHIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 3 | tail -n 1) &&\n+\t\tBATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) &&\n+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 6 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 6 midx-list\n+\t)\n+'\n+\n+test_expect_success 'expire removes repacked packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls -al .git/objects/pack/*pack &&\n+\t\tls -S .git/objects/pack/*pack | head -n 4 >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/*pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 4 midx-list\n+\t)\n+'\n+\n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374402","messageId":"20190424151428.170316-10-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 09/11] multi-pack-index: test expire while adding packs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:26Z","receivedAt":"2019-04-24T15:14:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"During development of the multi-pack-index expire subcommand, a\nversion went out that improperly computed the pack order if a new\npack was introduced while other packs were being removed. Part of\nthe subtlety of the bug involved the new pack being placed before\nother packs that already existed in the multi-pack-index.\n\nAdd a test to t5319-multi-pack-index.sh that catches this issue.\nThe test adds new packs that cause another pack to be expired, and\ncreates new packs that are lexicographically sorted before and\nafter the existing packs.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex d6c1353514..19b769eea0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -463,4 +463,36 @@ test_expect_success 'expire removes repacked packs' '\n \t)\n '\n \n+test_expect_success 'expire works when adding new packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/a-pack <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/z-pack <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack/ | grep idx >expect &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374404","messageId":"20190424151428.170316-11-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 10/11] midx: add test that 'expire' respects .keep files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:27Z","receivedAt":"2019-04-24T15:14:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'git multi-pack-index expire' subcommand may delete packs that\nare not needed from the perspective of the multi-pack-index. If\na pack has a .keep file, then we should not delete that pack. Add\na test that ensures we preserve a pack that would otherwise be\nexpired. First, create a new pack that contains every object in\nthe repo, then add it to the multi-pack-index. Then create a .keep\nfile for a pack starting with \"a-pack\" that was added in the\nprevious test. Finally, expire and verify that the pack remains\nand the other packs were expired.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 18 ++++++++++++++++++\n 1 file changed, 18 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 19b769eea0..bcfa520401 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -495,4 +495,22 @@ test_expect_success 'expire works when adding new packs' '\n \t)\n '\n \n+test_expect_success 'expire respects .keep files' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-all <<-EOF &&\n+\t\trefs/heads/A\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n+\t\ttouch $PACKA.keep &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n+\t\ttest_line_count = 3 a-pack-files &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 2 midx-list\n+\t)\n+'\n+\n+\n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374403","messageId":"20190424151428.170316-12-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 11/11] t5319-multi-pack-index.sh: test batch size zero","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:28Z","receivedAt":"2019-04-24T15:14:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'git multi-pack-index repack' command can take a batch size of\nzero, which creates a new pack-file containing all objects in the\nmulti-pack-index. The first 'repack' command will create one new\npack-file, and an 'expire' command after that will delete the old\npack-files, as they no longer contain any referenced objects in the\nmulti-pack-index.\n\nWe must remove the .keep file that was added in the previous test\nin order to expire that pack-file.\n\nAlso test that a 'repack' will do nothing if there is only one\npack-file.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex bcfa520401..0f116b4b92 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -512,5 +512,24 @@ test_expect_success 'expire respects .keep files' '\n \t)\n '\n \n+test_expect_success 'repack --batch-size=0 repacks everything' '\n+\t(\n+\t\tcd dup &&\n+\t\trm .git/objects/pack/*.keep &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 2 idx-list &&\n+\t\tgit multi-pack-index repack --batch-size=0 &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 3 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 3 midx-list &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -al .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 1 idx-list &&\n+\t\tgit multi-pack-index repack --batch-size=0 &&\n+\t\tls -al .git/objects/pack/*idx >new-idx-list &&\n+\t\ttest_cmp idx-list new-idx-list\n+\t)\n+'\n \n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374405","messageId":"20190424151428.170316-7-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 06/11] multi-pack-index: implement 'expire' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:23Z","receivedAt":"2019-04-24T15:15:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'git multi-pack-index expire' subcommand looks at the existing\nmult-pack-index, counts the number of objects referenced in each\npack-file, deletes the pack-fils with no referenced objects, and\nrewrites the multi-pack-index to no longer reference those packs.\n\nRefactor the write_midx_file() method to call write_midx_internal()\nwhich now takes an existing 'struct multi_pack_index' and a list\nof pack-files to drop (as specified by the names of their pack-\nindexes). As we write the new multi-pack-index, we drop those\nfile names from the list of known pack-files.\n\nThe expire_midx_packs() method removes the unreferenced pack-files\nafter carefully closing the packs to avoid open handles.\n\nTest that a new pack-file that covers the contents of two other\npack-files leads to those pack-files being deleted during the\nexpire subcommand. Be sure to read the multi-pack-index to ensure\nit no longer references those packs.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 120 +++++++++++++++++++++++++++++++++---\n t/t5319-multi-pack-index.sh | 20 ++++++\n 2 files changed, 130 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 95c39106b2..299e9b2e8f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,8 @@\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n+#define PACK_EXPIRED UINT_MAX\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -381,6 +383,7 @@ struct pack_info {\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n+\tunsigned expired : 1;\n };\n \n static int pack_info_compare(const void *_a, const void *_b)\n@@ -428,6 +431,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n \t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n+\t\tpacks->info[packs->nr].expired = 0;\n \t\tpacks->nr++;\n \t}\n }\n@@ -587,13 +591,17 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n+\t\tsize_t writelen;\n+\n+\t\tif (info[i].expired)\n+\t\t\tcontinue;\n \n \t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n \t\t\t info[i - 1].pack_name,\n \t\t\t info[i].pack_name);\n \n+\t\twritelen = strlen(info[i].pack_name) + 1;\n \t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n@@ -675,6 +683,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n+\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->pack_int_id);\n+\n \t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n@@ -721,7 +734,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n-int write_midx_file(const char *object_dir)\n+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+\t\t\t struct string_list *packs_to_drop)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -737,6 +751,8 @@ int write_midx_file(const char *object_dir)\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n+\tint dropped_packs = 0;\n+\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -745,7 +761,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n-\tpacks.m = load_multi_pack_index(object_dir, 1);\n+\tif (m)\n+\t\tpacks.m = m;\n+\telse\n+\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n \tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n@@ -759,13 +778,14 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n \t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n \t\t\tpacks.info[packs.nr].p = NULL;\n+\t\t\tpacks.info[packs.nr].expired = 0;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs)\n+\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n \tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n@@ -779,6 +799,34 @@ int write_midx_file(const char *object_dir)\n \n \tQSORT(packs.info, packs.nr, pack_info_compare);\n \n+\tif (packs_to_drop && packs_to_drop->nr) {\n+\t\tint drop_index = 0;\n+\t\tint missing_drops = 0;\n+\n+\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\tif (!cmp) {\n+\t\t\t\tdrop_index++;\n+\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t} else if (cmp > 0) {\n+\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\tdrop_index++;\n+\t\t\t\tmissing_drops++;\n+\t\t\t\ti--;\n+\t\t\t} else {\n+\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (missing_drops) {\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \t/*\n \t * pack_perm stores a permutation between pack-int-ids from the\n \t * previous multi-pack-index to the new one we are writing:\n@@ -787,11 +835,18 @@ int write_midx_file(const char *object_dir)\n \t */\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t\tif (packs.info[i].expired) {\n+\t\t\tdropped_packs++;\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t} else {\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tif (!packs.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -807,7 +862,7 @@ int write_midx_file(const char *object_dir)\n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -908,7 +963,12 @@ int write_midx_file(const char *object_dir)\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-\treturn 0;\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1010,5 +1070,45 @@ int verify_midx_file(const char *object_dir)\n \n int expire_midx_packs(const char *object_dir)\n {\n-\treturn 0;\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t}\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\t\tFREE_AND_NULL(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex a8528f7da0..65e85debec 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -397,4 +397,24 @@ test_expect_success 'expire does not remove any packs' '\n \t)\n '\n \n+test_expect_success 'expire removes unreferenced packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n+\t\ttest_cmp expect-idx actual-midx &&\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374406","messageId":"20190424151428.170316-6-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v5 05/11] midx: refactor permutation logic and pack sorting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-24T15:14:22Z","receivedAt":"2019-04-24T15:15:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of the expire subcommand, refactor the way we sort\nthe packfiles by name. This will greatly simplify our approach to\ndropping expired packs from the list.\n\nFirst, create 'struct pack_info' to replace 'struct pack_pair'.\nThis struct contains the necessary information about a pack,\nincluding its name, a pointer to its packfile struct (if not\nalready in the multi-pack-index), and the original pack-int-id.\n\nSecond, track the pack information using an array of pack_info\nstructs in the pack_list struct. This simplifies the logic around\nthe multiple arrays we were tracking in that struct.\n\nFinally, update get_sorted_entries() to not permute the pack-int-id\nand instead supply the permutation to write_midx_object_offsets().\nThis requires sorting the packs after get_sorted_entries().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 156 +++++++++++++++++++++++++--------------------------------\n 1 file changed, 69 insertions(+), 87 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f087bbbe82..95c39106b2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -377,12 +377,23 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_info {\n+\tuint32_t orig_pack_int_id;\n+\tchar *pack_name;\n+\tstruct packed_git *p;\n+};\n+\n+static int pack_info_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_info *a = (struct pack_info *)_a;\n+\tstruct pack_info *b = (struct pack_info *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n struct pack_list {\n-\tstruct packed_git **list;\n-\tchar **names;\n+\tstruct pack_info *info;\n \tuint32_t nr;\n-\tuint32_t alloc_list;\n-\tuint32_t alloc_names;\n+\tuint32_t alloc;\n \tstruct multi_pack_index *m;\n };\n \n@@ -395,66 +406,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n-\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n+\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n \n-\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n-\t\t\t\t\t\t\tfull_path_len,\n-\t\t\t\t\t\t\t0);\n+\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->list[packs->nr]) {\n+\t\tif (!packs->info[packs->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->list[packs->nr]);\n-\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t\tclose_pack(packs->info[packs->nr].p);\n+\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n \t\tpacks->nr++;\n \t}\n }\n \n-struct pack_pair {\n-\tuint32_t pack_int_id;\n-\tchar *pack_name;\n-};\n-\n-static int pack_pair_compare(const void *_a, const void *_b)\n-{\n-\tstruct pack_pair *a = (struct pack_pair *)_a;\n-\tstruct pack_pair *b = (struct pack_pair *)_b;\n-\treturn strcmp(a->pack_name, b->pack_name);\n-}\n-\n-static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n-{\n-\tuint32_t i;\n-\tstruct pack_pair *pairs;\n-\n-\tALLOC_ARRAY(pairs, nr_packs);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpairs[i].pack_int_id = i;\n-\t\tpairs[i].pack_name = pack_names[i];\n-\t}\n-\n-\tQSORT(pairs, nr_packs, pack_pair_compare);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpack_names[i] = pairs[i].pack_name;\n-\t\tperm[pairs[i].pack_int_id] = i;\n-\t}\n-\n-\tfree(pairs);\n-}\n-\n struct pack_midx_entry {\n \tstruct object_id oid;\n \tuint32_t pack_int_id;\n@@ -480,7 +457,6 @@ static int midx_oid_compare(const void *_a, const void *_b)\n }\n \n static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n-\t\t\t\t uint32_t *pack_perm,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n@@ -488,7 +464,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n-\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n \te->offset = nth_midxed_offset(m, pos);\n \n \t/* consider objects in midx to be from \"old\" packs */\n@@ -522,8 +498,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * of a packfile containing the object).\n */\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n-\t\t\t\t\t\t struct packed_git **p,\n-\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n {\n@@ -534,7 +509,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \tuint32_t start_pack = m ? m->num_packs : 0;\n \n \tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n-\t\ttotal_objects += p[cur_pack]->num_objects;\n+\t\ttotal_objects += info[cur_pack].p->num_objects;\n \n \t/*\n \t * As we de-duplicate by fanout value, we expect the fanout\n@@ -559,7 +534,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n \t\t\t\tnr_fanout++;\n@@ -570,12 +545,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n-\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n-\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(info[cur_pack].p, cur_fanout);\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -604,7 +579,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n }\n \n static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t char **pack_names,\n+\t\t\t\t struct pack_info *info,\n \t\t\t\t uint32_t num_packs)\n {\n \tuint32_t i;\n@@ -612,14 +587,14 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n \n-\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t pack_names[i - 1],\n-\t\t\t pack_names[i]);\n+\t\t\t info[i - 1].pack_name,\n+\t\t\t info[i].pack_name);\n \n-\t\thashwrite(f, pack_names[i], writelen);\n+\t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -690,6 +665,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n }\n \n static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tuint32_t *perm,\n \t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n {\n \tstruct pack_midx_entry *list = objects;\n@@ -699,7 +675,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\thashwrite_be32(f, obj->pack_int_id);\n+\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -772,20 +748,17 @@ int write_midx_file(const char *object_dir)\n \tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n-\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n-\tpacks.alloc_names = packs.alloc_list;\n-\tpacks.list = NULL;\n-\tpacks.names = NULL;\n-\tALLOC_ARRAY(packs.list, packs.alloc_list);\n-\tALLOC_ARRAY(packs.names, packs.alloc_names);\n+\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n+\tpacks.info = NULL;\n+\tALLOC_ARRAY(packs.info, packs.alloc);\n \n \tif (packs.m) {\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n-\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n \n-\t\t\tpacks.list[packs.nr] = NULL;\n-\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n+\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].p = NULL;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -795,10 +768,7 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n-\n-\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -807,8 +777,21 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\n+\t/*\n+\t * pack_perm stores a permutation between pack-int-ids from the\n+\t * previous multi-pack-index to the new one we are writing:\n+\t *\n+\t * pack_perm[old_id] = new_id\n+\t */\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t}\n+\n \tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -879,7 +862,7 @@ int write_midx_file(const char *object_dir)\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -891,7 +874,7 @@ int write_midx_file(const char *object_dir)\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -914,15 +897,14 @@ int write_midx_file(const char *object_dir)\n \n cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.list[i]) {\n-\t\t\tclose_pack(packs.list[i]);\n-\t\t\tfree(packs.list[i]);\n+\t\tif (packs.info[i].p) {\n+\t\t\tclose_pack(packs.info[i].p);\n+\t\t\tfree(packs.info[i].p);\n \t\t}\n-\t\tfree(packs.names[i]);\n+\t\tfree(packs.info[i].pack_name);\n \t}\n \n-\tfree(packs.list);\n-\tfree(packs.names);\n+\tfree(packs.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \n2.21.0.1096.g1c91fdc207\n\n"},{"id":"374457","messageId":"xmqqbm0uy78e.fsf@gitster-ct.c.googlers.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"Re: [PATCH v5 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-25T05:38:41Z","receivedAt":"2019-04-25T05:38:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Updates in V5:\n>\n> * Fixed the error in PATCH 7 due to a missing line that existed in PATCH 8. Thanks, Josh Steadmon!\n>\n> * The 'repack' subcommand now computes the \"expected size\" of a pack instead of\n> relying on the total size of the pack. This is actually really important to\n> the way VFS for Git uses prefetch packs, and some packs are not being\n> repacked because the pack size is larger than the batch size, but really\n> there are only a few referenced objects.\n>\n> * The 'repack' subcommand now allows a batch size of zero to mean \"create one\n> pack containing all objects in the multi-pack-index\". A new commit adds a\n> test that hits the boundary cases here, but follows the 'expire' subcommand\n> so we can show that cycle of repack-then-expire to safely replace the packs.\n\nI guess all of them need to tweak the authorship from the gmail\naddress to the work address on the Signed-off-by: trailer, which I\ncan do (as I noticed it before applying).\n\n"},{"id":"374493","messageId":"651dd41f-4d16-8adf-e128-7ca2f5eff907@gmail.com","threadId":"49993","inReplyTo":"xmqqbm0uy78e.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-25T11:06:06Z","receivedAt":"2019-04-25T11:06:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/25/2019 1:38 AM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> Updates in V5:\n>>\n>> * Fixed the error in PATCH 7 due to a missing line that existed in PATCH 8. Thanks, Josh Steadmon!\n>>\n>> * The 'repack' subcommand now computes the \"expected size\" of a pack instead of\n>> relying on the total size of the pack. This is actually really important to\n>> the way VFS for Git uses prefetch packs, and some packs are not being\n>> repacked because the pack size is larger than the batch size, but really\n>> there are only a few referenced objects.\n>>\n>> * The 'repack' subcommand now allows a batch size of zero to mean \"create one\n>> pack containing all objects in the multi-pack-index\". A new commit adds a\n>> test that hits the boundary cases here, but follows the 'expire' subcommand\n>> so we can show that cycle of repack-then-expire to safely replace the packs.\n> \n> I guess all of them need to tweak the authorship from the gmail\n> address to the work address on the Signed-off-by: trailer, which I\n> can do (as I noticed it before applying).\n\nSorry. Due to the conflicts, GitGitGadget prevented me from submitting in\nmy normal way, so I pulled out format-patch and send-email for the first\ntime in a very long time. I manually added new \"From: \" lines in the bodies\nof the patch files, but they got suppressed, I guess.\n\nThanks,\n-Stolee\n"},{"id":"375552","messageId":"20190514184754.3196-1-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190424151428.170316-1-dstolee@microsoft.com","subject":"[PATCH v6 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:43Z","receivedAt":"2019-05-14T18:48:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index provides a fast way to find an object among a large list\nof pack-files. It stores a single pack-reference for each object id, so\nduplicate objects are ignored. Among a list of pack-files storing the same\nobject, the most-recently modified one is used.\n\nCreate new subcommands for the multi-pack-index builtin.\n\n* 'git multi-pack-index expire': If we have a pack-file indexed by the\n multi-pack-index, but all objects in that pack are duplicated in\n more-recently modified packs, then delete that pack (and any others like it).\n Delete the reference to that pack in the multi-pack-index.\n\n* 'git multi-pack-index repack --batch-size=': Starting from the oldest\n pack-files covered by the multi-pack-index, find those whose \"expected size\"\n is below the batch size until we have a collection of packs whose expected\n sizes add up to the batch size. We compute the expected size by multiplying\n the number of referenced objects by the pack-size and dividing by the total\n number of objects in the pack. If the batch-size is zero, then select all\n packs. Create a new pack containing all objects that the multi-pack-index\n references to those packs.\n\nThis allows us to create a new pattern for repacking objects: run 'repack'.\nAfter enough time has passed that all Git commands that started before the last\n'repack' are finished, run 'expire' again. This approach has some advantages\nover the existing \"repack everything\" model:\n\n1. Incremental. We can repack a small batch of objects at a time, instead of\nrepacking all reachable objects. We can also limit ourselves to the objects\nthat do not appear in newer pack-files.\n\n2. Highly Available. By adding a new pack-file (and not deleting the old\npack-files) we do not interrupt concurrent Git commands, and do not suffer\nperformance degradation. By expiring only pack-files that have no referenced\nobjects, we know that Git commands that are doing normal object lookups* will\nnot be interrupted.\n\n* Note: if someone concurrently runs a Git command that uses get_all_packs(),\n* then that command could try to read the pack-files and pack-indexes that we\n* are deleting during an expire command. Such commands are usually related to\n* object maintenance (i.e. fsck, gc, pack-objects) or are related to\n* less-often-used features (i.e. fast-import, http-backend, server-info).\n\nWe **are using** this approach in VFS for Git to do background maintenance of\nthe \"shared object cache\" which is a Git alternate directory filled with\npackfiles containing commits and trees. We currently download pack-files on an\nhourly basis to keep up-to-date with the central server. The cache servers\nsupply packs on an hourly and daily basis, so most of the hourly packs become\nuseless after a new daily pack is downloaded. The 'expire' command would clear\nout most of those packs, but many will still remain with fewer than 100 objects\nremaining. The 'repack' command (with a batch size of 1-3gb, probably) can\ncondense the remaining packs in commands that run for 1-3 min at a time. Since\nthe daily packs range from 100-250mb, we will also combine and condense those\npacks.\n\nUpdates in V6:\n\nI rebased onto ds/midx-too-many-packs. Thanks, Junio for taking that\nchange first. There were several subtle things that needed to change to\nput this change on top:\n\n* We need a repository struct everywhere since we add pack-files to the\n packed_git list now.\n\n* A FREE_AND_NULL() was dropped after closing a pack because the pack\n is still in the packed_git list after opening.\n\n* I noticed some whitespace problems.\n\nI also expect GMail to munge my added \"From:\" tags, so it will look\nlike the author is \"stolee@gmail.com\" instead of\n\"dstolee@microsoft.com\". Sorry for the continued inconvenience here.\n\nThanks,\n-Stolee\n\nDerrick Stolee (11):\n repack: refactor pack deletion for future use\n Docs: rearrange subcommands for multi-pack-index\n multi-pack-index: prepare for 'expire' subcommand\n midx: simplify computation of pack name lengths\n midx: refactor permutation logic and pack sorting\n multi-pack-index: implement 'expire' subcommand\n multi-pack-index: prepare 'repack' subcommand\n midx: implement midx_repack()\n multi-pack-index: test expire while adding packs\n midx: add test that 'expire' respects .keep files\n t5319-multi-pack-index.sh: test batch size zero\n\n Documentation/git-multi-pack-index.txt | 32 +-\n builtin/multi-pack-index.c | 14 +-\n builtin/repack.c | 14 +-\n midx.c | 440 +++++++++++++++++++------\n midx.h | 2 +\n packfile.c | 28 ++\n packfile.h | 7 +\n t/t5319-multi-pack-index.sh | 184 +++++++++++\n 8 files changed, 602 insertions(+), 119 deletions(-)\n\n-- \n2.22.0.rc0\n\n 1: d8d7629fc0 ! 1: 3b424f7c2a repack: refactor pack deletion for future use\n @@ -81,8 +81,8 @@\n --- a/packfile.h\n +++ b/packfile.h\n @@\n - extern void clear_delta_base_cache(void);\n - extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n + void clear_delta_base_cache(void);\n + struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n +/*\n + * Unlink the .pack and associated extension files.\n 2: d8ed299705 = 2: fe047db570 Docs: rearrange subcommands for multi-pack-index\n 3: 166e03dd77 ! 3: b78967a052 multi-pack-index: prepare for 'expire' subcommand\n @@ -42,7 +42,7 @@\n --- a/builtin/multi-pack-index.c\n +++ b/builtin/multi-pack-index.c\n @@\n - #include \"midx.h\"\n + #include \"trace2.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n -\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n @@ -53,9 +53,9 @@\n @@\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n - \t\treturn verify_midx_file(opts.object_dir);\n + \t\treturn verify_midx_file(the_repository, opts.object_dir);\n +\tif (!strcmp(argv[0], \"expire\"))\n -+\t\treturn expire_midx_packs(opts.object_dir);\n ++\t\treturn expire_midx_packs(the_repository, opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\n @@ -68,7 +68,7 @@\n \treturn verify_midx_error;\n }\n +\n -+int expire_midx_packs(const char *object_dir)\n ++int expire_midx_packs(struct repository *r, const char *object_dir)\n +{\n +\treturn 0;\n +}\n @@ -79,8 +79,8 @@\n @@\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n - int verify_midx_file(const char *object_dir);\n -+int expire_midx_packs(const char *object_dir);\n + int verify_midx_file(struct repository *r, const char *object_dir);\n ++int expire_midx_packs(struct repository *r, const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \n 4: f82ccd0e16 = 4: dec7f384ee midx: simplify computation of pack name lengths\n 5: a4ea2a0fe0 = 5: 989d49d0b2 midx: refactor permutation logic and pack sorting\n 6: 28b99a74da ! 6: 8213541052 multi-pack-index: implement 'expire' subcommand\n @@ -211,7 +211,7 @@\n void clear_midx_file(struct repository *r)\n @@\n \n - int expire_midx_packs(const char *object_dir)\n + int expire_midx_packs(struct repository *r, const char *object_dir)\n {\n -\treturn 0;\n +\tuint32_t i, *count, result = 0;\n @@ -233,7 +233,7 @@\n +\t\tif (count[i])\n +\t\t\tcontinue;\n +\n -+\t\tif (prepare_midx_pack(m, i))\n ++\t\tif (prepare_midx_pack(r, m, i))\n +\t\t\tcontinue;\n +\n +\t\tif (m->packs[i]->pack_keep)\n @@ -241,7 +241,6 @@\n +\n +\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n +\t\tclose_pack(m->packs[i]);\n -+\t\tFREE_AND_NULL(m->packs[i]);\n +\n +\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n +\t\tunlink_pack_path(pack_name, 0);\n 7: b1f7b66948 ! 7: 1776e36f19 multi-pack-index: prepare 'repack' subcommand\n @@ -65,7 +65,7 @@\n --- a/builtin/multi-pack-index.c\n +++ b/builtin/multi-pack-index.c\n @@\n - #include \"midx.h\"\n + #include \"trace2.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n -\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n @@ -89,11 +89,11 @@\n \t};\n \n @@\n - \t\treturn 1;\n - \t}\n + \n + \ttrace2_cmd_mode(argv[0]);\n \n +\tif (!strcmp(argv[0], \"repack\"))\n -+\t\treturn midx_repack(opts.object_dir, (size_t)opts.batch_size);\n ++\t\treturn midx_repack(the_repository, opts.object_dir, (size_t)opts.batch_size);\n +\tif (opts.batch_size)\n +\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n +\n @@ -102,7 +102,7 @@\n \tif (!strcmp(argv[0], \"verify\"))\n @@\n \tif (!strcmp(argv[0], \"expire\"))\n - \t\treturn expire_midx_packs(opts.object_dir);\n + \t\treturn expire_midx_packs(the_repository, opts.object_dir);\n \n -\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n +\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n @@ -116,7 +116,7 @@\n \treturn result;\n }\n +\n -+int midx_repack(const char *object_dir, size_t batch_size)\n ++int midx_repack(struct repository *r, const char *object_dir, size_t batch_size)\n +{\n +\treturn 0;\n +}\n @@ -126,9 +126,9 @@\n +++ b/midx.h\n @@\n void clear_midx_file(struct repository *r);\n - int verify_midx_file(const char *object_dir);\n - int expire_midx_packs(const char *object_dir);\n -+int midx_repack(const char *object_dir, size_t batch_size);\n + int verify_midx_file(struct repository *r, const char *object_dir);\n + int expire_midx_packs(struct repository *r, const char *object_dir);\n ++int midx_repack(struct repository *r, const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \n 8: 6e962fd947 ! 8: ab77ce8afe midx: implement midx_repack()\n @@ -38,9 +38,9 @@\n --- a/midx.c\n +++ b/midx.c\n @@\n - #include \"sha1-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n + #include \"trace2.h\"\n +#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n @@ -49,7 +49,7 @@\n \treturn result;\n }\n \n --int midx_repack(const char *object_dir, size_t batch_size)\n +-int midx_repack(struct repository *r, const char *object_dir, size_t batch_size)\n +struct repack_info {\n +\ttimestamp_t mtime;\n +\tuint32_t referenced_objects;\n @@ -81,7 +81,8 @@\n +\treturn m->num_packs < 2;\n +}\n +\n -+static int fill_included_packs_batch(struct multi_pack_index *m,\n ++static int fill_included_packs_batch(struct repository *r,\n ++\t\t\t\t struct multi_pack_index *m,\n +\t\t\t\t unsigned char *include_pack,\n +\t\t\t\t size_t batch_size)\n +{\n @@ -92,7 +93,7 @@\n +\tfor (i = 0; i < m->num_packs; i++) {\n +\t\tpack_info[i].pack_int_id = i;\n +\n -+\t\tif (prepare_midx_pack(m, i))\n ++\t\tif (prepare_midx_pack(r, m, i))\n +\t\t\tcontinue;\n +\n +\t\tpack_info[i].mtime = m->packs[i]->mtime;\n @@ -135,9 +136,9 @@\n +\t\treturn 1;\n +\n \treturn 0;\n -+}\t\n + }\n +\n -+int midx_repack(const char *object_dir, size_t batch_size)\n ++int midx_repack(struct repository *r, const char *object_dir, size_t batch_size)\n +{\n +\tint result = 0;\n +\tuint32_t i;\n @@ -152,7 +153,7 @@\n +\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n +\n +\tif (batch_size) {\n -+\t\tif (fill_included_packs_batch(m, include_pack, batch_size))\n ++\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n +\t\t\tgoto cleanup;\n +\t} else if (fill_included_packs_all(m, include_pack))\n +\t\tgoto cleanup;\n @@ -200,7 +201,7 @@\n +\t\tclose_midx(m);\n +\tfree(include_pack);\n +\treturn result;\n - }\n ++}\n \n diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n --- a/t/t5319-multi-pack-index.sh\n 9: 321c8990f8 = 9: 80c7d2e581 multi-pack-index: test expire while adding packs\n10: f7e4e76bfe = 10: 8e243939ef midx: add test that 'expire' respects .keep files\n11: 5da2603ed5 = 11: 3ed388f0a8 t5319-multi-pack-index.sh: test batch size zero\n \n"},{"id":"375553","messageId":"20190514184754.3196-2-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 01/11] repack: refactor pack deletion for future use","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:44Z","receivedAt":"2019-05-14T18:48:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The repack builtin deletes redundant pack-files and their\nassociated .idx, .promisor, .bitmap, and .keep files. We will want\nto re-use this logic in the future for other types of repack, so\npull the logic into 'unlink_pack_path()' in packfile.c.\n\nThe 'ignore_keep' parameter is enabled for the use in repack, but\nwill be important for a future caller.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/repack.c | 14 ++------------\n packfile.c | 28 ++++++++++++++++++++++++++++\n packfile.h | 7 +++++++\n 3 files changed, 37 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 67f8978043..1f9e6fad1b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \n static void remove_redundant_pack(const char *dir_name, const char *base_name)\n {\n-\tconst char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n-\tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tsize_t plen;\n-\n-\tstrbuf_addf(&buf, \"%s/%s\", dir_name, base_name);\n-\tplen = buf.len;\n-\n-\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tstrbuf_setlen(&buf, plen);\n-\t\tstrbuf_addstr(&buf, exts[i]);\n-\t\tunlink(buf.buf);\n-\t}\n+\tstrbuf_addf(&buf, \"%s/%s.pack\", dir_name, base_name);\n+\tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex 060de420d1..683ce5674c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o)\n \t}\n }\n \n+void unlink_pack_path(const char *pack_name, int force_delete)\n+{\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tint i;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t plen;\n+\n+\tstrbuf_addstr(&buf, pack_name);\n+\tstrip_suffix_mem(buf.buf, &buf.len, \".pack\");\n+\tplen = buf.len;\n+\n+\tif (!force_delete) {\n+\t\tstrbuf_addstr(&buf, \".keep\");\n+\t\tif (!access(buf.buf, F_OK)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n+\t\tstrbuf_setlen(&buf, plen);\n+\t\tstrbuf_addstr(&buf, exts[i]);\n+\t\tunlink(buf.buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\ndiff --git a/packfile.h b/packfile.h\nindex 12baa6118a..09f5222113 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -94,6 +94,13 @@ void unuse_pack(struct pack_window **);\n void clear_delta_base_cache(void);\n struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Unlink the .pack and associated extension files.\n+ * Does not unlink if 'force_delete' is false and the pack-file is\n+ * marked as \".keep\".\n+ */\n+extern void unlink_pack_path(const char *pack_name, int force_delete);\n+\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n-- \n2.22.0.rc0\n\n"},{"id":"375554","messageId":"20190514184754.3196-3-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 02/11] Docs: rearrange subcommands for multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:45Z","receivedAt":"2019-05-14T18:48:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We will add new subcommands to the multi-pack-index, and that will\nmake the documentation a bit messier. Clean up the 'verb'\ndescriptions by renaming the concept to 'subcommand' and removing\nthe reference to the object directory.\n\nHelped-by: Stefan Beller \nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex f7778a2c85..1af406aca2 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] \n+'git multi-pack-index' [--object-dir=] \n \n DESCRIPTION\n -----------\n@@ -23,13 +23,13 @@ OPTIONS\n \t`/packs/multi-pack-index` for the current MIDX file, and\n \t`/packs` for the pack-files to index.\n \n+The following subcommands are available:\n+\n write::\n-\tWhen given as the verb, write a new MIDX file to\n-\t`/packs/multi-pack-index`.\n+\tWrite a new MIDX file.\n \n verify::\n-\tWhen given as the verb, verify the contents of the MIDX file\n-\tat `/packs/multi-pack-index`.\n+\tVerify the contents of the MIDX file.\n \n \n EXAMPLES\n-- \n2.22.0.rc0\n\n"},{"id":"375555","messageId":"20190514184754.3196-6-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 05/11] midx: refactor permutation logic and pack sorting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:48Z","receivedAt":"2019-05-14T18:48:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of the expire subcommand, refactor the way we sort\nthe packfiles by name. This will greatly simplify our approach to\ndropping expired packs from the list.\n\nFirst, create 'struct pack_info' to replace 'struct pack_pair'.\nThis struct contains the necessary information about a pack,\nincluding its name, a pointer to its packfile struct (if not\nalready in the multi-pack-index), and the original pack-int-id.\n\nSecond, track the pack information using an array of pack_info\nstructs in the pack_list struct. This simplifies the logic around\nthe multiple arrays we were tracking in that struct.\n\nFinally, update get_sorted_entries() to not permute the pack-int-id\nand instead supply the permutation to write_midx_object_offsets().\nThis requires sorting the packs after get_sorted_entries().\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 156 +++++++++++++++++++++++++--------------------------------\n 1 file changed, 69 insertions(+), 87 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 62404620ad..6d4b84e243 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -427,12 +427,23 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+struct pack_info {\n+\tuint32_t orig_pack_int_id;\n+\tchar *pack_name;\n+\tstruct packed_git *p;\n+};\n+\n+static int pack_info_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_info *a = (struct pack_info *)_a;\n+\tstruct pack_info *b = (struct pack_info *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n struct pack_list {\n-\tstruct packed_git **list;\n-\tchar **names;\n+\tstruct pack_info *info;\n \tuint32_t nr;\n-\tuint32_t alloc_list;\n-\tuint32_t alloc_names;\n+\tuint32_t alloc;\n \tstruct multi_pack_index *m;\n };\n \n@@ -445,66 +456,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tif (packs->m && midx_contains_pack(packs->m, file_name))\n \t\t\treturn;\n \n-\t\tALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);\n-\t\tALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);\n+\t\tALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);\n \n-\t\tpacks->list[packs->nr] = add_packed_git(full_path,\n-\t\t\t\t\t\t\tfull_path_len,\n-\t\t\t\t\t\t\t0);\n+\t\tpacks->info[packs->nr].p = add_packed_git(full_path,\n+\t\t\t\t\t\t\t full_path_len,\n+\t\t\t\t\t\t\t 0);\n \n-\t\tif (!packs->list[packs->nr]) {\n+\t\tif (!packs->info[packs->nr].p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(packs->list[packs->nr])) {\n+\t\tif (open_pack_index(packs->info[packs->nr].p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(packs->list[packs->nr]);\n-\t\t\tFREE_AND_NULL(packs->list[packs->nr]);\n+\t\t\tclose_pack(packs->info[packs->nr].p);\n+\t\t\tFREE_AND_NULL(packs->info[packs->nr].p);\n \t\t\treturn;\n \t\t}\n \n-\t\tpacks->names[packs->nr] = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n+\t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n \t\tpacks->nr++;\n \t}\n }\n \n-struct pack_pair {\n-\tuint32_t pack_int_id;\n-\tchar *pack_name;\n-};\n-\n-static int pack_pair_compare(const void *_a, const void *_b)\n-{\n-\tstruct pack_pair *a = (struct pack_pair *)_a;\n-\tstruct pack_pair *b = (struct pack_pair *)_b;\n-\treturn strcmp(a->pack_name, b->pack_name);\n-}\n-\n-static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)\n-{\n-\tuint32_t i;\n-\tstruct pack_pair *pairs;\n-\n-\tALLOC_ARRAY(pairs, nr_packs);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpairs[i].pack_int_id = i;\n-\t\tpairs[i].pack_name = pack_names[i];\n-\t}\n-\n-\tQSORT(pairs, nr_packs, pack_pair_compare);\n-\n-\tfor (i = 0; i < nr_packs; i++) {\n-\t\tpack_names[i] = pairs[i].pack_name;\n-\t\tperm[pairs[i].pack_int_id] = i;\n-\t}\n-\n-\tfree(pairs);\n-}\n-\n struct pack_midx_entry {\n \tstruct object_id oid;\n \tuint32_t pack_int_id;\n@@ -530,7 +507,6 @@ static int midx_oid_compare(const void *_a, const void *_b)\n }\n \n static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n-\t\t\t\t uint32_t *pack_perm,\n \t\t\t\t struct pack_midx_entry *e,\n \t\t\t\t uint32_t pos)\n {\n@@ -538,7 +514,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n \t\treturn 1;\n \n \tnth_midxed_object_oid(&e->oid, m, pos);\n-\te->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];\n+\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n \te->offset = nth_midxed_offset(m, pos);\n \n \t/* consider objects in midx to be from \"old\" packs */\n@@ -572,8 +548,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n * of a packfile containing the object).\n */\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n-\t\t\t\t\t\t struct packed_git **p,\n-\t\t\t\t\t\t uint32_t *perm,\n+\t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects)\n {\n@@ -584,7 +559,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \tuint32_t start_pack = m ? m->num_packs : 0;\n \n \tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n-\t\ttotal_objects += p[cur_pack]->num_objects;\n+\t\ttotal_objects += info[cur_pack].p->num_objects;\n \n \t/*\n \t * As we de-duplicate by fanout value, we expect the fanout\n@@ -609,7 +584,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tnth_midxed_pack_midx_entry(m, perm,\n+\t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n \t\t\t\tnr_fanout++;\n@@ -620,12 +595,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\tuint32_t start = 0, end;\n \n \t\t\tif (cur_fanout)\n-\t\t\t\tstart = get_pack_fanout(p[cur_pack], cur_fanout - 1);\n-\t\t\tend = get_pack_fanout(p[cur_pack], cur_fanout);\n+\t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n+\t\t\tend = get_pack_fanout(info[cur_pack].p, cur_fanout);\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -654,7 +629,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n }\n \n static size_t write_midx_pack_names(struct hashfile *f,\n-\t\t\t\t char **pack_names,\n+\t\t\t\t struct pack_info *info,\n \t\t\t\t uint32_t num_packs)\n {\n \tuint32_t i;\n@@ -662,14 +637,14 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(pack_names[i]) + 1;\n+\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n \n-\t\tif (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)\n+\t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t pack_names[i - 1],\n-\t\t\t pack_names[i]);\n+\t\t\t info[i - 1].pack_name,\n+\t\t\t info[i].pack_name);\n \n-\t\thashwrite(f, pack_names[i], writelen);\n+\t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n \n@@ -740,6 +715,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,\n }\n \n static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,\n+\t\t\t\t\tuint32_t *perm,\n \t\t\t\t\tstruct pack_midx_entry *objects, uint32_t nr_objects)\n {\n \tstruct pack_midx_entry *list = objects;\n@@ -749,7 +725,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\thashwrite_be32(f, obj->pack_int_id);\n+\t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -822,20 +798,17 @@ int write_midx_file(const char *object_dir)\n \tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n-\tpacks.alloc_list = packs.m ? packs.m->num_packs : 16;\n-\tpacks.alloc_names = packs.alloc_list;\n-\tpacks.list = NULL;\n-\tpacks.names = NULL;\n-\tALLOC_ARRAY(packs.list, packs.alloc_list);\n-\tALLOC_ARRAY(packs.names, packs.alloc_names);\n+\tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n+\tpacks.info = NULL;\n+\tALLOC_ARRAY(packs.info, packs.alloc);\n \n \tif (packs.m) {\n \t\tfor (i = 0; i < packs.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);\n-\t\t\tALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);\n+\t\t\tALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);\n \n-\t\t\tpacks.list[packs.nr] = NULL;\n-\t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n+\t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n+\t\t\tpacks.info[packs.nr].p = NULL;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -845,10 +818,7 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tALLOC_ARRAY(pack_perm, packs.nr);\n-\tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n-\n-\tentries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);\n+\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -857,8 +827,21 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tQSORT(packs.info, packs.nr, pack_info_compare);\n+\n+\t/*\n+\t * pack_perm stores a permutation between pack-int-ids from the\n+\t * previous multi-pack-index to the new one we are writing:\n+\t *\n+\t * pack_perm[old_id] = new_id\n+\t */\n+\tALLOC_ARRAY(pack_perm, packs.nr);\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t}\n+\n \tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -929,7 +912,7 @@ int write_midx_file(const char *object_dir)\n \n \t\tswitch (chunk_ids[i]) {\n \t\t\tcase MIDX_CHUNKID_PACKNAMES:\n-\t\t\t\twritten += write_midx_pack_names(f, packs.names, packs.nr);\n+\t\t\t\twritten += write_midx_pack_names(f, packs.info, packs.nr);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OIDFANOUT:\n@@ -941,7 +924,7 @@ int write_midx_file(const char *object_dir)\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_OBJECTOFFSETS:\n-\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);\n+\t\t\t\twritten += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);\n \t\t\t\tbreak;\n \n \t\t\tcase MIDX_CHUNKID_LARGEOFFSETS:\n@@ -964,15 +947,14 @@ int write_midx_file(const char *object_dir)\n \n cleanup:\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tif (packs.list[i]) {\n-\t\t\tclose_pack(packs.list[i]);\n-\t\t\tfree(packs.list[i]);\n+\t\tif (packs.info[i].p) {\n+\t\t\tclose_pack(packs.info[i].p);\n+\t\t\tfree(packs.info[i].p);\n \t\t}\n-\t\tfree(packs.names[i]);\n+\t\tfree(packs.info[i].pack_name);\n \t}\n \n-\tfree(packs.list);\n-\tfree(packs.names);\n+\tfree(packs.info);\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-- \n2.22.0.rc0\n\n"},{"id":"375556","messageId":"20190514184754.3196-7-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 06/11] multi-pack-index: implement 'expire' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:49Z","receivedAt":"2019-05-14T18:48:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'git multi-pack-index expire' subcommand looks at the existing\nmult-pack-index, counts the number of objects referenced in each\npack-file, deletes the pack-fils with no referenced objects, and\nrewrites the multi-pack-index to no longer reference those packs.\n\nRefactor the write_midx_file() method to call write_midx_internal()\nwhich now takes an existing 'struct multi_pack_index' and a list\nof pack-files to drop (as specified by the names of their pack-\nindexes). As we write the new multi-pack-index, we drop those\nfile names from the list of known pack-files.\n\nThe expire_midx_packs() method removes the unreferenced pack-files\nafter carefully closing the packs to avoid open handles.\n\nTest that a new pack-file that covers the contents of two other\npack-files leads to those pack-files being deleted during the\nexpire subcommand. Be sure to read the multi-pack-index to ensure\nit no longer references those packs.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 119 +++++++++++++++++++++++++++++++++---\n t/t5319-multi-pack-index.sh | 20 ++++++\n 2 files changed, 129 insertions(+), 10 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 6d4b84e243..9b0b4c1520 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -34,6 +34,8 @@\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n+#define PACK_EXPIRED UINT_MAX\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n@@ -431,6 +433,7 @@ struct pack_info {\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n+\tunsigned expired : 1;\n };\n \n static int pack_info_compare(const void *_a, const void *_b)\n@@ -478,6 +481,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tpacks->info[packs->nr].pack_name = xstrdup(file_name);\n \t\tpacks->info[packs->nr].orig_pack_int_id = packs->nr;\n+\t\tpacks->info[packs->nr].expired = 0;\n \t\tpacks->nr++;\n \t}\n }\n@@ -637,13 +641,17 @@ static size_t write_midx_pack_names(struct hashfile *f,\n \tsize_t written = 0;\n \n \tfor (i = 0; i < num_packs; i++) {\n-\t\tsize_t writelen = strlen(info[i].pack_name) + 1;\n+\t\tsize_t writelen;\n+\n+\t\tif (info[i].expired)\n+\t\t\tcontinue;\n \n \t\tif (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n \t\t\t info[i - 1].pack_name,\n \t\t\t info[i].pack_name);\n \n+\t\twritelen = strlen(info[i].pack_name) + 1;\n \t\thashwrite(f, info[i].pack_name, writelen);\n \t\twritten += writelen;\n \t}\n@@ -725,6 +733,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n+\t\tif (perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n+\t\t\t oid_to_hex(&obj->oid),\n+\t\t\t obj->pack_int_id);\n+\n \t\thashwrite_be32(f, perm[obj->pack_int_id]);\n \n \t\tif (large_offset_needed && obj->offset >> 31)\n@@ -771,7 +784,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n-int write_midx_file(const char *object_dir)\n+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n+\t\t\t struct string_list *packs_to_drop)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -787,6 +801,8 @@ int write_midx_file(const char *object_dir)\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n \tint pack_name_concat_len = 0;\n+\tint dropped_packs = 0;\n+\tint result = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -795,7 +811,10 @@ int write_midx_file(const char *object_dir)\n \t\t\t midx_name);\n \t}\n \n-\tpacks.m = load_multi_pack_index(object_dir, 1);\n+\tif (m)\n+\t\tpacks.m = m;\n+\telse\n+\t\tpacks.m = load_multi_pack_index(object_dir, 1);\n \n \tpacks.nr = 0;\n \tpacks.alloc = packs.m ? packs.m->num_packs : 16;\n@@ -809,13 +828,14 @@ int write_midx_file(const char *object_dir)\n \t\t\tpacks.info[packs.nr].orig_pack_int_id = i;\n \t\t\tpacks.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);\n \t\t\tpacks.info[packs.nr].p = NULL;\n+\t\t\tpacks.info[packs.nr].expired = 0;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n \n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);\n \n-\tif (packs.m && packs.nr == packs.m->num_packs)\n+\tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n \tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n@@ -829,6 +849,34 @@ int write_midx_file(const char *object_dir)\n \n \tQSORT(packs.info, packs.nr, pack_info_compare);\n \n+\tif (packs_to_drop && packs_to_drop->nr) {\n+\t\tint drop_index = 0;\n+\t\tint missing_drops = 0;\n+\n+\t\tfor (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(packs.info[i].pack_name,\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\tif (!cmp) {\n+\t\t\t\tdrop_index++;\n+\t\t\t\tpacks.info[i].expired = 1;\n+\t\t\t} else if (cmp > 0) {\n+\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\tdrop_index++;\n+\t\t\t\tmissing_drops++;\n+\t\t\t\ti--;\n+\t\t\t} else {\n+\t\t\t\tpacks.info[i].expired = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (missing_drops) {\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \t/*\n \t * pack_perm stores a permutation between pack-int-ids from the\n \t * previous multi-pack-index to the new one we are writing:\n@@ -837,11 +885,18 @@ int write_midx_file(const char *object_dir)\n \t */\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tfor (i = 0; i < packs.nr; i++) {\n-\t\tpack_perm[packs.info[i].orig_pack_int_id] = i;\n+\t\tif (packs.info[i].expired) {\n+\t\t\tdropped_packs++;\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t} else {\n+\t\t\tpack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t}\n \t}\n \n-\tfor (i = 0; i < packs.nr; i++)\n-\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\tfor (i = 0; i < packs.nr; i++) {\n+\t\tif (!packs.info[i].expired)\n+\t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n+\t}\n \n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n@@ -857,7 +912,7 @@ int write_midx_file(const char *object_dir)\n \tcur_chunk = 0;\n \tnum_chunks = large_offsets_needed ? 5 : 4;\n \n-\twritten = write_midx_header(f, num_chunks, packs.nr);\n+\twritten = write_midx_header(f, num_chunks, packs.nr - dropped_packs);\n \n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;\n \tchunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;\n@@ -958,7 +1013,12 @@ int write_midx_file(const char *object_dir)\n \tfree(entries);\n \tfree(pack_perm);\n \tfree(midx_name);\n-\treturn 0;\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1125,5 +1185,44 @@ int verify_midx_file(struct repository *r, const char *object_dir)\n \n int expire_midx_packs(struct repository *r, const char *object_dir)\n {\n-\treturn 0;\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t}\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\treturn result;\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1b2d32f475..12570fe7ac 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -412,4 +412,24 @@ test_expect_success 'expire does not remove any packs' '\n \t)\n '\n \n+test_expect_success 'expire removes unreferenced packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tls .git/objects/pack | grep -v -e pack-[AB] >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tls .git/objects/pack/ | grep idx >expect-idx &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual-midx &&\n+\t\ttest_cmp expect-idx actual-midx &&\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375557","messageId":"20190514184754.3196-4-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 03/11] multi-pack-index: prepare for 'expire' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:46Z","receivedAt":"2019-05-14T18:48:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The multi-pack-index tracks objects in a collection of pack-files.\nOnly one copy of each object is indexed, using the modified time\nof the pack-files to determine tie-breakers. It is possible to\nhave a pack-file with no referenced objects because all objects\nhave a duplicate in a newer pack-file.\n\nIntroduce a new 'expire' subcommand to the multi-pack-index builtin.\nThis subcommand will delete these unused pack-files and rewrite the\nmulti-pack-index to no longer refer to those files. More details\nabout the specifics will follow as the method is implemented.\n\nAdd a test that verifies the 'expire' subcommand is correctly wired,\nbut will still be valid when the verb is implemented. Specifically,\ncreate a set of packs that should all have referenced objects and\nshould not be removed during an 'expire' operation. The packs are\ncreated carefully to ensure they have a specific order when sorted\nby size. This will be important in a later test.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 5 +++\n builtin/multi-pack-index.c | 4 ++-\n midx.c | 5 +++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 49 ++++++++++++++++++++++++++\n 5 files changed, 63 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 1af406aca2..6186c4c936 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -31,6 +31,11 @@ write::\n verify::\n \tVerify the contents of the MIDX file.\n \n+expire::\n+\tDelete the pack-files that are tracked \tby the MIDX file, but\n+\thave no objects referenced by the MIDX. Rewrite the MIDX file\n+\tafterward to remove all references to these pack-files.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 72dfd3dadc..ad10d40512 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -6,7 +6,7 @@\n #include \"trace2.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n \tNULL\n };\n \n@@ -47,6 +47,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(the_repository, opts.object_dir);\n+\tif (!strcmp(argv[0], \"expire\"))\n+\t\treturn expire_midx_packs(the_repository, opts.object_dir);\n \n \tdie(_(\"unrecognized verb: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex e7e1fe4d65..3b7da1a360 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1140,3 +1140,8 @@ int verify_midx_file(struct repository *r, const char *object_dir)\n \n \treturn verify_midx_error;\n }\n+\n+int expire_midx_packs(struct repository *r, const char *object_dir)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 3eb29731f2..505f1431b7 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -50,6 +50,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir);\n+int expire_midx_packs(struct repository *r, const char *object_dir);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 1ebf19ec3c..1b2d32f475 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -363,4 +363,53 @@ test_expect_success 'verify incorrect 64-bit offset' '\n \t\t\"incorrect object offset\"\n '\n \n+test_expect_success 'setup expire tests' '\n+\tmkdir dup &&\n+\t(\n+\t\tcd dup &&\n+\t\tgit init &&\n+\t\ttest-tool genrandom \"data\" 4096 >large_file.txt &&\n+\t\tgit update-index --add large_file.txt &&\n+\t\tfor i in $(test_seq 1 20)\n+\t\tdo\n+\t\t\ttest_commit $i\n+\t\tdone &&\n+\t\tgit branch A HEAD &&\n+\t\tgit branch B HEAD~8 &&\n+\t\tgit branch C HEAD~13 &&\n+\t\tgit branch D HEAD~16 &&\n+\t\tgit branch E HEAD~18 &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-A <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-B <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-C <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-D <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n+test_expect_success 'expire does not remove any packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375558","messageId":"20190514184754.3196-8-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 07/11] multi-pack-index: prepare 'repack' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:50Z","receivedAt":"2019-05-14T18:48:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In an environment where the multi-pack-index is useful, it is due\nto many pack-files and an inability to repack the object store\ninto a single pack-file. However, it is likely that many of these\npack-files are rather small, and could be repacked into a slightly\nlarger pack-file without too much effort. It may also be important\nto ensure the object store is highly available and the repack\noperation does not interrupt concurrent git commands.\n\nIntroduce a 'repack' subcommand to 'git multi-pack-index' that\ntakes a '--batch-size' option. The subcommand will inspect the\nmulti-pack-index for referenced pack-files whose size is smaller\nthan the batch size, until collecting a list of pack-files whose\nsizes sum to larger than the batch size. Then, a new pack-file\nwill be created containing the objects from those pack-files that\nare referenced by the multi-pack-index. The resulting pack is\nlikely to actually be smaller than the batch size due to\ncompression and the fact that there may be objects in the pack-\nfiles that have duplicate copies in other pack-files.\n\nThe current change introduces the command-line arguments, and we\nadd a test that ensures we parse these options properly. Since\nwe specify a small batch size, we will guarantee that future\nimplementations do not change the list of pack-files.\n\nIn addition, we hard-code the modified times of the packs in\nthe pack directory to ensure the list of packs sorted by modified\ntime matches the order if sorted by size (ascending). This will\nbe important in a future test.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-multi-pack-index.txt | 17 +++++++++++++++++\n builtin/multi-pack-index.c | 12 ++++++++++--\n midx.c | 5 +++++\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 20 +++++++++++++++++++-\n 5 files changed, 52 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 6186c4c936..233b2b7862 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -36,6 +36,23 @@ expire::\n \thave no objects referenced by the MIDX. Rewrite the MIDX file\n \tafterward to remove all references to these pack-files.\n \n+repack::\n+\tCreate a new pack-file containing objects in small pack-files\n+\treferenced by the multi-pack-index. If the size given by the\n+\t`--batch-size=` argument is zero, then create a pack\n+\tcontaining all objects referenced by the multi-pack-index. For\n+\ta non-zero batch size, Select the pack-files by examining packs\n+\tfrom oldest-to-newest, computing the \"expected size\" by counting\n+\tthe number of objects in the pack referenced by the\n+\tmulti-pack-index, then divide by the total number of objects in\n+\tthe pack and multiply by the pack size. We select packs with\n+\texpected size below the batch size until the set of packs have\n+\ttotal expected size at least the batch size. If the total size\n+\tdoes not reach the batch size, then do nothing. If a new pack-\n+\tfile is created, rewrite the multi-pack-index to reference the\n+\tnew pack-file. A later run of 'git multi-pack-index expire' will\n+\tdelete the pack-files that were part of this batch.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex ad10d40512..b1ea1a6aa1 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -6,12 +6,13 @@\n #include \"trace2.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire)\"),\n+\tN_(\"git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)\"),\n \tNULL\n };\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tunsigned long batch_size;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n@@ -20,6 +21,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n+\t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n@@ -43,6 +46,11 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \ttrace2_cmd_mode(argv[0]);\n \n+\tif (!strcmp(argv[0], \"repack\"))\n+\t\treturn midx_repack(the_repository, opts.object_dir, (size_t)opts.batch_size);\n+\tif (opts.batch_size)\n+\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n+\n \tif (!strcmp(argv[0], \"write\"))\n \t\treturn write_midx_file(opts.object_dir);\n \tif (!strcmp(argv[0], \"verify\"))\n@@ -50,5 +58,5 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!strcmp(argv[0], \"expire\"))\n \t\treturn expire_midx_packs(the_repository, opts.object_dir);\n \n-\tdie(_(\"unrecognized verb: %s\"), argv[0]);\n+\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\ndiff --git a/midx.c b/midx.c\nindex 9b0b4c1520..fbed8a8adb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1226,3 +1226,8 @@ int expire_midx_packs(struct repository *r, const char *object_dir)\n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n }\n+\n+int midx_repack(struct repository *r, const char *object_dir, size_t batch_size)\n+{\n+\treturn 0;\n+}\ndiff --git a/midx.h b/midx.h\nindex 505f1431b7..f0ae656b5d 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -51,6 +51,7 @@ int write_midx_file(const char *object_dir);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir);\n int expire_midx_packs(struct repository *r, const char *object_dir);\n+int midx_repack(struct repository *r, const char *object_dir, size_t batch_size);\n \n void close_midx(struct multi_pack_index *m);\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 12570fe7ac..133d5b7068 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -398,7 +398,8 @@ test_expect_success 'setup expire tests' '\n \t\tgit pack-objects --revs .git/objects/pack/pack-E <<-EOF &&\n \t\trefs/heads/E\n \t\tEOF\n-\t\tgit multi-pack-index write\n+\t\tgit multi-pack-index write &&\n+\t\tcp -r .git/objects/pack .git/objects/pack-backup\n \t)\n '\n \n@@ -432,4 +433,21 @@ test_expect_success 'expire removes unreferenced packs' '\n \t)\n '\n \n+test_expect_success 'repack with minimum size does not alter existing packs' '\n+\t(\n+\t\tcd dup &&\n+\t\trm -rf .git/objects/pack &&\n+\t\tmv .git/objects/pack-backup .git/objects/pack &&\n+\t\ttouch -m -t 201901010000 .git/objects/pack/pack-D* &&\n+\t\ttouch -m -t 201901010001 .git/objects/pack/pack-C* &&\n+\t\ttouch -m -t 201901010002 .git/objects/pack/pack-B* &&\n+\t\ttouch -m -t 201901010003 .git/objects/pack/pack-A* &&\n+\t\tls .git/objects/pack >expect &&\n+\t\tMINSIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 1) &&\n+\t\tgit multi-pack-index repack --batch-size=$MINSIZE &&\n+\t\tls .git/objects/pack >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375559","messageId":"20190514184754.3196-10-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 09/11] multi-pack-index: test expire while adding packs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:52Z","receivedAt":"2019-05-14T18:48:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"During development of the multi-pack-index expire subcommand, a\nversion went out that improperly computed the pack order if a new\npack was introduced while other packs were being removed. Part of\nthe subtlety of the bug involved the new pack being placed before\nother packs that already existed in the multi-pack-index.\n\nAdd a test to t5319-multi-pack-index.sh that catches this issue.\nThe test adds new packs that cause another pack to be expired, and\ncreates new packs that are lexicographically sorted before and\nafter the existing packs.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 6e47e5d0b2..8e04ce2821 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -478,4 +478,36 @@ test_expect_success 'expire removes repacked packs' '\n \t)\n '\n \n+test_expect_success 'expire works when adding new packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/A\n+\t\t^refs/heads/B\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/B\n+\t\t^refs/heads/C\n+\t\tEOF\n+\t\tgit pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&\n+\t\trefs/heads/C\n+\t\t^refs/heads/D\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/a-pack <<-EOF &&\n+\t\trefs/heads/D\n+\t\t^refs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tgit pack-objects --revs .git/objects/pack/z-pack <<-EOF &&\n+\t\trefs/heads/E\n+\t\tEOF\n+\t\tgit multi-pack-index expire &&\n+\t\tls .git/objects/pack/ | grep idx >expect &&\n+\t\ttest-tool read-midx .git/objects | grep idx >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375560","messageId":"20190514184754.3196-11-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 10/11] midx: add test that 'expire' respects .keep files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:53Z","receivedAt":"2019-05-14T18:48:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'git multi-pack-index expire' subcommand may delete packs that\nare not needed from the perspective of the multi-pack-index. If\na pack has a .keep file, then we should not delete that pack. Add\na test that ensures we preserve a pack that would otherwise be\nexpired. First, create a new pack that contains every object in\nthe repo, then add it to the multi-pack-index. Then create a .keep\nfile for a pack starting with \"a-pack\" that was added in the\nprevious test. Finally, expire and verify that the pack remains\nand the other packs were expired.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 18 ++++++++++++++++++\n 1 file changed, 18 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 8e04ce2821..c288901401 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -510,4 +510,22 @@ test_expect_success 'expire works when adding new packs' '\n \t)\n '\n \n+test_expect_success 'expire respects .keep files' '\n+\t(\n+\t\tcd dup &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack-all <<-EOF &&\n+\t\trefs/heads/A\n+\t\tEOF\n+\t\tgit multi-pack-index write &&\n+\t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n+\t\ttouch $PACKA.keep &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n+\t\ttest_line_count = 3 a-pack-files &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 2 midx-list\n+\t)\n+'\n+\n+\n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375561","messageId":"20190514184754.3196-12-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 11/11] t5319-multi-pack-index.sh: test batch size zero","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:54Z","receivedAt":"2019-05-14T18:48:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'git multi-pack-index repack' command can take a batch size of\nzero, which creates a new pack-file containing all objects in the\nmulti-pack-index. The first 'repack' command will create one new\npack-file, and an 'expire' command after that will delete the old\npack-files, as they no longer contain any referenced objects in the\nmulti-pack-index.\n\nWe must remove the .keep file that was added in the previous test\nin order to expire that pack-file.\n\nAlso test that a 'repack' will do nothing if there is only one\npack-file.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5319-multi-pack-index.sh | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex c288901401..79bfaeafa9 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -527,5 +527,24 @@ test_expect_success 'expire respects .keep files' '\n \t)\n '\n \n+test_expect_success 'repack --batch-size=0 repacks everything' '\n+\t(\n+\t\tcd dup &&\n+\t\trm .git/objects/pack/*.keep &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 2 idx-list &&\n+\t\tgit multi-pack-index repack --batch-size=0 &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 3 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 3 midx-list &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -al .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 1 idx-list &&\n+\t\tgit multi-pack-index repack --batch-size=0 &&\n+\t\tls -al .git/objects/pack/*idx >new-idx-list &&\n+\t\ttest_cmp idx-list new-idx-list\n+\t)\n+'\n \n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375562","messageId":"20190514184754.3196-9-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 08/11] midx: implement midx_repack()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:51Z","receivedAt":"2019-05-14T18:48:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"To repack with a non-zero batch-size, first sort all pack-files by\ntheir modified time. Second, walk those pack-files from oldest\nto newest, compute their expected size, and add the packs to a list\nif they are smaller than the given batch-size. Stop when the total\nexpected size is at least the batch size.\n\nIf the batch size is zero, select all packs in the multi-pack-index.\n\nFinally, collect the objects from the multi-pack-index that are in\nthe selected packs and send them to 'git pack-objects'. Write a new\nmulti-pack-index that includes the new pack.\n\nUsing a batch size of zero is very similar to a standard 'git repack'\ncommand, except that we do not delete the old packs and instead rely\non the new multi-pack-index to prevent new processes from reading the\nold packs. This does not disrupt other Git processes that are currently\nreading the old packs based on the old multi-pack-index.\n\nWhile first designing a 'git multi-pack-index repack' operation, I\nstarted by collecting the batches based on the actual size of the\nobjects instead of the size of the pack-files. This allows repacking\na large pack-file that has very few referencd objects. However, this\ncame at a significant cost of parsing pack-files instead of simply\nreading the multi-pack-index and getting the file information for\nthe pack-files. The \"expected size\" version provides similar\nbehavior, but could skip a pack-file if the average object size is\nmuch larger than the actual size of the referenced objects, or\ncan create a large pack if the actual size of the referenced objects\nis larger than the expected size.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 151 +++++++++++++++++++++++++++++++++++-\n t/t5319-multi-pack-index.sh | 28 +++++++\n 2 files changed, 178 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex fbed8a8adb..d649644420 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -9,6 +9,7 @@\n #include \"midx.h\"\n #include \"progress.h\"\n #include \"trace2.h\"\n+#include \"run-command.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -1227,7 +1228,155 @@ int expire_midx_packs(struct repository *r, const char *object_dir)\n \treturn result;\n }\n \n-int midx_repack(struct repository *r, const char *object_dir, size_t batch_size)\n+struct repack_info {\n+\ttimestamp_t mtime;\n+\tuint32_t referenced_objects;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n {\n+\tconst struct repack_info *a, *b;\n+\n+\ta = (const struct repack_info *)a_;\n+\tb = (const struct repack_info *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fill_included_packs_all(struct multi_pack_index *m,\n+\t\t\t\t unsigned char *include_pack)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < m->num_packs; i++)\n+\t\tinclude_pack[i] = 1;\n+\n+\treturn m->num_packs < 2;\n+}\n+\n+static int fill_included_packs_batch(struct repository *r,\n+\t\t\t\t struct multi_pack_index *m,\n+\t\t\t\t unsigned char *include_pack,\n+\t\t\t\t size_t batch_size)\n+{\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct repack_info *pack_info = xcalloc(m->num_packs, sizeof(struct repack_info));\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_info[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_info[i].mtime = m->packs[i]->mtime;\n+\t}\n+\n+\tfor (i = 0; batch_size && i < m->num_objects; i++) {\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tpack_info[pack_int_id].referenced_objects++;\n+\t}\n+\n+\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_info[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\t\tsize_t expected_size;\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\t\tcontinue;\n+\n+\t\texpected_size = (size_t)(p->pack_size\n+\t\t\t\t\t * pack_info[i].referenced_objects);\n+\t\texpected_size /= p->num_objects;\n+\n+\t\tif (expected_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += expected_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tfree(pack_info);\n+\n+\tif (total_size < batch_size || packs_to_repack < 2)\n+\t\treturn 1;\n+\n \treturn 0;\n }\n+\n+int midx_repack(struct repository *r, const char *object_dir, size_t batch_size)\n+{\n+\tint result = 0;\n+\tuint32_t i;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = load_multi_pack_index(object_dir, 1);\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tinclude_pack = xcalloc(m->num_packs, sizeof(unsigned char));\n+\n+\tif (batch_size) {\n+\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n+\t\t\tgoto cleanup;\n+\t} else if (fill_included_packs_all(m, include_pack))\n+\t\tgoto cleanup;\n+\n+\targv_array_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\targv_array_push(&cmd.args, base_name.buf);\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\txwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);\n+\t\txwrite(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, m, NULL);\n+\tm = NULL;\n+\n+cleanup:\n+\tif (m)\n+\t\tclose_midx(m);\n+\tfree(include_pack);\n+\treturn result;\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 133d5b7068..6e47e5d0b2 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -450,4 +450,32 @@ test_expect_success 'repack with minimum size does not alter existing packs' '\n \t)\n '\n \n+test_expect_success 'repack creates a new pack' '\n+\t(\n+\t\tcd dup &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 5 idx-list &&\n+\t\tTHIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk \"{print \\$5;}\" | sort -n | head -n 3 | tail -n 1) &&\n+\t\tBATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) &&\n+\t\tgit multi-pack-index repack --batch-size=$BATCH_SIZE &&\n+\t\tls .git/objects/pack/*idx >idx-list &&\n+\t\ttest_line_count = 6 idx-list &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 6 midx-list\n+\t)\n+'\n+\n+test_expect_success 'expire removes repacked packs' '\n+\t(\n+\t\tcd dup &&\n+\t\tls -al .git/objects/pack/*pack &&\n+\t\tls -S .git/objects/pack/*pack | head -n 4 >expect &&\n+\t\tgit multi-pack-index expire &&\n+\t\tls -S .git/objects/pack/*pack >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n+\t\ttest_line_count = 4 midx-list\n+\t)\n+'\n+\n test_done\n-- \n2.22.0.rc0\n\n"},{"id":"375563","messageId":"20190514184754.3196-5-dstolee@microsoft.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"[PATCH v6 04/11] midx: simplify computation of pack name lengths","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-14T18:47:47Z","receivedAt":"2019-05-14T18:48:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before writing the multi-pack-index, we compute the length of the\npack-index names concatenated together. This forms the data in the\npack name chunk, and we precompute it to compute chunk offsets.\nThe value is also modified to fit alignment needs.\n\nPreviously, this computation was coupled with adding packs from\nthe existing multi-pack-index and the remaining packs in the object\ndir not already covered by the multi-pack-index.\n\nIn anticipation of this becoming more complicated with the 'expire'\nsubcommand, simplify the computation by centralizing it to a single\nloop before writing the file.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3b7da1a360..62404620ad 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -433,7 +433,6 @@ struct pack_list {\n \tuint32_t nr;\n \tuint32_t alloc_list;\n \tuint32_t alloc_names;\n-\tsize_t pack_name_concat_len;\n \tstruct multi_pack_index *m;\n };\n \n@@ -468,7 +467,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t}\n \n \t\tpacks->names[packs->nr] = xstrdup(file_name);\n-\t\tpacks->pack_name_concat_len += strlen(file_name) + 1;\n \t\tpacks->nr++;\n \t}\n }\n@@ -812,6 +810,7 @@ int write_midx_file(const char *object_dir)\n \tuint32_t nr_entries, num_large_offsets = 0;\n \tstruct pack_midx_entry *entries = NULL;\n \tint large_offsets_needed = 0;\n+\tint pack_name_concat_len = 0;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name)) {\n@@ -827,7 +826,6 @@ int write_midx_file(const char *object_dir)\n \tpacks.alloc_names = packs.alloc_list;\n \tpacks.list = NULL;\n \tpacks.names = NULL;\n-\tpacks.pack_name_concat_len = 0;\n \tALLOC_ARRAY(packs.list, packs.alloc_list);\n \tALLOC_ARRAY(packs.names, packs.alloc_names);\n \n@@ -838,7 +836,6 @@ int write_midx_file(const char *object_dir)\n \n \t\t\tpacks.list[packs.nr] = NULL;\n \t\t\tpacks.names[packs.nr] = xstrdup(packs.m->pack_names[i]);\n-\t\t\tpacks.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;\n \t\t\tpacks.nr++;\n \t\t}\n \t}\n@@ -848,10 +845,6 @@ int write_midx_file(const char *object_dir)\n \tif (packs.m && packs.nr == packs.m->num_packs)\n \t\tgoto cleanup;\n \n-\tif (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n-\t\tpacks.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n-\n \tALLOC_ARRAY(pack_perm, packs.nr);\n \tsort_packs_by_name(packs.names, packs.nr, pack_perm);\n \n@@ -864,6 +857,13 @@ int write_midx_file(const char *object_dir)\n \t\t\tlarge_offsets_needed = 1;\n \t}\n \n+\tfor (i = 0; i < packs.nr; i++)\n+\t\tpack_name_concat_len += strlen(packs.names[i]) + 1;\n+\n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \tFREE_AND_NULL(midx_name);\n@@ -881,7 +881,7 @@ int write_midx_file(const char *object_dir)\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;\n-\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;\n+\tchunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;\n \n \tcur_chunk++;\n \tchunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;\n-- \n2.22.0.rc0\n\n"},{"id":"376928","messageId":"5fadd5ee-0993-ad94-2cda-f8dfe3795336@gmail.com","threadId":"49993","inReplyTo":"20190514184754.3196-1-dstolee@microsoft.com","subject":"Re: [PATCH v6 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-10T14:15:44Z","receivedAt":"2019-06-10T14:15:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/14/2019 2:47 PM, Derrick Stolee wrote:\n> Updates in V6:\n> \n> I rebased onto ds/midx-too-many-packs. Thanks, Junio for taking that\n> change first. There were several subtle things that needed to change to\n> put this change on top:\n> \n> * We need a repository struct everywhere since we add pack-files to the\n> packed_git list now.\n> \n> * A FREE_AND_NULL() was dropped after closing a pack because the pack\n> is still in the packed_git list after opening.\n> \n> * I noticed some whitespace problems.\n> \n> I also expect GMail to munge my added \"From:\" tags, so it will look\n> like the author is \"stolee@gmail.com\" instead of\n> \"dstolee@microsoft.com\". Sorry for the continued inconvenience here.\n\nJunio: thank you for taking ds/midx-too-many-packs into v2.22.0. That\nwas a helpful bugfix.\n\nHowever, this series was dropped from the cooking emails, and never\nincluded this v6. Now that the release is complete, could this be\nreconsidered?\n\nThanks,\n-Stolee\n"},{"id":"376953","messageId":"xmqqy329mjns.fsf@gitster-ct.c.googlers.com","threadId":"49993","inReplyTo":"5fadd5ee-0993-ad94-2cda-f8dfe3795336@gmail.com","subject":"Re: [PATCH v6 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-10T17:31:35Z","receivedAt":"2019-06-10T17:31:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 5/14/2019 2:47 PM, Derrick Stolee wrote:\n>> Updates in V6:\n>> \n>> I rebased onto ds/midx-too-many-packs. Thanks, Junio for taking that\n>> change first. There were several subtle things that needed to change to\n>> put this change on top:\n>> ...\n> However, this series was dropped from the cooking emails, and never\n> included this v6. Now that the release is complete, could this be\n> reconsidered?\n\n\"reconsider\" is a bit strong word, as (at least as far as I recall)\nit was never \"rejected\" as an unwanted topic, but was merely\npostponed to give way to other topics in flight. Thanks for keeping\nan eye on it and finding the right moment to raising it again.\n\nI could go back to the list archive and dig it up, but because it\nhas been a while since it was posted, it may not be a bad idea to\nsend it for a review, after making sure it cleanly applies to\n'master', to make it one of the early topics to go 'next' during\nthis cycle, I would think.\n\nThanks.\n"},{"id":"376959","messageId":"39b0c277-0ab4-971e-bf10-3f232f0df8e4@gmail.com","threadId":"49993","inReplyTo":"xmqqy329mjns.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v6 00/11] Create 'expire' and 'repack' verbs for git-multi-pack-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-10T17:57:56Z","receivedAt":"2019-06-10T17:58:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/10/2019 1:31 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 5/14/2019 2:47 PM, Derrick Stolee wrote:\n>>> Updates in V6:\n>>>\n>>> I rebased onto ds/midx-too-many-packs. Thanks, Junio for taking that\n>>> change first. There were several subtle things that needed to change to\n>>> put this change on top:\n>>> ...\n>> However, this series was dropped from the cooking emails, and never\n>> included this v6. Now that the release is complete, could this be\n>> reconsidered?\n> \n> \"reconsider\" is a bit strong word, as (at least as far as I recall)\n> it was never \"rejected\" as an unwanted topic, but was merely\n> postponed to give way to other topics in flight. Thanks for keeping\n> an eye on it and finding the right moment to raising it again.\n> \n> I could go back to the list archive and dig it up, but because it\n> has been a while since it was posted, it may not be a bad idea to\n> send it for a review, after making sure it cleanly applies to\n> 'master', to make it one of the early topics to go 'next' during\n> this cycle, I would think.\n\nSure, I'll create a brand new thread and point to this thread for\nhistory.\n\nThanks,\n-Stolee\n"}]}