{"thread":{"id":"50055","subject":"[PATCH 1/2] pack-redundant: new algorithm to find min packs","startedAt":"2018-12-18T09:58:40Z","lastAt":"2019-02-02T13:31:36Z","messageCount":83,"participants":["Jiang Xin","16657101987@163.com","Junio C Hamano","SZEDER Gábor","Johannes Sixt","Torsten Bögershausen","tboegi@web.de","Eric Sunshine","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"365522","messageId":"20181218095829.20092-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":null,"subject":"[PATCH 1/2] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2018-12-18T09:58:28Z","receivedAt":"2018-12-18T09:58:40Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 116 ++++++++++++++++++++++++---------------\n 1 file changed, 73 insertions(+), 43 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..19dcf74750 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -421,14 +421,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n-static void minimize(struct pack_list **min)\n+static int cmp_pack_list_reverse(const void *a, const void *b)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n+static void minimize(struct pack_list **min, struct llist *ignore)\n+{\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -446,49 +484,41 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove all the ignored objects and unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile ((pl = non_unique)) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\twhile ((pl = pl->next)) {\n+\t\t\tif (pl->all_objects->size == 0)\n+\t\t\t\tbreak;\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\t\t}\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -603,7 +633,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\n@@ -667,7 +697,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \t\tpl = pl->next;\n \t}\n \n-\tminimize(&min);\n+\tminimize(&min, ignore);\n \n \tif (verbose) {\n \t\tfprintf(stderr, \"There are %lu packs available in alt-odbs.\\n\",\n-- \n2.20.0.2.g660e9286fc\n\n"},{"id":"365523","messageId":"20181218095829.20092-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181218095829.20092-1-worldhello.net@gmail.com","subject":"[PATCH 2/2] pack-redundant: remove unused functions","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2018-12-18T09:58:29Z","receivedAt":"2018-12-18T09:58:43Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 81 ----------------------------------------\n 1 file changed, 81 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 19dcf74750..d0ff2377f3 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -63,15 +63,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -285,78 +276,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.20.0.2.g660e9286fc\n\n"},{"id":"365569","messageId":"20181219121451.21697-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181218095829.20092-2-worldhello.net@gmail.com","subject":"[PATCH v2 0/3] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2018-12-19T12:14:48Z","receivedAt":"2018-12-19T12:15:03Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Sun Chao is my former colleague at Huawei. He finds a bug of git-pack-redundant.\n\nWhen I was in Huawei, I develop a program to manage fork tree of repositories,\nusing alternate repo for forks to save disk spaces. \n\nSun Chao finds if there are too many packs and many of them overlap each\nother, running `git pack-redundant --all` will exhaust all memories and the\nprocess will be killed by kernel.\n\nThere is a script in commit log of commit 2/3, which can be used to create a\nrepository with lots of redundant packs. Running `git pack-redundant\n--all` in it can reproduce this issue.\n\nUpdates of reroll v2:\n\n* Add test cases in t5322.\n* Fix a bug in patch 2/3.\n\n--\n\nJiang Xin (1):\n t5322: test cases for git-pack-redundant\n\nSun Chao (2):\n pack-redundant: new algorithm to find min packs\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 181 ++++++++++++++------------------------\n t/t5322-pack-redundant.sh | 69 +++++++++++++++\n 2 files changed, 137 insertions(+), 113 deletions(-)\n create mode 100755 t/t5322-pack-redundant.sh\n\n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"365570","messageId":"20181219121451.21697-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181218095829.20092-2-worldhello.net@gmail.com","subject":"[PATCH v2 1/3] t5322: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2018-12-19T12:14:49Z","receivedAt":"2018-12-19T12:15:06Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \n---\n t/t5322-pack-redundant.sh | 69 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 69 insertions(+)\n create mode 100755 t/t5322-pack-redundant.sh\n\ndiff --git a/t/t5322-pack-redundant.sh b/t/t5322-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..4add9c2bb1\n--- /dev/null\n+++ b/t/t5322-pack-redundant.sh\n@@ -0,0 +1,69 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='git redundant test'\n+\n+. ./test-lib.sh\n+\n+create_commits()\n+{\n+\tset -e\n+\tparent=\n+\tfor name in A B C D E F G H I J K L M\n+\tdo\n+\t\ttest_tick\n+\t\tT=$(git write-tree)\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\tsha1=$(echo $name | git commit-tree $T)\n+\t\telse\n+\t\t\tsha1=$(echo $name | git commit-tree -p $parent $T)\n+\t\tfi\n+\t\teval $name=$sha1\n+\t\tparent=$sha1\n+\tdone\n+\tgit update-ref refs/heads/master $M\n+}\n+\n+create_redundant_packs()\n+{\n+\tset -e\n+\tcd .git/objects/pack\n+\tP1=$(printf \"$T\\n$A\\n\" | git pack-objects pack 2>/dev/null)\n+\tP2=$(printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n\" | git pack-objects pack 2>/dev/null)\n+\tP3=$(printf \"$C\\n$D\\n$F\\n$G\\n$I\\n$J\\n\" | git pack-objects pack 2>/dev/null)\n+\tP4=$(printf \"$D\\n$E\\n$G\\n$H\\n$J\\n$K\\n\" | git pack-objects pack 2>/dev/null)\n+\tP5=$(printf \"$F\\n$G\\n$H\\n\" | git pack-objects pack 2>/dev/null)\n+\tP6=$(printf \"$F\\n$I\\n$L\\n\" | git pack-objects pack 2>/dev/null)\n+\tP7=$(printf \"$H\\n$K\\n$M\\n\" | git pack-objects pack 2>/dev/null)\n+\tP8=$(printf \"$L\\n$M\\n\" | git pack-objects pack 2>/dev/null)\n+\tcd -\n+}\n+\n+# Create commits and packs\n+create_commits\n+create_redundant_packs\n+\n+test_expect_success 'clear loose objects' '\n+\tgit prune-packed &&\n+\ttest $(find .git/objects -type f | grep -v pack | wc -l) -eq 0\n+'\n+\n+printf \"$P1\\n$P4\\n$P5\\n$P6\\n\" | sort >expected\n+\n+test_expect_success 'git pack-redundant --all' '\n+\tgit pack-redundant --all | \\\n+\t\tsed -e \"s#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g\" | \\\n+\t\tsort -u >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'remove redundant packs' '\n+\tgit pack-redundant --all | xargs rm &&\n+\tgit fsck &&\n+\ttest $(git pack-redundant --all | wc -l) -eq 0\n+'\n+\n+test_done\n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"365571","messageId":"20181219121451.21697-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181218095829.20092-2-worldhello.net@gmail.com","subject":"[PATCH v2 2/3] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2018-12-19T12:14:50Z","receivedAt":"2018-12-19T12:15:10Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 109 ++++++++++++++++++++++++---------------\n 1 file changed, 68 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..3655cc7dc6 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -421,14 +421,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -446,49 +484,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -603,7 +629,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\n@@ -664,6 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tpl = local_packs;\n \twhile (pl) {\n \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"365572","messageId":"20181219121451.21697-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181218095829.20092-2-worldhello.net@gmail.com","subject":"[PATCH v2 3/3] pack-redundant: remove unused functions","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2018-12-19T12:14:51Z","receivedAt":"2018-12-19T12:15:14Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 72 ----------------------------------------\n 1 file changed, 72 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 3655cc7dc6..9630117c90 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -285,78 +285,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"366026","messageId":"20190102043456.15652-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181219121451.21697-1-worldhello.net@gmail.com","subject":"[PATCH v3 0/3] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-02T04:34:53Z","receivedAt":"2019-01-02T04:37:32Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Sun Chao (my former colleague at Huawei) found a bug of\ngit-pack-redundant. If there are too many packs and many of them overlap\neach other, running `git pack-redundant --all` will exhaust all memories\nand the process will be killed by kernel.\n\nThere is a script in commit log of commit 2/3, which can be used to\ncreate a repository with lots of redundant packs. Running `git\npack-redundant --all` in it can reproduce this issue.\n\nUpdates of reroll v3:\n\n* Rename test case file from t5322 to t5323, for I see t5322 exist in\n commit 404dead121: \"pack-objects: add --sparse option\".\n\nJiang Xin (1):\n t5323: test cases for git-pack-redundant\n\nSun Chao (2):\n pack-redundant: new algorithm to find min packs\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 181 +++++++++++++++++-----------------------------\n t/t5323-pack-redundant.sh | 84 +++++++++++++++++++++\n 2 files changed, 152 insertions(+), 113 deletions(-)\n create mode 100755 t/t5323-pack-redundant.sh\n\n-- \n2.14.5.agit.2\n\n"},{"id":"366027","messageId":"20190102043456.15652-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181219121451.21697-1-worldhello.net@gmail.com","subject":"[PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-02T04:34:54Z","receivedAt":"2019-01-02T04:37:38Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \n---\n t/t5323-pack-redundant.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n create mode 100755 t/t5323-pack-redundant.sh\n\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..ef6076f065\n--- /dev/null\n+++ b/t/t5323-pack-redundant.sh\n@@ -0,0 +1,84 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='git redundant test'\n+\n+. ./test-lib.sh\n+\n+create_commits()\n+{\n+\tset -e\n+\tparent=\n+\tfor name in A B C D E F G H I J K L M\n+\tdo\n+\t\ttest_tick\n+\t\tT=$(git write-tree)\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\tsha1=$(echo $name | git commit-tree $T)\n+\t\telse\n+\t\t\tsha1=$(echo $name | git commit-tree -p $parent $T)\n+\t\tfi\n+\t\teval $name=$sha1\n+\t\tparent=$sha1\n+\tdone\n+\tgit update-ref refs/heads/master $M\n+}\n+\n+create_redundant_packs()\n+{\n+\tset -e\n+\tcd .git/objects/pack\n+\tP1=$(printf \"$T\\n$A\\n\" | git pack-objects pack 2>/dev/null)\n+\tP2=$(printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n\" | git pack-objects pack 2>/dev/null)\n+\tP3=$(printf \"$C\\n$D\\n$F\\n$G\\n$I\\n$J\\n\" | git pack-objects pack 2>/dev/null)\n+\tP4=$(printf \"$D\\n$E\\n$G\\n$H\\n$J\\n$K\\n\" | git pack-objects pack 2>/dev/null)\n+\tP5=$(printf \"$F\\n$G\\n$H\\n\" | git pack-objects pack 2>/dev/null)\n+\tP6=$(printf \"$F\\n$I\\n$L\\n\" | git pack-objects pack 2>/dev/null)\n+\tP7=$(printf \"$H\\n$K\\n$M\\n\" | git pack-objects pack 2>/dev/null)\n+\tP8=$(printf \"$L\\n$M\\n\" | git pack-objects pack 2>/dev/null)\n+\tcd -\n+\teval P$P1=P1:$P1\n+\teval P$P2=P2:$P2\n+\teval P$P3=P3:$P3\n+\teval P$P4=P4:$P4\n+\teval P$P5=P5:$P5\n+\teval P$P6=P6:$P6\n+\teval P$P7=P7:$P7\n+\teval P$P8=P8:$P8\n+}\n+\n+# Create commits and packs\n+create_commits\n+create_redundant_packs\n+\n+test_expect_success 'clear loose objects' '\n+\tgit prune-packed &&\n+\ttest $(find .git/objects -type f | grep -v pack | wc -l) -eq 0\n+'\n+\n+cat >expected < actual && \\\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'remove redundant packs' '\n+\tgit pack-redundant --all | xargs rm &&\n+\tgit fsck &&\n+\ttest $(git pack-redundant --all | wc -l) -eq 0\n+'\n+\n+test_done\n-- \n2.14.5.agit.2\n\n"},{"id":"366028","messageId":"20190102043456.15652-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181219121451.21697-1-worldhello.net@gmail.com","subject":"[PATCH v3 2/3] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-02T04:34:55Z","receivedAt":"2019-01-02T04:37:38Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 109 +++++++++++++++++++++++++++++------------------\n 1 file changed, 68 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..3655cc7dc6 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -421,14 +421,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -446,49 +484,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -603,7 +629,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\n@@ -664,6 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tpl = local_packs;\n \twhile (pl) {\n \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.14.5.agit.2\n\n"},{"id":"366029","messageId":"20190102043456.15652-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20181219121451.21697-1-worldhello.net@gmail.com","subject":"[PATCH v3 3/3] pack-redundant: remove unused functions","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-02T04:34:56Z","receivedAt":"2019-01-02T04:37:42Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 72 ------------------------------------------------\n 1 file changed, 72 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 3655cc7dc6..9630117c90 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -285,78 +285,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.14.5.agit.2\n\n"},{"id":"366338","messageId":"20190108164333.73425-1-16657101987@163.com","threadId":"50055","inReplyTo":"20190102043456.15652-4-worldhello.net@gmail.com","subject":"[PATCH v4 1/1] pack-redundant: remove unused functions","fromName":"","fromEmail":"16657101987@163.com","sentAt":"2019-01-08T16:43:33Z","receivedAt":"2019-01-08T16:43:53Z","isPatch":true,"sender":{"key":"16657101987@163.com","avatar":"https://avatars.githubusercontent.com/u/192864724?v=4"},"body":"From: Sun Chao \n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 86 ------------------------------------------------\n 1 file changed, 86 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 3655cc7..eac2350 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -35,11 +35,6 @@ static struct pack_list {\n \tstruct llist *all_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n-struct pll {\n-\tstruct pll *next;\n-\tstruct pack_list *pl;\n-};\n-\n static struct llist_item *free_nodes;\n \n static inline void llist_item_put(struct llist_item *item)\n@@ -63,15 +58,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -285,78 +271,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.8.1\n\n\n"},{"id":"366341","messageId":"20190108164100.73219-1-16657101987@163.com","threadId":"50055","inReplyTo":"20190102043456.15652-4-worldhello.net@gmail.com","subject":"[PATCH v4 0/1] pack-redundant: remove unused functions","fromName":"","fromEmail":"16657101987@163.com","sentAt":"2019-01-08T16:40:59Z","receivedAt":"2019-01-08T16:56:55Z","isPatch":true,"sender":{"key":"16657101987@163.com","avatar":"https://avatars.githubusercontent.com/u/192864724?v=4"},"body":"From: Sun Chao \n\nI'm particularly grateful to Junio and JiangXin for fixing the patches,\nand I noticed Junio send a new commit to remove more unused codes and\nsuggest to SQUASH it.\n\nSo I create this new version of patches to do this work, I also have\nchecked the left codes and remove a unused struct based on Junio's\nlast commit of `https://github.com/gitster/git/commits/sc/pack-redundant`.\n\n--\n\nSun Chao (1):\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 86 ------------------------------------------------\n 1 file changed, 86 deletions(-)\n\n-- \n2.8.1\n\n\n"},{"id":"366342","messageId":"20190108164525.73592-1-16657101987@163.com","threadId":"50055","inReplyTo":"20190102043456.15652-4-worldhello.net@gmail.com","subject":"[PATCH v4 0/1] pack-redundant: remove unused functions","fromName":"","fromEmail":"16657101987@163.com","sentAt":"2019-01-08T16:45:25Z","receivedAt":"2019-01-08T17:00:59Z","isPatch":true,"sender":{"key":"16657101987@163.com","avatar":"https://avatars.githubusercontent.com/u/192864724?v=4"},"body":"From: Sun Chao \n\nI'm particularly grateful to Junio and JiangXin for fixing the patches,\nand I noticed Junio send a new commit to remove more unused codes and\nsuggest to SQUASH it.\n\nSo I create this new version of patches to do this work, I also have\nchecked the left codes and remove a unused struct based on Junio's\nlast commit of `https://github.com/gitster/git/commits/sc/pack-redundant`.\n\n--\n\nSun Chao (1):\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 86 ------------------------------------------------\n 1 file changed, 86 deletions(-)\n\n-- \n2.8.1\n\n\n"},{"id":"366361","messageId":"xmqqzhsb3q1c.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190108164100.73219-1-16657101987@163.com","subject":"Re: [PATCH v4 0/1] pack-redundant: remove unused functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-08T19:30:55Z","receivedAt":"2019-01-08T19:53:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"16657101987@163.com writes:\n\n> From: Sun Chao \n>\n> I'm particularly grateful to Junio and JiangXin for fixing the patches,\n> and I noticed Junio send a new commit to remove more unused codes and\n> suggest to SQUASH it.\n>\n> So I create this new version of patches to do this work, I also have\n> checked the left codes and remove a unused struct based on Junio's\n> last commit of `https://github.com/gitster/git/commits/sc/pack-redundant`.\n>\n> --\n>\n> Sun Chao (1):\n> pack-redundant: remove unused functions\n\nIs this meant to replace [v3 3/3]?\n\n"},{"id":"366389","messageId":"20190109002911.78405-1-16657101987@163.com","threadId":"50055","inReplyTo":"xmqqzhsb3q1c.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 0/1] pack-redundant: remove unused functions","fromName":"","fromEmail":"16657101987@163.com","sentAt":"2019-01-09T00:29:11Z","receivedAt":"2019-01-09T00:29:57Z","isPatch":true,"sender":{"key":"16657101987@163.com","avatar":"https://avatars.githubusercontent.com/u/192864724?v=4"},"body":"> 16657101987@163.com writes:\n> \n>> From: Sun Chao \n>>\n>> I'm particularly grateful to Junio and JiangXin for fixing the patches,\n>> and I noticed Junio send a new commit to remove more unused codes and\n>> suggest to SQUASH it.\n>>\n>> So I create this new version of patches to do this work, I also have\n>> checked the left codes and remove a unused struct based on Junio's\n>> last commit of `https://github.com/gitster/git/commits/sc/pack-redundant`.\n>>\n>> --\n>>\n>> Sun Chao (1):\n>> pack-redundant: remove unused functions\n> \n> Is this meant to replace [v3 3/3]?\n\nI'm Sun Chao and because my huawei email account can't send\nemail outside from company, so I used 163 email account to\nsend new path at home. I'm sorry for not explaining that.\n\nYes, this is meant to replace [v3 3/3], and I have noticed the\npatch is applied, Thanks very much.\n\n"},{"id":"366411","messageId":"20190109125628.GG4673@szeder.dev","threadId":"50055","inReplyTo":"20190102043456.15652-2-worldhello.net@gmail.com","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-09T12:56:28Z","receivedAt":"2019-01-09T12:56:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 02, 2019 at 12:34:54PM +0800, Jiang Xin wrote:\n> From: Jiang Xin \n> \n> Add test cases for git pack-redundant to validate new algorithm for git\n> pack-redundant.\n> \n> Signed-off-by: Jiang Xin \n> ---\n> t/t5323-pack-redundant.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 84 insertions(+)\n> create mode 100755 t/t5323-pack-redundant.sh\n> \n> diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n> new file mode 100755\n> index 0000000000..ef6076f065\n> --- /dev/null\n> +++ b/t/t5323-pack-redundant.sh\n> @@ -0,0 +1,84 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2018 Jiang Xin\n> +#\n> +\n> +test_description='git redundant test'\n\ns/redundant/pack-redundant/ ?\n\n> +\n> +. ./test-lib.sh\n> +\n> +create_commits()\n> +{\n> +\tset -e\n> +\tparent=\n> +\tfor name in A B C D E F G H I J K L M\n> +\tdo\n> +\t\ttest_tick\n> +\t\tT=$(git write-tree)\n> +\t\tif test -z \"$parent\"\n> +\t\tthen\n> +\t\t\tsha1=$(echo $name | git commit-tree $T)\n\nThere is a considerable effort going on to switch from SHA-1 to a\ndifferent hash function, so please don't add any new $sha1 variable;\ncall it $oid or $commit instead.\n\n> +\t\telse\n> +\t\t\tsha1=$(echo $name | git commit-tree -p $parent $T)\n> +\t\tfi\n> +\t\teval $name=$sha1\n> +\t\tparent=$sha1\n> +\tdone\n> +\tgit update-ref refs/heads/master $M\n> +}\n> +\n> +create_redundant_packs()\n> +{\n> +\tset -e\n> +\tcd .git/objects/pack\n> +\tP1=$(printf \"$T\\n$A\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP2=$(printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP3=$(printf \"$C\\n$D\\n$F\\n$G\\n$I\\n$J\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP4=$(printf \"$D\\n$E\\n$G\\n$H\\n$J\\n$K\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP5=$(printf \"$F\\n$G\\n$H\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP6=$(printf \"$F\\n$I\\n$L\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP7=$(printf \"$H\\n$K\\n$M\\n\" | git pack-objects pack 2>/dev/null)\n> +\tP8=$(printf \"$L\\n$M\\n\" | git pack-objects pack 2>/dev/null)\n> +\tcd -\n> +\teval P$P1=P1:$P1\n> +\teval P$P2=P2:$P2\n> +\teval P$P3=P3:$P3\n> +\teval P$P4=P4:$P4\n> +\teval P$P5=P5:$P5\n> +\teval P$P6=P6:$P6\n> +\teval P$P7=P7:$P7\n> +\teval P$P8=P8:$P8\n> +}\n> +\n> +# Create commits and packs\n> +create_commits\n> +create_redundant_packs\n\nPlease perform all setup tasks in a test_expect_success block, so we\nget verbose and trace output about what's going on.\n\nDon't use 'set -e', use an &&-chain instead. To fail the test if a\ncommand in the for loop were to fail you could do something like this:\n\n for ....\n do\n do-this &&\n do-that ||\n return 1\n done\n\n> +\n> +test_expect_success 'clear loose objects' '\n> +\tgit prune-packed &&\n> +\ttest $(find .git/objects -type f | grep -v pack | wc -l) -eq 0\n\nUse something like\n\n find .git/objects -type f | grep -v pack >out &&\n test_must_be_empty out\n\ninstead, so we get an informative error message on failure.\n\n> +'\n> +\n> +cat >expected < +P1:$P1\n> +P4:$P4\n> +P5:$P5\n> +P6:$P6\n> +EOF\n> +\n> +test_expect_success 'git pack-redundant --all' '\n> +\tgit pack-redundant --all | \\\n\nDon't run a git command (especially the particular command the test\nscript focuses on) upstream of a pipe, because it hides the command's\nexit code. Use an intermediate file instead.\n\n> +\t\tsed -e \"s#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g\" | \\\n\nThis sed command doesn't seem to work on macOS (on Travis CI), and\ncauses the test to fail with:\n\n ++git pack-redundant --all\n ++sed -e 's#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g'\n ++sort -u\n ++read p\n ++sort\n ++eval echo '${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}'\n ./test-lib.sh: line 697: ${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}: bad substitution\n ++test_cmp expected actual\n ++diff -u expected actual\n --- expected 2019-01-09 01:53:45.000000000 +0000\n +++ actual 2019-01-09 01:53:45.000000000 +0000\n @@ -1,4 +0,0 @@\n -P1:24ee080366509364d04a138cd4e168dc4ff33354\n -P4:139d8b0cfe7e8970a8f3533835f90278d88de474\n -P5:23e0f02d822fa4bfe5ee63337ba5632cd7be208e\n -P6:deeb289f1749972f1cd57c3b9f359ece2361f60a\n error: last command exited with $?=1\n not ok 2 - git pack-redundant --all\n\nI'm not sure what's wrong with it, though.\n\nMinor nit: 'git pack-redundant' prints one filename per line, so the\n'g' at the end of the 's###g' is not necessary.\n\n> +\t\tsort -u | \\\n> +\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n> +\t\tsort > actual && \\\n\nStyle nit: no space between redirection operator and filename\n\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'remove redundant packs' '\n> +\tgit pack-redundant --all | xargs rm &&\n> +\tgit fsck &&\n> +\ttest $(git pack-redundant --all | wc -l) -eq 0\n> +'\n> +\n> +test_done\n> -- \n> 2.14.5.agit.2\n> \n"},{"id":"366428","messageId":"20190109164731.GJ4673@szeder.dev","threadId":"50055","inReplyTo":"20190109125628.GG4673@szeder.dev","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-09T16:47:31Z","receivedAt":"2019-01-09T16:47:38Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 09, 2019 at 01:56:28PM +0100, SZEDER Gábor wrote:\n> On Wed, Jan 02, 2019 at 12:34:54PM +0800, Jiang Xin wrote:\n> > +cat >expected < > +P1:$P1\n> > +P4:$P4\n> > +P5:$P5\n> > +P6:$P6\n> > +EOF\n\nCreating the expected results could be moved into the\ntest_expect_success block as well.\n\n> > +\n> > +test_expect_success 'git pack-redundant --all' '\n> > +\tgit pack-redundant --all | \\\n> \n> Don't run a git command (especially the particular command the test\n> script focuses on) upstream of a pipe, because it hides the command's\n> exit code. Use an intermediate file instead.\n> \n> > +\t\tsed -e \"s#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g\" | \\\n> \n> This sed command doesn't seem to work on macOS (on Travis CI), and\n> causes the test to fail with:\n> \n> ++git pack-redundant --all\n> ++sed -e 's#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g'\n> ++sort -u\n> ++read p\n> ++sort\n> ++eval echo '${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}'\n> ./test-lib.sh: line 697: ${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}: bad substitution\n> ++test_cmp expected actual\n> ++diff -u expected actual\n> --- expected 2019-01-09 01:53:45.000000000 +0000\n> +++ actual 2019-01-09 01:53:45.000000000 +0000\n> @@ -1,4 +0,0 @@\n> -P1:24ee080366509364d04a138cd4e168dc4ff33354\n> -P4:139d8b0cfe7e8970a8f3533835f90278d88de474\n> -P5:23e0f02d822fa4bfe5ee63337ba5632cd7be208e\n> -P6:deeb289f1749972f1cd57c3b9f359ece2361f60a\n> error: last command exited with $?=1\n> not ok 2 - git pack-redundant --all\n> \n> I'm not sure what's wrong with it, though.\n\nSo, it appears that 'sed' in macOS doesn't understand the\n'\\(idx\\|pack\\)' part of that regex. Turning that command into\n\n sed -e \"s#^.git/objects/pack/pack-\\($OID_REGEX\\)\\..*#\\1#\" out | \\\n\nmakes it work even on macOS, but note that those 40 hexdigits are not\nactual OIDs but file content checksums, so using $OID_REGEX is not the\nright thing to do here (though I'm not sure what is supposed to be\nused instead, as $_x40 hardcodes the number of hexdigits).\n\nAlas, the test as a whole still fails with the following on macOS:\n\n ++diff -u expected actual\n --- expected 2019-01-09 15:54:49.000000000 +0000\n +++ actual 2019-01-09 15:54:49.000000000 +0000\n @@ -1,4 +1,4 @@\n P1:24ee080366509364d04a138cd4e168dc4ff33354\n -P4:139d8b0cfe7e8970a8f3533835f90278d88de474\n +P3:0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e\n P5:23e0f02d822fa4bfe5ee63337ba5632cd7be208e\n -P6:deeb289f1749972f1cd57c3b9f359ece2361f60a\n +P7:4ecc1eb138516a26654cd4e3570b322c0820f170\n error: last command exited with $?=1\n\n"},{"id":"366469","messageId":"CANYiYbGqLHr-t+f6m6gyY3QiYgxbzbqsmmRw-afKe6NG_mxhPQ@mail.gmail.com","threadId":"50055","inReplyTo":"20190109125628.GG4673@szeder.dev","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-10T03:28:34Z","receivedAt":"2019-01-10T03:30:21Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"SZEDER Gábor 于2019年1月9日周三 下午8:56写道：\n>\n> On Wed, Jan 02, 2019 at 12:34:54PM +0800, Jiang Xin wrote:\n> > From: Jiang Xin \n> > +test_description='git redundant test'\n>\n> s/redundant/pack-redundant/ ?\n\nYes, will correct it.\n\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +create_commits()\n> > +{\n> > + set -e\n> > + parent=\n> > + for name in A B C D E F G H I J K L M\n> > + do\n> > + test_tick\n> > + T=$(git write-tree)\n> > + if test -z \"$parent\"\n> > + then\n> > + sha1=$(echo $name | git commit-tree $T)\n>\n> There is a considerable effort going on to switch from SHA-1 to a\n> different hash function, so please don't add any new $sha1 variable;\n> call it $oid or $commit instead.\n>\n\nWill do.\n\n> > +\n> > +# Create commits and packs\n> > +create_commits\n> > +create_redundant_packs\n>\n> Please perform all setup tasks in a test_expect_success block, so we\n> get verbose and trace output about what's going on.\n\nWill do like this:\n\n test_expect_success 'setup' '\n create_commits &&\n create_redundant_packs\n '\n\n> Don't use 'set -e', use an &&-chain instead. To fail the test if a\n> command in the for loop were to fail you could do something like this:\n>\n> for ....\n> do\n> do-this &&\n> do-that ||\n> return 1\n> done\n\nWill do.\n\n> > +\n> > +test_expect_success 'clear loose objects' '\n> > + git prune-packed &&\n> > + test $(find .git/objects -type f | grep -v pack | wc -l) -eq 0\n>\n> Use something like\n>\n> find .git/objects -type f | grep -v pack >out &&\n> test_must_be_empty out\n>\n> instead, so we get an informative error message on failure.\n\nif `grep -v pack` return empty output, it will return error, so\nI will use `sed -e \"/objects\\/pack\\//d\" >out` instead.\n\n>\n> > +'\n> > +\n> > +cat >expected < > +P1:$P1\n> > +P4:$P4\n> > +P5:$P5\n> > +P6:$P6\n> > +EOF\n> > +\n> > +test_expect_success 'git pack-redundant --all' '\n> > + git pack-redundant --all | \\\n>\n> Don't run a git command (especially the particular command the test\n> script focuses on) upstream of a pipe, because it hides the command's\n> exit code. Use an intermediate file instead.\n>\n> > + sed -e \"s#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g\" | \\\n>\n> This sed command doesn't seem to work on macOS (on Travis CI), and\n> causes the test to fail with:\n>\n\nIt works if rewrite as follows:\n\n git pack-redundant --all >out &&\n sed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n\nWithout `-E`, MasOS has to write two seperate sed commands, such as:\n\n git pack-redundant --all >out &&\n sed -e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" out | \\\n sed -e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\"\n\nOption '-E' is an alias for -r in GNU sed 4.2 (added in 4.2, not documented\nunti 4.3), released on May 11 2009. I prefer the `-E` version.\n\n>\n> Minor nit: 'git pack-redundant' prints one filename per line, so the\n> 'g' at the end of the 's###g' is not necessary.\n>\n> > + sort -u | \\\n> > + while read p; do eval echo \"\\${P$p}\"; done | \\\n> > + sort > actual && \\\n>\n> Style nit: no space between redirection operator and filename\n\nWill do.\n"},{"id":"366492","messageId":"74412c28-936a-5276-d6ca-db7d8c5c2124@kdbg.org","threadId":"50055","inReplyTo":"CANYiYbGqLHr-t+f6m6gyY3QiYgxbzbqsmmRw-afKe6NG_mxhPQ@mail.gmail.com","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2019-01-10T07:11:28Z","receivedAt":"2019-01-10T07:11:32Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 10.01.19 um 04:28 schrieb Jiang Xin:\n> SZEDER Gábor 于2019年1月9日周三 下午8:56写道：\n>> Use something like\n>>\n>> find .git/objects -type f | grep -v pack >out &&\n>> test_must_be_empty out\n>>\n>> instead, so we get an informative error message on failure.\n> \n> if `grep -v pack` return empty output, it will return error, so\n> I will use `sed -e \"/objects\\/pack\\//d\" >out` instead.\n\nSo, you could even write this as\n\n\tfind .git/objects -type f >out &&\n\t! grep -v pack out\t# must be empty\nor\n\t! find .git/objects -type f | grep -v pack\n\nif you want to be terse.\n\n-- Hannes\n"},{"id":"366494","messageId":"20190110115704.GL4673@szeder.dev","threadId":"50055","inReplyTo":"CANYiYbGqLHr-t+f6m6gyY3QiYgxbzbqsmmRw-afKe6NG_mxhPQ@mail.gmail.com","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-10T11:57:04Z","receivedAt":"2019-01-10T11:57:11Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jan 10, 2019 at 11:28:34AM +0800, Jiang Xin wrote:\n> SZEDER Gábor 于2019年1月9日周三 下午8:56写道：\n> > > + sed -e \"s#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g\" | \\\n> >\n> > This sed command doesn't seem to work on macOS (on Travis CI), and\n> > causes the test to fail with:\n> >\n> \n> It works if rewrite as follows:\n> \n> git pack-redundant --all >out &&\n> sed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n> \n> Without `-E`, MasOS has to write two seperate sed commands, such as:\n> \n> git pack-redundant --all >out &&\n> sed -e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" out | \\\n> sed -e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\"\n> \n> Option '-E' is an alias for -r in GNU sed 4.2 (added in 4.2, not documented\n> unti 4.3), released on May 11 2009. I prefer the `-E` version.\n\nIs 'sed -E' portable enough, e.g. to the various BSDs, Solaris, and\nwhatnot? I don't know, but POSIX doesn't mention it, there is not a\nsingle instance of it in our current codebase, and it appears that\nwe've never used it before, either. OTOH,\n't/check-non-portable-shell.pl' doesn't catch it as non-portable\nconstruct...\n\n\n"},{"id":"366495","messageId":"20190110120142.22271-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190109164731.GJ4673@szeder.dev","subject":"[PATCH v5 0/5] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-10T12:01:37Z","receivedAt":"2019-01-10T12:05:11Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"> Sun Chao (my former colleague at Huawei) found a bug of\n> git-pack-redundant. If there are too many packs and many of them\n> overlap\n> each other, running `git pack-redundant --all` will exhaust all memories\n> and the process will be killed by kernel.\n> \n> There is a script in commit log of commit 2/3, which can be used to\n> create a repository with lots of redundant packs. Running `git\n> pack-redundant --all` in it can reproduce this issue.\n\nSZEDER reported that t5233 won't pass for MacOS. See solution in patch\n4/5.\n\nChanges since reroll v4:\n\n* Rewrite t5323, add more test cases.\n* Add two new patches, one for refactor, and another changed sorting\n method and fixed t5323 for the new algorithm.\n\nRange diff with sc/pack-redundant feature branch:\n\n 1: 702267a888 < -: ---------- t5323: test cases for git-pack-redundant\n -: ---------- > 1: 40fea5d67f t5323: test cases for git-pack-redundant\n 2: c4b133d858 = 2: 50cd5a5b47 pack-redundant: new algorithm to find min packs\n -: ---------- > 3: 6338c6fad4 pack-redundant: rename pack_list.all_objects\n -: ---------- > 4: 734f4d8a8b pack-redundant: consistent sort method\n 3: 2351d7e8b5 ! 5: b7ccdea1ad pack-redundant: remove unused functions\n @@ -13,7 +13,7 @@\n --- a/builtin/pack-redundant.c\n +++ b/builtin/pack-redundant.c\n @@\n - \tstruct llist *all_objects;\n + \tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n -struct pll {\n @@ -105,7 +105,7 @@\n -\tdiff = llist_copy(list);\n -\n -\twhile (pl) {\n --\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n +-\t\tllist_sorted_difference_inplace(diff, pl->remaining_objects);\n -\t\tif (diff->size == 0) { /* we're done */\n -\t\t\tllist_free(diff);\n -\t\t\treturn 1;\n\n\nJiang Xin (3):\n t5323: test cases for git-pack-redundant\n pack-redundant: rename pack_list.all_objects\n pack-redundant: consistent sort method\n\nSun Chao (2):\n pack-redundant: new algorithm to find min packs\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 221 +++++++++++++++-----------------------\n t/t5323-pack-redundant.sh | 157 +++++++++++++++++++++++++++\n 2 files changed, 242 insertions(+), 136 deletions(-)\n create mode 100755 t/t5323-pack-redundant.sh\n\n\n-- \n2.20.1.101.gc01fadde4e\n\n"},{"id":"366496","messageId":"20190110120142.22271-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190109164731.GJ4673@szeder.dev","subject":"[PATCH v5 1/5] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-10T12:01:38Z","receivedAt":"2019-01-10T12:05:13Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \nReviewed-by: SZEDER Gábor \n---\n t/t5323-pack-redundant.sh | 157 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 157 insertions(+)\n create mode 100755 t/t5323-pack-redundant.sh\n\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..7410426dee\n--- /dev/null\n+++ b/t/t5323-pack-redundant.sh\n@@ -0,0 +1,157 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='git pack-redundant test'\n+\n+. ./test-lib.sh\n+\n+create_commits()\n+{\n+\tparent=\n+\tfor name in A B C D E F G H I J K L M N O P Q R\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\toid=$(echo $name | git commit-tree $T)\n+\t\telse\n+\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n+\t\tfi &&\n+\t\teval $name=$oid &&\n+\t\tparent=$oid ||\n+\t\treturn 1\n+\tdone\n+\tgit update-ref refs/heads/master $M\n+}\n+\n+create_pack_1()\n+{\n+\tP1=$(cd .git/objects/pack; printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n$F\\n$R\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P1=P1:$P1\n+}\n+\n+create_pack_2()\n+{\n+\tP2=$(cd .git/objects/pack; printf \"$B\\n$C\\n$D\\n$E\\n$G\\n$H\\n$I\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P2=P2:$P2\n+}\n+\n+create_pack_3()\n+{\n+\tP3=$(cd .git/objects/pack; printf \"$F\\n$I\\n$J\\n$K\\n$L\\n$M\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P3=P3:$P3\n+}\n+\n+create_pack_4()\n+{\n+\tP4=$(cd .git/objects/pack; printf \"$J\\n$K\\n$L\\n$M\\n$P\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P4=P4:$P4\n+}\n+\n+create_pack_5()\n+{\n+\tP5=$(cd .git/objects/pack; printf \"$G\\n$H\\n$N\\n$O\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P5=P5:$P5\n+}\n+\n+create_pack_6()\n+{\n+\tP6=$(cd .git/objects/pack; printf \"$N\\n$O\\n$Q\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P6=P6:$P6\n+}\n+\n+create_pack_7()\n+{\n+\tP7=$(cd .git/objects/pack; printf \"$P\\n$Q\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P7=P7:$P7\n+}\n+\n+create_pack_8()\n+{\n+\tP8=$(cd .git/objects/pack; printf \"$A\\n\" | git pack-objects pack 2>/dev/null) &&\n+\teval P$P8=P8:$P8\n+}\n+\n+test_expect_success 'setup' '\n+\tcreate_commits\n+'\n+\n+test_expect_success 'no redundant packs' '\n+\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'create pack 4, 5' '\n+\tcreate_pack_4 && create_pack_5\n+'\n+\n+cat >expected <out &&\n+\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n+\t\tsort -u | \\\n+\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n+\t\tsort >actual && \\\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'create pack 6, 7' '\n+\tcreate_pack_6 && create_pack_7\n+'\n+\n+cat >expected <out &&\n+\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n+\t\tsort -u | \\\n+\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n+\t\tsort >actual && \\\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'create pack 8' '\n+\tcreate_pack_8\n+'\n+\n+cat >expected <out &&\n+\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n+\t\tsort -u | \\\n+\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n+\t\tsort >actual && \\\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'clear loose objects' '\n+\tgit prune-packed &&\n+\tfind .git/objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'remove redundant packs' '\n+\tgit pack-redundant --all | xargs rm &&\n+\tgit fsck &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_done\n-- \n2.20.1.101.gc01fadde4e\n\n"},{"id":"366497","messageId":"20190110120142.22271-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190109164731.GJ4673@szeder.dev","subject":"[PATCH v5 2/5] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-10T12:01:39Z","receivedAt":"2019-01-10T12:05:16Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 109 ++++++++++++++++++++++++---------------\n 1 file changed, 68 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..3655cc7dc6 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -421,14 +421,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -446,49 +484,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -603,7 +629,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\n@@ -664,6 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tpl = local_packs;\n \twhile (pl) {\n \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.1.101.gc01fadde4e\n\n"},{"id":"366498","messageId":"20190110120142.22271-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190109164731.GJ4673@szeder.dev","subject":"[PATCH v5 3/5] pack-redundant: rename pack_list.all_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-10T12:01:40Z","receivedAt":"2019-01-10T12:05:19Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nNew algorithm uses `pack_list.all_objects` to track remaining objects,\nso rename it to `pack_list.remaining_objects`.\n\nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 3655cc7dc6..56591d283f 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -32,7 +32,7 @@ static struct pack_list {\n \tstruct pack_list *next;\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n-\tstruct llist *all_objects;\n+\tstruct llist *remaining_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n struct pll {\n@@ -346,7 +346,7 @@ static int is_superset(struct pack_list *pl, struct llist *list)\n \tdiff = llist_copy(list);\n \n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(diff, pl->remaining_objects);\n \t\tif (diff->size == 0) { /* we're done */\n \t\t\tllist_free(diff);\n \t\t\treturn 1;\n@@ -425,8 +425,8 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->all_objects->size;\n-\tsize_t sz_b = pl_b->all_objects->size;\n+\tsize_t sz_a = pl_a->remaining_objects->size;\n+\tsize_t sz_b = pl_b->remaining_objects->size;\n \n \tif (sz_a == sz_b)\n \t\treturn 0;\n@@ -436,7 +436,7 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n \t\treturn -1;\n }\n \n-/* Sort pack_list, greater size of all_objects first */\n+/* Sort pack_list, greater size of remaining_objects first */\n static void sort_pack_list(struct pack_list **pl)\n {\n \tstruct pack_list **ary, *p;\n@@ -480,7 +480,7 @@ static void minimize(struct pack_list **min)\n \tmissing = llist_copy(all_objects);\n \tpl = unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(missing, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(missing, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n \n@@ -498,20 +498,20 @@ static void minimize(struct pack_list **min)\n \t/* remove unique pack objects from the non_unique packs */\n \tpl = non_unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n \n \twhile (non_unique) {\n-\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\t/* sort the non_unique packs, greater size of remaining_objects first */\n \t\tsort_pack_list(&non_unique);\n-\t\tif (non_unique->all_objects->size == 0)\n+\t\tif (non_unique->remaining_objects->size == 0)\n \t\t\tbreak;\n \n \t\tpack_list_insert(min, non_unique);\n \n-\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n-\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\t\tfor (pl = non_unique->next; pl && pl->remaining_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->remaining_objects, non_unique->remaining_objects);\n \n \t\tnon_unique = non_unique->next;\n \t}\n@@ -526,7 +526,7 @@ static void load_all_objects(void)\n \n \twhile (pl) {\n \t\thint = NULL;\n-\t\tl = pl->all_objects->front;\n+\t\tl = pl->remaining_objects->front;\n \t\twhile (l) {\n \t\t\thint = llist_insert_sorted_unique(all_objects,\n \t\t\t\t\t\t\t l->oid, hint);\n@@ -537,7 +537,7 @@ static void load_all_objects(void)\n \t/* remove objects present in remote packs */\n \tpl = altodb_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(all_objects, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n }\n@@ -563,10 +563,10 @@ static void scan_alt_odb_packs(void)\n \t\tlocal = local_packs;\n \t\twhile (local) {\n \t\t\tllist_sorted_difference_inplace(local->unique_objects,\n-\t\t\t\t\t\t\talt->all_objects);\n+\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n-\t\tllist_sorted_difference_inplace(all_objects, alt->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, alt->remaining_objects);\n \t\talt = alt->next;\n \t}\n }\n@@ -581,7 +581,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\treturn NULL;\n \n \tl.pack = p;\n-\tllist_init(&l.all_objects);\n+\tllist_init(&l.remaining_objects);\n \n \tif (open_pack_index(p))\n \t\treturn NULL;\n@@ -590,11 +590,11 @@ static struct pack_list * add_pack(struct packed_git *p)\n \tbase += 256 * 4 + ((p->index_version < 2) ? 4 : 8);\n \tstep = the_hash_algo->rawsz + ((p->index_version < 2) ? 4 : 0);\n \twhile (off < p->num_objects * step) {\n-\t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n+\t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n \t/* this list will be pruned in cmp_two_packs later */\n-\tl.unique_objects = llist_copy(l.all_objects);\n+\tl.unique_objects = llist_copy(l.remaining_objects);\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\n \telse\n@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tpl = local_packs;\n \twhile (pl) {\n \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n-\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.1.101.gc01fadde4e\n\n"},{"id":"366499","messageId":"20190110120142.22271-5-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190109164731.GJ4673@szeder.dev","subject":"[PATCH v5 4/5] pack-redundant: consistent sort method","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-10T12:01:41Z","receivedAt":"2019-01-10T12:05:22Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nSZEDER reported that test case t5323 has different test result on MacOS.\nThis is because `cmp_pack_list_reverse` cannot give identical result\nwhen two pack being sorted has the same size of remaining_objects.\n\nChanges to the sorting function will make consistent test result for\nt5323.\n\nThe new algorithm to find redundant packs is a trade-off to save memory\nresources, and the result of it may be different with old one, and may\nbe not the best result sometimes. Update t5323 for the new algorithm.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 22 +++++++++++++++-------\n t/t5323-pack-redundant.sh | 2 +-\n 2 files changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 56591d283f..e9d2586e2e 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -33,6 +33,7 @@ static struct pack_list {\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n \tstruct llist *remaining_objects;\n+\tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n struct pll {\n@@ -421,16 +422,22 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n-static int cmp_pack_list_reverse(const void *a, const void *b)\n+static int cmp_remaining_objects(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->remaining_objects->size;\n-\tsize_t sz_b = pl_b->remaining_objects->size;\n \n-\tif (sz_a == sz_b)\n-\t\treturn 0;\n-\telse if (sz_a < sz_b)\n+\t/* if have the same remaining_objects, big pack first */\n+\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size)\n+\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n+\t\t\treturn 0;\n+\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n+\t\t\treturn 1;\n+\t\telse\n+\t\t\treturn -1;\n+\n+\t/* sort according to remaining objects, more remaining objects first */\n+\tif (pl_a->remaining_objects->size < pl_b->remaining_objects->size)\n \t\treturn 1;\n \telse\n \t\treturn -1;\n@@ -451,7 +458,7 @@ static void sort_pack_list(struct pack_list **pl)\n \tfor (n = 0, p = *pl; p; p = p->next)\n \t\tary[n++] = p;\n \n-\tQSORT(ary, n, cmp_pack_list_reverse);\n+\tQSORT(ary, n, cmp_remaining_objects);\n \n \t/* link them back again */\n \tfor (i = 0; i < n - 1; i++)\n@@ -593,6 +600,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n+\tl.all_objects_size = l.remaining_objects->size;\n \t/* this list will be pruned in cmp_two_packs later */\n \tl.unique_objects = llist_copy(l.remaining_objects);\n \tif (p->pack_local)\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex 7410426dee..2a09ff1bfb 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -90,7 +90,7 @@ test_expect_success 'create pack 4, 5' '\n '\n \n cat >expected <\n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 86 ----------------------------------------\n 1 file changed, 86 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex e9d2586e2e..dd71fdd435 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -36,11 +36,6 @@ static struct pack_list {\n \tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n-struct pll {\n-\tstruct pll *next;\n-\tstruct pack_list *pl;\n-};\n-\n static struct llist_item *free_nodes;\n \n static inline void llist_item_put(struct llist_item *item)\n@@ -64,15 +59,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -286,78 +272,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->remaining_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.20.1.101.gc01fadde4e\n\n"},{"id":"366501","messageId":"700045da-2d91-a3c7-aaf2-c9d7bdd86a51@web.de","threadId":"50055","inReplyTo":"20190110115704.GL4673@szeder.dev","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-01-10T12:25:35Z","receivedAt":"2019-01-10T12:26:25Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 10.01.19 12:57, SZEDER Gábor wrote:\n> On Thu, Jan 10, 2019 at 11:28:34AM +0800, Jiang Xin wrote:\n>> SZEDER Gábor 于2019年1月9日周三 下午8:56写道：\n>>>> + sed -e \"s#^.*/pack-\\(.*\\)\\.\\(idx\\|pack\\)#\\1#g\" | \\\n>>>\n>>> This sed command doesn't seem to work on macOS (on Travis CI), and\n>>> causes the test to fail with:\n>>>\n>>\n>> It works if rewrite as follows:\n>>\n>> git pack-redundant --all >out &&\n>> sed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n>>\n>> Without `-E`, MasOS has to write two seperate sed commands, such as:\n>>\n>> git pack-redundant --all >out &&\n>> sed -e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" out | \\\n>> sed -e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\"\n>>\n>> Option '-E' is an alias for -r in GNU sed 4.2 (added in 4.2, not documented\n>> unti 4.3), released on May 11 2009. I prefer the `-E` version.\n> \n> Is 'sed -E' portable enough, e.g. to the various BSDs, Solaris, and\n> whatnot? I don't know, but POSIX doesn't mention it, there is not a\n> single instance of it in our current codebase, and it appears that\n> we've never used it before, either. OTOH,\n\nIf we can use \"two seperate sed commands\" i would (really) prefer to so,\nto avoid \"sed -E\".\nMy conclusion is that it is not portable enough.\n> 't/check-non-portable-shell.pl' doesn't catch it as non-portable\n> construct...\n\nGood point.\nActually that script only checks \"known non-portable\" options.\nEvery time somebody finds a non-portable option, we update it.\nA growing blacklist, so to say.\nMay be we should have a white list instead.\n\n\n\n"},{"id":"366508","messageId":"xmqq5zuw4dp9.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190110115704.GL4673@szeder.dev","subject":"Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-10T17:36:34Z","receivedAt":"2019-01-10T17:36:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n>> Without `-E`, MasOS has to write two seperate sed commands, such as:\n>> \n>> git pack-redundant --all >out &&\n>> sed -e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" out | \\\n>> sed -e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\"\n\nTwo commands, perhaps, but does it have to be two separate sed\nprocesses piped together? Why won't something like this work?\n\n\tsed -e 's|.*/pack-\\([0-9a-f]*\\)\\.idx$|\\1' \\\n\t-e 's|.*/pack-\\([0-9a-f]*\\)\\.pack$|\\1'\n\n"},{"id":"366529","messageId":"20190110200503.GM4673@szeder.dev","threadId":"50055","inReplyTo":"20190110120142.22271-5-worldhello.net@gmail.com","subject":"Re: [PATCH v5 4/5] pack-redundant: consistent sort method","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-10T20:05:03Z","receivedAt":"2019-01-10T20:05:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jan 10, 2019 at 08:01:41PM +0800, Jiang Xin wrote:\n> diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\n> index 56591d283f..e9d2586e2e 100644\n> --- a/builtin/pack-redundant.c\n> +++ b/builtin/pack-redundant.c\n\n> @@ -421,16 +422,22 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n> \treturn ret;\n> }\n> \n> -static int cmp_pack_list_reverse(const void *a, const void *b)\n> +static int cmp_remaining_objects(const void *a, const void *b)\n> {\n> \tstruct pack_list *pl_a = *((struct pack_list **)a);\n> \tstruct pack_list *pl_b = *((struct pack_list **)b);\n> -\tsize_t sz_a = pl_a->remaining_objects->size;\n> -\tsize_t sz_b = pl_b->remaining_objects->size;\n> \n> -\tif (sz_a == sz_b)\n> -\t\treturn 0;\n> -\telse if (sz_a < sz_b)\n> +\t/* if have the same remaining_objects, big pack first */\n> +\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size)\n> +\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n> +\t\t\treturn 0;\n> +\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n> +\t\t\treturn 1;\n> +\t\telse\n> +\t\t\treturn -1;\n\nMy compiler complains about the above nested if statements:\n\n builtin/pack-redundant.c: In function ‘cmp_remaining_objects’:\n builtin/pack-redundant.c:345:5: error: suggest explicit braces to avoid ambiguous ‘else’ [-Werror=parentheses]\n if (pl_a->remaining_objects->size == pl_b->remaining_objects->size)\n ^\n cc1: all warnings being treated as errors\n Makefile:2302: recipe for target 'builtin/pack-redundant.o' failed\n\nAfter adding a pair of {} to the outer if statement\n't5323-pack-redundant.sh' passed successfully even on macOS (on Travis\nCI).\n\n"},{"id":"366533","messageId":"xmqqimyw1am8.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190110120142.22271-2-worldhello.net@gmail.com","subject":"Re: [PATCH v5 1/5] t5323: test cases for git-pack-redundant","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-10T21:11:27Z","receivedAt":"2019-01-10T21:11:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jiang Xin writes:\n\n> From: Jiang Xin \n>\n> Add test cases for git pack-redundant to validate new algorithm for git\n> pack-redundant.\n>\n> Signed-off-by: Jiang Xin \n> Reviewed-by: SZEDER Gábor \n> ---\n> t/t5323-pack-redundant.sh | 157 ++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 157 insertions(+)\n> create mode 100755 t/t5323-pack-redundant.sh\n>\n> diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n> new file mode 100755\n> index 0000000000..7410426dee\n> --- /dev/null\n> +++ b/t/t5323-pack-redundant.sh\n> @@ -0,0 +1,157 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2018 Jiang Xin\n> +#\n> +\n> +test_description='git pack-redundant test'\n> +\n> +. ./test-lib.sh\n> +\n> +create_commits()\n> +{\n\nStyle (see Documentation/CodingGuidelines).\n\n> +\tparent=\n> +\tfor name in A B C D E F G H I J K L M N O P Q R\n> +\tdo\n> +\t\ttest_tick &&\n> +\t\tT=$(git write-tree) &&\n> +\t\tif test -z \"$parent\"\n> +\t\tthen\n> +\t\t\toid=$(echo $name | git commit-tree $T)\n> +\t\telse\n> +\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n> +\t\tfi &&\n> +\t\teval $name=$oid &&\n> +\t\tparent=$oid ||\n> +\t\treturn 1\n> +\tdone\n> +\tgit update-ref refs/heads/master $M\n> +}\n> +\n> +create_pack_1()\n> +{\n> +\tP1=$(cd .git/objects/pack; printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n$F\\n$R\\n\" | git pack-objects pack 2>/dev/null) &&\n\nYikes. Can't \"git pack-objects\" get the input directly without\noverlong printf, something along the lines of...\n\n\tP1=$(git -C .git/objects/pack pack-objects pack <<-EOF\n\t\t$A\n\t\t$B\n\t\t$C\n\t\t...\n\t\t$R\n\t\tEOF\n\t)\n\n> +\teval P$P1=P1:$P1\n> +}\n> ...\n> +test_expect_success 'setup' '\n> +\tcreate_commits\n> +'\n> +\n> +test_expect_success 'no redundant packs' '\n> +\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n> +\tgit pack-redundant --all >out &&\n> +\ttest_must_be_empty out\n> +'\n> +\n> +test_expect_success 'create pack 4, 5' '\n> +\tcreate_pack_4 && create_pack_5\n> +'\n> +\n> +cat >expected < +P2:$P2\n> +EOF\n\nMove this to the next \"expect success\" block?\n\n> +test_expect_success 'one of pack-2/pack-3 is redundant' '\n> +\tgit pack-redundant --all >out &&\n> +\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n\nHow portable is \"sed -E\" (it is not even in POSIX.1)? Wouldn't it\nbe easier to work with to have two \"-e\" fed to a single sed\ninvocation instead?\n"},{"id":"366551","messageId":"20190111011941.GB840@szeder.dev","threadId":"50055","inReplyTo":"20190110120142.22271-3-worldhello.net@gmail.com","subject":"Re: [PATCH v5 2/5] pack-redundant: new algorithm to find min packs","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-11T01:19:41Z","receivedAt":"2019-01-11T01:19:53Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jan 10, 2019 at 08:01:39PM +0800, Jiang Xin wrote:\n> diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\n> index cf9a9aabd4..3655cc7dc6 100644\n> --- a/builtin/pack-redundant.c\n> +++ b/builtin/pack-redundant.c\n\n> @@ -446,49 +484,37 @@ static void minimize(struct pack_list **min)\n> \t\tpl = pl->next;\n> \t}\n> \n> +\t*min = unique;\n> +\n> \t/* return if there are no objects missing from the unique set */\n> \tif (missing->size == 0) {\n> -\t\t*min = unique;\n> \t\tfree(missing);\n> \t\treturn;\n> \t}\n> \n> -\t/* find the permutations which contain all missing objects */\n> -\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n> -\t\tperm_all = perm = get_permutations(non_unique, n);\n> -\t\twhile (perm) {\n> -\t\t\tif (is_superset(perm->pl, missing)) {\n> -\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n> -\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n> -\t\t\t\tnew_perm->next = perm_ok;\n> -\t\t\t\tperm_ok = new_perm;\n> -\t\t\t}\n> -\t\t\tperm = perm->next;\n> -\t\t}\n> -\t\tif (perm_ok)\n> -\t\t\tbreak;\n> -\t\tpll_free(perm_all);\n> -\t}\n\nPlease make sure that all commits in the patch series can be build\ncleanly without any warnings (with '-Werror' or preferably with 'make\nDEVELOPER=1') and pass the test suite. This is important, because\nunbuildable commits will cause trouble later on, when e.g. 'git\nbisect' happens to pick such a commit.\n\nIn this case, the removal of the above loop removes all callsites of\nthe static functions get_permutations(), is_superset(), and\npll_free(), resulting the following compiler error:\n\n builtin/pack-redundant.c: At top level:\n builtin/pack-redundant.c:289:13: error: ‘pll_free’ defined but not used [-Werror=unused-function]\n static void pll_free(struct pll *l)\n ^\n builtin/pack-redundant.c:309:21: error: ‘get_permutations’ defined but not used [-Werror=unused-function]\n static struct pll * get_permutations(struct pack_list *list, int n)\n ^\n builtin/pack-redundant.c:343:12: error: ‘is_superset’ defined but not used [-Werror=unused-function]\n static int is_superset(struct pack_list *pl, struct llist *list)\n ^\n\nI see that the last patch in this series removes those three\nunused functions, but that patch should be squashed into this one to\nkeep Git buildable with '-Werror' or DEVELOPER=1.\n\nFurthermore, after building this patch (without '-Werror'), several\ntests in 't5323-pack-redundant.sh' fail. To avoid the test failure I\nthink the fourth patch ensuring a consistent sort order should be\nsquashed in as well.\n\n\n"},{"id":"366552","messageId":"CANYiYbFxCYSt0yQtt5xsFSshVCDuVU58hWBA6AF2e+QzmdUViQ@mail.gmail.com","threadId":"50055","inReplyTo":"xmqqimyw1am8.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 1/5] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-11T01:59:22Z","receivedAt":"2019-01-11T01:59:36Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Junio C Hamano 于2019年1月11日周五 上午5:11写道：\n>\n> Jiang Xin writes:\n>\n> > From: Jiang Xin \n> > +create_commits()\n> > +{\n>\n> Style (see Documentation/CodingGuidelines).\n\nOK, parenthese after function name.\n>\n> > +create_pack_1()\n> > +{\n> > + P1=$(cd .git/objects/pack; printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n$F\\n$R\\n\" | git pack-objects pack 2>/dev/null) &&\n>\n> Yikes. Can't \"git pack-objects\" get the input directly without\n> overlong printf, something along the lines of...\n>\n> P1=$(git -C .git/objects/pack pack-objects pack <<-EOF\n> $A\n> $B\n> $C\n> ...\n> $R\n> EOF\n> )\n\nFind that no space before , because git-pack-objects not allow that,\nand mached parentheses should in the same line.\nSo Will write like this:\n\n create_pack_1() {\n P1=$(git -C .git/objects/pack pack-objects pack <<-EOF) &&\n $T\n $A\n $B\n $R\n EOF\n eval P$P1=P1:$P1\n }\n\n> > +test_expect_success 'no redundant packs' '\n> > + create_pack_1 && create_pack_2 && create_pack_3 &&\n> > + git pack-redundant --all >out &&\n> > + test_must_be_empty out\n> > +'\n> > +\n> > +test_expect_success 'create pack 4, 5' '\n> > + create_pack_4 && create_pack_5\n> > +'\n> > +\n> > +cat >expected < > +P2:$P2\n> > +EOF\n>\n> Move this to the next \"expect success\" block?\n\n$P4 and $P5 are defined after calling `create_pack_4` and `create_pack_5`,\nso create pack functions should be called before write `expected` file,\nif puts $P4 and/or $P5 in the expected file.\n\nFor this case, $P4 and $P5 not in expected file, we can move\ncreate_pack_4 and 5 to the following test_expect_success block,\nbut the new algorithm may change the expected file.\n>\n> > +test_expect_success 'one of pack-2/pack-3 is redundant' '\n> > + git pack-redundant --all >out &&\n> > + sed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n>\n> How portable is \"sed -E\" (it is not even in POSIX.1)? Wouldn't it\n> be easier to work with to have two \"-e\" fed to a single sed\n> invocation instead?\n\nwill fix using two '-e' commands.\n"},{"id":"366567","messageId":"xmqq8szrysz4.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"CANYiYbFxCYSt0yQtt5xsFSshVCDuVU58hWBA6AF2e+QzmdUViQ@mail.gmail.com","subject":"Re: [PATCH v5 1/5] t5323: test cases for git-pack-redundant","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-11T18:00:47Z","receivedAt":"2019-01-11T18:00:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jiang Xin writes:\n\n> Junio C Hamano 于2019年1月11日周五 上午5:11写道：\n>>\n>> Jiang Xin writes:\n>>\n>> > From: Jiang Xin \n>> > +create_commits()\n>> > +{\n>>\n>> Style (see Documentation/CodingGuidelines).\n>\n> OK, parenthese after function name.\n>>\n>> > +create_pack_1()\n>> > +{\n>> > + P1=$(cd .git/objects/pack; printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n$F\\n$R\\n\" | git pack-objects pack 2>/dev/null) &&\n>>\n>> Yikes. Can't \"git pack-objects\" get the input directly without\n>> overlong printf, something along the lines of...\n>>\n>> P1=$(git -C .git/objects/pack pack-objects pack <<-EOF\n>> $A\n>> $B\n>> $C\n>> ...\n>> $R\n>> EOF\n>> )\n>\n> Find that no space before , because git-pack-objects not allow that,\n> and mached parentheses should in the same line.\n> So Will write like this:\n>\n> create_pack_1() {\n> P1=$(git -C .git/objects/pack pack-objects pack <<-EOF) &&\n> $T\n\nIsn't the whole point of <<-EOF (notice the leading dash) to allow\nus to indent the here-doc with horizontal tab?\n\n"},{"id":"366614","messageId":"20190112091754.30985-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190110120142.22271-1-worldhello.net@gmail.com","subject":"[PATCH v6 0/5] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-12T09:17:49Z","receivedAt":"2019-01-12T09:19:52Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"> Sun Chao (my former colleague at Huawei) found a bug of\n> git-pack-redundant. If there are too many packs and many of them\n> overlap each other, running `git pack-redundant --all` will\n> exhaust all memories and the process will be killed by kernel.\n> \n> There is a script in commit log of commit 2/5, which can be used to\n> create a repository with lots of redundant packs. Running `git\n> pack-redundant --all` in it can reproduce this issue.\n\n\nJunio C Hamano 于2019年1月12日周六 上午2:00写道：\n> >> Yikes. Can't \"git pack-objects\" get the input directly without\n> >> overlong printf, something along the lines of...\n> >>\n> >> P1=$(git -C .git/objects/pack pack-objects pack <<-EOF\n> >> $A\n> >> $B\n> >> $C\n> >> ...\n> >> $R\n> >> EOF\n> >> )\n> >\n> > Find that no space before , because git-pack-objects not allow that,\n> > and mached parentheses should in the same line.\n> > So Will write like this:\n> >\n> > create_pack_1() {\n> > P1=$(git -C .git/objects/pack pack-objects pack <<-EOF) &&\n> > $T\n>\n> Isn't the whole point of <<-EOF (notice the leading dash) to allow\n> us to indent the here-doc with horizontal tab?\n\nThe reason that indents are not stripped even with `<<-EOF` is I mixed\ntabs and spaces to make a better align.\n\nIf put the heredoc outside the parentheses, it will failed on MacOS, so\nuse the syntax Junio previously suggested.\n\n\nSZEDER Gábor 于2019年1月11日周五 上午9:19写道：\n> I see that the last patch in this series removes those three\n> unused functions, but that patch should be squashed into this one to\n> keep Git buildable with '-Werror' or DEVELOPER=1.\n>\n> Furthermore, after building this patch (without '-Werror'), several\n> tests in 't5323-pack-redundant.sh' fail. To avoid the test failure I\n> think the fourth patch ensuring a consistent sort order should be\n> squashed in as well.\nPatch 3/5 to 5/5 can be squashed to patch 2/5.\n\n\n## Changes since reroll v5\n\n\n1: 40fea5d67f ! 1: 7e4e703083 t5323: test cases for git-pack-redundant\n @@ -22,8 +22,7 @@\n +\n +. ./test-lib.sh\n +\n -+create_commits()\n -+{\n ++create_commits() {\n +\tparent=\n +\tfor name in A B C D E F G H I J K L M N O P Q R\n +\tdo\n @@ -39,54 +38,98 @@\n +\t\tparent=$oid ||\n +\t\treturn 1\n +\tdone\n -+\tgit update-ref refs/heads/master $M\n ++\tgit update-ref refs/heads/master $R\n +}\n +\n -+create_pack_1()\n -+{\n -+\tP1=$(cd .git/objects/pack; printf \"$T\\n$A\\n$B\\n$C\\n$D\\n$E\\n$F\\n$R\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_1() {\n ++\tP1=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$T\n ++\t\t$A\n ++\t\t$B\n ++\t\t$C\n ++\t\t$D\n ++\t\t$E\n ++\t\t$F\n ++\t\t$R\n ++\t\tEOF\n ++\t) &&\n +\teval P$P1=P1:$P1\n +}\n +\n -+create_pack_2()\n -+{\n -+\tP2=$(cd .git/objects/pack; printf \"$B\\n$C\\n$D\\n$E\\n$G\\n$H\\n$I\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_2() {\n ++\tP2=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$B\n ++\t\t$C\n ++\t\t$D\n ++\t\t$E\n ++\t\t$G\n ++\t\t$H\n ++\t\t$I\n ++\t\tEOF\n ++\t) &&\n +\teval P$P2=P2:$P2\n +}\n +\n -+create_pack_3()\n -+{\n -+\tP3=$(cd .git/objects/pack; printf \"$F\\n$I\\n$J\\n$K\\n$L\\n$M\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_3() {\n ++\tP3=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$F\n ++\t\t$I\n ++\t\t$J\n ++\t\t$K\n ++\t\t$L\n ++\t\t$M\n ++\t\tEOF\n ++\t) &&\n +\teval P$P3=P3:$P3\n +}\n +\n -+create_pack_4()\n -+{\n -+\tP4=$(cd .git/objects/pack; printf \"$J\\n$K\\n$L\\n$M\\n$P\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_4() {\n ++\tP4=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$J\n ++\t\t$K\n ++\t\t$L\n ++\t\t$M\n ++\t\t$P\n ++\t\tEOF\n ++\t) &&\n +\teval P$P4=P4:$P4\n +}\n +\n -+create_pack_5()\n -+{\n -+\tP5=$(cd .git/objects/pack; printf \"$G\\n$H\\n$N\\n$O\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_5() {\n ++\tP5=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$G\n ++\t\t$H\n ++\t\t$N\n ++\t\t$O\n ++\t\tEOF\n ++\t) &&\n +\teval P$P5=P5:$P5\n +}\n +\n -+create_pack_6()\n -+{\n -+\tP6=$(cd .git/objects/pack; printf \"$N\\n$O\\n$Q\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_6() {\n ++\tP6=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$N\n ++\t\t$O\n ++\t\t$Q\n ++\t\tEOF\n ++\t) &&\n +\teval P$P6=P6:$P6\n +}\n +\n -+create_pack_7()\n -+{\n -+\tP7=$(cd .git/objects/pack; printf \"$P\\n$Q\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_7() {\n ++\tP7=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$P\n ++\t\t$Q\n ++\t\tEOF\n ++\t) &&\n +\teval P$P7=P7:$P7\n +}\n +\n -+create_pack_8()\n -+{\n -+\tP8=$(cd .git/objects/pack; printf \"$A\\n\" | git pack-objects pack 2>/dev/null) &&\n ++create_pack_8() {\n ++\tP8=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\t\t$A\n ++\t\tEOF\n ++\t) &&\n +\teval P$P8=P8:$P8\n +}\n +\n @@ -110,10 +153,12 @@\n +\n +test_expect_success 'one of pack-2/pack-3 is redundant' '\n +\tgit pack-redundant --all >out &&\n -+\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n -+\t\tsort -u | \\\n -+\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n -+\t\tsort >actual && \\\n ++\tsed \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n ++\t\tsort -u |\n ++\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n ++\t\tsort >actual &&\n +\ttest_cmp expected actual\n +'\n +\n @@ -121,6 +166,7 @@\n +\tcreate_pack_6 && create_pack_7\n +'\n +\n ++# Only after calling create_pack_6, we can use $P6 variable.\n +cat >expected <out &&\n -+\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n -+\t\tsort -u | \\\n -+\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n -+\t\tsort >actual && \\\n ++\tsed \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n ++\t\tsort -u |\n ++\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n ++\t\tsort >actual &&\n +\ttest_cmp expected actual\n +'\n +\n @@ -147,24 +195,26 @@\n +P8:$P8\n +EOF\n +\n -+test_expect_success 'pack-8, subset of pack-1, is also redundant' '\n ++test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n +\tgit pack-redundant --all >out &&\n -+\tsed -E -e \"s#.*/pack-(.*)\\.(idx|pack)#\\1#\" out | \\\n -+\t\tsort -u | \\\n -+\t\twhile read p; do eval echo \"\\${P$p}\"; done | \\\n -+\t\tsort >actual && \\\n ++\tsed \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n ++\t\tsort -u |\n ++\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n ++\t\tsort >actual &&\n +\ttest_cmp expected actual\n +'\n +\n -+test_expect_success 'clear loose objects' '\n ++test_expect_success 'clean loose objects' '\n +\tgit prune-packed &&\n +\tfind .git/objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n +\ttest_must_be_empty out\n +'\n +\n -+test_expect_success 'remove redundant packs' '\n ++test_expect_success 'remove redundant packs and pass fsck' '\n +\tgit pack-redundant --all | xargs rm &&\n -+\tgit fsck &&\n ++\tgit fsck --no-progress &&\n +\tgit pack-redundant --all >out &&\n +\ttest_must_be_empty out\n +'\n2: 50cd5a5b47 ! 2: 51a9c2d8a5 pack-redundant: new algorithm to find min packs\n @@ -67,7 +67,7 @@\n Original PR and discussions: https://github.com/jiangxin/git/pull/25\n \n Signed-off-by: Sun Chao \n - Signed-off-by: Jiang Xin \n + Signed-off-by: Jiang Xin \n Signed-off-by: Junio C Hamano \n \n diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\n5: b7ccdea1ad ! 3: c5eb21c23c pack-redundant: remove unused functions\n @@ -6,14 +6,14 @@\n `pll_free`, etc.\n \n Signed-off-by: Sun Chao \n - Signed-off-by: Jiang Xin \n + Signed-off-by: Jiang Xin \n Signed-off-by: Junio C Hamano \n \n diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\n --- a/builtin/pack-redundant.c\n +++ b/builtin/pack-redundant.c\n @@\n - \tsize_t all_objects_size;\n + \tstruct llist *all_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n -struct pll {\n @@ -105,7 +105,7 @@\n -\tdiff = llist_copy(list);\n -\n -\twhile (pl) {\n --\t\tllist_sorted_difference_inplace(diff, pl->remaining_objects);\n +-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n -\t\tif (diff->size == 0) { /* we're done */\n -\t\t\tllist_free(diff);\n -\t\t\treturn 1;\n3: 6338c6fad4 ! 4: 1acdd0af1e pack-redundant: rename pack_list.all_objects\n @@ -18,16 +18,7 @@\n +\tstruct llist *remaining_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n - struct pll {\n -@@\n - \tdiff = llist_copy(list);\n - \n - \twhile (pl) {\n --\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n -+\t\tllist_sorted_difference_inplace(diff, pl->remaining_objects);\n - \t\tif (diff->size == 0) { /* we're done */\n - \t\t\tllist_free(diff);\n - \t\t\treturn 1;\n + static struct llist_item *free_nodes;\n @@\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n4: 734f4d8a8b ! 5: 306d515cda pack-redundant: consistent sort method\n @@ -26,7 +26,7 @@\n +\tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n - struct pll {\n + static struct llist_item *free_nodes;\n @@\n \treturn ret;\n }\n @@ -42,20 +42,24 @@\n -\tif (sz_a == sz_b)\n -\t\treturn 0;\n -\telse if (sz_a < sz_b)\n -+\t/* if have the same remaining_objects, big pack first */\n -+\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size)\n ++\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size) {\n ++\t\t/* have the same remaining_objects, big pack first */\n +\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n +\t\t\treturn 0;\n +\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n +\t\t\treturn 1;\n +\t\telse\n +\t\t\treturn -1;\n -+\n -+\t/* sort according to remaining objects, more remaining objects first */\n -+\tif (pl_a->remaining_objects->size < pl_b->remaining_objects->size)\n ++\t} else if (pl_a->remaining_objects->size < pl_b->remaining_objects->size) {\n ++\t\t/* sort by remaining objects, more objects first */\n \t\treturn 1;\n - \telse\n +-\telse\n ++\t} else {\n \t\treturn -1;\n ++\t}\n + }\n + \n + /* Sort pack_list, greater size of remaining_objects first */\n @@\n \tfor (n = 0, p = *pl; p; p = p->next)\n \t\tary[n++] = p;\n\n## This reroll has the following commits:\n\nJiang Xin (3):\n t5323: test cases for git-pack-redundant\n pack-redundant: rename pack_list.all_objects\n pack-redundant: consistent sort method\n\nSun Chao (2):\n pack-redundant: new algorithm to find min packs\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 221 +++++++++++++++-----------------------\n t/t5323-pack-redundant.sh | 207 +++++++++++++++++++++++++++++++++++\n 2 files changed, 292 insertions(+), 136 deletions(-)\n create mode 100755 t/t5323-pack-redundant.sh\n\n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"366615","messageId":"20190112091754.30985-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190110120142.22271-1-worldhello.net@gmail.com","subject":"[PATCH v6 1/5] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-12T09:17:50Z","receivedAt":"2019-01-12T09:19:59Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \nReviewed-by: SZEDER Gábor \n---\n t/t5323-pack-redundant.sh | 207 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 207 insertions(+)\n create mode 100755 t/t5323-pack-redundant.sh\n\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..407838f0e8\n--- /dev/null\n+++ b/t/t5323-pack-redundant.sh\n@@ -0,0 +1,207 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='git pack-redundant test'\n+\n+. ./test-lib.sh\n+\n+create_commits() {\n+\tparent=\n+\tfor name in A B C D E F G H I J K L M N O P Q R\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\toid=$(echo $name | git commit-tree $T)\n+\t\telse\n+\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n+\t\tfi &&\n+\t\teval $name=$oid &&\n+\t\tparent=$oid ||\n+\t\treturn 1\n+\tdone\n+\tgit update-ref refs/heads/master $R\n+}\n+\n+create_pack_1() {\n+\tP1=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$T\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\t$R\n+\t\tEOF\n+\t) &&\n+\teval P$P1=P1:$P1\n+}\n+\n+create_pack_2() {\n+\tP2=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$G\n+\t\t$H\n+\t\t$I\n+\t\tEOF\n+\t) &&\n+\teval P$P2=P2:$P2\n+}\n+\n+create_pack_3() {\n+\tP3=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$F\n+\t\t$I\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\tEOF\n+\t) &&\n+\teval P$P3=P3:$P3\n+}\n+\n+create_pack_4() {\n+\tP4=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\t$P\n+\t\tEOF\n+\t) &&\n+\teval P$P4=P4:$P4\n+}\n+\n+create_pack_5() {\n+\tP5=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$G\n+\t\t$H\n+\t\t$N\n+\t\t$O\n+\t\tEOF\n+\t) &&\n+\teval P$P5=P5:$P5\n+}\n+\n+create_pack_6() {\n+\tP6=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$N\n+\t\t$O\n+\t\t$Q\n+\t\tEOF\n+\t) &&\n+\teval P$P6=P6:$P6\n+}\n+\n+create_pack_7() {\n+\tP7=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$P\n+\t\t$Q\n+\t\tEOF\n+\t) &&\n+\teval P$P7=P7:$P7\n+}\n+\n+create_pack_8() {\n+\tP8=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n+\t\t$A\n+\t\tEOF\n+\t) &&\n+\teval P$P8=P8:$P8\n+}\n+\n+test_expect_success 'setup' '\n+\tcreate_commits\n+'\n+\n+test_expect_success 'no redundant packs' '\n+\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'create pack 4, 5' '\n+\tcreate_pack_4 && create_pack_5\n+'\n+\n+cat >expected <out &&\n+\tsed \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n+\t\tsort -u |\n+\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n+\t\tsort >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'create pack 6, 7' '\n+\tcreate_pack_6 && create_pack_7\n+'\n+\n+# Only after calling create_pack_6, we can use $P6 variable.\n+cat >expected <out &&\n+\tsed \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n+\t\tsort -u |\n+\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n+\t\tsort >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'create pack 8' '\n+\tcreate_pack_8\n+'\n+\n+cat >expected <out &&\n+\tsed \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n+\t\tsort -u |\n+\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n+\t\tsort >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'clean loose objects' '\n+\tgit prune-packed &&\n+\tfind .git/objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'remove redundant packs and pass fsck' '\n+\tgit pack-redundant --all | xargs rm &&\n+\tgit fsck --no-progress &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_done\n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"366616","messageId":"20190112091754.30985-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190110120142.22271-1-worldhello.net@gmail.com","subject":"[PATCH v6 2/5] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-12T09:17:51Z","receivedAt":"2019-01-12T09:19:59Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 109 ++++++++++++++++++++++++---------------\n 1 file changed, 68 insertions(+), 41 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..3655cc7dc6 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -421,14 +421,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -446,49 +484,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -603,7 +629,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\n@@ -664,6 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tpl = local_packs;\n \twhile (pl) {\n \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"366617","messageId":"20190112091754.30985-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190110120142.22271-1-worldhello.net@gmail.com","subject":"[PATCH v6 3/5] pack-redundant: remove unused functions","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-12T09:17:52Z","receivedAt":"2019-01-12T09:19:59Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 86 ----------------------------------------\n 1 file changed, 86 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 3655cc7dc6..eac23500ee 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -35,11 +35,6 @@ static struct pack_list {\n \tstruct llist *all_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n-struct pll {\n-\tstruct pll *next;\n-\tstruct pack_list *pl;\n-};\n-\n static struct llist_item *free_nodes;\n \n static inline void llist_item_put(struct llist_item *item)\n@@ -63,15 +58,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -285,78 +271,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"366618","messageId":"20190112091754.30985-5-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190110120142.22271-1-worldhello.net@gmail.com","subject":"[PATCH v6 4/5] pack-redundant: rename pack_list.all_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-12T09:17:53Z","receivedAt":"2019-01-12T09:20:02Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nNew algorithm uses `pack_list.all_objects` to track remaining objects,\nso rename it to `pack_list.remaining_objects`.\n\nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 36 ++++++++++++++++++------------------\n 1 file changed, 18 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex eac23500ee..64eec3e297 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -32,7 +32,7 @@ static struct pack_list {\n \tstruct pack_list *next;\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n-\tstruct llist *all_objects;\n+\tstruct llist *remaining_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -339,8 +339,8 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->all_objects->size;\n-\tsize_t sz_b = pl_b->all_objects->size;\n+\tsize_t sz_a = pl_a->remaining_objects->size;\n+\tsize_t sz_b = pl_b->remaining_objects->size;\n \n \tif (sz_a == sz_b)\n \t\treturn 0;\n@@ -350,7 +350,7 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n \t\treturn -1;\n }\n \n-/* Sort pack_list, greater size of all_objects first */\n+/* Sort pack_list, greater size of remaining_objects first */\n static void sort_pack_list(struct pack_list **pl)\n {\n \tstruct pack_list **ary, *p;\n@@ -394,7 +394,7 @@ static void minimize(struct pack_list **min)\n \tmissing = llist_copy(all_objects);\n \tpl = unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(missing, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(missing, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n \n@@ -412,20 +412,20 @@ static void minimize(struct pack_list **min)\n \t/* remove unique pack objects from the non_unique packs */\n \tpl = non_unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n \n \twhile (non_unique) {\n-\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\t/* sort the non_unique packs, greater size of remaining_objects first */\n \t\tsort_pack_list(&non_unique);\n-\t\tif (non_unique->all_objects->size == 0)\n+\t\tif (non_unique->remaining_objects->size == 0)\n \t\t\tbreak;\n \n \t\tpack_list_insert(min, non_unique);\n \n-\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n-\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\t\tfor (pl = non_unique->next; pl && pl->remaining_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->remaining_objects, non_unique->remaining_objects);\n \n \t\tnon_unique = non_unique->next;\n \t}\n@@ -440,7 +440,7 @@ static void load_all_objects(void)\n \n \twhile (pl) {\n \t\thint = NULL;\n-\t\tl = pl->all_objects->front;\n+\t\tl = pl->remaining_objects->front;\n \t\twhile (l) {\n \t\t\thint = llist_insert_sorted_unique(all_objects,\n \t\t\t\t\t\t\t l->oid, hint);\n@@ -451,7 +451,7 @@ static void load_all_objects(void)\n \t/* remove objects present in remote packs */\n \tpl = altodb_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(all_objects, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n }\n@@ -477,10 +477,10 @@ static void scan_alt_odb_packs(void)\n \t\tlocal = local_packs;\n \t\twhile (local) {\n \t\t\tllist_sorted_difference_inplace(local->unique_objects,\n-\t\t\t\t\t\t\talt->all_objects);\n+\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n-\t\tllist_sorted_difference_inplace(all_objects, alt->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, alt->remaining_objects);\n \t\talt = alt->next;\n \t}\n }\n@@ -495,7 +495,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\treturn NULL;\n \n \tl.pack = p;\n-\tllist_init(&l.all_objects);\n+\tllist_init(&l.remaining_objects);\n \n \tif (open_pack_index(p))\n \t\treturn NULL;\n@@ -504,11 +504,11 @@ static struct pack_list * add_pack(struct packed_git *p)\n \tbase += 256 * 4 + ((p->index_version < 2) ? 4 : 8);\n \tstep = the_hash_algo->rawsz + ((p->index_version < 2) ? 4 : 0);\n \twhile (off < p->num_objects * step) {\n-\t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n+\t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n \t/* this list will be pruned in cmp_two_packs later */\n-\tl.unique_objects = llist_copy(l.all_objects);\n+\tl.unique_objects = llist_copy(l.remaining_objects);\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\n \telse\n@@ -604,7 +604,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tpl = local_packs;\n \twhile (pl) {\n \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n-\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.0.3.gc45e608566\n\n"},{"id":"366619","messageId":"20190112091754.30985-6-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190110120142.22271-1-worldhello.net@gmail.com","subject":"[PATCH v6 5/5] pack-redundant: consistent sort method","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-12T09:17:54Z","receivedAt":"2019-01-12T09:20:04Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nSZEDER reported that test case t5323 has different test result on MacOS.\nThis is because `cmp_pack_list_reverse` cannot give identical result\nwhen two pack being sorted has the same size of remaining_objects.\n\nChanges to the sorting function will make consistent test result for\nt5323.\n\nThe new algorithm to find redundant packs is a trade-off to save memory\nresources, and the result of it may be different with old one, and may\nbe not the best result sometimes. Update t5323 for the new algorithm.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 24 ++++++++++++++++--------\n t/t5323-pack-redundant.sh | 2 +-\n 2 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 64eec3e297..4448e58a10 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -33,6 +33,7 @@ static struct pack_list {\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n \tstruct llist *remaining_objects;\n+\tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -335,19 +336,25 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n-static int cmp_pack_list_reverse(const void *a, const void *b)\n+static int cmp_remaining_objects(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->remaining_objects->size;\n-\tsize_t sz_b = pl_b->remaining_objects->size;\n \n-\tif (sz_a == sz_b)\n-\t\treturn 0;\n-\telse if (sz_a < sz_b)\n+\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size) {\n+\t\t/* have the same remaining_objects, big pack first */\n+\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n+\t\t\treturn 0;\n+\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n+\t\t\treturn 1;\n+\t\telse\n+\t\t\treturn -1;\n+\t} else if (pl_a->remaining_objects->size < pl_b->remaining_objects->size) {\n+\t\t/* sort by remaining objects, more objects first */\n \t\treturn 1;\n-\telse\n+\t} else {\n \t\treturn -1;\n+\t}\n }\n \n /* Sort pack_list, greater size of remaining_objects first */\n@@ -365,7 +372,7 @@ static void sort_pack_list(struct pack_list **pl)\n \tfor (n = 0, p = *pl; p; p = p->next)\n \t\tary[n++] = p;\n \n-\tQSORT(ary, n, cmp_pack_list_reverse);\n+\tQSORT(ary, n, cmp_remaining_objects);\n \n \t/* link them back again */\n \tfor (i = 0; i < n - 1; i++)\n@@ -507,6 +514,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n+\tl.all_objects_size = l.remaining_objects->size;\n \t/* this list will be pruned in cmp_two_packs later */\n \tl.unique_objects = llist_copy(l.remaining_objects);\n \tif (p->pack_local)\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex 407838f0e8..663328ab30 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -133,7 +133,7 @@ test_expect_success 'create pack 4, 5' '\n '\n \n cat >expected <\n\nFrom `man sed` (on a Mac OS X box):\nThe -E, -a and -i options are non-standard FreeBSD extensions and may not be available\non other operating systems.\n\nFrom `man sed` on a Linux box:\nREGULAR EXPRESSIONS\n POSIX.2 BREs should be supported, but they aren't completely because of\n performance problems. The \\n sequence in a regular expression matches\n the newline character, and similarly for \\a, \\t, and other sequences.\n The -E option switches to using extended regular expressions instead;\n the -E option has been supported for years by GNU sed, and is now\n included in POSIX.\n\nWell, there are still a lot of systems out there, which don't support it.\n\nBeside that, see IEEE Std 1003.1TM-2017\nhttp://pubs.opengroup.org/onlinepubs/9699919799/\ndoes not mention -E either.\n\nTo be on the safe side, don't allow it.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Torsten Bögershausen \n---\n\nI am somewhat unsure if we should disable all options except -e -f -n\ninstead ?\n/\\bsed\\s+-[^efn]/ and err 'Not portable option with sed. Only -n -e -f are portable';\n\nThat would cause a false positive in t9001 here:\n\"--cc-cmd=./cccmd-sed --suppress-cc=self\"\n\nwhich could either be fixed by an anchor:\n/^\\s*sed\\s+-[^efn]/\n\nOr by allowing '--' like this:\n/\\bsed\\s+-[^-efn]/\n\nAny thoughts, please ?\n\nt/check-non-portable-shell.pl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/check-non-portable-shell.pl b/t/check-non-portable-shell.pl\nindex b45bdac688..96b6afdeb8 100755\n--- a/t/check-non-portable-shell.pl\n+++ b/t/check-non-portable-shell.pl\n@@ -35,7 +35,7 @@ sub err {\n \t\tchomp;\n \t}\n\n-\t/\\bsed\\s+-i/ and err 'sed -i is not portable';\n+\t/\\bsed\\s+-[Eail]/ and err 'Not portable option with sed. Only -e -f -n are portable';\n \t/\\becho\\s+-[neE]/ and err 'echo with option is not portable (use printf)';\n \t/^\\s*declare\\s+/ and err 'arrays/declare not portable';\n \t/^\\s*[^#]\\s*which\\s/ and err 'which is not portable (use type)';\n--\n2.20.1.2.gb21ebb671\n\n"},{"id":"366766","messageId":"CAPig+cSeDNYFGYC2WznjW3zYMJCWZbZFY1KM5H5ir2L=Jxwy7w@mail.gmail.com","threadId":"50055","inReplyTo":"20190115203038.10574-1-tboegi@web.de","subject":"Re: [PATCH/RFC v1 1/1] test-lint: sed -E (or -a, -l) are not portable","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-01-15T21:09:22Z","receivedAt":"2019-01-15T21:09:35Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jan 15, 2019 at 3:31 PM wrote:\n> From `man sed` (on a Mac OS X box):\n> The -E, -a and -i options are non-standard FreeBSD extensions and may not be available\n> on other operating systems.\n> [...]\n> To be on the safe side, don't allow it.\n>\n> Signed-off-by: Torsten Bögershausen \n> ---\n> diff --git a/t/check-non-portable-shell.pl b/t/check-non-portable-shell.pl\n> @@ -35,7 +35,7 @@ sub err {\n> - /\\bsed\\s+-i/ and err 'sed -i is not portable';\n> + /\\bsed\\s+-[Eail]/ and err 'Not portable option with sed. Only -e -f -n are portable';\n> /\\becho\\s+-[neE]/ and err 'echo with option is not portable (use printf)';\n> /^\\s*declare\\s+/ and err 'arrays/declare not portable';\n> /^\\s*[^#]\\s*which\\s/ and err 'which is not portable (use type)';\n\nPlease update the new message to be more consistent with existing\nsurrounding error messages. For instance:\n\n err 'sed -i/-a/-l/-E not portable (use only -e/-f/-n)'\n\nor something. Thanks.\n"},{"id":"366815","messageId":"87pnsw4zha.fsf@evledraar.gmail.com","threadId":"50055","inReplyTo":"20190115203038.10574-1-tboegi@web.de","subject":"Re: [PATCH/RFC v1 1/1] test-lint: sed -E (or -a, -l) are not portable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T11:24:17Z","receivedAt":"2019-01-16T11:24:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 15 2019, tboegi@web.de wrote:\n\n> From: Torsten Bögershausen \n>\n> From `man sed` (on a Mac OS X box):\n> The -E, -a and -i options are non-standard FreeBSD extensions and may not be available\n> on other operating systems.\n>\n> From `man sed` on a Linux box:\n> REGULAR EXPRESSIONS\n> POSIX.2 BREs should be supported, but they aren't completely because of\n> performance problems. The \\n sequence in a regular expression matches\n> the newline character, and similarly for \\a, \\t, and other sequences.\n> The -E option switches to using extended regular expressions instead;\n> the -E option has been supported for years by GNU sed, and is now\n> included in POSIX.\n>\n> Well, there are still a lot of systems out there, which don't support it.\n>\n> Beside that, see IEEE Std 1003.1TM-2017\n> http://pubs.opengroup.org/onlinepubs/9699919799/\n> does not mention -E either.\n>\n> To be on the safe side, don't allow it.\n>\n> Reported-by: SZEDER Gábor \n> Signed-off-by: Torsten Bögershausen \n> ---\n>\n> I am somewhat unsure if we should disable all options except -e -f -n\n> instead ?\n> /\\bsed\\s+-[^efn]/ and err 'Not portable option with sed. Only -n -e -f are portable';\n>\n> That would cause a false positive in t9001 here:\n> \"--cc-cmd=./cccmd-sed --suppress-cc=self\"\n>\n> which could either be fixed by an anchor:\n> /^\\s*sed\\s+-[^efn]/\n>\n> Or by allowing '--' like this:\n> /\\bsed\\s+-[^-efn]/\n>\n> Any thoughts, please ?\n>\n> t/check-non-portable-shell.pl | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/check-non-portable-shell.pl b/t/check-non-portable-shell.pl\n> index b45bdac688..96b6afdeb8 100755\n> --- a/t/check-non-portable-shell.pl\n> +++ b/t/check-non-portable-shell.pl\n> @@ -35,7 +35,7 @@ sub err {\n> \t\tchomp;\n> \t}\n>\n> -\t/\\bsed\\s+-i/ and err 'sed -i is not portable';\n> +\t/\\bsed\\s+-[Eail]/ and err 'Not portable option with sed. Only -e -f -n are portable';\n> \t/\\becho\\s+-[neE]/ and err 'echo with option is not portable (use printf)';\n> \t/^\\s*declare\\s+/ and err 'arrays/declare not portable';\n> \t/^\\s*[^#]\\s*which\\s/ and err 'which is not portable (use type)';\n\nI'd just go for your /\\bsed\\s+-[^-efn]/ suggestion. Just a note if we do\ngo for the whitelist: According to GNU sed's manpage -E is also known as\n-r, so /\\bsed\\s+-[Erail]/ would be better.\n"},{"id":"367197","messageId":"20190120075350.5950-1-tboegi@web.de","threadId":"50055","inReplyTo":"20190110115704.GL4673@szeder.dev","subject":"[PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2019-01-20T07:53:50Z","receivedAt":"2019-01-20T07:54:04Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nFrom `man sed` (on a Mac OS X box):\nThe -E, -a and -i options are non-standard FreeBSD extensions and may not be available\non other operating systems.\n\nFrom `man sed` on a Linux box:\nREGULAR EXPRESSIONS\n POSIX.2 BREs should be supported, but they aren't completely because of\n performance problems. The \\n sequence in a regular expression matches the newline\n character, and similarly for \\a, \\t, and other sequences.\n The -E option switches to using extended regular expressions instead; the -E option\n has been supported for years by GNU sed, and is now included in POSIX.\n\nWell, there are still a lot of systems out there, which don't support it.\nBeside that, IEEE Std 1003.1TM-2017, see\nhttp://pubs.opengroup.org/onlinepubs/9699919799/\ndoes not mention -E either.\n\nTo be on the safe side, don't allow -E (or -r, which is GNU).\nChange check-non-portable-shell.pl to only accept the portable options:\nsed [-n] [-e command] [-f command_file]\n\nReported-by: SZEDER Gábor \nHelped-by: Eric Sunshine \nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Torsten Bögershausen \n---\n t/check-non-portable-shell.pl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/check-non-portable-shell.pl b/t/check-non-portable-shell.pl\nindex b45bdac688..6c798608a9 100755\n--- a/t/check-non-portable-shell.pl\n+++ b/t/check-non-portable-shell.pl\n@@ -35,7 +35,7 @@ sub err {\n \t\tchomp;\n \t}\n\n-\t/\\bsed\\s+-i/ and err 'sed -i is not portable';\n+\t/\\bsed\\s+-[^efn]\\s+/ and err 'Not portable option with sed (use only [-n] [-e command] [-f command_file])';\n \t/\\becho\\s+-[neE]/ and err 'echo with option is not portable (use printf)';\n \t/^\\s*declare\\s+/ and err 'arrays/declare not portable';\n \t/^\\s*[^#]\\s*which\\s/ and err 'which is not portable (use type)';\n--\n2.20.1.2.gb21ebb671\n\n"},{"id":"367346","messageId":"xmqqo988fpag.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190120075350.5950-1-tboegi@web.de","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-22T19:47:03Z","receivedAt":"2019-01-22T19:47:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"tboegi@web.de writes:\n\n> From: Torsten Bögershausen \n>\n> From `man sed` (on a Mac OS X box):\n> The -E, -a and -i options are non-standard FreeBSD extensions and may not be available\n> on other operating systems.\n>\n> -\t/\\bsed\\s+-i/ and err 'sed -i is not portable';\n> +\t/\\bsed\\s+-[^efn]\\s+/ and err 'Not portable option with sed (use only [-n] [-e command] [-f command_file])';\n\n\"sed -n -i -E -e 's/foo/bar/p'\" won't be caught with this as an\nerror, but that's OK ;-).\n\nIs this still an RFC patch?\n"},{"id":"367348","messageId":"799c5063-b0a7-df6b-6f86-a92b8b6d679b@web.de","threadId":"50055","inReplyTo":"xmqqo988fpag.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-01-22T20:00:09Z","receivedAt":"2019-01-22T20:00:23Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 22.01.19 20:47, Junio C Hamano wrote:\n> tboegi@web.de writes:\n>\n>> From: Torsten Bögershausen \n>>\n>> From `man sed` (on a Mac OS X box):\n>> The -E, -a and -i options are non-standard FreeBSD extensions and may not be available\n>> on other operating systems.\n>>\n>> -\t/\\bsed\\s+-i/ and err 'sed -i is not portable';\n>> +\t/\\bsed\\s+-[^efn]\\s+/ and err 'Not portable option with sed (use only [-n] [-e command] [-f command_file])';\n>\n> \"sed -n -i -E -e 's/foo/bar/p'\" won't be caught with this as an\n> error, but that's OK ;-).\n>\n> Is this still an RFC patch?\n>\n\nIt seems as if everybody is happy with it,\nso it may be ready for for pu.\n\n"},{"id":"367361","messageId":"CAPig+cSDnhvVCDE15koO9M1A8TBg+Mbn2OBsfXVvshmRyybApw@mail.gmail.com","threadId":"50055","inReplyTo":"799c5063-b0a7-df6b-6f86-a92b8b6d679b@web.de","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-01-22T21:15:57Z","receivedAt":"2019-01-22T21:16:10Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jan 22, 2019 at 3:00 PM Torsten Bögershausen wrote:\n> On 22.01.19 20:47, Junio C Hamano wrote:\n> > tboegi@web.de writes:\n> >> - /\\bsed\\s+-i/ and err 'sed -i is not portable';\n> >> + /\\bsed\\s+-[^efn]\\s+/ and err 'Not portable option with sed (use only [-n] [-e command] [-f command_file])';\n> >\n> > \"sed -n -i -E -e 's/foo/bar/p'\" won't be caught with this as an\n> > error, but that's OK ;-).\n> > Is this still an RFC patch?\n>\n> It seems as if everybody is happy with it,\n> so it may be ready for for pu.\n\nI'd still prefer to see a more terse[1] (and not capitalized) message\nto be consistent with existing error messages and to keep the reported\nerrors more compact overall to make them easier to digest[2,3]:\n\n err 'sed option not portable (use only -n, -e, -f)'\n\nBut that's just a very minor nit.\n\n[1]: http://public-inbox.org/git/CAPig+cSeDNYFGYC2WznjW3zYMJCWZbZFY1KM5H5ir2L=Jxwy7w@mail.gmail.com/\n[2]: http://public-inbox.org/git/20180713055205.32351-3-sunshine@sunshineco.com/\n[3]: http://public-inbox.org/git/20180713055205.32351-4-sunshine@sunshineco.com/\n"},{"id":"367405","messageId":"f7871d8b-a19b-5816-120b-ce583debc4ca@web.de","threadId":"50055","inReplyTo":"CAPig+cSDnhvVCDE15koO9M1A8TBg+Mbn2OBsfXVvshmRyybApw@mail.gmail.com","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-01-23T06:35:24Z","receivedAt":"2019-01-23T06:35:44Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 22.01.19 22:15, Eric Sunshine wrote:\n> On Tue, Jan 22, 2019 at 3:00 PM Torsten Bögershausen wrote:\n>> On 22.01.19 20:47, Junio C Hamano wrote:\n>>> tboegi@web.de writes:\n>>>> - /\\bsed\\s+-i/ and err 'sed -i is not portable';\n>>>> + /\\bsed\\s+-[^efn]\\s+/ and err 'Not portable option with sed (use only [-n] [-e command] [-f command_file])';\n>>>\n>>> \"sed -n -i -E -e 's/foo/bar/p'\" won't be caught with this as an\n>>> error, but that's OK ;-).\n>>> Is this still an RFC patch?\n>>\n>> It seems as if everybody is happy with it,\n>> so it may be ready for for pu.\n>\n> I'd still prefer to see a more terse[1] (and not capitalized) message\n> to be consistent with existing error messages and to keep the reported\n> errors more compact overall to make them easier to digest[2,3]:\n>\n> err 'sed option not portable (use only -n, -e, -f)'\n>\n\nThat's OK for me - lets see if there are more comments.\n\n\n> But that's just a very minor nit.\n>\n> [1]: http://public-inbox.org/git/CAPig+cSeDNYFGYC2WznjW3zYMJCWZbZFY1KM5H5ir2L=Jxwy7w@mail.gmail.com/\n> [2]: http://public-inbox.org/git/20180713055205.32351-3-sunshine@sunshineco.com/\n> [3]: http://public-inbox.org/git/20180713055205.32351-4-sunshine@sunshineco.com/\n>\n\n"},{"id":"367447","messageId":"xmqq8szbcl9q.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"f7871d8b-a19b-5816-120b-ce583debc4ca@web.de","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T17:54:25Z","receivedAt":"2019-01-23T17:54:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> I'd still prefer to see a more terse[1] (and not capitalized) message\n>> to be consistent with existing error messages and to keep the reported\n>> errors more compact overall to make them easier to digest[2,3]:\n>>\n>> err 'sed option not portable (use only -n, -e, -f)'\n>\n> That's OK for me - lets see if there are more comments.\n\nThanks, both. Eric's proposed message looks good to me too.\n"},{"id":"367672","messageId":"20190125191224.pgugeqskuhzi34hv@tb-raspi4","threadId":"50055","inReplyTo":"xmqq8szbcl9q.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-01-25T19:12:24Z","receivedAt":"2019-01-25T19:12:43Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Jan 23, 2019 at 09:54:25AM -0800, Junio C Hamano wrote:\n> Torsten Bögershausen writes:\n>\n> >> I'd still prefer to see a more terse[1] (and not capitalized) message\n> >> to be consistent with existing error messages and to keep the reported\n> >> errors more compact overall to make them easier to digest[2,3]:\n> >>\n> >> err 'sed option not portable (use only -n, -e, -f)'\n> >\n> > That's OK for me - lets see if there are more comments.\n>\n> Thanks, both. Eric's proposed message looks good to me too.\n\nDo you want to ammend the patch locally ?\nOr should I send a new version ?\n"},{"id":"367779","messageId":"xmqq1s4xg26x.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190125191224.pgugeqskuhzi34hv@tb-raspi4","subject":"Re: [PATCH/RFC v2 1/1] test-lint: Only use only sed [-n] [-e command] [-f command_file]","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-27T22:34:14Z","receivedAt":"2019-01-27T22:34:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> Thanks, both. Eric's proposed message looks good to me too.\n>\n> Do you want to ammend the patch locally ?\n\nI just amended it; thanks.\n\n\n"},{"id":"368114","messageId":"20190130114736.30357-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 0/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:30Z","receivedAt":"2019-01-30T11:48:20Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Sun Chao (my former colleague at Huawei) found a bug of\ngit-pack-redundant. If there are too many packs and many of them\noverlap each other, running `git pack-redundant --all` will\nexhaust all memories and the process will be killed by kernel.\n\nThere is a script in commit log of commit 3/6, which can be used to\ncreate a repository with lots of redundant packs. Running `git\npack-redundant --all` in it can reproduce this issue.\n\nDerrick Stolee 于2019年1月20日周日 上午9:08写道：\n>\n> Here is today's test coverage report.\n>\n> builtin/pack-redundant.c\n> a338d10395 builtin/pack-redundant.c 339) static int cmp_remaining_objects(const void *a, const void *b)\n> e4e2c2884e builtin/pack-redundant.c 341) struct pack_list *pl_a = *((struct pack_list **)a);\n> e4e2c2884e builtin/pack-redundant.c 342) struct pack_list *pl_b = *((struct pack_list **)b);\n> ...\n\nAdd new test cases in t5323 for better test coverage.\n\n## Changes since reroll v6\n\n* Add new test cases in t5323.\n\n* Add new patch 2/6 (pack-redundant: delay creation of unique_objects), \n which will fix a bug which fail to find redundant packs if turn on alt-odb\n searching with `--alt-odb` option. This is because alt-odb objects are\n only remove in unique_objects fields but not in all_objects fields of\n pack_list.\n\n## Range diff\n\n1: be6555ae60 ! 1: 799e804d5e t5323: test cases for git-pack-redundant\n @@ -43,7 +43,7 @@\n +}\n +\n +create_pack_1 () {\n -+\tP1=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP1=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$T\n +\t\t$A\n +\t\t$B\n @@ -58,7 +58,7 @@\n +}\n +\n +create_pack_2 () {\n -+\tP2=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP2=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$B\n +\t\t$C\n +\t\t$D\n @@ -72,7 +72,7 @@\n +}\n +\n +create_pack_3 () {\n -+\tP3=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP3=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$F\n +\t\t$I\n +\t\t$J\n @@ -85,7 +85,7 @@\n +}\n +\n +create_pack_4 () {\n -+\tP4=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP4=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$J\n +\t\t$K\n +\t\t$L\n @@ -97,7 +97,7 @@\n +}\n +\n +create_pack_5 () {\n -+\tP5=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP5=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$G\n +\t\t$H\n +\t\t$N\n @@ -108,7 +108,7 @@\n +}\n +\n +create_pack_6 () {\n -+\tP6=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP6=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$N\n +\t\t$O\n +\t\t$Q\n @@ -118,7 +118,7 @@\n +}\n +\n +create_pack_7 () {\n -+\tP7=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP7=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$P\n +\t\t$Q\n +\t\tEOF\n @@ -127,18 +127,37 @@\n +}\n +\n +create_pack_8 () {\n -+\tP8=$(git -C .git/objects/pack pack-objects -q pack <<-EOF\n ++\tP8=$(git -C objects/pack pack-objects -q pack <<-EOF\n +\t\t$A\n +\t\tEOF\n +\t) &&\n +\teval P$P8=P8:$P8\n +}\n +\n -+test_expect_success 'setup' '\n ++format_packfiles () {\n ++\tsed \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n ++\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" |\n ++\tsort -u |\n ++\twhile read p\n ++\tdo\n ++\t\tif test -z \"$(eval echo \\${P$p})\"\n ++\t\tthen\n ++\t\t\techo $p\n ++\t\telse\n ++\t\t\teval echo \"\\${P$p}\"\n ++\t\tfi\n ++\tdone |\n ++\tsort\n ++}\n ++\n ++test_expect_success 'setup master.git' '\n ++\tgit init --bare master.git &&\n ++\tcd master.git &&\n +\tcreate_commits\n +'\n +\n -+test_expect_success 'no redundant packs' '\n ++test_expect_success 'no redundant for pack 1, 2, 3' '\n +\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n +\tgit pack-redundant --all >out &&\n +\ttest_must_be_empty out\n @@ -154,12 +173,7 @@\n +\n +test_expect_success 'one of pack-2/pack-3 is redundant' '\n +\tgit pack-redundant --all >out &&\n -+\tsed \\\n -+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n -+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n -+\t\tsort -u |\n -+\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n -+\t\tsort >actual &&\n ++\tformat_packfiles actual &&\n +\ttest_cmp expected actual\n +'\n +\n @@ -176,12 +190,7 @@\n +\n +test_expect_success 'pack 2, 4, and 6 are redundant' '\n +\tgit pack-redundant --all >out &&\n -+\tsed \\\n -+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n -+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n -+\t\tsort -u |\n -+\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n -+\t\tsort >actual &&\n ++\tformat_packfiles actual &&\n +\ttest_cmp expected actual\n +'\n +\n @@ -198,18 +207,13 @@\n +\n +test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n +\tgit pack-redundant --all >out &&\n -+\tsed \\\n -+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n -+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" out |\n -+\t\tsort -u |\n -+\t\twhile read p; do eval echo \"\\${P$p}\"; done |\n -+\t\tsort >actual &&\n ++\tformat_packfiles actual &&\n +\ttest_cmp expected actual\n +'\n +\n +test_expect_success 'clean loose objects' '\n +\tgit prune-packed &&\n -+\tfind .git/objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n ++\tfind objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n +\ttest_must_be_empty out\n +'\n +\n @@ -220,4 +224,115 @@\n +\ttest_must_be_empty out\n +'\n +\n ++test_expect_success 'setup shared.git' '\n ++\tcd \"$TRASH_DIRECTORY\" &&\n ++\tgit clone -q --mirror master.git shared.git &&\n ++\tcd shared.git &&\n ++\tprintf \"../../master.git/objects\" >objects/info/alternates\n ++'\n ++\n ++test_expect_success 'no redundant packs without --alt-odb' '\n ++\tgit pack-redundant --all >out &&\n ++\ttest_must_be_empty out\n ++'\n ++\n ++cat >expected <out 2>out.err &&\n ++\ttest_must_be_empty out &&\n ++\tgrep \"pack$\" out.err | format_packfiles >actual &&\n ++\ttest_cmp expected actual\n ++'\n ++\n ++cat >expected <actual 2>&1 &&\n ++\ttest_cmp expected actual\n ++'\n ++\n ++create_commits_others () {\n ++\tparent=$(git rev-parse HEAD)\n ++\tfor name in X Y Z\n ++\tdo\n ++\t\ttest_tick &&\n ++\t\tT=$(git write-tree) &&\n ++\t\tif test -z \"$parent\"\n ++\t\tthen\n ++\t\t\toid=$(echo $name | git commit-tree $T)\n ++\t\telse\n ++\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n ++\t\tfi &&\n ++\t\teval $name=$oid &&\n ++\t\tparent=$oid ||\n ++\t\treturn 1\n ++\tdone\n ++\tgit update-ref refs/heads/master $Z\n ++}\n ++\n ++create_pack_x1 () {\n ++\tPx1=$(git -C objects/pack pack-objects -q pack <<-EOF\n ++\t\t$X\n ++\t\t$Y\n ++\t\t$Z\n ++\t\t$A\n ++\t\t$B\n ++\t\t$C\n ++\t\tEOF\n ++\t) &&\n ++\teval P${Px1}=Px1:${Px1}\n ++}\n ++\n ++create_pack_x2 () {\n ++\tPx2=$(git -C objects/pack pack-objects -q pack <<-EOF\n ++\t\t$X\n ++\t\t$Y\n ++\t\t$Z\n ++\t\t$D\n ++\t\t$E\n ++\t\t$F\n ++\t\tEOF\n ++\t) &&\n ++\teval P${Px2}=Px2:${Px2}\n ++}\n ++\n ++test_expect_success 'new objects and packs in shared.git' '\n ++\tcreate_commits_others &&\n ++\tcreate_pack_x1 &&\n ++\tcreate_pack_x2 &&\n ++\tgit pack-redundant --all >out &&\n ++\ttest_must_be_empty out\n ++'\n ++\n ++test_expect_success 'one pack is redundant' '\n ++\tgit pack-redundant --all --alt-odb >out &&\n ++\tformat_packfiles actual &&\n ++\ttest_line_count = 1 actual\n ++'\n ++\n ++cat >expected <out <<-EOF &&\n ++\t\t$X\n ++\t\t$Y\n ++\t\t$Z\n ++\t\tEOF\n ++\tformat_packfiles actual &&\n ++\ttest_cmp expected actual\n ++'\n ++\n +test_done\n-: ---------- > 2: 520f6277fb pack-redundant: delay creation of unique_objects\n2: e4e2c2884e ! 3: ab1c2c4950 pack-redundant: new algorithm to find min packs\n @@ -64,6 +64,9 @@\n 4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n \n + Some test cases will fail on Mac OS X. Mark them and will resolve in\n + later commit.\n +\n Original PR and discussions: https://github.com/jiangxin/git/pull/25\n \n Signed-off-by: Sun Chao \n @@ -213,11 +216,61 @@\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\n +\n + diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n + --- a/t/t5323-pack-redundant.sh\n + +++ b/t/t5323-pack-redundant.sh\n @@\n - \tpl = local_packs;\n - \twhile (pl) {\n - \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n -+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n - \t\tpl = pl->next;\n - \t}\n + P2:$P2\n + EOF\n + \n +-test_expect_success 'one of pack-2/pack-3 is redundant' '\n ++test_expect_failure 'one of pack-2/pack-3 is redundant' '\n + \tgit pack-redundant --all >out &&\n + \tformat_packfiles actual &&\n + \ttest_cmp expected actual\n +@@\n + P6:$P6\n + EOF\n + \n +-test_expect_success 'pack 2, 4, and 6 are redundant' '\n ++test_expect_failure 'pack 2, 4, and 6 are redundant' '\n + \tgit pack-redundant --all >out &&\n + \tformat_packfiles actual &&\n + \ttest_cmp expected actual\n +@@\n + P8:$P8\n + EOF\n + \n +-test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n ++test_expect_failure 'pack-8 (subset of pack-1) is also redundant' '\n + \tgit pack-redundant --all >out &&\n + \tformat_packfiles actual &&\n + \ttest_cmp expected actual\n +@@\n + \ttest_must_be_empty out\n + '\n + \n +-test_expect_success 'remove redundant packs and pass fsck' '\n ++test_expect_failure 'remove redundant packs and pass fsck' '\n + \tgit pack-redundant --all | xargs rm &&\n + \tgit fsck --no-progress &&\n + \tgit pack-redundant --all >out &&\n +@@\n + \tprintf \"../../master.git/objects\" >objects/info/alternates\n + '\n + \n +-test_expect_success 'no redundant packs without --alt-odb' '\n ++test_expect_failure 'no redundant packs without --alt-odb' '\n + \tgit pack-redundant --all >out &&\n + \ttest_must_be_empty out\n + '\n +@@\n + P7:$P7\n + EOF\n \n +-test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n ++test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr' '\n + \tgit pack-redundant --all --verbose >out 2>out.err &&\n + \ttest_must_be_empty out &&\n + \tgrep \"pack$\" out.err | format_packfiles >actual &&\n3: e60b134e66 = 4: 3c3a7ea40f pack-redundant: remove unused functions\n4: cb7e0336fc ! 5: bc4b681f40 pack-redundant: rename pack_list.all_objects\n @@ -20,6 +20,18 @@\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n +@@\n + \tconst unsigned int hashsz = the_hash_algo->rawsz;\n + \n + \tif (!p1->unique_objects)\n +-\t\tp1->unique_objects = llist_copy(p1->all_objects);\n ++\t\tp1->unique_objects = llist_copy(p1->remaining_objects);\n + \tif (!p2->unique_objects)\n +-\t\tp2->unique_objects = llist_copy(p2->all_objects);\n ++\t\tp2->unique_objects = llist_copy(p2->remaining_objects);\n + \n + \tp1_base = p1->pack->index_data;\n + \tp2_base = p2->pack->index_data;\n @@\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n @@ -94,10 +106,12 @@\n \t}\n }\n @@\n + \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n - \t\t\tllist_sorted_difference_inplace(local->unique_objects,\n +-\t\t\tllist_sorted_difference_inplace(local->all_objects,\n -\t\t\t\t\t\t\talt->all_objects);\n ++\t\t\tllist_sorted_difference_inplace(local->remaining_objects,\n +\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n @@ -123,16 +137,11 @@\n +\t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n - \t/* this list will be pruned in cmp_two_packs later */\n --\tl.unique_objects = llist_copy(l.all_objects);\n -+\tl.unique_objects = llist_copy(l.remaining_objects);\n - \tif (p->pack_local)\n - \t\treturn pack_list_insert(&local_packs, &l);\n - \telse\n + \tl.unique_objects = NULL;\n @@\n + \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n - \t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n -\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n +\t\tllist_sorted_difference_inplace(pl->remaining_objects, ignore);\n \t\tpl = pl->next;\n5: a338d10395 ! 6: 6cfba5b4b2 pack-redundant: consistent sort method\n @@ -75,9 +75,9 @@\n \t\toff += step;\n \t}\n +\tl.all_objects_size = l.remaining_objects->size;\n - \t/* this list will be pruned in cmp_two_packs later */\n - \tl.unique_objects = llist_copy(l.remaining_objects);\n + \tl.unique_objects = NULL;\n \tif (p->pack_local)\n + \t\treturn pack_list_insert(&local_packs, &l);\n \n diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n --- a/t/t5323-pack-redundant.sh\n @@ -90,4 +90,53 @@\n +P3:$P3\n EOF\n \n - test_expect_success 'one of pack-2/pack-3 is redundant' '\n +-test_expect_failure 'one of pack-2/pack-3 is redundant' '\n ++test_expect_success 'one of pack-2/pack-3 is redundant' '\n + \tgit pack-redundant --all >out &&\n + \tformat_packfiles actual &&\n + \ttest_cmp expected actual\n +@@\n + P6:$P6\n + EOF\n + \n +-test_expect_failure 'pack 2, 4, and 6 are redundant' '\n ++test_expect_success 'pack 2, 4, and 6 are redundant' '\n + \tgit pack-redundant --all >out &&\n + \tformat_packfiles actual &&\n + \ttest_cmp expected actual\n +@@\n + P8:$P8\n + EOF\n + \n +-test_expect_failure 'pack-8 (subset of pack-1) is also redundant' '\n ++test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n + \tgit pack-redundant --all >out &&\n + \tformat_packfiles actual &&\n + \ttest_cmp expected actual\n +@@\n + \ttest_must_be_empty out\n + '\n + \n +-test_expect_failure 'remove redundant packs and pass fsck' '\n ++test_expect_success 'remove redundant packs and pass fsck' '\n + \tgit pack-redundant --all | xargs rm &&\n + \tgit fsck --no-progress &&\n + \tgit pack-redundant --all >out &&\n +@@\n + \tprintf \"../../master.git/objects\" >objects/info/alternates\n + '\n + \n +-test_expect_failure 'no redundant packs without --alt-odb' '\n ++test_expect_success 'no redundant packs without --alt-odb' '\n + \tgit pack-redundant --all >out &&\n + \ttest_must_be_empty out\n + '\n +@@\n + P7:$P7\n + EOF\n + \n +-test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr' '\n ++test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n + \tgit pack-redundant --all --verbose >out 2>out.err &&\n + \ttest_must_be_empty out &&\n + \tgrep \"pack$\" out.err | format_packfiles >actual &&\n\n--\nJiang Xin (4):\n t5323: test cases for git-pack-redundant\n pack-redundant: delay creation of unique_objects\n pack-redundant: rename pack_list.all_objects\n pack-redundant: consistent sort method\n\nSun Chao (2):\n pack-redundant: new algorithm to find min packs\n pack-redundant: remove unused functions\n\n builtin/pack-redundant.c | 233 +++++++++++----------------\n t/t5323-pack-redundant.sh | 322 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 415 insertions(+), 140 deletions(-)\n create mode 100755 t/t5323-pack-redundant.sh\n\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368115","messageId":"20190130114736.30357-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:31Z","receivedAt":"2019-01-30T11:48:28Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \nReviewed-by: SZEDER Gábor \nSigned-off-by: Junio C Hamano \n---\n t/t5323-pack-redundant.sh | 322 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 322 insertions(+)\n create mode 100755 t/t5323-pack-redundant.sh\n\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..710fe9884c\n--- /dev/null\n+++ b/t/t5323-pack-redundant.sh\n@@ -0,0 +1,322 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='git pack-redundant test'\n+\n+. ./test-lib.sh\n+\n+create_commits () {\n+\tparent=\n+\tfor name in A B C D E F G H I J K L M N O P Q R\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\toid=$(echo $name | git commit-tree $T)\n+\t\telse\n+\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n+\t\tfi &&\n+\t\teval $name=$oid &&\n+\t\tparent=$oid ||\n+\t\treturn 1\n+\tdone\n+\tgit update-ref refs/heads/master $R\n+}\n+\n+create_pack_1 () {\n+\tP1=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$T\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\t$R\n+\t\tEOF\n+\t) &&\n+\teval P$P1=P1:$P1\n+}\n+\n+create_pack_2 () {\n+\tP2=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$G\n+\t\t$H\n+\t\t$I\n+\t\tEOF\n+\t) &&\n+\teval P$P2=P2:$P2\n+}\n+\n+create_pack_3 () {\n+\tP3=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$F\n+\t\t$I\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\tEOF\n+\t) &&\n+\teval P$P3=P3:$P3\n+}\n+\n+create_pack_4 () {\n+\tP4=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\t$P\n+\t\tEOF\n+\t) &&\n+\teval P$P4=P4:$P4\n+}\n+\n+create_pack_5 () {\n+\tP5=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$G\n+\t\t$H\n+\t\t$N\n+\t\t$O\n+\t\tEOF\n+\t) &&\n+\teval P$P5=P5:$P5\n+}\n+\n+create_pack_6 () {\n+\tP6=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$N\n+\t\t$O\n+\t\t$Q\n+\t\tEOF\n+\t) &&\n+\teval P$P6=P6:$P6\n+}\n+\n+create_pack_7 () {\n+\tP7=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$P\n+\t\t$Q\n+\t\tEOF\n+\t) &&\n+\teval P$P7=P7:$P7\n+}\n+\n+create_pack_8 () {\n+\tP8=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$A\n+\t\tEOF\n+\t) &&\n+\teval P$P8=P8:$P8\n+}\n+\n+format_packfiles () {\n+\tsed \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" |\n+\tsort -u |\n+\twhile read p\n+\tdo\n+\t\tif test -z \"$(eval echo \\${P$p})\"\n+\t\tthen\n+\t\t\techo $p\n+\t\telse\n+\t\t\teval echo \"\\${P$p}\"\n+\t\tfi\n+\tdone |\n+\tsort\n+}\n+\n+test_expect_success 'setup master.git' '\n+\tgit init --bare master.git &&\n+\tcd master.git &&\n+\tcreate_commits\n+'\n+\n+test_expect_success 'no redundant for pack 1, 2, 3' '\n+\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'create pack 4, 5' '\n+\tcreate_pack_4 && create_pack_5\n+'\n+\n+cat >expected <out &&\n+\tformat_packfiles actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'create pack 6, 7' '\n+\tcreate_pack_6 && create_pack_7\n+'\n+\n+# Only after calling create_pack_6, we can use $P6 variable.\n+cat >expected <out &&\n+\tformat_packfiles actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'create pack 8' '\n+\tcreate_pack_8\n+'\n+\n+cat >expected <out &&\n+\tformat_packfiles actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'clean loose objects' '\n+\tgit prune-packed &&\n+\tfind objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'remove redundant packs and pass fsck' '\n+\tgit pack-redundant --all | xargs rm &&\n+\tgit fsck --no-progress &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'setup shared.git' '\n+\tcd \"$TRASH_DIRECTORY\" &&\n+\tgit clone -q --mirror master.git shared.git &&\n+\tcd shared.git &&\n+\tprintf \"../../master.git/objects\" >objects/info/alternates\n+'\n+\n+test_expect_success 'no redundant packs without --alt-odb' '\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+cat >expected <out 2>out.err &&\n+\ttest_must_be_empty out &&\n+\tgrep \"pack$\" out.err | format_packfiles >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+cat >expected <actual 2>&1 &&\n+\ttest_cmp expected actual\n+'\n+\n+create_commits_others () {\n+\tparent=$(git rev-parse HEAD)\n+\tfor name in X Y Z\n+\tdo\n+\t\ttest_tick &&\n+\t\tT=$(git write-tree) &&\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\toid=$(echo $name | git commit-tree $T)\n+\t\telse\n+\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n+\t\tfi &&\n+\t\teval $name=$oid &&\n+\t\tparent=$oid ||\n+\t\treturn 1\n+\tdone\n+\tgit update-ref refs/heads/master $Z\n+}\n+\n+create_pack_x1 () {\n+\tPx1=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\tEOF\n+\t) &&\n+\teval P${Px1}=Px1:${Px1}\n+}\n+\n+create_pack_x2 () {\n+\tPx2=$(git -C objects/pack pack-objects -q pack <<-EOF\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\tEOF\n+\t) &&\n+\teval P${Px2}=Px2:${Px2}\n+}\n+\n+test_expect_success 'new objects and packs in shared.git' '\n+\tcreate_commits_others &&\n+\tcreate_pack_x1 &&\n+\tcreate_pack_x2 &&\n+\tgit pack-redundant --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'one pack is redundant' '\n+\tgit pack-redundant --all --alt-odb >out &&\n+\tformat_packfiles actual &&\n+\ttest_line_count = 1 actual\n+'\n+\n+cat >expected <out <<-EOF &&\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\tEOF\n+\tformat_packfiles actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_done\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368116","messageId":"20190130114736.30357-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 2/6] pack-redundant: delay creation of unique_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:32Z","receivedAt":"2019-01-30T11:48:36Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nInstead of initializing unique_objects in `add_pack()`, copy from\nall_objects in `cmp_two_packs()`, when unwanted objects are removed from\nall_objects.\n\nThis will save memory (no allocate memory for alt-odb packs), and run\n`llist_sorted_difference_inplace()` only once when removing ignored\nobjects and removing objects in alt-odb in `scan_alt_odb_packs()`.\n\nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..f7dab0ec60 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -254,6 +254,11 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \tstruct llist_item *p1_hint = NULL, *p2_hint = NULL;\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n+\tif (!p1->unique_objects)\n+\t\tp1->unique_objects = llist_copy(p1->all_objects);\n+\tif (!p2->unique_objects)\n+\t\tp2->unique_objects = llist_copy(p2->all_objects);\n+\n \tp1_base = p1->pack->index_data;\n \tp2_base = p2->pack->index_data;\n \tp1_base += 256 * 4 + ((p1->pack->index_version < 2) ? 4 : 8);\n@@ -536,7 +541,7 @@ static void scan_alt_odb_packs(void)\n \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n-\t\t\tllist_sorted_difference_inplace(local->unique_objects,\n+\t\t\tllist_sorted_difference_inplace(local->all_objects,\n \t\t\t\t\t\t\talt->all_objects);\n \t\t\tlocal = local->next;\n \t\t}\n@@ -567,8 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n-\t/* this list will be pruned in cmp_two_packs later */\n-\tl.unique_objects = llist_copy(l.all_objects);\n+\tl.unique_objects = NULL;\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\n \telse\n@@ -646,7 +650,6 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \n \tload_all_objects();\n \n-\tcmp_local_packs();\n \tif (alt_odb)\n \t\tscan_alt_odb_packs();\n \n@@ -663,10 +666,12 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n+\tcmp_local_packs();\n+\n \tminimize(&min);\n \n \tif (verbose) {\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368117","messageId":"20190130114736.30357-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 3/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:33Z","receivedAt":"2019-01-30T11:48:47Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nSome test cases will fail on Mac OS X. Mark them and will resolve in\nlater commit.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 108 +++++++++++++++++++++++---------------\n t/t5323-pack-redundant.sh | 12 ++---\n 2 files changed, 73 insertions(+), 47 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex f7dab0ec60..b3f3bb5e09 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -426,14 +426,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -451,49 +489,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -607,7 +633,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex 710fe9884c..d176a6eb91 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -155,7 +155,7 @@ cat >expected <out &&\n \tformat_packfiles actual &&\n \ttest_cmp expected actual\n@@ -172,7 +172,7 @@ P4:$P4\n P6:$P6\n EOF\n \n-test_expect_success 'pack 2, 4, and 6 are redundant' '\n+test_expect_failure 'pack 2, 4, and 6 are redundant' '\n \tgit pack-redundant --all >out &&\n \tformat_packfiles actual &&\n \ttest_cmp expected actual\n@@ -189,7 +189,7 @@ P6:$P6\n P8:$P8\n EOF\n \n-test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n+test_expect_failure 'pack-8 (subset of pack-1) is also redundant' '\n \tgit pack-redundant --all >out &&\n \tformat_packfiles actual &&\n \ttest_cmp expected actual\n@@ -201,7 +201,7 @@ test_expect_success 'clean loose objects' '\n \ttest_must_be_empty out\n '\n \n-test_expect_success 'remove redundant packs and pass fsck' '\n+test_expect_failure 'remove redundant packs and pass fsck' '\n \tgit pack-redundant --all | xargs rm &&\n \tgit fsck --no-progress &&\n \tgit pack-redundant --all >out &&\n@@ -215,7 +215,7 @@ test_expect_success 'setup shared.git' '\n \tprintf \"../../master.git/objects\" >objects/info/alternates\n '\n \n-test_expect_success 'no redundant packs without --alt-odb' '\n+test_expect_failure 'no redundant packs without --alt-odb' '\n \tgit pack-redundant --all >out &&\n \ttest_must_be_empty out\n '\n@@ -227,7 +227,7 @@ P5:$P5\n P7:$P7\n EOF\n \n-test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n+test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr' '\n \tgit pack-redundant --all --verbose >out 2>out.err &&\n \ttest_must_be_empty out &&\n \tgrep \"pack$\" out.err | format_packfiles >actual &&\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368118","messageId":"20190130114736.30357-5-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 4/6] pack-redundant: remove unused functions","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:34Z","receivedAt":"2019-01-30T11:48:55Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nRemove unused functions to find `min` packs, such as `get_permutations`,\n`pll_free`, etc.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 86 ----------------------------------------\n 1 file changed, 86 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex b3f3bb5e09..b83d55031a 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -35,11 +35,6 @@ static struct pack_list {\n \tstruct llist *all_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n-struct pll {\n-\tstruct pll *next;\n-\tstruct pack_list *pl;\n-};\n-\n static struct llist_item *free_nodes;\n \n static inline void llist_item_put(struct llist_item *item)\n@@ -63,15 +58,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -290,78 +276,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368119","messageId":"20190130114736.30357-6-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 5/6] pack-redundant: rename pack_list.all_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:35Z","receivedAt":"2019-01-30T11:49:05Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nNew algorithm uses `pack_list.all_objects` to track remaining objects,\nso rename it to `pack_list.remaining_objects`.\n\nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 40 ++++++++++++++++++++--------------------\n 1 file changed, 20 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex b83d55031a..c145852b8b 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -32,7 +32,7 @@ static struct pack_list {\n \tstruct pack_list *next;\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n-\tstruct llist *all_objects;\n+\tstruct llist *remaining_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -241,9 +241,9 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n \tif (!p1->unique_objects)\n-\t\tp1->unique_objects = llist_copy(p1->all_objects);\n+\t\tp1->unique_objects = llist_copy(p1->remaining_objects);\n \tif (!p2->unique_objects)\n-\t\tp2->unique_objects = llist_copy(p2->all_objects);\n+\t\tp2->unique_objects = llist_copy(p2->remaining_objects);\n \n \tp1_base = p1->pack->index_data;\n \tp2_base = p2->pack->index_data;\n@@ -344,8 +344,8 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->all_objects->size;\n-\tsize_t sz_b = pl_b->all_objects->size;\n+\tsize_t sz_a = pl_a->remaining_objects->size;\n+\tsize_t sz_b = pl_b->remaining_objects->size;\n \n \tif (sz_a == sz_b)\n \t\treturn 0;\n@@ -355,7 +355,7 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n \t\treturn -1;\n }\n \n-/* Sort pack_list, greater size of all_objects first */\n+/* Sort pack_list, greater size of remaining_objects first */\n static void sort_pack_list(struct pack_list **pl)\n {\n \tstruct pack_list **ary, *p;\n@@ -399,7 +399,7 @@ static void minimize(struct pack_list **min)\n \tmissing = llist_copy(all_objects);\n \tpl = unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(missing, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(missing, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n \n@@ -417,20 +417,20 @@ static void minimize(struct pack_list **min)\n \t/* remove unique pack objects from the non_unique packs */\n \tpl = non_unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n \n \twhile (non_unique) {\n-\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\t/* sort the non_unique packs, greater size of remaining_objects first */\n \t\tsort_pack_list(&non_unique);\n-\t\tif (non_unique->all_objects->size == 0)\n+\t\tif (non_unique->remaining_objects->size == 0)\n \t\t\tbreak;\n \n \t\tpack_list_insert(min, non_unique);\n \n-\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n-\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\t\tfor (pl = non_unique->next; pl && pl->remaining_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->remaining_objects, non_unique->remaining_objects);\n \n \t\tnon_unique = non_unique->next;\n \t}\n@@ -445,7 +445,7 @@ static void load_all_objects(void)\n \n \twhile (pl) {\n \t\thint = NULL;\n-\t\tl = pl->all_objects->front;\n+\t\tl = pl->remaining_objects->front;\n \t\twhile (l) {\n \t\t\thint = llist_insert_sorted_unique(all_objects,\n \t\t\t\t\t\t\t l->oid, hint);\n@@ -456,7 +456,7 @@ static void load_all_objects(void)\n \t/* remove objects present in remote packs */\n \tpl = altodb_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(all_objects, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n }\n@@ -481,11 +481,11 @@ static void scan_alt_odb_packs(void)\n \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n-\t\t\tllist_sorted_difference_inplace(local->all_objects,\n-\t\t\t\t\t\t\talt->all_objects);\n+\t\t\tllist_sorted_difference_inplace(local->remaining_objects,\n+\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n-\t\tllist_sorted_difference_inplace(all_objects, alt->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, alt->remaining_objects);\n \t\talt = alt->next;\n \t}\n }\n@@ -500,7 +500,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\treturn NULL;\n \n \tl.pack = p;\n-\tllist_init(&l.all_objects);\n+\tllist_init(&l.remaining_objects);\n \n \tif (open_pack_index(p))\n \t\treturn NULL;\n@@ -509,7 +509,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \tbase += 256 * 4 + ((p->index_version < 2) ? 4 : 8);\n \tstep = the_hash_algo->rawsz + ((p->index_version < 2) ? 4 : 0);\n \twhile (off < p->num_objects * step) {\n-\t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n+\t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n \tl.unique_objects = NULL;\n@@ -606,7 +606,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368120","messageId":"20190130114736.30357-7-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190112091754.30985-1-worldhello.net@gmail.com","subject":"[PATCH v7 6/6] pack-redundant: consistent sort method","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-01-30T11:47:36Z","receivedAt":"2019-01-30T11:49:12Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nSZEDER reported that test case t5323 has different test result on MacOS.\nThis is because `cmp_pack_list_reverse` cannot give identical result\nwhen two pack being sorted has the same size of remaining_objects.\n\nChanges to the sorting function will make consistent test result for\nt5323.\n\nThe new algorithm to find redundant packs is a trade-off to save memory\nresources, and the result of it may be different with old one, and may\nbe not the best result sometimes. Update t5323 for the new algorithm.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 24 ++++++++++++++++--------\n t/t5323-pack-redundant.sh | 14 +++++++-------\n 2 files changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex c145852b8b..0316a400ad 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -33,6 +33,7 @@ static struct pack_list {\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n \tstruct llist *remaining_objects;\n+\tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -340,19 +341,25 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n-static int cmp_pack_list_reverse(const void *a, const void *b)\n+static int cmp_remaining_objects(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->remaining_objects->size;\n-\tsize_t sz_b = pl_b->remaining_objects->size;\n \n-\tif (sz_a == sz_b)\n-\t\treturn 0;\n-\telse if (sz_a < sz_b)\n+\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size) {\n+\t\t/* have the same remaining_objects, big pack first */\n+\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n+\t\t\treturn 0;\n+\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n+\t\t\treturn 1;\n+\t\telse\n+\t\t\treturn -1;\n+\t} else if (pl_a->remaining_objects->size < pl_b->remaining_objects->size) {\n+\t\t/* sort by remaining objects, more objects first */\n \t\treturn 1;\n-\telse\n+\t} else {\n \t\treturn -1;\n+\t}\n }\n \n /* Sort pack_list, greater size of remaining_objects first */\n@@ -370,7 +377,7 @@ static void sort_pack_list(struct pack_list **pl)\n \tfor (n = 0, p = *pl; p; p = p->next)\n \t\tary[n++] = p;\n \n-\tQSORT(ary, n, cmp_pack_list_reverse);\n+\tQSORT(ary, n, cmp_remaining_objects);\n \n \t/* link them back again */\n \tfor (i = 0; i < n - 1; i++)\n@@ -512,6 +519,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n+\tl.all_objects_size = l.remaining_objects->size;\n \tl.unique_objects = NULL;\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex d176a6eb91..4469128203 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -152,10 +152,10 @@ test_expect_success 'create pack 4, 5' '\n '\n \n cat >expected <out &&\n \tformat_packfiles actual &&\n \ttest_cmp expected actual\n@@ -172,7 +172,7 @@ P4:$P4\n P6:$P6\n EOF\n \n-test_expect_failure 'pack 2, 4, and 6 are redundant' '\n+test_expect_success 'pack 2, 4, and 6 are redundant' '\n \tgit pack-redundant --all >out &&\n \tformat_packfiles actual &&\n \ttest_cmp expected actual\n@@ -189,7 +189,7 @@ P6:$P6\n P8:$P8\n EOF\n \n-test_expect_failure 'pack-8 (subset of pack-1) is also redundant' '\n+test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n \tgit pack-redundant --all >out &&\n \tformat_packfiles actual &&\n \ttest_cmp expected actual\n@@ -201,7 +201,7 @@ test_expect_success 'clean loose objects' '\n \ttest_must_be_empty out\n '\n \n-test_expect_failure 'remove redundant packs and pass fsck' '\n+test_expect_success 'remove redundant packs and pass fsck' '\n \tgit pack-redundant --all | xargs rm &&\n \tgit fsck --no-progress &&\n \tgit pack-redundant --all >out &&\n@@ -215,7 +215,7 @@ test_expect_success 'setup shared.git' '\n \tprintf \"../../master.git/objects\" >objects/info/alternates\n '\n \n-test_expect_failure 'no redundant packs without --alt-odb' '\n+test_expect_success 'no redundant packs without --alt-odb' '\n \tgit pack-redundant --all >out &&\n \ttest_must_be_empty out\n '\n@@ -227,7 +227,7 @@ P5:$P5\n P7:$P7\n EOF\n \n-test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr' '\n+test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n \tgit pack-redundant --all --verbose >out 2>out.err &&\n \ttest_must_be_empty out &&\n \tgrep \"pack$\" out.err | format_packfiles >actual &&\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368138","messageId":"20190130150334.20598-1-16657101987@163.com","threadId":"50055","inReplyTo":"20190130114736.30357-5-worldhello.net@gmail.com","subject":"[PATCH v8 1/1] pack-redundant: delete redundant code","fromName":"","fromEmail":"16657101987@163.com","sentAt":"2019-01-30T15:03:34Z","receivedAt":"2019-01-30T15:04:10Z","isPatch":true,"sender":{"key":"16657101987@163.com","avatar":"https://avatars.githubusercontent.com/u/192864724?v=4"},"body":"From: Sun Chao \n\nThe objects in alt-odb are removed from `all_objects` twice in `load_all_objects`\nand `scan_alt_odb_packs`, remove it from the later function.\n\nSigned-off-by: Sun Chao \n---\n builtin/pack-redundant.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 0316a400ad..29ff5e99cb 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -492,7 +492,6 @@ static void scan_alt_odb_packs(void)\n \t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n-\t\tllist_sorted_difference_inplace(all_objects, alt->remaining_objects);\n \t\talt = alt->next;\n \t}\n }\n-- \n2.20.1\n\n\n"},{"id":"368260","messageId":"xmqqva241v6d.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190130114736.30357-4-worldhello.net@gmail.com","subject":"Re: [PATCH v7 3/6] pack-redundant: new algorithm to find min packs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-31T19:30:50Z","receivedAt":"2019-01-31T19:30:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jiang Xin writes:\n\n> From: Sun Chao \n>\n> When calling `git pack-redundant --all`, if there are too many local\n> packs and too many redundant objects within them, the too deep iteration\n> of `get_permutations` will exhaust all the resources, and the process of\n> `git pack-redundant` will be killed.\n\nBuild each step of the series with \"make DEVELOPER=YesPlease\" and\nyour compiler hopefully would notice...\n\n CC builtin/pack-redundant.o\nbuiltin/pack-redundant.c:347:12: error: 'is_superset' defined but not used [-Werror=unused-function]\n static int is_superset(struct pack_list *pl, struct llist *list)\n ^~~~~~~~~~~\nbuiltin/pack-redundant.c:313:21: error: 'get_permutations' defined but not used [-Werror=unused-function]\n static struct pll * get_permutations(struct pack_list *list, int n)\n ^~~~~~~~~~~~~~~~\nbuiltin/pack-redundant.c:293:13: error: 'pll_free' defined but not used [-Werror=unused-function]\n static void pll_free(struct pll *l)\n ^~~~~~~~\ncc1: all warnings being treated as errors\nMakefile:2302: recipe for target 'builtin/pack-redundant.o' failed\nmake: *** [builtin/pack-redundant.o] Error 1\n\n"},{"id":"368274","messageId":"xmqq7eek1ozm.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"20190130114736.30357-2-worldhello.net@gmail.com","subject":"Re: [PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-31T21:44:29Z","receivedAt":"2019-01-31T21:44:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jiang Xin writes:\n\n> diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n> new file mode 100755\n> index 0000000000..710fe9884c\n> --- /dev/null\n> +++ b/t/t5323-pack-redundant.sh\n> @@ -0,0 +1,322 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2018 Jiang Xin\n> +#\n> +\n> +test_description='git pack-redundant test'\n> +\n> +. ./test-lib.sh\n> +\n> +create_commits () {\n> +\tparent=\n> +\tfor name in A B C D E F G H I J K L M N O P Q R\n> +\tdo\n> +\t\ttest_tick &&\n> +\t\tT=$(git write-tree) &&\n\nMove this outside loop, not for efficiency but for clarity. This\nhelper function creates a single empty tree and bunch of commits\nthat hold the same empty tree, arranged as a single strand of\npearls.\n\nBy the way, I had to draw a table like this to figure out ...\n\n T A B C D E F G H I J K L M N O P Q R\n1 x x x x x x x x\n2 x x x x x x x\n3 x x x x x x\n4 x x x x x\n5 x x x x\n6 x x x\n7 x x\n8 x\n\n... what is going on. Perhaps something like this would help other\nreaders near the top of the file (or in test_description)?\n\n\n> +format_packfiles () {\n> +\tsed \\\n> +\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n> +\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" |\n> +\tsort -u |\n> +\twhile read p\n> +\tdo\n> +\t\tif test -z \"$(eval echo \\${P$p})\"\n> +\t\tthen\n> +\t\t\techo $p\n\nAll the \"expected output\" below will expect P$n:${P$n} prepared by\nvarious create_pack_$n helpers we saw earlier, so an unknown\npackfile would be detected as a line that this emits. Is that the\nidea?\n\n> +\t\telse\n> +\t\t\teval echo \"\\${P$p}\"\n> +\t\tfi\n> +\tdone |\n> +\tsort\n> +}\n> +\n> +test_expect_success 'setup master.git' '\n> +\tgit init --bare master.git &&\n> +\tcd master.git &&\n> +\tcreate_commits\n> +'\n\nEverything below will be done inside master.git? Avoid cd'ing\naround in random places in the test script, as a failure in any of\nthe steps that does cd would start later tests in an unexpected\nplace, if you can.\n\n> +test_expect_success 'no redundant for pack 1, 2, 3' '\n> +\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n> +\tgit pack-redundant --all >out &&\n> +\ttest_must_be_empty out\n> +'\n> +\n> +test_expect_success 'create pack 4, 5' '\n> +\tcreate_pack_4 && create_pack_5\n> +'\n> +\n\n> +cat >expected < +P2:$P2\n> +EOF\n> +\n> +test_expect_success 'one of pack-2/pack-3 is redundant' '\n> +\tgit pack-redundant --all >out &&\n> +\tformat_packfiles actual &&\n> +\ttest_cmp expected actual\n> +'\n\nDo the preparation of file \"expect\" (most of the tests compare\n'expect' vs 'actual', not 'expected') _inside_ the next test that\nuses it. i.e.\n\n\ttest_expect_success 'with 1 4 and 5, either 2 or 3 can be omitted' '\n\t\tcat >expect <<-EOF &&\n\t\tP2:$P2\n\t\tEOF\n\t\tgit pack-redundant --all >out &&\n\t\tformat ... >actual &&\n\t\ttest_cmp expect actual\n\t'\n\nAgain, I needed to draw this to see if the \"one of ... is redundant\"\nin the title is a valid claim. Something like it would help future\nreaders.\n\n T A B C D E F G H I J K L M N O P Q R\n1245 x x x x x x x x x x x x x x x x x\n3 x x x x x x\n\n T A B C D E F G H I J K L M N O P Q R\n1345 x x x x x x x x x x x x x x x x x\n2 x x x x x x x\n\nI won't repeat the same for tests that appear later in this file,\nbut they share the same issue.\n\n> +test_expect_success 'setup shared.git' '\n> +\tcd \"$TRASH_DIRECTORY\" &&\n> +\tgit clone -q --mirror master.git shared.git &&\n\nWhy \"-q\"?\n\n> +\tcd shared.git &&\n> +\tprintf \"../../master.git/objects\" >objects/info/alternates\n> +'\n\nWhy not echo? I recall designing the alternates file to be a plain\ntext file. Is it necessary to leave the line incomplete?\n\n> +test_expect_success 'remove redundant packs by alt-odb, no packs left' '\n> +\tgit pack-redundant --all --alt-odb | xargs rm &&\n> +\tgit fsck --no-progress &&\n\nWhy \"--no-progress\"?\n\n> +\ttest_must_fail git pack-redundant --all --alt-odb >actual 2>&1 &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +create_commits_others () {\n> +\tparent=$(git rev-parse HEAD)\n\nIf this fails, you'd still go ahead and enter the loop, which is not\nwhat you want.\n\n> +\tfor name in X Y Z\n> +\tdo\n> +\t\ttest_tick &&\n> +\t\tT=$(git write-tree) &&\n\nLift this outside the loop.\n\n> +\t\tif test -z \"$parent\"\n> +\t\tthen\n> +\t\t\toid=$(echo $name | git commit-tree $T)\n> +\t\telse\n> +\t\t\toid=$(echo $name | git commit-tree -p $parent $T)\n> +\t\tfi &&\n> +\t\teval $name=$oid &&\n> +\t\tparent=$oid ||\n> +\t\treturn 1\n> +\tdone\n> +\tgit update-ref refs/heads/master $Z\n> +}\n"},{"id":"368297","messageId":"CANYiYbG5vbV1bqbHu-u19XFy7JTpcopg0s=JWAhJBRNGvP7j3g@mail.gmail.com","threadId":"50055","inReplyTo":"xmqq7eek1ozm.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T05:44:16Z","receivedAt":"2019-02-01T05:44:31Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Junio C Hamano 于2019年2月1日周五 上午5:44写道：\n> > +create_commits () {\n> > + parent=\n> > + for name in A B C D E F G H I J K L M N O P Q R\n> > + do\n> > + test_tick &&\n> > + T=$(git write-tree) &&\n>\n> Move this outside loop, not for efficiency but for clarity. This\n> helper function creates a single empty tree and bunch of commits\n> that hold the same empty tree, arranged as a single strand of\n> pearls.\n\nWill rewrite as:\n\n create_commits () {\n parent=\n T=$(git write-tree) &&\n for name in A B C D E F G H I J K L M N O P Q R\n\n>\n> By the way, I had to draw a table like this to figure out ...\n>\n> T A B C D E F G H I J K L M N O P Q R\n> 1 x x x x x x x x\n> 2 x x x x x x x\n> 3 x x x x x x\n> 4 x x x x x\n> 5 x x x x\n> 6 x x x\n> 7 x x\n> 8 x\n>\n> ... what is going on. Perhaps something like this would help other\n> readers near the top of the file (or in test_description)?\n\nNice chart, will edit test_description as follows:\n\n test_description='git pack-redundant test\n\n In order to test git-pack-redundant, we will create a number of\nredundant\n packs in the repository `master.git`. The relationship between\npacks (P1-P8)\n and objects (T,A-R) is show in the following chart:\n\n | T A B C D E F G H I J K L M N O P Q R\n ---+--------------------------------------\n P1 | x x x x x x x x\n P2 | x x x x x x x\n P3 | x x x x x x\n P4 | x x x x x\n P5 | x x x x\n P6 | x x x\n P7 | x x\n P8 | x\n\n Another repoisitory `shared.git` has unique objects (X-Z), while\nshare others\n objects through alt-odb (of `master.git`). The relationship\nbetween packs\n and objects is as follows:\n\n | T A B C D E F G H I J K L M N O P Q R X Y Z\n ---+----------------------------------------------\n Px1| x x x x x x\n Px2| x x x x x x\n '\n\n>\n>\n> > +format_packfiles () {\n> > + sed \\\n> > + -e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n> > + -e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" |\n> > + sort -u |\n> > + while read p\n> > + do\n> > + if test -z \"$(eval echo \\${P$p})\"\n> > + then\n> > + echo $p\n>\n> All the \"expected output\" below will expect P$n:${P$n} prepared by\n> various create_pack_$n helpers we saw earlier, so an unknown\n> packfile would be detected as a line that this emits. Is that the\n> idea?\n\nRight. During the reroll, a typo makes an empty output, so I decide\nto make this change.\n\n\n> > + else\n> > + eval echo \"\\${P$p}\"\n> > + fi\n> > + done |\n> > + sort\n> > +}\n> > +\n> > +test_expect_success 'setup master.git' '\n> > + git init --bare master.git &&\n> > + cd master.git &&\n> > + create_commits\n> > +'\n>\n> Everything below will be done inside master.git? Avoid cd'ing\n> around in random places in the test script, as a failure in any of\n> the steps that does cd would start later tests in an unexpected\n> place, if you can.\n\nThe first 10 test cases will run inside master.git, and others will\nrun inside shared.git. Only run cd inside the two `setup` test cases.\n\n> > +cat >expected < > +P2:$P2\n> > +EOF\n> > +\n> > +test_expect_success 'one of pack-2/pack-3 is redundant' '\n> > + git pack-redundant --all >out &&\n> > + format_packfiles actual &&\n> > + test_cmp expected actual\n> > +'\n>\n> Do the preparation of file \"expect\" (most of the tests compare\n> 'expect' vs 'actual', not 'expected') _inside_ the next test that\n> uses it. i.e.\n>\n> test_expect_success 'with 1 4 and 5, either 2 or 3 can be omitted' '\n> cat >expect <<-EOF &&\n> P2:$P2\n> EOF\n> git pack-redundant --all >out &&\n> format ... >actual &&\n> test_cmp expect actual\n> '\n\nWill do.\n\n> > +test_expect_success 'setup shared.git' '\n> > + cd \"$TRASH_DIRECTORY\" &&\n> > + git clone -q --mirror master.git shared.git &&\n>\n> Why \"-q\"?\n\nTo make verbose output cleaner.\n\n> > + cd shared.git &&\n> > + printf \"../../master.git/objects\" >objects/info/alternates\n> > +'\n>\n> Why not echo? I recall designing the alternates file to be a plain\n> text file. Is it necessary to leave the line incomplete?\n\nForgot \"\\n\", will append.\n\n>\n> > +test_expect_success 'remove redundant packs by alt-odb, no packs left' '\n> > + git pack-redundant --all --alt-odb | xargs rm &&\n> > + git fsck --no-progress &&\n>\n> Why \"--no-progress\"?\n\nTo make verbose output cleaner.\n\n>\n> > + test_must_fail git pack-redundant --all --alt-odb >actual 2>&1 &&\n> > + test_cmp expected actual\n> > +'\n> > +\n> > +create_commits_others () {\n> > + parent=$(git rev-parse HEAD)\n\nWill append \"&&\".\n"},{"id":"368299","messageId":"CAPig+cS84BOPdrZ+e1Z9ystx6SDyON0MJxVySQyxYE=LYcNRdg@mail.gmail.com","threadId":"50055","inReplyTo":"CANYiYbG5vbV1bqbHu-u19XFy7JTpcopg0s=JWAhJBRNGvP7j3g@mail.gmail.com","subject":"Re: [PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-02-01T06:11:32Z","receivedAt":"2019-02-01T06:13:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Feb 1, 2019 at 12:44 AM Jiang Xin wrote:\n>> Junio C Hamano 于2019年2月1日周五 上午5:44写道：\n> > Move this outside loop, not for efficiency but for clarity. This\n> > helper function creates a single empty tree and bunch of commits\n> > that hold the same empty tree, arranged as a single strand of\n> > pearls.\n>\n> Will rewrite as:\n>\n> create_commits () {\n> parent=\n> T=$(git write-tree) &&\n> for name in A B C D E F G H I J K L M N O P Q R\n\nDon't forget the && at the end of the 'parent=' line to protect\nagainst someone later adding code above that line. So:\n\n create_commits () {\n parent= &&\n T=$(git write-tree) &&\n ...\n\n> Nice chart, will edit test_description as follows:\n>\n> test_description='git pack-redundant test\n>\n> In order to test git-pack-redundant, we will create a number of\n> redundant\n> packs in the repository `master.git`. The relationship between\n> packs (P1-P8)\n> and objects (T,A-R) is show in the following chart:\n>\n> | T A B C D E F G H I J K L M N O P Q R\n> ---+--------------------------------------\n> P1 | x x x x x x x x\n> P2 | x x x x x x x\n> P3 | x x x x x x\n> P4 | x x x x x\n> P5 | x x x x\n> P6 | x x x\n> P7 | x x\n> P8 | x\n\ntest_description should be a meaningful one-liner; it should not\ncontain this other information, but this information should appear as\ncomments in the test script.\n\n> Another repoisitory `shared.git` has unique objects (X-Z), while\n> share others\n\ns/repoisitory/repository/\n\n> > > +test_expect_success 'setup master.git' '\n> > > + git init --bare master.git &&\n> > > + cd master.git &&\n> > > + create_commits\n> > > +'\n> >\n> > Everything below will be done inside master.git? Avoid cd'ing\n> > around in random places in the test script, as a failure in any of\n> > the steps that does cd would start later tests in an unexpected\n> > place, if you can.\n>\n> The first 10 test cases will run inside master.git, and others will\n> run inside shared.git. Only run cd inside the two `setup` test cases.\n\nThat's not what Junio meant. It's okay for tests to 'cd', but each\ntest which does so _must_ ensure that the 'cd' is undone at the end of\nthe test, even if the test fails. The correct way to do this within\neach test is by using 'cd' in a subhsell, like this:\n\n test_expect_success 'setup master.git' '\n git init --bare master.git &&\n (\n cd master.git &&\n create_commits\n )\n '\n\nThen, each test which needs to use \"master.git\" would 'cd' itself, like this:\n\n test_expect_success 'some test' '\n (\n cd master.git &&\n ...\n )\n '\n\n> > > +test_expect_success 'setup shared.git' '\n> > > + cd \"$TRASH_DIRECTORY\" &&\n> > > + git clone -q --mirror master.git shared.git &&\n> >\n> > Why \"-q\"?\n>\n> To make verbose output cleaner.\n\nWhat Junio really meant by asking that question was that you should\nnot do this. When something goes wrong with a test, we want as much\noutput as possible to help diagnose the problem, so suppressing output\nis undesirable. To summarize, don't use -q, --no-progress, or any\nother such option and don't redirect to /dev/null.\n"},{"id":"368304","messageId":"CANYiYbHK9Qozqzpnwu3y9Gq=ieMPUgXfZxr8Ofxwe6mJox9Ohg@mail.gmail.com","threadId":"50055","inReplyTo":"CAPig+cS84BOPdrZ+e1Z9ystx6SDyON0MJxVySQyxYE=LYcNRdg@mail.gmail.com","subject":"Re: [PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T07:23:46Z","receivedAt":"2019-02-01T07:24:01Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Eric Sunshine 于2019年2月1日周五 下午2:11写道：\n>\n> On Fri, Feb 1, 2019 at 12:44 AM Jiang Xin wrote:\n> >> Junio C Hamano 于2019年2月1日周五 上午5:44写道：\n> > > Move this outside loop, not for efficiency but for clarity. This\n> > > helper function creates a single empty tree and bunch of commits\n> > > that hold the same empty tree, arranged as a single strand of\n> > > pearls.\n> >\n> > Will rewrite as:\n> >\n> > create_commits () {\n> > parent=\n> > T=$(git write-tree) &&\n> > for name in A B C D E F G H I J K L M N O P Q R\n>\n> Don't forget the && at the end of the 'parent=' line to protect\n> against someone later adding code above that line. So:\n>\n> create_commits () {\n> parent= &&\n> T=$(git write-tree) &&\n> ...\n\nWill do.\n\n> > Nice chart, will edit test_description as follows:\n> >\n> > test_description='git pack-redundant test\n> >\n> > In order to test git-pack-redundant, we will create a number of\n> > redundant\n> > packs in the repository `master.git`. The relationship between\n> > packs (P1-P8)\n> > and objects (T,A-R) is show in the following chart:\n> >\n> > | T A B C D E F G H I J K L M N O P Q R\n> > ---+--------------------------------------\n> > P1 | x x x x x x x x\n> > P2 | x x x x x x x\n> > P3 | x x x x x x\n> > P4 | x x x x x\n> > P5 | x x x x\n> > P6 | x x x\n> > P7 | x x\n> > P8 | x\n>\n> test_description should be a meaningful one-liner; it should not\n> contain this other information, but this information should appear as\n> comments in the test script.\n\nIn 't/t0000-basic.sh', there is also a very long test_description.\nAfter read 't/test-lib.sh', the only usage of test_description\nis showing it as help, when runing:\n\n sh ./t0000-basic.sh\n\nSo write a long test_description is ok, I think.\n\n> > Another repoisitory `shared.git` has unique objects (X-Z), while\n> > share others\n>\n> s/repoisitory/repository/\n\nThanks, will fix.\n\n> > > > +test_expect_success 'setup master.git' '\n> > > > + git init --bare master.git &&\n> > > > + cd master.git &&\n> > > > + create_commits\n> > > > +'\n> > >\n> > > Everything below will be done inside master.git? Avoid cd'ing\n> > > around in random places in the test script, as a failure in any of\n> > > the steps that does cd would start later tests in an unexpected\n> > > place, if you can.\n> >\n> > The first 10 test cases will run inside master.git, and others will\n> > run inside shared.git. Only run cd inside the two `setup` test cases.\n>\n> That's not what Junio meant. It's okay for tests to 'cd', but each\n> test which does so _must_ ensure that the 'cd' is undone at the end of\n> the test, even if the test fails. The correct way to do this within\n> each test is by using 'cd' in a subhsell, like this:\n>\n> test_expect_success 'setup master.git' '\n> git init --bare master.git &&\n> (\n> cd master.git &&\n> create_commits\n> )\n> '\n>\n> Then, each test which needs to use \"master.git\" would 'cd' itself, like this:\n>\n> test_expect_success 'some test' '\n> (\n> cd master.git &&\n> ...\n> )\n> '\n\nNice explaination, will do.\n\n> > > > +test_expect_success 'setup shared.git' '\n> > > > + cd \"$TRASH_DIRECTORY\" &&\n> > > > + git clone -q --mirror master.git shared.git &&\n> > >\n> > > Why \"-q\"?\n> >\n> > To make verbose output cleaner.\n>\n> What Junio really meant by asking that question was that you should\n> not do this. When something goes wrong with a test, we want as much\n> output as possible to help diagnose the problem, so suppressing output\n> is undesirable. To summarize, don't use -q, --no-progress, or any\n> other such option and don't redirect to /dev/null.\n\nThanks.\n"},{"id":"368305","messageId":"CANYiYbGi+Hk6Hk=_ooQtPV=wrdHTJpzocctE-yEt4N8KKkvdGQ@mail.gmail.com","threadId":"50055","inReplyTo":"CANYiYbHK9Qozqzpnwu3y9Gq=ieMPUgXfZxr8Ofxwe6mJox9Ohg@mail.gmail.com","subject":"Re: [PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T07:25:50Z","receivedAt":"2019-02-01T07:26:05Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Jiang Xin 于2019年2月1日周五 下午3:23写道：\n>\n> Eric Sunshine 于2019年2月1日周五 下午2:11写道：\n> > > Nice chart, will edit test_description as follows:\n> > >\n> > > test_description='git pack-redundant test\n> > >\n> > > In order to test git-pack-redundant, we will create a number of\n> > > redundant\n> > > packs in the repository `master.git`. The relationship between\n> > > packs (P1-P8)\n> > > and objects (T,A-R) is show in the following chart:\n> > >\n> > > | T A B C D E F G H I J K L M N O P Q R\n> > > ---+--------------------------------------\n> > > P1 | x x x x x x x x\n> > > P2 | x x x x x x x\n> > > P3 | x x x x x x\n> > > P4 | x x x x x\n> > > P5 | x x x x\n> > > P6 | x x x\n> > > P7 | x x\n> > > P8 | x\n> >\n> > test_description should be a meaningful one-liner; it should not\n> > contain this other information, but this information should appear as\n> > comments in the test script.\n>\n> In 't/t0000-basic.sh', there is also a very long test_description.\n> After read 't/test-lib.sh', the only usage of test_description\n> is showing it as help, when runing:\n>\n> sh ./t0000-basic.sh\n\nsh ./t0000-basic.sh --help\n"},{"id":"368318","messageId":"CANYiYbGtXnhLdFCHFfQxT=tWyXGaq1gfuKuhxe1D7crXo6LSHQ@mail.gmail.com","threadId":"50055","inReplyTo":"CAPig+cS84BOPdrZ+e1Z9ystx6SDyON0MJxVySQyxYE=LYcNRdg@mail.gmail.com","subject":"Re: [PATCH v7 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T09:51:38Z","receivedAt":"2019-02-01T09:51:53Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Eric Sunshine 于2019年2月1日周五 下午2:11写道：\n> > Everything below will be done inside master.git? Avoid cd'ing\n> > > around in random places in the test script, as a failure in any of\n> > > the steps that does cd would start later tests in an unexpected\n> > > place, if you can.\n> >\n> > The first 10 test cases will run inside master.git, and others will\n> > run inside shared.git. Only run cd inside the two `setup` test cases.\n>\n> That's not what Junio meant. It's okay for tests to 'cd', but each\n> test which does so _must_ ensure that the 'cd' is undone at the end of\n> the test, even if the test fails. The correct way to do this within\n> each test is by using 'cd' in a subhsell, like this:\n>\n> test_expect_success 'setup master.git' '\n> git init --bare master.git &&\n> (\n> cd master.git &&\n> create_commits\n> )\n> '\n\ncreate_commits should not run in sub-shell, or variables set are lost.\nI write a commit_commits_in function :\n # Usage: create_commits_in A B C ...\n # Note: DO NOT run it in sub shell, or variables are not set\n create_commits_in () {\n repo=\"$1\" &&\n parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null) || parent=\n T=$(git -C \"$repo\" write-tree) &&\n shift &&\n while test $# -gt 0\n do\n name=$1 &&\n test_tick &&\n if test -z \"$parent\"\n then\n oid=$(echo $name | git -C \"$repo\" commit-tree $T)\n else\n oid=$(echo $name | git -C \"$repo\"\ncommit-tree -p $parent $T)\n fi &&\n eval $name=$oid &&\n parent=$oid &&\n shift ||\n return 1\n done\n git -C \"$repo\" update-ref refs/heads/master $oid\n }\n\nand use it to create commits like:\n\n create_commits_in master.git A B C D E F G ...\n"},{"id":"368319","messageId":"CANYiYbGpz57J9wY6+O17jgYw7rwnEr7=X73+_O61J_Cp+fVRyg@mail.gmail.com","threadId":"50055","inReplyTo":"xmqqva241v6d.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v7 3/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T09:55:08Z","receivedAt":"2019-02-01T09:55:22Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Junio C Hamano 于2019年2月1日周五 上午3:30写道：\n>\n> Jiang Xin writes:\n>\n> > From: Sun Chao \n> >\n> > When calling `git pack-redundant --all`, if there are too many local\n> > packs and too many redundant objects within them, the too deep iteration\n> > of `get_permutations` will exhaust all the resources, and the process of\n> > `git pack-redundant` will be killed.\n>\n> Build each step of the series with \"make DEVELOPER=YesPlease\" and\n> your compiler hopefully would notice...\n>\n> CC builtin/pack-redundant.o\n> builtin/pack-redundant.c:347:12: error: 'is_superset' defined but not used [-Werror=unused-function]\n> static int is_superset(struct pack_list *pl, struct llist *list)\n> ^~~~~~~~~~~\n> builtin/pack-redundant.c:313:21: error: 'get_permutations' defined but not used [-Werror=unused-function]\n> static struct pll * get_permutations(struct pack_list *list, int n)\n> ^~~~~~~~~~~~~~~~\n> builtin/pack-redundant.c:293:13: error: 'pll_free' defined but not used [-Werror=unused-function]\n> static void pll_free(struct pll *l)\n> ^~~~~~~~\n> cc1: all warnings being treated as errors\n> Makefile:2302: recipe for target 'builtin/pack-redundant.o' failed\n> make: *** [builtin/pack-redundant.o] Error 1\n>\n\nThese unused code are deleted in another, and will be squashed to\nprevious patch in next reroll.\n"},{"id":"368336","messageId":"20190201162152.31136-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 0/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:46Z","receivedAt":"2019-02-01T16:22:15Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Sun Chao (my former colleague at Huawei) found a bug of\ngit-pack-redundant. If there are too many packs and many of them\noverlap each other, running `git pack-redundant --all` will\nexhaust all memories and the process will be killed by kernel.\n\nThere is a script in commit log of commit 3/6, which can be used to\ncreate a repository with lots of redundant packs. Running `git\npack-redundant --all` in it can reproduce this issue.\n\n## Changes since reroll v7\n\n1. Rewrite [PATCH v9 1/6] (t5323: test cases for git-pack-redundant)\n\n * Add many tables for relationship of packs and objects.\n * Change dir in subshell and fixed other issues.\n\n2. New patch file from Sun Chao: [PATCH v9 3/6] (pack-redundant: delete redundant code)\n\n3. Squash patches (remove unused functions) to patch 4/6 (new algorithm to find min packs).\n\n## Range diff\n\n1: 799e804d5e < -: ---------- t5323: test cases for git-pack-redundant\n-: ---------- > 1: c8dbf8cef2 t5323: test cases for git-pack-redundant\n2: 520f6277fb = 2: a6300516d7 pack-redundant: delay creation of unique_objects\n-: ---------- > 3: fb71973df5 pack-redundant: delete redundant code\n3: ab1c2c4950 ! 4: 9963d1c49f pack-redundant: new algorithm to find min packs\n @@ -76,6 +76,113 @@\n diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\n --- a/builtin/pack-redundant.c\n +++ b/builtin/pack-redundant.c\n +@@\n + \tstruct llist *all_objects;\n + } *local_packs = NULL, *altodb_packs = NULL;\n + \n +-struct pll {\n +-\tstruct pll *next;\n +-\tstruct pack_list *pl;\n +-};\n +-\n + static struct llist_item *free_nodes;\n + \n + static inline void llist_item_put(struct llist_item *item)\n +@@\n + \treturn new_item;\n + }\n + \n +-static void llist_free(struct llist *list)\n +-{\n +-\twhile ((list->back = list->front)) {\n +-\t\tlist->front = list->front->next;\n +-\t\tllist_item_put(list->back);\n +-\t}\n +-\tfree(list);\n +-}\n +-\n + static inline void llist_init(struct llist **list)\n + {\n + \t*list = xmalloc(sizeof(struct llist));\n +@@\n + \t}\n + }\n + \n +-static void pll_free(struct pll *l)\n +-{\n +-\tstruct pll *old;\n +-\tstruct pack_list *opl;\n +-\n +-\twhile (l) {\n +-\t\told = l;\n +-\t\twhile (l->pl) {\n +-\t\t\topl = l->pl;\n +-\t\t\tl->pl = opl->next;\n +-\t\t\tfree(opl);\n +-\t\t}\n +-\t\tl = l->next;\n +-\t\tfree(old);\n +-\t}\n +-}\n +-\n +-/* all the permutations have to be free()d at the same time,\n +- * since they refer to each other\n +- */\n +-static struct pll * get_permutations(struct pack_list *list, int n)\n +-{\n +-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n +-\n +-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n +-\t\treturn NULL;\n +-\n +-\tif (n == 1) {\n +-\t\twhile (list) {\n +-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n +-\t\t\tnew_pll->pl = NULL;\n +-\t\t\tpack_list_insert(&new_pll->pl, list);\n +-\t\t\tnew_pll->next = ret;\n +-\t\t\tret = new_pll;\n +-\t\t\tlist = list->next;\n +-\t\t}\n +-\t\treturn ret;\n +-\t}\n +-\n +-\twhile (list->next) {\n +-\t\tsubset = get_permutations(list->next, n - 1);\n +-\t\twhile (subset) {\n +-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n +-\t\t\tnew_pll->pl = subset->pl;\n +-\t\t\tpack_list_insert(&new_pll->pl, list);\n +-\t\t\tnew_pll->next = ret;\n +-\t\t\tret = new_pll;\n +-\t\t\tsubset = subset->next;\n +-\t\t}\n +-\t\tlist = list->next;\n +-\t}\n +-\treturn ret;\n +-}\n +-\n +-static int is_superset(struct pack_list *pl, struct llist *list)\n +-{\n +-\tstruct llist *diff;\n +-\n +-\tdiff = llist_copy(list);\n +-\n +-\twhile (pl) {\n +-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n +-\t\tif (diff->size == 0) { /* we're done */\n +-\t\t\tllist_free(diff);\n +-\t\t\treturn 1;\n +-\t\t}\n +-\t\tpl = pl->next;\n +-\t}\n +-\tllist_free(diff);\n +-\treturn 0;\n +-}\n +-\n + static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n + {\n + \tsize_t ret = 0;\n @@\n \treturn ret;\n }\n @@ -221,56 +328,56 @@\n --- a/t/t5323-pack-redundant.sh\n +++ b/t/t5323-pack-redundant.sh\n @@\n - P2:$P2\n - EOF\n - \n + # ALL | x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n -test_expect_success 'one of pack-2/pack-3 is redundant' '\n -+test_expect_failure 'one of pack-2/pack-3 is redundant' '\n - \tgit pack-redundant --all >out &&\n - \tformat_packfiles actual &&\n - \ttest_cmp expected actual\n ++test_expect_failure 'one of pack-2/pack-3 is redundant (failed on Mac)' '\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tcat >expect <<-EOF &&\n @@\n - P6:$P6\n - EOF\n - \n + # ALL | x x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n -test_expect_success 'pack 2, 4, and 6 are redundant' '\n -+test_expect_failure 'pack 2, 4, and 6 are redundant' '\n - \tgit pack-redundant --all >out &&\n - \tformat_packfiles actual &&\n - \ttest_cmp expected actual\n ++test_expect_failure 'pack 2, 4, and 6 are redundant (failed on Mac)' '\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tcat >expect <<-EOF &&\n @@\n - P8:$P8\n - EOF\n - \n + # ALL | x x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n -test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n -+test_expect_failure 'pack-8 (subset of pack-1) is also redundant' '\n - \tgit pack-redundant --all >out &&\n - \tformat_packfiles actual &&\n - \ttest_cmp expected actual\n ++test_expect_failure 'pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tcat >expect <<-EOF &&\n @@\n - \ttest_must_be_empty out\n + \t)\n '\n \n -test_expect_success 'remove redundant packs and pass fsck' '\n -+test_expect_failure 'remove redundant packs and pass fsck' '\n - \tgit pack-redundant --all | xargs rm &&\n - \tgit fsck --no-progress &&\n - \tgit pack-redundant --all >out &&\n ++test_expect_failure 'remove redundant packs and pass fsck (failed on Mac)' '\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tgit pack-redundant --all | xargs rm &&\n @@\n - \tprintf \"../../master.git/objects\" >objects/info/alternates\n + \t)\n '\n \n -test_expect_success 'no redundant packs without --alt-odb' '\n -+test_expect_failure 'no redundant packs without --alt-odb' '\n - \tgit pack-redundant --all >out &&\n - \ttest_must_be_empty out\n - '\n ++test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n + \t(\n + \t\tcd \"$shared_repo\" &&\n + \t\tgit pack-redundant --all >out &&\n @@\n - P7:$P7\n - EOF\n - \n + # ALL | x x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n -test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n -+test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr' '\n - \tgit pack-redundant --all --verbose >out 2>out.err &&\n - \ttest_must_be_empty out &&\n - \tgrep \"pack$\" out.err | format_packfiles >actual &&\n ++test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr (failed on Mac)' '\n + \t(\n + \t\tcd \"$shared_repo\" &&\n + \t\tcat >expect <<-EOF &&\n4: 3c3a7ea40f < -: ---------- pack-redundant: remove unused functions\n5: bc4b681f40 ! 5: b8f80ad454 pack-redundant: rename pack_list.all_objects\n @@ -115,11 +115,7 @@\n +\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n --\t\tllist_sorted_difference_inplace(all_objects, alt->all_objects);\n -+\t\tllist_sorted_difference_inplace(all_objects, alt->remaining_objects);\n \t\talt = alt->next;\n - \t}\n - }\n @@\n \t\treturn NULL;\n \n6: 6cfba5b4b2 ! 6: 8a12ad699e pack-redundant: consistent sort method\n @@ -83,60 +83,71 @@\n --- a/t/t5323-pack-redundant.sh\n +++ b/t/t5323-pack-redundant.sh\n @@\n - '\n - \n - cat >expected <out &&\n - \tformat_packfiles actual &&\n - \ttest_cmp expected actual\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tcat >expect <<-EOF &&\n +-\t\t\tP2:$P2\n ++\t\t\tP3:$P3\n + \t\t\tEOF\n + \t\tgit pack-redundant --all >out &&\n + \t\tformat_packfiles actual &&\n @@\n - P6:$P6\n - EOF\n - \n --test_expect_failure 'pack 2, 4, and 6 are redundant' '\n + # ALL | x x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n +-test_expect_failure 'pack 2, 4, and 6 are redundant (failed on Mac)' '\n +test_expect_success 'pack 2, 4, and 6 are redundant' '\n - \tgit pack-redundant --all >out &&\n - \tformat_packfiles actual &&\n - \ttest_cmp expected actual\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tcat >expect <<-EOF &&\n @@\n - P8:$P8\n - EOF\n - \n --test_expect_failure 'pack-8 (subset of pack-1) is also redundant' '\n + # ALL | x x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n +-test_expect_failure 'pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n +test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n - \tgit pack-redundant --all >out &&\n - \tformat_packfiles actual &&\n - \ttest_cmp expected actual\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tcat >expect <<-EOF &&\n @@\n - \ttest_must_be_empty out\n + \t)\n '\n \n --test_expect_failure 'remove redundant packs and pass fsck' '\n +-test_expect_failure 'remove redundant packs and pass fsck (failed on Mac)' '\n +test_expect_success 'remove redundant packs and pass fsck' '\n - \tgit pack-redundant --all | xargs rm &&\n - \tgit fsck --no-progress &&\n - \tgit pack-redundant --all >out &&\n + \t(\n + \t\tcd \"$master_repo\" &&\n + \t\tgit pack-redundant --all | xargs rm &&\n @@\n - \tprintf \"../../master.git/objects\" >objects/info/alternates\n + \t)\n '\n \n --test_expect_failure 'no redundant packs without --alt-odb' '\n +-test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n +test_expect_success 'no redundant packs without --alt-odb' '\n - \tgit pack-redundant --all >out &&\n - \ttest_must_be_empty out\n - '\n + \t(\n + \t\tcd \"$shared_repo\" &&\n + \t\tgit pack-redundant --all >out &&\n @@\n - P7:$P7\n - EOF\n - \n --test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr' '\n + # ALL | x x x x x x x x x x x x x x x x x x x\n + #\n + #############################################################################\n +-test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr (failed on Mac)' '\n +test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n - \tgit pack-redundant --all --verbose >out 2>out.err &&\n - \ttest_must_be_empty out &&\n - \tgrep \"pack$\" out.err | format_packfiles >actual &&\n + \t(\n + \t\tcd \"$shared_repo\" &&\n + \t\tcat >expect <<-EOF &&\n\n\nJiang Xin (4):\n t5323: test cases for git-pack-redundant\n pack-redundant: delay creation of unique_objects\n pack-redundant: rename pack_list.all_objects\n pack-redundant: consistent sort method\n\nSun Chao (2):\n pack-redundant: delete redundant code\n pack-redundant: new algorithm to find min packs\n\n builtin/pack-redundant.c | 232 +++++++----------\n t/t5323-pack-redundant.sh | 510 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 602 insertions(+), 140 deletions(-)\n create mode 100755 t/t5323-pack-redundant.sh\n\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368337","messageId":"20190201162152.31136-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:47Z","receivedAt":"2019-02-01T16:22:24Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \nReviewed-by: SZEDER Gábor \nHelped-by: Eric Sunshine \nReviewed-by: Sun Chao \nSigned-off-by: Junio C Hamano \n---\n t/t5323-pack-redundant.sh | 510 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 510 insertions(+)\n create mode 100755 t/t5323-pack-redundant.sh\n\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..d224ff3c50\n--- /dev/null\n+++ b/t/t5323-pack-redundant.sh\n@@ -0,0 +1,510 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='Test git pack-redundant\n+\n+In order to test git-pack-redundant, we will create a number of objects and\n+packs in the repository `master.git`. The relationship between packs (P1-P8)\n+and objects (T, A-R) is showed in the following chart. Objects of a pack will\n+be marked with letter x, while objects of redundant packs will be marked with\n+exclamation point, and redundant pack itself will be marked with asterisk.\n+\n+ | T A B C D E F G H I J K L M N O P Q R\n+ ----+--------------------------------------\n+ P1 | x x x x x x x x\n+ P2* | ! ! ! ! ! ! !\n+ P3 | x x x x x x\n+ P4* | ! ! ! ! !\n+ P5 | x x x x\n+ P6* | ! ! !\n+ P7 | x x\n+ P8* | !\n+ ----+--------------------------------------\n+ ALL | x x x x x x x x x x x x x x x x x x x\n+\n+Another repository `shared.git` has unique objects (X-Z), while other objects\n+(marked with letter s) are shared through alt-odb (of `master.git`). The\n+relationship between packs and objects is as follows:\n+\n+ | T A B C D E F G H I J K L M N O P Q R X Y Z\n+ ----+----------------------------------------------\n+ Px1 | s s s x x x\n+ Px2 | s s s x x x\n+'\n+\n+. ./test-lib.sh\n+\n+master_repo=master.git\n+shared_repo=shared.git\n+\n+# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n+# Usage: create_commits_in A B C ...\n+create_commits_in () {\n+\trepo=\"$1\" &&\n+\tparent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null) || parent=\n+\tT=$(git -C \"$repo\" write-tree) &&\n+\tshift &&\n+\twhile test $# -gt 0\n+\tdo\n+\t\tname=$1 &&\n+\t\ttest_tick &&\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\toid=$(echo $name | git -C \"$repo\" commit-tree $T)\n+\t\telse\n+\t\t\toid=$(echo $name | git -C \"$repo\" commit-tree -p $parent $T)\n+\t\tfi &&\n+\t\teval $name=$oid &&\n+\t\tparent=$oid &&\n+\t\tshift ||\n+\t\treturn 1\n+\tdone\n+\tgit -C \"$repo\" update-ref refs/heads/master $oid\n+}\n+\n+# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n+create_pack_1 () {\n+\tP1=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$T\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\t$R\n+\t\tEOF\n+\t) &&\n+\teval P$P1=P1:$P1\n+}\n+\n+create_pack_2 () {\n+\tP2=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$G\n+\t\t$H\n+\t\t$I\n+\t\tEOF\n+\t) &&\n+\teval P$P2=P2:$P2\n+}\n+\n+create_pack_3 () {\n+\tP3=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$F\n+\t\t$I\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\tEOF\n+\t) &&\n+\teval P$P3=P3:$P3\n+}\n+\n+create_pack_4 () {\n+\tP4=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\t$P\n+\t\tEOF\n+\t) &&\n+\teval P$P4=P4:$P4\n+}\n+\n+create_pack_5 () {\n+\tP5=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$G\n+\t\t$H\n+\t\t$N\n+\t\t$O\n+\t\tEOF\n+\t) &&\n+\teval P$P5=P5:$P5\n+}\n+\n+create_pack_6 () {\n+\tP6=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$N\n+\t\t$O\n+\t\t$Q\n+\t\tEOF\n+\t) &&\n+\teval P$P6=P6:$P6\n+}\n+\n+create_pack_7 () {\n+\tP7=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$P\n+\t\t$Q\n+\t\tEOF\n+\t) &&\n+\teval P$P7=P7:$P7\n+}\n+\n+create_pack_8 () {\n+\tP8=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$A\n+\t\tEOF\n+\t) &&\n+\teval P$P8=P8:$P8\n+}\n+\n+format_packfiles () {\n+\tsed \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" |\n+\tsort -u |\n+\twhile read p\n+\tdo\n+\t\tif test -z \"$(eval echo \\${P$p})\"\n+\t\tthen\n+\t\t\techo $p\n+\t\telse\n+\t\t\teval echo \"\\${P$p}\"\n+\t\tfi\n+\tdone |\n+\tsort\n+}\n+\n+test_expect_success 'setup master repo' '\n+\tgit init --bare \"$master_repo\" &&\n+\tcreate_commits_in \"$master_repo\" A B C D E F G H I J K L M N O P Q R\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2 | x x x x x x x\n+# P3 | x x x x x x\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'no redundant for pack 1, 2, 3' '\n+\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+test_expect_success 'create pack 4, 5' '\n+\tcreate_pack_4 && create_pack_5\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2* | ! ! ! ! ! ! !\n+# P3 | x x x x x x\n+# P4 | x x x x x\n+# P5 | x x x x\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'one of pack-2/pack-3 is redundant' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP2:$P2\n+\t\t\tEOF\n+\t\tgit pack-redundant --all >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'create pack 6, 7' '\n+\tcreate_pack_6 && create_pack_7\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2* | ! ! ! ! ! ! !\n+# P3 | x x x x x x\n+# P4* | ! ! ! ! !\n+# P5 | x x x x\n+# P6* | ! ! !\n+# P7 | x x\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'pack 2, 4, and 6 are redundant' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP2:$P2\n+\t\t\tP4:$P4\n+\t\t\tP6:$P6\n+\t\t\tEOF\n+\t\tgit pack-redundant --all >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'create pack 8' '\n+\tcreate_pack_8\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2* | ! ! ! ! ! ! !\n+# P3 | x x x x x x\n+# P4* | ! ! ! ! !\n+# P5 | x x x x\n+# P6* | ! ! !\n+# P7 | x x\n+# P8* | !\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP2:$P2\n+\t\t\tP4:$P4\n+\t\t\tP6:$P6\n+\t\t\tP8:$P8\n+\t\t\tEOF\n+\t\tgit pack-redundant --all >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'clean loose objects' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tgit prune-packed &&\n+\t\tfind objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+test_expect_success 'remove redundant packs and pass fsck' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tgit pack-redundant --all | xargs rm &&\n+\t\tgit fsck &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+# The following test cases will execute inside `shared.git`, instead of\n+# inside `master.git`.\n+test_expect_success 'setup shared.git' '\n+\tgit clone --mirror \"$master_repo\" \"$shared_repo\" &&\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tprintf \"../../$master_repo/objects\\n\" >objects/info/alternates\n+\t)\n+'\n+\n+test_expect_success 'no redundant packs without --alt-odb' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# ================ master.git ===============\n+# | T A B C D E F G H I J K L M N O P Q R <----------+\n+# ----+-------------------------------------- |\n+# P1 | x x x x x x x x |\n+# P3 | x x x x x x |\n+# P5 | x x x x |\n+# P7 | x x |\n+# ----+-------------------------------------- |\n+# ALL | x x x x x x x x x x x x x x x x x x x |\n+# |\n+# |\n+# ================ shared.git =============== |\n+# | T A B C D E F G H I J K L M N O P Q R \n+# ----+--------------------------------------\n+# P1* | s s s s s s s s\n+# P3* | s s s s s s\n+# P5* | s s s s\n+# P7* | s s\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP1:$P1\n+\t\t\tP3:$P3\n+\t\t\tP5:$P5\n+\t\t\tP7:$P7\n+\t\t\tEOF\n+\t\tgit pack-redundant --all --verbose >out 2>out.err &&\n+\t\ttest_must_be_empty out &&\n+\t\tgrep \"pack$\" out.err | format_packfiles >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'remove redundant packs by alt-odb, no packs left' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tfatal: Zero packs found!\n+\t\t\tEOF\n+\t\tgit pack-redundant --all --alt-odb | xargs rm &&\n+\t\tgit fsck &&\n+\t\ttest_must_fail git pack-redundant --all --alt-odb >actual 2>&1 &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+# Note: DO NOT run function `create_pack_*` in sub shell, or variables are not set\n+create_pack_x1_in () {\n+\trepo=\"$1\" &&\n+\tPx1=$(git -C \"$repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\tEOF\n+\t) &&\n+\teval P${Px1}=Px1:${Px1}\n+}\n+\n+create_pack_x2_in () {\n+\trepo=\"$1\" &&\n+\tPx2=$(git -C \"$repo/objects/pack\" pack-objects -q pack <<-EOF\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\tEOF\n+\t) &&\n+\teval P${Px2}=Px2:${Px2}\n+}\n+\n+test_expect_success 'create new objects and packs in shared.git' '\n+\tcreate_commits_in \"$shared_repo\" X Y Z &&\n+\tcreate_pack_x1_in \"$shared_repo\" &&\n+\tcreate_pack_x2_in \"$shared_repo\"\n+'\n+\n+test_expect_success 'no redundant without --alt-odb' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# ================ master.git ===============\n+# | T A B C D E F G H I J K L M N O P Q R <----------------+\n+# ----+-------------------------------------- |\n+# P1 | x x x x x x x x |\n+# P3 | x x x x x x |\n+# P5 | x x x x |\n+# P7 | x x |\n+# ----+-------------------------------------- |\n+# ALL | x x x x x x x x x x x x x x x x x x x |\n+# |\n+# |\n+# ================ shared.git ======================= |\n+# | T A B C D E F G H I J K L M N O P Q R X Y Z \n+# ----+----------------------------------------------\n+# Px1 | s s s x x x\n+# Px2*| s s s ! ! !\n+# ----+----------------------------------------------\n+# ALL | s s s s s s s s s s s s s s s s s s s x x x\n+#\n+#############################################################################\n+test_expect_success 'one pack is redundant' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tgit pack-redundant --all --alt-odb >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_line_count = 1 actual\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# ================ master.git ===============\n+# | T A B C D E F G H I J K L M N O P Q R <----------------+\n+# ----+-------------------------------------- |\n+# P1 | x x x x x x x x |\n+# P3 | x x x x x x |\n+# P5 | x x x x |\n+# P7 | x x |\n+# ----+-------------------------------------- |\n+# ALL | x x x x x x x x x x x x x x x x x x x |\n+# |\n+# |\n+# ================ shared.git ======================= |\n+# | T A B C D E F G H I J K L M N O P Q R X Y Z \n+# ----+----------------------------------------------\n+# Px1*| s s s i i i\n+# Px2*| s s s i i i\n+# ----+----------------------------------------------\n+# ALL | s s s s s s s s s s s s s s s s s s s i i i\n+# (ignored objects, marked with i)\n+#\n+#############################################################################\n+test_expect_success 'set ignore objects and all two packs are redundant' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tPx1:$Px1\n+\t\t\tPx2:$Px2\n+\t\t\tEOF\n+\t\tgit pack-redundant --all --alt-odb >out <<-EOF &&\n+\t\t\t$X\n+\t\t\t$Y\n+\t\t\t$Z\n+\t\t\tEOF\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368338","messageId":"20190201162152.31136-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 2/6] pack-redundant: delay creation of unique_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:48Z","receivedAt":"2019-02-01T16:22:30Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nInstead of initializing unique_objects in `add_pack()`, copy from\nall_objects in `cmp_two_packs()`, when unwanted objects are removed from\nall_objects.\n\nThis will save memory (no allocate memory for alt-odb packs), and run\n`llist_sorted_difference_inplace()` only once when removing ignored\nobjects and removing objects in alt-odb in `scan_alt_odb_packs()`.\n\nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..f7dab0ec60 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -254,6 +254,11 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \tstruct llist_item *p1_hint = NULL, *p2_hint = NULL;\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n+\tif (!p1->unique_objects)\n+\t\tp1->unique_objects = llist_copy(p1->all_objects);\n+\tif (!p2->unique_objects)\n+\t\tp2->unique_objects = llist_copy(p2->all_objects);\n+\n \tp1_base = p1->pack->index_data;\n \tp2_base = p2->pack->index_data;\n \tp1_base += 256 * 4 + ((p1->pack->index_version < 2) ? 4 : 8);\n@@ -536,7 +541,7 @@ static void scan_alt_odb_packs(void)\n \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n-\t\t\tllist_sorted_difference_inplace(local->unique_objects,\n+\t\t\tllist_sorted_difference_inplace(local->all_objects,\n \t\t\t\t\t\t\talt->all_objects);\n \t\t\tlocal = local->next;\n \t\t}\n@@ -567,8 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n-\t/* this list will be pruned in cmp_two_packs later */\n-\tl.unique_objects = llist_copy(l.all_objects);\n+\tl.unique_objects = NULL;\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\n \telse\n@@ -646,7 +650,6 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \n \tload_all_objects();\n \n-\tcmp_local_packs();\n \tif (alt_odb)\n \t\tscan_alt_odb_packs();\n \n@@ -663,10 +666,12 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n+\tcmp_local_packs();\n+\n \tminimize(&min);\n \n \tif (verbose) {\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368339","messageId":"20190201162152.31136-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 3/6] pack-redundant: delete redundant code","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:49Z","receivedAt":"2019-02-01T16:22:37Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nThe objects in alt-odb are removed from `all_objects` twice in `load_all_objects`\nand `scan_alt_odb_packs`, remove it from the later function.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex f7dab0ec60..4a06f057dd 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -545,7 +545,6 @@ static void scan_alt_odb_packs(void)\n \t\t\t\t\t\t\talt->all_objects);\n \t\t\tlocal = local->next;\n \t\t}\n-\t\tllist_sorted_difference_inplace(all_objects, alt->all_objects);\n \t\talt = alt->next;\n \t}\n }\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368340","messageId":"20190201162152.31136-5-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 4/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:50Z","receivedAt":"2019-02-01T16:22:47Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nSome test cases will fail on Mac OS X. Mark them and will resolve in\nlater commit.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 194 +++++++++++++-------------------------\n t/t5323-pack-redundant.sh | 12 +--\n 2 files changed, 73 insertions(+), 133 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 4a06f057dd..d6d9a66e46 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -35,11 +35,6 @@ static struct pack_list {\n \tstruct llist *all_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n-struct pll {\n-\tstruct pll *next;\n-\tstruct pack_list *pl;\n-};\n-\n static struct llist_item *free_nodes;\n \n static inline void llist_item_put(struct llist_item *item)\n@@ -63,15 +58,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -290,78 +276,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n@@ -426,14 +340,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -451,49 +403,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -606,7 +546,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex d224ff3c50..897acaa365 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -218,7 +218,7 @@ test_expect_success 'create pack 4, 5' '\n # ALL | x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'one of pack-2/pack-3 is redundant' '\n+test_expect_failure 'one of pack-2/pack-3 is redundant (failed on Mac)' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n@@ -250,7 +250,7 @@ test_expect_success 'create pack 6, 7' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'pack 2, 4, and 6 are redundant' '\n+test_expect_failure 'pack 2, 4, and 6 are redundant (failed on Mac)' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n@@ -285,7 +285,7 @@ test_expect_success 'create pack 8' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n+test_expect_failure 'pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n@@ -309,7 +309,7 @@ test_expect_success 'clean loose objects' '\n \t)\n '\n \n-test_expect_success 'remove redundant packs and pass fsck' '\n+test_expect_failure 'remove redundant packs and pass fsck (failed on Mac)' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tgit pack-redundant --all | xargs rm &&\n@@ -329,7 +329,7 @@ test_expect_success 'setup shared.git' '\n \t)\n '\n \n-test_expect_success 'no redundant packs without --alt-odb' '\n+test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tgit pack-redundant --all >out &&\n@@ -362,7 +362,7 @@ test_expect_success 'no redundant packs without --alt-odb' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n+test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr (failed on Mac)' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tcat >expect <<-EOF &&\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368341","messageId":"20190201162152.31136-6-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 5/6] pack-redundant: rename pack_list.all_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:51Z","receivedAt":"2019-02-01T16:22:54Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nNew algorithm uses `pack_list.all_objects` to track remaining objects,\nso rename it to `pack_list.remaining_objects`.\n\nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex d6d9a66e46..15cdf233c4 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -32,7 +32,7 @@ static struct pack_list {\n \tstruct pack_list *next;\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n-\tstruct llist *all_objects;\n+\tstruct llist *remaining_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -241,9 +241,9 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n \tif (!p1->unique_objects)\n-\t\tp1->unique_objects = llist_copy(p1->all_objects);\n+\t\tp1->unique_objects = llist_copy(p1->remaining_objects);\n \tif (!p2->unique_objects)\n-\t\tp2->unique_objects = llist_copy(p2->all_objects);\n+\t\tp2->unique_objects = llist_copy(p2->remaining_objects);\n \n \tp1_base = p1->pack->index_data;\n \tp2_base = p2->pack->index_data;\n@@ -344,8 +344,8 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->all_objects->size;\n-\tsize_t sz_b = pl_b->all_objects->size;\n+\tsize_t sz_a = pl_a->remaining_objects->size;\n+\tsize_t sz_b = pl_b->remaining_objects->size;\n \n \tif (sz_a == sz_b)\n \t\treturn 0;\n@@ -355,7 +355,7 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n \t\treturn -1;\n }\n \n-/* Sort pack_list, greater size of all_objects first */\n+/* Sort pack_list, greater size of remaining_objects first */\n static void sort_pack_list(struct pack_list **pl)\n {\n \tstruct pack_list **ary, *p;\n@@ -399,7 +399,7 @@ static void minimize(struct pack_list **min)\n \tmissing = llist_copy(all_objects);\n \tpl = unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(missing, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(missing, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n \n@@ -417,20 +417,20 @@ static void minimize(struct pack_list **min)\n \t/* remove unique pack objects from the non_unique packs */\n \tpl = non_unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n \n \twhile (non_unique) {\n-\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\t/* sort the non_unique packs, greater size of remaining_objects first */\n \t\tsort_pack_list(&non_unique);\n-\t\tif (non_unique->all_objects->size == 0)\n+\t\tif (non_unique->remaining_objects->size == 0)\n \t\t\tbreak;\n \n \t\tpack_list_insert(min, non_unique);\n \n-\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n-\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\t\tfor (pl = non_unique->next; pl && pl->remaining_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->remaining_objects, non_unique->remaining_objects);\n \n \t\tnon_unique = non_unique->next;\n \t}\n@@ -445,7 +445,7 @@ static void load_all_objects(void)\n \n \twhile (pl) {\n \t\thint = NULL;\n-\t\tl = pl->all_objects->front;\n+\t\tl = pl->remaining_objects->front;\n \t\twhile (l) {\n \t\t\thint = llist_insert_sorted_unique(all_objects,\n \t\t\t\t\t\t\t l->oid, hint);\n@@ -456,7 +456,7 @@ static void load_all_objects(void)\n \t/* remove objects present in remote packs */\n \tpl = altodb_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(all_objects, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n }\n@@ -481,8 +481,8 @@ static void scan_alt_odb_packs(void)\n \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n-\t\t\tllist_sorted_difference_inplace(local->all_objects,\n-\t\t\t\t\t\t\talt->all_objects);\n+\t\t\tllist_sorted_difference_inplace(local->remaining_objects,\n+\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n \t\talt = alt->next;\n@@ -499,7 +499,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\treturn NULL;\n \n \tl.pack = p;\n-\tllist_init(&l.all_objects);\n+\tllist_init(&l.remaining_objects);\n \n \tif (open_pack_index(p))\n \t\treturn NULL;\n@@ -508,7 +508,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \tbase += 256 * 4 + ((p->index_version < 2) ? 4 : 8);\n \tstep = the_hash_algo->rawsz + ((p->index_version < 2) ? 4 : 0);\n \twhile (off < p->num_objects * step) {\n-\t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n+\t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n \tl.unique_objects = NULL;\n@@ -605,7 +605,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368342","messageId":"20190201162152.31136-7-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"20190130114736.30357-1-worldhello.net@gmail.com","subject":"[PATCH v9 6/6] pack-redundant: consistent sort method","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-01T16:21:52Z","receivedAt":"2019-02-01T16:23:03Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nSZEDER reported that test case t5323 has different test result on MacOS.\nThis is because `cmp_pack_list_reverse` cannot give identical result\nwhen two pack being sorted has the same size of remaining_objects.\n\nChanges to the sorting function will make consistent test result for\nt5323.\n\nThe new algorithm to find redundant packs is a trade-off to save memory\nresources, and the result of it may be different with old one, and may\nbe not the best result sometimes. Update t5323 for the new algorithm.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 24 ++++++++++++++++--------\n t/t5323-pack-redundant.sh | 18 +++++++++---------\n 2 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 15cdf233c4..29ff5e99cb 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -33,6 +33,7 @@ static struct pack_list {\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n \tstruct llist *remaining_objects;\n+\tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -340,19 +341,25 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n-static int cmp_pack_list_reverse(const void *a, const void *b)\n+static int cmp_remaining_objects(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->remaining_objects->size;\n-\tsize_t sz_b = pl_b->remaining_objects->size;\n \n-\tif (sz_a == sz_b)\n-\t\treturn 0;\n-\telse if (sz_a < sz_b)\n+\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size) {\n+\t\t/* have the same remaining_objects, big pack first */\n+\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n+\t\t\treturn 0;\n+\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n+\t\t\treturn 1;\n+\t\telse\n+\t\t\treturn -1;\n+\t} else if (pl_a->remaining_objects->size < pl_b->remaining_objects->size) {\n+\t\t/* sort by remaining objects, more objects first */\n \t\treturn 1;\n-\telse\n+\t} else {\n \t\treturn -1;\n+\t}\n }\n \n /* Sort pack_list, greater size of remaining_objects first */\n@@ -370,7 +377,7 @@ static void sort_pack_list(struct pack_list **pl)\n \tfor (n = 0, p = *pl; p; p = p->next)\n \t\tary[n++] = p;\n \n-\tQSORT(ary, n, cmp_pack_list_reverse);\n+\tQSORT(ary, n, cmp_remaining_objects);\n \n \t/* link them back again */\n \tfor (i = 0; i < n - 1; i++)\n@@ -511,6 +518,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n+\tl.all_objects_size = l.remaining_objects->size;\n \tl.unique_objects = NULL;\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex 897acaa365..e642240d71 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -210,19 +210,19 @@ test_expect_success 'create pack 4, 5' '\n # | T A B C D E F G H I J K L M N O P Q R\n # ----+--------------------------------------\n # P1 | x x x x x x x x\n-# P2* | ! ! ! ! ! ! !\n-# P3 | x x x x x x\n+# P2 | x x x x x x x\n+# P3* | ! ! ! ! ! !\n # P4 | x x x x x\n # P5 | x x x x\n # ----+--------------------------------------\n # ALL | x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'one of pack-2/pack-3 is redundant (failed on Mac)' '\n+test_expect_success 'one of pack-2/pack-3 is redundant' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n-\t\t\tP2:$P2\n+\t\t\tP3:$P3\n \t\t\tEOF\n \t\tgit pack-redundant --all >out &&\n \t\tformat_packfiles actual &&\n@@ -250,7 +250,7 @@ test_expect_success 'create pack 6, 7' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'pack 2, 4, and 6 are redundant (failed on Mac)' '\n+test_expect_success 'pack 2, 4, and 6 are redundant' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n@@ -285,7 +285,7 @@ test_expect_success 'create pack 8' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n+test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n@@ -309,7 +309,7 @@ test_expect_success 'clean loose objects' '\n \t)\n '\n \n-test_expect_failure 'remove redundant packs and pass fsck (failed on Mac)' '\n+test_expect_success 'remove redundant packs and pass fsck' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tgit pack-redundant --all | xargs rm &&\n@@ -329,7 +329,7 @@ test_expect_success 'setup shared.git' '\n \t)\n '\n \n-test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n+test_expect_success 'no redundant packs without --alt-odb' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tgit pack-redundant --all >out &&\n@@ -362,7 +362,7 @@ test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr (failed on Mac)' '\n+test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tcat >expect <<-EOF &&\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368366","messageId":"CAPig+cRX=ttHWe1C6m1u9-02oT-TpBtHWb48N=-wX4jSYaFuew@mail.gmail.com","threadId":"50055","inReplyTo":"20190201162152.31136-2-worldhello.net@gmail.com","subject":"Re: [PATCH v9 1/6] t5323: test cases for git-pack-redundant","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-02-01T19:42:53Z","receivedAt":"2019-02-01T19:43:06Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Feb 1, 2019 at 11:22 AM Jiang Xin wrote:\n> Add test cases for git pack-redundant to validate new algorithm for git\n> pack-redundant.\n>\n> Signed-off-by: Jiang Xin \n> ---\n> diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n> @@ -0,0 +1,510 @@\n> +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n\nWhich variables won't be set? It's not clear what this restriction is about.\n\n> +# Usage: create_commits_in A B C ...\n> +create_commits_in () {\n> + repo=\"$1\" &&\n> + parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null) || parent=\n\nBroken &&-chain. Instead, perhaps:\n\n if ! parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null)\n then\n parent=\n fi &&\n\nor something simpler.\n\n> + T=$(git -C \"$repo\" write-tree) &&\n> + shift &&\n> + while test $# -gt 0\n> + do\n> + name=$1 &&\n> + test_tick &&\n> + if test -z \"$parent\"\n> + then\n> + oid=$(echo $name | git -C \"$repo\" commit-tree $T)\n> + else\n> + oid=$(echo $name | git -C \"$repo\" commit-tree -p $parent $T)\n> + fi &&\n> + eval $name=$oid &&\n> + parent=$oid &&\n> + shift ||\n> + return 1\n> + done\n\nBroken &&-chain. Use:\n\n done &&\n\n> + git -C \"$repo\" update-ref refs/heads/master $oid\n> +}\n> +\n> +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n> +create_pack_1 () {\n> + P1=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n\nWhich variables? Note that you can capture output of a subshell into a\nvariable, if necessary.\n"},{"id":"368373","messageId":"xmqqva23xlun.fsf@gitster-ct.c.googlers.com","threadId":"50055","inReplyTo":"CAPig+cRX=ttHWe1C6m1u9-02oT-TpBtHWb48N=-wX4jSYaFuew@mail.gmail.com","subject":"Re: [PATCH v9 1/6] t5323: test cases for git-pack-redundant","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-01T21:03:28Z","receivedAt":"2019-02-01T21:03:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Fri, Feb 1, 2019 at 11:22 AM Jiang Xin wrote:\n>> Add test cases for git pack-redundant to validate new algorithm for git\n>> pack-redundant.\n>>\n>> Signed-off-by: Jiang Xin \n>> ---\n>> diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n>> @@ -0,0 +1,510 @@\n>> +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n>\n> Which variables won't be set? It's not clear what this restriction is about.\n\n>> + git -C \"$repo\" update-ref refs/heads/master $oid\n>> +}\n>> +\n>> +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n>> +create_pack_1 () {\n>> + P1=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n>\n> Which variables? Note that you can capture output of a subshell into a\n> variable, if necessary.\n\nThese helper functions set a bunch of variables $P1, $P2, etc. as\nwell as variables whose name begin with P and followed by 40-hex.\nThe script wants to use them later when preparing expected output,\nand with the most natural way to organize the code, that \"later\"\nhappens in the process that would have spawned a subshell to run\nthis function.\n\nIt would have been easier for you to grok if the note instead said\n\"this function sets two global shell variables\" or something,\nperhaps? Such a variable would certainly not be visible if this\nfunction is called inside a subshell to the main process.\n"},{"id":"368376","messageId":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","threadId":"50055","inReplyTo":"xmqqva23xlun.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v9 1/6] t5323: test cases for git-pack-redundant","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-02-01T21:49:30Z","receivedAt":"2019-02-01T21:49:44Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Feb 1, 2019 at 4:03 PM Junio C Hamano wrote:\n> Eric Sunshine writes:\n> > On Fri, Feb 1, 2019 at 11:22 AM Jiang Xin wrote:\n> >> +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n> >\n> > Which variables won't be set? It's not clear what this restriction is about.\n>\n> >> +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n> >> +create_pack_1 () {\n> >> + P1=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n> >\n> > Which variables? Note that you can capture output of a subshell into a\n> > variable, if necessary.\n>\n> These helper functions set a bunch of variables $P1, $P2, etc. as\n> well as variables whose name begin with P and followed by 40-hex.\n> The script wants to use them later when preparing expected output,\n> and with the most natural way to organize the code, that \"later\"\n> happens in the process that would have spawned a subshell to run\n> this function.\n>\n> It would have been easier for you to grok if the note instead said\n> \"this function sets two global shell variables\" or something,\n> perhaps? Such a variable would certainly not be visible if this\n> function is called inside a subshell to the main process.\n\nYes, better function comments would facilitate comprehension both for\nthe reviewer and those working on the code in the future. For\ninstance:\n\n # Create commit for each argument [...with blah properties...] and\n # assign [...] to shell variable of same name as argument.\n # NOTE: Avoid calling this function from a subshell since variable\n # assignments will disappear when subshell exits.\n\nor something.\n\nBut, looking more closely at the patch, I'm wondering why the various\ncreate_pack_#() functions are defined at all since they are each only\never called from a single place.\n\n create_pack_4 () {\n ...\n eval P$P4=P4:$P4\n }\n ...\n test_expect_success 'create pack 4, 5' '\n create_pack_4 && create_pack_5\n '\n\nI haven't been able to convince myself that this helps readability --\nespecially since the function definition is often far removed from the\nsingle point of use -- over merely inlining the function body directly\nin the sole test which calls it.\n\nAnyhow, this is all pretty minor, not necessarily worth a re-roll.\n"},{"id":"368407","messageId":"20190202133017.1039-1-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 0/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:11Z","receivedAt":"2019-02-02T13:30:45Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"Sun Chao (my former colleague at Huawei) found a bug of\ngit-pack-redundant. If there are too many packs and many of them\noverlap each other, running `git pack-redundant --all` will\nexhaust all memories and the process will be killed by kernel.\n\nThere is a script in commit log of commit 3/6, which can be used to\ncreate a repository with lots of redundant packs. Running `git\npack-redundant --all` in it can reproduce this issue.\n\n## Changes since re-roll v9\n\nEric Sunshine 于2019年2月2日周六 上午3:43写道：\n>\n> On Fri, Feb 1, 2019 at 11:22 AM Jiang Xin wrote:\n> > Add test cases for git pack-redundant to validate new algorithm for git\n> > pack-redundant.\n> >\n> > Signed-off-by: Jiang Xin \n> > ---\n> > diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\n> > @@ -0,0 +1,510 @@\n> > +# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n>\n> Which variables won't be set? It's not clear what this restriction is about.\n>\n> > +# Usage: create_commits_in A B C ...\n> > +create_commits_in () {\n> > + repo=\"$1\" &&\n> > + parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null) || parent=\n>\n> Broken &&-chain. Instead, perhaps:\n>\n> if ! parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null)\n> then\n> parent=\n> fi &&\n>\n> or something simpler.\n\nFixed.\n\n> > + T=$(git -C \"$repo\" write-tree) &&\n> > + shift &&\n> > + while test $# -gt 0\n> > + do\n> > + name=$1 &&\n> > + test_tick &&\n> > + if test -z \"$parent\"\n> > + then\n> > + oid=$(echo $name | git -C \"$repo\" commit-tree $T)\n> > + else\n> > + oid=$(echo $name | git -C \"$repo\" commit-tree -p $parent $T)\n> > + fi &&\n> > + eval $name=$oid &&\n> > + parent=$oid &&\n> > + shift ||\n> > + return 1\n> > + done\n>\n> Broken &&-chain. Use:\n>\n> done &&\n>\n\nFixed, thanks.\n\n> > It would have been easier for you to grok if the note instead said\n> > \"this function sets two global shell variables\" or something,\n> > perhaps? Such a variable would certainly not be visible if this\n> > function is called inside a subshell to the main process.\n>\n> Yes, better function comments would facilitate comprehension both for\n> the reviewer and those working on the code in the future. For\n> instance:\n>\n> # Create commit for each argument [...with blah properties...] and\n> # assign [...] to shell variable of same name as argument.\n> # NOTE: Avoid calling this function from a subshell since variable\n> # assignments will disappear when subshell exits.\n\nPolished comments for `create_commits_in` and `create_pack_in` helper\nfunction.\n\n> create_pack_4 () {\n> ...\n> eval P$P4=P4:$P4\n> }\n> ...\n> test_expect_success 'create pack 4, 5' '\n> create_pack_4 && create_pack_5\n> '\n>\n> I haven't been able to convince myself that this helps readability --\n> especially since the function definition is often far removed from the\n> single point of use -- over merely inlining the function body directly\n> in the sole test which calls it.\n\nUse a new helper function `create_pack_in` to create packs near test \nfunctions.\n\n\n## Range diff since v9:\n\n1: c8dbf8cef2 ! 1: 4719043603 t5323: test cases for git-pack-redundant\n @@ -43,8 +43,8 @@\n + ALL | x x x x x x x x x x x x x x x x x x x\n +\n +Another repository `shared.git` has unique objects (X-Z), while other objects\n -+(marked with letter s) are shared through alt-odb (of `master.git`). The\n -+relationship between packs and objects is as follows:\n ++(marked with letter s) can be found in the shared alt-odb (of `master.git`).\n ++The relationship between packs and objects is as follows:\n +\n + | T A B C D E F G H I J K L M N O P Q R X Y Z\n + ----+----------------------------------------------\n @@ -57,11 +57,19 @@\n +master_repo=master.git\n +shared_repo=shared.git\n +\n -+# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n -+# Usage: create_commits_in A B C ...\n ++# Create commits in and assign each commit's oid to shell variables\n ++# given in the arguments (A, B, and C). E.g.:\n ++#\n ++# create_commits_in A B C\n ++#\n ++# NOTE: Avoid calling this function from a subshell since variable\n ++# assignments will disappear when subshell exits.\n +create_commits_in () {\n +\trepo=\"$1\" &&\n -+\tparent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null) || parent=\n ++\tif ! parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null)\n ++\tthen\n ++\t\tparent=\n ++\tfi &&\n +\tT=$(git -C \"$repo\" write-tree) &&\n +\tshift &&\n +\twhile test $# -gt 0\n @@ -78,101 +86,26 @@\n +\t\tparent=$oid &&\n +\t\tshift ||\n +\t\treturn 1\n -+\tdone\n ++\tdone &&\n +\tgit -C \"$repo\" update-ref refs/heads/master $oid\n +}\n +\n -+# Note: DO NOT run it in a subshell, otherwise the variables will not be set\n -+create_pack_1 () {\n -+\tP1=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$T\n -+\t\t$A\n -+\t\t$B\n -+\t\t$C\n -+\t\t$D\n -+\t\t$E\n -+\t\t$F\n -+\t\t$R\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P1=P1:$P1\n -+}\n -+\n -+create_pack_2 () {\n -+\tP2=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$B\n -+\t\t$C\n -+\t\t$D\n -+\t\t$E\n -+\t\t$G\n -+\t\t$H\n -+\t\t$I\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P2=P2:$P2\n -+}\n -+\n -+create_pack_3 () {\n -+\tP3=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$F\n -+\t\t$I\n -+\t\t$J\n -+\t\t$K\n -+\t\t$L\n -+\t\t$M\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P3=P3:$P3\n -+}\n -+\n -+create_pack_4 () {\n -+\tP4=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$J\n -+\t\t$K\n -+\t\t$L\n -+\t\t$M\n -+\t\t$P\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P4=P4:$P4\n -+}\n -+\n -+create_pack_5 () {\n -+\tP5=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$G\n -+\t\t$H\n -+\t\t$N\n -+\t\t$O\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P5=P5:$P5\n -+}\n -+\n -+create_pack_6 () {\n -+\tP6=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$N\n -+\t\t$O\n -+\t\t$Q\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P6=P6:$P6\n -+}\n -+\n -+create_pack_7 () {\n -+\tP7=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$P\n -+\t\t$Q\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P7=P7:$P7\n -+}\n -+\n -+create_pack_8 () {\n -+\tP8=$(git -C \"$master_repo/objects/pack\" pack-objects -q pack <<-EOF\n -+\t\t$A\n -+\t\tEOF\n -+\t) &&\n -+\teval P$P8=P8:$P8\n ++# Create pack in and assign pack id to variable given in the 2nd argument\n ++# (). Commits in the pack will be read from stdin. E.g.:\n ++#\n ++# create_pack_in <<-EOF\n ++# ...\n ++# EOF\n ++#\n ++# NOTE: commits from stdin should be given using heredoc, not using pipe, and\n ++# avoid calling this function from a subshell since variable assignments will\n ++# disappear when subshell exits.\n ++create_pack_in () {\n ++\trepo=\"$1\" &&\n ++\tname=\"$2\" &&\n ++\tpack=$(git -C \"$repo/objects/pack\" pack-objects -q pack) &&\n ++\teval $name=$pack &&\n ++\teval P$pack=$name:$pack\n +}\n +\n +format_packfiles () {\n @@ -209,8 +142,34 @@\n +# ALL | x x x x x x x x x x x x x x x\n +#\n +#############################################################################\n -+test_expect_success 'no redundant for pack 1, 2, 3' '\n -+\tcreate_pack_1 && create_pack_2 && create_pack_3 &&\n ++test_expect_success 'master: no redundant for pack 1, 2, 3' '\n ++\tcreate_pack_in \"$master_repo\" P1 <<-EOF &&\n ++\t\t$T\n ++\t\t$A\n ++\t\t$B\n ++\t\t$C\n ++\t\t$D\n ++\t\t$E\n ++\t\t$F\n ++\t\t$R\n ++\t\tEOF\n ++\tcreate_pack_in \"$master_repo\" P2 <<-EOF &&\n ++\t\t$B\n ++\t\t$C\n ++\t\t$D\n ++\t\t$E\n ++\t\t$G\n ++\t\t$H\n ++\t\t$I\n ++\t\tEOF\n ++\tcreate_pack_in \"$master_repo\" P3 <<-EOF &&\n ++\t\t$F\n ++\t\t$I\n ++\t\t$J\n ++\t\t$K\n ++\t\t$L\n ++\t\t$M\n ++\t\tEOF\n +\t(\n +\t\tcd \"$master_repo\" &&\n +\t\tgit pack-redundant --all >out &&\n @@ -218,10 +177,6 @@\n +\t)\n +'\n +\n -+test_expect_success 'create pack 4, 5' '\n -+\tcreate_pack_4 && create_pack_5\n -+'\n -+\n +#############################################################################\n +# Chart of packs and objects for this test case\n +#\n @@ -236,7 +191,20 @@\n +# ALL | x x x x x x x x x x x x x x x x x x\n +#\n +#############################################################################\n -+test_expect_success 'one of pack-2/pack-3 is redundant' '\n ++test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n ++\tcreate_pack_in \"$master_repo\" P4 <<-EOF &&\n ++\t\t$J\n ++\t\t$K\n ++\t\t$L\n ++\t\t$M\n ++\t\t$P\n ++\t\tEOF\n ++\tcreate_pack_in \"$master_repo\" P5 <<-EOF &&\n ++\t\t$G\n ++\t\t$H\n ++\t\t$N\n ++\t\t$O\n ++\t\tEOF\n +\t(\n +\t\tcd \"$master_repo\" &&\n +\t\tcat >expect <<-EOF &&\n @@ -248,10 +216,6 @@\n +\t)\n +'\n +\n -+test_expect_success 'create pack 6, 7' '\n -+\tcreate_pack_6 && create_pack_7\n -+'\n -+\n +#############################################################################\n +# Chart of packs and objects for this test case\n +#\n @@ -268,7 +232,16 @@\n +# ALL | x x x x x x x x x x x x x x x x x x x\n +#\n +#############################################################################\n -+test_expect_success 'pack 2, 4, and 6 are redundant' '\n ++test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n ++\tcreate_pack_in \"$master_repo\" P6 <<-EOF &&\n ++\t\t$N\n ++\t\t$O\n ++\t\t$Q\n ++\t\tEOF\n ++\tcreate_pack_in \"$master_repo\" P7 <<-EOF &&\n ++\t\t$P\n ++\t\t$Q\n ++\t\tEOF\n +\t(\n +\t\tcd \"$master_repo\" &&\n +\t\tcat >expect <<-EOF &&\n @@ -282,10 +255,6 @@\n +\t)\n +'\n +\n -+test_expect_success 'create pack 8' '\n -+\tcreate_pack_8\n -+'\n -+\n +#############################################################################\n +# Chart of packs and objects for this test case\n +#\n @@ -303,7 +272,10 @@\n +# ALL | x x x x x x x x x x x x x x x x x x x\n +#\n +#############################################################################\n -+test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n ++test_expect_success 'master: pack-8 (subset of pack-1) is also redundant' '\n ++\tcreate_pack_in \"$master_repo\" P8 <<-EOF &&\n ++\t\t$A\n ++\t\tEOF\n +\t(\n +\t\tcd \"$master_repo\" &&\n +\t\tcat >expect <<-EOF &&\n @@ -318,7 +290,7 @@\n +\t)\n +'\n +\n -+test_expect_success 'clean loose objects' '\n ++test_expect_success 'master: clean loose objects' '\n +\t(\n +\t\tcd \"$master_repo\" &&\n +\t\tgit prune-packed &&\n @@ -327,7 +299,7 @@\n +\t)\n +'\n +\n -+test_expect_success 'remove redundant packs and pass fsck' '\n ++test_expect_success 'master: remove redundant packs and pass fsck' '\n +\t(\n +\t\tcd \"$master_repo\" &&\n +\t\tgit pack-redundant --all | xargs rm &&\n @@ -347,7 +319,7 @@\n +\t)\n +'\n +\n -+test_expect_success 'no redundant packs without --alt-odb' '\n ++test_expect_success 'shared: all packs are redundant, but no output without --alt-odb' '\n +\t(\n +\t\tcd \"$shared_repo\" &&\n +\t\tgit pack-redundant --all >out &&\n @@ -380,7 +352,7 @@\n +# ALL | x x x x x x x x x x x x x x x x x x x\n +#\n +#############################################################################\n -+test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n ++test_expect_success 'shared: show redundant packs in stderr for verbose mode' '\n +\t(\n +\t\tcd \"$shared_repo\" &&\n +\t\tcat >expect <<-EOF &&\n @@ -396,7 +368,7 @@\n +\t)\n +'\n +\n -+test_expect_success 'remove redundant packs by alt-odb, no packs left' '\n ++test_expect_success 'shared: remove redundant packs, no packs left' '\n +\t(\n +\t\tcd \"$shared_repo\" &&\n +\t\tcat >expect <<-EOF &&\n @@ -409,10 +381,9 @@\n +\t)\n +'\n +\n -+# Note: DO NOT run function `create_pack_*` in sub shell, or variables are not set\n -+create_pack_x1_in () {\n -+\trepo=\"$1\" &&\n -+\tPx1=$(git -C \"$repo/objects/pack\" pack-objects -q pack <<-EOF\n ++test_expect_success 'shared: create new objects and packs' '\n ++\tcreate_commits_in \"$shared_repo\" X Y Z &&\n ++\tcreate_pack_in \"$shared_repo\" Px1 <<-EOF &&\n +\t\t$X\n +\t\t$Y\n +\t\t$Z\n @@ -420,13 +391,7 @@\n +\t\t$B\n +\t\t$C\n +\t\tEOF\n -+\t) &&\n -+\teval P${Px1}=Px1:${Px1}\n -+}\n -+\n -+create_pack_x2_in () {\n -+\trepo=\"$1\" &&\n -+\tPx2=$(git -C \"$repo/objects/pack\" pack-objects -q pack <<-EOF\n ++\tcreate_pack_in \"$shared_repo\" Px2 <<-EOF\n +\t\t$X\n +\t\t$Y\n +\t\t$Z\n @@ -434,17 +399,9 @@\n +\t\t$E\n +\t\t$F\n +\t\tEOF\n -+\t) &&\n -+\teval P${Px2}=Px2:${Px2}\n -+}\n -+\n -+test_expect_success 'create new objects and packs in shared.git' '\n -+\tcreate_commits_in \"$shared_repo\" X Y Z &&\n -+\tcreate_pack_x1_in \"$shared_repo\" &&\n -+\tcreate_pack_x2_in \"$shared_repo\"\n +'\n +\n -+test_expect_success 'no redundant without --alt-odb' '\n ++test_expect_success 'shared: no redundant without --alt-odb' '\n +\t(\n +\t\tcd \"$shared_repo\" &&\n +\t\tgit pack-redundant --all >out &&\n @@ -475,7 +432,7 @@\n +# ALL | s s s s s s s s s s s s s s s s s s s x x x\n +#\n +#############################################################################\n -+test_expect_success 'one pack is redundant' '\n ++test_expect_success 'shared: one pack is redundant with --alt-odb' '\n +\t(\n +\t\tcd \"$shared_repo\" &&\n +\t\tgit pack-redundant --all --alt-odb >out &&\n @@ -508,7 +465,7 @@\n +# (ignored objects, marked with i)\n +#\n +#############################################################################\n -+test_expect_success 'set ignore objects and all two packs are redundant' '\n ++test_expect_success 'shared: ignore unique objects and all two packs are redundant' '\n +\t(\n +\t\tcd \"$shared_repo\" &&\n +\t\tcat >expect <<-EOF &&\n2: a6300516d7 = 2: 4feb1eaa40 pack-redundant: delay creation of unique_objects\n3: fb71973df5 = 3: 875367d7b4 pack-redundant: delete redundant code\n4: 9963d1c49f ! 4: 50cb2854f1 pack-redundant: new algorithm to find min packs\n @@ -331,35 +331,35 @@\n # ALL | x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_success 'one of pack-2/pack-3 is redundant' '\n -+test_expect_failure 'one of pack-2/pack-3 is redundant (failed on Mac)' '\n - \t(\n - \t\tcd \"$master_repo\" &&\n - \t\tcat >expect <<-EOF &&\n +-test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n ++test_expect_failure 'master: one of pack-2/pack-3 is redundant (failed on Mac)' '\n + \tcreate_pack_in \"$master_repo\" P4 <<-EOF &&\n + \t\t$J\n + \t\t$K\n @@\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_success 'pack 2, 4, and 6 are redundant' '\n -+test_expect_failure 'pack 2, 4, and 6 are redundant (failed on Mac)' '\n - \t(\n - \t\tcd \"$master_repo\" &&\n - \t\tcat >expect <<-EOF &&\n +-test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n ++test_expect_failure 'master: pack 2, 4, and 6 are redundant (failed on Mac)' '\n + \tcreate_pack_in \"$master_repo\" P6 <<-EOF &&\n + \t\t$N\n + \t\t$O\n @@\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n -+test_expect_failure 'pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n - \t(\n - \t\tcd \"$master_repo\" &&\n - \t\tcat >expect <<-EOF &&\n +-test_expect_success 'master: pack-8 (subset of pack-1) is also redundant' '\n ++test_expect_failure 'master: pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n + \tcreate_pack_in \"$master_repo\" P8 <<-EOF &&\n + \t\t$A\n + \t\tEOF\n @@\n \t)\n '\n \n --test_expect_success 'remove redundant packs and pass fsck' '\n -+test_expect_failure 'remove redundant packs and pass fsck (failed on Mac)' '\n +-test_expect_success 'master: remove redundant packs and pass fsck' '\n ++test_expect_failure 'master: remove redundant packs and pass fsck (failed on Mac)' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tgit pack-redundant --all | xargs rm &&\n @@ -367,8 +367,8 @@\n \t)\n '\n \n --test_expect_success 'no redundant packs without --alt-odb' '\n -+test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n +-test_expect_success 'shared: all packs are redundant, but no output without --alt-odb' '\n ++test_expect_failure 'shared: all packs are redundant, but no output without --alt-odb (failed on Mac)' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tgit pack-redundant --all >out &&\n @@ -376,8 +376,8 @@\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n -+test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr (failed on Mac)' '\n +-test_expect_success 'shared: show redundant packs in stderr for verbose mode' '\n ++test_expect_failure 'shared: show redundant packs in stderr for verbose mode (failed on Mac)' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tcat >expect <<-EOF &&\n5: b8f80ad454 = 5: 4af03876d4 pack-redundant: rename pack_list.all_objects\n6: 8a12ad699e ! 6: 89ed4fb2a5 pack-redundant: consistent sort method\n @@ -96,8 +96,12 @@\n # ALL | x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_failure 'one of pack-2/pack-3 is redundant (failed on Mac)' '\n -+test_expect_success 'one of pack-2/pack-3 is redundant' '\n +-test_expect_failure 'master: one of pack-2/pack-3 is redundant (failed on Mac)' '\n ++test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n + \tcreate_pack_in \"$master_repo\" P4 <<-EOF &&\n + \t\t$J\n + \t\t$K\n +@@\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n @@ -110,26 +114,26 @@\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_failure 'pack 2, 4, and 6 are redundant (failed on Mac)' '\n -+test_expect_success 'pack 2, 4, and 6 are redundant' '\n - \t(\n - \t\tcd \"$master_repo\" &&\n - \t\tcat >expect <<-EOF &&\n +-test_expect_failure 'master: pack 2, 4, and 6 are redundant (failed on Mac)' '\n ++test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n + \tcreate_pack_in \"$master_repo\" P6 <<-EOF &&\n + \t\t$N\n + \t\t$O\n @@\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_failure 'pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n -+test_expect_success 'pack-8 (subset of pack-1) is also redundant' '\n - \t(\n - \t\tcd \"$master_repo\" &&\n - \t\tcat >expect <<-EOF &&\n +-test_expect_failure 'master: pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n ++test_expect_success 'master: pack-8 (subset of pack-1) is also redundant' '\n + \tcreate_pack_in \"$master_repo\" P8 <<-EOF &&\n + \t\t$A\n + \t\tEOF\n @@\n \t)\n '\n \n --test_expect_failure 'remove redundant packs and pass fsck (failed on Mac)' '\n -+test_expect_success 'remove redundant packs and pass fsck' '\n +-test_expect_failure 'master: remove redundant packs and pass fsck (failed on Mac)' '\n ++test_expect_success 'master: remove redundant packs and pass fsck' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tgit pack-redundant --all | xargs rm &&\n @@ -137,8 +141,8 @@\n \t)\n '\n \n --test_expect_failure 'no redundant packs without --alt-odb (failed on Mac)' '\n -+test_expect_success 'no redundant packs without --alt-odb' '\n +-test_expect_failure 'shared: all packs are redundant, but no output without --alt-odb (failed on Mac)' '\n ++test_expect_success 'shared: all packs are redundant, but no output without --alt-odb' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tgit pack-redundant --all >out &&\n @@ -146,8 +150,8 @@\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n --test_expect_failure 'pack-redundant --verbose: show duplicate packs in stderr (failed on Mac)' '\n -+test_expect_success 'pack-redundant --verbose: show duplicate packs in stderr' '\n +-test_expect_failure 'shared: show redundant packs in stderr for verbose mode (failed on Mac)' '\n ++test_expect_success 'shared: show redundant packs in stderr for verbose mode' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tcat >expect <<-EOF &&\n\n--\n\nJiang Xin (4):\n t5323: test cases for git-pack-redundant\n pack-redundant: delay creation of unique_objects\n pack-redundant: rename pack_list.all_objects\n pack-redundant: consistent sort method\n\nSun Chao (2):\n pack-redundant: delete redundant code\n pack-redundant: new algorithm to find min packs\n\n builtin/pack-redundant.c | 232 ++++++++-----------\n t/t5323-pack-redundant.sh | 467 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 559 insertions(+), 140 deletions(-)\n create mode 100755 t/t5323-pack-redundant.sh\n\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368408","messageId":"20190202133017.1039-2-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 1/6] t5323: test cases for git-pack-redundant","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:12Z","receivedAt":"2019-02-02T13:30:57Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nAdd test cases for git pack-redundant to validate new algorithm for git\npack-redundant.\n\nSigned-off-by: Jiang Xin \nReviewed-by: SZEDER Gábor \nHelped-by: Eric Sunshine \nReviewed-by: Sun Chao \nSigned-off-by: Junio C Hamano \n---\n t/t5323-pack-redundant.sh | 467 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 467 insertions(+)\n create mode 100755 t/t5323-pack-redundant.sh\n\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nnew file mode 100755\nindex 0000000000..f18db06d91\n--- /dev/null\n+++ b/t/t5323-pack-redundant.sh\n@@ -0,0 +1,467 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2018 Jiang Xin\n+#\n+\n+test_description='Test git pack-redundant\n+\n+In order to test git-pack-redundant, we will create a number of objects and\n+packs in the repository `master.git`. The relationship between packs (P1-P8)\n+and objects (T, A-R) is showed in the following chart. Objects of a pack will\n+be marked with letter x, while objects of redundant packs will be marked with\n+exclamation point, and redundant pack itself will be marked with asterisk.\n+\n+ | T A B C D E F G H I J K L M N O P Q R\n+ ----+--------------------------------------\n+ P1 | x x x x x x x x\n+ P2* | ! ! ! ! ! ! !\n+ P3 | x x x x x x\n+ P4* | ! ! ! ! !\n+ P5 | x x x x\n+ P6* | ! ! !\n+ P7 | x x\n+ P8* | !\n+ ----+--------------------------------------\n+ ALL | x x x x x x x x x x x x x x x x x x x\n+\n+Another repository `shared.git` has unique objects (X-Z), while other objects\n+(marked with letter s) are shared through alt-odb (of `master.git`). The\n+relationship between packs and objects is as follows:\n+\n+ | T A B C D E F G H I J K L M N O P Q R X Y Z\n+ ----+----------------------------------------------\n+ Px1 | s s s x x x\n+ Px2 | s s s x x x\n+'\n+\n+. ./test-lib.sh\n+\n+master_repo=master.git\n+shared_repo=shared.git\n+\n+# Create commits in and assign each commit's oid to shell variables\n+# given in the arguments (A, B, and C). E.g.:\n+#\n+# create_commits_in A B C\n+#\n+# NOTE: Avoid calling this function from a subshell since variable\n+# assignments will disappear when subshell exits.\n+create_commits_in () {\n+\trepo=\"$1\" &&\n+\tif ! parent=$(git -C \"$repo\" rev-parse HEAD^{} 2>/dev/null)\n+\tthen\n+\t\tparent=\n+\tfi &&\n+\tT=$(git -C \"$repo\" write-tree) &&\n+\tshift &&\n+\twhile test $# -gt 0\n+\tdo\n+\t\tname=$1 &&\n+\t\ttest_tick &&\n+\t\tif test -z \"$parent\"\n+\t\tthen\n+\t\t\toid=$(echo $name | git -C \"$repo\" commit-tree $T)\n+\t\telse\n+\t\t\toid=$(echo $name | git -C \"$repo\" commit-tree -p $parent $T)\n+\t\tfi &&\n+\t\teval $name=$oid &&\n+\t\tparent=$oid &&\n+\t\tshift ||\n+\t\treturn 1\n+\tdone &&\n+\tgit -C \"$repo\" update-ref refs/heads/master $oid\n+}\n+\n+# Create pack in and assign pack id to variable given in the 2nd argument\n+# (). Commits in the pack will be read from stdin. E.g.:\n+#\n+# create_pack_in <<-EOF\n+# ...\n+# EOF\n+#\n+# NOTE: commits from stdin should be given using heredoc, not using pipe, and\n+# avoid calling this function from a subshell since variable assignments will\n+# disappear when subshell exits.\n+create_pack_in () {\n+\trepo=\"$1\" &&\n+\tname=\"$2\" &&\n+\tpack=$(git -C \"$repo/objects/pack\" pack-objects -q pack) &&\n+\teval $name=$pack &&\n+\teval P$pack=$name:$pack\n+}\n+\n+format_packfiles () {\n+\tsed \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.idx#\\1#\" \\\n+\t\t-e \"s#.*/pack-\\(.*\\)\\.pack#\\1#\" |\n+\tsort -u |\n+\twhile read p\n+\tdo\n+\t\tif test -z \"$(eval echo \\${P$p})\"\n+\t\tthen\n+\t\t\techo $p\n+\t\telse\n+\t\t\teval echo \"\\${P$p}\"\n+\t\tfi\n+\tdone |\n+\tsort\n+}\n+\n+test_expect_success 'setup master repo' '\n+\tgit init --bare \"$master_repo\" &&\n+\tcreate_commits_in \"$master_repo\" A B C D E F G H I J K L M N O P Q R\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2 | x x x x x x x\n+# P3 | x x x x x x\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'master: no redundant for pack 1, 2, 3' '\n+\tcreate_pack_in \"$master_repo\" P1 <<-EOF &&\n+\t\t$T\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\t$R\n+\t\tEOF\n+\tcreate_pack_in \"$master_repo\" P2 <<-EOF &&\n+\t\t$B\n+\t\t$C\n+\t\t$D\n+\t\t$E\n+\t\t$G\n+\t\t$H\n+\t\t$I\n+\t\tEOF\n+\tcreate_pack_in \"$master_repo\" P3 <<-EOF &&\n+\t\t$F\n+\t\t$I\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\tEOF\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2* | ! ! ! ! ! ! !\n+# P3 | x x x x x x\n+# P4 | x x x x x\n+# P5 | x x x x\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n+\tcreate_pack_in \"$master_repo\" P4 <<-EOF &&\n+\t\t$J\n+\t\t$K\n+\t\t$L\n+\t\t$M\n+\t\t$P\n+\t\tEOF\n+\tcreate_pack_in \"$master_repo\" P5 <<-EOF &&\n+\t\t$G\n+\t\t$H\n+\t\t$N\n+\t\t$O\n+\t\tEOF\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP2:$P2\n+\t\t\tEOF\n+\t\tgit pack-redundant --all >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2* | ! ! ! ! ! ! !\n+# P3 | x x x x x x\n+# P4* | ! ! ! ! !\n+# P5 | x x x x\n+# P6* | ! ! !\n+# P7 | x x\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n+\tcreate_pack_in \"$master_repo\" P6 <<-EOF &&\n+\t\t$N\n+\t\t$O\n+\t\t$Q\n+\t\tEOF\n+\tcreate_pack_in \"$master_repo\" P7 <<-EOF &&\n+\t\t$P\n+\t\t$Q\n+\t\tEOF\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP2:$P2\n+\t\t\tP4:$P4\n+\t\t\tP6:$P6\n+\t\t\tEOF\n+\t\tgit pack-redundant --all >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# | T A B C D E F G H I J K L M N O P Q R\n+# ----+--------------------------------------\n+# P1 | x x x x x x x x\n+# P2* | ! ! ! ! ! ! !\n+# P3 | x x x x x x\n+# P4* | ! ! ! ! !\n+# P5 | x x x x\n+# P6* | ! ! !\n+# P7 | x x\n+# P8* | !\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'master: pack-8 (subset of pack-1) is also redundant' '\n+\tcreate_pack_in \"$master_repo\" P8 <<-EOF &&\n+\t\t$A\n+\t\tEOF\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP2:$P2\n+\t\t\tP4:$P4\n+\t\t\tP6:$P6\n+\t\t\tP8:$P8\n+\t\t\tEOF\n+\t\tgit pack-redundant --all >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'master: clean loose objects' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tgit prune-packed &&\n+\t\tfind objects -type f | sed -e \"/objects\\/pack\\//d\" >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+test_expect_success 'master: remove redundant packs and pass fsck' '\n+\t(\n+\t\tcd \"$master_repo\" &&\n+\t\tgit pack-redundant --all | xargs rm &&\n+\t\tgit fsck &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+# The following test cases will execute inside `shared.git`, instead of\n+# inside `master.git`.\n+test_expect_success 'setup shared.git' '\n+\tgit clone --mirror \"$master_repo\" \"$shared_repo\" &&\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tprintf \"../../$master_repo/objects\\n\" >objects/info/alternates\n+\t)\n+'\n+\n+test_expect_success 'shared: all packs are redundant, but no output without --alt-odb' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# ================ master.git ===============\n+# | T A B C D E F G H I J K L M N O P Q R <----------+\n+# ----+-------------------------------------- |\n+# P1 | x x x x x x x x |\n+# P3 | x x x x x x |\n+# P5 | x x x x |\n+# P7 | x x |\n+# ----+-------------------------------------- |\n+# ALL | x x x x x x x x x x x x x x x x x x x |\n+# |\n+# |\n+# ================ shared.git =============== |\n+# | T A B C D E F G H I J K L M N O P Q R \n+# ----+--------------------------------------\n+# P1* | s s s s s s s s\n+# P3* | s s s s s s\n+# P5* | s s s s\n+# P7* | s s\n+# ----+--------------------------------------\n+# ALL | x x x x x x x x x x x x x x x x x x x\n+#\n+#############################################################################\n+test_expect_success 'shared: show redundant packs in stderr for verbose mode' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tP1:$P1\n+\t\t\tP3:$P3\n+\t\t\tP5:$P5\n+\t\t\tP7:$P7\n+\t\t\tEOF\n+\t\tgit pack-redundant --all --verbose >out 2>out.err &&\n+\t\ttest_must_be_empty out &&\n+\t\tgrep \"pack$\" out.err | format_packfiles >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'shared: remove redundant packs, no packs left' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tfatal: Zero packs found!\n+\t\t\tEOF\n+\t\tgit pack-redundant --all --alt-odb | xargs rm &&\n+\t\tgit fsck &&\n+\t\ttest_must_fail git pack-redundant --all --alt-odb >actual 2>&1 &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'shared: create new objects and packs' '\n+\tcreate_commits_in \"$shared_repo\" X Y Z &&\n+\tcreate_pack_in \"$shared_repo\" Px1 <<-EOF &&\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\t$A\n+\t\t$B\n+\t\t$C\n+\t\tEOF\n+\tcreate_pack_in \"$shared_repo\" Px2 <<-EOF\n+\t\t$X\n+\t\t$Y\n+\t\t$Z\n+\t\t$D\n+\t\t$E\n+\t\t$F\n+\t\tEOF\n+'\n+\n+test_expect_success 'shared: no redundant without --alt-odb' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tgit pack-redundant --all >out &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# ================ master.git ===============\n+# | T A B C D E F G H I J K L M N O P Q R <----------------+\n+# ----+-------------------------------------- |\n+# P1 | x x x x x x x x |\n+# P3 | x x x x x x |\n+# P5 | x x x x |\n+# P7 | x x |\n+# ----+-------------------------------------- |\n+# ALL | x x x x x x x x x x x x x x x x x x x |\n+# |\n+# |\n+# ================ shared.git ======================= |\n+# | T A B C D E F G H I J K L M N O P Q R X Y Z \n+# ----+----------------------------------------------\n+# Px1 | s s s x x x\n+# Px2*| s s s ! ! !\n+# ----+----------------------------------------------\n+# ALL | s s s s s s s s s s s s s s s s s s s x x x\n+#\n+#############################################################################\n+test_expect_success 'shared: one pack is redundant with --alt-odb' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tgit pack-redundant --all --alt-odb >out &&\n+\t\tformat_packfiles actual &&\n+\t\ttest_line_count = 1 actual\n+\t)\n+'\n+\n+#############################################################################\n+# Chart of packs and objects for this test case\n+#\n+# ================ master.git ===============\n+# | T A B C D E F G H I J K L M N O P Q R <----------------+\n+# ----+-------------------------------------- |\n+# P1 | x x x x x x x x |\n+# P3 | x x x x x x |\n+# P5 | x x x x |\n+# P7 | x x |\n+# ----+-------------------------------------- |\n+# ALL | x x x x x x x x x x x x x x x x x x x |\n+# |\n+# |\n+# ================ shared.git ======================= |\n+# | T A B C D E F G H I J K L M N O P Q R X Y Z \n+# ----+----------------------------------------------\n+# Px1*| s s s i i i\n+# Px2*| s s s i i i\n+# ----+----------------------------------------------\n+# ALL | s s s s s s s s s s s s s s s s s s s i i i\n+# (ignored objects, marked with i)\n+#\n+#############################################################################\n+test_expect_success 'shared: ignore unique objects and all two packs are redundant' '\n+\t(\n+\t\tcd \"$shared_repo\" &&\n+\t\tcat >expect <<-EOF &&\n+\t\t\tPx1:$Px1\n+\t\t\tPx2:$Px2\n+\t\t\tEOF\n+\t\tgit pack-redundant --all --alt-odb >out <<-EOF &&\n+\t\t\t$X\n+\t\t\t$Y\n+\t\t\t$Z\n+\t\t\tEOF\n+\t\tformat_packfiles actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368409","messageId":"20190202133017.1039-3-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 2/6] pack-redundant: delay creation of unique_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:13Z","receivedAt":"2019-02-02T13:31:03Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nInstead of initializing unique_objects in `add_pack()`, copy from\nall_objects in `cmp_two_packs()`, when unwanted objects are removed from\nall_objects.\n\nThis will save memory (no allocate memory for alt-odb packs), and run\n`llist_sorted_difference_inplace()` only once when removing ignored\nobjects and removing objects in alt-odb in `scan_alt_odb_packs()`.\n\nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cf9a9aabd4..f7dab0ec60 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -254,6 +254,11 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \tstruct llist_item *p1_hint = NULL, *p2_hint = NULL;\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n+\tif (!p1->unique_objects)\n+\t\tp1->unique_objects = llist_copy(p1->all_objects);\n+\tif (!p2->unique_objects)\n+\t\tp2->unique_objects = llist_copy(p2->all_objects);\n+\n \tp1_base = p1->pack->index_data;\n \tp2_base = p2->pack->index_data;\n \tp1_base += 256 * 4 + ((p1->pack->index_version < 2) ? 4 : 8);\n@@ -536,7 +541,7 @@ static void scan_alt_odb_packs(void)\n \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n-\t\t\tllist_sorted_difference_inplace(local->unique_objects,\n+\t\t\tllist_sorted_difference_inplace(local->all_objects,\n \t\t\t\t\t\t\talt->all_objects);\n \t\t\tlocal = local->next;\n \t\t}\n@@ -567,8 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n-\t/* this list will be pruned in cmp_two_packs later */\n-\tl.unique_objects = llist_copy(l.all_objects);\n+\tl.unique_objects = NULL;\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\n \telse\n@@ -646,7 +650,6 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \n \tload_all_objects();\n \n-\tcmp_local_packs();\n \tif (alt_odb)\n \t\tscan_alt_odb_packs();\n \n@@ -663,10 +666,12 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->unique_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n+\tcmp_local_packs();\n+\n \tminimize(&min);\n \n \tif (verbose) {\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368410","messageId":"20190202133017.1039-4-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 3/6] pack-redundant: delete redundant code","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:14Z","receivedAt":"2019-02-02T13:31:12Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nThe objects in alt-odb are removed from `all_objects` twice in `load_all_objects`\nand `scan_alt_odb_packs`, remove it from the later function.\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \n---\n builtin/pack-redundant.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex f7dab0ec60..4a06f057dd 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -545,7 +545,6 @@ static void scan_alt_odb_packs(void)\n \t\t\t\t\t\t\talt->all_objects);\n \t\t\tlocal = local->next;\n \t\t}\n-\t\tllist_sorted_difference_inplace(all_objects, alt->all_objects);\n \t\talt = alt->next;\n \t}\n }\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368411","messageId":"20190202133017.1039-5-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 4/6] pack-redundant: new algorithm to find min packs","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:15Z","receivedAt":"2019-02-02T13:31:18Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Sun Chao \n\nWhen calling `git pack-redundant --all`, if there are too many local\npacks and too many redundant objects within them, the too deep iteration\nof `get_permutations` will exhaust all the resources, and the process of\n`git pack-redundant` will be killed.\n\nThe following script could create a repository with too many redundant\npacks, and running `git pack-redundant --all` in the `test.git` repo\nwill die soon.\n\n #!/bin/sh\n\n repo=\"$(pwd)/test.git\"\n work=\"$(pwd)/test\"\n i=1\n max=199\n\n if test -d \"$repo\" || test -d \"$work\"; then\n \techo >&2 \"ERROR: '$repo' or '$work' already exist\"\n \texit 1\n fi\n\n git init -q --bare \"$repo\"\n git --git-dir=\"$repo\" config gc.auto 0\n git --git-dir=\"$repo\" config transfer.unpackLimit 0\n git clone -q \"$repo\" \"$work\" 2>/dev/null\n\n while :; do\n cd \"$work\"\n echo \"loop $i: $(date +%s)\" >$i\n git add $i\n git commit -q -sm \"loop $i\"\n git push -q origin HEAD:master\n printf \"\\rCreate pack %4d/%d\\t\" $i $max\n if test $i -ge $max; then break; fi\n\n cd \"$repo\"\n git repack -q\n if test $(($i % 2)) -eq 0; then\n git repack -aq\n pack=$(ls -t $repo/objects/pack/*.pack | head -1)\n touch \"${pack%.pack}.keep\"\n fi\n i=$((i+1))\n done\n printf \"\\ndone\\n\"\n\nTo get the `min` unique pack list, we can replace the iteration in\n`minimize` function with a new algorithm, and this could solve this\nissue:\n\n1. Get the unique and non_uniqe packs, add the unique packs to the\n `min` list.\n\n2. Remove the objects of unique packs from non_unique packs, then each\n object left in the non_unique packs will have at least two copies.\n\n3. Sort the non_unique packs by the objects' size, more objects first,\n and add the first non_unique pack to `min` list.\n\n4. Drop the duplicated objects from other packs in the ordered\n non_unique pack list, and repeat step 3.\n\nSome test cases will fail on Mac OS X. Mark them and will resolve in\nlater commit.\n\nOriginal PR and discussions: https://github.com/jiangxin/git/pull/25\n\nSigned-off-by: Sun Chao \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 194 +++++++++++++-------------------------\n t/t5323-pack-redundant.sh | 12 +--\n 2 files changed, 73 insertions(+), 133 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 4a06f057dd..d6d9a66e46 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -35,11 +35,6 @@ static struct pack_list {\n \tstruct llist *all_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n-struct pll {\n-\tstruct pll *next;\n-\tstruct pack_list *pl;\n-};\n-\n static struct llist_item *free_nodes;\n \n static inline void llist_item_put(struct llist_item *item)\n@@ -63,15 +58,6 @@ static inline struct llist_item *llist_item_get(void)\n \treturn new_item;\n }\n \n-static void llist_free(struct llist *list)\n-{\n-\twhile ((list->back = list->front)) {\n-\t\tlist->front = list->front->next;\n-\t\tllist_item_put(list->back);\n-\t}\n-\tfree(list);\n-}\n-\n static inline void llist_init(struct llist **list)\n {\n \t*list = xmalloc(sizeof(struct llist));\n@@ -290,78 +276,6 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t}\n }\n \n-static void pll_free(struct pll *l)\n-{\n-\tstruct pll *old;\n-\tstruct pack_list *opl;\n-\n-\twhile (l) {\n-\t\told = l;\n-\t\twhile (l->pl) {\n-\t\t\topl = l->pl;\n-\t\t\tl->pl = opl->next;\n-\t\t\tfree(opl);\n-\t\t}\n-\t\tl = l->next;\n-\t\tfree(old);\n-\t}\n-}\n-\n-/* all the permutations have to be free()d at the same time,\n- * since they refer to each other\n- */\n-static struct pll * get_permutations(struct pack_list *list, int n)\n-{\n-\tstruct pll *subset, *ret = NULL, *new_pll = NULL;\n-\n-\tif (list == NULL || pack_list_size(list) < n || n == 0)\n-\t\treturn NULL;\n-\n-\tif (n == 1) {\n-\t\twhile (list) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = NULL;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tlist = list->next;\n-\t\t}\n-\t\treturn ret;\n-\t}\n-\n-\twhile (list->next) {\n-\t\tsubset = get_permutations(list->next, n - 1);\n-\t\twhile (subset) {\n-\t\t\tnew_pll = xmalloc(sizeof(*new_pll));\n-\t\t\tnew_pll->pl = subset->pl;\n-\t\t\tpack_list_insert(&new_pll->pl, list);\n-\t\t\tnew_pll->next = ret;\n-\t\t\tret = new_pll;\n-\t\t\tsubset = subset->next;\n-\t\t}\n-\t\tlist = list->next;\n-\t}\n-\treturn ret;\n-}\n-\n-static int is_superset(struct pack_list *pl, struct llist *list)\n-{\n-\tstruct llist *diff;\n-\n-\tdiff = llist_copy(list);\n-\n-\twhile (pl) {\n-\t\tllist_sorted_difference_inplace(diff, pl->all_objects);\n-\t\tif (diff->size == 0) { /* we're done */\n-\t\t\tllist_free(diff);\n-\t\t\treturn 1;\n-\t\t}\n-\t\tpl = pl->next;\n-\t}\n-\tllist_free(diff);\n-\treturn 0;\n-}\n-\n static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n@@ -426,14 +340,52 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n+static int cmp_pack_list_reverse(const void *a, const void *b)\n+{\n+\tstruct pack_list *pl_a = *((struct pack_list **)a);\n+\tstruct pack_list *pl_b = *((struct pack_list **)b);\n+\tsize_t sz_a = pl_a->all_objects->size;\n+\tsize_t sz_b = pl_b->all_objects->size;\n+\n+\tif (sz_a == sz_b)\n+\t\treturn 0;\n+\telse if (sz_a < sz_b)\n+\t\treturn 1;\n+\telse\n+\t\treturn -1;\n+}\n+\n+/* Sort pack_list, greater size of all_objects first */\n+static void sort_pack_list(struct pack_list **pl)\n+{\n+\tstruct pack_list **ary, *p;\n+\tint i;\n+\tsize_t n = pack_list_size(*pl);\n+\n+\tif (n < 2)\n+\t\treturn;\n+\n+\t/* prepare an array of packed_list for easier sorting */\n+\tary = xcalloc(n, sizeof(struct pack_list *));\n+\tfor (n = 0, p = *pl; p; p = p->next)\n+\t\tary[n++] = p;\n+\n+\tQSORT(ary, n, cmp_pack_list_reverse);\n+\n+\t/* link them back again */\n+\tfor (i = 0; i < n - 1; i++)\n+\t\tary[i]->next = ary[i + 1];\n+\tary[n - 1]->next = NULL;\n+\t*pl = ary[0];\n+\n+\tfree(ary);\n+}\n+\n+\n static void minimize(struct pack_list **min)\n {\n-\tstruct pack_list *pl, *unique = NULL,\n-\t\t*non_unique = NULL, *min_perm = NULL;\n-\tstruct pll *perm, *perm_all, *perm_ok = NULL, *new_perm;\n-\tstruct llist *missing;\n-\toff_t min_perm_size = 0, perm_size;\n-\tint n;\n+\tstruct pack_list *pl, *unique = NULL, *non_unique = NULL;\n+\tstruct llist *missing, *unique_pack_objects;\n \n \tpl = local_packs;\n \twhile (pl) {\n@@ -451,49 +403,37 @@ static void minimize(struct pack_list **min)\n \t\tpl = pl->next;\n \t}\n \n+\t*min = unique;\n+\n \t/* return if there are no objects missing from the unique set */\n \tif (missing->size == 0) {\n-\t\t*min = unique;\n \t\tfree(missing);\n \t\treturn;\n \t}\n \n-\t/* find the permutations which contain all missing objects */\n-\tfor (n = 1; n <= pack_list_size(non_unique) && !perm_ok; n++) {\n-\t\tperm_all = perm = get_permutations(non_unique, n);\n-\t\twhile (perm) {\n-\t\t\tif (is_superset(perm->pl, missing)) {\n-\t\t\t\tnew_perm = xmalloc(sizeof(struct pll));\n-\t\t\t\tmemcpy(new_perm, perm, sizeof(struct pll));\n-\t\t\t\tnew_perm->next = perm_ok;\n-\t\t\t\tperm_ok = new_perm;\n-\t\t\t}\n-\t\t\tperm = perm->next;\n-\t\t}\n-\t\tif (perm_ok)\n-\t\t\tbreak;\n-\t\tpll_free(perm_all);\n-\t}\n-\tif (perm_ok == NULL)\n-\t\tdie(\"Internal error: No complete sets found!\");\n-\n-\t/* find the permutation with the smallest size */\n-\tperm = perm_ok;\n-\twhile (perm) {\n-\t\tperm_size = pack_set_bytecount(perm->pl);\n-\t\tif (!min_perm_size || min_perm_size > perm_size) {\n-\t\t\tmin_perm_size = perm_size;\n-\t\t\tmin_perm = perm->pl;\n-\t\t}\n-\t\tperm = perm->next;\n-\t}\n-\t*min = min_perm;\n-\t/* add the unique packs to the list */\n-\tpl = unique;\n+\tunique_pack_objects = llist_copy(all_objects);\n+\tllist_sorted_difference_inplace(unique_pack_objects, missing);\n+\n+\t/* remove unique pack objects from the non_unique packs */\n+\tpl = non_unique;\n \twhile (pl) {\n-\t\tpack_list_insert(min, pl);\n+\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n+\n+\twhile (non_unique) {\n+\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\tsort_pack_list(&non_unique);\n+\t\tif (non_unique->all_objects->size == 0)\n+\t\t\tbreak;\n+\n+\t\tpack_list_insert(min, non_unique);\n+\n+\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\n+\t\tnon_unique = non_unique->next;\n+\t}\n }\n \n static void load_all_objects(void)\n@@ -606,7 +546,7 @@ static void load_all(void)\n int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct pack_list *min, *red, *pl;\n+\tstruct pack_list *min = NULL, *red, *pl;\n \tstruct llist *ignore;\n \tstruct object_id *oid;\n \tchar buf[GIT_MAX_HEXSZ + 2]; /* hex hash + \\n + \\0 */\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex f18db06d91..3e62e8663f 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -173,7 +173,7 @@ test_expect_success 'master: no redundant for pack 1, 2, 3' '\n # ALL | x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n+test_expect_failure 'master: one of pack-2/pack-3 is redundant (failed on Mac)' '\n \tcreate_pack_in \"$master_repo\" P4 <<-EOF &&\n \t\t$J\n \t\t$K\n@@ -214,7 +214,7 @@ test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n+test_expect_failure 'master: pack 2, 4, and 6 are redundant (failed on Mac)' '\n \tcreate_pack_in \"$master_repo\" P6 <<-EOF &&\n \t\t$N\n \t\t$O\n@@ -254,7 +254,7 @@ test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'master: pack-8 (subset of pack-1) is also redundant' '\n+test_expect_failure 'master: pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n \tcreate_pack_in \"$master_repo\" P8 <<-EOF &&\n \t\t$A\n \t\tEOF\n@@ -281,7 +281,7 @@ test_expect_success 'master: clean loose objects' '\n \t)\n '\n \n-test_expect_success 'master: remove redundant packs and pass fsck' '\n+test_expect_failure 'master: remove redundant packs and pass fsck (failed on Mac)' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tgit pack-redundant --all | xargs rm &&\n@@ -301,7 +301,7 @@ test_expect_success 'setup shared.git' '\n \t)\n '\n \n-test_expect_success 'shared: all packs are redundant, but no output without --alt-odb' '\n+test_expect_failure 'shared: all packs are redundant, but no output without --alt-odb (failed on Mac)' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tgit pack-redundant --all >out &&\n@@ -334,7 +334,7 @@ test_expect_success 'shared: all packs are redundant, but no output without --al\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_success 'shared: show redundant packs in stderr for verbose mode' '\n+test_expect_failure 'shared: show redundant packs in stderr for verbose mode (failed on Mac)' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tcat >expect <<-EOF &&\n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368412","messageId":"20190202133017.1039-6-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 5/6] pack-redundant: rename pack_list.all_objects","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:16Z","receivedAt":"2019-02-02T13:31:25Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nNew algorithm uses `pack_list.all_objects` to track remaining objects,\nso rename it to `pack_list.remaining_objects`.\n\nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex d6d9a66e46..15cdf233c4 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -32,7 +32,7 @@ static struct pack_list {\n \tstruct pack_list *next;\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n-\tstruct llist *all_objects;\n+\tstruct llist *remaining_objects;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -241,9 +241,9 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \tconst unsigned int hashsz = the_hash_algo->rawsz;\n \n \tif (!p1->unique_objects)\n-\t\tp1->unique_objects = llist_copy(p1->all_objects);\n+\t\tp1->unique_objects = llist_copy(p1->remaining_objects);\n \tif (!p2->unique_objects)\n-\t\tp2->unique_objects = llist_copy(p2->all_objects);\n+\t\tp2->unique_objects = llist_copy(p2->remaining_objects);\n \n \tp1_base = p1->pack->index_data;\n \tp2_base = p2->pack->index_data;\n@@ -344,8 +344,8 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->all_objects->size;\n-\tsize_t sz_b = pl_b->all_objects->size;\n+\tsize_t sz_a = pl_a->remaining_objects->size;\n+\tsize_t sz_b = pl_b->remaining_objects->size;\n \n \tif (sz_a == sz_b)\n \t\treturn 0;\n@@ -355,7 +355,7 @@ static int cmp_pack_list_reverse(const void *a, const void *b)\n \t\treturn -1;\n }\n \n-/* Sort pack_list, greater size of all_objects first */\n+/* Sort pack_list, greater size of remaining_objects first */\n static void sort_pack_list(struct pack_list **pl)\n {\n \tstruct pack_list **ary, *p;\n@@ -399,7 +399,7 @@ static void minimize(struct pack_list **min)\n \tmissing = llist_copy(all_objects);\n \tpl = unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(missing, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(missing, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n \n@@ -417,20 +417,20 @@ static void minimize(struct pack_list **min)\n \t/* remove unique pack objects from the non_unique packs */\n \tpl = non_unique;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, unique_pack_objects);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, unique_pack_objects);\n \t\tpl = pl->next;\n \t}\n \n \twhile (non_unique) {\n-\t\t/* sort the non_unique packs, greater size of all_objects first */\n+\t\t/* sort the non_unique packs, greater size of remaining_objects first */\n \t\tsort_pack_list(&non_unique);\n-\t\tif (non_unique->all_objects->size == 0)\n+\t\tif (non_unique->remaining_objects->size == 0)\n \t\t\tbreak;\n \n \t\tpack_list_insert(min, non_unique);\n \n-\t\tfor (pl = non_unique->next; pl && pl->all_objects->size > 0; pl = pl->next)\n-\t\t\tllist_sorted_difference_inplace(pl->all_objects, non_unique->all_objects);\n+\t\tfor (pl = non_unique->next; pl && pl->remaining_objects->size > 0; pl = pl->next)\n+\t\t\tllist_sorted_difference_inplace(pl->remaining_objects, non_unique->remaining_objects);\n \n \t\tnon_unique = non_unique->next;\n \t}\n@@ -445,7 +445,7 @@ static void load_all_objects(void)\n \n \twhile (pl) {\n \t\thint = NULL;\n-\t\tl = pl->all_objects->front;\n+\t\tl = pl->remaining_objects->front;\n \t\twhile (l) {\n \t\t\thint = llist_insert_sorted_unique(all_objects,\n \t\t\t\t\t\t\t l->oid, hint);\n@@ -456,7 +456,7 @@ static void load_all_objects(void)\n \t/* remove objects present in remote packs */\n \tpl = altodb_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(all_objects, pl->all_objects);\n+\t\tllist_sorted_difference_inplace(all_objects, pl->remaining_objects);\n \t\tpl = pl->next;\n \t}\n }\n@@ -481,8 +481,8 @@ static void scan_alt_odb_packs(void)\n \twhile (alt) {\n \t\tlocal = local_packs;\n \t\twhile (local) {\n-\t\t\tllist_sorted_difference_inplace(local->all_objects,\n-\t\t\t\t\t\t\talt->all_objects);\n+\t\t\tllist_sorted_difference_inplace(local->remaining_objects,\n+\t\t\t\t\t\t\talt->remaining_objects);\n \t\t\tlocal = local->next;\n \t\t}\n \t\talt = alt->next;\n@@ -499,7 +499,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\treturn NULL;\n \n \tl.pack = p;\n-\tllist_init(&l.all_objects);\n+\tllist_init(&l.remaining_objects);\n \n \tif (open_pack_index(p))\n \t\treturn NULL;\n@@ -508,7 +508,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \tbase += 256 * 4 + ((p->index_version < 2) ? 4 : 8);\n \tstep = the_hash_algo->rawsz + ((p->index_version < 2) ? 4 : 0);\n \twhile (off < p->num_objects * step) {\n-\t\tllist_insert_back(l.all_objects, (const struct object_id *)(base + off));\n+\t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n \tl.unique_objects = NULL;\n@@ -605,7 +605,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)\n \tllist_sorted_difference_inplace(all_objects, ignore);\n \tpl = local_packs;\n \twhile (pl) {\n-\t\tllist_sorted_difference_inplace(pl->all_objects, ignore);\n+\t\tllist_sorted_difference_inplace(pl->remaining_objects, ignore);\n \t\tpl = pl->next;\n \t}\n \n-- \n2.20.1.103.ged0fc2ca7b\n\n"},{"id":"368413","messageId":"20190202133017.1039-7-worldhello.net@gmail.com","threadId":"50055","inReplyTo":"CAPig+cQh5TDKVaDi0gg9LZTo1Og_Qw6S2sH9cPABR9q05gEUfg@mail.gmail.com","subject":"[PATCH v10 6/6] pack-redundant: consistent sort method","fromName":"Jiang Xin","fromEmail":"worldhello.net@gmail.com","sentAt":"2019-02-02T13:30:17Z","receivedAt":"2019-02-02T13:31:36Z","isPatch":true,"sender":{"key":"worldhello.net@gmail.com","avatar":"https://avatars.githubusercontent.com/u/183860?v=4"},"body":"From: Jiang Xin \n\nSZEDER reported that test case t5323 has different test result on MacOS.\nThis is because `cmp_pack_list_reverse` cannot give identical result\nwhen two pack being sorted has the same size of remaining_objects.\n\nChanges to the sorting function will make consistent test result for\nt5323.\n\nThe new algorithm to find redundant packs is a trade-off to save memory\nresources, and the result of it may be different with old one, and may\nbe not the best result sometimes. Update t5323 for the new algorithm.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Jiang Xin \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-redundant.c | 24 ++++++++++++++++--------\n t/t5323-pack-redundant.sh | 18 +++++++++---------\n 2 files changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex 15cdf233c4..29ff5e99cb 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -33,6 +33,7 @@ static struct pack_list {\n \tstruct packed_git *pack;\n \tstruct llist *unique_objects;\n \tstruct llist *remaining_objects;\n+\tsize_t all_objects_size;\n } *local_packs = NULL, *altodb_packs = NULL;\n \n static struct llist_item *free_nodes;\n@@ -340,19 +341,25 @@ static inline off_t pack_set_bytecount(struct pack_list *pl)\n \treturn ret;\n }\n \n-static int cmp_pack_list_reverse(const void *a, const void *b)\n+static int cmp_remaining_objects(const void *a, const void *b)\n {\n \tstruct pack_list *pl_a = *((struct pack_list **)a);\n \tstruct pack_list *pl_b = *((struct pack_list **)b);\n-\tsize_t sz_a = pl_a->remaining_objects->size;\n-\tsize_t sz_b = pl_b->remaining_objects->size;\n \n-\tif (sz_a == sz_b)\n-\t\treturn 0;\n-\telse if (sz_a < sz_b)\n+\tif (pl_a->remaining_objects->size == pl_b->remaining_objects->size) {\n+\t\t/* have the same remaining_objects, big pack first */\n+\t\tif (pl_a->all_objects_size == pl_b->all_objects_size)\n+\t\t\treturn 0;\n+\t\telse if (pl_a->all_objects_size < pl_b->all_objects_size)\n+\t\t\treturn 1;\n+\t\telse\n+\t\t\treturn -1;\n+\t} else if (pl_a->remaining_objects->size < pl_b->remaining_objects->size) {\n+\t\t/* sort by remaining objects, more objects first */\n \t\treturn 1;\n-\telse\n+\t} else {\n \t\treturn -1;\n+\t}\n }\n \n /* Sort pack_list, greater size of remaining_objects first */\n@@ -370,7 +377,7 @@ static void sort_pack_list(struct pack_list **pl)\n \tfor (n = 0, p = *pl; p; p = p->next)\n \t\tary[n++] = p;\n \n-\tQSORT(ary, n, cmp_pack_list_reverse);\n+\tQSORT(ary, n, cmp_remaining_objects);\n \n \t/* link them back again */\n \tfor (i = 0; i < n - 1; i++)\n@@ -511,6 +518,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \t\tllist_insert_back(l.remaining_objects, (const struct object_id *)(base + off));\n \t\toff += step;\n \t}\n+\tl.all_objects_size = l.remaining_objects->size;\n \tl.unique_objects = NULL;\n \tif (p->pack_local)\n \t\treturn pack_list_insert(&local_packs, &l);\ndiff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh\nindex 3e62e8663f..384b244314 100755\n--- a/t/t5323-pack-redundant.sh\n+++ b/t/t5323-pack-redundant.sh\n@@ -165,15 +165,15 @@ test_expect_success 'master: no redundant for pack 1, 2, 3' '\n # | T A B C D E F G H I J K L M N O P Q R\n # ----+--------------------------------------\n # P1 | x x x x x x x x\n-# P2* | ! ! ! ! ! ! !\n-# P3 | x x x x x x\n+# P2 | x x x x x x x\n+# P3* | ! ! ! ! ! !\n # P4 | x x x x x\n # P5 | x x x x\n # ----+--------------------------------------\n # ALL | x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'master: one of pack-2/pack-3 is redundant (failed on Mac)' '\n+test_expect_success 'master: one of pack-2/pack-3 is redundant' '\n \tcreate_pack_in \"$master_repo\" P4 <<-EOF &&\n \t\t$J\n \t\t$K\n@@ -190,7 +190,7 @@ test_expect_failure 'master: one of pack-2/pack-3 is redundant (failed on Mac)'\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tcat >expect <<-EOF &&\n-\t\t\tP2:$P2\n+\t\t\tP3:$P3\n \t\t\tEOF\n \t\tgit pack-redundant --all >out &&\n \t\tformat_packfiles actual &&\n@@ -214,7 +214,7 @@ test_expect_failure 'master: one of pack-2/pack-3 is redundant (failed on Mac)'\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'master: pack 2, 4, and 6 are redundant (failed on Mac)' '\n+test_expect_success 'master: pack 2, 4, and 6 are redundant' '\n \tcreate_pack_in \"$master_repo\" P6 <<-EOF &&\n \t\t$N\n \t\t$O\n@@ -254,7 +254,7 @@ test_expect_failure 'master: pack 2, 4, and 6 are redundant (failed on Mac)' '\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'master: pack-8 (subset of pack-1) is also redundant (failed on Mac)' '\n+test_expect_success 'master: pack-8 (subset of pack-1) is also redundant' '\n \tcreate_pack_in \"$master_repo\" P8 <<-EOF &&\n \t\t$A\n \t\tEOF\n@@ -281,7 +281,7 @@ test_expect_success 'master: clean loose objects' '\n \t)\n '\n \n-test_expect_failure 'master: remove redundant packs and pass fsck (failed on Mac)' '\n+test_expect_success 'master: remove redundant packs and pass fsck' '\n \t(\n \t\tcd \"$master_repo\" &&\n \t\tgit pack-redundant --all | xargs rm &&\n@@ -301,7 +301,7 @@ test_expect_success 'setup shared.git' '\n \t)\n '\n \n-test_expect_failure 'shared: all packs are redundant, but no output without --alt-odb (failed on Mac)' '\n+test_expect_success 'shared: all packs are redundant, but no output without --alt-odb' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tgit pack-redundant --all >out &&\n@@ -334,7 +334,7 @@ test_expect_failure 'shared: all packs are redundant, but no output without --al\n # ALL | x x x x x x x x x x x x x x x x x x x\n #\n #############################################################################\n-test_expect_failure 'shared: show redundant packs in stderr for verbose mode (failed on Mac)' '\n+test_expect_success 'shared: show redundant packs in stderr for verbose mode' '\n \t(\n \t\tcd \"$shared_repo\" &&\n \t\tcat >expect <<-EOF &&\n-- \n2.20.1.103.ged0fc2ca7b\n\n"}]}