{"thread":{"id":"48285","subject":"[PATCH 3/7] gc: add --keep-largest-pack option","startedAt":"2018-04-14T15:27:12Z","lastAt":"2018-04-16T04:51:19Z","messageCount":19,"participants":["Nguyễn Thái Ngọc Duy","Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"344663","messageId":"20180414152642.4666-4-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 3/7] gc: add --keep-largest-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:38Z","receivedAt":"2018-04-14T15:27:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This adds a new repack mode that combines everything into a secondary\npack, leaving the largest pack alone.\n\nThis could help reduce memory pressure. On linux-2.6.git, valgrind\nmassif reports 1.6GB heap in \"pack all\" case, and 535MB in \"pack\nall except the base pack\" case. We save roughly 1GB memory by\nexcluding the base pack.\n\nThis should also lower I/O because we don't have to rewrite a giant\npack every time (e.g. for linux-2.6.git that's a 1.4GB pack file)..\n\nPS. The use of string_list here seems overkill, but we'll need it in\nthe next patch...\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 6 +++++-\n builtin/gc.c | 45 ++++++++++++++++++++++++++++++++++++----\n t/t6500-gc.sh | 25 ++++++++++++++++++++++\n 3 files changed, 71 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 3126e0dd00..8f903231da 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force]\n+'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -84,6 +84,10 @@ be performed as well.\n \tForce `git gc` to run even if there may be another `git gc`\n \tinstance running on this repository.\n \n+--keep-largest-pack::\n+\tAll packs except the largest pack and those marked with a\n+\t`.keep` files are consolidated into a single pack.\n+\n Configuration\n -------------\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3e67124eaa..f251662a8f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -166,6 +166,22 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n+static void find_base_packs(struct string_list *packs)\n+{\n+\tstruct packed_git *p, *base = NULL;\n+\n+\tfor (p = get_packed_git(the_repository); p; p = p->next) {\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\t\tbase = p;\n+\t\t}\n+\t}\n+\n+\tif (base)\n+\t\tstring_list_append(packs, base->pack_name);\n+}\n+\n static int too_many_packs(void)\n {\n \tstruct packed_git *p;\n@@ -188,7 +204,13 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static void add_repack_all_option(void)\n+static int keep_one_pack(struct string_list_item *item, void *data)\n+{\n+\targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n+\treturn 0;\n+}\n+\n+static void add_repack_all_option(struct string_list *keep_pack)\n {\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n@@ -197,6 +219,9 @@ static void add_repack_all_option(void)\n \t\tif (prune_expire)\n \t\t\targv_array_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n \t}\n+\n+\tif (keep_pack)\n+\t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n }\n \n static void add_repack_incremental_option(void)\n@@ -220,7 +245,7 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs())\n-\t\tadd_repack_all_option();\n+\t\tadd_repack_all_option(NULL);\n \telse if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n@@ -354,6 +379,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tconst char *name;\n \tpid_t pid;\n \tint daemonized = 0;\n+\tint keep_base_pack = -1;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n@@ -366,6 +392,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL_F(0, \"force\", &force,\n \t\t\t N_(\"force running gc even if there may be another gc running\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL(0, \"keep-largest-pack\", &keep_base_pack,\n+\t\t\t N_(\"repack all other packs except the largest pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -431,8 +459,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t */\n \t\t\tdaemonized = !daemonize();\n \t\t}\n-\t} else\n-\t\tadd_repack_all_option();\n+\t} else {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (keep_base_pack != -1) {\n+\t\t\tif (keep_base_pack)\n+\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t}\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t}\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex d5255dd576..c42f60bc5b 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -43,6 +43,31 @@ test_expect_success 'gc is not aborted due to a stale symref' '\n \t)\n '\n \n+test_expect_success 'gc --keep-largest-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\t\ttest_commit three &&\n+\t\tgit gc &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 1 pack-list &&\n+\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d &&\n+\t\ttest_commit five &&\n+\t\tgit repack -d &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 3 pack-list &&\n+\t\tgit gc --keep-largest-pack &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 2 pack-list &&\n+\t\ttest_path_is_file $BASE_PACK &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_expect_success 'auto gc with too many loose objects does not attempt to create bitmaps' '\n \ttest_config gc.auto 3 &&\n \ttest_config gc.autodetach false &&\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344664","messageId":"20180414152642.4666-3-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 2/7] repack: add --keep-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:37Z","receivedAt":"2018-04-14T15:27:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We allow to keep existing packs by having companion .keep files. This\nis helpful when a pack is permanently kept. In the next patch, git-gc\njust wants to keep a pack temporarily, for one pack-objects\nrun. git-gc can use --keep-pack for this use case.\n\nA note about why the pack_keep field cannot be reused and\npack_keep_in_core has to be added. This is about the case when\n--keep-pack is specified together with either --keep-unreachable or\n--unpack-unreachable, but --honor-pack-keep is NOT specified.\n\nIn this case, we want to exclude objects from the packs specified on\ncommand line, not from ones with .keep files. If only one bit flag is\nused, we have to clear pack_keep on pack files with the .keep file.\n\nBut we can't make any assumption about unreachable objects in .keep\npacks. If \"pack_keep\" field is false for .keep packs, we could\npotentially pull lots of unreachable objects into the new pack, or\nunpack them loose. The safer approach is ignore all packs with either\n.keep file or --keep-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++-\n Documentation/git-repack.txt | 9 ++++-\n builtin/pack-objects.c | 63 ++++++++++++++++++++++++------\n builtin/repack.c | 21 ++++++++--\n object-store.h | 1 +\n t/t7700-repack.sh | 25 ++++++++++++\n 6 files changed, 110 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..403524652a 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n-\t[--revs [--unpacked | --all]]\n+\t[--revs [--unpacked | --all]] [--keep-pack=]\n \t[--stdout [--filter=] | base-name]\n \t[--shallow] [--keep-true-parents] < object-list\n \n@@ -126,6 +126,13 @@ base-name::\n \thas a .keep file to be ignored, even if it would have\n \totherwise been packed.\n \n+--keep-pack=::\n+\tThis flag causes an object already in the given pack to be\n+\tignored, even if it would have otherwise been\n+\tpacked. `` is the the pack file name without\n+\tleading directory (e.g. `pack-123.pack`). The option could be\n+\tspecified multiple times to keep multiple packs.\n+\n --incremental::\n \tThis flag causes an object already in a pack to be ignored\n \teven if it would have otherwise been packed.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..ce497d9d12 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=] [--keep-pack=]\n \n DESCRIPTION\n -----------\n@@ -133,6 +133,13 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n+--keep-pack=::\n+\tExclude the given pack from repacking. This is the equivalent\n+\tof having `.keep` file on the pack. `` is the the\n+\tpack file name without leading directory (e.g. `pack-123.pack`).\n+\tThe option could be specified multiple times to keep multiple\n+\tpacks.\n+\n --unpack-unreachable=::\n \tWhen loosening unreachable objects, do not bother loosening any\n \tobjects older than ``. This can be used to optimize out\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4bdae5a1d8..9b9a6d6268 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,7 @@\n #include \"list.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"dir.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -55,7 +56,8 @@ static int pack_loose_unreachable;\n static int local;\n static int have_non_local_packs;\n static int incremental;\n-static int ignore_packed_keep;\n+static int ignore_packed_keep_on_disk;\n+static int ignore_packed_keep_in_core;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -982,13 +984,16 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n \t * not know either way, and it needs to check more packs.\n \t */\n-\tif (!ignore_packed_keep &&\n+\tif (!ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs))\n \t\treturn 1;\n \n \tif (local && !p->pack_local)\n \t\treturn 0;\n-\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\tif (p->pack_local &&\n+\t ((ignore_packed_keep_on_disk && p->pack_keep) ||\n+\t (ignore_packed_keep_in_core && p->pack_keep_in_core)))\n \t\treturn 0;\n \n \t/* we don't know yet; keep looking for more packs */\n@@ -2675,7 +2680,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \t\tstruct object_id oid;\n \t\tstruct object *o;\n \n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \t\tif (open_pack_index(p))\n \t\t\tdie(\"cannot open pack index\");\n@@ -2738,7 +2743,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\t\t\tget_packed_git(the_repository);\n \n \twhile (p) {\n-\t\tif ((!p->pack_local || p->pack_keep) &&\n+\t\tif ((!p->pack_local || p->pack_keep ||\n+\t\t\t\tp->pack_keep_in_core) &&\n \t\t\tfind_pack_entry_one(oid->hash, p)) {\n \t\t\tlast_found = p;\n \t\t\treturn 1;\n@@ -2781,7 +2787,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \tstruct object_id oid;\n \n \tfor (p = get_packed_git(the_repository); p; p = p->next) {\n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n \t\tif (open_pack_index(p))\n@@ -2807,7 +2813,8 @@ static int pack_options_allow_reuse(void)\n {\n \treturn pack_to_stdout &&\n \t allow_ofs_delta &&\n-\t !ignore_packed_keep &&\n+\t !ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs) &&\n \t !incremental;\n }\n@@ -2916,6 +2923,32 @@ static void get_object_list(int ac, const char **av)\n \toid_array_clear(&recent_objects);\n }\n \n+static void add_extra_kept_packs(const struct string_list *names)\n+{\n+\tstruct packed_git *p;\n+\n+\tif (!names->nr)\n+\t\treturn;\n+\n+\tfor (p = get_packed_git(the_repository); p; p = p->next) {\n+\t\tconst char *name = basename(p->pack_name);\n+\t\tint i;\n+\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < names->nr; i++)\n+\t\t\tif (!fspathcmp(name, names->items[i].string))\n+\t\t\t\tbreak;\n+\n+\t\tif (i < names->nr) {\n+\t\t\tp->pack_keep_in_core = 1;\n+\t\t\tignore_packed_keep_in_core = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+}\n+\n static int option_parse_index_version(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -2955,6 +2988,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct option pack_objects_options[] = {\n \t\tOPT_SET_INT('q', \"quiet\", &progress,\n \t\t\t N_(\"do not show progress meter\"), 0),\n@@ -3019,8 +3053,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n-\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n+\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"ignore this pack\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n \t\t\t N_(\"pack compression level\")),\n \t\tOPT_SET_INT(0, \"keep-true-parents\", &grafts_replace_parents,\n@@ -3148,19 +3184,20 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (progress && all_progress_implied)\n \t\tprogress = 2;\n \n-\tif (ignore_packed_keep) {\n+\tadd_extra_kept_packs(&keep_pack_list);\n+\tif (ignore_packed_keep_on_disk) {\n \t\tstruct packed_git *p;\n \t\tfor (p = get_packed_git(the_repository); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n-\t\t\tignore_packed_keep = 0;\n+\t\t\tignore_packed_keep_on_disk = 0;\n \t}\n \tif (local) {\n \t\t/*\n-\t\t * unlike ignore_packed_keep above, we do not want to\n-\t\t * unset \"local\" based on looking at packs, as it\n-\t\t * also covers non-local objects\n+\t\t * unlike ignore_packed_keep_on_disk above, we do not\n+\t\t * want to unset \"local\" based on looking at packs, as\n+\t\t * it also covers non-local objects\n \t\t */\n \t\tstruct packed_git *p;\n \t\tfor (p = get_packed_git(the_repository); p; p = p->next) {\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 7bdb40142f..6c636e159e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -86,7 +86,8 @@ static void remove_pack_on_signal(int signo)\n * have a corresponding .keep or .promisor file. These packs are not to\n * be kept if we are going to pack everything into one file.\n */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list)\n+static void get_non_kept_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t\tconst struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -97,6 +98,14 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n \n \twhile ((e = readdir(dir)) != NULL) {\n \t\tsize_t len;\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < extra_keep->nr; i++)\n+\t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n+\t\t\t\tbreak;\n+\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n+\t\t\tcontinue;\n+\n \t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n \t\t\tcontinue;\n \n@@ -148,7 +157,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tint ext, ret, failed;\n+\tint i, ext, ret, failed;\n \tFILE *out;\n \n \t/* variables to be filled by option parsing */\n@@ -160,6 +169,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tconst char *depth = NULL;\n \tconst char *threads = NULL;\n \tconst char *max_pack_size = NULL;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tint no_reuse_delta = 0, no_reuse_object = 0;\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n@@ -200,6 +210,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"maximum size of each packfile\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -230,6 +242,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--keep-true-parents\");\n \tif (!pack_kept_objects)\n \t\targv_array_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list.nr; i++)\n+\t\targv_array_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t\t keep_pack_list.items[i].string);\n \targv_array_push(&cmd.args, \"--non-empty\");\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n@@ -254,7 +269,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&cmd.args, \"--write-bitmap-index\");\n \n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs);\n+\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n \t\t\tif (unpack_unreachable) {\ndiff --git a/object-store.h b/object-store.h\nindex fef33f345f..0a4dbb74d3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -71,6 +71,7 @@ struct packed_git {\n \tint pack_fd;\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n+\t\t pack_keep_in_core:1,\n \t\t freshened:1,\n \t\t do_not_close:1,\n \t\t pack_promisor:1;\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 38247afbec..6162e2a8e6 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -4,6 +4,12 @@ test_description='git repack works correctly'\n \n . ./test-lib.sh\n \n+commit_and_pack() {\n+\ttest_commit \"$@\" >/dev/null &&\n+\tSHA1=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack file1 &&\n \techo content2 > file2 &&\n@@ -196,5 +202,24 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit cat-file -t $H1\n '\n \n+test_expect_success 'repack --keep-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\tP1=$(commit_and_pack 1) &&\n+\t\tP2=$(commit_and_pack 2) &&\n+\t\tP3=$(commit_and_pack 3) &&\n+\t\tP4=$(commit_and_pack 4) &&\n+\t\tls .git/objects/pack/*.pack >old-counts &&\n+\t\ttest_line_count = 4 old-counts &&\n+\t\tgit repack -a -d --keep-pack $P1 --keep-pack $P4 &&\n+\t\tls .git/objects/pack/*.pack >new-counts &&\n+\t\tgrep -q $P1 new-counts &&\n+\t\tgrep -q $P4 new-counts &&\n+\t\ttest_line_count = 3 new-counts &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n \n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344665","messageId":"20180414152642.4666-8-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 7/7] pack-objects: show some progress when counting kept objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:42Z","receivedAt":"2018-04-14T15:27:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only show progress when there are new objects to be packed. But\nwhen --keep-pack is specified on the base pack, we will exclude most\nof objects. This makes 'pack-objects' stay silent for a long time\nwhile the counting phase is going.\n\nLet's show some progress whenever we visit an object instead. The old\n\"Counting objects\" is renamed to \"Enumerating objects\" and a new\nprogress \"Counting objects\" line is added.\n\nThis new \"Counting objects\" line should progress pretty quick when the\nsystem is beefy. But when the system is under pressure, the reading\nobject header done in this phase could be slow and showing progress is\nan improvement over staying silent in the current code.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c77bea404d..6a1346c41f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -46,7 +46,7 @@ static const char *pack_usage[] = {\n static struct packing_data to_pack;\n \n static struct pack_idx_entry **written_list;\n-static uint32_t nr_result, nr_written;\n+static uint32_t nr_result, nr_written, nr_seen;\n \n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n@@ -1096,6 +1096,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, exclude, &index_pos))\n \t\treturn 0;\n \n@@ -1111,8 +1113,6 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tcreate_object_entry(oid, type, pack_name_hash(name),\n \t\t\t exclude, name && no_try_delta(name),\n \t\t\t index_pos, found_pack, found_offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1123,6 +1123,8 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n {\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, 0, &index_pos))\n \t\treturn 0;\n \n@@ -1130,8 +1132,6 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, index_pos, pack, offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1716,6 +1716,10 @@ static void get_object_details(void)\n \tuint32_t i;\n \tstruct object_entry **sorted_by_offset;\n \n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Counting objects\"),\n+\t\t\t\t\t\tto_pack.nr_objects);\n+\n \tsorted_by_offset = xcalloc(to_pack.nr_objects, sizeof(struct object_entry *));\n \tfor (i = 0; i < to_pack.nr_objects; i++)\n \t\tsorted_by_offset[i] = to_pack.objects + i;\n@@ -1726,7 +1730,9 @@ static void get_object_details(void)\n \t\tcheck_object(entry);\n \t\tif (big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n+\t\tdisplay_progress(progress_state, i + 1);\n \t}\n+\tstop_progress(&progress_state);\n \n \t/*\n \t * This must happen in a second pass, since we rely on the delta\n@@ -3209,7 +3215,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (progress)\n-\t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n+\t\tprogress_state = start_progress(_(\"Enumerating objects\"), 0);\n \tif (!use_internal_rev_list)\n \t\tread_object_list_from_stdin();\n \telse {\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344666","messageId":"20180414152642.4666-6-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 5/7] gc: handle a corner case in gc.bigPackThreshold","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:40Z","receivedAt":"2018-04-14T15:27:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This config allows us to keep packs back if their size is larger\nthan a limit. But if this N >= gc.autoPackLimit, we may have a\nproblem. We are supposed to reduce the number of packs after a\nthreshold because it affects performance.\n\nWe could tell the user that they have incompatible gc.bigPackThreshold\nand gc.autoPackLimit, but it's kinda hard when 'git gc --auto' runs in\nbackground. Instead let's fall back to the next best stategy: try to\nreduce the number of packs anyway, but keep the base pack out. This\nreduces the number of packs to two and hopefully won't take up too\nmuch resources to repack (the assumption still is the base pack takes\nmost resources to handle).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 6 +++++-\n builtin/gc.c | 8 +++++++-\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 40516675b6..4c3f1d7651 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1564,7 +1564,11 @@ gc.bigPackThreshold::\n \texcept that all packs that meet the threshold are kept, not\n \tjust the base pack. Defaults to zero. Common unit suffixes of\n \t'k', 'm', or 'g' are supported.\n-\n++\n+Note that if the number of kept packs is more than gc.autoPackLimit,\n+this configuration variable is ignored, all packs except the base pack\n+will be repacked. After this the number of packs should go below\n+gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 81ecdc5ffa..23c17ba7e9 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -253,8 +253,14 @@ static int need_to_gc(void)\n \tif (too_many_packs()) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold)\n+\t\tif (big_pack_threshold) {\n \t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n+\t\t\t\tbig_pack_threshold = 0;\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t}\n+\t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344667","messageId":"20180414152642.4666-7-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 6/7] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:41Z","receivedAt":"2018-04-14T15:27:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack-objects could be a big memory hog especially on large repos,\neverybody knows that. The suggestion to stick a .keep file on the\ngiant base pack to avoid this problem is also known for a long time.\n\nRecent patches add an option to do just this, but it has to be either\nconfigured or activated manually. This patch lets `git gc --auto`\nactivate this mode automatically when it thinks `repack -ad` will use\na lot of memory and start affecting the system due to swapping or\nflushing OS cache.\n\ngc --auto decides to do this based on an estimation of pack-objects\nmemory usage, which is quite accurate at least for the heap part, and\nwhether that fits in half of system memory (the assumption here is for\ndesktop environment where there are many other applications running).\n\nThis mechanism only kicks in if gc.bigBasePackThreshold is not configured.\nIf it is, it is assumed that the user already knows what they want.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 9 +++-\n builtin/gc.c | 98 +++++++++++++++++++++++++++++++++++++++-\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 ++\n pack-objects.h | 2 +\n t/t6500-gc.sh | 7 +++\n 7 files changed, 119 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 649faddfa6..0fb1d43b2c 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -59,8 +59,13 @@ If the number of packs exceeds the value of `gc.autoPackLimit`,\n then existing packs (except those marked with a `.keep` file\n or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n-'git repack'. Setting `gc.autoPackLimit` to 0 disables\n-automatic consolidation of packs.\n+'git repack'.\n+If the amount of memory is estimated not enough for `git repack` to\n+run smoothly and `gc.bigPackThreshold` is not set, the largest\n+pack will also be excluded (this is the equivalent of running `git gc`\n+with `--keep-base-pack`).\n+Setting `gc.autoPackLimit` to 0 disables automatic consolidation of\n+packs.\n +\n If houskeeping is required due to many loose objects or packs, all\n other housekeeping tasks (e.g. rerere, working trees, reflog...) will\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 23c17ba7e9..3c7c93e961 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -22,6 +22,10 @@\n #include \"commit.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"pack.h\"\n+#include \"pack-objects.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -42,6 +46,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static unsigned long big_pack_threshold;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -130,6 +135,7 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -169,7 +175,8 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs, unsigned long limit)\n+static struct packed_git *find_base_packs(struct string_list *packs,\n+\t\t\t\t\t unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n@@ -186,6 +193,8 @@ static void find_base_packs(struct string_list *packs, unsigned long limit)\n \n \tif (base)\n \t\tstring_list_append(packs, base->pack_name);\n+\n+\treturn base;\n }\n \n static int too_many_packs(void)\n@@ -210,6 +219,79 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n+static uint64_t total_ram(void)\n+{\n+#if defined(HAVE_SYSINFO)\n+\tstruct sysinfo si;\n+\n+\tif (!sysinfo(&si))\n+\t\treturn si.totalram;\n+#elif defined(HAVE_BSD_SYSCTL) && (defined(HW_MEMSIZE) || defined(HW_PHYSMEM))\n+\tint64_t physical_memory;\n+\tint mib[2];\n+\tsize_t length;\n+\n+\tmib[0] = CTL_HW;\n+# if defined(HW_MEMSIZE)\n+\tmib[1] = HW_MEMSIZE;\n+# else\n+\tmib[1] = HW_PHYSMEM;\n+# endif\n+\tlength = sizeof(int64_t);\n+\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n+\t\treturn physical_memory;\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\tMEMORYSTATUSEX memInfo;\n+\n+\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n+\tif (GlobalMemoryStatusEx(&memInfo))\n+\t\treturn memInfo.ullTotalPhys;\n+#endif\n+\treturn 0;\n+}\n+\n+static uint64_t estimate_repack_memory(struct packed_git *pack)\n+{\n+\tunsigned long nr_objects = approximate_object_count();\n+\tsize_t os_cache, heap;\n+\n+\tif (!pack || !nr_objects)\n+\t\treturn 0;\n+\n+\t/*\n+\t * First we have to scan through at least one pack.\n+\t * Assume enough room in OS file cache to keep the entire pack\n+\t * or we may accidentally evict data of other processes from\n+\t * the cache.\n+\t */\n+\tos_cache = pack->pack_size + pack->index_size;\n+\t/* then pack-objects needs lots more for book keeping */\n+\theap = sizeof(struct object_entry) * nr_objects;\n+\t/*\n+\t * internal rev-list --all --objects takes up some memory too,\n+\t * let's say half of it is for blobs\n+\t */\n+\theap += sizeof(struct blob) * nr_objects / 2;\n+\t/*\n+\t * and the other half is for trees (commits and tags are\n+\t * usually insignificant)\n+\t */\n+\theap += sizeof(struct tree) * nr_objects / 2;\n+\t/* and then obj_hash[], underestimated in fact */\n+\theap += sizeof(struct object *) * nr_objects;\n+\t/* revindex is used also */\n+\theap += sizeof(struct revindex_entry) * nr_objects;\n+\t/*\n+\t * read_sha1_file() (either at delta calculation phase, or\n+\t * writing phase) also fills up the delta base cache\n+\t */\n+\theap += delta_base_cache_limit;\n+\t/* and of course pack-objects has its own delta cache */\n+\theap += max_delta_cache_size;\n+\n+\treturn os_cache + heap;\n+}\n+\n static int keep_one_pack(struct string_list_item *item, void *data)\n {\n \targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n@@ -260,6 +342,20 @@ static int need_to_gc(void)\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n+\t\t} else {\n+\t\t\tstruct packed_git *p = find_base_packs(&keep_pack, 0);\n+\t\t\tuint64_t mem_have, mem_want;\n+\n+\t\t\tmem_have = total_ram();\n+\t\t\tmem_want = estimate_repack_memory(p);\n+\n+\t\t\t/*\n+\t\t\t * Only allow 1/2 of memory for pack-objects, leave\n+\t\t\t * the rest for the OS and other processes in the\n+\t\t\t * system.\n+\t\t\t */\n+\t\t\tif (!mem_have || mem_want < mem_have / 2)\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9b9a6d6268..c77bea404d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -82,7 +82,7 @@ static uint16_t write_bitmap_options;\n static int exclude_promisor_objects;\n \n static unsigned long delta_cache_size = 0;\n-static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 6a1d0de0cc..ae9cbccec1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n+\tBASIC_CFLAGS += -DHAVE_SYSINFO\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 07e383257b..e373af48b8 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -284,6 +284,10 @@ extern char *gitdirname(char *);\n #include \n #endif\n \n+#ifdef HAVE_SYSINFO\n+# include \n+#endif\n+\n /* On most systems would have given us this, but\n * not on some systems (e.g. z/OS).\n */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..af4f46c026 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024)\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex c42f60bc5b..818435f04e 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -5,6 +5,13 @@ test_description='basic git gc tests\n \n . ./test-lib.sh\n \n+test_expect_success 'setup' '\n+\t# do not let the amount of physical memory affects gc\n+\t# behavior, make sure we always pack everything to one pack by\n+\t# default\n+\tgit config gc.bigPackThreshold 2g\n+'\n+\n test_expect_success 'gc empty repository' '\n \tgit gc\n '\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344668","messageId":"20180414152642.4666-5-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 4/7] gc: add gc.bigPackThreshold config","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:39Z","receivedAt":"2018-04-14T15:27:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The --keep-largest-pack option is not very convenient to use because\nyou need to tell gc to do this explicitly (and probably on just a few\nlarge repos).\n\nAdd a config key that enables this mode when packs larger than a limit\nare found. Note that there's a slight behavior difference compared to\n--keep-largest-pack: all packs larger than the threshold are kept, not\njust the largest one.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 8 ++++++++\n Documentation/git-gc.txt | 6 ++++--\n builtin/gc.c | 26 ++++++++++++++++++++------\n 3 files changed, 32 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2659153cb3..40516675b6 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1558,6 +1558,14 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.bigPackThreshold::\n+\tIf non-zero, all packs larger than this limit are kept when\n+\t`git gc` is run. This is very similar to `--keep-base-pack`\n+\texcept that all packs that meet the threshold are kept, not\n+\tjust the base pack. Defaults to zero. Common unit suffixes of\n+\t'k', 'm', or 'g' are supported.\n+\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 8f903231da..649faddfa6 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -56,7 +56,8 @@ single pack using `git repack -d -l`. Setting the value of `gc.auto`\n to 0 disables automatic packing of loose objects.\n +\n If the number of packs exceeds the value of `gc.autoPackLimit`,\n-then existing packs (except those marked with a `.keep` file)\n+then existing packs (except those marked with a `.keep` file\n+or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n 'git repack'. Setting `gc.autoPackLimit` to 0 disables\n automatic consolidation of packs.\n@@ -86,7 +87,8 @@ be performed as well.\n \n --keep-largest-pack::\n \tAll packs except the largest pack and those marked with a\n-\t`.keep` files are consolidated into a single pack.\n+\t`.keep` files are consolidated into a single pack. When this\n+\toption is used, `gc.bigPackThreshold` is ignored.\n \n Configuration\n -------------\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f251662a8f..81ecdc5ffa 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -41,6 +41,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static unsigned long big_pack_threshold;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -128,6 +129,8 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -166,14 +169,17 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs)\n+static void find_base_packs(struct string_list *packs, unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n \tfor (p = get_packed_git(the_repository); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n-\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\tif (limit) {\n+\t\t\tif (p->pack_size >= limit)\n+\t\t\t\tstring_list_append(packs, p->pack_name);\n+\t\t} else if (!base || base->pack_size < p->pack_size) {\n \t\t\tbase = p;\n \t\t}\n \t}\n@@ -244,9 +250,15 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs())\n-\t\tadd_repack_all_option(NULL);\n-\telse if (too_many_loose_objects())\n+\tif (too_many_packs()) {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (big_pack_threshold)\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t} else if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -464,7 +476,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\tif (keep_base_pack != -1) {\n \t\t\tif (keep_base_pack)\n-\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t} else if (big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344669","messageId":"20180414152642.4666-1-pclouds@gmail.com","threadId":"48285","inReplyTo":null,"subject":"[PATCH 0/7] nd/repack-keep-pack update","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:35Z","receivedAt":"2018-04-14T15:27:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is basically a resend from the last round but rebased on\n'master'. The old base results in some conflicts with packfile and oid\nconversion series. This one should reduce merge conflicts on 'pu' to\ntrivial ones.\n\nNguyễn Thái Ngọc Duy (7):\n t7700: have closing quote of a test at the beginning of line\n repack: add --keep-pack option\n gc: add --keep-largest-pack option\n gc: add gc.bigPackThreshold config\n gc: handle a corner case in gc.bigPackThreshold\n gc --auto: exclude base pack if not enough mem to \"repack -ad\"\n pack-objects: show some progress when counting kept objects\n\n Documentation/config.txt | 12 +++\n Documentation/git-gc.txt | 19 +++-\n Documentation/git-pack-objects.txt | 9 +-\n Documentation/git-repack.txt | 9 +-\n builtin/gc.c | 165 +++++++++++++++++++++++++++--\n builtin/pack-objects.c | 83 +++++++++++----\n builtin/repack.c | 21 +++-\n config.mak.uname | 1 +\n git-compat-util.h | 4 +\n object-store.h | 1 +\n pack-objects.h | 2 +\n t/t6500-gc.sh | 32 ++++++\n t/t7700-repack.sh | 27 ++++-\n 13 files changed, 349 insertions(+), 36 deletions(-)\n\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344670","messageId":"20180414152642.4666-2-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH 1/7] t7700: have closing quote of a test at the beginning of line","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:26:36Z","receivedAt":"2018-04-14T15:27:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The closing quote of a test body by convention is always at the start\nof line.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n t/t7700-repack.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 6061a04147..38247afbec 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -194,7 +194,7 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&\n \tgit repack -a -d &&\n \tgit cat-file -t $H1\n-\t'\n+'\n \n test_done\n \n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344698","messageId":"878t9pfu4f.fsf@evledraar.gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"Re: [PATCH 0/7] nd/repack-keep-pack update","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-04-14T19:47:12Z","receivedAt":"2018-04-14T19:47:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Apr 14 2018, Nguyễn Thái Ngọc Duy wrote:\n\n> This is basically a resend from the last round but rebased on\n> 'master'. The old base results in some conflicts with packfile and oid\n> conversion series. This one should reduce merge conflicts on 'pu' to\n> trivial ones.\n\nThanks. I've been running this at work and as noted in\nhttps://public-inbox.org/git/87vadpxv27.fsf@evledraar.gmail.com/ it's\nhad big performance impact to the better, users even started noticing it\n(they'd previously get noticeable slowdowns while doing other task on\nGC).\n\nI also tried to see just how much worse this was making performance, my\nhunch was that the difference should be trivial but noticeable since\nwe'll produce a less efficient pack.\n\nWhat I found was the opposite, under real-world conditions it seems to\nbe making things 1-2% better on common git operations, which I suspect\nis because once we've done a few pulls and coalesced those into their\nown pack(s) there's more cache locality for the data we're actually\nlooking at.\n\nI.e. once you've got a repo has a big pack you're not touching, and a\nfew weeks of updates from upstream that you've coalesced into another\npack there's a higher density of stuff you care about near HEAD per FS\npage in the recent smaller pack, which if you're pressed for memory and\nparts of your pack are getting paged out of the FS cache is a win. I\nhaven't confirmed that, it's just a hypothesis.\n\nThe only (trivial) issue I found in the patches themselves was that\nbetween 4/5 and 5/7 you're adding an empty line to config.txt in 4/7\njust to erase it in 5/7, better not to add it to begin with, but\nhopefully Junio can fix that up (if he cares).\n"},{"id":"344705","messageId":"CAPc5daXHJCbuxv2NE1KUt3pYtP1p1RQhOT5ceb0++spEJZ5dTQ@mail.gmail.com","threadId":"48285","inReplyTo":"878t9pfu4f.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/7] nd/repack-keep-pack update","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-14T22:38:41Z","receivedAt":"2018-04-14T22:39:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Sun, Apr 15, 2018 at 4:47 AM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n> The only (trivial) issue I found in the patches themselves was that\n> between 4/5 and 5/7 you're adding an empty line to config.txt in 4/7\n> just to erase it in 5/7, better not to add it to begin with, but\n> hopefully Junio can fix that up (if he cares).\n\nI do care but I'd wish I do not have to waste time and concentration to spend\non doing so, even though I would be fully capable of skipping this round and\nremembering to queue a rerolled one.\n\nI've seen mentions like the above one a few times on the list recently, so let\nme make it clear. Some things are easier to tweak locally than others, and\nI'd rather not to waste my time on cleaning other people's mess. A simple\ntypofix that does not cascade through to later steps in a series is one thing.\nA tweak that changes number of lines in a hunk that forces a later step to\ncompensate is more involved.\n\nDon't expect your traffic cop to wash your care while you're stopping at a\nred signal.\n"},{"id":"344717","messageId":"20180415153618.32019-1-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180414152642.4666-1-pclouds@gmail.com","subject":"[PATCH v2 0/7] nd/repack-keep-pack update","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:11Z","receivedAt":"2018-04-15T15:36:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Apr 14, 2018 at 9:47 PM, Ævar Arnfjörð Bjarmason wrote:\n> The only (trivial) issue I found in the patches themselves was that\n> between 4/5 and 5/7 you're adding an empty line to config.txt in 4/7\n> just to erase it in 5/7, better not to add it to begin with, but\n> hopefully Junio can fix that up (if he cares).\n\nFixed (and is the only change in v2) in case Junio has not picked up\nthe last round and fixed it himself yet.\n\nNguyễn Thái Ngọc Duy (7):\n t7700: have closing quote of a test at the beginning of line\n repack: add --keep-pack option\n gc: add --keep-largest-pack option\n gc: add gc.bigPackThreshold config\n gc: handle a corner case in gc.bigPackThreshold\n gc --auto: exclude base pack if not enough mem to \"repack -ad\"\n pack-objects: show some progress when counting kept objects\n\n Documentation/config.txt | 12 +++\n Documentation/git-gc.txt | 19 +++-\n Documentation/git-pack-objects.txt | 9 +-\n Documentation/git-repack.txt | 9 +-\n builtin/gc.c | 165 +++++++++++++++++++++++++++--\n builtin/pack-objects.c | 83 +++++++++++----\n builtin/repack.c | 21 +++-\n config.mak.uname | 1 +\n git-compat-util.h | 4 +\n object-store.h | 1 +\n pack-objects.h | 2 +\n t/t6500-gc.sh | 32 ++++++\n t/t7700-repack.sh | 27 ++++-\n 13 files changed, 349 insertions(+), 36 deletions(-)\n\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344718","messageId":"20180415153618.32019-2-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 1/7] t7700: have closing quote of a test at the beginning of line","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:12Z","receivedAt":"2018-04-15T15:36:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The closing quote of a test body by convention is always at the start\nof line.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n t/t7700-repack.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 6061a04147..38247afbec 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -194,7 +194,7 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&\n \tgit repack -a -d &&\n \tgit cat-file -t $H1\n-\t'\n+'\n \n test_done\n \n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344719","messageId":"20180415153618.32019-3-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 2/7] repack: add --keep-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:13Z","receivedAt":"2018-04-15T15:36:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We allow to keep existing packs by having companion .keep files. This\nis helpful when a pack is permanently kept. In the next patch, git-gc\njust wants to keep a pack temporarily, for one pack-objects\nrun. git-gc can use --keep-pack for this use case.\n\nA note about why the pack_keep field cannot be reused and\npack_keep_in_core has to be added. This is about the case when\n--keep-pack is specified together with either --keep-unreachable or\n--unpack-unreachable, but --honor-pack-keep is NOT specified.\n\nIn this case, we want to exclude objects from the packs specified on\ncommand line, not from ones with .keep files. If only one bit flag is\nused, we have to clear pack_keep on pack files with the .keep file.\n\nBut we can't make any assumption about unreachable objects in .keep\npacks. If \"pack_keep\" field is false for .keep packs, we could\npotentially pull lots of unreachable objects into the new pack, or\nunpack them loose. The safer approach is ignore all packs with either\n.keep file or --keep-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 9 ++++-\n Documentation/git-repack.txt | 9 ++++-\n builtin/pack-objects.c | 63 ++++++++++++++++++++++++------\n builtin/repack.c | 21 ++++++++--\n object-store.h | 1 +\n t/t7700-repack.sh | 25 ++++++++++++\n 6 files changed, 110 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 81bc490ac5..403524652a 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -12,7 +12,7 @@ SYNOPSIS\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n-\t[--revs [--unpacked | --all]]\n+\t[--revs [--unpacked | --all]] [--keep-pack=]\n \t[--stdout [--filter=] | base-name]\n \t[--shallow] [--keep-true-parents] < object-list\n \n@@ -126,6 +126,13 @@ base-name::\n \thas a .keep file to be ignored, even if it would have\n \totherwise been packed.\n \n+--keep-pack=::\n+\tThis flag causes an object already in the given pack to be\n+\tignored, even if it would have otherwise been\n+\tpacked. `` is the the pack file name without\n+\tleading directory (e.g. `pack-123.pack`). The option could be\n+\tspecified multiple times to keep multiple packs.\n+\n --incremental::\n \tThis flag causes an object already in a pack to be ignored\n \teven if it would have otherwise been packed.\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ae750e9e11..ce497d9d12 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=] [--depth=] [--threads=] [--keep-pack=]\n \n DESCRIPTION\n -----------\n@@ -133,6 +133,13 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n+--keep-pack=::\n+\tExclude the given pack from repacking. This is the equivalent\n+\tof having `.keep` file on the pack. `` is the the\n+\tpack file name without leading directory (e.g. `pack-123.pack`).\n+\tThe option could be specified multiple times to keep multiple\n+\tpacks.\n+\n --unpack-unreachable=::\n \tWhen loosening unreachable objects, do not bother loosening any\n \tobjects older than ``. This can be used to optimize out\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4bdae5a1d8..9b9a6d6268 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -30,6 +30,7 @@\n #include \"list.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"dir.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -55,7 +56,8 @@ static int pack_loose_unreachable;\n static int local;\n static int have_non_local_packs;\n static int incremental;\n-static int ignore_packed_keep;\n+static int ignore_packed_keep_on_disk;\n+static int ignore_packed_keep_in_core;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -982,13 +984,16 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n \t * not know either way, and it needs to check more packs.\n \t */\n-\tif (!ignore_packed_keep &&\n+\tif (!ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs))\n \t\treturn 1;\n \n \tif (local && !p->pack_local)\n \t\treturn 0;\n-\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\tif (p->pack_local &&\n+\t ((ignore_packed_keep_on_disk && p->pack_keep) ||\n+\t (ignore_packed_keep_in_core && p->pack_keep_in_core)))\n \t\treturn 0;\n \n \t/* we don't know yet; keep looking for more packs */\n@@ -2675,7 +2680,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \t\tstruct object_id oid;\n \t\tstruct object *o;\n \n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \t\tif (open_pack_index(p))\n \t\t\tdie(\"cannot open pack index\");\n@@ -2738,7 +2743,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n \t\t\t\t\tget_packed_git(the_repository);\n \n \twhile (p) {\n-\t\tif ((!p->pack_local || p->pack_keep) &&\n+\t\tif ((!p->pack_local || p->pack_keep ||\n+\t\t\t\tp->pack_keep_in_core) &&\n \t\t\tfind_pack_entry_one(oid->hash, p)) {\n \t\t\tlast_found = p;\n \t\t\treturn 1;\n@@ -2781,7 +2787,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \tstruct object_id oid;\n \n \tfor (p = get_packed_git(the_repository); p; p = p->next) {\n-\t\tif (!p->pack_local || p->pack_keep)\n+\t\tif (!p->pack_local || p->pack_keep || p->pack_keep_in_core)\n \t\t\tcontinue;\n \n \t\tif (open_pack_index(p))\n@@ -2807,7 +2813,8 @@ static int pack_options_allow_reuse(void)\n {\n \treturn pack_to_stdout &&\n \t allow_ofs_delta &&\n-\t !ignore_packed_keep &&\n+\t !ignore_packed_keep_on_disk &&\n+\t !ignore_packed_keep_in_core &&\n \t (!local || !have_non_local_packs) &&\n \t !incremental;\n }\n@@ -2916,6 +2923,32 @@ static void get_object_list(int ac, const char **av)\n \toid_array_clear(&recent_objects);\n }\n \n+static void add_extra_kept_packs(const struct string_list *names)\n+{\n+\tstruct packed_git *p;\n+\n+\tif (!names->nr)\n+\t\treturn;\n+\n+\tfor (p = get_packed_git(the_repository); p; p = p->next) {\n+\t\tconst char *name = basename(p->pack_name);\n+\t\tint i;\n+\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < names->nr; i++)\n+\t\t\tif (!fspathcmp(name, names->items[i].string))\n+\t\t\t\tbreak;\n+\n+\t\tif (i < names->nr) {\n+\t\t\tp->pack_keep_in_core = 1;\n+\t\t\tignore_packed_keep_in_core = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+}\n+\n static int option_parse_index_version(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n@@ -2955,6 +2988,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct option pack_objects_options[] = {\n \t\tOPT_SET_INT('q', \"quiet\", &progress,\n \t\t\t N_(\"do not show progress meter\"), 0),\n@@ -3019,8 +3053,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n-\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n+\t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"ignore this pack\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n \t\t\t N_(\"pack compression level\")),\n \t\tOPT_SET_INT(0, \"keep-true-parents\", &grafts_replace_parents,\n@@ -3148,19 +3184,20 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (progress && all_progress_implied)\n \t\tprogress = 2;\n \n-\tif (ignore_packed_keep) {\n+\tadd_extra_kept_packs(&keep_pack_list);\n+\tif (ignore_packed_keep_on_disk) {\n \t\tstruct packed_git *p;\n \t\tfor (p = get_packed_git(the_repository); p; p = p->next)\n \t\t\tif (p->pack_local && p->pack_keep)\n \t\t\t\tbreak;\n \t\tif (!p) /* no keep-able packs found */\n-\t\t\tignore_packed_keep = 0;\n+\t\t\tignore_packed_keep_on_disk = 0;\n \t}\n \tif (local) {\n \t\t/*\n-\t\t * unlike ignore_packed_keep above, we do not want to\n-\t\t * unset \"local\" based on looking at packs, as it\n-\t\t * also covers non-local objects\n+\t\t * unlike ignore_packed_keep_on_disk above, we do not\n+\t\t * want to unset \"local\" based on looking at packs, as\n+\t\t * it also covers non-local objects\n \t\t */\n \t\tstruct packed_git *p;\n \t\tfor (p = get_packed_git(the_repository); p; p = p->next) {\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 7bdb40142f..6c636e159e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -86,7 +86,8 @@ static void remove_pack_on_signal(int signo)\n * have a corresponding .keep or .promisor file. These packs are not to\n * be kept if we are going to pack everything into one file.\n */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list)\n+static void get_non_kept_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t\tconst struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -97,6 +98,14 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n \n \twhile ((e = readdir(dir)) != NULL) {\n \t\tsize_t len;\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < extra_keep->nr; i++)\n+\t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n+\t\t\t\tbreak;\n+\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n+\t\t\tcontinue;\n+\n \t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n \t\t\tcontinue;\n \n@@ -148,7 +157,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tint ext, ret, failed;\n+\tint i, ext, ret, failed;\n \tFILE *out;\n \n \t/* variables to be filled by option parsing */\n@@ -160,6 +169,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tconst char *depth = NULL;\n \tconst char *threads = NULL;\n \tconst char *max_pack_size = NULL;\n+\tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tint no_reuse_delta = 0, no_reuse_object = 0;\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n@@ -200,6 +210,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"maximum size of each packfile\")),\n \t\tOPT_BOOL(0, \"pack-kept-objects\", &pack_kept_objects,\n \t\t\t\tN_(\"repack objects in packs marked with .keep\")),\n+\t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n+\t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -230,6 +242,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--keep-true-parents\");\n \tif (!pack_kept_objects)\n \t\targv_array_push(&cmd.args, \"--honor-pack-keep\");\n+\tfor (i = 0; i < keep_pack_list.nr; i++)\n+\t\targv_array_pushf(&cmd.args, \"--keep-pack=%s\",\n+\t\t\t\t keep_pack_list.items[i].string);\n \targv_array_push(&cmd.args, \"--non-empty\");\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n@@ -254,7 +269,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&cmd.args, \"--write-bitmap-index\");\n \n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs);\n+\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n \t\t\tif (unpack_unreachable) {\ndiff --git a/object-store.h b/object-store.h\nindex fef33f345f..0a4dbb74d3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -71,6 +71,7 @@ struct packed_git {\n \tint pack_fd;\n \tunsigned pack_local:1,\n \t\t pack_keep:1,\n+\t\t pack_keep_in_core:1,\n \t\t freshened:1,\n \t\t do_not_close:1,\n \t\t pack_promisor:1;\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 38247afbec..6162e2a8e6 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -4,6 +4,12 @@ test_description='git repack works correctly'\n \n . ./test-lib.sh\n \n+commit_and_pack() {\n+\ttest_commit \"$@\" >/dev/null &&\n+\tSHA1=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack file1 &&\n \techo content2 > file2 &&\n@@ -196,5 +202,24 @@ test_expect_success 'objects made unreachable by grafts only are kept' '\n \tgit cat-file -t $H1\n '\n \n+test_expect_success 'repack --keep-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\tP1=$(commit_and_pack 1) &&\n+\t\tP2=$(commit_and_pack 2) &&\n+\t\tP3=$(commit_and_pack 3) &&\n+\t\tP4=$(commit_and_pack 4) &&\n+\t\tls .git/objects/pack/*.pack >old-counts &&\n+\t\ttest_line_count = 4 old-counts &&\n+\t\tgit repack -a -d --keep-pack $P1 --keep-pack $P4 &&\n+\t\tls .git/objects/pack/*.pack >new-counts &&\n+\t\tgrep -q $P1 new-counts &&\n+\t\tgrep -q $P4 new-counts &&\n+\t\ttest_line_count = 3 new-counts &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\n \n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344720","messageId":"20180415153618.32019-6-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 5/7] gc: handle a corner case in gc.bigPackThreshold","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:16Z","receivedAt":"2018-04-15T15:36:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This config allows us to keep packs back if their size is larger\nthan a limit. But if this N >= gc.autoPackLimit, we may have a\nproblem. We are supposed to reduce the number of packs after a\nthreshold because it affects performance.\n\nWe could tell the user that they have incompatible gc.bigPackThreshold\nand gc.autoPackLimit, but it's kinda hard when 'git gc --auto' runs in\nbackground. Instead let's fall back to the next best stategy: try to\nreduce the number of packs anyway, but keep the base pack out. This\nreduces the number of packs to two and hopefully won't take up too\nmuch resources to repack (the assumption still is the base pack takes\nmost resources to handle).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 5 +++++\n builtin/gc.c | 8 +++++++-\n 2 files changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 728ae902e6..4c3f1d7651 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1564,6 +1564,11 @@ gc.bigPackThreshold::\n \texcept that all packs that meet the threshold are kept, not\n \tjust the base pack. Defaults to zero. Common unit suffixes of\n \t'k', 'm', or 'g' are supported.\n++\n+Note that if the number of kept packs is more than gc.autoPackLimit,\n+this configuration variable is ignored, all packs except the base pack\n+will be repacked. After this the number of packs should go below\n+gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 81ecdc5ffa..23c17ba7e9 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -253,8 +253,14 @@ static int need_to_gc(void)\n \tif (too_many_packs()) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold)\n+\t\tif (big_pack_threshold) {\n \t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n+\t\t\t\tbig_pack_threshold = 0;\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t}\n+\t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344721","messageId":"20180415153618.32019-8-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 7/7] pack-objects: show some progress when counting kept objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:18Z","receivedAt":"2018-04-15T15:36:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We only show progress when there are new objects to be packed. But\nwhen --keep-pack is specified on the base pack, we will exclude most\nof objects. This makes 'pack-objects' stay silent for a long time\nwhile the counting phase is going.\n\nLet's show some progress whenever we visit an object instead. The old\n\"Counting objects\" is renamed to \"Enumerating objects\" and a new\nprogress \"Counting objects\" line is added.\n\nThis new \"Counting objects\" line should progress pretty quick when the\nsystem is beefy. But when the system is under pressure, the reading\nobject header done in this phase could be slow and showing progress is\nan improvement over staying silent in the current code.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c77bea404d..6a1346c41f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -46,7 +46,7 @@ static const char *pack_usage[] = {\n static struct packing_data to_pack;\n \n static struct pack_idx_entry **written_list;\n-static uint32_t nr_result, nr_written;\n+static uint32_t nr_result, nr_written, nr_seen;\n \n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n@@ -1096,6 +1096,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, exclude, &index_pos))\n \t\treturn 0;\n \n@@ -1111,8 +1113,6 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tcreate_object_entry(oid, type, pack_name_hash(name),\n \t\t\t exclude, name && no_try_delta(name),\n \t\t\t index_pos, found_pack, found_offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1123,6 +1123,8 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n {\n \tuint32_t index_pos;\n \n+\tdisplay_progress(progress_state, ++nr_seen);\n+\n \tif (have_duplicate_entry(oid, 0, &index_pos))\n \t\treturn 0;\n \n@@ -1130,8 +1132,6 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, index_pos, pack, offset);\n-\n-\tdisplay_progress(progress_state, nr_result);\n \treturn 1;\n }\n \n@@ -1716,6 +1716,10 @@ static void get_object_details(void)\n \tuint32_t i;\n \tstruct object_entry **sorted_by_offset;\n \n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Counting objects\"),\n+\t\t\t\t\t\tto_pack.nr_objects);\n+\n \tsorted_by_offset = xcalloc(to_pack.nr_objects, sizeof(struct object_entry *));\n \tfor (i = 0; i < to_pack.nr_objects; i++)\n \t\tsorted_by_offset[i] = to_pack.objects + i;\n@@ -1726,7 +1730,9 @@ static void get_object_details(void)\n \t\tcheck_object(entry);\n \t\tif (big_file_threshold < entry->size)\n \t\t\tentry->no_try_delta = 1;\n+\t\tdisplay_progress(progress_state, i + 1);\n \t}\n+\tstop_progress(&progress_state);\n \n \t/*\n \t * This must happen in a second pass, since we rely on the delta\n@@ -3209,7 +3215,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (progress)\n-\t\tprogress_state = start_progress(_(\"Counting objects\"), 0);\n+\t\tprogress_state = start_progress(_(\"Enumerating objects\"), 0);\n \tif (!use_internal_rev_list)\n \t\tread_object_list_from_stdin();\n \telse {\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344722","messageId":"20180415153618.32019-7-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 6/7] gc --auto: exclude base pack if not enough mem to \"repack -ad\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:17Z","receivedAt":"2018-04-15T15:36:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack-objects could be a big memory hog especially on large repos,\neverybody knows that. The suggestion to stick a .keep file on the\ngiant base pack to avoid this problem is also known for a long time.\n\nRecent patches add an option to do just this, but it has to be either\nconfigured or activated manually. This patch lets `git gc --auto`\nactivate this mode automatically when it thinks `repack -ad` will use\na lot of memory and start affecting the system due to swapping or\nflushing OS cache.\n\ngc --auto decides to do this based on an estimation of pack-objects\nmemory usage, which is quite accurate at least for the heap part, and\nwhether that fits in half of system memory (the assumption here is for\ndesktop environment where there are many other applications running).\n\nThis mechanism only kicks in if gc.bigBasePackThreshold is not configured.\nIf it is, it is assumed that the user already knows what they want.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 9 +++-\n builtin/gc.c | 98 +++++++++++++++++++++++++++++++++++++++-\n builtin/pack-objects.c | 2 +-\n config.mak.uname | 1 +\n git-compat-util.h | 4 ++\n pack-objects.h | 2 +\n t/t6500-gc.sh | 7 +++\n 7 files changed, 119 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 649faddfa6..0fb1d43b2c 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -59,8 +59,13 @@ If the number of packs exceeds the value of `gc.autoPackLimit`,\n then existing packs (except those marked with a `.keep` file\n or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n-'git repack'. Setting `gc.autoPackLimit` to 0 disables\n-automatic consolidation of packs.\n+'git repack'.\n+If the amount of memory is estimated not enough for `git repack` to\n+run smoothly and `gc.bigPackThreshold` is not set, the largest\n+pack will also be excluded (this is the equivalent of running `git gc`\n+with `--keep-base-pack`).\n+Setting `gc.autoPackLimit` to 0 disables automatic consolidation of\n+packs.\n +\n If houskeeping is required due to many loose objects or packs, all\n other housekeeping tasks (e.g. rerere, working trees, reflog...) will\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 23c17ba7e9..3c7c93e961 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -22,6 +22,10 @@\n #include \"commit.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"pack.h\"\n+#include \"pack-objects.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -42,6 +46,7 @@ static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n static unsigned long big_pack_threshold;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -130,6 +135,7 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -169,7 +175,8 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs, unsigned long limit)\n+static struct packed_git *find_base_packs(struct string_list *packs,\n+\t\t\t\t\t unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n@@ -186,6 +193,8 @@ static void find_base_packs(struct string_list *packs, unsigned long limit)\n \n \tif (base)\n \t\tstring_list_append(packs, base->pack_name);\n+\n+\treturn base;\n }\n \n static int too_many_packs(void)\n@@ -210,6 +219,79 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n+static uint64_t total_ram(void)\n+{\n+#if defined(HAVE_SYSINFO)\n+\tstruct sysinfo si;\n+\n+\tif (!sysinfo(&si))\n+\t\treturn si.totalram;\n+#elif defined(HAVE_BSD_SYSCTL) && (defined(HW_MEMSIZE) || defined(HW_PHYSMEM))\n+\tint64_t physical_memory;\n+\tint mib[2];\n+\tsize_t length;\n+\n+\tmib[0] = CTL_HW;\n+# if defined(HW_MEMSIZE)\n+\tmib[1] = HW_MEMSIZE;\n+# else\n+\tmib[1] = HW_PHYSMEM;\n+# endif\n+\tlength = sizeof(int64_t);\n+\tif (!sysctl(mib, 2, &physical_memory, &length, NULL, 0))\n+\t\treturn physical_memory;\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\tMEMORYSTATUSEX memInfo;\n+\n+\tmemInfo.dwLength = sizeof(MEMORYSTATUSEX);\n+\tif (GlobalMemoryStatusEx(&memInfo))\n+\t\treturn memInfo.ullTotalPhys;\n+#endif\n+\treturn 0;\n+}\n+\n+static uint64_t estimate_repack_memory(struct packed_git *pack)\n+{\n+\tunsigned long nr_objects = approximate_object_count();\n+\tsize_t os_cache, heap;\n+\n+\tif (!pack || !nr_objects)\n+\t\treturn 0;\n+\n+\t/*\n+\t * First we have to scan through at least one pack.\n+\t * Assume enough room in OS file cache to keep the entire pack\n+\t * or we may accidentally evict data of other processes from\n+\t * the cache.\n+\t */\n+\tos_cache = pack->pack_size + pack->index_size;\n+\t/* then pack-objects needs lots more for book keeping */\n+\theap = sizeof(struct object_entry) * nr_objects;\n+\t/*\n+\t * internal rev-list --all --objects takes up some memory too,\n+\t * let's say half of it is for blobs\n+\t */\n+\theap += sizeof(struct blob) * nr_objects / 2;\n+\t/*\n+\t * and the other half is for trees (commits and tags are\n+\t * usually insignificant)\n+\t */\n+\theap += sizeof(struct tree) * nr_objects / 2;\n+\t/* and then obj_hash[], underestimated in fact */\n+\theap += sizeof(struct object *) * nr_objects;\n+\t/* revindex is used also */\n+\theap += sizeof(struct revindex_entry) * nr_objects;\n+\t/*\n+\t * read_sha1_file() (either at delta calculation phase, or\n+\t * writing phase) also fills up the delta base cache\n+\t */\n+\theap += delta_base_cache_limit;\n+\t/* and of course pack-objects has its own delta cache */\n+\theap += max_delta_cache_size;\n+\n+\treturn os_cache + heap;\n+}\n+\n static int keep_one_pack(struct string_list_item *item, void *data)\n {\n \targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n@@ -260,6 +342,20 @@ static int need_to_gc(void)\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n+\t\t} else {\n+\t\t\tstruct packed_git *p = find_base_packs(&keep_pack, 0);\n+\t\t\tuint64_t mem_have, mem_want;\n+\n+\t\t\tmem_have = total_ram();\n+\t\t\tmem_want = estimate_repack_memory(p);\n+\n+\t\t\t/*\n+\t\t\t * Only allow 1/2 of memory for pack-objects, leave\n+\t\t\t * the rest for the OS and other processes in the\n+\t\t\t * system.\n+\t\t\t */\n+\t\t\tif (!mem_have || mem_want < mem_have / 2)\n+\t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9b9a6d6268..c77bea404d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -82,7 +82,7 @@ static uint16_t write_bitmap_options;\n static int exclude_promisor_objects;\n \n static unsigned long delta_cache_size = 0;\n-static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n+static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 6a1d0de0cc..ae9cbccec1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n+\tBASIC_CFLAGS += -DHAVE_SYSINFO\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 07e383257b..e373af48b8 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -284,6 +284,10 @@ extern char *gitdirname(char *);\n #include \n #endif\n \n+#ifdef HAVE_SYSINFO\n+# include \n+#endif\n+\n /* On most systems would have given us this, but\n * not on some systems (e.g. z/OS).\n */\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 03f1191659..af4f46c026 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_OBJECTS_H\n #define PACK_OBJECTS_H\n \n+#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024)\n+\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex c42f60bc5b..818435f04e 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -5,6 +5,13 @@ test_description='basic git gc tests\n \n . ./test-lib.sh\n \n+test_expect_success 'setup' '\n+\t# do not let the amount of physical memory affects gc\n+\t# behavior, make sure we always pack everything to one pack by\n+\t# default\n+\tgit config gc.bigPackThreshold 2g\n+'\n+\n test_expect_success 'gc empty repository' '\n \tgit gc\n '\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344723","messageId":"20180415153618.32019-4-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 3/7] gc: add --keep-largest-pack option","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:14Z","receivedAt":"2018-04-15T15:36:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This adds a new repack mode that combines everything into a secondary\npack, leaving the largest pack alone.\n\nThis could help reduce memory pressure. On linux-2.6.git, valgrind\nmassif reports 1.6GB heap in \"pack all\" case, and 535MB in \"pack\nall except the base pack\" case. We save roughly 1GB memory by\nexcluding the base pack.\n\nThis should also lower I/O because we don't have to rewrite a giant\npack every time (e.g. for linux-2.6.git that's a 1.4GB pack file)..\n\nPS. The use of string_list here seems overkill, but we'll need it in\nthe next patch...\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-gc.txt | 6 +++++-\n builtin/gc.c | 45 ++++++++++++++++++++++++++++++++++++----\n t/t6500-gc.sh | 25 ++++++++++++++++++++++\n 3 files changed, 71 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 3126e0dd00..8f903231da 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force]\n+'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -84,6 +84,10 @@ be performed as well.\n \tForce `git gc` to run even if there may be another `git gc`\n \tinstance running on this repository.\n \n+--keep-largest-pack::\n+\tAll packs except the largest pack and those marked with a\n+\t`.keep` files are consolidated into a single pack.\n+\n Configuration\n -------------\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3e67124eaa..f251662a8f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -166,6 +166,22 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n+static void find_base_packs(struct string_list *packs)\n+{\n+\tstruct packed_git *p, *base = NULL;\n+\n+\tfor (p = get_packed_git(the_repository); p; p = p->next) {\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\t\tbase = p;\n+\t\t}\n+\t}\n+\n+\tif (base)\n+\t\tstring_list_append(packs, base->pack_name);\n+}\n+\n static int too_many_packs(void)\n {\n \tstruct packed_git *p;\n@@ -188,7 +204,13 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit < cnt;\n }\n \n-static void add_repack_all_option(void)\n+static int keep_one_pack(struct string_list_item *item, void *data)\n+{\n+\targv_array_pushf(&repack, \"--keep-pack=%s\", basename(item->string));\n+\treturn 0;\n+}\n+\n+static void add_repack_all_option(struct string_list *keep_pack)\n {\n \tif (prune_expire && !strcmp(prune_expire, \"now\"))\n \t\targv_array_push(&repack, \"-a\");\n@@ -197,6 +219,9 @@ static void add_repack_all_option(void)\n \t\tif (prune_expire)\n \t\t\targv_array_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n \t}\n+\n+\tif (keep_pack)\n+\t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n }\n \n static void add_repack_incremental_option(void)\n@@ -220,7 +245,7 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs())\n-\t\tadd_repack_all_option();\n+\t\tadd_repack_all_option(NULL);\n \telse if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n@@ -354,6 +379,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tconst char *name;\n \tpid_t pid;\n \tint daemonized = 0;\n+\tint keep_base_pack = -1;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n@@ -366,6 +392,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL_F(0, \"force\", &force,\n \t\t\t N_(\"force running gc even if there may be another gc running\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL(0, \"keep-largest-pack\", &keep_base_pack,\n+\t\t\t N_(\"repack all other packs except the largest pack\")),\n \t\tOPT_END()\n \t};\n \n@@ -431,8 +459,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t */\n \t\t\tdaemonized = !daemonize();\n \t\t}\n-\t} else\n-\t\tadd_repack_all_option();\n+\t} else {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (keep_base_pack != -1) {\n+\t\t\tif (keep_base_pack)\n+\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t}\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t}\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex d5255dd576..c42f60bc5b 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -43,6 +43,31 @@ test_expect_success 'gc is not aborted due to a stale symref' '\n \t)\n '\n \n+test_expect_success 'gc --keep-largest-pack' '\n+\ttest_create_repo keep-pack &&\n+\t(\n+\t\tcd keep-pack &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\t\ttest_commit three &&\n+\t\tgit gc &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 1 pack-list &&\n+\t\tBASE_PACK=.git/objects/pack/pack-*.pack &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d &&\n+\t\ttest_commit five &&\n+\t\tgit repack -d &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 3 pack-list &&\n+\t\tgit gc --keep-largest-pack &&\n+\t\t( cd .git/objects/pack && ls *.pack ) >pack-list &&\n+\t\ttest_line_count = 2 pack-list &&\n+\t\ttest_path_is_file $BASE_PACK &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_expect_success 'auto gc with too many loose objects does not attempt to create bitmaps' '\n \ttest_config gc.auto 3 &&\n \ttest_config gc.autodetach false &&\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344724","messageId":"20180415153618.32019-5-pclouds@gmail.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"[PATCH v2 4/7] gc: add gc.bigPackThreshold config","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-15T15:36:15Z","receivedAt":"2018-04-15T15:36:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The --keep-largest-pack option is not very convenient to use because\nyou need to tell gc to do this explicitly (and probably on just a few\nlarge repos).\n\nAdd a config key that enables this mode when packs larger than a limit\nare found. Note that there's a slight behavior difference compared to\n--keep-largest-pack: all packs larger than the threshold are kept, not\njust the largest one.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 7 +++++++\n Documentation/git-gc.txt | 6 ++++--\n builtin/gc.c | 26 ++++++++++++++++++++------\n 3 files changed, 31 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2659153cb3..728ae902e6 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1558,6 +1558,13 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.bigPackThreshold::\n+\tIf non-zero, all packs larger than this limit are kept when\n+\t`git gc` is run. This is very similar to `--keep-base-pack`\n+\texcept that all packs that meet the threshold are kept, not\n+\tjust the base pack. Defaults to zero. Common unit suffixes of\n+\t'k', 'm', or 'g' are supported.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 8f903231da..649faddfa6 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -56,7 +56,8 @@ single pack using `git repack -d -l`. Setting the value of `gc.auto`\n to 0 disables automatic packing of loose objects.\n +\n If the number of packs exceeds the value of `gc.autoPackLimit`,\n-then existing packs (except those marked with a `.keep` file)\n+then existing packs (except those marked with a `.keep` file\n+or over `gc.bigPackThreshold` limit)\n are consolidated into a single pack by using the `-A` option of\n 'git repack'. Setting `gc.autoPackLimit` to 0 disables\n automatic consolidation of packs.\n@@ -86,7 +87,8 @@ be performed as well.\n \n --keep-largest-pack::\n \tAll packs except the largest pack and those marked with a\n-\t`.keep` files are consolidated into a single pack.\n+\t`.keep` files are consolidated into a single pack. When this\n+\toption is used, `gc.bigPackThreshold` is ignored.\n \n Configuration\n -------------\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f251662a8f..81ecdc5ffa 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -41,6 +41,7 @@ static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n static const char *prune_worktrees_expire = \"3.months.ago\";\n+static unsigned long big_pack_threshold;\n \n static struct argv_array pack_refs_cmd = ARGV_ARRAY_INIT;\n static struct argv_array reflog = ARGV_ARRAY_INIT;\n@@ -128,6 +129,8 @@ static void gc_config(void)\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n \n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n+\n \tgit_config(git_default_config, NULL);\n }\n \n@@ -166,14 +169,17 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static void find_base_packs(struct string_list *packs)\n+static void find_base_packs(struct string_list *packs, unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n \tfor (p = get_packed_git(the_repository); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n-\t\tif (!base || base->pack_size < p->pack_size) {\n+\t\tif (limit) {\n+\t\t\tif (p->pack_size >= limit)\n+\t\t\t\tstring_list_append(packs, p->pack_name);\n+\t\t} else if (!base || base->pack_size < p->pack_size) {\n \t\t\tbase = p;\n \t\t}\n \t}\n@@ -244,9 +250,15 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs())\n-\t\tadd_repack_all_option(NULL);\n-\telse if (too_many_loose_objects())\n+\tif (too_many_packs()) {\n+\t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n+\n+\t\tif (big_pack_threshold)\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\n+\t\tadd_repack_all_option(&keep_pack);\n+\t\tstring_list_clear(&keep_pack, 0);\n+\t} else if (too_many_loose_objects())\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -464,7 +476,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\tif (keep_base_pack != -1) {\n \t\t\tif (keep_base_pack)\n-\t\t\t\tfind_base_packs(&keep_pack);\n+\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t} else if (big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n-- \n2.17.0.367.g5dd2e386c3\n\n"},{"id":"344771","messageId":"xmqqmuy3ycsg.fsf@gitster-ct.c.googlers.com","threadId":"48285","inReplyTo":"20180415153618.32019-1-pclouds@gmail.com","subject":"Re: [PATCH v2 0/7] nd/repack-keep-pack update","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-16T04:51:11Z","receivedAt":"2018-04-16T04:51:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> On Sat, Apr 14, 2018 at 9:47 PM, Ævar Arnfjörð Bjarmason wrote:\n>> The only (trivial) issue I found in the patches themselves was that\n>> between 4/5 and 5/7 you're adding an empty line to config.txt in 4/7\n>> just to erase it in 5/7, better not to add it to begin with, but\n>> hopefully Junio can fix that up (if he cares).\n>\n> Fixed (and is the only change in v2) in case Junio has not picked up\n> the last round and fixed it himself yet.\n\nThanks.\n"}]}