{"thread":{"id":"30178","subject":"[RFC/PATCH] gc: do not explode objects which will be immediately pruned","startedAt":"2012-04-07T10:30:09Z","lastAt":"2012-04-11T18:31:37Z","messageCount":3,"participants":["Jeff King","Nicolas Pitre","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"188727","messageId":"20120407103008.GA11242@sigill.intra.peff.net","threadId":"30178","inReplyTo":null,"subject":"[RFC/PATCH] gc: do not explode objects which will be immediately pruned","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-04-07T10:30:09Z","receivedAt":"2012-04-07T10:30:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"When we pack everything into one big pack with \"git repack\n-Ad\", any unreferenced objects in to-be-deleted packs are\nexploded into loose objects, with the intent that they will\nbe examined and possibly cleaned up by the next run of \"git\nprune\".\n\nSince the exploded objects will receive the mtime of the\npack from which they come, if the source pack is old, those\nloose objects will end up pruned immediately. In that case,\nit is much more efficient to skip the exploding step\nentirely for these objects.\n\nThis patch teaches pack-objects to receive the expiration\ninformation and avoid writing these objects out. It also\nteaches \"git gc\" to pass the value of gc.pruneexpire to\nrepack (which in turn learns to pass it along to\npack-objects) so that this optimization happens\nautomatically during \"git gc\" and \"git gc --auto\".\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n builtin/gc.c                         |   23 +++++++++++++++--------\n builtin/pack-objects.c               |   25 +++++++++++++++++++++++--\n git-repack.sh                        |   10 +++++++++-\n t/t7701-repack-unpack-unreachable.sh |   14 ++++++++++++++\n 4 files changed, 61 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 271376d..1bc2fe3 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -144,6 +144,19 @@ static int too_many_packs(void)\n \treturn gc_auto_pack_limit <= cnt;\n }\n \n+static void add_repack_all_option(void)\n+{\n+\tif (prune_expire && !strcmp(prune_expire, \"now\"))\n+\t\tappend_option(argv_repack, \"-a\", MAX_ADD);\n+\telse {\n+\t\tappend_option(argv_repack, \"-A\", MAX_ADD);\n+\t\tif (prune_expire) {\n+\t\t\tappend_option(argv_repack, \"--unpack-unreachable\", MAX_ADD);\n+\t\t\tappend_option(argv_repack, prune_expire, MAX_ADD);\n+\t\t}\n+\t}\n+}\n+\n static int need_to_gc(void)\n {\n \t/*\n@@ -160,10 +173,7 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs())\n-\t\tappend_option(argv_repack,\n-\t\t\t      prune_expire && !strcmp(prune_expire, \"now\") ?\n-\t\t\t      \"-a\" : \"-A\",\n-\t\t\t      MAX_ADD);\n+\t\tadd_repack_all_option();\n \telse if (!too_many_loose_objects())\n \t\treturn 0;\n \n@@ -227,10 +237,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t\"run \\\"git gc\\\" manually. See \"\n \t\t\t\t\t\"\\\"git help gc\\\" for more information.\\n\"));\n \t} else\n-\t\tappend_option(argv_repack,\n-\t\t\t      prune_expire && !strcmp(prune_expire, \"now\")\n-\t\t\t      ? \"-a\" : \"-A\",\n-\t\t\t      MAX_ADD);\n+\t\tadd_repack_all_option();\n \n \tif (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))\n \t\treturn error(FAILED_RUN, argv_pack_refs[0]);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7b07c09..1861093 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -63,6 +63,7 @@ static uint32_t nr_objects, nr_alloc, nr_result, nr_written;\n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n static int keep_unreachable, unpack_unreachable, include_tag;\n+static unsigned long unpack_unreachable_expiration;\n static int local;\n static int incremental;\n static int ignore_packed_keep;\n@@ -2249,6 +2250,10 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \t\tif (!p->pack_local || p->pack_keep)\n \t\t\tcontinue;\n \n+\t\tif (unpack_unreachable_expiration &&\n+\t\t    p->mtime < unpack_unreachable_expiration)\n+\t\t\tcontinue;\n+\n \t\tif (open_pack_index(p))\n \t\t\tdie(\"cannot open pack index\");\n \n@@ -2315,6 +2320,21 @@ static int option_parse_index_version(const struct option *opt,\n \treturn 0;\n }\n \n+static int option_parse_unpack_unreachable(const struct option *opt,\n+\t\t\t\t\t   const char *arg, int unset)\n+{\n+\tif (unset) {\n+\t\tunpack_unreachable = 0;\n+\t\tunpack_unreachable_expiration = 0;\n+\t}\n+\telse {\n+\t\tunpack_unreachable = 1;\n+\t\tif (arg)\n+\t\t\tunpack_unreachable_expiration = approxidate(arg);\n+\t}\n+\treturn 0;\n+}\n+\n static int option_parse_ulong(const struct option *opt,\n \t\t\t      const char *arg, int unset)\n {\n@@ -2392,8 +2412,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t \"include tag objects that refer to objects to be packed\"),\n \t\tOPT_BOOL(0, \"keep-unreachable\", &keep_unreachable,\n \t\t\t \"keep unreachable objects\"),\n-\t\tOPT_BOOL(0, \"unpack-unreachable\", &unpack_unreachable,\n-\t\t\t \"unpack unreachable objects\"),\n+\t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, \"time\",\n+\t\t  \"unpack unreachable objects newer than <time>\",\n+\t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t \"create thin packs\"),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 624feec..7579331 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -15,6 +15,7 @@ F               pass --no-reuse-object to git-pack-objects\n n               do not run git-update-server-info\n q,quiet         be quiet\n l               pass --local to git-pack-objects\n+unpack-unreachable=  with -A, do not loosen objects older than this\n  Packing constraints\n window=         size of the window used for delta compression\n window-memory=  same as the above, but limit memory size instead of entries count\n@@ -33,6 +34,8 @@ do\n \t-a)\tall_into_one=t ;;\n \t-A)\tall_into_one=t\n \t\tunpack_unreachable=--unpack-unreachable ;;\n+\t--unpack-unreachable)\n+\t\tunpack_unreachable=\"--unpack-unreachable=$2\"; shift ;;\n \t-d)\tremove_redundant=t ;;\n \t-q)\tGIT_QUIET=t ;;\n \t-f)\tno_reuse=--no-reuse-delta ;;\n@@ -76,7 +79,12 @@ case \",$all_into_one,\" in\n \t\tif test -n \"$existing\" -a -n \"$unpack_unreachable\" -a \\\n \t\t\t-n \"$remove_redundant\"\n \t\tthen\n-\t\t\targs=\"$args $unpack_unreachable\"\n+\t\t\t# This may have arbitrary user arguments, so we\n+\t\t\t# have to protect it against whitespace splitting\n+\t\t\t# when it gets run as \"pack-objects $args\" later.\n+\t\t\t# Fortunately, we know it's an approxidate, so we\n+\t\t\t# can just use dots instead.\n+\t\t\targs=\"$args $(echo \"$unpack_unreachable\" | tr ' ' .)\"\n \t\tfi\n \tfi\n \t;;\ndiff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh\nindex 200ab61..b8d4cde 100755\n--- a/t/t7701-repack-unpack-unreachable.sh\n+++ b/t/t7701-repack-unpack-unreachable.sh\n@@ -95,4 +95,18 @@ test_expect_success 'unpacked objects receive timestamp of pack file' '\n \tcompare_mtimes < mtimes\n '\n \n+test_expect_success 'do not bother loosening old objects' '\n+\tobj1=$(echo one | git hash-object -w --stdin) &&\n+\tobj2=$(echo two | git hash-object -w --stdin) &&\n+\tpack1=$(echo $obj1 | git pack-objects .git/objects/pack/pack) &&\n+\tpack2=$(echo $obj2 | git pack-objects .git/objects/pack/pack) &&\n+\tgit prune-packed &&\n+\tgit cat-file -p $obj1 &&\n+\tgit cat-file -p $obj2 &&\n+\ttest-chmtime =-86400 .git/objects/pack/pack-$pack2.pack &&\n+\tgit repack -A -d --unpack-unreachable=1.hour.ago &&\n+\tgit cat-file -p $obj1 &&\n+\ttest_must_fail git cat-file -p $obj2\n+'\n+\n test_done\n-- \n1.7.10.11.g901cee\n"},{"id":"188905","messageId":"alpine.LFD.2.02.1204101623010.16314@xanadu.home","threadId":"30178","inReplyTo":"20120407103008.GA11242@sigill.intra.peff.net","subject":"Re: [RFC/PATCH] gc: do not explode objects which will be immediately pruned","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2012-04-10T20:24:17Z","receivedAt":"2012-04-10T20:24:17Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 7 Apr 2012, Jeff King wrote:\n\n> When we pack everything into one big pack with \"git repack\n> -Ad\", any unreferenced objects in to-be-deleted packs are\n> exploded into loose objects, with the intent that they will\n> be examined and possibly cleaned up by the next run of \"git\n> prune\".\n> \n> Since the exploded objects will receive the mtime of the\n> pack from which they come, if the source pack is old, those\n> loose objects will end up pruned immediately. In that case,\n> it is much more efficient to skip the exploding step\n> entirely for these objects.\n> \n> This patch teaches pack-objects to receive the expiration\n> information and avoid writing these objects out. It also\n> teaches \"git gc\" to pass the value of gc.pruneexpire to\n> repack (which in turn learns to pass it along to\n> pack-objects) so that this optimization happens\n> automatically during \"git gc\" and \"git gc --auto\".\n> \n> Signed-off-by: Jeff King <peff@peff.net>\n\nLooks fine to me.\n\nAcked-by: Nicolas Pitre <nico@fluxnic.net>\n\n\n> ---\n>  builtin/gc.c                         |   23 +++++++++++++++--------\n>  builtin/pack-objects.c               |   25 +++++++++++++++++++++++--\n>  git-repack.sh                        |   10 +++++++++-\n>  t/t7701-repack-unpack-unreachable.sh |   14 ++++++++++++++\n>  4 files changed, 61 insertions(+), 11 deletions(-)\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 271376d..1bc2fe3 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -144,6 +144,19 @@ static int too_many_packs(void)\n>  \treturn gc_auto_pack_limit <= cnt;\n>  }\n>  \n> +static void add_repack_all_option(void)\n> +{\n> +\tif (prune_expire && !strcmp(prune_expire, \"now\"))\n> +\t\tappend_option(argv_repack, \"-a\", MAX_ADD);\n> +\telse {\n> +\t\tappend_option(argv_repack, \"-A\", MAX_ADD);\n> +\t\tif (prune_expire) {\n> +\t\t\tappend_option(argv_repack, \"--unpack-unreachable\", MAX_ADD);\n> +\t\t\tappend_option(argv_repack, prune_expire, MAX_ADD);\n> +\t\t}\n> +\t}\n> +}\n> +\n>  static int need_to_gc(void)\n>  {\n>  \t/*\n> @@ -160,10 +173,7 @@ static int need_to_gc(void)\n>  \t * there is no need.\n>  \t */\n>  \tif (too_many_packs())\n> -\t\tappend_option(argv_repack,\n> -\t\t\t      prune_expire && !strcmp(prune_expire, \"now\") ?\n> -\t\t\t      \"-a\" : \"-A\",\n> -\t\t\t      MAX_ADD);\n> +\t\tadd_repack_all_option();\n>  \telse if (!too_many_loose_objects())\n>  \t\treturn 0;\n>  \n> @@ -227,10 +237,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n>  \t\t\t\t\t\"run \\\"git gc\\\" manually. See \"\n>  \t\t\t\t\t\"\\\"git help gc\\\" for more information.\\n\"));\n>  \t} else\n> -\t\tappend_option(argv_repack,\n> -\t\t\t      prune_expire && !strcmp(prune_expire, \"now\")\n> -\t\t\t      ? \"-a\" : \"-A\",\n> -\t\t\t      MAX_ADD);\n> +\t\tadd_repack_all_option();\n>  \n>  \tif (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))\n>  \t\treturn error(FAILED_RUN, argv_pack_refs[0]);\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 7b07c09..1861093 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -63,6 +63,7 @@ static uint32_t nr_objects, nr_alloc, nr_result, nr_written;\n>  static int non_empty;\n>  static int reuse_delta = 1, reuse_object = 1;\n>  static int keep_unreachable, unpack_unreachable, include_tag;\n> +static unsigned long unpack_unreachable_expiration;\n>  static int local;\n>  static int incremental;\n>  static int ignore_packed_keep;\n> @@ -2249,6 +2250,10 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n>  \t\tif (!p->pack_local || p->pack_keep)\n>  \t\t\tcontinue;\n>  \n> +\t\tif (unpack_unreachable_expiration &&\n> +\t\t    p->mtime < unpack_unreachable_expiration)\n> +\t\t\tcontinue;\n> +\n>  \t\tif (open_pack_index(p))\n>  \t\t\tdie(\"cannot open pack index\");\n>  \n> @@ -2315,6 +2320,21 @@ static int option_parse_index_version(const struct option *opt,\n>  \treturn 0;\n>  }\n>  \n> +static int option_parse_unpack_unreachable(const struct option *opt,\n> +\t\t\t\t\t   const char *arg, int unset)\n> +{\n> +\tif (unset) {\n> +\t\tunpack_unreachable = 0;\n> +\t\tunpack_unreachable_expiration = 0;\n> +\t}\n> +\telse {\n> +\t\tunpack_unreachable = 1;\n> +\t\tif (arg)\n> +\t\t\tunpack_unreachable_expiration = approxidate(arg);\n> +\t}\n> +\treturn 0;\n> +}\n> +\n>  static int option_parse_ulong(const struct option *opt,\n>  \t\t\t      const char *arg, int unset)\n>  {\n> @@ -2392,8 +2412,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n>  \t\t\t \"include tag objects that refer to objects to be packed\"),\n>  \t\tOPT_BOOL(0, \"keep-unreachable\", &keep_unreachable,\n>  \t\t\t \"keep unreachable objects\"),\n> -\t\tOPT_BOOL(0, \"unpack-unreachable\", &unpack_unreachable,\n> -\t\t\t \"unpack unreachable objects\"),\n> +\t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, \"time\",\n> +\t\t  \"unpack unreachable objects newer than <time>\",\n> +\t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n>  \t\tOPT_BOOL(0, \"thin\", &thin,\n>  \t\t\t \"create thin packs\"),\n>  \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n> diff --git a/git-repack.sh b/git-repack.sh\n> index 624feec..7579331 100755\n> --- a/git-repack.sh\n> +++ b/git-repack.sh\n> @@ -15,6 +15,7 @@ F               pass --no-reuse-object to git-pack-objects\n>  n               do not run git-update-server-info\n>  q,quiet         be quiet\n>  l               pass --local to git-pack-objects\n> +unpack-unreachable=  with -A, do not loosen objects older than this\n>   Packing constraints\n>  window=         size of the window used for delta compression\n>  window-memory=  same as the above, but limit memory size instead of entries count\n> @@ -33,6 +34,8 @@ do\n>  \t-a)\tall_into_one=t ;;\n>  \t-A)\tall_into_one=t\n>  \t\tunpack_unreachable=--unpack-unreachable ;;\n> +\t--unpack-unreachable)\n> +\t\tunpack_unreachable=\"--unpack-unreachable=$2\"; shift ;;\n>  \t-d)\tremove_redundant=t ;;\n>  \t-q)\tGIT_QUIET=t ;;\n>  \t-f)\tno_reuse=--no-reuse-delta ;;\n> @@ -76,7 +79,12 @@ case \",$all_into_one,\" in\n>  \t\tif test -n \"$existing\" -a -n \"$unpack_unreachable\" -a \\\n>  \t\t\t-n \"$remove_redundant\"\n>  \t\tthen\n> -\t\t\targs=\"$args $unpack_unreachable\"\n> +\t\t\t# This may have arbitrary user arguments, so we\n> +\t\t\t# have to protect it against whitespace splitting\n> +\t\t\t# when it gets run as \"pack-objects $args\" later.\n> +\t\t\t# Fortunately, we know it's an approxidate, so we\n> +\t\t\t# can just use dots instead.\n> +\t\t\targs=\"$args $(echo \"$unpack_unreachable\" | tr ' ' .)\"\n>  \t\tfi\n>  \tfi\n>  \t;;\n> diff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh\n> index 200ab61..b8d4cde 100755\n> --- a/t/t7701-repack-unpack-unreachable.sh\n> +++ b/t/t7701-repack-unpack-unreachable.sh\n> @@ -95,4 +95,18 @@ test_expect_success 'unpacked objects receive timestamp of pack file' '\n>  \tcompare_mtimes < mtimes\n>  '\n>  \n> +test_expect_success 'do not bother loosening old objects' '\n> +\tobj1=$(echo one | git hash-object -w --stdin) &&\n> +\tobj2=$(echo two | git hash-object -w --stdin) &&\n> +\tpack1=$(echo $obj1 | git pack-objects .git/objects/pack/pack) &&\n> +\tpack2=$(echo $obj2 | git pack-objects .git/objects/pack/pack) &&\n> +\tgit prune-packed &&\n> +\tgit cat-file -p $obj1 &&\n> +\tgit cat-file -p $obj2 &&\n> +\ttest-chmtime =-86400 .git/objects/pack/pack-$pack2.pack &&\n> +\tgit repack -A -d --unpack-unreachable=1.hour.ago &&\n> +\tgit cat-file -p $obj1 &&\n> +\ttest_must_fail git cat-file -p $obj2\n> +'\n> +\n>  test_done\n> -- \n> 1.7.10.11.g901cee\n> \n"},{"id":"189012","messageId":"7v398aktme.fsf@alter.siamese.dyndns.org","threadId":"30178","inReplyTo":"alpine.LFD.2.02.1204101623010.16314@xanadu.home","subject":"Re: [RFC/PATCH] gc: do not explode objects which will be immediately pruned","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-04-11T18:31:37Z","receivedAt":"2012-04-11T18:31:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@fluxnic.net> writes:\n\n>> ...\n>> This patch teaches pack-objects to receive the expiration\n>> information and avoid writing these objects out. It also\n>> teaches \"git gc\" to pass the value of gc.pruneexpire to\n>> repack (which in turn learns to pass it along to\n>> pack-objects) so that this optimization happens\n>> automatically during \"git gc\" and \"git gc --auto\".\n>> \n>> Signed-off-by: Jeff King <peff@peff.net>\n>\n> Looks fine to me.\n>\n> Acked-by: Nicolas Pitre <nico@fluxnic.net>\n\nThanks, both.\n"}]}