{"thread":{"id":"13506","subject":"[PATCH 2/2] let pack-objects do the writing of unreachable objects as loose objects","startedAt":"2008-05-14T05:33:53Z","lastAt":"2008-05-14T21:06:10Z","messageCount":3,"participants":["Nicolas Pitre","Juergen Ruehle","Alex Riesen"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"76915","messageId":"alpine.LFD.1.10.0805140132500.23581@xanadu.home","threadId":"13506","inReplyTo":null,"subject":"[PATCH 2/2] let pack-objects do the writing of unreachable objects as loose objects","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-14T05:33:53Z","receivedAt":"2008-05-14T05:33:53Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\nCommit ccc1297226b184c40459e9d373cc9eebfb7bd898 changed the behavior\nof 'git repack -A' so unreachable objects are stored as loose objects.\nHowever it did so in a naive and inn efficient way by making packs\nabout to be deleted inaccessible and feeding their content through\n'git unpack-objects'.  While this works, there are major flaws with\nthis approach:\n\n- It is unacceptably sloooooooooooooow.\n\n  In the Linux kernel repository with no actual unreachable objects,\n  doing 'git repack -A -d' before:\n\n\treal    2m33.220s\n\tuser    2m21.675s\n\tsys     0m3.510s\n\n  And with this change:\n\n\treal    0m36.849s\n\tuser    0m24.365s\n\tsys     0m1.950s\n\n  For reference, here's the timing for 'git repack -a -d':\n\n\treal    0m35.816s\n\tuser    0m22.571s\n\tsys     0m2.011s\n\n  This is explained by the fact that 'git unpack-objects' was used to\n  unpack _every_ objects even if (almost) 100% of them were thrown away.\n\n- There is a black out period.\n\n  Between the removal of the .idx file for the redundant pack and the\n  completion of its unpacking, the unreachable objects become completely\n  unaccessible.  This is not a big issue as we're talking about unreachable\n  objects, but some consistency is always good.\n\n- There is no way to easily set a sensible mtime for the newly created\n  unreachable loose objects.\n\nSo, while having a command called \"pack-objects\" to perform object\nunpacking looks really odd, this is probably the best compromize to be\nable to solve the above issues in an efficient way.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex e20851e..f43eb67 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -28,7 +28,8 @@ git-pack-objects [{ -q | --progress | --all-progress }] \\n\\\n \t[--window=N] [--window-memory=N] [--depth=N] \\n\\\n \t[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \\n\\\n \t[--threads=N] [--non-empty] [--revs [--unpacked | --all]*] [--reflog] \\n\\\n-\t[--stdout | base-name] [--include-tag] [--keep-unreachable] \\n\\\n+\t[--stdout | base-name] [--include-tag] \\n\\\n+\t[--keep-unreachable | --unpack-unreachable] \\n\\\n \t[<ref-list | <object-list]\";\n \n struct object_entry {\n@@ -67,7 +68,7 @@ static uint32_t nr_objects, nr_alloc, nr_result, nr_written;\n \n static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n-static int keep_unreachable, include_tag;\n+static int keep_unreachable, unpack_unreachable, include_tag;\n static int local;\n static int incremental;\n static int allow_ofs_delta;\n@@ -1946,6 +1947,32 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \tfree(in_pack.array);\n }\n \n+static void loosen_unused_packed_objects(struct rev_info *revs)\n+{\n+\tstruct packed_git *p;\n+\tuint32_t i;\n+\tconst unsigned char *sha1;\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tfor (i = 0; i < revs->num_ignore_packed; i++) {\n+\t\t\tif (matches_pack_name(p, revs->ignore_packed[i]))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (revs->num_ignore_packed <= i)\n+\t\t\tcontinue;\n+\n+\t\tif (open_pack_index(p))\n+\t\t\tdie(\"cannot open pack index\");\n+\n+\t\tfor (i = 0; i < p->num_objects; i++) {\n+\t\t\tsha1 = nth_packed_object_sha1(p, i);\n+\t\t\tif (!locate_object_entry(sha1))\n+\t\t\t\tif (force_object_loose(sha1, p->mtime))\n+\t\t\t\t\tdie(\"unable to force loose object\");\n+\t\t}\n+\t}\n+}\n+\n static void get_object_list(int ac, const char **av)\n {\n \tstruct rev_info revs;\n@@ -1980,6 +2007,8 @@ static void get_object_list(int ac, const char **av)\n \n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs(&revs);\n+\tif (unpack_unreachable)\n+\t\tloosen_unused_packed_objects(&revs);\n }\n \n static int adjust_perm(const char *path, mode_t mode)\n@@ -2114,6 +2143,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\tkeep_unreachable = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(\"--unpack-unreachable\", arg)) {\n+\t\t\tunpack_unreachable = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(\"--include-tag\", arg)) {\n \t\t\tinclude_tag = 1;\n \t\t\tcontinue;\n@@ -2179,6 +2212,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!pack_to_stdout && thin)\n \t\tdie(\"--thin cannot be used to build an indexable pack.\");\n \n+\tif (keep_unreachable && unpack_unreachable)\n+\t\tdie(\"--keep-unreachable and --unpack-unreachable are incompatible.\");\n+\n #ifdef THREADED_DELTA_SEARCH\n \tif (!delta_search_threads)\t/* --threads=0 means autodetect */\n \t\tdelta_search_threads = online_cpus();\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 0aede89..10f735c 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -8,7 +8,7 @@ OPTIONS_SPEC=\"\\\n git-repack [options]\n --\n a               pack everything in a single pack\n-A               same as -a, and keep unreachable objects too\n+A               same as -a, and turn unreachable objects loose\n d               remove redundant packs, and run git-prune-packed\n f               pass --no-reuse-delta to git-pack-objects\n n               do not run git-update-server-info\n@@ -23,7 +23,7 @@ max-pack-size=  maximum size of each packfile\n SUBDIRECTORY_OK='Yes'\n . git-sh-setup\n \n-no_update_info= all_into_one= remove_redundant= keep_unreachable=\n+no_update_info= all_into_one= remove_redundant= unpack_unreachable=\n local= quiet= no_reuse= extra=\n while test $# != 0\n do\n@@ -31,7 +31,7 @@ do\n \t-n)\tno_update_info=t ;;\n \t-a)\tall_into_one=t ;;\n \t-A)\tall_into_one=t\n-\t\tkeep_unreachable=t ;;\n+\t\tunpack_unreachable=--unpack-unreachable ;;\n \t-d)\tremove_redundant=t ;;\n \t-q)\tquiet=-q ;;\n \t-f)\tno_reuse=--no-reuse-object ;;\n@@ -79,6 +79,9 @@ case \",$all_into_one,\" in\n \tif test -z \"$args\"\n \tthen\n \t\targs='--unpacked --incremental'\n+\telif test -n \"$unpack_unreachable\"\n+\tthen\n+\t\targs=\"$args $unpack_unreachable\"\n \tfi\n \t;;\n esac\n@@ -128,18 +131,7 @@ then\n \t\t  do\n \t\t\tcase \" $fullbases \" in\n \t\t\t*\" $e \"*) ;;\n-\t\t\t*)\n-\t\t\t\trm -f \"$e.idx\" \"$e.keep\"\n-\t\t\t\tif test -n \"$keep_unreachable\" &&\n-\t\t\t\t   test -f \"$e.pack\"\n-\t\t\t\tthen\n-\t\t\t\t\tgit unpack-objects < \"$e.pack\" || {\n-\t\t\t\t\t\techo >&2 \"Failed unpacking unreachable objects from redundant pack file $e.pack\"\n-\t\t\t\t\t\texit 1\n-\t\t\t\t\t}\n-\t\t\t\tfi\n-\t\t\t\trm -f \"$e.pack\"\n-\t\t\t;;\n+\t\t\t*)\trm -f \"$e.pack\" \"$e.idx\" \"$e.keep\" ;;\n \t\t\tesac\n \t\t  done\n \t\t)\n"},{"id":"76933","messageId":"18474.44264.479000.403927@lapjr.intranet.kiel.bmiag.de","threadId":"13506","inReplyTo":"alpine.LFD.1.10.0805140132500.23581@xanadu.home","subject":"Re: [PATCH 2/2] let pack-objects do the writing of unreachable objects as loose objects","fromName":"Juergen Ruehle","fromEmail":"j.ruehle@bmiag.de","sentAt":"2008-05-14T09:12:08Z","receivedAt":"2008-05-14T09:12:08Z","isPatch":true,"sender":{"key":"j.ruehle@bmiag.de","avatar":null},"body":"Nicolas Pitre writes:\n > \n > Commit ccc1297226b184c40459e9d373cc9eebfb7bd898 changed the behavior\n > of 'git repack -A' so unreachable objects are stored as loose objects.\n > However it did so in a naive and inn efficient way by making packs\n > about to be deleted inaccessible and feeding their content through\n > 'git unpack-objects'.  While this works, there are major flaws with\n > this approach:\n\nDoesn't this make git gc --auto a dangerous operation?\n"},{"id":"76992","messageId":"20080514210610.GA11109@steel.home","threadId":"13506","inReplyTo":"alpine.LFD.1.10.0805140132500.23581@xanadu.home","subject":"Re: [PATCH 2/2] let pack-objects do the writing of unreachable objects as loose objects","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2008-05-14T21:06:10Z","receivedAt":"2008-05-14T21:06:10Z","isPatch":true,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Nicolas Pitre, Wed, May 14, 2008 07:33:53 +0200:\n> @@ -67,7 +68,7 @@ static uint32_t nr_objects, nr_alloc, nr_result, nr_written;\n>  \n>  static int non_empty;\n>  static int reuse_delta = 1, reuse_object = 1;\n> -static int keep_unreachable, include_tag;\n> +static int keep_unreachable, unpack_unreachable, include_tag;\n\nIf the options are incompatible anyway, why not\n\nenum\n{\n    NO_UNREACHABLE,\n    KEEP_UNREACHABLE,\n    UNPACK_UNREACHABLE,\n};\n\nstatic int unreachable_handling, include_tag;\n\n?\n"}]}