{"thread":{"id":"42586","subject":"Repacking a repository uses up all available disk space","startedAt":"2016-06-16T02:19:53Z","lastAt":"2016-06-16T02:19:53Z","messageCount":11,"participants":["Konstantin Ryabitsev","Jeff King","Duy Nguyen","Nasser Grainawi"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"289061","messageId":"20160612212514.GA4584@gmail.com","threadId":"42586","inReplyTo":null,"subject":"Repacking a repository uses up all available disk space","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2016-06-12T21:25:14Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"Hello:\n\nI have a problematic repository that:\n\n- Takes up 9GB on disk\n- Passes 'git fsck --full' with no errors\n- When cloned with --mirror, takes up 38M on the target system\n- When attempting to repack, creates millions of files and eventually\n eats up all available disk space\n\nRepacking the result of 'git clone --mirror' shows no problem, so it's\ngot to be something really weird with that particular instance of the\nrepository.\n\nIf anyone is interested in poking at this particular problem to figure\nout what causes the repack process to eat up all available disk space,\nyou can find the tarball of the problematic repository here:\n\nhttp://mricon.com/misc/src.git.tar.xz (warning: 6.6GB)\n\nYou can clone the non-problematic version of this repository from\ngit://codeaurora.org/quic/chrome4sdp/breakpad/breakpad/src.git\n\nBest,\n-- \nKonstantin Ryabitsev\nLinux Foundation Collab Projects\nMontréal, Québec\n"},{"id":"289062","messageId":"20160612213804.GA5428@sigill.intra.peff.net","threadId":"42586","inReplyTo":"20160612212514.GA4584@gmail.com","subject":"Re: Repacking a repository uses up all available disk space","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-12T21:38:04Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jun 12, 2016 at 05:25:14PM -0400, Konstantin Ryabitsev wrote:\n\n> Hello:\n> \n> I have a problematic repository that:\n> \n> - Takes up 9GB on disk\n> - Passes 'git fsck --full' with no errors\n> - When cloned with --mirror, takes up 38M on the target system\n\nCloning will only copy the objects that are reachable from the refs. So\npresumably the other 8.9GB is either reachable from reflogs, or not\nreachable at all (due to rewinding history or deleting branches).\n\n> - When attempting to repack, creates millions of files and eventually\n> eats up all available disk space\n\nThat means these objects fall into the unreachable category. Git will\nprune unreachable loose objects after a grace period based on the\nfilesystem mtime of the objects; the default is 2 weeks.\n\nFor unreachable packed objects, their mtime is jumbled in with the rest\nof the objects in the packfile. So Git's strategy is to \"eject\" such\nobjects from the packfiles into individual loose objects, and let them\n\"age out\" of the grace period individually.\n\nGenerally this works just fine, but there are corner cases where you\nmight have a very large number of such objects, and the loose storage is\nmuch more expensive than the packed (e.g., because each object is stored\nindividually, not as a delta).\n\nIt sounds like this is the case you're running into.\n\nThe solution is to lower the grace period time, with something like:\n\n git gc --prune=5.minutes.ago\n\nor even:\n\n git gc --prune=now\n\nThat will prune the unreachable objects immediately (and the packfile\nejector is smart enough to skip ejecting any file that would just get\ndeleted immediately anyway).\n\n-Peff\n"},{"id":"289063","messageId":"20160612215436.GB4584@gmail.com","threadId":"42586","inReplyTo":"20160612213804.GA5428@sigill.intra.peff.net","subject":"Re: Repacking a repository uses up all available disk space","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2016-06-12T21:54:36Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Sun, Jun 12, 2016 at 05:38:04PM -0400, Jeff King wrote:\n> > - When attempting to repack, creates millions of files and eventually\n> > eats up all available disk space\n> \n> That means these objects fall into the unreachable category. Git will\n> prune unreachable loose objects after a grace period based on the\n> filesystem mtime of the objects; the default is 2 weeks.\n> \n> For unreachable packed objects, their mtime is jumbled in with the rest\n> of the objects in the packfile. So Git's strategy is to \"eject\" such\n> objects from the packfiles into individual loose objects, and let them\n> \"age out\" of the grace period individually.\n> \n> Generally this works just fine, but there are corner cases where you\n> might have a very large number of such objects, and the loose storage is\n> much more expensive than the packed (e.g., because each object is stored\n> individually, not as a delta).\n> \n> It sounds like this is the case you're running into.\n> \n> The solution is to lower the grace period time, with something like:\n> \n> git gc --prune=5.minutes.ago\n> \n> or even:\n> \n> git gc --prune=now\n\nYou are correct, this solves the problem, however I'm curious. The usual\nmaintenance for these repositories is a regular run of:\n\n- git fsck --full\n- git repack -Adl -b --pack-kept-objects\n- git pack-refs --all\n- git prune\n\nThe reason it's split into repack + prune instead of just gc is because\nwe use alternates to save on disk space and try not to prune repos that\nare used as alternates by other repos in order to avoid potential\ncorruption.\n\nAm I not doing something that needs to be doing in order to avoid the\nsame problem?\n\nThanks for your help.\n\nRegards,\n-- \nKonstantin Ryabitsev\nLinux Foundation Collab Projects\nMontréal, Québec\n"},{"id":"289065","messageId":"20160612221309.GC5428@sigill.intra.peff.net","threadId":"42586","inReplyTo":"20160612215436.GB4584@gmail.com","subject":"Re: Repacking a repository uses up all available disk space","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-12T22:13:09Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jun 12, 2016 at 05:54:36PM -0400, Konstantin Ryabitsev wrote:\n\n> > git gc --prune=now\n> \n> You are correct, this solves the problem, however I'm curious. The usual\n> maintenance for these repositories is a regular run of:\n> \n> - git fsck --full\n> - git repack -Adl -b --pack-kept-objects\n> - git pack-refs --all\n> - git prune\n> \n> The reason it's split into repack + prune instead of just gc is because\n> we use alternates to save on disk space and try not to prune repos that\n> are used as alternates by other repos in order to avoid potential\n> corruption.\n> \n> Am I not doing something that needs to be doing in order to avoid the\n> same problem?\n\nYour approach makes sense; we do the same thing at GitHub for the same\nreasons[1]. The main thing you are missing that gc will do is that it\nknows the prune-time it is going to feed to git-prune[2], and passes\nthat along to repack. That's what enables the \"don't bother ejecting\nthese, because I'm about to delete them\" optimization.\n\nThat option is not documented, because it was always assumed to be an\ninternal thing to git-gc, but it is:\n\n git repack ... --unpack-unreachable=5.minutes.ago\n\nor whatever.\n\n-Peff\n\n[1] We don't run the fsck at the front, though, because it's really\n expensive. I'm not sure it buys you much, either. The repack\n will do a full walk of the graph, so it gets you a connectivity\n check, as well as a full content check of the commits and trees. The\n blobs are copied as-is from the old pack, but there is a checksum on\n the pack data (to catch any bit flips by the disk storage). So the\n only thing the fsck is getting you is that it fully reconstructs the\n deltas for each blob and checks their sha1. That's more robust than\n a checksum, but it's a lot more expensive.\n\n[2] It's unclear to me if you're passing any options to git-prune, but\n you may want to pass \"--expire\" with a short grace period. Without\n any options it prunes every unreachable thing, which can lead to\n races if the repository is actively being used.\n\n At GitHub we actually have a patch to `repack` that keeps all\n objects, reachable or not, in the pack, and use it for all of our\n automated maintenance. Since we don't drop objects at all, we can't\n ever have such a race. Aside from some pathological cases, it wastes\n much less space than you'd expect. We turn the flag off for special\n cases (e.g., somebody has rewound history and wants to expunge a\n sensitive object).\n\n I'm happy to share the \"keep everything\" patch if you're interested.\n"},{"id":"289067","messageId":"CACsJy8Awd2oCm0puh=bnKu9snOZr85+kVRe0D5DUhP6NhmiwcQ@mail.gmail.com","threadId":"42586","inReplyTo":"20160612221309.GC5428@sigill.intra.peff.net","subject":"Re: Repacking a repository uses up all available disk space","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-13T00:24:51Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 13, 2016 at 5:13 AM, Jeff King wrote:\n> On Sun, Jun 12, 2016 at 05:54:36PM -0400, Konstantin Ryabitsev wrote:\n>\n>> > git gc --prune=now\n>>\n>> You are correct, this solves the problem, however I'm curious. The usual\n>> maintenance for these repositories is a regular run of:\n>>\n>> - git fsck --full\n>> - git repack -Adl -b --pack-kept-objects\n>> - git pack-refs --all\n>> - git prune\n>>\n>> The reason it's split into repack + prune instead of just gc is because\n>> we use alternates to save on disk space and try not to prune repos that\n>> are used as alternates by other repos in order to avoid potential\n>> corruption.\n\nIsn't this what extensions.preciousObjects is for? It looks like prune\njust refuses to run in precious objects mode though, and repack is\nskipped by gc, but if that repack command works, maybe we should do\nsomething like that in git-gc?\n\nBTW Jeff, I think we need more documentation for\nextensions.preciousObjects. It's only documented in technical/ which\nis practically invisible to all users. Maybe\ninclude::repository-version.txt in config.txt, or somewhere close to\nalternates?\n\n> [2] It's unclear to me if you're passing any options to git-prune, but\n> you may want to pass \"--expire\" with a short grace period. Without\n> any options it prunes every unreachable thing, which can lead to\n> races if the repository is actively being used.\n>\n> At GitHub we actually have a patch to `repack` that keeps all\n> objects, reachable or not, in the pack, and use it for all of our\n> automated maintenance. Since we don't drop objects at all, we can't\n> ever have such a race. Aside from some pathological cases, it wastes\n> much less space than you'd expect. We turn the flag off for special\n> cases (e.g., somebody has rewound history and wants to expunge a\n> sensitive object).\n>\n> I'm happy to share the \"keep everything\" patch if you're interested.\n\nAh ok, I guess this is why we just skip repack. I guess '-Adl -b\n--pack-kept-objects' is not enough then.\n-- \nDuy\n"},{"id":"289069","messageId":"C04883EB-2170-47C3-94E7-AE13516FD0C0@codeaurora.org","threadId":"42586","inReplyTo":"20160612221309.GC5428@sigill.intra.peff.net","subject":"Re: Repacking a repository uses up all available disk space","fromName":"Nasser Grainawi","fromEmail":"nasser@codeaurora.org","sentAt":"2016-06-13T01:43:27Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"nasser@codeaurora.org","avatar":"https://avatars.githubusercontent.com/u/757421?v=4"},"body":"On Jun 12, 2016, at 4:13 PM, Jeff King wrote:\n> \n> At GitHub we actually have a patch to `repack` that keeps all\n> objects, reachable or not, in the pack, and use it for all of our\n> automated maintenance. Since we don't drop objects at all, we can't\n> ever have such a race. Aside from some pathological cases, it wastes\n> much less space than you'd expect. We turn the flag off for special\n> cases (e.g., somebody has rewound history and wants to expunge a\n> sensitive object).\n> \n> I'm happy to share the \"keep everything\" patch if you're interested.\n\nWe have the same kind of patch actually (for the same reason), but back on the shell implementation of repack. It'd be great if you could share your modern version.\n\nNasser\n\n-- \nQualcomm Innovation Center, Inc.\nThe Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, \na Linux Foundation Collaborative Project\n"},{"id":"289070","messageId":"20160613043313.GA29422@sigill.intra.peff.net","threadId":"42586","inReplyTo":"C04883EB-2170-47C3-94E7-AE13516FD0C0@codeaurora.org","subject":"[PATCH 0/3] repack --keep-unreachable","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-13T04:33:14Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jun 12, 2016 at 07:43:27PM -0600, Nasser Grainawi wrote:\n\n> On Jun 12, 2016, at 4:13 PM, Jeff King wrote:\n> > \n> > At GitHub we actually have a patch to `repack` that keeps all\n> > objects, reachable or not, in the pack, and use it for all of our\n> > automated maintenance. Since we don't drop objects at all, we can't\n> > ever have such a race. Aside from some pathological cases, it wastes\n> > much less space than you'd expect. We turn the flag off for special\n> > cases (e.g., somebody has rewound history and wants to expunge a\n> > sensitive object).\n> > \n> > I'm happy to share the \"keep everything\" patch if you're interested.\n> \n> We have the same kind of patch actually (for the same reason), but\n> back on the shell implementation of repack. It'd be great if you could\n> share your modern version.\n\nHere is a cleaned-up version of what we run at GitHub (so this is a\nconcept that has been exercised for a few years in production, but I had\nto forward port the patches a bit; I _probably_ didn't introduce any\nbugs. :) ).\n\nThe heavy lifting is done by the existing --keep-unreachable option to\npack-objects, which Junio added a long time ago[1] in support of a safer\n\"gc --auto\". But it doesn't look like we ever documented or exercised\nit, and \"gc --auto\" ended up using the loosen-unreachable strategy\ninstead. In fact, the rest of that series seems to have been dropped; I\ncouldn't find any discussion on the list explaining it, or why this one\npatch was kept (so I don't think anybody upstream has ever used this\ncode, but as I said, we have been doing so for a few years, so I feel\nconfident in it).\n\n [1/3]: repack: document --unpack-unreachable option\n [2/3]: repack: add --keep-unreachable option\n [3/3]: repack: extend --keep-unreachable to loose objects\n\n-Peff\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/58413\n"},{"id":"289071","messageId":"20160613043354.GA3902@sigill.intra.peff.net","threadId":"42586","inReplyTo":"20160613043313.GA29422@sigill.intra.peff.net","subject":"[PATCH 1/3] repack: document --unpack-unreachable option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-13T04:33:54Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This was added back in 7e52f56 (gc: do not explode objects\nwhich will be immediately pruned, 2012-04-07), but not\ndocumented at the time, since it was an internal detail\nbetween git-gc and git-repack. However, as people with\ncomplicated setups may want to effectively reimplement the\nsteps of git-gc themselves, it is nice for us to document\nthese interfaces.\n\nSigned-off-by: Jeff King \n---\n Documentation/git-repack.txt | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex b9c02ce..cde7b44 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -128,6 +128,12 @@ other objects in that pack they already have locally.\n \twith `-b` or `repack.writeBitmaps`, as it ensures that the\n \tbitmapped packfile has the necessary objects.\n \n+--unpack-unreachable=::\n+\tWhen loosening unreachable objects, do not bother loosening any\n+\tobjects older than ``. This can be used to optimize out\n+\tthe write of any objects that would be immediately pruned by\n+\ta follow-up `git prune`.\n+\n Configuration\n -------------\n \n-- \n2.9.0.rc2.149.gd580ccd\n"},{"id":"289072","messageId":"20160613043628.GB3902@sigill.intra.peff.net","threadId":"42586","inReplyTo":"20160613043313.GA29422@sigill.intra.peff.net","subject":"[PATCH 2/3] repack: add --keep-unreachable option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-13T04:36:28Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The usual way to do a full repack (and what is done by\ngit-gc) is to run \"repack -Ad --unpack-unreachable=\",\nwhich will loosen any unreachable objects newer than\n\"\", and drop any older ones.\n\nThis is a safer alternative to \"repack -ad\", because\n\"\" becomes a grace period during which we will not\ndrop any new objects that are about to be referenced.\nHowever, it isn't perfectly safe. It's always possible that\na process is about to reference an old object. Even if that\nprocess were to take care to update the timestamp on the\nobject, there is no atomicity with a simultaneously running\n\"repack\" process.\n\nSo while unlikely, there is a small race wherein we may drop\nan object that is in the process of being referenced. If you\ndo automated repacking on a large number of active\nrepositories, you may hit it eventually, and the result is a\ncorrupted repository.\n\nIt would be nice to fix that race in the long run, but it's\ncomplicated. In the meantime, there is a much simpler\nstrategy for automated repository maintenance: do not drop\nobjects at all. We already have a \"--keep-unreachable\"\noption in pack-objects; we just need to plumb it through\nfrom git-repack.\n\nNote that this _isn't_ plumbed through from git-gc, so at\nthis point it's strictly a tool for people doing their own\nadvanced repository maintenance strategy.\n\nSigned-off-by: Jeff King \n---\n Documentation/git-repack.txt | 6 ++++++\n builtin/repack.c | 9 +++++++++\n t/t7701-repack-unpack-unreachable.sh | 15 +++++++++++++++\n 3 files changed, 30 insertions(+)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex cde7b44..68702ea 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -134,6 +134,12 @@ other objects in that pack they already have locally.\n \tthe write of any objects that would be immediately pruned by\n \ta follow-up `git prune`.\n \n+-k::\n+--keep-unreachable::\n+\tWhen used with `-ad`, any unreachable objects from existing\n+\tpacks will be appended to the end of the packfile instead of\n+\tbeing removed.\n+\n Configuration\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 858db38..573e66c 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -146,6 +146,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint pack_everything = 0;\n \tint delete_redundant = 0;\n \tconst char *unpack_unreachable = NULL;\n+\tint keep_unreachable = 0;\n \tconst char *window = NULL, *window_memory = NULL;\n \tconst char *depth = NULL;\n \tconst char *max_pack_size = NULL;\n@@ -175,6 +176,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"write bitmap index\")),\n \t\tOPT_STRING(0, \"unpack-unreachable\", &unpack_unreachable, N_(\"approxidate\"),\n \t\t\t\tN_(\"with -A, do not loosen objects older than this\")),\n+\t\tOPT_BOOL('k', \"keep-unreachable\", &keep_unreachable,\n+\t\t\t\tN_(\"with -a, repack unreachable objects\")),\n \t\tOPT_STRING(0, \"window\", &window, N_(\"n\"),\n \t\t\t\tN_(\"size of the window used for delta compression\")),\n \t\tOPT_STRING(0, \"window-memory\", &window_memory, N_(\"bytes\"),\n@@ -196,6 +199,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (delete_redundant && repository_format_precious_objects)\n \t\tdie(_(\"cannot delete packs in a precious-objects repo\"));\n \n+\tif (keep_unreachable &&\n+\t (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE)))\n+\t\tdie(_(\"--keep-unreachable and -A are incompatible\"));\n+\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps;\n \n@@ -239,6 +246,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t} else if (pack_everything & LOOSEN_UNREACHABLE) {\n \t\t\t\targv_array_push(&cmd.args,\n \t\t\t\t\t\t\"--unpack-unreachable\");\n+\t\t\t} else if (keep_unreachable) {\n+\t\t\t\targv_array_push(&cmd.args, \"--keep-unreachable\");\n \t\t\t} else {\n \t\t\t\targv_array_push(&cmd.env_array, \"GIT_REF_PARANOIA=1\");\n \t\t\t}\ndiff --git a/t/t7701-repack-unpack-unreachable.sh b/t/t7701-repack-unpack-unreachable.sh\nindex b66e383..f13df43 100755\n--- a/t/t7701-repack-unpack-unreachable.sh\n+++ b/t/t7701-repack-unpack-unreachable.sh\n@@ -122,4 +122,19 @@ test_expect_success 'keep packed objects found only in index' '\n \tgit cat-file blob :file\n '\n \n+test_expect_success 'repack -k keeps unreachable packed objects' '\n+\t# create packed-but-unreachable object\n+\tsha1=$(echo unreachable-packed | git hash-object -w --stdin) &&\n+\tpack=$(echo $sha1 | git pack-objects .git/objects/pack/pack) &&\n+\tgit prune-packed &&\n+\n+\t# -k should keep it\n+\tgit repack -adk &&\n+\tgit cat-file -p $sha1 &&\n+\n+\t# and double check that without -k it would have been removed\n+\tgit repack -ad &&\n+\ttest_must_fail git cat-file -p $sha1\n+'\n+\n test_done\n-- \n2.9.0.rc2.149.gd580ccd\n"},{"id":"289073","messageId":"20160613043804.GC3902@sigill.intra.peff.net","threadId":"42586","inReplyTo":"20160613043313.GA29422@sigill.intra.peff.net","subject":"[PATCH 3/3] repack: extend --keep-unreachable to loose objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-13T04:38:04Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"If you use \"repack -adk\" currently, we will pack all objects\nthat are already packed into the new pack, and then drop the\nold packs. However, loose unreachable objects will be left\nas-is. In theory these are meant to expire eventually with\n\"git prune\". But if you are using \"repack -k\", you probably\nwant to keep things forever and therefore do not run \"git\nprune\" at all. Meaning those loose objects may build up over\ntime and end up fooling any object-count heuristics (such as\nthe one done by \"gc --auto\", though since git-gc does not\nsupport \"repack -k\", this really applies to whatever custom\nscripts people might have driving \"repack -k\").\n\nWith this patch, we instead stuff any loose unreachable\nobjects into the pack along with the already-packed\nunreachable objects. This may seem wasteful, but it is\nreally no more so than using \"repack -k\" in the first place.\nWe are at a slight disadvantage, in that we have no useful\nordering for the result, or names to hand to the delta code.\nHowever, this is again no worse than what \"repack -k\" is\nalready doing for the packed objects. The packing of these\nobjects doesn't matter much because they should not be\naccessed frequently (unless they actually _do_ become\nreferenced, but then they would get moved to a different\npart of the packfile during the next repack).\n\nSigned-off-by: Jeff King \n---\n Documentation/git-repack.txt | 3 ++-\n builtin/pack-objects.c | 31 +++++++++++++++++++++++++++++++\n builtin/repack.c | 1 +\n t/t7701-repack-unpack-unreachable.sh | 13 +++++++++++++\n 4 files changed, 47 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 68702ea..b58b6b5 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -138,7 +138,8 @@ other objects in that pack they already have locally.\n --keep-unreachable::\n \tWhen used with `-ad`, any unreachable objects from existing\n \tpacks will be appended to the end of the packfile instead of\n-\tbeing removed.\n+\tbeing removed. In addition, any unreachable loose objects will\n+\tbe packed (and their loose counterparts removed).\n \n Configuration\n -------------\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8f5e358..a2f8cfd 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -44,6 +44,7 @@ static int non_empty;\n static int reuse_delta = 1, reuse_object = 1;\n static int keep_unreachable, unpack_unreachable, include_tag;\n static unsigned long unpack_unreachable_expiration;\n+static int pack_loose_unreachable;\n static int local;\n static int incremental;\n static int ignore_packed_keep;\n@@ -2378,6 +2379,32 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \tfree(in_pack.array);\n }\n \n+static int add_loose_object(const unsigned char *sha1, const char *path,\n+\t\t\t void *data)\n+{\n+\tenum object_type type = sha1_object_info(sha1, NULL);\n+\n+\tif (type < 0) {\n+\t\twarning(\"loose object at %s could not be examined\", path);\n+\t\treturn 0;\n+\t}\n+\n+\tadd_object_entry(sha1, type, \"\", 0);\n+\treturn 0;\n+}\n+\n+/*\n+ * We actually don't even have to worry about reachability here.\n+ * add_object_entry will weed out duplicates, so we just add every\n+ * loose object we find.\n+ */\n+static void add_unreachable_loose_objects(void)\n+{\n+\tfor_each_loose_file_in_objdir(get_object_directory(),\n+\t\t\t\t add_loose_object,\n+\t\t\t\t NULL, NULL, NULL);\n+}\n+\n static int has_sha1_pack_kept_or_nonlocal(const unsigned char *sha1)\n {\n \tstatic struct packed_git *last_found = (void *)1;\n@@ -2547,6 +2574,8 @@ static void get_object_list(int ac, const char **av)\n \n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs(&revs);\n+\tif (pack_loose_unreachable)\n+\t\tadd_unreachable_loose_objects();\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects(&revs);\n \n@@ -2647,6 +2676,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"include tag objects that refer to objects to be packed\")),\n \t\tOPT_BOOL(0, \"keep-unreachable\", &keep_unreachable,\n \t\t\t N_(\"keep unreachable objects\")),\n+\t\tOPT_BOOL(0, \"pack-loose-unreachable\", &pack_loose_unreachable,\n+\t\t\t N_(\"pack loose unreachable objects\")),\n \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n \t\t N_(\"unpack unreachable objects newer than