{"thread":{"id":"55702","subject":"git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","startedAt":"2021-05-16T13:10:03Z","lastAt":"2021-05-22T12:11:34Z","messageCount":6,"participants":["Bagas Sanjaya","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"424683","messageId":"c98e51b0-d89b-9170-e3bc-69ac9ec5e114@gmail.com","threadId":"55702","inReplyTo":null,"subject":"git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-05-16T13:09:56Z","receivedAt":"2021-05-16T13:10:03Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"Hi,\n\nI have a shallow clone of linux-stable repo [1] on my computer. Now\nI'm trying to repack with `git repack -A -d`.\n\nBefore repacking, here is the object counts on my clone\n(`git count-objects -v`):\n\n> count: 0\n> size: 0\n> in-pack: 3162206\n> packs: 17\n> size-pack: 3120393\n> prune-packable: 0\n> garbage: 0\n> size-garbage: 0\n\nAnd I have 41496 commits (only on master).\n\nAnd here are relevant config used:\n\n> pack.deltacachesize=120M\n> pack.windowmemory=400M\n> pack.packsizelimit=650M\n> pack.autopacklimit=0\n\nWhen I trigger repack operation, I expected that all objects on 17 packs\nare consolidated into several 650M-sized packs. However, in my case, repacking\nwas hang at \"Enumerating objects\" stage, that is I stuck at:\n\n\"Eumerating objects: 902036\"\n\nI also try to reproduce the issue with other repo, such as with local copy\nof GCC repo on my computer [2] using similar config, and the repack succeed\nwithout any errors.\n\nAm I missing something?\n\n[1]: https://github.com/gregkh/linux\n[2]: https://github.com/gcc-mirror/gcc\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"424716","messageId":"YKIulliGp8hyf5Y6@coredump.intra.peff.net","threadId":"55702","inReplyTo":"c98e51b0-d89b-9170-e3bc-69ac9ec5e114@gmail.com","subject":"Re: git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-17T08:51:34Z","receivedAt":"2021-05-17T08:51:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 16, 2021 at 08:09:56PM +0700, Bagas Sanjaya wrote:\n\n> I have a shallow clone of linux-stable repo [1] on my computer. Now\n> I'm trying to repack with `git repack -A -d`.\n> \n> Before repacking, here is the object counts on my clone\n> (`git count-objects -v`):\n> \n> > count: 0\n> > size: 0\n> > in-pack: 3162206\n> > packs: 17\n> > size-pack: 3120393\n> > prune-packable: 0\n> > garbage: 0\n> > size-garbage: 0\n> \n> And I have 41496 commits (only on master).\n> \n> And here are relevant config used:\n> \n> > pack.deltacachesize=120M\n> > pack.windowmemory=400M\n> > pack.packsizelimit=650M\n> > pack.autopacklimit=0\n> \n> When I trigger repack operation, I expected that all objects on 17 packs\n> are consolidated into several 650M-sized packs. However, in my case, repacking\n> was hang at \"Enumerating objects\" stage, that is I stuck at:\n> \n> \"Eumerating objects: 902036\"\n\nYou could try using strace or gdb to see what it's doing.\n\nBut as a guess, one thing that sometimes causes a stall near the end of\n\"enumerating objects\" is loosening unreachable objects that are\ncurrently packed. You told repack to use \"-A\", which asks to loosen\nthose objects so they aren't lost when the old packs are deleted (as\nopposed to \"-a\").\n\nYou'd probably want to at least say \"--unpack-unreachable=some.time\" to\navoid writing out ones which are not even recent (and which is what \"git\ngc\" will do under the hood).\n\nBut if you don't care about keeping them at all (e.g., because this is\nnot an active repository where other simultaneous operations may be\ntaking place, so you know it is safe to delete even recent ones), then\njust \"git repack -a -d\" is probably your best bet.\n\n-Peff\n"},{"id":"424854","messageId":"158a2f8c-95c0-075a-4cda-6328324c2261@gmail.com","threadId":"55702","inReplyTo":"YKIulliGp8hyf5Y6@coredump.intra.peff.net","subject":"Re: git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-05-18T11:23:40Z","receivedAt":"2021-05-18T11:23:55Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"Hi Jeff,\n\n> You could try using strace or gdb to see what it's doing.\n> \n> But as a guess, one thing that sometimes causes a stall near the end of\n> \"enumerating objects\" is loosening unreachable objects that are\n> currently packed. You told repack to use \"-A\", which asks to loosen\n> those objects so they aren't lost when the old packs are deleted (as\n> opposed to \"-a\").\n> \n\nI attached two strace logs, one for \"git repack -A -d\" and one for \"git\nrepack -a -d\".\n\nFor the former, I got following excerpt before I had to SIGINT the process:\n\n> stat(\"/opt/git/libexec/git-core/git\", {st_mode=S_IFREG|0755, st_size=22096480, ...}) = 0\n> pipe([5, 7])                            = 0\n> openat(AT_FDCWD, \"/dev/null\", O_RDWR|O_CLOEXEC) = 8\n> fcntl(8, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)\n> fcntl(8, F_SETFD, FD_CLOEXEC)           = 0\n> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0\n> clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7feb5ecbfa10) = 13691\n> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0\n> close(7)                                = 0\n> read(5, \"\", 8)                          = 0\n> close(5)                                = 0\n> close(8)                                = 0\n> close(4)                                = 0\n> fcntl(3, F_GETFL)                       = 0 (flags O_RDONLY)\n> fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0\n> read(3, 0x55a540de5250, 4096)           = ? ERESTARTSYS (To be restarted if SA_RESTART is set)\n\nI thought that in the case of \"git repack -A -d\", it is stucked at the\nlast read() before I ctrl-c'ed to trigger SIGINT.\n\n> You'd probably want to at least say \"--unpack-unreachable=some.time\" to\n> avoid writing out ones which are not even recent (and which is what \"git\n> gc\" will do under the hood).\n>> But if you don't care about keeping them at all (e.g., because this is\n> not an active repository where other simultaneous operations may be\n> taking place, so you know it is safe to delete even recent ones), then\n> just \"git repack -a -d\" is probably your best bet.\n> \n> -Peff\n> \nUsing \"-a\" instead of \"-A\" on git repack works. Thanks.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"424857","messageId":"YKOuHLWmJ3jktpyA@coredump.intra.peff.net","threadId":"55702","inReplyTo":"158a2f8c-95c0-075a-4cda-6328324c2261@gmail.com","subject":"Re: git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-18T12:07:56Z","receivedAt":"2021-05-18T12:08:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 18, 2021 at 06:23:40PM +0700, Bagas Sanjaya wrote:\n\n> > You could try using strace or gdb to see what it's doing.\n> > \n> > But as a guess, one thing that sometimes causes a stall near the end of\n> > \"enumerating objects\" is loosening unreachable objects that are\n> > currently packed. You told repack to use \"-A\", which asks to loosen\n> > those objects so they aren't lost when the old packs are deleted (as\n> > opposed to \"-a\").\n> > \n> \n> I attached two strace logs, one for \"git repack -A -d\" and one for \"git\n> repack -a -d\".\n> \n> For the former, I got following excerpt before I had to SIGINT the process:\n> \n> > stat(\"/opt/git/libexec/git-core/git\", {st_mode=S_IFREG|0755, st_size=22096480, ...}) = 0\n> > pipe([5, 7])                            = 0\n> > openat(AT_FDCWD, \"/dev/null\", O_RDWR|O_CLOEXEC) = 8\n> > fcntl(8, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)\n> > fcntl(8, F_SETFD, FD_CLOEXEC)           = 0\n> > rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0\n> > clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7feb5ecbfa10) = 13691\n> > rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0\n> > close(7)                                = 0\n> > read(5, \"\", 8)                          = 0\n> > close(5)                                = 0\n> > close(8)                                = 0\n> > close(4)                                = 0\n> > fcntl(3, F_GETFL)                       = 0 (flags O_RDONLY)\n> > fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0\n> > read(3, 0x55a540de5250, 4096)           = ? ERESTARTSYS (To be restarted if SA_RESTART is set)\n> \n> I thought that in the case of \"git repack -A -d\", it is stucked at the\n> last read() before I ctrl-c'ed to trigger SIGINT.\n\nYou need \"strace -f\". The parent repack process spawns pack-objects (via\nthe clone() call above), and then waits for it to print the name of the\ngenerated pack at the end. So it will stall on that read() for quite a\nwhile, even under normal circumstances.\n\n-Peff\n"},{"id":"425297","messageId":"168567f7-cec1-a4f1-5b2e-b51637ab7668@gmail.com","threadId":"55702","inReplyTo":"YKOuHLWmJ3jktpyA@coredump.intra.peff.net","subject":"Re: git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-05-22T11:16:15Z","receivedAt":"2021-05-22T11:16:23Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"Hi Jeff, sorry for long reply.\n\nOn 18/05/21 19.07, Jeff King wrote:\n> You need \"strace -f\". The parent repack process spawns pack-objects (via\n> the clone() call above), and then waits for it to print the name of the\n> generated pack at the end. So it will stall on that read() for quite a\n> while, even under normal circumstances.\n> \n> -Peff\n> \n\nNow on my clone, git count-objects -v says:\n\n> count: 0\n> size: 0\n> in-pack: 3469192\n> packs: 7\n> size-pack: 2181233\n> prune-packable: 0\n> garbage: 0\n> size-garbage: 0\n\nAnd I have 60211 commits, with the oldest tag available is v3.15-rc3.\n\nHere is excerpt from strace -f just before I triggered SIGINT due to hang:\n\n> 22903 openat(AT_FDCWD, \".git/objects/3b/tmp_obj_24pz93\", O_RDWR|O_CREAT|O_EXCL, 0444) = 3\n> 22903 write(3, \"x\\1+)JMU0421d040031Q\\320K\\316I\\314K\\327M\\313/\\312M,\"..., 1113) = 1113\n> 22903 close(3)                          = 0\n> 22903 utime(\".git/objects/3b/tmp_obj_24pz93\", {actime=1621599665 /* 2021-05-21T19:21:05+0700 */, modtime=1621599665 /* 2021-05-21T19:21:05+0700 */}) = 0\n> 22903 link(\".git/objects/3b/tmp_obj_24pz93\", \".git/objects/3b/816f00a02062692e95a9a756247fca34abb911\") = 0\n> 22903 unlink(\".git/objects/3b/tmp_obj_24pz93\") = 0\n> 22903 access(\".git/objects/3b/819396230eda4ce9be9fbb2c91c13ebb28e8d3\", F_OK) = -1 ENOENT (No such file or directory)\n> 22903 getpid()                          = 22903\n> 22903 openat(AT_FDCWD, \".git/objects/3b/tmp_obj_yrHAe4\", O_RDWR|O_CREAT|O_EXCL, 0444) = 3\n> 22903 write(3, \"x\\1\\224}y<\\324\\337\\367\\377(\\205\\262E\\226\\262D\\205T\\0233\\214=\\n!\\25\\205R\\22\\306\\314`\"..., 4096) = 4096\n> 22903 write(3, \"\\365\\16\\236\\215!&\\37 \\v\\300\\240\\334\\33r\\266/_\\317Hg,\\333\\356w\\366\\235\\307\\245<E\\347\\t\"..., 4096) = 4096\n> 22903 write(3, \"\\362\\367\\267\\224Pc\\207\\252!\\240$\\\"\\26\\212\\204\\22\\2315\\245\\255\\357j\\373f\\316G9\\352[\\r!\\326\"..., 4096) = 4096\n> 22903 write(3, \"\\305\\vC3\\253\\217\\343\\17kl>v\\200_\\226Y\\206\\247\\21@\\327mi\\310\\177{7{i\\214w\\243\"..., 4096) = 4096\n> 22903 write(3, \"@X\\17\\213\\304\\307z\\232v\\355\\313y\\316,\\342\\202p\\\"\\343|x\\306\\306<\\335DM\\350\\315!m\\207\"..., 4096) = 4096\n> 22903 write(3, \"'\\253R\\33\\267\\310p\\3\\332\\201\\333\\316\\254'\\3114\\347\\277\\0321\\35\\223\\314i\\255v\\252X{[r\\33\"..., 4096) = 4096\n> 22903 write(3, \"\\261\\341a\\252\\355\\360\\356\\340\\375A$\\214\\377\\336O\\311A\\215\\327\\5g2\\\\\\324\\332\\276\\363\\323\\306S\\215\\342\"..., 4096) = 4096\n> 22903 write(3, \"s_n83S\\326\\224]H\\271\\246-\\267\\35\\32\\234\\6s!\\273\\177\\30\\256k\\\\eW\\355E\\264^\"..., 4096) = 4096\n> 22903 write(3, \"E\\0266\\37\\30244Mtk9\\310\\305\\227\\370\\245\\355qwUi\\363\\237\\302\\347S\\4Xp\\342\\330\\370\"..., 4096) = 4096\n> 22903 write(3, \"v\\267\\303B\\177\\232v{\\342\\353\\1\\340A\\205[\\336\\266\\250_e\\346\\347e)\\212^\\241]\\263\\233\\270\\323\"..., 4096) = 4096\n> 22903 write(3, \"\\325dA\\266\\333\\221\\274\\32\\367\\277\\271nh\\327\\276\\272C\\323-E\\242\\351\\232\\352\\303]|\\335\\227\\227\\200\\301\"..., 4096 <unfinished ...>\n> 22902 <... read resumed>0x5594d878b7b0, 4096) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)\n> 22903 <... write resumed>)              = 4096\n\nThe last write() sequence seems running repeatedly, what it means? Infinite loop?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"425301","messageId":"YKj085Nn/uQYomZG@coredump.intra.peff.net","threadId":"55702","inReplyTo":"168567f7-cec1-a4f1-5b2e-b51637ab7668@gmail.com","subject":"Re: git repack on shallow clone of large repo (linux kernel) hangs at \"Enumerating objects\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-22T12:11:31Z","receivedAt":"2021-05-22T12:11:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 22, 2021 at 06:16:15PM +0700, Bagas Sanjaya wrote:\n\n> Here is excerpt from strace -f just before I triggered SIGINT due to hang:\n> \n> > 22903 openat(AT_FDCWD, \".git/objects/3b/tmp_obj_24pz93\", O_RDWR|O_CREAT|O_EXCL, 0444) = 3\n> > 22903 write(3, \"x\\1+)JMU0421d040031Q\\320K\\316I\\314K\\327M\\313/\\312M,\"..., 1113) = 1113\n> > 22903 close(3)                          = 0\n> > 22903 utime(\".git/objects/3b/tmp_obj_24pz93\", {actime=1621599665 /* 2021-05-21T19:21:05+0700 */, modtime=1621599665 /* 2021-05-21T19:21:05+0700 */}) = 0\n> > 22903 link(\".git/objects/3b/tmp_obj_24pz93\", \".git/objects/3b/816f00a02062692e95a9a756247fca34abb911\") = 0\n> > 22903 unlink(\".git/objects/3b/tmp_obj_24pz93\") = 0\n\nOK, so this is writing out loose objects. It's exactly the case I\nsuspected.\n\n> The last write() sequence seems running repeatedly, what it means? Infinite loop?\n\nIf it's a lot of just write(), it's probably just a large object. If\nit's a lot of writes interspersed with opens and links, as above, then\nit's just a lot of objects. It probably would finish eventually, but\nit's just slow.\n\n-Peff\n"}]}