{"thread":{"id":"22444","subject":"unable to run gc (or git repack -Adl )","startedAt":"2010-01-29T22:29:01Z","lastAt":"2010-01-30T02:45:55Z","messageCount":3,"participants":["Jon Nelson","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"133027","messageId":"cccedfc61001291429q7dfa2045x18134439ad1e7c94@mail.gmail.com","threadId":"22444","inReplyTo":null,"subject":"unable to run gc (or git repack -Adl )","fromName":"Jon Nelson","fromEmail":"jnelson@jamponi.net","sentAt":"2010-01-29T22:29:01Z","receivedAt":"2010-01-29T22:29:01Z","isPatch":false,"sender":{"key":"jnelson@jamponi.net","avatar":null},"body":"Using 1.6.4.2 on openSUSE 11.2 (x86_64).\n\nI have a beefy repo (du of 14GB) that I can't seem to run 'gc' on.\n\nAfter running for over 2 hours, this is what I get:\n\nCounting objects: 267676, done.\nCompressing objects: 100% (217424/217424), done.\nfatal: Unable to create temporary file: Too many open files\nerror: failed to run repack\n\nUgh!\n\nI have 3 GB of memory (and 1GB of swap).\nWhen I strace the various processes, I see some things I don't understand:\n\n1. I see the 'git-repack' shell process scanning for .keep files. I\ndon't have any. Is there a shortcut to this?\n\nIt's also hugely inefficient. In this case, the code to identify non\n.keep packs takes *4 minutes, 45 seconds*, lots of disk I/O, and lots\nof CPU (it pegs one CPU at 100% for the entire duration). With a wee\nbit of awk, I have reduced that to 2.3 seconds with VASTLY reduced I/O\nand CPU requirements. Patch attached.\n\n2. When git-pack-objects is being run, around the time it's 85% done\n\"compressing\" it's very very very slow. Like, 2-5 objects every\nsecond. The largest object in the repo is about 1MB.\n\n3. When git pack objects is running and counting up the number of\nobjects, it is stat'ing files that aren't in the working directly, and\nshould not be, according to the index. If I switch the repo to be a\n\"bare\" repository, then it doesn't do that, however, why is it doing\nthat in the first place?\n\n4. Should git-pack-objects be reading the pack.idx files for counting\nobjects instead of the .pack files themselves?\n\n5. There is no 5\n\n6. Should git-pack-objects be closing .pack files after opening them?\nI have 6559 .pack files.\n\n7. Ultimately, how do I get \"git gc\" to work on this repo?\n\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 1eb3bca..4358f96 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -62,15 +62,7 @@ case \",$all_into_one,\" in\n ,t,)\n        args= existing=\n        if [ -d \"$PACKDIR\" ]; then\n-               for e in `cd \"$PACKDIR\" && find . -type f -name '*.pack' \\\n-                       | sed -e 's/^\\.\\///' -e 's/\\.pack$//'`\n-               do\n-                       if [ -e \"$PACKDIR/$e.keep\" ]; then\n-                               : keep\n-                       else\n-                               existing=\"$existing $e\"\n-                       fi\n-               done\n+        existing=$( find . -type f -name '*.pack' -o -name\n'*.pack.keep' | sed -e 's/^\\.\\///' | sort | awk '{ if ($0 ~ /\\.keep$/)\n{ N=substr($0, 0, length($0)-5); K[N]=0; } else { K[$0]=1; } } END {\nfor (k in K) { if (K[k] == 1) { printf \"%s \", k; } } } ' )\n                if test -n \"$existing\" -a -n \"$unpack_unreachable\" -a \\\n                        -n \"$remove_redundant\"\n                then\n\n\n\n-- \nJon\n"},{"id":"133064","messageId":"alpine.LFD.2.00.1001292025260.1681@xanadu.home","threadId":"22444","inReplyTo":"cccedfc61001291429q7dfa2045x18134439ad1e7c94@mail.gmail.com","subject":"Re: unable to run gc (or git repack -Adl )","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-01-30T02:14:47Z","receivedAt":"2010-01-30T02:14:47Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 29 Jan 2010, Jon Nelson wrote:\n\n> Using 1.6.4.2 on openSUSE 11.2 (x86_64).\n> \n> I have a beefy repo (du of 14GB) that I can't seem to run 'gc' on.\n> \n> After running for over 2 hours, this is what I get:\n> \n> Counting objects: 267676, done.\n> Compressing objects: 100% (217424/217424), done.\n> fatal: Unable to create temporary file: Too many open files\n> error: failed to run repack\n\nOuch!!  Impressive.\n\n> Ugh!\n\nIndeed.\n\n> I have 3 GB of memory (and 1GB of swap).\n> When I strace the various processes, I see some things I don't understand:\n> \n> 1. I see the 'git-repack' shell process scanning for .keep files. I\n> don't have any. Is there a shortcut to this?\n> \n> It's also hugely inefficient. In this case, the code to identify non\n> .keep packs takes *4 minutes, 45 seconds*, lots of disk I/O, and lots\n> of CPU (it pegs one CPU at 100% for the entire duration). With a wee\n> bit of awk, I have reduced that to 2.3 seconds with VASTLY reduced I/O\n> and CPU requirements. Patch attached.\n\nYour patch will pick any .pack file in the repo not only from the \n.git/objects/pack directory.  There is no such thing as *.pack.keep \neither.\n\n> 2. When git-pack-objects is being run, around the time it's 85% done\n> \"compressing\" it's very very very slow. Like, 2-5 objects every\n> second. The largest object in the repo is about 1MB.\n\nYou probably consumed all RAM and started swapping at that point.\n\nOr... you have many of those 1MB objects.  If so try \nusing --window-memory=8M or similar.\n\n> 3. When git pack objects is running and counting up the number of\n> objects, it is stat'ing files that aren't in the working directly, and\n> should not be, according to the index. If I switch the repo to be a\n> \"bare\" repository, then it doesn't do that, however, why is it doing\n> that in the first place?\n\nA bare repository has no index.  When the index is present though, it is \nnecessary to also pack objects it references.  Why working directory \nfiles would be stat()'d in that case I don't know.\n\n> 4. Should git-pack-objects be reading the pack.idx files for counting\n> objects instead of the .pack files themselves?\n\nNo.  The whole point when \"counting objects\" is to perform a walk of the \nhistory graph and capture the set of objects that are actually \nreferenced from your branches/tags and leave the unreferenced objects \nbehind.  Also the order in which those objects are encountered during \nthat history walk is very important for efficient object placement in \nthe final pack.  So this is much more involved than only listing the \nobjects contained in every packs.\n\n> 5. There is no 5\n\nI'm a flying buldozer.\n\n> 6. Should git-pack-objects be closing .pack files after opening them?\n> I have 6559 .pack files.\n\nNo wonder why you exhausted your file handles.  And your repository must \nbe _horribly_ slow to work with, which might explain the \nslowness/swappiness.\n\n> 7. Ultimately, how do I get \"git gc\" to work on this repo?\n\n... because you really really want to repack this mess ASAP of course.\n\nHaving so many packs means they must be relatively small.  Yet, Git \nallows up to 8GB of pack data to be mmap()'d at once on x86_64.  This \nmeans that an average of 3700 packs might be mapped at once, plus their \nrespective .idx files.\n\nYou could try:\n\n\tgit config core.packedGitLimit 256m\n\tgit config core.packedGitWindowSize 32m\n\tgit config pack.deltaCacheSize 1\n\nand try repacking again with 'git gc --prune=now'.  After the repack \nsucceeds, you should be able to remove the above configs from your \n.git/config file.\n\n\nNicolas\n"},{"id":"133068","messageId":"cccedfc61001291845n5703b606icb75bb2093ae51b2@mail.gmail.com","threadId":"22444","inReplyTo":"alpine.LFD.2.00.1001292025260.1681@xanadu.home","subject":"Re: unable to run gc (or git repack -Adl )","fromName":"Jon Nelson","fromEmail":"jnelson@jamponi.net","sentAt":"2010-01-30T02:45:55Z","receivedAt":"2010-01-30T02:45:55Z","isPatch":false,"sender":{"key":"jnelson@jamponi.net","avatar":null},"body":"On Fri, Jan 29, 2010 at 8:14 PM, Nicolas Pitre <nico@fluxnic.net> wrote:\n> On Fri, 29 Jan 2010, Jon Nelson wrote:\n...\n>> 1. I see the 'git-repack' shell process scanning for .keep files. I\n>> don't have any. Is there a shortcut to this?\n>>\n>> It's also hugely inefficient. In this case, the code to identify non\n>> .keep packs takes *4 minutes, 45 seconds*, lots of disk I/O, and lots\n>> of CPU (it pegs one CPU at 100% for the entire duration). With a wee\n>> bit of awk, I have reduced that to 2.3 seconds with VASTLY reduced I/O\n>> and CPU requirements. Patch attached.\n>\n> Your patch will pick any .pack file in the repo not only from the\n> .git/objects/pack directory.  There is no such thing as *.pack.keep\n> either.\n\nUgh. Yep. Patch amended. Still fast. Still wrong?\n\n>> 3. When git pack objects is running and counting up the number of\n>> objects, it is stat'ing files that aren't in the working directly, and\n>> should not be, according to the index. If I switch the repo to be a\n>> \"bare\" repository, then it doesn't do that, however, why is it doing\n>> that in the first place?\n>\n> A bare repository has no index.  When the index is present though, it is\n> necessary to also pack objects it references.  Why working directory\n> files would be stat()'d in that case I don't know.\n\nInquiring minds want to know.\n\n>> 4. Should git-pack-objects be reading the pack.idx files for counting\n>> objects instead of the .pack files themselves?\n>\n> No.  The whole point when \"counting objects\" is to perform a walk of the\n> history graph and capture the set of objects that are actually\n> referenced from your branches/tags and leave the unreferenced objects\n> behind.  Also the order in which those objects are encountered during\n> that history walk is very important for efficient object placement in\n> the final pack.  So this is much more involved than only listing the\n> objects contained in every packs.\n\nAh. For some reason I thought the .idx files contained not just a\nstraight listing but also the parent/child relationships as well.\n\n> You could try:\n>\n>        git config core.packedGitLimit 256m\n>        git config core.packedGitWindowSize 32m\n>        git config pack.deltaCacheSize 1\n>\n> and try repacking again with 'git gc --prune=now'.  After the repack\n> succeeds, you should be able to remove the above configs from your\n> .git/config file.\n\nI have since thrown out the repo and started over on this particular\nexperiment, issuing a 'git gc' rather more often. The config options\nabove are now dutifully scribbled down.  Thanks!\n\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 1eb3bca..3cef57d 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -62,15 +62,7 @@ case \",$all_into_one,\" in\n ,t,)\n        args= existing=\n        if [ -d \"$PACKDIR\" ]; then\n-               for e in `cd \"$PACKDIR\" && find . -type f -name '*.pack' \\\n-                       | sed -e 's/^\\.\\///' -e 's/\\.pack$//'`\n-               do\n-                       if [ -e \"$PACKDIR/$e.keep\" ]; then\n-                               : keep\n-                       else\n-                               existing=\"$existing $e\"\n-                       fi\n-               done\n+               existing=$( cd \"$PACKDIR\" && find . -type f -name\n'*.pack' -o -name '*.keep' | sed -e 's/^\\.\\///' | sort | awk '{ if ($0\n~ /\\.keep$/) { N=substr($0, 0, length($0)-4) \"pack\"; K[N]=0; } else {\nif ($0 in K) { } else { K[$0]=1; } } } END { for (k in K) { if (K[k]\n== 1) { printf \"%s \", k; } } } ' )\n                if test -n \"$existing\" -a -n \"$unpack_unreachable\" -a \\\n                        -n \"$remove_redundant\"\n                then\n\n\n-- \nJon\n"}]}