{"thread":{"id":"35903","subject":"git gc --aggressive led to about 40 times slower \"git log --raw\"","startedAt":"2014-02-18T07:25:10Z","lastAt":"2014-02-24T09:27:26Z","messageCount":31,"participants":["Christian Jaeger","David Kastrup","Duy Nguyen","Jonathan Nieder","Junio C Hamano","Philippe Vaucher","Andreas Schwab"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"234942","messageId":"CAEjYwfU==yYtQBDzZzEPdvbqz1N=gZtbMr5ccRaC_U7NfViQLA@mail.gmail.com","threadId":"35903","inReplyTo":null,"subject":"git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Christian Jaeger","fromEmail":"chrjae@gmail.com","sentAt":"2014-02-18T07:25:10Z","receivedAt":"2014-02-18T07:25:10Z","isPatch":false,"sender":{"key":"chrjae@gmail.com","avatar":null},"body":"Hi\n\nI've got a repository where \"git log --raw > _somefile\" took a few\nseconds in the past, but after an attempt at merging some commits that\nwere collected in a clone of the same repo that was created about a\nyear ago, I noticed that this command was now taking 3 minutes 7\nseconds. \"git gc\", \"git fsck\", \"git clone file:///the/repo/.git\" also\nnow each took between ~4-10 minutes, also \"git log --raw somefile\" got\nequally unusably slow. With the help of the people on the IRC, I\ntracked it down to my recent use of \"git gc --aggressive\" in this\nrepo. Running \"git repack -a -d -f\" solved it, now it's again taking\n4-5 seconds. After running \"git gc --aggressive\" again for\nconfirmation, \"git log --raw > _somefile\" was again slowed down,\nalthough now 'only' to 1 minute 34 seconds; did perhaps my \"git remote\nadd -f other-repo\", which I remember was also running rather slowly,\nexacerbate the problem (to the > 3 minutes I was seeing)?\n\nThe repo has about 6000 commits, about 12'000 files in the current\nHEAD, and about 43 MB packed .git contents. The files are (almost) all\nplain text, about half of them are about 42 bytes long, the rest up to\nabout 2 MB although most of them are just around 5-50 KB. Most files\nmostly grow at the end. The biggest files (500KB-2MB) are quite\nlong-lived and don't stop growing, again mostly at the end. Also,\nabout 2*5K files are each in the same directory, meaning that the tree\nobjects representing those 2 directories are big but changing only in\na few places.\n\nI've now learned to avoid \"git gc --aggressive\". Perhaps there are\nsome other conclusions to be drawn, I don't know.\n\nChristian.\n"},{"id":"234955","messageId":"87r470ssuc.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"CAEjYwfU==yYtQBDzZzEPdvbqz1N=gZtbMr5ccRaC_U7NfViQLA@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-18T08:55:23Z","receivedAt":"2014-02-18T08:55:23Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Christian Jaeger writes:\n\n> I've got a repository where \"git log --raw > _somefile\" took a few\n> seconds in the past, but after an attempt at merging some commits that\n> were collected in a clone of the same repo that was created about a\n> year ago, I noticed that this command was now taking 3 minutes 7\n> seconds. \"git gc\", \"git fsck\", \"git clone file:///the/repo/.git\" also\n> now each took between ~4-10 minutes, also \"git log --raw somefile\" got\n> equally unusably slow. With the help of the people on the IRC, I\n> tracked it down to my recent use of \"git gc --aggressive\" in this\n> repo. Running \"git repack -a -d -f\" solved it, now it's again taking\n> 4-5 seconds. After running \"git gc --aggressive\" again for\n> confirmation, \"git log --raw > _somefile\" was again slowed down,\n> although now 'only' to 1 minute 34 seconds;\n\n[...]\n\n> I've now learned to avoid \"git gc --aggressive\". Perhaps there are\n> some other conclusions to be drawn, I don't know.\n\nI've seen the same with my ongoing work on git-blame with the current\nEmacs Git mirror. Aggressive packing reduces the repository size to\nabout a quarter, but it blows up the system time (mainly I/O)\nsignificantly, quite reducing the total benefits of my algorithmic\nimprovements there.\n\nThere is also some quite visible additional time spent in zlib, so a\nwild guess would be that zlib is not really suited to the massive amount\nof directory entries of a Git object store. Since the system time still\ndominates, this guess would only make sense if Git over zlib kept\nrereading the directory section of whatever compressed file we are\ntalking about. But that's really a rather handwavy wild guess without\nanything better than a hunch to back it up. I don't even know what kind\nof compression and/or packs are used: I've only ever messed myself with\nthe delta coding of the normal \"unpacked\" operation (there are a few\nolder commits from me on that).\n\n-- \nDavid Kastrup\n"},{"id":"234957","messageId":"CACsJy8D9tws_gu6yWVdz3t+Vfg5-9iorptn4BLnTL3b+YWcHzQ@mail.gmail.com","threadId":"35903","inReplyTo":"87r470ssuc.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-18T09:45:25Z","receivedAt":"2014-02-18T09:45:25Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 18, 2014 at 3:55 PM, David Kastrup wrote:\n> Christian Jaeger writes:\n>\n>> I've got a repository where \"git log --raw > _somefile\" took a few\n>> seconds in the past, but after an attempt at merging some commits that\n>> were collected in a clone of the same repo that was created about a\n>> year ago, I noticed that this command was now taking 3 minutes 7\n>> seconds. \"git gc\", \"git fsck\", \"git clone file:///the/repo/.git\" also\n>> now each took between ~4-10 minutes, also \"git log --raw somefile\" got\n>> equally unusably slow. With the help of the people on the IRC, I\n>> tracked it down to my recent use of \"git gc --aggressive\" in this\n>> repo. Running \"git repack -a -d -f\" solved it, now it's again taking\n>> 4-5 seconds. After running \"git gc --aggressive\" again for\n>> confirmation, \"git log --raw > _somefile\" was again slowed down,\n>> although now 'only' to 1 minute 34 seconds;\n>\n> [...]\n>\n>> I've now learned to avoid \"git gc --aggressive\". Perhaps there are\n>> some other conclusions to be drawn, I don't know.\n>\n> I've seen the same with my ongoing work on git-blame with the current\n> Emacs Git mirror. Aggressive packing reduces the repository size to\n> about a quarter, but it blows up the system time (mainly I/O)\n> significantly, quite reducing the total benefits of my algorithmic\n> improvements there.\n\nLikely because --aggressive passes --depth=250 to pack-objects. Long\ndelta chains could reduce pack size and increase I/O as well as zlib\nprocessing signficantly. Christian can try \"git repack -adf\" which is\nreally close to --aggressive (except it uses default --depth=50) and\nsee if it makes any difference.\n\n> There is also some quite visible additional time spent in zlib, so a\n> wild guess would be that zlib is not really suited to the massive amount\n> of directory entries of a Git object store. Since the system time still\n> dominates, this guess would only make sense if Git over zlib kept\n> rereading the directory section of whatever compressed file we are\n> talking about. But that's really a rather handwavy wild guess without\n> anything better than a hunch to back it up. I don't even know what kind\n> of compression and/or packs are used: I've only ever messed myself with\n> the delta coding of the normal \"unpacked\" operation (there are a few\n> older commits from me on that).\n>\n> --\n> David Kastrup\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n\n\n\n-- \nDuy\n"},{"id":"234958","messageId":"87ioscsoow.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"CACsJy8D9tws_gu6yWVdz3t+Vfg5-9iorptn4BLnTL3b+YWcHzQ@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-18T10:25:03Z","receivedAt":"2014-02-18T10:25:03Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Duy Nguyen writes:\n\n> On Tue, Feb 18, 2014 at 3:55 PM, David Kastrup wrote:\n>\n>> I've seen the same with my ongoing work on git-blame with the current\n>> Emacs Git mirror. Aggressive packing reduces the repository size to\n>> about a quarter, but it blows up the system time (mainly I/O)\n>> significantly, quite reducing the total benefits of my algorithmic\n>> improvements there.\n>\n> Likely because --aggressive passes --depth=250 to pack-objects. Long\n> delta chains could reduce pack size and increase I/O as well as zlib\n> processing signficantly.\n\nIncreased zlib processing time is one thing, but if it _increases_ I/O,\nthen it would seem there is a serious impedance mismatch between the\ncompression scheme and the code relying on it, leading to repeated reads\nof blocks only needed for reconstructing dynamic compression\ndictionaries.\n\nCompression should reduce rather than increase the total amount of\nreads. So it would seem that either better caching and/or smaller\nindependent block sizes and/or strategies for sorting the delta chain to\nmake its resolution require mostly linear reads, and then make sure to\ndo this in a manner that does not reinitialize the decompression for\naccessing each delta that happens to be more or less \"in sequence\".\n\nOf course, this is assuming that the additional time is spent\nuncompressing data rather than navigating directories.\n\nIt's actually conceivable that there is quite a bit of potential to get\nbetter performance from unchanged readers by packing stuff in a\ndifferent order while still using the same delta chain depth.\n\n-- \nDavid Kastrup\n"},{"id":"234994","messageId":"20140218155842.GA7855@google.com","threadId":"35903","inReplyTo":"87ioscsoow.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-02-18T15:59:22Z","receivedAt":"2014-02-18T15:59:22Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"David Kastrup wrote:\n> Duy Nguyen writes:\n\n>> Likely because --aggressive passes --depth=250 to pack-objects. Long\n>> delta chains could reduce pack size and increase I/O as well as zlib\n>> processing signficantly.\n[...]\n> Compression should reduce rather than increase the total amount of\n> reads.\n\n--depth=250 means to allow chains of \"To get this object, first\ninflate this object, then apply this delta\" of length 250.\n\nThat's absurdly long, and doesn't even help compression much in\npractice (many short chains referring to the same objects tends to\nwork fine). We probably shouldn't make --aggressive do that.\nSomething like --depth=10 would make more sense.\n\nHoping that clarifies,\nJonathan\n"},{"id":"234995","messageId":"CAEjYwfUfOo1huAJE2oLrMorV0tQy0Y79y2WPGEK8R0T_wq4k9g@mail.gmail.com","threadId":"35903","inReplyTo":"CACsJy8D9tws_gu6yWVdz3t+Vfg5-9iorptn4BLnTL3b+YWcHzQ@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Christian Jaeger","fromEmail":"chrjae@gmail.com","sentAt":"2014-02-18T16:43:26Z","receivedAt":"2014-02-18T16:43:26Z","isPatch":false,"sender":{"key":"chrjae@gmail.com","avatar":null},"body":"2014-02-18 9:45 GMT+00:00 Duy Nguyen :\n> Christian can try \"git repack -adf\"\n\nThat's what I already mentioned in my first mail is what I used to fix\nthe problem.\n\nHere are some 'hard' numbers, FWIW:\n\n- both ~/scr and swap are on the same SSD;\n\n$ free\n total used free shared buffers cached\nMem: 3996748 3800828 195920 0 377176 1078848\n-/+ buffers/cache: 2344804 1651944\nSwap: 2097148 169760 1927388\n\ngit only used up to about 100 MB of VIRT or RSS when I checked, there\nwas an ulimit of \"-S -v 1200000\".\n\n- this is git version 1.7.10.4 (1:1.7.10.4-1+wheezy1 i386 Debian)\n\n- after my attempted merge (which had conflicts and I had then\ncancelled by way of git reset --hard), and then a \"git gc\", the times\nwere:\n\n~/scr$ time git log --raw > _THELOG\n\nreal 3m7.002s\nuser 2m0.252s\nsys 1m6.008s\n\n- on a copy:\n\n/dev/shm/scr$ time git repack -a -d -f\nCounting objects: 34917, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (27038/27038), done.\nWriting objects: 100% (34917/34917), done.\nTotal 34917 (delta 13928), reused 0 (delta 0)\n\nreal 4m33.193s\nuser 3m42.950s\nsys 1m13.821s\n\n/dev/shm/scr$ time git log --raw > _THELOG2\n\nreal 0m8.276s\nuser 0m7.192s\nsys 0m1.052s\n\n(not sure why it took 8s here, perhaps I had another process running\nat the same time? Compare with the \"0m4.913s\" below.)\n\n/dev/shm/scr$ time g-gc --aggressive\nCounting objects: 36066, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (27812/27812), done.\nWriting objects: 100% (36066/36066), done.\nTotal 36066 (delta 14367), reused 21699 (delta 0)\nChecking connectivity: 36066, done.\n\nreal 5m52.013s\nuser 8m28.652s\nsys 1m4.308s\n\n/dev/shm/scr$ time git log --raw > _THELOG2\n\nreal 1m34.430s\nuser 0m47.291s\nsys 0m46.615s\n\n/dev/shm/scr$ time git repack -adf\nCounting objects: 36066, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (27812/27812), done.\nWriting objects: 100% (36066/36066), done.\nTotal 36066 (delta 14256), reused 21699 (delta 0)\n\nreal 2m32.083s\nuser 1m51.295s\nsys 1m4.940s\n\n/dev/shm/scr$ time git log --raw > _THELOG3\n\nreal 0m4.913s\nuser 0m3.944s\nsys 0m0.944s\n\n/dev/shm/scr$ du -s .git\n43728 .git\n\n- back in the original place:\n\n~/scr$ time git repack -a -d -f\nCounting objects: 36066, done.\nDelta compression using up to 2 threads.\nCompressing objects: 100% (27812/27812), done.\nWriting objects: 100% (36066/36066), done.\nTotal 36066 (delta 14257), reused 21700 (delta 0)\n\nreal 4m6.503s\nuser 3m16.568s\nsys 1m11.640s\n\n~/scr$ time git log --raw > _THELOG2\n\nreal 0m5.002s\nuser 0m4.032s\nsys 0m0.952s\n"},{"id":"235017","messageId":"xmqqzjlocf28.fsf@gitster.dls.corp.google.com","threadId":"35903","inReplyTo":"20140218155842.GA7855@google.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-02-18T20:59:43Z","receivedAt":"2014-02-18T20:59:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> David Kastrup wrote:\n>> Duy Nguyen writes:\n>\n>>> Likely because --aggressive passes --depth=250 to pack-objects. Long\n>>> delta chains could reduce pack size and increase I/O as well as zlib\n>>> processing signficantly.\n> [...]\n>> Compression should reduce rather than increase the total amount of\n>> reads.\n>\n> --depth=250 means to allow chains of \"To get this object, first\n> inflate this object, then apply this delta\" of length 250.\n>\n> That's absurdly long, and doesn't even help compression much in\n> practice (many short chains referring to the same objects tends to\n> work fine). We probably shouldn't make --aggressive do that.\n> Something like --depth=10 would make more sense.\n\nYes, my thinking indeed.\n\nI didn't know --agressive was so aggressive myself, as I personally\nnever use it. \"git repack -a -d -f --depth=32 window=4000\" is what I\noften use, but I suspect most people would not be patient enough for\nthat 4k window.\n\nLet's do something like this first and then later make --depth\nconfigurable just like --width, perhaps? For \"aggressive\", I think\nthe default width (hardcoded to 250 but configurable) is a bit too\nnarrow.\n\n builtin/gc.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 6be6c8d..0d010f0 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -204,7 +204,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tif (aggressive) {\n \t\targv_array_push(&repack, \"-f\");\n-\t\targv_array_push(&repack, \"--depth=250\");\n+\t\targv_array_push(&repack, \"--depth=20\");\n \t\tif (aggressive_window > 0)\n \t\t\targv_array_pushf(&repack, \"--window=%d\", aggressive_window);\n \t}\n"},{"id":"235029","messageId":"CACsJy8AEXP45K+r3gGVTWbn4uuPLeHOkf-an20rj77QSfG1-ew@mail.gmail.com","threadId":"35903","inReplyTo":"xmqqzjlocf28.fsf@gitster.dls.corp.google.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-18T22:46:10Z","receivedAt":"2014-02-18T22:46:10Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 19, 2014 at 3:59 AM, Junio C Hamano wrote:\n> Let's do something like this first and then later make --depth\n> configurable just like --width, perhaps? For \"aggressive\", I think\n> the default width (hardcoded to 250 but configurable) is a bit too\n> narrow.\n>\n> builtin/gc.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 6be6c8d..0d010f0 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -204,7 +204,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n>\n> if (aggressive) {\n> argv_array_push(&repack, \"-f\");\n> - argv_array_push(&repack, \"--depth=250\");\n> + argv_array_push(&repack, \"--depth=20\");\n> if (aggressive_window > 0)\n> argv_array_pushf(&repack, \"--window=%d\", aggressive_window);\n> }\n\nLower depth than default (50) does not sound \"aggressive\" to me, at\nleast from disk space utilization. I agree it should be configurable\nthough.\n-- \nDuy\n"},{"id":"235041","messageId":"xmqq4n3warni.fsf@gitster.dls.corp.google.com","threadId":"35903","inReplyTo":"CACsJy8AEXP45K+r3gGVTWbn4uuPLeHOkf-an20rj77QSfG1-ew@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-02-19T00:10:41Z","receivedAt":"2014-02-19T00:10:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Lower depth than default (50) does not sound \"aggressive\" to me, at\n> least from disk space utilization. I agree it should be configurable\n> though.\n\nDo you mean you want to keep \"--aggressive\" to mean \"too aggressive\nin resulting size, to the point that it is not useful to anybody\"?\n\nShallow and wide will give us, with a large window, the most\naggressively efficient packfiles that are useful, and we would\nrather want to fix it to be usable, I would think.\n"},{"id":"235045","messageId":"CACsJy8C+wGd9WxnsML6-_G_S5GtN2pCPf09kcFtBVu-SDfP8YA@mail.gmail.com","threadId":"35903","inReplyTo":"xmqq4n3warni.fsf@gitster.dls.corp.google.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-19T00:33:15Z","receivedAt":"2014-02-19T00:33:15Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 19, 2014 at 7:10 AM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>> Lower depth than default (50) does not sound \"aggressive\" to me, at\n>> least from disk space utilization. I agree it should be configurable\n>> though.\n>\n> Do you mean you want to keep \"--aggressive\" to mean \"too aggressive\n> in resulting size, to the point that it is not useful to anybody\"?\n\ngit-gc.txt is pretty vague about this --aggressive. I assume we would\nwant both, better disk utilization and performance. But if it produces\na tiny pack that takes forever to access, then it's definitely bad\naggression.\n\n> Shallow and wide will give us, with a large window, the most\n> aggressively efficient packfiles that are useful, and we would\n> rather want to fix it to be usable, I would think.\n\nfwiw this is the thread that added --depth=250\n\nhttp://thread.gmane.org/gmane.comp.gcc.devel/94565/focus=94626\n\nyes, if reducing depth leads to better performance and does not use\nmuch disk in general case, then of course we should do it. \"General\ncase\" may be hard to define though. It'd be best if we have some sort\nof heuristics to try out different combinations on a specific repo and\nreturn the \"best\" combination of parameters. It could even take longer\ntime, but once we have good parameters, they should remain good for a\nlong time, I think.\n-- \nDuy\n"},{"id":"235047","messageId":"CAGK7Mr4wpwUK6UF6vTmgszX4sajPDvQazY2QagFfH9BEJx_9Ow@mail.gmail.com","threadId":"35903","inReplyTo":"CACsJy8C+wGd9WxnsML6-_G_S5GtN2pCPf09kcFtBVu-SDfP8YA@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Philippe Vaucher","fromEmail":"philippe.vaucher@gmail.com","sentAt":"2014-02-19T08:38:31Z","receivedAt":"2014-02-19T08:38:31Z","isPatch":false,"sender":{"key":"philippe.vaucher@gmail.com","avatar":null},"body":"> fwiw this is the thread that added --depth=250\n>\n> http://thread.gmane.org/gmane.comp.gcc.devel/94565/focus=94626\n\nThis post is quite interesting:\nhttp://article.gmane.org/gmane.comp.gcc.devel/94637\n\nPhilippe\n"},{"id":"235048","messageId":"87bny3qxwr.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"CAGK7Mr4wpwUK6UF6vTmgszX4sajPDvQazY2QagFfH9BEJx_9Ow@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-19T09:01:08Z","receivedAt":"2014-02-19T09:01:08Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Philippe Vaucher writes:\n\n>> fwiw this is the thread that added --depth=250\n>>\n>> http://thread.gmane.org/gmane.comp.gcc.devel/94565/focus=94626\n>\n> This post is quite interesting:\n> http://article.gmane.org/gmane.comp.gcc.devel/94637\n\nYes. Of course I am prejudiced because I volunteered fixing git-blame\non the Emacs developer list in order to make it more feasible to\ntransfer the Emacs repository to Git.\n\nCalling git blame via C-x v g is a rather important part of the\nworkflow, and it's currently intolerable to work with on a number of\nfiles.\n\nWhile I'm fixing the basic shortcomings in builtin/blame.c itself, the\noperation \"fetch the objects\" is necessary for all objects at least\nonce. It's conceivable that some nice caching strategy would help with\navoiding the repeated traversal of long delta chain tails. That could\nalso help defusing the operation of basic stuff like git-log.\n\nBut the short and long end of it is that there are valid operations\naccessing a large amount of past history, and one point of having a\ndistributed version control system with non-shallow repository by\ndefault is to have history and ways of working with it at one's hand.\n\nAnd git's default modus of operation is _not_ to store things like\ncopies and moves and renames in commits, but deduce them from looking at\nthe stored data. So making looking at stored data including old data\nexpensive means that Git does not work well in the way it is designed to\noperate.\n\n-- \nDavid Kastrup\n"},{"id":"235051","messageId":"CACsJy8DsC9X=13iEpONcT6bw6qTw_O586_vZ2W_3O42ajEPF4A@mail.gmail.com","threadId":"35903","inReplyTo":"CAGK7Mr4wpwUK6UF6vTmgszX4sajPDvQazY2QagFfH9BEJx_9Ow@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-19T10:14:46Z","receivedAt":"2014-02-19T10:14:46Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 19, 2014 at 3:38 PM, Philippe Vaucher\n wrote:\n>> fwiw this is the thread that added --depth=250\n>>\n>> http://thread.gmane.org/gmane.comp.gcc.devel/94565/focus=94626\n>\n> This post is quite interesting:\n> http://article.gmane.org/gmane.comp.gcc.devel/94637\n\nEspecially this part\n\n-- 8< --\nAnd quite frankly, a delta depth\nof 250 is likely going to cause overflows in the delta cache (which is\nonly 256 entries in size *and* it's a hash, so it's going to start having\nhash conflicts long before hitting the 250 depth limit).\n-- 8< --\n\nSo in order to get file A's content, we go through its 250 level chain\n(and fill the cache), then we get to file B and do the same, which\nevicts nearly everything from A. By the time we go to the next commit,\nwe have to go through 250 levels for A again because the cache is\npretty much useless.\n\nI can think of two improvements we could make, either increase cache\nsize dynamically (within limits) or make it configurable. If we have N\nentries in worktree (both trees and blobs) and depth M, then we might\nneed to cache N*M objects for it to be effective. Christian, if you\nwant to experiment this, update MAX_DELTA_CACHE in sha1_file.c and\nrebuild.\n\nThe other is smarter eviction, instead of throwing all A's cached\nitems out (based on recent order), keep the last few items of A and\nevict B's oldest cached items. Hopefully by the next comit, we can\nstill reuse some cache for A and other files/trees. Delta cache needs\nto learn about grouping to achieve this.\n-- \nDuy\n"},{"id":"235052","messageId":"CACsJy8AQksxT-QROqxCKCjCzfqfz+SNN3=HTCjtRx3G-9GdnKg@mail.gmail.com","threadId":"35903","inReplyTo":"87bny3qxwr.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-19T10:24:20Z","receivedAt":"2014-02-19T10:24:20Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 19, 2014 at 4:01 PM, David Kastrup wrote:\n> Calling git blame via C-x v g is a rather important part of the\n> workflow, and it's currently intolerable to work with on a number of\n> files.\n>\n> While I'm fixing the basic shortcomings in builtin/blame.c itself, the\n> operation \"fetch the objects\" is necessary for all objects at least\n> once. It's conceivable that some nice caching strategy would help with\n> avoiding the repeated traversal of long delta chain tails. That could\n> also help defusing the operation of basic stuff like git-log.\n\nPack v4 is supposed to tackle this delta chain thing, but its future\nis a bit uncertain (you can give a hand btw). If you often do \"git\nblame\", you might consider unpack most accessed objects (make it part\nof \"blame\" process), which would function exactly like a cache with no\nextra code. The downside is git-gc --auto is more likely to kick in\nbecause of too many loose objects and pack everything up again.\n-- \nDuy\n"},{"id":"235061","messageId":"xmqqd2ij9be1.fsf@gitster.dls.corp.google.com","threadId":"35903","inReplyTo":"CAGK7Mr4wpwUK6UF6vTmgszX4sajPDvQazY2QagFfH9BEJx_9Ow@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-02-19T18:59:34Z","receivedAt":"2014-02-19T18:59:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Philippe Vaucher writes:\n\n>> fwiw this is the thread that added --depth=250\n>>\n>> http://thread.gmane.org/gmane.comp.gcc.devel/94565/focus=94626\n>\n> This post is quite interesting:\n> http://article.gmane.org/gmane.comp.gcc.devel/94637\n\nYes, it most clearly says that --depth=250 was *not* a\nrecommendation, with technical background to explain why such a long\ndelta chain is a bad idea.\n"},{"id":"235087","messageId":"CAEjYwfX_vZ1hYC1RKV7JXYhvd2Ru1kLSbNre-kOp_koK-GJMgg@mail.gmail.com","threadId":"35903","inReplyTo":"CACsJy8DsC9X=13iEpONcT6bw6qTw_O586_vZ2W_3O42ajEPF4A@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Christian Jaeger","fromEmail":"chrjae@gmail.com","sentAt":"2014-02-20T04:09:09Z","receivedAt":"2014-02-20T04:09:09Z","isPatch":false,"sender":{"key":"chrjae@gmail.com","avatar":null},"body":"2014-02-19 10:14 GMT+00:00 Duy Nguyen :\n> Christian, if you\n> want to experiment this, update MAX_DELTA_CACHE in sha1_file.c and\n> rebuild.\n\nI don't have the time right now. (Perhaps next week?)\n"},{"id":"235098","messageId":"8738jdspbe.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"CACsJy8DsC9X=13iEpONcT6bw6qTw_O586_vZ2W_3O42ajEPF4A@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-20T16:48:21Z","receivedAt":"2014-02-20T16:48:21Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Duy Nguyen writes:\n\n> I can think of two improvements we could make, either increase cache\n> size dynamically (within limits) or make it configurable. If we have N\n> entries in worktree (both trees and blobs) and depth M, then we might\n> need to cache N*M objects for it to be effective. Christian, if you\n> want to experiment this, update MAX_DELTA_CACHE in sha1_file.c and\n> rebuild.\n\nWell, my optimized \"git-blame\" code is considerably hit by an\naggressively packed Emacs repository so I took a look at it with the\nMAX_DELTA_CACHE value set to the default 256, and then 512, 1024, 2048.\n\nHere are the results:\n\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t1m17.496s\nuser\t0m30.552s\nsys\t0m46.496s\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t1m13.888s\nuser\t0m30.060s\nsys\t0m43.420s\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t1m16.415s\nuser\t0m31.436s\nsys\t0m44.564s\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t1m24.732s\nuser\t0m34.416s\nsys\t0m49.808s\n\nSo using a value of 512 helps a bit (7% or so), but further increases\nalready cause a hit. My machine has 4G of memory (32bit x86), so it is\nunlikely that memory is running out. I have no idea why this would be\nso: either memory locality plays a role here, or the cache for some\nreason gets reinitialized or scanned/copied/accessed as a whole\nrepeatedly, defeating the idea of a cache. Or the access pattern are\nsuch that it's entirely useless as a cache even at this size.\n\nTrying with 16384:\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t2m8.000s\nuser\t0m54.968s\nsys\t1m12.624s\n\nAnd memory consumption did not exceed about 200m all the while, so is\nfar lower than what would have been available.\n\nSomething's _really_ fishy about that cache behavior. Note that the\n_system_ time goes up considerably, not just user time. Since the packs\nare zlib-packed, it's reasonable that more I/O time is also associated\nwith more user time and it is well possible that the user time increase\nis entirely explainable by the larger amount of compressed data to\naccess.\n\nBut this stinks. I doubt that the additional time is spent in memory\nallocation: most of that would register only as user time. And the\ntotal allocated memory is not large enough that one can explain this\naway with fewer available disk buffers for the kernel: the aggressively\npacked repo takes about 300m so it would fine into memory together with\nthe git process.\n\n-- \nDavid Kastrup\n"},{"id":"235099","messageId":"87y515r9wb.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"8738jdspbe.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-20T17:06:44Z","receivedAt":"2014-02-20T17:06:44Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"David Kastrup writes:\n\n> Duy Nguyen writes:\n>\n>> I can think of two improvements we could make, either increase cache\n>> size dynamically (within limits) or make it configurable. If we have N\n>> entries in worktree (both trees and blobs) and depth M, then we might\n>> need to cache N*M objects for it to be effective. Christian, if you\n>> want to experiment this, update MAX_DELTA_CACHE in sha1_file.c and\n>> rebuild.\n>\n> Well, my optimized \"git-blame\" code is considerably hit by an\n> aggressively packed Emacs repository so I took a look at it with the\n> MAX_DELTA_CACHE value set to the default 256, and then 512, 1024, 2048.\n\n[...]\n\n> Trying with 16384:\n> dak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n>\n> real\t2m8.000s\n> user\t0m54.968s\n> sys\t1m12.624s\n>\n> And memory consumption did not exceed about 200m all the while, so is\n> far lower than what would have been available.\n\nOf course, this has to do with delta_base_cache_limit defaulting to 16m.\n\n> Something's _really_ fishy about that cache behavior. Note that the\n> _system_ time goes up considerably, not just user time. Since the\n> packs are zlib-packed, it's reasonable that more I/O time is also\n> associated with more user time and it is well possible that the user\n> time increase is entirely explainable by the larger amount of\n> compressed data to access.\n>\n> But this stinks.\n\nAnd an obvious contender for the stinking is that the \"LRU\" scheme used\nhere is _strictly_ freeing memory based on which cache entry has been\n_created_ the longest time ago, not which cache entry has been\n_accessed_ the longest time ago. Which means a pure round-robin\nstrategy for freeing memory rather than LRU.\n\nLet's see what happens when changing this.\n\n-- \nDavid Kastrup\n"},{"id":"235113","messageId":"87ppmhr72d.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"87y515r9wb.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-20T18:07:54Z","receivedAt":"2014-02-20T18:07:54Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"David Kastrup writes:\n\n> David Kastrup writes:\n>\n>> Duy Nguyen writes:\n>>\n>> Something's _really_ fishy about that cache behavior. Note that the\n>> _system_ time goes up considerably, not just user time. Since the\n>> packs are zlib-packed, it's reasonable that more I/O time is also\n>> associated with more user time and it is well possible that the user\n>> time increase is entirely explainable by the larger amount of\n>> compressed data to access.\n>>\n>> But this stinks.\n>\n> And an obvious contender for the stinking is that the \"LRU\" scheme used\n> here is _strictly_ freeing memory based on which cache entry has been\n> _created_ the longest time ago, not which cache entry has been\n> _accessed_ the longest time ago. Which means a pure round-robin\n> strategy for freeing memory rather than LRU.\n>\n> Let's see what happens when changing this.\n\nNot much. With any cache size, using a \"true\" LRU scheme does not buy\nmore than 2%. On the other hand, increasing core.deltaBaseCacheLimit\nfrom its default of 16m to 128m in the config file results in the\nfollowing difference (with default #define MAX_DELTA_CACHE (256)):\n\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t1m17.446s\nuser\t0m30.696s\nsys\t0m46.332s\ndak@lola:/usr/local/tmp/emacs$ time ../git/git blame src/xdisp.c >/dev/null\n\nreal\t0m27.519s\nuser\t0m20.248s\nsys\t0m7.156s\n\nSo it would seem that the default available cache slots are not utilized\nanyway when operating on this file (about 1MB in size) with the default\nof core.deltaBaseCacheLimit.\n\nIt is still irritating that the performance drops quite a bit with a\nconsiderably larger number of cache slots.\n\n-- \nDavid Kastrup\n"},{"id":"235114","messageId":"CACsJy8AeZWPz=39ySQr9MrSUiLyJDQbs02sumS9VjbbGWzP9pw@mail.gmail.com","threadId":"35903","inReplyTo":"xmqqd2ij9be1.fsf@gitster.dls.corp.google.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-20T23:35:06Z","receivedAt":"2014-02-20T23:35:06Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Feb 20, 2014 at 1:59 AM, Junio C Hamano wrote:\n> Philippe Vaucher writes:\n>\n>>> fwiw this is the thread that added --depth=250\n>>>\n>>> http://thread.gmane.org/gmane.comp.gcc.devel/94565/focus=94626\n>>\n>> This post is quite interesting:\n>> http://article.gmane.org/gmane.comp.gcc.devel/94637\n>\n> Yes, it most clearly says that --depth=250 was *not* a\n> recommendation, with technical background to explain why such a long\n> delta chain is a bad idea.\n\nOn the other hand, the size reduction is really nice (320MB vs 500MB).\nI don't know if we can do this, but does it make sense to apply\n--depth=250 for old commits only and shallow depth for recent commits?\n\nFor old projects, commits older than 1-2 years is probably less often\naccessed and could use some aggressive packing. This still hits\ngit-blame badly. We could even make sure all objects \"on the blame\nsurface\" have short delta chain. But that may be pushing pack-objects\ntoo much.\n-- \nDuy\n"},{"id":"235118","messageId":"CAEjYwfU7un6wuA0Z-hSycROSWNdfQq1NawC7=+cNZeNu4-DpJg@mail.gmail.com","threadId":"35903","inReplyTo":"CACsJy8AeZWPz=39ySQr9MrSUiLyJDQbs02sumS9VjbbGWzP9pw@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Christian Jaeger","fromEmail":"chrjae@gmail.com","sentAt":"2014-02-21T00:32:48Z","receivedAt":"2014-02-21T00:32:48Z","isPatch":false,"sender":{"key":"chrjae@gmail.com","avatar":null},"body":"2014-02-20 23:35 GMT+00:00 Duy Nguyen :\n> does it make sense to apply\n> --depth=250 for old commits only\n\nJust wondering: would it be difficult to fix the problems that lead to\nworse than linear slowdown with the --depth? (I.e. adaptive cache/hash\ntable size.) If the performance difference between say --depth=25 and\n--depth=250 could be reduced from a factor 40 to 10 (or better if\nthings are back to other things taking more time than the object\naccess), that would seem like a nice gain in any case.\n\nAlso, in \"man git-gc\" document --aggressive that it leads to slower\n*read* performance after the gc, I remember having red that option's\ndocs when I ran it, and since it didn't mention that it makes reads\nslower, I didn't expect it to, and thus didn't remember this as the\nsource of the problem when I noticed that things were slow.\n\n(But, I took from the discussion that increasing the gzip window size\n(?) would make things smaller anyway, so perhaps all that isn't even\nnecessary?)\n\nI can test next week if you have particular suggestions to test.\n\nChristian.\n"},{"id":"235120","messageId":"20140221050947.GA12510@duynguyen-vnpc.dek-tpc.internal","threadId":"35903","inReplyTo":"CACsJy8AeZWPz=39ySQr9MrSUiLyJDQbs02sumS9VjbbGWzP9pw@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-21T05:09:47Z","receivedAt":"2014-02-21T05:09:47Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Feb 21, 2014 at 06:35:06AM +0700, Duy Nguyen wrote:\n> On the other hand, the size reduction is really nice (320MB vs 500MB).\n> I don't know if we can do this, but does it make sense to apply\n> --depth=250 for old commits only and shallow depth for recent commits?\n> \n> For old projects, commits older than 1-2 years is probably less often\n> accessed and could use some aggressive packing. This still hits\n> git-blame badly. We could even make sure all objects \"on the blame\n> surface\" have short delta chain. But that may be pushing pack-objects\n> too much.\n\nWe can have a \"moderately aggressive\" mode like this. With the patch\nbelow, first you repack all and remove all loose objects. Then replay\nyour favourite use cases with GIT_LOOSE_THEM=1. For example, if I'm\nmost interested in commits from a yearq ago\n\n$ GIT_LOOSE_THEM=1 ../git log --raw --since=1.year.ago >/dev/null\n\nall relevant trees will be unpacked. Put --stat there too if you want\nto unpack blobs. blame-heavy users may want to blame a few (or all)\nfiles here too to unpack more. Now we can repack aggressively all\nnon-loose objects:\n\n$ git repack -adf --exclude-loose --depth=250\n\nand repack again, this time with normal depth, which would only affect\nloose objects\n\n$ git repack -ad\n\nThe end result is a pack with ancient history with potentially long\ndelta chains, tightly packed, and nearer history with shorter\nchains. You will not notice any performance degradation (unless I run\npast 1 year history in my case). And the result pack of git.git is 39M\nrather than 64M with standard depth.\n\nThe use of loose objects to mark recent objects is not efficient (but\nfast for this prototype). We could store an SHA-1 map instead.\n\n-- 8< --\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 541667f..0e9dc8c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -82,6 +82,7 @@ static int num_preferred_base;\n static struct progress *progress_state;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n+static int no_loose;\n \n static unsigned long delta_cache_size = 0;\n static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n@@ -2204,7 +2205,12 @@ static void show_object(struct object *obj,\n \t\t\tconst struct name_path *path, const char *last,\n \t\t\tvoid *data)\n {\n-\tchar *name = path_name(path, last);\n+\tchar *name;\n+\n+\tif (no_loose && has_loose_object(obj->sha1))\n+\t\treturn;\n+\n+\tname = path_name(path, last);\n \n \tadd_preferred_base_object(name);\n \tadd_object_entry(obj->sha1, obj->type, name, 0);\n@@ -2487,6 +2493,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_SET_INT, 0, \"reflog\", &rev_list_reflog, NULL,\n \t\t N_(\"include objects referred by reflog entries\"),\n \t\t PARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, 1 },\n+\t\tOPT_BOOL(0, \"exclude-loose\", &no_loose, \"\"),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n \t\t\t N_(\"output pack to stdout\")),\n \t\tOPT_BOOL(0, \"include-tag\", &include_tag,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex bb2314c..9b8bb35 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -137,6 +137,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tint quiet = 0;\n \tint local = 0;\n+\tint no_loose = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -152,6 +153,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n \t\tOPT_BOOL('n', NULL, &no_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\")),\n+\t\tOPT_BOOL(0, \"exclude-loose\", &no_loose, \"\"),\n \t\tOPT__QUIET(&quiet, N_(\"be quiet\")),\n \t\tOPT_BOOL('l', \"local\", &local,\n \t\t\t\tN_(\"pass --local to git-pack-objects\")),\n@@ -184,6 +186,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd_args, \"--non-empty\");\n \targv_array_push(&cmd_args, \"--all\");\n \targv_array_push(&cmd_args, \"--reflog\");\n+\tif (no_loose)\n+\t\targv_array_push(&cmd_args, \"--exclude-loose\");\n \tif (window)\n \t\targv_array_pushf(&cmd_args, \"--window=%s\", window);\n \tif (window_memory)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 6e8c05d..d0988f2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -454,7 +454,7 @@ int has_loose_object_nonlocal(const unsigned char *sha1)\n \treturn 0;\n }\n \n-static int has_loose_object(const unsigned char *sha1)\n+int has_loose_object(const unsigned char *sha1)\n {\n \treturn has_loose_object_local(sha1) ||\n \t has_loose_object_nonlocal(sha1);\n@@ -2114,6 +2114,11 @@ struct unpack_entry_stack_ent {\n \tunsigned long size;\n };\n \n+static void write_sha1_file_prepare(const void *buf, unsigned long len,\n+\t\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t\t char *hdr, int *hdrlen);\n+static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n+\t\t\t const void *buf, unsigned long len, time_t mtime);\n void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \t\t enum object_type *final_type, unsigned long *final_size)\n {\n@@ -2126,6 +2131,7 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \tstruct unpack_entry_stack_ent *delta_stack = small_delta_stack;\n \tint delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;\n \tint base_from_cache = 0;\n+\tstatic int let_them_loose = -1;\n \n \tif (log_pack_access != no_log_pack_access)\n \t\twrite_pack_access_log(p, obj_offset);\n@@ -2288,6 +2294,17 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \t*final_type = type;\n \t*final_size = size;\n \n+\tif (let_them_loose == -1)\n+\t\tlet_them_loose = getenv(\"GIT_LOOSE_THEM\") != NULL;\n+\tif (let_them_loose && (type == OBJ_TREE || type == OBJ_BLOB)) {\n+\t\tunsigned char sha1[20];\n+\t\tchar hdr[32];\n+\t\tint hdrlen;\n+\t\twrite_sha1_file_prepare(data, size, typename(type), sha1, hdr, &hdrlen);\n+\t\tif (!has_loose_object(sha1))\n+\t\t\twrite_loose_object(sha1, hdr, hdrlen, data, size, 0);\n+\t}\n+\n \tunuse_pack(&w_curs);\n \treturn data;\n }\n-- 8< --\n\n--\nDuy\n"},{"id":"235142","messageId":"xmqq4n3s5pwx.fsf@gitster.dls.corp.google.com","threadId":"35903","inReplyTo":"CAEjYwfU7un6wuA0Z-hSycROSWNdfQq1NawC7=+cNZeNu4-DpJg@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-02-21T17:36:14Z","receivedAt":"2014-02-21T17:36:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Jaeger writes:\n\n> Also, in \"man git-gc\" document --aggressive that it leads to slower\n> *read* performance after the gc, I remember having red that option's\n> docs when I ran it, and since it didn't mention that it makes reads\n> slower, I didn't expect it to, and thus didn't remember this as the\n> source of the problem when I noticed that things were slow.\n\nGood point. We would at least need such a documentation update to\nwarn users.\n\n> (But, I took from the discussion that increasing the gzip window size\n> (?) would make things smaller anyway, so perhaps all that isn't even\n> necessary?)\n\nIf you are talking about \"--window\" in \"git repack --window=xxxx\",\nthat is not related to gzip. It is how many other \"similar\" objects\nan object will be tried to delta against to find a smallest delta\nthat can represent it in the pack. Such a better delta, if found,\ncan give you a packfile with a smaller depth that is as small as\nanother packfile created with a larger depth, which is an overall\nwin, and using a wider window is a way to achieve such a result.\n"},{"id":"235144","messageId":"xmqqzjlk4atj.fsf@gitster.dls.corp.google.com","threadId":"35903","inReplyTo":"CACsJy8AeZWPz=39ySQr9MrSUiLyJDQbs02sumS9VjbbGWzP9pw@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-02-21T17:47:36Z","receivedAt":"2014-02-21T17:47:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> For old projects, commits older than 1-2 years is probably less often\n> accessed and could use some aggressive packing.\n\nI used to repack older part of history manually with a deeper depth,\nmark the result with the .keep bit, and then repack the whole thing\nagain to have the remainder in a shallower depth. Something like:\n\n\tgit rev-list --objects v1.5.3 |\n git pack-objects --depth=128 --delta-base-offset pack\n\nwould give me the first pack (in real life, I would use a larger\nwindow size like 4096), and then after placing the resulting .pack\nand .idx files along with a .keep file in .git/objects/pack/,\nrunning \"git repack -a -d\" to pack the rest.\n\n> This still hits git-blame badly. We could even make sure all\n> objects \"on the blame surface\" have short delta chain. But that\n> may be pushing pack-objects too much.\n\nYes, you can do a similar trick by blaming all the paths that ever\nexisted in the project, parse its --porcelain output to learn all\nthe commits and paths involved, to find the objects that need\nquicker access. Pack such objects in a pack with a shallow depth,\ntentatively mark that pack with .keep, repack the remainder with a\ndeep depth, remove .keep from the first pack and mark the new pack\nwith .keep to prevent it from getting repacked, or something like\nthat.\n"},{"id":"235158","messageId":"CACsJy8DnjQyzY2ym7=fAQzThuhMuFzGLuKc35JJXn5FfB7r4Gg@mail.gmail.com","threadId":"35903","inReplyTo":"xmqqzjlocf28.fsf@gitster.dls.corp.google.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-22T00:36:06Z","receivedAt":"2014-02-22T00:36:06Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 19, 2014 at 3:59 AM, Junio C Hamano wrote:\n> I didn't know --agressive was so aggressive myself, as I personally\n> never use it. \"git repack -a -d -f --depth=32 window=4000\" is what I\n> often use, but I suspect most people would not be patient enough for\n> that 4k window.\n>\n> Let's do something like this first and then later make --depth\n> configurable just like --width, perhaps? For \"aggressive\", I think\n> the default width (hardcoded to 250 but configurable) is a bit too\n> narrow.\n\nOK with git://git.savannah.gnu.org/emacs.git we have\n\n - a 209MB pack with --aggressive\n - 1.3GB with --depth=50\n - 1.3GB with --window=4000 --depth=32\n - 1.3GB with --depth=20\n - 821MB with --depth=250 for commits --before=2.years.ago, --depth=50\nfor the rest\n\nSo I don't think we should go with your following patch because the\nsize explosion is just too much no matter how faster it could be. An\nimmediate action could be just make --depth=250 configurable and let\npeople deal with it. A better option is something like \"3 repack\nsteps\" you described where we pack deep depth first, mark .keep, pack\nshallower depth and combine them all into one.\n\nI'm not really happy with --depth=250 producing 209MB while\n--depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe\nI did something wrong)\n\n>\n> builtin/gc.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 6be6c8d..0d010f0 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -204,7 +204,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n>\n> if (aggressive) {\n> argv_array_push(&repack, \"-f\");\n> - argv_array_push(&repack, \"--depth=250\");\n> + argv_array_push(&repack, \"--depth=20\");\n> if (aggressive_window > 0)\n> argv_array_pushf(&repack, \"--window=%d\", aggressive_window);\n> }\n\n\n\n-- \nDuy\n"},{"id":"235173","messageId":"87fvnbhdn7.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"CACsJy8DnjQyzY2ym7=fAQzThuhMuFzGLuKc35JJXn5FfB7r4Gg@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-22T06:20:28Z","receivedAt":"2014-02-22T06:20:28Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Duy Nguyen writes:\n\n> OK with git://git.savannah.gnu.org/emacs.git we have\n>\n> - a 209MB pack with --aggressive\n> - 1.3GB with --depth=50\n> - 1.3GB with --window=4000 --depth=32\n> - 1.3GB with --depth=20\n> - 821MB with --depth=250 for commits --before=2.years.ago, --depth=50\n> for the rest\n>\n> So I don't think we should go with your following patch because the\n> size explosion is just too much no matter how faster it could be. An\n> immediate action could be just make --depth=250 configurable and let\n> people deal with it. A better option is something like \"3 repack\n> steps\" you described where we pack deep depth first, mark .keep, pack\n> shallower depth and combine them all into one.\n>\n> I'm not really happy with --depth=250 producing 209MB while\n> --depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe\n> I did something wrong)\n\nThat does look strange: Emacs has a history of more than 30 years. But\nthe Git mirror is quite younger. Maybe one needs to make sure to use\nthe author date rather than the commit date here?\n\n-- \nDavid Kastrup\n"},{"id":"235174","messageId":"877g8nh6k8.fsf@fencepost.gnu.org","threadId":"35903","inReplyTo":"87fvnbhdn7.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2014-02-22T08:53:27Z","receivedAt":"2014-02-22T08:53:27Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"David Kastrup writes:\n\n> Duy Nguyen writes:\n>\n>> OK with git://git.savannah.gnu.org/emacs.git we have\n>>\n>> - a 209MB pack with --aggressive\n>> - 1.3GB with --depth=50\n>> - 1.3GB with --window=4000 --depth=32\n>> - 1.3GB with --depth=20\n>> - 821MB with --depth=250 for commits --before=2.years.ago, --depth=50\n>> for the rest\n>>\n>> So I don't think we should go with your following patch because the\n>> size explosion is just too much no matter how faster it could be. An\n>> immediate action could be just make --depth=250 configurable and let\n>> people deal with it. A better option is something like \"3 repack\n>> steps\" you described where we pack deep depth first, mark .keep, pack\n>> shallower depth and combine them all into one.\n>>\n>> I'm not really happy with --depth=250 producing 209MB while\n>> --depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe\n>> I did something wrong)\n>\n> That does look strange: Emacs has a history of more than 30 years. But\n> the Git mirror is quite younger. Maybe one needs to make sure to use\n> the author date rather than the commit date here?\n\nAnother thing: did you really use --depth=250 here or did you use\n--aggressive? It may be that the latter also sets other options?\n\n-- \nDavid Kastrup\n"},{"id":"235176","messageId":"CACsJy8Cyf6Mu3q1VWH7srCK4m=+UgR4m7RiNkMv-nr8eF4YAJA@mail.gmail.com","threadId":"35903","inReplyTo":"877g8nh6k8.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-22T09:14:33Z","receivedAt":"2014-02-22T09:14:33Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Feb 22, 2014 at 3:53 PM, David Kastrup wrote:\n> David Kastrup writes:\n>\n>> Duy Nguyen writes:\n>>\n>>> OK with git://git.savannah.gnu.org/emacs.git we have\n>>>\n>>> - a 209MB pack with --aggressive\n>>> - 1.3GB with --depth=50\n>>> - 1.3GB with --window=4000 --depth=32\n>>> - 1.3GB with --depth=20\n>>> - 821MB with --depth=250 for commits --before=2.years.ago, --depth=50\n>>> for the rest\n>>>\n>>> So I don't think we should go with your following patch because the\n>>> size explosion is just too much no matter how faster it could be. An\n>>> immediate action could be just make --depth=250 configurable and let\n>>> people deal with it. A better option is something like \"3 repack\n>>> steps\" you described where we pack deep depth first, mark .keep, pack\n>>> shallower depth and combine them all into one.\n>>>\n>>> I'm not really happy with --depth=250 producing 209MB while\n>>> --depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe\n>>> I did something wrong)\n>>\n>> That does look strange: Emacs has a history of more than 30 years. But\n>> the Git mirror is quite younger. Maybe one needs to make sure to use\n>> the author date rather than the commit date here?\n\nI think commit date is fine because it covers a large portion of\nobjects (649946 per total 739990) and it does not (or should not)\naffect object ordering in pack-objects/rev-list.\n\n> Another thing: did you really use --depth=250 here or did you use\n> --aggressive? It may be that the latter also sets other options?\n\nI can't use --aggressive because I need to feed revisions directly to\npack-objects. --aggressive also sets --window=250. Thanks for\nchecking. My machine will have another workout session.\n-- \nDuy\n"},{"id":"235186","messageId":"m261o7a2rm.fsf@linux-m68k.org","threadId":"35903","inReplyTo":"87fvnbhdn7.fsf@fencepost.gnu.org","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2014-02-22T09:57:17Z","receivedAt":"2014-02-22T09:57:17Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"David Kastrup writes:\n\n> That does look strange: Emacs has a history of more than 30 years. But\n> the Git mirror is quite younger. Maybe one needs to make sure to use\n> the author date rather than the commit date here?\n\nThere is no difference between commit and author date in the Emacs git\nmirror since bzr doesn't keep that distinction (and cvs didn't either).\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"235188","messageId":"CACsJy8AS6FMqMXcsDtUvrC2bgZ90irMXDCh58KjmgQK8+yFwVA@mail.gmail.com","threadId":"35903","inReplyTo":"CACsJy8Cyf6Mu3q1VWH7srCK4m=+UgR4m7RiNkMv-nr8eF4YAJA@mail.gmail.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-22T13:00:47Z","receivedAt":"2014-02-22T13:00:47Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Feb 22, 2014 at 4:14 PM, Duy Nguyen wrote:\n> On Sat, Feb 22, 2014 at 3:53 PM, David Kastrup wrote:\n>> David Kastrup writes:\n>>\n>>> Duy Nguyen writes:\n>>>\n>>>> OK with git://git.savannah.gnu.org/emacs.git we have\n>>>>\n>>>> - a 209MB pack with --aggressive\n>>>> - 1.3GB with --depth=50\n>>>> - 1.3GB with --window=4000 --depth=32\n>>>> - 1.3GB with --depth=20\n>>>> - 821MB with --depth=250 for commits --before=2.years.ago, --depth=50\n>>>> for the rest\n...\n>>>>\n>>>> I'm not really happy with --depth=250 producing 209MB while\n>>>> --depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe\n>>>> I did something wrong)\n....\n>> Another thing: did you really use --depth=250 here or did you use\n>> --aggressive? It may be that the latter also sets other options?\n>\n> I can't use --aggressive because I need to feed revisions directly to\n> pack-objects. --aggressive also sets --window=250. Thanks for\n> checking. My machine will have another workout session.\n\nAnd 800MB is reduced to 177MB, containing history older than 2 years.\nThe final pack is 199MB, within the size range of current --aggressive\nand should be reasonably fast on most operations. Again blame could\nstill hit long delta chains but I think we should just unpack some\ntrees/blobs when we hit long delta chains.\n\nI think we should update --aggressive to do it this way. So\n\n - gc.aggressiveDepth defaults to 50 (or 20?), this is used for recent history\n - gc.aggressiveDeepDepth defaults to 250 (or smaller??), used for\nancient history\n - gc.aggressiveDeepOption is rev-list a rev-list option to define\n\"ancient history\", default to --before=2.years.ago. This option could\nbe specified multiple times.\n\nBoth packing phases use the same gc.aggressiveWindow. We could add\ngc.aggressiveDeepWindow too.\n\nGSoC project?\n-- \nDuy\n"},{"id":"235236","messageId":"CAGK7Mr71S608FtNWYvxHdSrCiEUuQicVy3xojNmwDSODufmsxw@mail.gmail.com","threadId":"35903","inReplyTo":"xmqqzjlk4atj.fsf@gitster.dls.corp.google.com","subject":"Re: git gc --aggressive led to about 40 times slower \"git log --raw\"","fromName":"Philippe Vaucher","fromEmail":"philippe.vaucher@gmail.com","sentAt":"2014-02-24T09:27:26Z","receivedAt":"2014-02-24T09:27:26Z","isPatch":false,"sender":{"key":"philippe.vaucher@gmail.com","avatar":null},"body":"> I used to repack older part of history manually with a deeper depth,\n> mark the result with the .keep bit, and then repack the whole thing\n> again to have the remainder in a shallower depth. Something like:\n>\n> git rev-list --objects v1.5.3 |\n> git pack-objects --depth=128 --delta-base-offset pack\n>\n> would give me the first pack (in real life, I would use a larger\n> window size like 4096), and then after placing the resulting .pack\n> and .idx files along with a .keep file in .git/objects/pack/,\n> running \"git repack -a -d\" to pack the rest.\n\nI'm curious, after these repacking, how do you guys publish these\npacks? git push? if yes, on what criteria does the remote repo know\nwhich pack it should fetch?\n\nOr maybe it's only a local operation and thus you cannot do it on the\nremote without ssh access?\n\nPhilippe\n"}]}