{"thread":{"id":"31199","subject":"git repack vs git gc --aggressive","startedAt":"2012-08-07T18:22:21Z","lastAt":"2012-08-13T17:19:56Z","messageCount":7,"participants":["Felix Natter","Jeff King","Junio C Hamano","Marc Branchaud"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"196609","messageId":"87zk66r28y.fsf@bitburger.home.felix","threadId":"31199","inReplyTo":null,"subject":"git repack vs git gc --aggressive","fromName":"Felix Natter","fromEmail":"fnatter@gmx.net","sentAt":"2012-08-07T18:22:21Z","receivedAt":"2012-08-07T18:22:21Z","isPatch":false,"sender":{"key":"fnatter@gmx.net","avatar":null},"body":"hello,\n\nI read this:\n http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/\nwhere\n git repack -a -d --depth=250 --window=250\nis mentioned as a (recommended) alternative to git gc --aggressive.\n\nI am a bit confused, because the page also mentions that git gc --aggressive\nis recommended when a repo has been imported using git fast-import.\n\nSo my questions are:\n\n1. is the above repack command (with --depth=500) safe? Of course I want\n to be absolutely sure that our repo will be consistent.\n Do I need another command (\"git gc\", \"git prune\") as well?\n\n2. is it the right tool for the job or shall I use git gc --aggressive?\n\nThanks!\n-- \nFelix Natter\n"},{"id":"196611","messageId":"20120807184405.GA440@sigill.intra.peff.net","threadId":"31199","inReplyTo":"87zk66r28y.fsf@bitburger.home.felix","subject":"Re: git repack vs git gc --aggressive","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-08-07T18:44:05Z","receivedAt":"2012-08-07T18:44:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 07, 2012 at 08:22:21PM +0200, Felix Natter wrote:\n\n> I read this:\n> http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/\n> where\n> git repack -a -d --depth=250 --window=250\n> is mentioned as a (recommended) alternative to git gc --aggressive.\n\nNote how old that post is. In fact, on the very same day it was posted,\nthe discussion on the mailing list resulted in this commit:\n\n commit 1c192f3442414a6ce83f9a524806fc26a0861d2d\n Author: Johannes Schindelin \n Date: Thu Dec 6 12:03:38 2007 +0000\n\n gc --aggressive: make it really aggressive\n\n The default was not to change the window or depth at all. As suggested\n by Jon Smirl, Linus Torvalds and others, default to\n\n --window=250 --depth=250\n\nSo the packing parameters are the same these days for either method.\nNote that \"git gc --aggressive\" will also use \"-f\" to recompute all\ndeltas. This is more expensive, but gives git more flexibility if the\nold deltas were sub-optimal (typically, this is the case if the existing\npack was generated by fast-import, which favors speed of import versus\ncoming up with an optimal storage pattern).\n\n> So my questions are:\n> \n> 1. is the above repack command (with --depth=500) safe? Of course I want\n> to be absolutely sure that our repo will be consistent.\n> Do I need another command (\"git gc\", \"git prune\") as well?\n\nYes, it's safe. Changing the depth parameter can never lose data.\nHowever, it's probably not a good idea for two reasons:\n\n 1. It probably does nothing. You're not likely to hit a 500-depth\n delta chain (the point of the \"250\" in --aggressive is that it is\n already ridiculously high).\n\n 2. Even if you did come up with a 500-depth delta chain, it may not be\n a good tradeoff. You might save a little bit of space, but keep in\n mind that to generate the object data, it means that git will have\n to follow a chain of 500 deltas to regenerate the object.\n\nOf course, every workload is different. One can develop pathological\ncases where --depth=500 saves a lot of space. But it's unlikely that it\nis the case for a normal repository. You can always try both and see the\nresult.\n\nIn fact, I'd also test how just \"git gc\" behaves versus \"git gc\n--aggressive\" for your repo. The former is much less expensive to run.\nYou really shouldn't need to be running \"--aggressive\" all the time, so\nif you are looking at doing a nightly repack or similar, just \"git gc\"\nis probably fine.\n\n-Peff\n"},{"id":"196613","messageId":"7vvcguv7y2.fsf@alter.siamese.dyndns.org","threadId":"31199","inReplyTo":"20120807184405.GA440@sigill.intra.peff.net","subject":"Re: git repack vs git gc --aggressive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-07T19:05:41Z","receivedAt":"2012-08-07T19:05:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> So the packing parameters are the same these days for either method.\n> Note that \"git gc --aggressive\" will also use \"-f\" to recompute all\n> deltas. This is more expensive, but gives git more flexibility if the\n> old deltas were sub-optimal (typically, this is the case if the existing\n> pack was generated by fast-import, which favors speed of import versus\n> coming up with an optimal storage pattern).\n\nAlso your fetch often results in storing the pack received from the\nother end straight to your local repository (with necessary objects\nto complete the pack the other end did not send appended at the\nend). If the server side hasn't been packed with \"-f\", you will\ninherit the badness until you repack with \"-f\".\n\n> Of course, every workload is different. One can develop pathological\n> cases where --depth=500 saves a lot of space. But it's unlikely that it\n> is the case for a normal repository. You can always try both and see the\n> result.\n\nFor a dataset where ridiculously large depth really is a win, these\nobjects would have to be reasonably large and cost of expanding the\nbase and then applying hundreds of delta to recover one object may\nnot be negligible. The user should consider if he is willing to pay\nthe price every time he does a local Git operation.\n\n> In fact, I'd also test how just \"git gc\" behaves versus \"git gc\n> --aggressive\" for your repo. The former is much less expensive to run.\n> You really shouldn't need to be running \"--aggressive\" all the time, so\n> if you are looking at doing a nightly repack or similar, just \"git gc\"\n> is probably fine.\n\nAs I am coming from \"large depth is harmful\" school, I would\nrecommend\n\n - \"git repack -a -d -f\" with large \"--window\" with reasonably short\n \"--depth\" once, and mark the result with .keep;\n \n - \"git repack -a -d -f\" once every several weeks; and\n\n - \"git gc\" or \"git repack\" (without any other options) daily.\n\nand ignore \"--aggressive\" entirely.\n"},{"id":"196823","messageId":"87ehnewolp.fsf@bitburger.home.felix","threadId":"31199","inReplyTo":"7vvcguv7y2.fsf@alter.siamese.dyndns.org","subject":"Re: git repack vs git gc --aggressive","fromName":"Felix Natter","fromEmail":"fnatter@gmx.net","sentAt":"2012-08-10T19:09:38Z","receivedAt":"2012-08-10T19:09:38Z","isPatch":false,"sender":{"key":"fnatter@gmx.net","avatar":null},"body":"Junio C Hamano writes:\n\n> Jeff King writes:\n>\n>> So the packing parameters are the same these days for either method.\n>> Note that \"git gc --aggressive\" will also use \"-f\" to recompute all\n>> deltas. This is more expensive, but gives git more flexibility if the\n>> old deltas were sub-optimal (typically, this is the case if the existing\n>> pack was generated by fast-import, which favors speed of import versus\n>> coming up with an optimal storage pattern).\n>\n> Also your fetch often results in storing the pack received from the\n> other end straight to your local repository (with necessary objects\n> to complete the pack the other end did not send appended at the\n> end). If the server side hasn't been packed with \"-f\", you will\n> inherit the badness until you repack with \"-f\".\n>\n>> Of course, every workload is different. One can develop pathological\n>> cases where --depth=500 saves a lot of space. But it's unlikely that it\n>> is the case for a normal repository. You can always try both and see the\n>> result.\n>\n> For a dataset where ridiculously large depth really is a win, these\n> objects would have to be reasonably large and cost of expanding the\n> base and then applying hundreds of delta to recover one object may\n> not be negligible. The user should consider if he is willing to pay\n> the price every time he does a local Git operation.\n>\n>> In fact, I'd also test how just \"git gc\" behaves versus \"git gc\n>> --aggressive\" for your repo. The former is much less expensive to run.\n>> You really shouldn't need to be running \"--aggressive\" all the time, so\n>> if you are looking at doing a nightly repack or similar, just \"git gc\"\n>> is probably fine.\n\nThank you both very much for your answers!\n\nI have a few questions about this:\n\n> As I am coming from \"large depth is harmful\" school, I would\n> recommend\n>\n> - \"git repack -a -d -f\" with large \"--window\" with reasonably short\n> \"--depth\" once, \n\nSo something like --depth=250 and --window=500? \n\n> and mark the result with .keep;\n\nI guess you refer to a toplevel '.keep' file. But what does\nthat do (sorry, couldn't find anything on google)?\n \n> - \"git repack -a -d -f\" once every several weeks; and\n>\n> - \"git gc\" or \"git repack\" (without any other options) daily.\n>\n> and ignore \"--aggressive\" entirely.\n\nOne more question: I use bzr fast-export | git fast-import to import\nbranches from bzr:\n\n bzr fast-export --marks=$MARKS_BZR --git-branch=\"$BRANCHNAME\" \"$BZR_FREEPLANE_REPO/$BRANCHNAME/\" | \\\n git fast-import --import-marks=$MARKS_GIT --export-marks=$MARKS_GIT\n\nWill those marks files (which remember which commits are already there in\nthe git repo) also work after I have done git repack / git gc?\nIn other words, can I import bzr-branches after I have run git repack /\ngit gc on the repo?\n\nThank you!\n-- \nFelix Natter\n"},{"id":"196830","messageId":"7v393ujypq.fsf@alter.siamese.dyndns.org","threadId":"31199","inReplyTo":"87ehnewolp.fsf@bitburger.home.felix","subject":"Re: git repack vs git gc --aggressive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-10T20:09:37Z","receivedAt":"2012-08-10T20:09:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Felix Natter writes:\n\n> I have a few questions about this:\n>\n>> As I am coming from \"large depth is harmful\" school, I would\n>> recommend\n>>\n>> - \"git repack -a -d -f\" with large \"--window\" with reasonably short\n>> \"--depth\" once, \n>\n> So something like --depth=250 and --window=500? \n\nI would use more like --depth=16 or 32 in my local repositories.\n\n>> and mark the result with .keep;\n>\n> I guess you refer to a toplevel '.keep' file.\n\nNot at all. And it is not documented, it seems X-<.\n\nTypically you have a pair of files in .git/objects/pack, e.g.\n\n .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.idx\n .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.pack\n\nAnd you can add another file next to them\n\n .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.keep\n\nto prevent the pack from getting repacked. I think \"git clone\" does\nthis for you after an initial import.\n"},{"id":"196910","messageId":"50290D1F.20007@xiplink.com","threadId":"31199","inReplyTo":"7v393ujypq.fsf@alter.siamese.dyndns.org","subject":"Re: git repack vs git gc --aggressive","fromName":"Marc Branchaud","fromEmail":"mbranchaud@xiplink.com","sentAt":"2012-08-13T14:20:15Z","receivedAt":"2012-08-13T14:20:15Z","isPatch":false,"sender":{"key":"mbranchaud@xiplink.com","avatar":null},"body":"On 12-08-10 04:09 PM, Junio C Hamano wrote:\n> Felix Natter writes:\n> \n>> I have a few questions about this:\n>>\n>>> As I am coming from \"large depth is harmful\" school, I would\n>>> recommend\n>>>\n>>> - \"git repack -a -d -f\" with large \"--window\" with reasonably short\n>>> \"--depth\" once, \n>>\n>> So something like --depth=250 and --window=500? \n> \n> I would use more like --depth=16 or 32 in my local repositories.\n> \n>>> and mark the result with .keep;\n>>\n>> I guess you refer to a toplevel '.keep' file.\n> \n> Not at all. And it is not documented, it seems X-<.\n> \n> Typically you have a pair of files in .git/objects/pack, e.g.\n> \n> .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.idx\n> .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.pack\n> \n> And you can add another file next to them\n> \n> .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.keep\n> \n> to prevent the pack from getting repacked. I think \"git clone\" does\n> this for you after an initial import.\n\n1.7.12.rc1 does not.\n\nI even cloned from a repo with a few .keep files, but ended up with only one\nbig .pack file.\n\nMaybe clone should preserve the packs it gets from the upstream repo? For\nexample, our main repo has a 690MB pack file that's marked .keep, but the\nclone just ends up with a single 725MB pack file. Would our clones see\nperformance improvements if they that big 690MB pack separate from the others?\n\nPerhaps the fact that clone creates a single pack file makes it impossible to\npreserve the .keep packs from the upstream?\n\n(I figure it's probably not a good idea for clone to .keep the single pack\nfile it creates.)\n\n\t\tM.\n"},{"id":"196919","messageId":"7vr4raemkj.fsf@alter.siamese.dyndns.org","threadId":"31199","inReplyTo":"50290D1F.20007@xiplink.com","subject":"Re: git repack vs git gc --aggressive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-13T17:19:56Z","receivedAt":"2012-08-13T17:19:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Marc Branchaud writes:\n\n> On 12-08-10 04:09 PM, Junio C Hamano wrote:\n>> Felix Natter writes:\n>> \n>>> I have a few questions about this:\n>>>\n>>>> As I am coming from \"large depth is harmful\" school, I would\n>>>> recommend\n>>>>\n>>>> - \"git repack -a -d -f\" with large \"--window\" with reasonably short\n>>>> \"--depth\" once, \n>>>\n>>> So something like --depth=250 and --window=500? \n>> \n>> I would use more like --depth=16 or 32 in my local repositories.\n>> \n>>>> and mark the result with .keep;\n>>>\n>>> I guess you refer to a toplevel '.keep' file.\n>> \n>> Not at all. And it is not documented, it seems X-<.\n>> \n>> Typically you have a pair of files in .git/objects/pack, e.g.\n>> \n>> .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.idx\n>> .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.pack\n>> \n>> And you can add another file next to them\n>> \n>> .git/objects/pack/pack-2e3e3b332b446278f9ff91c4f497bc6ed2626d00.keep\n>> \n>> to prevent the pack from getting repacked. I think \"git clone\" does\n>> this for you after an initial import.\n>\n> 1.7.12.rc1 does not.\n\nSorry, I misremembered. It was removed at 1db4a75 (Remove\nunnecessary pack-*.keep file after successful git-clone,\n2008-07-08), so even when the sender gave you a crappy pack, you can\nrepack locally to correct it.\n\n> Maybe clone should preserve the packs it gets from the upstream repo?\n\nThat was part of the intention of the code 1db4a75 removed.\n\n> For\n> example, our main repo has a 690MB pack file that's marked .keep, but the\n> clone just ends up with a single 725MB pack file. Would our clones see\n> performance improvements if they that big 690MB pack separate from the others?\n\nThere is no \"pack boundary\" in the object transfer protocol. What\ncomes out of the wire is a single stream of pack data, so the above\nis not feasible without major surgery and backward incompatible\nchange.\n"}]}