{"thread":{"id":"40642","subject":"git repack command on larger pack file","startedAt":"2015-10-26T05:57:36Z","lastAt":"2015-10-28T06:47:17Z","messageCount":9,"participants":["Sivakumar Selvam","Junio C Hamano","Philip Oakley","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"272225","messageId":"loom.20151026T065553-934@post.gmane.org","threadId":"40642","inReplyTo":null,"subject":"git repack command on larger pack file","fromName":"Sivakumar Selvam","fromEmail":"gerritcode@gmail.com","sentAt":"2015-10-26T05:57:36Z","receivedAt":"2015-10-26T05:57:36Z","isPatch":false,"sender":{"key":"gerritcode@gmail.com","avatar":null},"body":"Hi,\n I ran git repack on a single larger repository abc.git where the pack\nfile size 34 GB. Generally it used to take 20-25 minutes in my server to\ncomplete the repacking. During repacking I noticed, disk usage was more, So\nI thought of splitting the pack file into 4 GB chunks. I used the following\ncommand to do repacking.\n git repack -A -b -d -q --depth=50 --window=10 abc.git\n\n After adding --max-pack-size=4g to the above command again I ran to split\npack files..\n git repack -A -b -d -q --depth=50 --window=10 --max-pack-size=4g abc.git\n \n When I finished running, I found 12 pack files with each 4 GB and the\nsize is 48 GB. Now my disk usage has increased by 14 GB. Again, I ran to\ncheck the performance, but the size (48 GB) and time to repacking takes\nanother 35 minutes more. Why this issue? If we split a larger pack file,\nrepacking takes more time with more disk usage for storing pack files. Any\nthoughts on this why this happens?\n\nThanks,\nSivakumar Selvam.\n"},{"id":"272228","messageId":"xmqq611ujfn0.fsf@gitster.mtv.corp.google.com","threadId":"40642","inReplyTo":"loom.20151026T065553-934@post.gmane.org","subject":"Re: git repack command on larger pack file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-26T06:41:23Z","receivedAt":"2015-10-26T06:41:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sivakumar Selvam writes:\n\n> I ran git repack on a single larger repository abc.git where the pack\n> file size 34 GB. Generally it used to take 20-25 minutes in my server to\n> complete the repacking. During repacking I noticed, disk usage was more, So\n> I thought of splitting the pack file into 4 GB chunks. I used the following\n> command to do repacking.\n> git repack -A -b -d -q --depth=50 --window=10 abc.git\n>\n> After adding --max-pack-size=4g to the above command again I ran to split\n> pack files..\n> git repack -A -b -d -q --depth=50 --window=10 --max-pack-size=4g abc.git\n> \n> When I finished running, I found 12 pack files with each 4 GB and the\n> size is 48 GB. Now my disk usage has increased by 14 GB. Again, I ran to\n> check the performance, but the size (48 GB) and time to repacking takes\n> another 35 minutes more. Why this issue?\n\nHmmm, what is \"this issue\"? I do not see anything surprising.\n\nIf you have N objects and run repack with window=10, you would\n(roughly speaking, without taking various optimization we have and\nbootstrap conditions into account) check each of these N objects\nagainst 10 other objects to find good delta base, no matter how big\nyour max pack-size is set. And that takes the bulk of time in the\nrepack process. Also it has to write more data to disk (see below),\nit has to find a good place to split, it has to adjust bookkeeping\ndata at the pack boundary, in general it has to do more, not less,\nto produce split packs. It would be surprising if it took less\ntime.\n\nEach pack by definition has to be self-sufficient; all delta in the\npack must have its base object in the same pack. Now, imagine that\nan object (call it X) would have been expressed as a delta derived\nfrom another object (call it Y) if you were producing a single pack,\nand imagine that the pack has grown to be 4 GB big just before you\nwrite object X out. The current pack (which contains the base\nobject Y already) needs to be closed and then a new pack is opened.\nImagine how you would write X now into that new pack. You have to\ndiscard the deltified representation of X (which by definition is\nmuch smaller, because it is an instruction to reconstitute X given\nan object Y whose contents is very similar to X) and write the base\nrepresentation of X to the pack, because X can no longer be\nexpressed as a delta derived from Y. That is why you would need to\nwrite more.\n"},{"id":"272229","messageId":"xmqqziz6hzom.fsf@gitster.mtv.corp.google.com","threadId":"40642","inReplyTo":"xmqq611ujfn0.fsf@gitster.mtv.corp.google.com","subject":"Re: git repack command on larger pack file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-26T07:11:21Z","receivedAt":"2015-10-26T07:11:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Sivakumar Selvam writes:\n>\n>> ... So\n>> I thought of splitting the pack file into 4 GB chunks.\n> ...\n> Hmmm, what is \"this issue\"? I do not see anything surprising.\n\nWhile the explanation might have been enlightening, the knowledge\nconveyed by the explanation by itself would not be of much practical\nuse, and enlightment without practical use is never fun.\n\nSo let's do another tangent that may be more useful.\n\nIn many repositories, older parts of the history often hold the bulk\nof objects that do not change, and it is wasteful to repack them\nover and over. If your project is at around v40.0 today, and it was\nat around v36.0 6 months ago, for example, you may want to pack\neverything that happened before v36.0 into a single pack just once,\npack them really well, and have your \"repack\" not touch that old\npart of the history.\n\n $ git rev-list --objects v36.0 |\n git pack-objects --window=200 --depth=128 pack\n\nwould produce such a pack [*1*]\n\nThe standard output from the above pipeline will give you a 40-hex\nstring (e.g. 51c472761b4690a331c02c90ec364e47cca1b3ac, call it\n$HEX), and in the current directory you will find two files,\npack-$HEX.pack and pack-$HEX.idx.\n\nYou can then do this:\n\n $ echo \"v36.0 with W/D 200/128\" >pack-$HEX.keep\n $ mv pack-$HEX.* .git/objects/pack/.\n $ git repack -a -d\n\nA pack that has an accompanying .keep file is excempt from\nrepacking, so once you do this, your future \"git repack\" will only\nrepack objects that are not in the kept packs.\n\n\n\n[Footnote]\n\n*1* I won't say 200/128 gives you a good pack; you would need to\nexperiment. In general, larger depth will result in smaller pack\nbut it will result in bigger overhead while you use the repository\nevery day. Larger window will spend a lot of cycles while packing,\nbut will result in a smaller pack.\n"},{"id":"272287","messageId":"loom.20151027T025257-333@post.gmane.org","threadId":"40642","inReplyTo":"xmqqziz6hzom.fsf@gitster.mtv.corp.google.com","subject":"Re: git repack command on larger pack file","fromName":"Sivakumar Selvam","fromEmail":"gerritcode@gmail.com","sentAt":"2015-10-27T02:04:23Z","receivedAt":"2015-10-27T02:04:23Z","isPatch":false,"sender":{"key":"gerritcode@gmail.com","avatar":null},"body":"Junio C Hamano pobox.com> writes:\n\n> \n> Junio C Hamano pobox.com> writes:\n> \n> > Sivakumar Selvam gmail.com> writes:\n> >\n> >> ... So\n> >> I thought of splitting the pack file into 4 GB chunks.\n> > ...\n> > Hmmm, what is \"this issue\"? I do not see anything surprising.\n> \n> While the explanation might have been enlightening, the knowledge\n> conveyed by the explanation by itself would not be of much practical\n> use, and enlightment without practical use is never fun.\n> \n> So let's do another tangent that may be more useful.\n> \n> In many repositories, older parts of the history often hold the bulk\n> of objects that do not change, and it is wasteful to repack them\n> over and over. If your project is at around v40.0 today, and it was\n> at around v36.0 6 months ago, for example, you may want to pack\n> everything that happened before v36.0 into a single pack just once,\n> pack them really well, and have your \"repack\" not touch that old\n> part of the history.\n> \n> $ git rev-list --objects v36.0 |\n> git pack-objects --window=200 --depth=128 pack\n> \n> would produce such a pack [*1*]\n> \n> The standard output from the above pipeline will give you a 40-hex\n> string (e.g. 51c472761b4690a331c02c90ec364e47cca1b3ac, call it\n> $HEX), and in the current directory you will find two files,\n> pack-$HEX.pack and pack-$HEX.idx.\n> \n> You can then do this:\n> \n> $ echo \"v36.0 with W/D 200/128\" >pack-$HEX.keep\n> $ mv pack-$HEX.* .git/objects/pack/.\n> $ git repack -a -d\n> \n> A pack that has an accompanying .keep file is excempt from\n> repacking, so once you do this, your future \"git repack\" will only\n> repack objects that are not in the kept packs.\n> \n> [Footnote]\n> \n> *1* I won't say 200/128 gives you a good pack; you would need to\n> experiment. In general, larger depth will result in smaller pack\n> but it will result in bigger overhead while you use the repository\n> every day. Larger window will spend a lot of cycles while packing,\n> but will result in a smaller pack.\n> \n\n\nHi Junio,\n\n When I finished git repacking, I found 12 pack files with each 4 GB and\nthe total size is 48 GB. Again I ran the same git repack command by just\nremoving only --max-pack-size= parameter, the size of the single pack file\nis 66 GB.\n\ngit repack -A -b -d -q --depth=50 --window=10 abc.git\n\nNow, I see the total size of the single abc.git has become 66 GB. Initially\nit was 34 GB, After using --max-pack-size=4g it become 48 GB. When we\nremove the --max-pack-size=4g parameter and tried to create a single pack\nfile now it become 66 GB.\n \nLooks like once we do git repack with multiple pack files, we can't revert\nback to the original size. \n\nThanks,\nSivakumar Selvam.\n"},{"id":"272291","messageId":"D8169FC37A314B82930152F9DE817682@PhilipOakley","threadId":"40642","inReplyTo":"xmqqziz6hzom.fsf@gitster.mtv.corp.google.com","subject":"Re: git repack command on larger pack file","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2015-10-27T05:25:41Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n> Junio C Hamano writes:\n>\n>> Sivakumar Selvam writes:\n>>\n>>> ... So\n>>> I thought of splitting the pack file into 4 GB chunks.\n>> ...\n>> Hmmm, what is \"this issue\"? I do not see anything surprising.\n>\n> While the explanation might have been enlightening, the knowledge\n> conveyed by the explanation by itself would not be of much practical\n> use, and enlightment without practical use is never fun.\n>\n> So let's do another tangent that may be more useful.\n>\n> In many repositories, older parts of the history often hold the bulk\n> of objects that do not change, and it is wasteful to repack them\n> over and over. If your project is at around v40.0 today, and it was\n> at around v36.0 6 months ago, for example, you may want to pack\n> everything that happened before v36.0 into a single pack just once,\n> pack them really well, and have your \"repack\" not touch that old\n> part of the history.\n>\n> $ git rev-list --objects v36.0 |\n> git pack-objects --window=200 --depth=128 pack\n>\n> would produce such a pack [*1*]\n>\n> The standard output from the above pipeline will give you a 40-hex\n> string (e.g. 51c472761b4690a331c02c90ec364e47cca1b3ac, call it\n> $HEX), and in the current directory you will find two files,\n> pack-$HEX.pack and pack-$HEX.idx.\n>\n> You can then do this:\n>\n> $ echo \"v36.0 with W/D 200/128\" >pack-$HEX.keep\n> $ mv pack-$HEX.* .git/objects/pack/.\n> $ git repack -a -d\n>\n> A pack that has an accompanying .keep file is excempt from\n> repacking, so once you do this, your future \"git repack\" will only\n> repack objects that are not in the kept packs.\n>\n\nI had a quick look at the man pages and couln't find an explanation (such as \nthis one) to explain the purpose, highlight the use of and how to create \nsuch .keep packs.\n\nCould this form the basis of a short section on .keep packs?(or did I miss \nsomething)\n\n>\n>\n> [Footnote]\n>\n> *1* I won't say 200/128 gives you a good pack; you would need to\n> experiment. In general, larger depth will result in smaller pack\n> but it will result in bigger overhead while you use the repository\n> every day. Larger window will spend a lot of cycles while packing,\n> but will result in a smaller pack.\n> --\n\nPhilip \n"},{"id":"272362","messageId":"20151027234406.GB4172@sigill.intra.peff.net","threadId":"40642","inReplyTo":"loom.20151027T025257-333@post.gmane.org","subject":"Re: git repack command on larger pack file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-10-27T23:44:06Z","receivedAt":"2015-10-27T23:44:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 27, 2015 at 02:04:23AM +0000, Sivakumar Selvam wrote:\n\n> When I finished git repacking, I found 12 pack files with each 4 GB and\n> the total size is 48 GB. Again I ran the same git repack command by just\n> removing only --max-pack-size= parameter, the size of the single pack file\n> is 66 GB.\n> \n> git repack -A -b -d -q --depth=50 --window=10 abc.git\n> \n> Now, I see the total size of the single abc.git has become 66 GB. Initially\n> it was 34 GB, After using --max-pack-size=4g it become 48 GB. When we\n> remove the --max-pack-size=4g parameter and tried to create a single pack\n> file now it become 66 GB.\n> \n> Looks like once we do git repack with multiple pack files, we can't revert\n> back to the original size.\n\nGit tries to take some shortcuts when repacking: if two objects are in\nthe same pack but not deltas, it will not consider making deltas out of\nthem. The logic is we would already have tried that while making the\noriginal pack. But of course when you are doing weird things with the\npacking parameters, that is not always a good assumption.\n\nWhen doing experiments like this, add \"-f\" to your repack command-line\nto avoid reusing deltas. The result should be much smaller (at the\nexpense of more CPU time to do the repack).\n\nI'd also recommend increasing \"--window\" if you can afford the extra CPU\nduring the repack. It can often produce smaller packs. And it has less\ncost than you might think (e.g., window=20 is not twice as expensive as\nwindow=10, because the work to access the objects is cached). You can\nalso increase --depth, but I have never found it to be particularly\nhelpful for decreasing size[1].\n\n-Peff\n\n[1] This is all theory, and I don't know how well git actually finds\n such deltas, but it is probably better to have a dense tree of\n deltas rather than long chains. If you have a chain of N objects\n and would to add object N+1 to it, you are probably not much worse\n off to base it on object N-1, creating a \"fork\" at N. The resulting\n objects should be less expensive to access for subsequent operations\n (as any time you want the Nth object, you have to resolve all parts\n of the chain, so shorter chains are better, and you the delta cache\n is more likely to get a hit on that N-1 object).\n"},{"id":"272363","messageId":"20151027234752.GC4172@sigill.intra.peff.net","threadId":"40642","inReplyTo":"xmqq611ujfn0.fsf@gitster.mtv.corp.google.com","subject":"Re: git repack command on larger pack file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-10-27T23:47:52Z","receivedAt":"2015-10-27T23:47:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Oct 25, 2015 at 11:41:23PM -0700, Junio C Hamano wrote:\n\n> Also it has to write more data to disk (see below), it has to find a\n> good place to split, it has to adjust bookkeeping data at the pack\n> boundary, in general it has to do more, not less, to produce split\n> packs. It would be surprising if it took less time.\n\nThis may go without saying, but the main cost in the write is that we\nhave to zlib deflate the output. I don't have any numbers at hand, but\nwhen I've benchmarked serving fetches, it is often a balance game\nbetween CPU time spent on a more aggressive delta search and CPU time\nthat goes into deflating the results of the search. Spending more CPU on\nthe former may yield more and smaller deltas which pay for themselves in\ntime spent on the latter.\n\nThere's definitely a balance point, and it varies from repo to repo, and\neven within repos from fetch to fetch. I wish I had better heuristics to\nreport, but it's an ongoing thing I'm exploring. :)\n\n-Peff\n"},{"id":"272395","messageId":"xmqqlhanmrz7.fsf@gitster.mtv.corp.google.com","threadId":"40642","inReplyTo":"20151027234406.GB4172@sigill.intra.peff.net","subject":"Re: git repack command on larger pack file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-28T06:23:24Z","receivedAt":"2015-10-28T06:23:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Git tries to take some shortcuts when repacking: if two objects are in\n> the same pack but not deltas, it will not consider making deltas out of\n> them. The logic is we would already have tried that while making the\n> original pack. But of course when you are doing weird things with the\n> packing parameters, that is not always a good assumption.\n\nYup, that is http://thread.gmane.org/gmane.comp.version-control.git/16223/focus=16267\n\n> [1] This is all theory, and I don't know how well git actually finds\n> such deltas, but it is probably better to have a dense tree of\n> deltas rather than long chains. If you have a chain of N objects\n> and would to add object N+1 to it, you are probably not much worse\n> off to base it on object N-1, creating a \"fork\" at N.\n\nYes, your guess is perfectly correct here, and indeed we did an\nextensive work along that line in 2006/2007. For an example, see\nhttp://thread.gmane.org/gmane.comp.version-control.git/51949/focus=52003\n\nThe histogram \"verify-pack -v\" produces was in fact done primarily\nin order to make it easy to check the distribution of delta depth.\n\nThanks.\n"},{"id":"272396","messageId":"xmqqh9lbmqve.fsf@gitster.mtv.corp.google.com","threadId":"40642","inReplyTo":"xmqqlhanmrz7.fsf@gitster.mtv.corp.google.com","subject":"Re: git repack command on larger pack file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-10-28T06:47:17Z","receivedAt":"2015-10-28T06:47:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> [1] This is all theory, and I don't know how well git actually finds\n>> such deltas, but it is probably better to have a dense tree of\n>> deltas rather than long chains. If you have a chain of N objects\n>> and would to add object N+1 to it, you are probably not much worse\n>> off to base it on object N-1, creating a \"fork\" at N.\n>\n> Yes, your guess is perfectly correct here, and indeed we did an\n> extensive work along that line in 2006/2007. For an example, see\n> http://thread.gmane.org/gmane.comp.version-control.git/51949/focus=52003\n\nAnd here is another, which is probably one of the most important\nthread on pack-objects, before the bitmap was introduced:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/20056/focus=20134\n"}]}