{"thread":{"id":"55128","subject":"[PATCH] doc: mention bigFileThreshold for packing","startedAt":"2021-02-09T19:16:49Z","lastAt":"2021-02-21T13:24:59Z","messageCount":5,"participants":["Christian Walther via GitGitGadget","Junio C Hamano","Christian Walther"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"416546","messageId":"pull.872.git.1612897624121.gitgitgadget@gmail.com","threadId":"55128","inReplyTo":null,"subject":"[PATCH] doc: mention bigFileThreshold for packing","fromName":"Christian Walther via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-09T19:07:03Z","receivedAt":"2021-02-09T19:16:49Z","isPatch":true,"sender":{"key":"cwalther@gmx.ch","avatar":"https://avatars.githubusercontent.com/u/234094?v=4"},"body":"From: Christian Walther \n\nKnowing about the core.bigFileThreshold configuration variable is\nhelpful when examining pack file size differences between repositories.\nAdd a reference to it to the manpages a user is likely to read in this\nsituation.\n\nSigned-off-by: Christian Walther \n---\n doc: mention bigFileThreshold for packing\n \n I recently spent a lot of time trying to figure out why git repack would\n create huge packs on some clones of my repository and small ones on\n others, until I found out about the existence of the\n core.bigFileThreshold configuration variable, which happened to be set\n on some and not on others. It would have saved me a lot of time if that\n variable had been mentioned in the relevant manpages that I was reading,\n git-repack and git-pack-objects. So this patch adds that.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-872%2Fcwalther%2Fdeltadoc-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-872/cwalther/deltadoc-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/872\n\n Documentation/git-pack-objects.txt | 4 ++++\n Documentation/git-repack.txt | 4 ++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 54d715ead137..59150ded4bef 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -97,6 +97,10 @@ base-name::\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n +\n+Note that delta compression is never used on objects larger than the\n+`core.bigFileThreshold` configuration variable (see\n+linkgit:git-config[1]).\n++\n The default value for --window is 10 and --depth is 50. The maximum\n depth is 4095.\n \ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 92f146d27dc3..0a7038ec4ad8 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -96,6 +96,10 @@ to the new separate pack will be written.\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n +\n+Note that delta compression is never used on objects larger than the\n+`core.bigFileThreshold` configuration variable (see\n+linkgit:git-config[1]).\n++\n The default value for --window is 10 and --depth is 50. The maximum\n depth is 4095.\n \n\nbase-commit: fb7fa4a1fd273f22efcafdd13c7f897814fd1eb9\n-- \ngitgitgadget\n"},{"id":"416586","messageId":"xmqqim70zzaf.fsf@gitster.c.googlers.com","threadId":"55128","inReplyTo":"pull.872.git.1612897624121.gitgitgadget@gmail.com","subject":"Re: [PATCH] doc: mention bigFileThreshold for packing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-09T21:50:16Z","receivedAt":"2021-02-09T23:12:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Christian Walther via GitGitGadget\" \nwrites:\n\n> From: Christian Walther \n>\n> Knowing about the core.bigFileThreshold configuration variable is\n> helpful when examining pack file size differences between repositories.\n> Add a reference to it to the manpages a user is likely to read in this\n> situation.\n\nThanks.\n\nI doubt that the description of --window/--depth command line\noptions, for both repack and pack-objects, is the best place to add\nthis \"Note\". Even if we were to add it as an appendix to these\nplaces, please do not break the flow of explanation by inserting it\nbefore the description of the default values of these options.\n\n> I recently spent a lot of time trying to figure out why git repack would\n> create huge packs on some clones of my repository and small ones on\n> others, until I found out about the existence of the\n> core.bigFileThreshold configuration variable, which happened to be set\n> on some and not on others. It would have saved me a lot of time if that\n> variable had been mentioned in the relevant manpages that I was reading,\n> git-repack and git-pack-objects. So this patch adds that.\n\nNot related to the contents of the patch, but I am somewhat curious\nto know what configuration resulted in the \"huge\" ones and \"small\"\nones. Documentation/config/core.txt::core.bigFileThreashold may be\nhelped by addition of a success story, and the configuration for the\n\"small\" ones may be a good place to start.\n\nThanks\n"},{"id":"416666","messageId":"F63929A8-7BC9-43A7-9E7B-118433F62588@gmx.ch","threadId":"55128","inReplyTo":"xmqqim70zzaf.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] doc: mention bigFileThreshold for packing","fromName":"Christian Walther","fromEmail":"cwalther@gmx.ch","sentAt":"2021-02-10T21:43:14Z","receivedAt":"2021-02-10T21:44:56Z","isPatch":true,"sender":{"key":"cwalther@gmx.ch","avatar":"https://avatars.githubusercontent.com/u/234094?v=4"},"body":"Junio C Hamano wrote:\n\n> I doubt that the description of --window/--depth command line\n> options, for both repack and pack-objects, is the best place to add\n> this \"Note\". Even if we were to add it as an appendix to these\n> places, please do not break the flow of explanation by inserting it\n> before the description of the default values of these options.\n\nOK. That was where I would have looked for it, because it explains why --window wasn't effective in my attempts to get better compression, but I don't insist on it - any place would have worked, as I read both manpages back and forth several times.\n\nIn git-repack.txt, there is a \"Configuration\" section at the bottom, I guess it would fit there? There is none in git-pack-objects.txt, but I could add it. What do you think?\n\n\n>> I recently spent a lot of time trying to figure out why git repack would\n>> create huge packs on some clones of my repository and small ones on\n>> others\n> \n> Not related to the contents of the patch, but I am somewhat curious\n> to know what configuration resulted in the \"huge\" ones and \"small\"\n> ones. Documentation/config/core.txt::core.bigFileThreashold may be\n> helped by addition of a success story, and the configuration for the\n> \"small\" ones may be a good place to start.\n\nThe \"huge\" repository had bigFileThreshold = 1m. That was set by SubGit when converting from Subversion, for reasons unknown to me (see some discussion at https://support.tmatesoft.com/t/reduce-repository-size/2551 and https://issues.tmatesoft.com/issue/SGT-604). The result is a pack file of about 3 GB.\n\nThe \"small\" repository has it unset, so the default 512m applies, resulting in a pack file of about 50 MB.\n\nWhat causes the huge difference is that the repository contains a \"changelog\" file that changes in almost every commit and has grown to 2.4 MB over 10000 commits. So it exists in about that many different versions, of which about 6000 are larger than 1 MB, but they only differ from each other by successive addition of small pieces.\n\nI'm not sure if that makes for a good success story. 1m seems a rather extreme value to me. If you think so, I can try to come up with something.\n\nThanks\n\n Christian\n\n"},{"id":"416670","messageId":"xmqq5z2z1s7v.fsf@gitster.c.googlers.com","threadId":"55128","inReplyTo":"F63929A8-7BC9-43A7-9E7B-118433F62588@gmx.ch","subject":"Re: [PATCH] doc: mention bigFileThreshold for packing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-10T22:19:16Z","receivedAt":"2021-02-10T22:20:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Walther writes:\n\n> Junio C Hamano wrote:\n>\n>> I doubt that the description of --window/--depth command line\n>> options, for both repack and pack-objects, is the best place to add\n>> this \"Note\". Even if we were to add it as an appendix to these\n>> places, please do not break the flow of explanation by inserting it\n>> before the description of the default values of these options.\n>\n> OK. That was where I would have looked for it, because it explains\n> why --window wasn't effective in my attempts to get better\n> compression, but I don't insist on it - any place would have\n> worked, as I read both manpages back and forth several times.\n\nThe \"pack-objects\" command (and to some degree \"repack\", too) is\nabout packing throughout, and --depth/--window is not necessarily\nthe central piece of the puzzle, and that, together with disruption\nof the flow of the original explanation, was the reason why I found\nthe initial location a bit odd.\n\n> In git-repack.txt, there is a \"Configuration\" section at the\n> bottom, I guess it would fit there? There is none in\n> git-pack-objects.txt, but I could add it. What do you think?\n\nYou're right---if there is an existing CONFIGURATION section, that\nmay be a much better place. There are configuration variables that\naffect how the packing works other than the core.bigFileThreshold,\nand attributes like \"delta\" would also affect the outcome.\n\nDescribing all in one CONFIGURATION section would be valuable.\n\nWhat I queued is with the following ready to be squashed in,\nprimarily because I was lazy and didn't have time/inclination to\nlook for a better place myself ;-)\n\nThanks.\n\n---- >8 ----\nSubject: [PATCH] fixup! doc: mention bigFileThreshold for packing\n\n---\n Documentation/git-pack-objects.txt | 7 +++----\n Documentation/git-repack.txt | 7 +++----\n 2 files changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 59150ded4b..be0f953c35 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -97,12 +97,11 @@ base-name::\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n +\n-Note that delta compression is never used on objects larger than the\n-`core.bigFileThreshold` configuration variable (see\n-linkgit:git-config[1]).\n-+\n The default value for --window is 10 and --depth is 50. The maximum\n depth is 4095.\n++\n+Note that delta compression is never used on objects larger than the\n+`core.bigFileThreshold` configuration variable (see linkgit:git-config[1]).\n \n --window-memory=::\n \tThis option provides an additional limit on top of `--window`;\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 0a7038ec4a..145fff6e01 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -96,12 +96,11 @@ to the new separate pack will be written.\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n +\n-Note that delta compression is never used on objects larger than the\n-`core.bigFileThreshold` configuration variable (see\n-linkgit:git-config[1]).\n-+\n The default value for --window is 10 and --depth is 50. The maximum\n depth is 4095.\n++\n+Note that delta compression is never used on objects larger than the\n+`core.bigFileThreshold` configuration variable (see linkgit:git-config[1]).\n \n --threads=::\n \tThis option is passed through to `git pack-objects`.\n-- \n2.30.1-597-g82b686dd6a\n\n"},{"id":"417455","messageId":"pull.872.v2.git.1613913838248.gitgitgadget@gmail.com","threadId":"55128","inReplyTo":"pull.872.git.1612897624121.gitgitgadget@gmail.com","subject":"[PATCH v2] doc: mention bigFileThreshold for packing","fromName":"Christian Walther via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-21T13:23:57Z","receivedAt":"2021-02-21T13:24:59Z","isPatch":true,"sender":{"key":"cwalther@gmx.ch","avatar":"https://avatars.githubusercontent.com/u/234094?v=4"},"body":"From: Christian Walther \n\nKnowing about the core.bigFileThreshold configuration variable is\nhelpful when examining pack file size differences between repositories.\nAdd a reference to it to the manpages a user is likely to read in this\nsituation.\n\nCapitalize CONFIGURATION for consistency with other pages having such a\nsection.\n\nSigned-off-by: Christian Walther \n---\n doc: mention bigFileThreshold for packing\n \n I recently spent a lot of time trying to figure out why git repack would\n create huge packs on some clones of my repository and small ones on\n others, until I found out about the existence of the\n core.bigFileThreshold configuration variable, which happened to be set\n on some and not on others. It would have saved me a lot of time if that\n variable had been mentioned in the relevant manpages that I was reading,\n git-repack and git-pack-objects. So this patch adds that.\n \n Changes in v2:\n \n * Move additions to the CONFIGURATION section at the bottom.\n * Reword a little after realizing that there are more configuration\n variables affecting packing.\n * Capitalize CONFIGURATION for consistency with other pages having such\n a section.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-872%2Fcwalther%2Fdeltadoc-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-872/cwalther/deltadoc-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/872\n\nRange-diff vs v1:\n\n 1: 20b9a56d94b7 < -: ------------ doc: mention bigFileThreshold for packing\n -: ------------ > 1: 027d1038fbb1 doc: mention bigFileThreshold for packing\n\n\n Documentation/git-pack-objects.txt | 11 +++++++++++\n Documentation/git-repack.txt | 9 ++++++++-\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 54d715ead137..f85cb7ea934c 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -400,6 +400,17 @@ Note that we pick a single island for each regex to go into, using \"last\n one wins\" ordering (which allows repo-specific config to take precedence\n over user-wide config, and so forth).\n \n+\n+CONFIGURATION\n+-------------\n+\n+Various configuration variables affect packing, see\n+linkgit:git-config[1] (search for \"pack\" and \"delta\").\n+\n+Notably, delta compression is not used on objects larger than the\n+`core.bigFileThreshold` configuration variable and on files with the\n+attribute `delta` set to false.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 92f146d27dc3..fbd4b4ae0677 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -165,9 +165,12 @@ depth is 4095.\n \tPass the `--delta-islands` option to `git-pack-objects`, see\n \tlinkgit:git-pack-objects[1].\n \n-Configuration\n+CONFIGURATION\n -------------\n \n+Various configuration variables affect packing, see\n+linkgit:git-config[1] (search for \"pack\" and \"delta\").\n+\n By default, the command passes `--delta-base-offset` option to\n 'git pack-objects'; this typically results in slightly smaller packs,\n but the generated packs are incompatible with versions of Git older than\n@@ -178,6 +181,10 @@ need to set the configuration variable `repack.UseDeltaBaseOffset` to\n is unaffected by this option as the conversion is performed on the fly\n as needed in that case.\n \n+Delta compression is not used on objects larger than the\n+`core.bigFileThreshold` configuration variable and on files with the\n+attribute `delta` set to false.\n+\n SEE ALSO\n --------\n linkgit:git-pack-objects[1]\n\nbase-commit: 2283e0e9af55689215afa39c03beb2315ce18e83\n-- \ngitgitgadget\n"}]}