{"thread":{"id":"8025","subject":"[PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","startedAt":"2007-05-08T02:54:34Z","lastAt":"2007-06-11T21:40:38Z","messageCount":40,"participants":["Theodore Ts'o","Nicolas Pitre","Theodore Tso","Dana How","Junio C Hamano","Johannes Schindelin","Daniel Barkalow","Steven Grimm","Shawn O. Pearce","Sam Vilain"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"41394","messageId":"E1HlFqU-0002ir-GK@candygram.thunk.org","threadId":"8025","inReplyTo":null,"subject":"[PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T02:54:34Z","receivedAt":"2007-05-08T02:54:34Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"Sometimes users might want to use more aggressive packing options\nwhen doing a git-gc. This allows them to do so without having\nto use the low-level plumbing commands.\n\nSigned-off-by: \"Theodore Ts'o\" \n---\n Documentation/git-gc.txt | 3 ++-\n Documentation/git-pack-objects.txt | 19 +------------------\n Documentation/pack-options.txt | 18 ++++++++++++++++++\n builtin-gc.c | 30 ++++++++++++++++++++++++++++--\n 4 files changed, 49 insertions(+), 21 deletions(-)\n create mode 100644 Documentation/pack-options.txt\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex bc16584..3c807c2 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n \n SYNOPSIS\n --------\n-'git-gc' [--prune]\n+'git-gc' [--prune] [--no-reuse-delta] [--window=N] [--depth=N]\n \n DESCRIPTION\n -----------\n@@ -35,6 +35,7 @@ OPTIONS\n \trepository at the same time (e.g. never use this option\n \tin a cron script).\n \n+include::pack-options.txt[]\n \n Configuration\n -------------\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex d9e11c6..aac71f6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -73,18 +73,6 @@ base-name::\n \tas if all refs under `$GIT_DIR/refs` are specified to be\n \tincluded.\n \n---window=[N], --depth=[N]::\n-\tThese two options affect how the objects contained in\n-\tthe pack are stored using delta compression. The\n-\tobjects are first internally sorted by type, size and\n-\toptionally names and compared against the other objects\n-\twithin --window to see if using delta compression saves\n-\tspace. --depth limits the maximum delta depth; making\n-\tit too deep affects the performance on the unpacker\n-\tside, because delta data needs to be applied that many\n-\ttimes to get to the necessary object.\n-\tThe default value for both --window and --depth is 10.\n-\n --incremental::\n \tThis flag causes an object already in a pack ignored\n \teven if it appears in the standard input.\n@@ -120,12 +108,7 @@ base-name::\n \tThis flag makes the command not to report its progress\n \ton the standard error stream.\n \n---no-reuse-delta::\n-\tWhen creating a packed archive in a repository that\n-\thas existing packs, the command reuses existing deltas.\n-\tThis sometimes results in a slightly suboptimal pack.\n-\tThis flag tells the command not to reuse existing deltas\n-\tbut compute them from scratch.\n+include::pack-options.txt[]\n \n --delta-base-offset::\n \tA packed archive can express base object of a delta as\ndiff --git a/Documentation/pack-options.txt b/Documentation/pack-options.txt\nnew file mode 100644\nindex 0000000..7b0ae5f\n--- /dev/null\n+++ b/Documentation/pack-options.txt\n@@ -0,0 +1,18 @@\n+--window=[N], --depth=[N]::\n+\tThese two options affect how the objects contained in\n+\tthe pack are stored using delta compression. The\n+\tobjects are first internally sorted by type, size and\n+\toptionally names and compared against the other objects\n+\twithin --window to see if using delta compression saves\n+\tspace. --depth limits the maximum delta depth; making\n+\tit too deep affects the performance on the unpacker\n+\tside, because delta data needs to be applied that many\n+\ttimes to get to the necessary object.\n+\tThe default value for both --window and --depth is 10.\n+\n+--no-reuse-delta::\n+\tWhen creating a packed archive in a repository that\n+\thas existing packs, the command reuses existing deltas.\n+\tThis sometimes results in a slightly suboptimal pack.\n+\tThis flag tells the command not to reuse existing deltas\n+\tbut compute them from scratch.\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 3b1f8c2..7e7775d 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -15,13 +15,15 @@\n \n #define FAILED_RUN \"failed to run %s\"\n \n-static const char builtin_gc_usage[] = \"git-gc [--prune]\";\n+static const char builtin_gc_usage[] = \"git-gc [--prune] [--no-reuse-delta] [--window=N] [--depth=N]\";\n \n static int pack_refs = -1;\n \n+#define MAX_ADD 10\n+\n static const char *argv_pack_refs[] = {\"pack-refs\", \"--prune\", NULL};\n static const char *argv_reflog[] = {\"reflog\", \"expire\", \"--all\", NULL};\n-static const char *argv_repack[] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n+static const char *argv_repack[MAX_ADD] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n static const char *argv_prune[] = {\"prune\", NULL};\n static const char *argv_rerere[] = {\"rerere\", \"gc\", NULL};\n \n@@ -37,6 +39,21 @@ static int gc_config(const char *var, const char *value)\n \treturn git_default_config(var, value);\n }\n \n+static append_option(const char **cmd, const char *opt, int max_length)\n+{\n+\tint\ti;\n+\n+\tfor (i=0; cmd[i]; i++)\n+\t\t;\n+\n+\tif (i+2 >= max_length) {\n+\t\tfprintf(stderr, \"Too many options specified\\n\");\n+\t\texit(1);\n+\t}\n+\tcmd[i++] = opt;\n+\tcmd[i] = 0;\n+}\n+\n int cmd_gc(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -53,6 +70,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tprune = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--no-reuse-delta\")) {\n+\t\t\tappend_option(argv_repack, \"-f\", MAX_ADD);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strncmp(arg, \"--window\", 8) ||\n+\t\t !strncmp(arg, \"--depth\", 7)) {\n+\t\t\tappend_option(argv_repack, arg, MAX_ADD);\n+\t\t\tcontinue;\n+\t\t}\n \t\t/* perhaps other parameters later... */\n \t\tbreak;\n \t}\n-- \n1.5.2.rc1.20.g86b9-dirty\n"},{"id":"41396","messageId":"alpine.LFD.0.99.0705072305270.24220@xanadu.home","threadId":"8025","inReplyTo":"E1HlFqU-0002ir-GK@candygram.thunk.org","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T03:13:58Z","receivedAt":"2007-05-08T03:13:58Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 7 May 2007, Theodore Ts'o wrote:\n\n> Sometimes users might want to use more aggressive packing options\n> when doing a git-gc. This allows them to do so without having\n> to use the low-level plumbing commands.\n\nThe 'git repack' command isn't _that_ low level, is it? \ngit-pack-objects is plumbing for sure, but not git-repack?\n\nEspecially if you're aware and interested in those options, you won't be \nafraid of 'git repack -a -f -d --window=...\".\n\nIn the context of \"gc\", having an option that reads \"window\" looks a bit \nstrange too.\n\nMaybe it's just me...\n\n\nNicolas\n"},{"id":"41399","messageId":"20070508032122.GA10940@thunk.org","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0705072305270.24220@xanadu.home","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T03:21:22Z","receivedAt":"2007-05-08T03:21:22Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, May 07, 2007 at 11:13:58PM -0400, Nicolas Pitre wrote:\n> On Mon, 7 May 2007, Theodore Ts'o wrote:\n> \n> > Sometimes users might want to use more aggressive packing options\n> > when doing a git-gc. This allows them to do so without having\n> > to use the low-level plumbing commands.\n> \n> The 'git repack' command isn't _that_ low level, is it? \n> git-pack-objects is plumbing for sure, but not git-repack?\n> \n> Especially if you're aware and interested in those options, you won't be \n> afraid of 'git repack -a -f -d --window=...\".\n> \n> In the context of \"gc\", having an option that reads \"window\" looks a bit \n> strange too.\n\nI suppose, but you either need to then know all of the other commands\nwhich git-gc runs, and do them manually, skipping git-gc altogether,\nor use git-gc, and end up rewriting the pack twice, ince using the\ngit-repack in git-gc, and then once manually so you can give the\noptions that you really want to give to git-repack.\n\nMaybe the right approach is to have a way to specify default --window\nand --depth as git configuration variables? Looks like there is a\npack.window already, but not a pack.depth.\n\nWhat if we add a pack.depth configuration option, and add only\n--no-reuse-delta to git-gc? Would that be better?\n\n\t\t\t\t\t\t- Ted\n"},{"id":"41403","messageId":"56b7f5510705072038y17509aaamaf8bca00bb53102@mail.gmail.com","threadId":"8025","inReplyTo":"20070508032122.GA10940@thunk.org","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-08T03:38:51Z","receivedAt":"2007-05-08T03:38:51Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/7/07, Theodore Tso wrote:\n> On Mon, May 07, 2007 at 11:13:58PM -0400, Nicolas Pitre wrote:\n> > On Mon, 7 May 2007, Theodore Ts'o wrote:\n> > > Sometimes users might want to use more aggressive packing options\n> > > when doing a git-gc. This allows them to do so without having\n> > > to use the low-level plumbing commands.\n> > In the context of \"gc\", having an option that reads \"window\" looks a bit\n> > strange too.\n> I suppose, but you either need to then know all of the other commands\n> which git-gc runs, and do them manually, skipping git-gc altogether,\n> or use git-gc, and end up rewriting the pack twice, ince using the\n> git-repack in git-gc, and then once manually so you can give the\n> options that you really want to give to git-repack.\n>\n> Maybe the right approach is to have a way to specify default --window\n> and --depth as git configuration variables? Looks like there is a\n> pack.window already, but not a pack.depth.\n>\n> What if we add a pack.depth configuration option, and add only\n> --no-reuse-delta to git-gc? Would that be better?\n\nI would use pack.depth .\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"41410","messageId":"7vr6ps3oyk.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"20070508032122.GA10940@thunk.org","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-08T04:43:15Z","receivedAt":"2007-05-08T04:43:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso writes:\n\n> On Mon, May 07, 2007 at 11:13:58PM -0400, Nicolas Pitre wrote:\n>> ... \n>> Especially if you're aware and interested in those options, you won't be \n>> afraid of 'git repack -a -f -d --window=...\".\n>> \n>> In the context of \"gc\", having an option that reads \"window\" looks a bit \n>> strange too.\n>\n> I suppose, but you either need to then know all of the other commands\n> which git-gc runs, and do them manually, skipping git-gc altogether,\n> or use git-gc, and end up rewriting the pack twice,...\n\nIf the user really wants to tweak the parameters that much and\nthat often, I think what Nico says, plus your pack.depth/window\nconfiguration variables, make more sense. git-gc is meant to be\na shorthand with reasonable \"one size fits all\" default, and\nthere is something wrong if a user has to give customization\noption every time it is run. It could be that the default\nparameters are grossly off for _everybody_, in which case we\nshould fix the default.\n\nWith the recent introduction of delta base caching code, we\nmight want to tweak the default pack depth to larger value for\neverybody, by the way.\n"},{"id":"41452","messageId":"11786309073709-git-send-email-tytso@mit.edu","threadId":"8025","inReplyTo":"7vr6ps3oyk.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T13:28:25Z","receivedAt":"2007-05-08T13:28:25Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"OK, here's a patch to implement pack.depth (with the default tweaked\nto 50 --- is that too high?), followed by a simplified and reworked\npatch to git-gc that only implements --no-reuse-delta.\n\nI don't imagine that most users will want to use that feature most of\nthe time, hence the long option name, but occasionally, it might be\nuseful. Yes, the user could just run \"git-repack -a -d -f -l\" after\nrunning git-gc, but then the \"git-repack -a -d -l\" in git-gc is just a\nwasted disk i/o. I don't know if I'll manage to convince you, if not,\njust drop the second patch, I guess. :-)\n\n\t\t\t\t\t\t - Ted\n"},{"id":"41450","messageId":"11786309072612-git-send-email-tytso@mit.edu","threadId":"8025","inReplyTo":"11786309073709-git-send-email-tytso@mit.edu","subject":"[PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T13:28:26Z","receivedAt":"2007-05-08T13:28:26Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"Signed-off-by: \"Theodore Ts'o\" \n---\n Documentation/config.txt | 6 +++++-\n Documentation/git-pack-objects.txt | 2 +-\n Documentation/git-repack.txt | 2 +-\n builtin-pack-objects.c | 6 +++++-\n 4 files changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 24f9655..c7674c2 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -544,7 +544,11 @@ merge..recursive::\n \n pack.window::\n \tThe size of the window used by gitlink:git-pack-objects[1] when no\n-\twindow size is given on the command line. Defaults to 10.\n+\twindow size is given on the command line. Defaults to 10.\n+\n+pack.depth::\n+\tThe maximum delta depth used by gitlink:git-pack-objects[1] when no\n+\tmaximum depth is given on the command line. Defaults to 50.\n \n pull.octopus::\n \tThe default merge strategy to use when pulling multiple branches\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex d9e11c6..bd3ee45 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -83,7 +83,7 @@ base-name::\n \tit too deep affects the performance on the unpacker\n \tside, because delta data needs to be applied that many\n \ttimes to get to the necessary object.\n-\tThe default value for both --window and --depth is 10.\n+\tThe default value for --window is 10 and --depth is 50.\n \n --incremental::\n \tThis flag causes an object already in a pack ignored\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex d39abc1..cc3b0b2 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -63,7 +63,7 @@ OPTIONS\n \tspace. `--depth` limits the maximum delta depth; making it too deep\n \taffects the performance on the unpacker side, because delta data needs\n \tto be applied that many times to get to the necessary object.\n-\tThe default value for both --window and --depth is 10.\n+\tThe default value for --window is 10 and --depth is 50.\n \n \n Configuration\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 7bff8ea..966f843 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -64,6 +64,7 @@ static char tmpname[PATH_MAX];\n static unsigned char pack_file_sha1[20];\n static int progress = 1;\n static int window = 10;\n+static int depth = 50;\n static int pack_to_stdout;\n static int num_preferred_base;\n static struct progress progress_state;\n@@ -1489,6 +1490,10 @@ static int git_pack_config(const char *k, const char *v)\n \t\twindow = git_config_int(k, v);\n \t\treturn 0;\n \t}\n+\tif(!strcmp(k, \"pack.depth\")) {\n+\t\tdepth = git_config_int(k, v);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v);\n }\n \n@@ -1584,7 +1589,6 @@ static int adjust_perm(const char *path, mode_t mode)\n \n int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n-\tint depth = 10;\n \tint use_internal_rev_list = 0;\n \tint thin = 0;\n \tuint32_t i;\n-- \n1.5.2.rc2.22.ga39d\n"},{"id":"41451","messageId":"11786309071033-git-send-email-tytso@mit.edu","threadId":"8025","inReplyTo":"11786309072612-git-send-email-tytso@mit.edu","subject":"[PATCH] Add --no-reuse-delta option to git-gc","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T13:28:27Z","receivedAt":"2007-05-08T13:28:27Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"This allows the user to regenerate the deltas in packs while doing\na git-gc. The user could just run git-repack -a -d -f -l after\nrunning git-gc, but then the first git-repack run by git-gc is\na bit of waste.\n\nSigned-off-by: \"Theodore Ts'o\" \n---\n Documentation/git-gc.txt | 7 ++++++-\n builtin-gc.c | 24 ++++++++++++++++++++++--\n 2 files changed, 28 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex bc16584..0493d06 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n \n SYNOPSIS\n --------\n-'git-gc' [--prune]\n+'git-gc' [--prune] [--no-reuse-delta]\n \n DESCRIPTION\n -----------\n@@ -35,6 +35,11 @@ OPTIONS\n \trepository at the same time (e.g. never use this option\n \tin a cron script).\n \n+--no-reuse-delta::\n+\tThis causes deltas in existing packs to be recalculated instead\n+\tof reusing the existing deltas. This can save disk space at\n+\tthe cost of taking more time to recalculate them from scratch.\n+\n \n Configuration\n -------------\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 3b1f8c2..5cb7ffd 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -15,13 +15,14 @@\n \n #define FAILED_RUN \"failed to run %s\"\n \n-static const char builtin_gc_usage[] = \"git-gc [--prune]\";\n+static const char builtin_gc_usage[] = \"git-gc [--prune] [--no-reuse-delta]\";\n \n static int pack_refs = -1;\n \n+#define MAX_ADD 10\n static const char *argv_pack_refs[] = {\"pack-refs\", \"--prune\", NULL};\n static const char *argv_reflog[] = {\"reflog\", \"expire\", \"--all\", NULL};\n-static const char *argv_repack[] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n+static const char *argv_repack[MAX_ADD] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n static const char *argv_prune[] = {\"prune\", NULL};\n static const char *argv_rerere[] = {\"rerere\", \"gc\", NULL};\n \n@@ -37,6 +38,21 @@ static int gc_config(const char *var, const char *value)\n \treturn git_default_config(var, value);\n }\n \n+static append_option(const char **cmd, const char *opt, int max_length)\n+{\n+\tint\ti;\n+\n+\tfor (i=0; cmd[i]; i++)\n+\t\t;\n+\n+\tif (i+2 >= max_length) {\n+\t\tfprintf(stderr, \"Too many options specified\\n\");\n+\t\texit(1);\n+\t}\n+\tcmd[i++] = opt;\n+\tcmd[i] = 0;\n+}\n+\n int cmd_gc(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -53,6 +69,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tprune = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--no-reuse-delta\")) {\n+\t\t\tappend_option(argv_repack, \"-f\", MAX_ADD);\n+\t\t\tcontinue;\n+\t\t}\n \t\t/* perhaps other parameters later... */\n \t\tbreak;\n \t}\n-- \n1.5.2.rc2.22.ga39d\n"},{"id":"41453","messageId":"alpine.LFD.0.99.0705080941350.24220@xanadu.home","threadId":"8025","inReplyTo":"7vr6ps3oyk.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to git-gc","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T13:46:38Z","receivedAt":"2007-05-08T13:46:38Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 7 May 2007, Junio C Hamano wrote:\n\n> Theodore Tso writes:\n> \n> > On Mon, May 07, 2007 at 11:13:58PM -0400, Nicolas Pitre wrote:\n> >> ... \n> >> Especially if you're aware and interested in those options, you won't be \n> >> afraid of 'git repack -a -f -d --window=...\".\n> >> \n> >> In the context of \"gc\", having an option that reads \"window\" looks a bit \n> >> strange too.\n> >\n> > I suppose, but you either need to then know all of the other commands\n> > which git-gc runs, and do them manually, skipping git-gc altogether,\n> > or use git-gc, and end up rewriting the pack twice,...\n> \n> If the user really wants to tweak the parameters that much and\n> that often, I think what Nico says, plus your pack.depth/window\n> configuration variables, make more sense.\n\nAlso, once you've run git-repack with -f and the window/depth params you \nwant, those deltas will be reused as is by default afterwards even if \nyou use git-gc. So the big repack to tighten a large repo needs only to \nbe done once.\n\n\nNicolas\n"},{"id":"41465","messageId":"alpine.LFD.0.99.0705081005400.24220@xanadu.home","threadId":"8025","inReplyTo":"11786309073709-git-send-email-tytso@mit.edu","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T15:30:31Z","receivedAt":"2007-05-08T15:30:31Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2007, Theodore Ts'o wrote:\n\n> OK, here's a patch to implement pack.depth (with the default tweaked\n> to 50 --- is that too high?),\n\nThis is most likely to affect runtime performances, but some benchmarks \nwould be needed to find out how much.\n\nHaving both pack.depth and pack.window as config options (with current \ndefaults of 10) would certainly be a good thing. Tweaking those \ndefaults should probably be investigated separately.\n\n> followed by a simplified and reworked\n> patch to git-gc that only implements --no-reuse-delta.\n> \n> I don't imagine that most users will want to use that feature most of\n> the time, hence the long option name, but occasionally, it might be\n> useful. Yes, the user could just run \"git-repack -a -d -f -l\" after\n> running git-gc, but then the \"git-repack -a -d -l\" in git-gc is just a\n> wasted disk i/o.\n\nIn which case, it is git-gc that needs to get a bit smarter. Maybe \nsomething like this:\n\n----- >*\nAvoid running git-repack from git-gc if there is evidently nothing to \nrepack.\n\nSigned-off-by: Nicolas Pitre \n---\n\ndiff --git a/builtin-count-objects.c b/builtin-count-objects.c\nindex ff90ebd..8d79764 100644\n--- a/builtin-count-objects.c\n+++ b/builtin-count-objects.c\n@@ -67,13 +67,40 @@ static void count_objects(DIR *d, char *path, int len, int verbose,\n \t}\n }\n \n-int cmd_count_objects(int ac, const char **av, const char *prefix)\n+static void count_loose_objects(int verbose,\n+\t\t\t\tunsigned long *loose,\n+\t\t\t\tunsigned long *loose_size,\n+\t\t\t\tunsigned long *packed_loose,\n+\t\t\t\tunsigned long *garbage)\n {\n-\tint i;\n-\tint verbose = 0;\n \tconst char *objdir = get_object_directory();\n-\tint len = strlen(objdir);\n+\tint i, len = strlen(objdir);\n \tchar *path = xmalloc(len + 50);\n+\tmemcpy(path, objdir, len);\n+\tif (len && objdir[len-1] != '/')\n+\t\tpath[len++] = '/';\n+\tfor (i = 0; i < 256; i++) {\n+\t\tDIR *d;\n+\t\tsprintf(path + len, \"%02x\", i);\n+\t\td = opendir(path);\n+\t\tif (!d)\n+\t\t\tcontinue;\n+\t\tcount_objects(d, path, len, verbose,\n+\t\t\t loose, loose_size, packed_loose, garbage);\n+\t\tclosedir(d);\n+\t}\n+}\n+\n+unsigned long num_loose_objects(void)\n+{\n+\tunsigned long loose = 0, packed_loose = 0, garbage = 0, loose_size = 0;\n+\tcount_loose_objects(0, &loose, &loose_size, &packed_loose, &garbage);\n+\treturn loose;\n+}\n+\n+int cmd_count_objects(int ac, const char **av, const char *prefix)\n+{\n+\tint i, verbose = 0;\n \tunsigned long loose = 0, packed = 0, packed_loose = 0, garbage = 0;\n \tunsigned long loose_size = 0;\n \n@@ -90,19 +117,8 @@ int cmd_count_objects(int ac, const char **av, const char *prefix)\n \t/* we do not take arguments other than flags for now */\n \tif (i < ac)\n \t\tusage(count_objects_usage);\n-\tmemcpy(path, objdir, len);\n-\tif (len && objdir[len-1] != '/')\n-\t\tpath[len++] = '/';\n-\tfor (i = 0; i < 256; i++) {\n-\t\tDIR *d;\n-\t\tsprintf(path + len, \"%02x\", i);\n-\t\td = opendir(path);\n-\t\tif (!d)\n-\t\t\tcontinue;\n-\t\tcount_objects(d, path, len, verbose,\n-\t\t\t &loose, &loose_size, &packed_loose, &garbage);\n-\t\tclosedir(d);\n-\t}\n+\n+\tcount_loose_objects(verbose, &loose, &loose_size, &packed_loose, &garbage);\n \tif (verbose) {\n \t\tstruct packed_git *p;\n \t\tunsigned long num_pack = 0;\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 3b1f8c2..b9b3c05 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -40,7 +40,7 @@ static int gc_config(const char *var, const char *value)\n int cmd_gc(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tint prune = 0;\n+\tint prune = 0, do_repack = 0;\n \n \tgit_config(gc_config);\n \n@@ -65,7 +65,20 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (run_command_v_opt(argv_reflog, RUN_GIT_CMD))\n \t\treturn error(FAILED_RUN, argv_reflog[0]);\n \n-\tif (run_command_v_opt(argv_repack, RUN_GIT_CMD))\n+\tif (num_loose_objects() > 0) {\n+\t\tdo_repack = 1;\n+\t} else {\n+\t\tstruct packed_git *p;\n+\t\tunsigned long num_pack = 0;\n+\t\tif (!packed_git)\n+\t\t\tprepare_packed_git();\n+\t\tfor (p = packed_git; p; p = p->next)\n+\t\t\tif (p->pack_local)\n+\t\t\t\tnum_pack++;\n+\t\tif (num_pack > 1)\n+\t\t\tdo_repack = 1;\n+\t}\n+\tif (do_repack && run_command_v_opt(argv_repack, RUN_GIT_CMD))\n \t\treturn error(FAILED_RUN, argv_repack[0]);\n \n \tif (prune && run_command_v_opt(argv_prune, RUN_GIT_CMD))\ndiff --git a/cache.h b/cache.h\nindex 8e76152..3a140f1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -359,6 +359,8 @@ extern int legacy_loose_object(unsigned char *);\n extern int has_pack_file(const unsigned char *sha1);\n extern int has_pack_index(const unsigned char *sha1);\n \n+extern unsigned long num_loose_objects(void);\n+\n extern signed char hexval_table[256];\n static inline unsigned int hexval(unsigned int c)\n {\n"},{"id":"41466","messageId":"alpine.LFD.0.99.0705081131440.24220@xanadu.home","threadId":"8025","inReplyTo":"11786309071033-git-send-email-tytso@mit.edu","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T15:35:17Z","receivedAt":"2007-05-08T15:35:17Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2007, Theodore Ts'o wrote:\n\n> This allows the user to regenerate the deltas in packs while doing\n> a git-gc. The user could just run git-repack -a -d -f -l after\n> running git-gc, but then the first git-repack run by git-gc is\n> a bit of waste.\n\nGiven the patch I just posted and my previous arguments I don't think \nthis patch is in the spirit of git-gc. I'd prefer if git-gc remained as \nsimple with the least options as possible, but I don't care that much \neither.\n\n\nNicolas\n"},{"id":"41468","messageId":"alpine.LFD.0.99.0705081136050.24220@xanadu.home","threadId":"8025","inReplyTo":"11786309072612-git-send-email-tytso@mit.edu","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T15:38:46Z","receivedAt":"2007-05-08T15:38:46Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2007, Theodore Ts'o wrote:\n\n> Signed-off-by: \"Theodore Ts'o\" \n\nI'd prefer if tests were performed on the performance impact before \nchanging the default depth. If done separately from this patch then the \ncommit log could contain those results as well.\n\n\nNicolas\n"},{"id":"41474","messageId":"20070508163044.GA15796@thunk.org","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0705081136050.24220@xanadu.home","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T16:30:44Z","receivedAt":"2007-05-08T16:30:44Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Tue, May 08, 2007 at 11:38:46AM -0400, Nicolas Pitre wrote:\n> On Tue, 8 May 2007, Theodore Ts'o wrote:\n> \n> > Signed-off-by: \"Theodore Ts'o\" \n> \n> I'd prefer if tests were performed on the performance impact before \n> changing the default depth. If done separately from this patch then the \n> commit log could contain those results as well.\n\nThe following results are on a recent git repository, using time to\nrecord the real, user, and sys times on the two commands: \"git-gc\n--no-reuse-delta --window=X --depth=Y\" and \"git log --pretty=oneline\n-S'object' > /dev/null\". All of these tests were done with a hot\ncache, so disk speed didn't enter into the calculations.\n\n git-gc git log -S'object' pack size\nw=10,d=10 27.1s/25.2s/0.3s 20.8s/20.4s/0.1s 15292k\nw=10,d=30\t23.8s/22.3s/0.2s 21.2s/20.9s/0.1s 12996k\nw=10,d=50 24.8s/22.4s/0.4s 21.8s/21.2s/0.1s 12340k\nw=100,d=100 24.1s/22.8s/0.3s 22.4s/21.8s/0.2s 11772k\n\nw=30,d=10 45.0s/43.1s/0.4s 20.8s/20.5s/0.1s 14388k\nw=30,d=30 35.8s/34.1s/0.3s 21.6s/21.1s/0.1s 11800k\nw=30,d=50 34.6s/33.0s/0.3s 22.1s/21.4s/0.1s 11376k\nw=30,d=100 34.0s/32.2s/0.3s 22.2s/21.6s/0.1s 11012k\n\nw=50,d=10 56.1s/54.3s/0.4s 21.3s/20.5s/0.1s 14224k\nw=50,d=30 47.2s/45.4s/0.4s 21.6s/21.0s/0.1s 11496k \nw=50,d=50 44.5s/43.0s/0.3s 21.7s/21.2s/0.1s 11108k\nw=50,d=100 44.3s/42.7s/0.4s 22.4s/21.7s/0.1s 10824k\n\nSo a couple of things immediately become evident. First of all, as\nJunio predicted, changing --depth makes no difference to the git-gc or\ngit log times. The latter is thanks to the delta chaching. Secondly,\nchanging --depth does make a signficiant difference to the pack size.\n\nFinally, --window does help somewhat in reducing the pack size, but it\n_significantly_ increases the time to calculate the pack.\n\nMy conclusion given this quick benchmark is that it seems to me that\nchanging the defaults of --depth to 50, and keeping --window at 10, is\na reasonable thing to do.\n\nRegards,\n\n\t\t\t\t\t\t- Ted\n"},{"id":"41475","messageId":"Pine.LNX.4.64.0705081848330.4167@racer.site","threadId":"8025","inReplyTo":"20070508163044.GA15796@thunk.org","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-05-08T16:49:47Z","receivedAt":"2007-05-08T16:49:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 8 May 2007, Theodore Tso wrote:\n\n> On Tue, May 08, 2007 at 11:38:46AM -0400, Nicolas Pitre wrote:\n> > On Tue, 8 May 2007, Theodore Ts'o wrote:\n> > \n> > > Signed-off-by: \"Theodore Ts'o\" \n> > \n> > I'd prefer if tests were performed on the performance impact before \n> > changing the default depth. If done separately from this patch then the \n> > commit log could contain those results as well.\n> \n> The following results are on a recent git repository, using time to\n> record the real, user, and sys times on the two commands: \"git-gc\n> --no-reuse-delta --window=X --depth=Y\" and \"git log --pretty=oneline\n> -S'object' > /dev/null\". All of these tests were done with a hot\n> cache, so disk speed didn't enter into the calculations.\n> \n> git-gc git log -S'object' pack size\n> w=10,d=10 27.1s/25.2s/0.3s 20.8s/20.4s/0.1s 15292k\n> w=10,d=30\t23.8s/22.3s/0.2s 21.2s/20.9s/0.1s 12996k\n> w=10,d=50 24.8s/22.4s/0.4s 21.8s/21.2s/0.1s 12340k\n> w=100,d=100 24.1s/22.8s/0.3s 22.4s/21.8s/0.2s 11772k\n> \n> w=30,d=10 45.0s/43.1s/0.4s 20.8s/20.5s/0.1s 14388k\n> w=30,d=30 35.8s/34.1s/0.3s 21.6s/21.1s/0.1s 11800k\n> w=30,d=50 34.6s/33.0s/0.3s 22.1s/21.4s/0.1s 11376k\n> w=30,d=100 34.0s/32.2s/0.3s 22.2s/21.6s/0.1s 11012k\n> \n> w=50,d=10 56.1s/54.3s/0.4s 21.3s/20.5s/0.1s 14224k\n> w=50,d=30 47.2s/45.4s/0.4s 21.6s/21.0s/0.1s 11496k \n> w=50,d=50 44.5s/43.0s/0.3s 21.7s/21.2s/0.1s 11108k\n> w=50,d=100 44.3s/42.7s/0.4s 22.4s/21.7s/0.1s 10824k\n> \n> So a couple of things immediately become evident. First of all, as\n> Junio predicted, changing --depth makes no difference to the git-gc or\n> git log times. The latter is thanks to the delta chaching. Secondly,\n> changing --depth does make a signficiant difference to the pack size.\n> \n> Finally, --window does help somewhat in reducing the pack size, but it\n> _significantly_ increases the time to calculate the pack.\n> \n> My conclusion given this quick benchmark is that it seems to me that\n> changing the defaults of --depth to 50, and keeping --window at 10, is\n> a reasonable thing to do.\n\nI'd be happier if that test was done on _at least_ the kernel repo, if not \nsomething larger, _plus_ having the numbers on page faults. Swapping can \nkill performance substantially...\n\ngit.git is small.\n\nCiao,\nDscho\n"},{"id":"41476","messageId":"56b7f5510705081007v4840ff7cma754a75eb3d3377f@mail.gmail.com","threadId":"8025","inReplyTo":"20070508163044.GA15796@thunk.org","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-08T17:07:44Z","receivedAt":"2007-05-08T17:07:44Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 5/8/07, Theodore Tso wrote:\n> The following results are on a recent git repository, using time to\n> record the real, user, and sys times on the two commands: \"git-gc\n> --no-reuse-delta --window=X --depth=Y\" and \"git log --pretty=oneline\n> -S'object' > /dev/null\". All of these tests were done with a hot\n> cache, so disk speed didn't enter into the calculations.\n>\n> ...\n>\n> So a couple of things immediately become evident. First of all, as\n> Junio predicted, changing --depth makes no difference to the git-gc or\n> git log times. The latter is thanks to the delta chaching. Secondly,\n> changing --depth does make a signficiant difference to the pack size.\n>\n> Finally, --window does help somewhat in reducing the pack size, but it\n> _significantly_ increases the time to calculate the pack.\n>\n> My conclusion given this quick benchmark is that it seems to me that\n> changing the defaults of --depth to 50, and keeping --window at 10, is\n> a reasonable thing to do.\n\nIf you still have the packfiles around, the times for some non-pickaxe\ngit-log commands would be interesting, like from git-log's man page:\n git log v2.6.12.. include/scsi drivers/scsi\n git log --since=\"2 weeks ago\" -- gitk\nThese operations would be more dominated by processing smaller objects.\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"41478","messageId":"alpine.LFD.0.99.0705081323040.24220@xanadu.home","threadId":"8025","inReplyTo":"20070508163044.GA15796@thunk.org","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T17:35:40Z","receivedAt":"2007-05-08T17:35:40Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2007, Theodore Tso wrote:\n\n> On Tue, May 08, 2007 at 11:38:46AM -0400, Nicolas Pitre wrote:\n> > On Tue, 8 May 2007, Theodore Ts'o wrote:\n> > \n> > > Signed-off-by: \"Theodore Ts'o\" \n> > \n> > I'd prefer if tests were performed on the performance impact before \n> > changing the default depth. If done separately from this patch then the \n> > commit log could contain those results as well.\n> \n> The following results are on a recent git repository, using time to\n> record the real, user, and sys times on the two commands: \"git-gc\n> --no-reuse-delta --window=X --depth=Y\" and \"git log --pretty=oneline\n> -S'object' > /dev/null\". All of these tests were done with a hot\n> cache, so disk speed didn't enter into the calculations.\n[...]\n> My conclusion given this quick benchmark is that it seems to me that\n> changing the defaults of --depth to 50, and keeping --window at 10, is\n> a reasonable thing to do.\n\nEffectively.\n\nI'd still prefer to see the default changed in a patch of its own \nthough.\n\n\nNicolas\n"},{"id":"41481","messageId":"20070508180947.GB15796@thunk.org","threadId":"8025","inReplyTo":"Pine.LNX.4.64.0705081848330.4167@racer.site","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-05-08T18:09:47Z","receivedAt":"2007-05-08T18:09:47Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Tue, May 08, 2007 at 06:49:47PM +0200, Johannes Schindelin wrote:\n> I'd be happier if that test was done on _at least_ the kernel repo, if not \n> something larger, _plus_ having the numbers on page faults. Swapping can \n> kill performance substantially...\n\nGiven how the delta cache works, I really don't think it's going to\nmatter. In any case, my laptop has 2gigs of memory, and the kernel\npack file is only 134megs, so you're not going to see any major page\nfaults....\n\nIn any case, here is a quick run:\n\n git-gc git-log -S'object' \n real/user/sys/min.faults real/user/sys/min.faults pack size\nw=10,d=10 4:31/257.7/6.2/391711 5:53/326.9/1.7/255156 155940k\nw=10,d=30 4:16/242.7/6.5/378193 5:39/331.6/2.3/437283 143144k\nw=10,d=50 4:29/250.1/6.7/554493 5:43/334.5/1.9/362574 140080k\n\nYou'll note that it's the same thing; git-gc, git-log doesn't change\nmuch, while the pack size decreases as --depth increases. We're only\nseeing at 10% decrease in the pack size, compared to the 20% decrease\nwith the git repository, but that's probably because of the HTML and\nman branches, which no doubt delta compress really, really well.\n\nI can run a full set of benchmarks, varying both --window and --depth,\nand also including a non-pickaxe git-log test as requested, but not\nuntil tonight. I really don't think we'll see any surprises compared\nto the earlier runs, though.\n\nAfter all, if we just stop and think about how the delta caching\nworks, and how the repacking algorithm works, it's pretty clear that\nthere shouldn't be any scaling issues with increasing --depth, and\nthat increasing --window is just going to be painful, and these should\nhold true regardless of the size of the repo.\n\n\t\t\t\t\t- Ted\n"},{"id":"41482","messageId":"alpine.LFD.0.99.0705081439140.24220@xanadu.home","threadId":"8025","inReplyTo":"20070508180947.GB15796@thunk.org","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T18:46:35Z","receivedAt":"2007-05-08T18:46:35Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2007, Theodore Tso wrote:\n\n> After all, if we just stop and think about how the delta caching\n> works, and how the repacking algorithm works, it's pretty clear that\n> there shouldn't be any scaling issues with increasing --depth, and\n> that increasing --window is just going to be painful, and these should\n> hold true regardless of the size of the repo.\n\nThe window size has absolutely no effect on the runtime pack access, \nexcept maybe for the increased number of deltas. It is really a pack \ntime cost. The delta depth is the opposite: it has no effect on the \npacking time, but it has the potential to slow down runtime access. But \nthe delta base cache is apparently working really well to mitigate that \ncost, as long as it is big enough of course.\n\n\nNicolas\n"},{"id":"41503","messageId":"7vwszj10kn.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0705081005400.24220@xanadu.home","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-08T21:12:56Z","receivedAt":"2007-05-08T21:12:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> In which case, it is git-gc that needs to get a bit smarter. Maybe \n> something like this:\n\nI agree git-gc should be tuned to \"one size fits all well enough\"\ndefault, rather than getting more complicated parameters to fine\ntune its behaviour to satisfy power users.\n\n> @@ -65,7 +65,20 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \tif (run_command_v_opt(argv_reflog, RUN_GIT_CMD))\n> \t\treturn error(FAILED_RUN, argv_reflog[0]);\n> \n> -\tif (run_command_v_opt(argv_repack, RUN_GIT_CMD))\n> +\tif (num_loose_objects() > 0) {\n> +\t\tdo_repack = 1;\n> +\t} else {\n> +\t\tstruct packed_git *p;\n> +\t\tunsigned long num_pack = 0;\n> +\t\tif (!packed_git)\n> +\t\t\tprepare_packed_git();\n> +\t\tfor (p = packed_git; p; p = p->next)\n> +\t\t\tif (p->pack_local)\n> +\t\t\t\tnum_pack++;\n> +\t\tif (num_pack > 1)\n> +\t\t\tdo_repack = 1;\n> +\t}\n> +\tif (do_repack && run_command_v_opt(argv_repack, RUN_GIT_CMD))\n> \t\treturn error(FAILED_RUN, argv_repack[0]);\n> \n> \tif (prune && run_command_v_opt(argv_prune, RUN_GIT_CMD))\n\nIs this even correct?\n\nWhen your repository is fully packed, if you decided to discard\none of your topic branches with \"git branch -D\", what does this\ncode do? We see no loose objects, we see only one pack, so the\nunreachable objects are left in the pack?\n"},{"id":"41517","messageId":"alpine.LFD.0.99.0705081958240.24220@xanadu.home","threadId":"8025","inReplyTo":"7vwszj10kn.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add --no-reuse-delta, --window, and --depth options to","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-05-08T23:59:53Z","receivedAt":"2007-05-08T23:59:53Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 8 May 2007, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > @@ -65,7 +65,20 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> > \tif (run_command_v_opt(argv_reflog, RUN_GIT_CMD))\n> > \t\treturn error(FAILED_RUN, argv_reflog[0]);\n> > \n> > -\tif (run_command_v_opt(argv_repack, RUN_GIT_CMD))\n> > +\tif (num_loose_objects() > 0) {\n> > +\t\tdo_repack = 1;\n> > +\t} else {\n> > +\t\tstruct packed_git *p;\n> > +\t\tunsigned long num_pack = 0;\n> > +\t\tif (!packed_git)\n> > +\t\t\tprepare_packed_git();\n> > +\t\tfor (p = packed_git; p; p = p->next)\n> > +\t\t\tif (p->pack_local)\n> > +\t\t\t\tnum_pack++;\n> > +\t\tif (num_pack > 1)\n> > +\t\t\tdo_repack = 1;\n> > +\t}\n> > +\tif (do_repack && run_command_v_opt(argv_repack, RUN_GIT_CMD))\n> > \t\treturn error(FAILED_RUN, argv_repack[0]);\n> > \n> > \tif (prune && run_command_v_opt(argv_prune, RUN_GIT_CMD))\n> \n> Is this even correct?\n> \n> When your repository is fully packed, if you decided to discard\n> one of your topic branches with \"git branch -D\", what does this\n> code do? We see no loose objects, we see only one pack, so the\n> unreachable objects are left in the pack?\n\nRight. OK, scrap that.\n\n\nNicolas\n"},{"id":"41536","messageId":"7v8xbyzizh.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0705081323040.24220@xanadu.home","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-09T05:03:30Z","receivedAt":"2007-05-09T05:03:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> On Tue, 8 May 2007, Theodore Tso wrote:\n>\n>> On Tue, May 08, 2007 at 11:38:46AM -0400, Nicolas Pitre wrote:\n>> > On Tue, 8 May 2007, Theodore Ts'o wrote:\n>> > \n>> > > Signed-off-by: \"Theodore Ts'o\" \n>> > \n>> > I'd prefer if tests were performed on the performance impact before \n>> > changing the default depth. If done separately from this patch then the \n>> > commit log could contain those results as well.\n>> \n>> The following results are on a recent git repository, using time to\n>> record the real, user, and sys times on the two commands: \"git-gc\n>> --no-reuse-delta --window=X --depth=Y\" and \"git log --pretty=oneline\n>> -S'object' > /dev/null\". All of these tests were done with a hot\n>> cache, so disk speed didn't enter into the calculations.\n> [...]\n>> My conclusion given this quick benchmark is that it seems to me that\n>> changing the defaults of --depth to 50, and keeping --window at 10, is\n>> a reasonable thing to do.\n>\n> Effectively.\n>\n> I'd still prefer to see the default changed in a patch of its own \n> though.\n\nI'll split the patch into two and apply them separately.\n"},{"id":"41538","messageId":"Pine.LNX.4.64.0705090056231.18541@iabervon.org","threadId":"8025","inReplyTo":"11786309071033-git-send-email-tytso@mit.edu","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2007-05-09T05:05:37Z","receivedAt":"2007-05-09T05:05:37Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 8 May 2007, Theodore Ts'o wrote:\n\n> This allows the user to regenerate the deltas in packs while doing\n> a git-gc. The user could just run git-repack -a -d -f -l after\n> running git-gc, but then the first git-repack run by git-gc is\n> a bit of waste.\n\nMaybe git-gc should have an option for \"compress hard\"? It seems to me \nlike a two-sizes-fit-all solution would be good here; \"git gc\" for daily \nuse, and \"git gc --squeeze\" for when you want to make the result as small \nas possible, with compute time not being a major factor. If all you know \nis that you're going to burn this repository onto a stack of CDs and mail \nit to somebody (but don't know about delta reuse, window sizes, and \ndepths, let alone good values for these), it would be good to have an \noption where it picks an appropriate different set of defaults for you.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"41553","messageId":"7v3b26xvjo.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"Pine.LNX.4.64.0705090056231.18541@iabervon.org","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-09T08:15:07Z","receivedAt":"2007-05-09T08:15:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow writes:\n\n> On Tue, 8 May 2007, Theodore Ts'o wrote:\n>\n>> This allows the user to regenerate the deltas in packs while doing\n>> a git-gc. The user could just run git-repack -a -d -f -l after\n>> running git-gc, but then the first git-repack run by git-gc is\n>> a bit of waste.\n>\n> Maybe git-gc should have an option for \"compress hard\"? It seems to me \n> like a two-sizes-fit-all solution would be good here; \"git gc\" for daily \n> use, and \"git gc --squeeze\" for when you want to make the result as small \n> as possible, with compute time not being a major factor.\n\nI think that sounds saner and more user friendly than specific\nknob to tune \"window\", \"depth\" and friends which are too\ntechnical. It has an added attraction that we can redefine what\nexactly \"hard\" means later.\n"},{"id":"41560","messageId":"46418E24.9020309@midwinter.com","threadId":"8025","inReplyTo":"7v3b26xvjo.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-05-09T09:02:28Z","receivedAt":"2007-05-09T09:02:28Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> I think that sounds saner and more user friendly than specific\n> knob to tune \"window\", \"depth\" and friends which are too\n> technical. It has an added attraction that we can redefine what\n> exactly \"hard\" means later.\n> \n\nOn that note, has any thought been given to looking at other compression \nalgorithms? Gzip is a great high-speed compressor, but there are others \nout there (some a bit slower, some much slower at both compression and \ndecompression) that produce substantially smaller output.\n\nOne could even, if one were in a particularly twisted state of mind, \nenvision using CPU-intensive compression for less frequently-accessed \nobjects and using gzip for active ones, on the theory that the best \ntime/space tradeoff is not uniform across all the objects in a git \nrepository. Presumably most of us never actually unpack the vast \nmajority of objects in a git repository of reasonable age, so the fact \nthat it'd take a little longer if we *did* want to unpack them isn't \nmuch of a downside compared to the upside of reclaiming disk space. That \nwould mitigate the impact of using an algorithm that's slow at \ndecompression.\n\nI think it'd be kind of neat to have my .git directory shrink by another \n20+%. That's conservative; on maximumcompression.com's test of a mix of \ndifferent file types including images, gzip compresses 64% and the \nbest-scoring one does 80%. On English text gzip does 71% and the top \nscorer does 89%. Most of the top-tier compressors are proprietary, but \nthere are some open-source ones that do pretty well.\n\nMaybe not worth the added complexity, but I thought I'd toss it out \nthere. It probably makes more sense (if it makes any at all) after \nLinus's suggestion to not unpack after cloning is in place. Once the \nupstream has gone to the trouble of CPU-intensive compressing, you \ncertainly don't want to force clones to have to spend the time repeating \nthe same work.\n\n-Steve (who suspects this is a \"yes, we talked this over early in git's \nhistory\" question, but what the heck)\n"},{"id":"41573","messageId":"Pine.LNX.4.64.0705091333570.4167@racer.site","threadId":"8025","inReplyTo":"46418E24.9020309@midwinter.com","subject":"Re: Other compression?, was Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-05-09T11:35:06Z","receivedAt":"2007-05-09T11:35:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 9 May 2007, Steven Grimm wrote:\n\n> On that note, has any thought been given to looking at other compression\n> algorithms?\n\nI think you could do that. But you would lose compatibility with all \nexisting Git clients. IOW nobody could pull from you.\n\nCiao,\nDscho\n"},{"id":"41591","messageId":"20070509134958.GA21489@thunk.org","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0705081439140.24220@xanadu.home","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-05-09T13:49:58Z","receivedAt":"2007-05-09T13:49:58Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Tue, May 08, 2007 at 02:46:35PM -0400, Nicolas Pitre wrote:\n> The window size has absolutely no effect on the runtime pack access, \n> except maybe for the increased number of deltas. It is really a pack \n> time cost. The delta depth is the opposite: it has no effect on the \n> packing time, but it has the potential to slow down runtime access. But \n> the delta base cache is apparently working really well to mitigate that \n> cost, as long as it is big enough of course.\n\nExactly, and the numbers bear out with the theory. Junio's already\napplied the change to up the default to 50, but for the record, here\nare the results using a kernel git repository, with times for doing a\ngit-gc (with the --no-reuse-delta, although I like the suggestion to\nchange it to just be \"--hard\", or maybe \"--agressive\"), and with a git\npickaxe and a git-log with pathname restrictions. It essentially\nconfirms that the delta base cache is doing the job just fine, up to\neven a depth of 100. However, there isn't much difference in pack\nsize between a depth of 50 and 100.\n\nIncreasing the window size from 10 to 30 increases the pack run time\nby roughly 40%, and saves an extra 5% or so on the pack size.\nIncreasing the window beyond 30 has ever-smaller diminishing returns,\nwhile the time to repack gets bigger and bigger.\n\nThe timing information is real/user/system/minor pagefaults, and as\nbefore, the results are done using a hot cache. Apologies in advance\nfor the long lines in the benchmark results. These results were\ngenerated using a relatively recent, post-2.6.21 kernel git tree on a\nThinkpad T60p laptop with a 2GHz T2500 Intel Core Duo processor.\n\nRegards,\n\n git-gc pack size git-log -S'object' git-log include/scsi drivers/scsi\nw=10,d=10 5:08.70/293.20/5.56/1027802 162316k 5:31.12/322.20/4.14/153298 0:01.80/1.72/0.03/19077\nw=10,d=30 4:14.79/245.46/2.57/398754 149636k 5:41.56/331.21/4.86/517220 0:01.83/1.74/0.05/17880\nw=10,d=50 4:31.89/257.16/3.50/576538 146608k 5:51.58/336.63/5.07/259643 0:01.88/1.80/0.04/17658\nw=10,d=100 4:35.93/262.77/4.08/715195 144152k 5:58.23/341.26/6.22/624510 0:01.89/1.80/0.04/17571\n\nw=30,d=10 7:23.06/424.08/6.35/1222640 153752k 5:32.85/323.96/2.13/213150 0:01.73/1.64/0.06/18429\nw=30,d=30 6:15.76/364.27/3.53/407546 141200k 5:42.29/333.53/2.10/338301 0:01.81/1.71/0.04/17237\nw=30,d=50 6:26.85/372.24/5.38/578343 139408k 5:42.21/336.65/1.30/260234 0:01.77/1.70/0.05/17108\nw=30,d=100 6:34.31/381.72/5.40/744886 138040k 5:59.03/342.65/4.90/607681 0:01.93/1.79/0.06/17043\n\nw=50,d=10 8:51.08/508.80/5.75/1050358 152168k 5:35.43/327.48/3.70/209655 0:01.70/1.64/0.04/18264\nw=50,d=30 8:04.53/471.65/5.58/423755 139000k 5:42.06/335.67/1.55/335333 0:01.78/1.72/0.04/17046\nw=50,d=50 8:14.87/479.93/7.09/617310 137244k 5:47.22/339.26/2.73/462412 0:01.79/1.72/0.04/16871\nw=50,d=100 8:23.13/490.09/4.10/751742 136152k 5:51.10/343.89/2.29/503573 0:01.83/1.73/0.06/16920\n\nw=100,d=10 12:00.34/702.07/6.27/1167403 150736k 5:34.42/328.88/1.09/207522 0:01.72/1.63/0.07/18024\nw=100,d=30 11:56.34/702.36/2.93/436950 137240k 5:42.64/335.82/2.07/422743 0:01.77/1.67/0.05/16852\nw=100,d=50 12:20.39/722.99/3.94/671655 135488k 5:47.20/339.38/2.44/468974 0:01.78/1.68/0.03/16621\nw=100,d=100 12:37.69/740.63/4.12/733593 134492k 5:52.26/344.16/3.00/636120 0:01.85/1.80/0.03/16681\n\n\t\t\t\t\t\t\t- Ted\n"},{"id":"41594","messageId":"Pine.LNX.4.64.0705091617060.4167@racer.site","threadId":"8025","inReplyTo":"20070509134958.GA21489@thunk.org","subject":"Re: [PATCH] Add pack.depth option to git-pack-objects and change default depth to 50","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-05-09T14:17:34Z","receivedAt":"2007-05-09T14:17:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 9 May 2007, Theodore Tso wrote:\n\n> [...] here are the results using a kernel git repository [...]\n\nThank you very much,\nDscho\n"},{"id":"41600","messageId":"7vtzumuiyk.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"46418E24.9020309@midwinter.com","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-09T15:15:15Z","receivedAt":"2007-05-09T15:15:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Steven Grimm writes:\n\n> -Steve (who suspects this is a \"yes, we talked this over early in\n> git's history\" question, but what the heck)\n\nAFAICR, the answer is \"heck, we have never bothered about such\nlow level implementation details as gzip has been good enough\nfor us. If you feel so inclined, go wild.\"\n"},{"id":"41644","messageId":"20070509191052.GD3141@spearce.org","threadId":"8025","inReplyTo":"46418E24.9020309@midwinter.com","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-05-09T19:10:52Z","receivedAt":"2007-05-09T19:10:52Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Steven Grimm wrote:\n> On that note, has any thought been given to looking at other compression \n> algorithms? Gzip is a great high-speed compressor, but there are others \n> out there (some a bit slower, some much slower at both compression and \n> decompression) that produce substantially smaller output.\n\nIts been discussed once before on the list, in very recent history,\nbut not by a whole lot. As Junio pointed out, I don't think there\never really was any discussion of is gzip the best way to deflate the\nobjects. I think gzip was just chosen simply because it was readily\navailable in libz, stable, and has a pretty decent speed/size ratio.\n \n> I think it'd be kind of neat to have my .git directory shrink by another \n> 20+%. That's conservative; on maximumcompression.com's test of a mix of \n> different file types including images, gzip compresses 64% and the \n> best-scoring one does 80%. On English text gzip does 71% and the top \n> scorer does 89%. Most of the top-tier compressors are proprietary, but \n\nYes. But in many cases we might actually be able to do even better\nby going with a pack-wide dictionary. Why?\n\nThink about source code structure. E.g.\n\n $ git grep --cached 'struct object'| cut -d: -f1|wc -l\n 402\n\nSo 402 files in git.git use the term 'struct object', and that's just\nthe current revision I had in my index. With our current packfile\norganization we are likely to store this string at least 402 times.\nWe'll store it once in each file's delta chain, assuming each\nfile's blobs largely fall into a single delta chain for that file\n(reasonable assumption, but certainly not always true).\n\nThat's just one string that does appear somewhat frequently in any\nfile its used in. Now try 'unsigned char' (its 944 files, but an\neven higher frequency-per-file).\n\nSo anyway, for the past year I've been thinking about trying to\nimplement a blob-level dictionary prototype to see if it helps on a\nproject like linux-2.6.git, but I haven't gotten to it. The pack v4\nwork was about applying that basic dicationary principal to trees\nand commits, and I think it pays off nicely there. Just need to\nget it cleaned up, rebased onto current master, and submitted to\nthe list for wider testing. ;-)\n\n-- \nShawn.\n"},{"id":"41647","messageId":"20070509194839.GB10280@thunk.org","threadId":"8025","inReplyTo":"7v3b26xvjo.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] Add --aggressive option to 'git gc'","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-05-09T19:48:39Z","receivedAt":"2007-05-09T19:48:39Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, May 09, 2007 at 01:15:07AM -0700, Junio C Hamano wrote:\n> > Maybe git-gc should have an option for \"compress hard\"? It seems to me \n> > like a two-sizes-fit-all solution would be good here; \"git gc\" for daily \n> > use, and \"git gc --squeeze\" for when you want to make the result as small \n> > as possible, with compute time not being a major factor.\n> \n> I think that sounds saner and more user friendly than specific\n> knob to tune \"window\", \"depth\" and friends which are too\n> technical. It has an added attraction that we can redefine what\n> exactly \"hard\" means later.\n\nOK, here's a patch that does exactly that. I choose git-gc\n--aggressive, since I thought that was more descriptive than --hard or\n--squeeze. Junio, would you be willing to apply this?\n\n\t\t\t\t\t- Ted\n\n=== Cut here ===\n\nAdd --aggressive option to 'git gc'\n\nThis option causes 'git gc' to more aggressively optimize the\nrepository at the cost of taking much more wall clock and CPU time.\n\nToday this option causes git-pack-objects to use --no-use-delta\noption, and it allows the --window parameter to be set via the\ngc.aggressiveWindow configuration parameter.\n\nSigned-off-by: \"Theodore Ts'o\" \n---\n Documentation/config.txt | 5 +++++\n Documentation/git-gc.txt | 16 +++++++++++++++-\n builtin-gc.c | 35 +++++++++++++++++++++++++++++++++--\n 3 files changed, 53 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ea434af..efcf301 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -384,6 +384,11 @@ format.suffix::\n \t`.patch`. Use this variable to change that suffix (make sure to\n \tinclude the dot if you want it).\n \n+gc.aggressiveWindow::\n+\tThe window size parameter used in the delta compression\n+\talgorithm used by 'git gc --aggressive'. This defaults\n+\tto 10.\n+\n gc.packrefs::\n \t`git gc` does not run `git pack-refs` in a bare repository by\n \tdefault so that older dumb-transport clients can still fetch\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex bc16584..56575e8 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n \n SYNOPSIS\n --------\n-'git-gc' [--prune]\n+'git-gc' [--prune] [--aggressive]\n \n DESCRIPTION\n -----------\n@@ -35,6 +35,13 @@ OPTIONS\n \trepository at the same time (e.g. never use this option\n \tin a cron script).\n \n+--aggressive::\n+\tUsually 'git-gc' runs very quickly while providing good disk\n+\tspace utilization and performance. This option will cause\n+\tgit-gc to more aggressive optimize the repository at the expense\n+\tof taking much more time. The effects of this optimization are\n+\tpersistent, so this option only needs to be sporadically; every\n+\tfew hundred changesets or so.\n \n Configuration\n -------------\n@@ -67,6 +74,13 @@ The optional configuration variable 'gc.packrefs' determines if\n is not run in bare repositories by default, to allow older dumb-transport\n clients fetch from the repository, but this will change in the future.\n \n+The optional configuration variable 'gc.aggressiveWindow' controls how\n+much time is spent optimizing the delta compression of the objects in\n+the repository when the --aggressive option is specified. The larger\n+the value, the more time is spent optimizing the delta compression. See\n+the documentation for the --window' option in gitlink:git-repack[1] for\n+more details. This defaults to 10.\n+\n See Also\n --------\n gitlink:git-prune[1]\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 3b1f8c2..10f92f1 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -15,13 +15,15 @@\n \n #define FAILED_RUN \"failed to run %s\"\n \n-static const char builtin_gc_usage[] = \"git-gc [--prune]\";\n+static const char builtin_gc_usage[] = \"git-gc [--prune] [--aggressive]\";\n \n static int pack_refs = -1;\n+static int aggressive_window = -1;\n \n+#define MAX_ADD 10\n static const char *argv_pack_refs[] = {\"pack-refs\", \"--prune\", NULL};\n static const char *argv_reflog[] = {\"reflog\", \"expire\", \"--all\", NULL};\n-static const char *argv_repack[] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n+static const char *argv_repack[MAX_ADD] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n static const char *argv_prune[] = {\"prune\", NULL};\n static const char *argv_rerere[] = {\"rerere\", \"gc\", NULL};\n \n@@ -34,13 +36,34 @@ static int gc_config(const char *var, const char *value)\n \t\t\tpack_refs = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"gc.aggressiveWindow\")) {\n+\t\taggressive_window = git_config_int(var, value);\n+\t\tprintf(\"aggressive_window = %d\\n\", aggressive_window);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value);\n }\n \n+static append_option(const char **cmd, const char *opt, int max_length)\n+{\n+\tint\ti;\n+\n+\tfor (i=0; cmd[i]; i++)\n+\t\t;\n+\n+\tif (i+2 >= max_length) {\n+\t\tfprintf(stderr, \"Too many options specified\\n\");\n+\t\texit(1);\n+\t}\n+\tcmd[i++] = opt;\n+\tcmd[i] = 0;\n+}\n+\n int cmd_gc(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tint prune = 0;\n+\tchar buf[80];\n \n \tgit_config(gc_config);\n \n@@ -53,6 +76,14 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tprune = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--aggressive\")) {\n+\t\t\tappend_option(argv_repack, \"-f\", MAX_ADD);\n+\t\t\tif (aggressive_window > 0) {\n+\t\t\t\tsprintf(buf, \"--window=%d\", aggressive_window);\n+\t\t\t\tappend_option(argv_repack, buf, MAX_ADD);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n \t\t/* perhaps other parameters later... */\n \t\tbreak;\n \t}\n-- \n1.5.2.rc2.22.ga39d\n"},{"id":"41656","messageId":"7v4pmlu4ut.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"20070509194839.GB10280@thunk.org","subject":"Re: [PATCH] Add --aggressive option to 'git gc'","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-09T20:19:54Z","receivedAt":"2007-05-09T20:19:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso writes:\n\n> On Wed, May 09, 2007 at 01:15:07AM -0700, Junio C Hamano wrote:\n>> > Maybe git-gc should have an option for \"compress hard\"? It seems to me \n>> > like a two-sizes-fit-all solution would be good here; \"git gc\" for daily \n>> > use, and \"git gc --squeeze\" for when you want to make the result as small \n>> > as possible, with compute time not being a major factor.\n>> \n>> I think that sounds saner and more user friendly than specific\n>> knob to tune \"window\", \"depth\" and friends which are too\n>> technical. It has an added attraction that we can redefine what\n>> exactly \"hard\" means later.\n>\n> OK, here's a patch that does exactly that. I choose git-gc\n> --aggressive, since I thought that was more descriptive than --hard or\n> --squeeze. Junio, would you be willing to apply this?\n\nWilling? Yes.\n\nIt's tricky that it defaults to 10 and still called aggressive.\nWhen the configuration variable is left unspecified, the only\nreason it is called aggressive is because it passes '-f' to\nrepack, right? It was not very clear at the first sight and I\nwas about to ask why the default is 10, not higher.\n"},{"id":"41676","messageId":"20070509222221.GA10322@thunk.org","threadId":"8025","inReplyTo":"7v4pmlu4ut.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add --aggressive option to 'git gc'","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-05-09T22:22:21Z","receivedAt":"2007-05-09T22:22:21Z","isPatch":true,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, May 09, 2007 at 01:19:54PM -0700, Junio C Hamano wrote:\n> It's tricky that it defaults to 10 and still called aggressive.\n> When the configuration variable is left unspecified, the only\n> reason it is called aggressive is because it passes '-f' to\n> repack, right? It was not very clear at the first sight and I\n> was about to ask why the default is 10, not higher.\n\nYes, it's called aggressive because it causes git-gc to recalculate\nthe delta's. So that means that git-gc --aggressive will take around\n9-10 times longer than git-gc. If the user changes\ngc.aggressive-window to some larger value, say like 30, then git-gc\n--aggressive would take around 20 times longer than git-gc.\n\nSo that's why I didn't make the default bigger, but instead left it as\nsomething which could be configured by the user. Maybe the default\nshould be larger; I don't strong opposition towards making it be more\nlike 30, but it seemed to me that doubling the run time for a 5%\ndecrease in pack size wasn't worth it.\n\nRegards,\n\n\t\t\t\t\t- Ted\n"},{"id":"41711","messageId":"7vzm4dku0f.fsf@assigned-by-dhcp.cox.net","threadId":"8025","inReplyTo":"20070509194839.GB10280@thunk.org","subject":"Re: [PATCH] Add --aggressive option to 'git gc'","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-10T07:38:56Z","receivedAt":"2007-05-10T07:38:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso writes:\n\n> Junio, would you be willing to apply this?\n\nYes, but ;-).\n\n> static int pack_refs = -1;\n> +static int aggressive_window = -1;\n> \n> +#define MAX_ADD 10\n> static const char *argv_pack_refs[] = {\"pack-refs\", \"--prune\", NULL};\n> static const char *argv_reflog[] = {\"reflog\", \"expire\", \"--all\", NULL};\n> -static const char *argv_repack[] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n> +static const char *argv_repack[MAX_ADD] = {\"repack\", \"-a\", \"-d\", \"-l\", NULL};\n> static const char *argv_prune[] = {\"prune\", NULL};\n> static const char *argv_rerere[] = {\"rerere\", \"gc\", NULL};\n> \n> @@ -34,13 +36,34 @@ static int gc_config(const char *var, const char *value)\n> \t\t\tpack_refs = git_config_bool(var, value);\n> \t\treturn 0;\n> \t}\n> +\tif (!strcmp(var, \"gc.aggressiveWindow\")) {\n\nCallbacks to git_config() are called with variable names\ndowncased (except for the 2nd level for 3-level variables. E.g.\n[REMOTE \"Foo\"] URL = ...; becomes var = \"remote.Foo.url\", val =\n...).\n\n> +\t\taggressive_window = git_config_int(var, value);\n> +\t\tprintf(\"aggressive_window = %d\\n\", aggressive_window);\n\nDid you mean to leave this in? Looks like a debug remnant...\n\n> +\t\treturn 0;\n> +\t}\n> \treturn git_default_config(var, value);\n> }\n> \n> +static append_option(const char **cmd, const char *opt, int max_length)\n\nType is \"static void\" I presume.\n\n> +{\n> +\tint\ti;\n\nFunny tab here.\n\n> +\n> +\tfor (i=0; cmd[i]; i++)\n> +\t\t;\n\nSP around operator '='.\n\n> +\n> +\tif (i+2 >= max_length) {\n\nSame for '+'\n\n> +\t\tfprintf(stderr, \"Too many options specified\\n\");\n> +\t\texit(1);\n\ndie(\"Too many...specified\"); /* note the lack of \\n at the end */\n\n> +\t}\n> +\tcmd[i++] = opt;\n> +\tcmd[i] = 0;\n\nWe tend to spell out NULL, although we all are aware that C says\nliteral '0' is the null pointer.\n\nAll fixups are trivial so I'd take the patch and amend locally.\n"},{"id":"44535","messageId":"466BAAD0.9060408@vilain.net","threadId":"8025","inReplyTo":"20070509191052.GD3141@spearce.org","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-06-10T07:40:00Z","receivedAt":"2007-06-10T07:40:00Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n>> On that note, has any thought been given to looking at other compression \n>> algorithms? Gzip is a great high-speed compressor, but there are others \n>> out there (some a bit slower, some much slower at both compression and \n>> decompression) that produce substantially smaller output.\n>> \n> Its been discussed once before on the list, in very recent history,\n> but not by a whole lot. As Junio pointed out, I don't think there\n> ever really was any discussion of is gzip the best way to deflate the\n> objects. I think gzip was just chosen simply because it was readily\n> available in libz, stable, and has a pretty decent speed/size ratio.\n> \n\nI think it's the right tool. I just don't see any point in changing to\nanything slower for the sake of 20% space saving. Especially bzip2.\n\nConsider this.\n\nCompression works primarily through two things: huffman coding and\nstring matching. The larger the window for your string matching, the\nslower the compression and the more memory you need thrashing your CPU\nmemory cache when decompressing.\n\nNow I'm not an expert on compression algorithms but I think a large part\nof the reason gzip is blindingly faster than bzip2 is because gzip uses\na 64k buffer and bzip2 a 900k one. Only now are CPUs getting caches\nlarge enough to deal with that size of buffer, the rest of the time\nyou're waiting for your RAM. Moore's law was supposed to make bzip2 fast\none of these days but I'm still waiting.\n\nBut with git-repack the window is effectively the size of your\nrepository. So that blows bzip2 out of the water. Why else can git make\ncompressed packs smaller than a .bz2 of the raw files? This is the same\nobservation Shawn makes with the pack-wide dictionary, but he sounds\nlike he wants to apply it to the huffman coding stage as well as the\ncurrent delta/string matching stage. Now that would be interesting...\n\nAnyway it's a free world so be my guest to implement it, I guess if this\nwas selectable it would only be a minor annoyance waiting a bit longer\npulling from from some repositories, and it would be interesting to see\nif it did make a big difference with pack file sizes.\n\nSam\n"},{"id":"44684","messageId":"alpine.LFD.0.99.0706102144080.12885@xanadu.home","threadId":"8025","inReplyTo":"466BAAD0.9060408@vilain.net","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-11T01:51:03Z","receivedAt":"2007-06-11T01:51:03Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 10 Jun 2007, Sam Vilain wrote:\n\n> Now I'm not an expert on compression algorithms but I think a large part\n> of the reason gzip is blindingly faster than bzip2 is because gzip uses\n> a 64k buffer and bzip2 a 900k one. Only now are CPUs getting caches\n> large enough to deal with that size of buffer, the rest of the time\n> you're waiting for your RAM. Moore's law was supposed to make bzip2 fast\n> one of these days but I'm still waiting.\n> \n> Anyway it's a free world so be my guest to implement it, I guess if this\n> was selectable it would only be a minor annoyance waiting a bit longer\n> pulling from from some repositories, and it would be interesting to see\n> if it did make a big difference with pack file sizes.\n\nIt won't happen for a simple reason: to be backward compatible with \nolder GIT clients. If you have your repo compressed with bzip2 and an \nold client pulls it then the server would have to decompress and \nrecompress everything with gzip. If instead your repo remains with gzip \nand a new client asks for bzip2 then you have to recompress as well \n(slow). So in practice it is best to remain with a single compression \nmethod.\n\n\nNicolas\n"},{"id":"44694","messageId":"466CE998.9000706@midwinter.com","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0706102144080.12885@xanadu.home","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2007-06-11T06:20:08Z","receivedAt":"2007-06-11T06:20:08Z","isPatch":true,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n>\n> It won't happen for a simple reason: to be backward compatible with \n> older GIT clients. If you have your repo compressed with bzip2 and an \n> old client pulls it then the server would have to decompress and \n> recompress everything with gzip. If instead your repo remains with gzip \n> and a new client asks for bzip2 then you have to recompress as well \n> (slow). So in practice it is best to remain with a single compression \n> method.\n> \n\nNot that I really think this is all that important (my original question \nwas more out of curiosity than anything) but I don't think those are \nreally issues.\n\nEven with a better compression scheme, nobody would want to remove gzip \nsupport; if you are creating a repo that needs to be compatible with old \nclients -- and of course not all repositories need that -- you'd just \nconfigure it to use gzip (or more likely, *not* configure it to use the \nother method.) There are already options whose documentation says, \n\"Don't enable this if you want your repo to be accessed by git clients \nolder than version X.\" In other words, git already has an established \napproach for adding new non-backward-compatible features. Some projects, \ne.g. internal corporate ones, can mandate that everyone upgrade their \nclients, and more importantly, once any new git feature has been out for \na long enough time, even public projects can reasonably say, \"You need \nversion X to access our repo.\" If alternate compression were introduced \ntoday, in five years would anyone care that there'd be some ancient, \nancient clients that couldn't use it?\n\nAnd since the hypothetical new client would have support for both \ncompression types, pulling from a gzip-based repo could be accomplished \ntotally transparently; you could, one assumes, even pull from a gzip \nrepo and pack locally using the other scheme if you felt like it.\n\n-Steve\n"},{"id":"44696","messageId":"20070611063118.GF6073@spearce.org","threadId":"8025","inReplyTo":"466CE998.9000706@midwinter.com","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-06-11T06:31:18Z","receivedAt":"2007-06-11T06:31:18Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Steven Grimm wrote:\n> Nicolas Pitre wrote:\n> >\n> >It won't happen for a simple reason: to be backward compatible with \n> >older GIT clients. If you have your repo compressed with bzip2 and an \n> >old client pulls it then the server would have to decompress and \n> >recompress everything with gzip. If instead your repo remains with gzip \n> >and a new client asks for bzip2 then you have to recompress as well \n> >(slow). So in practice it is best to remain with a single compression \n> >method.\n> > \n> \n> Not that I really think this is all that important (my original question \n> was more out of curiosity than anything) but I don't think those are \n> really issues.\n...\n> And since the hypothetical new client would have support for both \n> compression types, pulling from a gzip-based repo could be accomplished \n> totally transparently; you could, one assumes, even pull from a gzip \n> repo and pack locally using the other scheme if you felt like it.\n\nRight, I agree with Steven here.\n\nPack v4 (which is rapidly becoming vaporware it seems) uses such\na radically different encoding for its commit and tree objects\nthat you cannot send them as-is to a client unless that client\nalso understands pack v4. Consequently you need to decompress and\nrecompress those objects when talking to an older peer; this isn't\nvery different from a switch to bzip2.\n\nThat said, I think switching to a different generic compressor isn't\nthat interesting. We can probably do a lot better by organizing\nfiles into clusters, where those clusters have large number of\nsymbols in common, and then compress everything in the same cluster\nwith the same pack-wide dictionary.\n\nSpecifically I'm thinking that you may be able to cluster the *.h/*.c\ninto one cluster, and the Makefile into another, and the .gitignore\ninto a third cluster, and then leverage the large commonalities\nbetween those blobs when building a dictionary and compressing them.\nNo, I haven't prototyped it out yet.\n\n-- \nShawn.\n"},{"id":"44714","messageId":"Pine.LNX.4.64.0706111109430.4059@racer.site","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0706102144080.12885@xanadu.home","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-11T10:20:57Z","receivedAt":"2007-06-11T10:20:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 10 Jun 2007, Nicolas Pitre wrote:\n\n> On Sun, 10 Jun 2007, Sam Vilain wrote:\n> \n> > Anyway it's a free world so be my guest to implement it, I guess if \n> > this was selectable it would only be a minor annoyance waiting a bit \n> > longer pulling from from some repositories, and it would be \n> > interesting to see if it did make a big difference with pack file \n> > sizes.\n> \n> It won't happen for a simple reason: to be backward compatible with \n> older GIT clients. If you have your repo compressed with bzip2 and an \n> old client pulls it then the server would have to decompress and \n> recompress everything with gzip. If instead your repo remains with gzip \n> and a new client asks for bzip2 then you have to recompress as well \n> (slow). So in practice it is best to remain with a single compression \n> method.\n\nWith the extension mechanism we have in place, the client can send what \nkind of compression it supports, and the server can actually refuse to \nsend anything if it does not want to recompress.\n\nWhat I am trying to say: you do not necessarily have to allow every client \nto access that particular repository. I agree that mixed-compression repos \nare evil, but nothing stands in the way of a flag allowing (or \ndisallowing) recompression in a different format when fetching.\n\nSo if you should decide someday to track data with Git (remember: Generic \nInformation Tracker, not just source code), that is particularly unfit for \ncompression with gzip, but that you _need_ to store in a different \ncompressed manner, you can set up a repository which will _only_ _ever_ \nuse that compression.\n\nOf course, you'd need to prepare Git for that, but I could imagine \nsomething like a music library, which stores everything as ogg encoded \nsnippets. It might even use some perception-based hash on small chunks of \nthe music, and store the music as tree objects, which concatenate the \nsmall chunks. I might even try to do this for fun, some day in the distant \nfuture.\n\nIt's a wild world,\nDscho\n"},{"id":"44726","messageId":"alpine.LFD.0.99.0706110930170.12885@xanadu.home","threadId":"8025","inReplyTo":"Pine.LNX.4.64.0706111109430.4059@racer.site","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-11T14:01:18Z","receivedAt":"2007-06-11T14:01:18Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 11 Jun 2007, Johannes Schindelin wrote:\n\n> Hi,\n> \n> On Sun, 10 Jun 2007, Nicolas Pitre wrote:\n> \n> > On Sun, 10 Jun 2007, Sam Vilain wrote:\n> > \n> > > Anyway it's a free world so be my guest to implement it, I guess if \n> > > this was selectable it would only be a minor annoyance waiting a bit \n> > > longer pulling from from some repositories, and it would be \n> > > interesting to see if it did make a big difference with pack file \n> > > sizes.\n> > \n> > It won't happen for a simple reason: to be backward compatible with \n> > older GIT clients. If you have your repo compressed with bzip2 and an \n> > old client pulls it then the server would have to decompress and \n> > recompress everything with gzip. If instead your repo remains with gzip \n> > and a new client asks for bzip2 then you have to recompress as well \n> > (slow). So in practice it is best to remain with a single compression \n> > method.\n> \n> With the extension mechanism we have in place, the client can send what \n> kind of compression it supports, and the server can actually refuse to \n> send anything if it does not want to recompress.\n> \n> What I am trying to say: you do not necessarily have to allow every client \n> to access that particular repository. I agree that mixed-compression repos \n> are evil, but nothing stands in the way of a flag allowing (or \n> disallowing) recompression in a different format when fetching.\n\nI know.\n\nBut is it worthwhile? I think not.\n\nHowever I won't stand in the way of anyone who wants to try and provide \nnumbers. I just don't believe this is worthwhile and am not inclined to \ndo it.\n\nOK... Well, I just performed a really quick test:\n\n$ mkdir test-bzip2\n$ mkdir test-gzip\n$ cp git/*.[cho] test-bzip2\n$ cp git/*.[cho] test-gzip\n$ bzip2 test-bzip2/*\n$ gzip test-gzip/*\n$ du -s test-bzip2 test-gzip\n5016 test-bzip2\n4956 test-gzip\n\nIt is true that bzip2 is better with large files, but we typically have \nvery few of them in a Git repo, and in the presence of large files bzip2 \nthen becomes _much_ slower than gzip. So, given that the nature of Git \nobjects are likely to be small in 98% of the cases due to deltas, it \nappears that bzip2 won't be a gain at all but rather a waste, making a \npoor case for supporting it forever afterwards.\n\n> So if you should decide someday to track data with Git (remember: Generic \n> Information Tracker, not just source code),\n\nBah... if you please.\n\n> that is particularly unfit for \n> compression with gzip, but that you _need_ to store in a different \n> compressed manner, you can set up a repository which will _only_ _ever_ \n> use that compression.\n\nMaybe. But you'd better have a concrete data set and result numbers to \nconvince me. Designing software for hypothetical situations before they \nactually exist leads to bloatware.\n\n\nNicolas\n"},{"id":"44762","messageId":"Pine.LNX.4.64.0706112216520.4059@racer.site","threadId":"8025","inReplyTo":"alpine.LFD.0.99.0706110930170.12885@xanadu.home","subject":"Re: [PATCH] Add --no-reuse-delta option to git-gc","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-06-11T21:40:38Z","receivedAt":"2007-06-11T21:40:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 11 Jun 2007, Nicolas Pitre wrote:\n\n> But you'd better have a concrete data set and result numbers to convince \n> me. Designing software for hypothetical situations before they actually \n> exist leads to bloatware.\n\nRight you are. So I'll not mention it until I have it ;-)\n\nCiao,\nDscho\n"}]}