{"thread":{"id":"25257","subject":"[PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects","startedAt":"2010-09-27T11:31:04Z","lastAt":"2010-09-28T09:49:34Z","messageCount":9,"participants":["Jan Krüger","Nicolas Pitre","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"151812","messageId":"20100927133104.25ce5285@jk.gs","threadId":"25257","inReplyTo":null,"subject":"[PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects","fromName":"Jan Krüger","fromEmail":"jk@jk.gs","sentAt":"2010-09-27T11:31:04Z","receivedAt":"2010-09-27T11:31:04Z","isPatch":true,"sender":{"key":"jk@jk.gs","avatar":"https://avatars.githubusercontent.com/u/1774?v=4"},"body":"In 479b56ba ('make \"repack -f\" imply \"pack-objects --no-reuse-object\"'),\ngit repack -f was changed to include recompressing all objects on the\nzlib level on the assumption that if the user wants to spend that much\ntime already, some more time won't hurt (and recompressing is useful if\nthe user changed the zlib compression level).\n\nHowever, \"some more time\" can be quite long with very big repositories,\nso some users are going to appreciate being able to choose. Hence, this\nadds a new -F option that uses the old behaviour of recalculating deltas\nonly and keeping the zlib compression intact.\n\nMeasurements taken using this patch on a current clone of git.git\nindicate a 17% decrease in time being made available to users:\n\ngit repack -Adf  38.79s user 0.56s system 133% cpu 29.394 total\ngit repack -AdF  34.84s user 0.56s system 145% cpu 24.388 total\n\nSigned-off-by: Jan Krüger <jk@jk.gs>\n---\n\nThe concrete case that prompted me to write this patch was a repository\nof 25 GB that some guys were trying to repack. 17% of the time needed to\nrepack -f that much data is... substantial.\n\nDiscussion point: it might make more sense to switch the meanings\naround, making -F do the 'bigger' routine and reverting -f to what it\nused to be. I don't feel strongly about that, however.\n\nCc'ing those who were involved in the discussion leading to 479b56ba.\n\n Documentation/git-repack.txt |    6 +++++-\n git-repack.sh                |    2 ++\n 2 files changed, 7 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 8c67d17..cce32e2 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository\n \n SYNOPSIS\n --------\n-'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N]\n \n DESCRIPTION\n -----------\n@@ -65,6 +65,10 @@ other objects in that pack they already have locally.\n \tPass the `--no-reuse-object` option to `git-pack-objects`, see\n \tlinkgit:git-pack-objects[1].\n \n+-F::\n+\tPass the `--no-reuse-delta` option to `git-pack-objects`, see\n+\tlinkgit:git-pack-objects[1].\n+\n -q::\n \tPass the `-q` option to 'git pack-objects'. See\n \tlinkgit:git-pack-objects[1].\ndiff --git a/git-repack.sh b/git-repack.sh\nindex 1eb3bca..a010406 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -11,6 +11,7 @@ a               pack everything in a single pack\n A               same as -a, and turn unreachable objects loose\n d               remove redundant packs, and run git-prune-packed\n f               pass --no-reuse-object to git-pack-objects\n+F               pass --no-reuse-delta to git-pack-objects\n n               do not run git-update-server-info\n q,quiet         be quiet\n l               pass --local to git-pack-objects\n@@ -35,6 +36,7 @@ do\n \t-d)\tremove_redundant=t ;;\n \t-q)\tGIT_QUIET=t ;;\n \t-f)\tno_reuse=--no-reuse-object ;;\n+\t-F)\tno_reuse=--no-reuse-delta ;;\n \t-l)\tlocal=--local ;;\n \t--max-pack-size|--window|--window-memory|--depth)\n \t\textra=\"$extra $1=$2\"; shift ;;\n-- \n1.7.2.3.392.g02377.dirty\n"},{"id":"151814","messageId":"alpine.LFD.2.00.1009270742250.1146@xanadu.home","threadId":"25257","inReplyTo":"20100927133104.25ce5285@jk.gs","subject":"Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-09-27T11:53:04Z","receivedAt":"2010-09-27T11:53:04Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 27 Sep 2010, Jan Krüger wrote:\n\n> In 479b56ba ('make \"repack -f\" imply \"pack-objects --no-reuse-object\"'),\n> git repack -f was changed to include recompressing all objects on the\n> zlib level on the assumption that if the user wants to spend that much\n> time already, some more time won't hurt (and recompressing is useful if\n> the user changed the zlib compression level).\n> \n> However, \"some more time\" can be quite long with very big repositories,\n> so some users are going to appreciate being able to choose. Hence, this\n> adds a new -F option that uses the old behaviour of recalculating deltas\n> only and keeping the zlib compression intact.\n> \n> Measurements taken using this patch on a current clone of git.git\n> indicate a 17% decrease in time being made available to users:\n> \n> git repack -Adf  38.79s user 0.56s system 133% cpu 29.394 total\n> git repack -AdF  34.84s user 0.56s system 145% cpu 24.388 total\n> \n> Signed-off-by: Jan Krüger <jk@jk.gs>\n> ---\n> \n> The concrete case that prompted me to write this patch was a repository\n> of 25 GB that some guys were trying to repack. 17% of the time needed to\n> repack -f that much data is... substantial.\n> \n> Discussion point: it might make more sense to switch the meanings\n> around, making -F do the 'bigger' routine and reverting -f to what it\n> used to be. I don't feel strongly about that, however.\n\nThat's exactly what I was about to propose before I read through your \nemail down to this part.\n\nI personally don't find --no-reuse-object particularly useful.  I hardly \nimagine that people are changing the pack compression level that often \nif at all.  So I doubt moving the current --no-reuse-object behavior to \n-F and reverting -f to  --no-reuse-delta would cause any serious \ninconvenience.  It certainly won't _break_ anything.  So you have my ACK \nto do that change.\n\nIn addition to that change, perhaps a note could be added to the \ndocumentation for pack.compression indicating that for the new setting \nto take effect for existing packs, they must be repacked with -F.\n\n\nNicolas\n"},{"id":"151816","messageId":"20100927141936.590d71b3@jk.gs","threadId":"25257","inReplyTo":"alpine.LFD.2.00.1009270742250.1146@xanadu.home","subject":"[Alt. PATCH 1/2] repack: add -F flag to let user choose between --no-reuse-delta/object","fromName":"Jan Krüger","fromEmail":"jk@jk.gs","sentAt":"2010-09-27T12:19:36Z","receivedAt":"2010-09-27T12:19:36Z","isPatch":true,"sender":{"key":"jk@jk.gs","avatar":"https://avatars.githubusercontent.com/u/1774?v=4"},"body":"In 479b56ba ('make \"repack -f\" imply \"pack-objects --no-reuse-object\"'),\ngit repack -f was changed to include recompressing all objects on the\nzlib level on the assumption that if the user wants to spend that much\ntime already, some more time won't hurt (and recompressing is useful if\nthe user changed the zlib compression level).\n\nHowever, \"some more time\" can be quite long with very big repositories,\nso some users are going to appreciate being able to choose. If we are\ngoing to give them the choice, --no-reuse-object will probably be\ninteresting a lot less frequently than --no-reuse-delta. Hence, this\nreverts -f to the old behaviour (--no-reuse-delta) and adds a new -F\noption that replaces the current -f.\n\nMeasurements taken using this patch on a current clone of git.git\nindicate a 17% decrease in time being made available to users:\n\ngit repack -Adf  34.84s user 0.56s system 145% cpu 24.388 total\ngit repack -AdF  38.79s user 0.56s system 133% cpu 29.394 total\n\nSigned-off-by: Jan Krüger <jk@jk.gs>\n---\n\n--- Nicolas Pitre <nico@fluxnic.net> wrote:\n\n> I personally don't find --no-reuse-object particularly useful.  I\n> hardly imagine that people are changing the pack compression level\n> that often if at all.  So I doubt moving the current\n> --no-reuse-object behavior to -F and reverting -f to\n> --no-reuse-delta would cause any serious inconvenience.  It certainly\n> won't _break_ anything.  So you have my ACK to do that change.\n\nHere we go.\n\n Documentation/git-repack.txt |    6 +++++-\n git-repack.sh                |    6 ++++--\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 8c67d17..9566727 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository\n \n SYNOPSIS\n --------\n-'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N]\n \n DESCRIPTION\n -----------\n@@ -62,6 +62,10 @@ other objects in that pack they already have locally.\n \tlinkgit:git-pack-objects[1].\n \n -f::\n+\tPass the `--no-reuse-delta` option to `git-pack-objects`, see\n+\tlinkgit:git-pack-objects[1].\n+\n+-F::\n \tPass the `--no-reuse-object` option to `git-pack-objects`, see\n \tlinkgit:git-pack-objects[1].\n \ndiff --git a/git-repack.sh b/git-repack.sh\nindex 1eb3bca..769baaf 100755\n--- a/git-repack.sh\n+++ b/git-repack.sh\n@@ -10,7 +10,8 @@ git repack [options]\n a               pack everything in a single pack\n A               same as -a, and turn unreachable objects loose\n d               remove redundant packs, and run git-prune-packed\n-f               pass --no-reuse-object to git-pack-objects\n+f               pass --no-reuse-delta to git-pack-objects\n+F               pass --no-reuse-object to git-pack-objects\n n               do not run git-update-server-info\n q,quiet         be quiet\n l               pass --local to git-pack-objects\n@@ -34,7 +35,8 @@ do\n \t\tunpack_unreachable=--unpack-unreachable ;;\n \t-d)\tremove_redundant=t ;;\n \t-q)\tGIT_QUIET=t ;;\n-\t-f)\tno_reuse=--no-reuse-object ;;\n+\t-f)\tno_reuse=--no-reuse-delta ;;\n+\t-F)\tno_reuse=--no-reuse-object ;;\n \t-l)\tlocal=--local ;;\n \t--max-pack-size|--window|--window-memory|--depth)\n \t\textra=\"$extra $1=$2\"; shift ;;\n-- \n1.7.2.3.392.g02377.dirty\n"},{"id":"151817","messageId":"20100927142158.487b8f19@jk.gs","threadId":"25257","inReplyTo":"20100927141936.590d71b3@jk.gs","subject":"[Alt. PATCH 2/2] Documentation: pack.compression: explain how to recompress","fromName":"Jan Krüger","fromEmail":"jk@jk.gs","sentAt":"2010-09-27T12:21:58Z","receivedAt":"2010-09-27T12:21:58Z","isPatch":true,"sender":{"key":"jk@jk.gs","avatar":"https://avatars.githubusercontent.com/u/1774?v=4"},"body":"Add a small remark about how to recompress all existing objects after\nchanging the compression level for pack files.\n\nSigned-off-by: Jan Krüger <jk@jk.gs>\n---\n\n--- Nicolas Pitre <nico@fluxnic.net> wrote:\n\n> In addition to that change, perhaps a note could be added to the \n> documentation for pack.compression indicating that for the new\n> setting to take effect for existing packs, they must be repacked with\n> -F.\n\nGood idea. Here it is.\n\n Documentation/config.txt |    4 ++++\n 1 files changed, 4 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d294dd6..506477b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1465,6 +1465,10 @@ pack.compression::\n \tnot set,  defaults to -1, the zlib default, which is \"a default\n \tcompromise between speed and compression (currently equivalent\n \tto level 6).\"\n++\n+Note that changing the compression level will not automatically recompress\n+all existing objects. You can force recompression by passing the -F option\n+to linkgit:git-repack[1].\n \n pack.deltaCacheSize::\n \tThe maximum memory in bytes used for caching deltas in\n-- \n1.7.2.3.392.g02377.dirty\n"},{"id":"151822","messageId":"alpine.LFD.2.00.1009271009490.1146@xanadu.home","threadId":"25257","inReplyTo":"20100927141936.590d71b3@jk.gs","subject":"Re: [Alt. PATCH 1/2] repack: add -F flag to let user choose between --no-reuse-delta/object","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-09-27T14:10:14Z","receivedAt":"2010-09-27T14:10:14Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 27 Sep 2010, Jan Krüger wrote:\n\n> In 479b56ba ('make \"repack -f\" imply \"pack-objects --no-reuse-object\"'),\n> git repack -f was changed to include recompressing all objects on the\n> zlib level on the assumption that if the user wants to spend that much\n> time already, some more time won't hurt (and recompressing is useful if\n> the user changed the zlib compression level).\n> \n> However, \"some more time\" can be quite long with very big repositories,\n> so some users are going to appreciate being able to choose. If we are\n> going to give them the choice, --no-reuse-object will probably be\n> interesting a lot less frequently than --no-reuse-delta. Hence, this\n> reverts -f to the old behaviour (--no-reuse-delta) and adds a new -F\n> option that replaces the current -f.\n> \n> Measurements taken using this patch on a current clone of git.git\n> indicate a 17% decrease in time being made available to users:\n> \n> git repack -Adf  34.84s user 0.56s system 145% cpu 24.388 total\n> git repack -AdF  38.79s user 0.56s system 133% cpu 29.394 total\n> \n> Signed-off-by: Jan Krüger <jk@jk.gs>\n\nAcked-by: Nicolas Pitre <nico@fluxnic.net>\n\n\n> ---\n> \n> --- Nicolas Pitre <nico@fluxnic.net> wrote:\n> \n> > I personally don't find --no-reuse-object particularly useful.  I\n> > hardly imagine that people are changing the pack compression level\n> > that often if at all.  So I doubt moving the current\n> > --no-reuse-object behavior to -F and reverting -f to\n> > --no-reuse-delta would cause any serious inconvenience.  It certainly\n> > won't _break_ anything.  So you have my ACK to do that change.\n> \n> Here we go.\n> \n>  Documentation/git-repack.txt |    6 +++++-\n>  git-repack.sh                |    6 ++++--\n>  2 files changed, 9 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> index 8c67d17..9566727 100644\n> --- a/Documentation/git-repack.txt\n> +++ b/Documentation/git-repack.txt\n> @@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository\n>  \n>  SYNOPSIS\n>  --------\n> -'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]\n> +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N]\n>  \n>  DESCRIPTION\n>  -----------\n> @@ -62,6 +62,10 @@ other objects in that pack they already have locally.\n>  \tlinkgit:git-pack-objects[1].\n>  \n>  -f::\n> +\tPass the `--no-reuse-delta` option to `git-pack-objects`, see\n> +\tlinkgit:git-pack-objects[1].\n> +\n> +-F::\n>  \tPass the `--no-reuse-object` option to `git-pack-objects`, see\n>  \tlinkgit:git-pack-objects[1].\n>  \n> diff --git a/git-repack.sh b/git-repack.sh\n> index 1eb3bca..769baaf 100755\n> --- a/git-repack.sh\n> +++ b/git-repack.sh\n> @@ -10,7 +10,8 @@ git repack [options]\n>  a               pack everything in a single pack\n>  A               same as -a, and turn unreachable objects loose\n>  d               remove redundant packs, and run git-prune-packed\n> -f               pass --no-reuse-object to git-pack-objects\n> +f               pass --no-reuse-delta to git-pack-objects\n> +F               pass --no-reuse-object to git-pack-objects\n>  n               do not run git-update-server-info\n>  q,quiet         be quiet\n>  l               pass --local to git-pack-objects\n> @@ -34,7 +35,8 @@ do\n>  \t\tunpack_unreachable=--unpack-unreachable ;;\n>  \t-d)\tremove_redundant=t ;;\n>  \t-q)\tGIT_QUIET=t ;;\n> -\t-f)\tno_reuse=--no-reuse-object ;;\n> +\t-f)\tno_reuse=--no-reuse-delta ;;\n> +\t-F)\tno_reuse=--no-reuse-object ;;\n>  \t-l)\tlocal=--local ;;\n>  \t--max-pack-size|--window|--window-memory|--depth)\n>  \t\textra=\"$extra $1=$2\"; shift ;;\n> -- \n> 1.7.2.3.392.g02377.dirty\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"151823","messageId":"alpine.LFD.2.00.1009271010180.1146@xanadu.home","threadId":"25257","inReplyTo":"20100927142158.487b8f19@jk.gs","subject":"Re: [Alt. PATCH 2/2] Documentation: pack.compression: explain how to recompress","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-09-27T14:10:47Z","receivedAt":"2010-09-27T14:10:47Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 27 Sep 2010, Jan Krüger wrote:\n\n> Add a small remark about how to recompress all existing objects after\n> changing the compression level for pack files.\n> \n> Signed-off-by: Jan Krüger <jk@jk.gs>\n\nAcked-by: Nicolas Pitre <nico@fluxnic.net>\n\n\n> ---\n> \n> --- Nicolas Pitre <nico@fluxnic.net> wrote:\n> \n> > In addition to that change, perhaps a note could be added to the \n> > documentation for pack.compression indicating that for the new\n> > setting to take effect for existing packs, they must be repacked with\n> > -F.\n> \n> Good idea. Here it is.\n> \n>  Documentation/config.txt |    4 ++++\n>  1 files changed, 4 insertions(+), 0 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index d294dd6..506477b 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1465,6 +1465,10 @@ pack.compression::\n>  \tnot set,  defaults to -1, the zlib default, which is \"a default\n>  \tcompromise between speed and compression (currently equivalent\n>  \tto level 6).\"\n> ++\n> +Note that changing the compression level will not automatically recompress\n> +all existing objects. You can force recompression by passing the -F option\n> +to linkgit:git-repack[1].\n>  \n>  pack.deltaCacheSize::\n>  \tThe maximum memory in bytes used for caching deltas in\n> -- \n> 1.7.2.3.392.g02377.dirty\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"151843","messageId":"7vbp7j2krn.fsf@alter.siamese.dyndns.org","threadId":"25257","inReplyTo":"20100927133104.25ce5285@jk.gs","subject":"Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-09-27T16:50:36Z","receivedAt":"2010-09-27T16:50:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jan Krüger <jk@jk.gs> writes:\n\n> Discussion point: it might make more sense to switch the meanings\n> around, making -F do the 'bigger' routine and reverting -f to what it\n> used to be. I don't feel strongly about that, however.\n\nThat sounds sensible.  reuse_object is used only to z-recompress and is\nnot involved in precomputed delta selection, and under normal\ncircumstances it should not have much effect on the outcome.\n\nPlease make it so.\n"},{"id":"151881","messageId":"20100928084451.78c13d17@jk.gs","threadId":"25257","inReplyTo":"7vbp7j2krn.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects","fromName":"Jan Krüger","fromEmail":"jk@jk.gs","sentAt":"2010-09-28T06:44:51Z","receivedAt":"2010-09-28T06:44:51Z","isPatch":true,"sender":{"key":"jk@jk.gs","avatar":"https://avatars.githubusercontent.com/u/1774?v=4"},"body":"--- Junio C Hamano <gitster@pobox.com> wrote:\n\n> Jan Krüger <jk@jk.gs> writes:\n> \n> > Discussion point: it might make more sense to switch the meanings\n> > around, making -F do the 'bigger' routine and reverting -f to what\n> > it used to be. I don't feel strongly about that, however.\n> \n> That sounds sensible.  reuse_object is used only to z-recompress and\n> is not involved in precomputed delta selection, and under normal\n> circumstances it should not have much effect on the outcome.\n> \n> Please make it so.\n\nAlready done. See the alternative patch (in a 2-series) at\n20100927141936.590d71b3@jk.gs\n<http://mid.gmane.org/20100927141936.590d71b3@jk.gs>. I Cc'd you and\nNicolas acked it. Just saying. ;-)\n\n-Jan\n"},{"id":"151890","messageId":"7vtylatcy9.fsf@alter.siamese.dyndns.org","threadId":"25257","inReplyTo":"20100928084451.78c13d17@jk.gs","subject":"Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-09-28T09:49:34Z","receivedAt":"2010-09-28T09:49:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jan Krüger <jk@jk.gs> writes:\n\n>> Please make it so.\n>\n> Already done.\n\nYeah, I saw and queued them ;-)\n\nThanks.\n"}]}