# [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects 9 messages from 2010-09-27 to 2010-09-28. Participants: Jan Krüger, Nicolas Pitre, Junio C Hamano. Thread: https://gitlist.dev/t/25257 ## Jan Krüger, 2010-09-27 11:31 Subject: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects Message-ID: <20100927133104.25ce5285@jk.gs> URL: https://gitlist.dev/e/20100927133104.25ce5285%40jk.gs ``` In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'), git repack -f was changed to include recompressing all objects on the zlib level on the assumption that if the user wants to spend that much time already, some more time won't hurt (and recompressing is useful if the user changed the zlib compression level). However, "some more time" can be quite long with very big repositories, so some users are going to appreciate being able to choose. Hence, this adds a new -F option that uses the old behaviour of recalculating deltas only and keeping the zlib compression intact. Measurements taken using this patch on a current clone of git.git indicate a 17% decrease in time being made available to users: git repack -Adf 38.79s user 0.56s system 133% cpu 29.394 total git repack -AdF 34.84s user 0.56s system 145% cpu 24.388 total Signed-off-by: Jan Krüger --- The concrete case that prompted me to write this patch was a repository of 25 GB that some guys were trying to repack. 17% of the time needed to repack -f that much data is... substantial. Discussion point: it might make more sense to switch the meanings around, making -F do the 'bigger' routine and reverting -f to what it used to be. I don't feel strongly about that, however. Cc'ing those who were involved in the discussion leading to 479b56ba. Documentation/git-repack.txt | 6 +++++- git-repack.sh | 2 ++ 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt index 8c67d17..cce32e2 100644 --- a/Documentation/git-repack.txt +++ b/Documentation/git-repack.txt @@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository SYNOPSIS -------- -'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N] +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N] DESCRIPTION ----------- @@ -65,6 +65,10 @@ other objects in that pack they already have locally. Pass the `--no-reuse-object` option to `git-pack-objects`, see linkgit:git-pack-objects[1]. +-F:: + Pass the `--no-reuse-delta` option to `git-pack-objects`, see + linkgit:git-pack-objects[1]. + -q:: Pass the `-q` option to 'git pack-objects'. See linkgit:git-pack-objects[1]. diff --git a/git-repack.sh b/git-repack.sh index 1eb3bca..a010406 100755 --- a/git-repack.sh +++ b/git-repack.sh @@ -11,6 +11,7 @@ a pack everything in a single pack A same as -a, and turn unreachable objects loose d remove redundant packs, and run git-prune-packed f pass --no-reuse-object to git-pack-objects +F pass --no-reuse-delta to git-pack-objects n do not run git-update-server-info q,quiet be quiet l pass --local to git-pack-objects @@ -35,6 +36,7 @@ do -d) remove_redundant=t ;; -q) GIT_QUIET=t ;; -f) no_reuse=--no-reuse-object ;; + -F) no_reuse=--no-reuse-delta ;; -l) local=--local ;; --max-pack-size|--window|--window-memory|--depth) extra="$extra $1=$2"; shift ;; -- 1.7.2.3.392.g02377.dirty ``` ## Nicolas Pitre, 2010-09-27 11:53 Subject: Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1009270742250.1146%40xanadu.home In-Reply-To: <20100927133104.25ce5285@jk.gs> ``` On Mon, 27 Sep 2010, Jan Krüger wrote: > In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'), > git repack -f was changed to include recompressing all objects on the > zlib level on the assumption that if the user wants to spend that much > time already, some more time won't hurt (and recompressing is useful if > the user changed the zlib compression level). > > However, "some more time" can be quite long with very big repositories, > so some users are going to appreciate being able to choose. Hence, this > adds a new -F option that uses the old behaviour of recalculating deltas > only and keeping the zlib compression intact. > > Measurements taken using this patch on a current clone of git.git > indicate a 17% decrease in time being made available to users: > > git repack -Adf 38.79s user 0.56s system 133% cpu 29.394 total > git repack -AdF 34.84s user 0.56s system 145% cpu 24.388 total > > Signed-off-by: Jan Krüger > --- > > The concrete case that prompted me to write this patch was a repository > of 25 GB that some guys were trying to repack. 17% of the time needed to > repack -f that much data is... substantial. > > Discussion point: it might make more sense to switch the meanings > around, making -F do the 'bigger' routine and reverting -f to what it > used to be. I don't feel strongly about that, however. That's exactly what I was about to propose before I read through your email down to this part. I personally don't find --no-reuse-object particularly useful. I hardly imagine that people are changing the pack compression level that often if at all. So I doubt moving the current --no-reuse-object behavior to -F and reverting -f to --no-reuse-delta would cause any serious inconvenience. It certainly won't _break_ anything. So you have my ACK to do that change. In addition to that change, perhaps a note could be added to the documentation for pack.compression indicating that for the new setting to take effect for existing packs, they must be repacked with -F. Nicolas ``` ## Jan Krüger, 2010-09-27 12:19 Subject: [Alt. PATCH 1/2] repack: add -F flag to let user choose between --no-reuse-delta/object Message-ID: <20100927141936.590d71b3@jk.gs> URL: https://gitlist.dev/e/20100927141936.590d71b3%40jk.gs In-Reply-To: ``` In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'), git repack -f was changed to include recompressing all objects on the zlib level on the assumption that if the user wants to spend that much time already, some more time won't hurt (and recompressing is useful if the user changed the zlib compression level). However, "some more time" can be quite long with very big repositories, so some users are going to appreciate being able to choose. If we are going to give them the choice, --no-reuse-object will probably be interesting a lot less frequently than --no-reuse-delta. Hence, this reverts -f to the old behaviour (--no-reuse-delta) and adds a new -F option that replaces the current -f. Measurements taken using this patch on a current clone of git.git indicate a 17% decrease in time being made available to users: git repack -Adf 34.84s user 0.56s system 145% cpu 24.388 total git repack -AdF 38.79s user 0.56s system 133% cpu 29.394 total Signed-off-by: Jan Krüger --- --- Nicolas Pitre wrote: > I personally don't find --no-reuse-object particularly useful. I > hardly imagine that people are changing the pack compression level > that often if at all. So I doubt moving the current > --no-reuse-object behavior to -F and reverting -f to > --no-reuse-delta would cause any serious inconvenience. It certainly > won't _break_ anything. So you have my ACK to do that change. Here we go. Documentation/git-repack.txt | 6 +++++- git-repack.sh | 6 ++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt index 8c67d17..9566727 100644 --- a/Documentation/git-repack.txt +++ b/Documentation/git-repack.txt @@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository SYNOPSIS -------- -'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N] +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N] DESCRIPTION ----------- @@ -62,6 +62,10 @@ other objects in that pack they already have locally. linkgit:git-pack-objects[1]. -f:: + Pass the `--no-reuse-delta` option to `git-pack-objects`, see + linkgit:git-pack-objects[1]. + +-F:: Pass the `--no-reuse-object` option to `git-pack-objects`, see linkgit:git-pack-objects[1]. diff --git a/git-repack.sh b/git-repack.sh index 1eb3bca..769baaf 100755 --- a/git-repack.sh +++ b/git-repack.sh @@ -10,7 +10,8 @@ git repack [options] a pack everything in a single pack A same as -a, and turn unreachable objects loose d remove redundant packs, and run git-prune-packed -f pass --no-reuse-object to git-pack-objects +f pass --no-reuse-delta to git-pack-objects +F pass --no-reuse-object to git-pack-objects n do not run git-update-server-info q,quiet be quiet l pass --local to git-pack-objects @@ -34,7 +35,8 @@ do unpack_unreachable=--unpack-unreachable ;; -d) remove_redundant=t ;; -q) GIT_QUIET=t ;; - -f) no_reuse=--no-reuse-object ;; + -f) no_reuse=--no-reuse-delta ;; + -F) no_reuse=--no-reuse-object ;; -l) local=--local ;; --max-pack-size|--window|--window-memory|--depth) extra="$extra $1=$2"; shift ;; -- 1.7.2.3.392.g02377.dirty ``` ## Jan Krüger, 2010-09-27 12:21 Subject: [Alt. PATCH 2/2] Documentation: pack.compression: explain how to recompress Message-ID: <20100927142158.487b8f19@jk.gs> URL: https://gitlist.dev/e/20100927142158.487b8f19%40jk.gs In-Reply-To: <20100927141936.590d71b3@jk.gs> ``` Add a small remark about how to recompress all existing objects after changing the compression level for pack files. Signed-off-by: Jan Krüger --- --- Nicolas Pitre wrote: > In addition to that change, perhaps a note could be added to the > documentation for pack.compression indicating that for the new > setting to take effect for existing packs, they must be repacked with > -F. Good idea. Here it is. Documentation/config.txt | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index d294dd6..506477b 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -1465,6 +1465,10 @@ pack.compression:: not set, defaults to -1, the zlib default, which is "a default compromise between speed and compression (currently equivalent to level 6)." ++ +Note that changing the compression level will not automatically recompress +all existing objects. You can force recompression by passing the -F option +to linkgit:git-repack[1]. pack.deltaCacheSize:: The maximum memory in bytes used for caching deltas in -- 1.7.2.3.392.g02377.dirty ``` ## Nicolas Pitre, 2010-09-27 14:10 Subject: Re: [Alt. PATCH 1/2] repack: add -F flag to let user choose between --no-reuse-delta/object Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1009271009490.1146%40xanadu.home In-Reply-To: <20100927141936.590d71b3@jk.gs> ``` On Mon, 27 Sep 2010, Jan Krüger wrote: > In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'), > git repack -f was changed to include recompressing all objects on the > zlib level on the assumption that if the user wants to spend that much > time already, some more time won't hurt (and recompressing is useful if > the user changed the zlib compression level). > > However, "some more time" can be quite long with very big repositories, > so some users are going to appreciate being able to choose. If we are > going to give them the choice, --no-reuse-object will probably be > interesting a lot less frequently than --no-reuse-delta. Hence, this > reverts -f to the old behaviour (--no-reuse-delta) and adds a new -F > option that replaces the current -f. > > Measurements taken using this patch on a current clone of git.git > indicate a 17% decrease in time being made available to users: > > git repack -Adf 34.84s user 0.56s system 145% cpu 24.388 total > git repack -AdF 38.79s user 0.56s system 133% cpu 29.394 total > > Signed-off-by: Jan Krüger Acked-by: Nicolas Pitre > --- > > --- Nicolas Pitre wrote: > > > I personally don't find --no-reuse-object particularly useful. I > > hardly imagine that people are changing the pack compression level > > that often if at all. So I doubt moving the current > > --no-reuse-object behavior to -F and reverting -f to > > --no-reuse-delta would cause any serious inconvenience. It certainly > > won't _break_ anything. So you have my ACK to do that change. > > Here we go. > > Documentation/git-repack.txt | 6 +++++- > git-repack.sh | 6 ++++-- > 2 files changed, 9 insertions(+), 3 deletions(-) > > diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt > index 8c67d17..9566727 100644 > --- a/Documentation/git-repack.txt > +++ b/Documentation/git-repack.txt > @@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository > > SYNOPSIS > -------- > -'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N] > +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N] > > DESCRIPTION > ----------- > @@ -62,6 +62,10 @@ other objects in that pack they already have locally. > linkgit:git-pack-objects[1]. > > -f:: > + Pass the `--no-reuse-delta` option to `git-pack-objects`, see > + linkgit:git-pack-objects[1]. > + > +-F:: > Pass the `--no-reuse-object` option to `git-pack-objects`, see > linkgit:git-pack-objects[1]. > > diff --git a/git-repack.sh b/git-repack.sh > index 1eb3bca..769baaf 100755 > --- a/git-repack.sh > +++ b/git-repack.sh > @@ -10,7 +10,8 @@ git repack [options] > a pack everything in a single pack > A same as -a, and turn unreachable objects loose > d remove redundant packs, and run git-prune-packed > -f pass --no-reuse-object to git-pack-objects > +f pass --no-reuse-delta to git-pack-objects > +F pass --no-reuse-object to git-pack-objects > n do not run git-update-server-info > q,quiet be quiet > l pass --local to git-pack-objects > @@ -34,7 +35,8 @@ do > unpack_unreachable=--unpack-unreachable ;; > -d) remove_redundant=t ;; > -q) GIT_QUIET=t ;; > - -f) no_reuse=--no-reuse-object ;; > + -f) no_reuse=--no-reuse-delta ;; > + -F) no_reuse=--no-reuse-object ;; > -l) local=--local ;; > --max-pack-size|--window|--window-memory|--depth) > extra="$extra $1=$2"; shift ;; > -- > 1.7.2.3.392.g02377.dirty > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ``` ## Nicolas Pitre, 2010-09-27 14:10 Subject: Re: [Alt. PATCH 2/2] Documentation: pack.compression: explain how to recompress Message-ID: URL: https://gitlist.dev/e/alpine.LFD.2.00.1009271010180.1146%40xanadu.home In-Reply-To: <20100927142158.487b8f19@jk.gs> ``` On Mon, 27 Sep 2010, Jan Krüger wrote: > Add a small remark about how to recompress all existing objects after > changing the compression level for pack files. > > Signed-off-by: Jan Krüger Acked-by: Nicolas Pitre > --- > > --- Nicolas Pitre wrote: > > > In addition to that change, perhaps a note could be added to the > > documentation for pack.compression indicating that for the new > > setting to take effect for existing packs, they must be repacked with > > -F. > > Good idea. Here it is. > > Documentation/config.txt | 4 ++++ > 1 files changed, 4 insertions(+), 0 deletions(-) > > diff --git a/Documentation/config.txt b/Documentation/config.txt > index d294dd6..506477b 100644 > --- a/Documentation/config.txt > +++ b/Documentation/config.txt > @@ -1465,6 +1465,10 @@ pack.compression:: > not set, defaults to -1, the zlib default, which is "a default > compromise between speed and compression (currently equivalent > to level 6)." > ++ > +Note that changing the compression level will not automatically recompress > +all existing objects. You can force recompression by passing the -F option > +to linkgit:git-repack[1]. > > pack.deltaCacheSize:: > The maximum memory in bytes used for caching deltas in > -- > 1.7.2.3.392.g02377.dirty > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ``` ## Junio C Hamano, 2010-09-27 16:50 Subject: Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects Message-ID: <7vbp7j2krn.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vbp7j2krn.fsf%40alter.siamese.dyndns.org In-Reply-To: <20100927133104.25ce5285@jk.gs> ``` Jan Krüger writes: > Discussion point: it might make more sense to switch the meanings > around, making -F do the 'bigger' routine and reverting -f to what it > used to be. I don't feel strongly about that, however. That sounds sensible. reuse_object is used only to z-recompress and is not involved in precomputed delta selection, and under normal circumstances it should not have much effect on the outcome. Please make it so. ``` ## Jan Krüger, 2010-09-28 06:44 Subject: Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects Message-ID: <20100928084451.78c13d17@jk.gs> URL: https://gitlist.dev/e/20100928084451.78c13d17%40jk.gs In-Reply-To: <7vbp7j2krn.fsf@alter.siamese.dyndns.org> ``` --- Junio C Hamano wrote: > Jan Krüger writes: > > > Discussion point: it might make more sense to switch the meanings > > around, making -F do the 'bigger' routine and reverting -f to what > > it used to be. I don't feel strongly about that, however. > > That sounds sensible. reuse_object is used only to z-recompress and > is not involved in precomputed delta selection, and under normal > circumstances it should not have much effect on the outcome. > > Please make it so. Already done. See the alternative patch (in a 2-series) at 20100927141936.590d71b3@jk.gs . I Cc'd you and Nicolas acked it. Just saying. ;-) -Jan ``` ## Junio C Hamano, 2010-09-28 09:49 Subject: Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects Message-ID: <7vtylatcy9.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vtylatcy9.fsf%40alter.siamese.dyndns.org In-Reply-To: <20100928084451.78c13d17@jk.gs> ``` Jan Krüger writes: >> Please make it so. > > Already done. Yeah, I saw and queued them ;-) Thanks. ```