threads / patch / 17776

patchDocumentation: pruning recipe for destructive filter-branch

Subject: [PATCH] Documentation: pruning recipe for destructive filter-branch

## tl;dr

17 messages between Feb 13, 2009 and Feb 15, 2009. Diffs are folded; open one to read it.

replies: 16people: 4as markdown or json

Thomas Rast· Feb 13, 2009, 17:14 UTC · lore

Add a section about how to shrink a repository's size after running git-filter-branch to remove large blobs from history.

This comes up every week or so on IRC, and the commands required to handle every case are not very newbie-friendly, so hopefully writing them down somewhere leads to fewer questions.

Thanks to doener (Björn Steinbrink) for comments and corrections.
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Or we could just add an option --prune-everything-I-have-a-backup-I-promise to git-filter-branch, so that users can get the same result with less effort.

 Documentation/git-filter-branch.txt |   36 +++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
Show changes to Documentation/git-filter-branch.txt +36 −0
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 1fbbbb4..737c555 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -339,6 +339,42 @@ git filter-branch --index-filter \
 ---------------------------------------------------------------
 
 
+
+Checklist for Shrinking a Repository
+------------------------------------
+
+git-filter-branch is often used to get rid of a subset of files,
+usually with some combination of `\--index-filter` and
+`\--subdirectory-filter`.  If you want to physically shrink the
+repository afterwards, you have some choices:
+
+* Clone it with `git clone file:///path/to/repo`.  The clone will not
+  have the removed objects.  See linkgit:git-clone[1].  (Note that
+  cloning with a plain path just hardlinks everything!)
+
+If you really don't want to clone it, for whatever reasons, check the
+following points (in this order).  This is a very destructive
+approach, so *make a backup* or go back to cloning it.  You have been
+warned.
+
+* Make sure you really removed all variants of a filename, if a blob
+  was moved over its lifetime.  `git log \--follow \--all \-- foo` can
+  help you find renames.
+
+* Make sure you really filtered all refs: use `\--tag-name-filter cat
+  \-- \--all` when calling git-filter-branch.
+
+* Make sure you remove the original refs backed up by
+  git-filter-branch: say `git for-each-ref \--format="%(refname)"
+  refs/original/ | xargs -n 1 git update-ref -d`.
+
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.
+
+* Repack to remove packed objects with `git repack -ad`.
+
+* Prune unpacked unreferenced objects with `git prune \--expire=now`.
+
+
 Author
 ------
 Written by Petr "Pasky" Baudis <pasky@suse.cz>,

-- 
1.6.2.rc0.274.g97213
Junio C Hamano· Feb 14, 2009, 01:07 UTC · re: Thomas Rast · lore

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

Thomas Rast <trast@student.ethz.ch> writes:
Show 25 quoted lines
> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
> index 1fbbbb4..737c555 100644
> --- a/Documentation/git-filter-branch.txt
> +++ b/Documentation/git-filter-branch.txt
> @@ -339,6 +339,42 @@ git filter-branch --index-filter \
>  ---------------------------------------------------------------
>  
>  
> +
> +Checklist for Shrinking a Repository
> +------------------------------------
> +
> +git-filter-branch is often used to get rid of a subset of files,
> +usually with some combination of `\--index-filter` and
> +`\--subdirectory-filter`.  If you want to physically shrink the
> +repository afterwards, you have some choices:
> +
> +* Clone it with `git clone file:///path/to/repo`.  The clone will not
> +  have the removed objects.  See linkgit:git-clone[1].  (Note that
> +  cloning with a plain path just hardlinks everything!)
> +
> +If you really don't want to clone it, for whatever reasons, check the
> +following points (in this order).  This is a very destructive
> +approach, so *make a backup* or go back to cloning it.  You have been
> +warned.
Sounds sane so far and even somewhat amusing.
Show 6 quoted lines
> +* Make sure you really removed all variants of a filename, if a blob
> +  was moved over its lifetime.  `git log \--follow \--all \-- foo` can
> +  help you find renames.
> +
> +* Make sure you really filtered all refs: use `\--tag-name-filter cat
> +  \-- \--all` when calling git-filter-branch.

Don't these two apply equally to "git clone" approach? Only the remaining four are specific to in-place pruning, if I am reading this correctly.

Show 21 quoted lines
> +* Make sure you remove the original refs backed up by
> +  git-filter-branch: say `git for-each-ref \--format="%(refname)"
> +  refs/original/ | xargs -n 1 git update-ref -d`.
> +
> +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> +
> +* Repack to remove packed objects with `git repack -ad`.
> +
> +* Prune unpacked unreferenced objects with `git prune \--expire=now`.
> +
> +
>  Author
>  ------
>  Written by Petr "Pasky" Baudis <pasky@suse.cz>,
> -- 
> 1.6.2.rc0.274.g97213
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jan Krüger· Feb 14, 2009, 01:51 UTC · re: Thomas Rast · lore

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

Hi,

On Fri, 13 Feb 2009 18:14:39 +0100, Thomas Rast <trast@student.ethz.ch> wrote:

> Add a section about how to shrink a repository's size after running
> git-filter-branch to remove large blobs from history.
Good idea.
Show 5 quoted lines
> +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> +
> +* Repack to remove packed objects with `git repack -ad`.
> +
> +* Prune unpacked unreferenced objects with `git prune \--expire=now`.

Is the --expire=now actually needed for prune? The way I read it git-prune(1), it defaults to pruning everything anyway.

-Jan
Johannes Schindelin· Feb 14, 2009, 05:46 UTC · re: Jan Krüger · lore

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

Hi,
On Sat, 14 Feb 2009, Jan Krüger wrote:
Show 7 quoted lines
> On Fri, 13 Feb 2009 18:14:39 +0100, Thomas Rast <trast@student.ethz.ch>
> wrote:
> 
> > Add a section about how to shrink a repository's size after running
> > git-filter-branch to remove large blobs from history.
> 
> Good idea.

Indeed. It is funny when some people ask, but reading it several times a week in #git makes that fun go away real quick.

Show 8 quoted lines
> > +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> > +
> > +* Repack to remove packed objects with `git repack -ad`.
> > +
> > +* Prune unpacked unreferenced objects with `git prune \--expire=now`.
> 
> Is the --expire=now actually needed for prune? The way I read it
> git-prune(1), it defaults to pruning everything anyway.
I hope not.  IIRC I set the default to 2 weeks.

*goes to read the source* Ah, that default is only set in "git gc", and "git prune" still prunes everything. So you're absolutely correct.

But maybe we do not want to direct the user to use "prune" here, but rather "gc"?

Ciao, Dscho

Jan Krüger· Feb 14, 2009, 06:49 UTC · re: Johannes Schindelin · lore

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

Hi,

On Sat, 14 Feb 2009 06:46:18 +0100 (CET), Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 6 quoted lines
> > Is the --expire=now actually needed for prune? The way I read it
> > git-prune(1), it defaults to pruning everything anyway.
> 
> [...] Ah, that default is only set in "git gc", and "git prune" still
> prunes everything. [...] But maybe we do not want to direct the user
> to use "prune" here, but rather "gc"?

I just read the source code of gc and noticed that the only way to influence its expiration limit for prune is to set it in the config file (gc.pruneExpire). Thus there is no immediate way to tell gc to prune *all* unreachable objects and we need to stick to the instructions that are in the patch (minus the unnecessary --expire=now for prune). The alternative would be to add a command line option to gc that corresponds to gc.pruneExpire. I don't really care about that, but a patch for it is below.

-Jan
-----8<-----
From 42226c6c542be4e9fff0817662115f180033e75a Mon Sep 17 00:00:00 2001
From: =?utf-8?q?Jan=20Kr=C3=BCger?= <jk@jk.gs>
Date: Sat, 14 Feb 2009 07:39:23 +0100
Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

In semi-regular intervals, people tend to want to get rid of huge objects they accidentally created in a repository. Of course it shouldn't be too easy to do but also not too hard. Thus we allow them to get rid of unreferenced objects more easily once they have managed to make the objects unreferenced, by allowing them to override the prune expiration limit.

Signed-off-by: Jan Krüger <jk@jk.gs>
---
 Documentation/git-gc.txt |    5 +++++
 builtin-gc.c             |    5 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)
Show changes to 2 files +8 −2

Documentation/git-gc.txt, builtin-gc.c

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 7086eea..a4d35fb 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -59,6 +59,11 @@ are consolidated into a single pack by using the `-A` option of
 'git-repack'. Setting `gc.autopacklimit` to 0 disables
 automatic consolidation of packs.
 
+--prune-expire <time>::
+	Expire unreachable objects older than <time>, rather than
+	considering the configuration variable or the default explained in
+	the next section.
+
 --quiet::
 	Suppress all progress reports.
 
diff --git a/builtin-gc.c b/builtin-gc.c
index a201438..a62c762 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -179,16 +179,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	int quiet = 0;
 	char buf[80];
 
+	git_config(gc_config, NULL);
+
 	struct option builtin_gc_options[] = {
 		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects (deprecated)"),
+		OPT_STRING(0, "prune-expire", &prune_expire, "time", "expire unreferenced objects older than <time>"),
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
 		OPT_BOOLEAN('q', "quiet", &quiet, "suppress progress reports"),
 		OPT_END()
 	};
 
-	git_config(gc_config, NULL);
-
 	if (pack_refs < 0)
 		pack_refs = !is_bare_repository();
 

-- 
1.6.2.rc0.61.g5cd12.dirty
Johannes Schindelin· Feb 14, 2009, 11:48 UTC · re: Jan Krüger · lore

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

Hi,
On Sat, 14 Feb 2009, Jan Krüger wrote:
> Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire

I like it, and if nobody beats me to it, I will add a test for this option tonight.

Ciao, Dscho

Thomas Rast· Feb 14, 2009, 14:22 UTC · re: Johannes Schindelin · lore

[PATCH v2] Documentation: pruning recipe for destructive filter-branch

Add a section about how to shrink a repository's size after running git-filter-branch to remove large blobs from history.

This comes up every week or so on IRC, and the commands required to handle every case are not very newbie-friendly, so hopefully writing them down somewhere leads to fewer questions.

It may seem contradictory to document fallbacks for older Gits in newer docs, but we want to point people at this as a FAQ answer, and they will frequently not have the newest version installed.

Thanks to Björn Steinbrink and Junio C Hamano for comments and corrections.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
Johannes Schindelin wrote:
Show 6 quoted lines
> On Sat, 14 Feb 2009, Jan Krüger wrote:
> 
> > Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
> 
> I like it, and if nobody beats me to it, I will add a test for this 
> option tonight.
Cool, then let's use that in the docs.
 Documentation/git-filter-branch.txt |   36 +++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
Show changes to Documentation/git-filter-branch.txt +36 −0
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 1fbbbb4..d148e58 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -339,6 +339,42 @@ git filter-branch --index-filter \
 ---------------------------------------------------------------
 
 
+
+Checklist for Shrinking a Repository
+------------------------------------
+
+git-filter-branch is often used to get rid of a subset of files,
+usually with some combination of `\--index-filter` and
+`\--subdirectory-filter`.  If you want to physically shrink the
+repository afterwards, you have some choices:
+
+* Make sure you really removed all variants of a filename, if a blob
+  was moved over its lifetime.  `git log \--name-only \--follow \--all
+  \-- filename` can help you find renames.
+
+* Make sure you really filtered all refs: use `\--tag-name-filter cat
+  \-- \--all` when calling git-filter-branch.
+
+* Clone it with `git clone file:///path/to/repo`.  The clone will not
+  have the removed objects.  See linkgit:git-clone[1].  (Note that
+  cloning with a plain path just hardlinks everything!)
+
+If you really don't want to clone it, for whatever reasons, check the
+following points instead (in this order).  This is a very destructive
+approach, so *make a backup* or go back to cloning it.  You have been
+warned.
+
+* Remove the original refs backed up by git-filter-branch: say `git
+  for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git
+  update-ref -d`.
+
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.
+
+* Garbage collect all unreferenced objects with `git gc
+  \--prune-expire=now` (or if your git-gc is not new enough to have
+  that option, `git repack -ad; git prune`).
+
+
 Author
 ------
 Written by Petr "Pasky" Baudis <pasky@suse.cz>,

-- 
1.6.2.rc0.283.g181d4
Johannes Schindelin· Feb 14, 2009, 19:12 UTC · re: Thomas Rast · lore

Re: [PATCH v2] Documentation: pruning recipe for destructive filter-branch

Hi,
On Sat, 14 Feb 2009, Thomas Rast wrote:
Show 9 quoted lines
> Johannes Schindelin wrote:
> > On Sat, 14 Feb 2009, Jan Krüger wrote:
> > 
> > > Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
> > 
> > I like it, and if nobody beats me to it, I will add a test for this 
> > option tonight.
> 
> Cool, then let's use that in the docs.

This would be needed in addition, incase that my --prune=<date> patch is taken:

Show changes to Documentation/git-filter-branch.txt +1 −1
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index d148e58..a8ad884 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -371,7 +371,7 @@ warned.
 * Expire all reflogs with `git reflog expire \--expire=now \--all`.
 
 * Garbage collect all unreferenced objects with `git gc
-  \--prune-expire=now` (or if your git-gc is not new enough to have
+  \--prune=now` (or if your git-gc is not new enough to have
   that option, `git repack -ad; git prune`).
 
 

-- 
1.6.2.rc0.367.g7f9a5
Junio C Hamano· Feb 14, 2009, 19:29 UTC · re: Thomas Rast · lore

Re: [PATCH v2] Documentation: pruning recipe for destructive filter-branch

Thomas Rast <trast@student.ethz.ch> writes:
Show 30 quoted lines
> +git-filter-branch is often used to get rid of a subset of files,
> +usually with some combination of `\--index-filter` and
> +`\--subdirectory-filter`.  If you want to physically shrink the
> +repository afterwards, you have some choices:
> +
> +* Make sure you really removed all variants of a filename, if a blob
> +  was moved over its lifetime.  `git log \--name-only \--follow \--all
> +  \-- filename` can help you find renames.
> +
> +* Make sure you really filtered all refs: use `\--tag-name-filter cat
> +  \-- \--all` when calling git-filter-branch.
> +
> +* Clone it with `git clone file:///path/to/repo`.  The clone will not
> +  have the removed objects.  See linkgit:git-clone[1].  (Note that
> +  cloning with a plain path just hardlinks everything!)
> +
> +If you really don't want to clone it, for whatever reasons, check the
> +following points instead (in this order).  This is a very destructive
> +approach, so *make a backup* or go back to cloning it.  You have been
> +warned.
> +
> +* Remove the original refs backed up by git-filter-branch: say `git
> +  for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git
> +  update-ref -d`.
> +
> +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> +
> +* Garbage collect all unreferenced objects with `git gc
> +  \--prune-expire=now` (or if your git-gc is not new enough to have
> +  that option, `git repack -ad; git prune`).

I am sorry, but the above does not flow well. The first two common preparatory steps appear as if it is only for the "clone" method.

	... often used to get rid of ... and people expect the resulting
        repository to be smaller than the original, but you need a few
        more steps to actually make it smaller, because git tries hard 
        not to lose your objects until you tell it to.  First make sure
	that:
	* you really removed all...
	* you reall filtered all...
	Then, there are two ways to get a smaller repository.  A safer way
	is to clone, that keeps your original intact.
	* Clone it with ...
	If you do not want to ...
	* Remove the original refs...
	* Expire all...
Thomas Rast· Feb 14, 2009, 20:56 UTC · re: Junio C Hamano · lore

[PATCH v3] Documentation: pruning recipe for destructive filter-branch

Add a section about how to shrink a repository's size after running git-filter-branch to remove large blobs from history.

This comes up every week or so on IRC, and the commands required to handle every case are not very newbie-friendly, so hopefully writing them down somewhere leads to fewer questions.

It may seem contradictory to document fallbacks for older Gits in newer docs, but we want to point people at this as a FAQ answer, and they will frequently not have the newest version installed.

Thanks to Björn Steinbrink and Junio C Hamano for comments and corrections.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
Junio C Hamano wrote:
> I am sorry, but the above does not flow well.  The first two common
> preparatory steps appear as if it is only for the "clone" method.

You're absolutely right. I should have re-read it more thoroughly after the rearrangement...

I also changed the --prune-expire to --prune, and changed the surrounding wording since it's now not a new option any more, just new semantics for an old one.

Furthermore, I noticed that asciidoc tries to turn file:///... into a link, mangling the entire paragraph's formatting in the process. Apparently +++quoting+++ helps.

 Documentation/git-filter-branch.txt |   41 +++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)
Show changes to Documentation/git-filter-branch.txt +41 −0
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 1fbbbb4..7ffe03f 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -339,6 +339,47 @@ git filter-branch --index-filter \
 ---------------------------------------------------------------
 
 
+
+Checklist for Shrinking a Repository
+------------------------------------
+
+git-filter-branch is often used to get rid of a subset of files,
+usually with some combination of `\--index-filter` and
+`\--subdirectory-filter`.  People expect the resulting repository to
+be smaller than the original, but you need a few more steps to
+actually make it smaller, because git tries hard not to lose your
+objects until you tell it to.  First make sure that:
+
+* You really removed all variants of a filename, if a blob was moved
+  over its lifetime.  `git log \--name-only \--follow \--all \--
+  filename` can help you find renames.
+
+* You really filtered all refs: use `\--tag-name-filter cat \--
+  \--all` when calling git-filter-branch.
+
+Then there are two ways to get a smaller repository.  A safer way is
+to clone, that keeps your original intact.
+
+* Clone it with `git clone +++file:///path/to/repo+++`.  The clone
+  will not have the removed objects.  See linkgit:git-clone[1].  (Note
+  that cloning with a plain path just hardlinks everything!)
+
+If you really don't want to clone it, for whatever reasons, check the
+following points instead (in this order).  This is a very destructive
+approach, so *make a backup* or go back to cloning it.  You have been
+warned.
+
+* Remove the original refs backed up by git-filter-branch: say `git
+  for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git
+  update-ref -d`.
+
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.
+
+* Garbage collect all unreferenced objects with `git gc \--prune=now`
+  (or if your git-gc is not new enough to support arguments to
+  `\--prune`, use `git repack -ad; git prune` instead).
+
+
 Author
 ------
 Written by Petr "Pasky" Baudis <pasky@suse.cz>,

-- 
1.6.2.rc0.296.ge2122
Johannes Schindelin· Feb 14, 2009, 19:02 UTC · re: Jan Krüger · lore

[PATCH] gc: make --prune useful again by accepting an optional parameter

With this patch, "git gc --no-prune" will not prune any loose (and dangling) object, and "git gc --prune=5.minutes.ago" will prune all loose objects older than 5 minutes.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
	Actually, when trying to add tests to Jan's patch, I realized that 
	I already have a patch in my personal tree, which makes --prune 
	accept an optional parameter...
	Jan, what do you think?  If you prefer --prune-expire, can you 
	please adapt my test?
 Documentation/git-gc.txt |    9 ++++++++-
 builtin-gc.c             |   19 ++++++++++++-------
 t/t5304-prune.sh         |   20 ++++++++++++++++++++
 3 files changed, 40 insertions(+), 8 deletions(-)
Show changes to 3 files +40 −8

Documentation/git-gc.txt, builtin-gc.c, t/t5304-prune.sh

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 7086eea..fcef5fb 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository
 
 SYNOPSIS
 --------
-'git gc' [--aggressive] [--auto] [--quiet]
+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]
 
 DESCRIPTION
 -----------
@@ -59,6 +59,13 @@ are consolidated into a single pack by using the `-A` option of
 'git-repack'. Setting `gc.autopacklimit` to 0 disables
 automatic consolidation of packs.
 
+--prune=<date>::
+	Prune loose objects older than date (default is 2 weeks ago).
+	This option is on by default.
+
+--no-prune::
+	Do not prune any loose objects.
+
 --quiet::
 	Suppress all progress reports.
 
diff --git a/builtin-gc.c b/builtin-gc.c
index a201438..a962f38 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -161,7 +161,8 @@ static int need_to_gc(void)
 	 */
 	if (too_many_packs())
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now") ?
+			      "-a" : "-A",
 			      MAX_ADD);
 	else if (!too_many_loose_objects())
 		return 0;
@@ -173,14 +174,15 @@ static int need_to_gc(void)
 
 int cmd_gc(int argc, const char **argv, const char *prefix)
 {
-	int prune = 0;
 	int aggressive = 0;
 	int auto_gc = 0;
 	int quiet = 0;
 	char buf[80];
 
 	struct option builtin_gc_options[] = {
-		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects (deprecated)"),
+		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
+			"prune unreferenced objects (deprecated)",
+			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
 		OPT_BOOLEAN('q', "quiet", &quiet, "suppress progress reports"),
@@ -218,7 +220,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 			"\"git help gc\" for more information.\n");
 	} else
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now")
+			      ? "-a" : "-A",
 			      MAX_ADD);
 
 	if (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))
@@ -230,9 +233,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	if (run_command_v_opt(argv_repack, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_repack[0]);
 
-	argv_prune[2] = prune_expire;
-	if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
-		return error(FAILED_RUN, argv_prune[0]);
+	if (prune_expire) {
+		argv_prune[2] = prune_expire;
+		if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
+			return error(FAILED_RUN, argv_prune[0]);
+	}
 
 	if (run_command_v_opt(argv_rerere, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_rerere[0]);
diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
index 771c0a0..2e9c4a9 100755
--- a/t/t5304-prune.sh
+++ b/t/t5304-prune.sh
@@ -112,4 +112,24 @@ test_expect_success 'prune: do not prune heads listed as an argument' '
 
 '
 
+test_expect_success 'gc --no-prune && gc --prune=<date>' '
+
+	before=$(git count-objects | sed "s/ .*//") &&
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git gc --no-prune &&
+	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	git gc --prune=5002.days.ago &&
+	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	git gc --prune=5000.days.ago &&
+	test 0 = $(git count-objects | sed "s/ .*//") &&
+	test ! -f $BLOB_FILE
+
+'
+
 test_done

-- 
1.6.2.rc0.367.g7f9a5
Thomas Rast· Feb 14, 2009, 21:33 UTC · re: Johannes Schindelin · lore

Re: [PATCH] gc: make --prune useful again by accepting an optional parameter

Johannes Schindelin wrote:
> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.

Prompted by discussion on IRC, I just spent some time staring at the code, and for the code part and FWIW,

  Acked-by: Thomas Rast <trast@student.ethz.ch>

Meaning that for all I can see, it handles every combination of config, --prune and --no-prune correctly.

I have some non-code nits however:
> +--prune=<date>::
> +	Prune loose objects older than date (default is 2 weeks ago).
> +	This option is on by default.

This isn't really true; to the user, it defaults to the value of 'gc.pruneExpire'. That this is done by defaulting the state of --prune to 2 weeks ago is an implementation detail.

> +		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
> +			"prune unreferenced objects (deprecated)",
> +			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
You still call the option "deprecated" :-)
> diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
[...]
Show 20 quoted lines
> +test_expect_success 'gc --no-prune && gc --prune=<date>' '
> +
> +	before=$(git count-objects | sed "s/ .*//") &&
> +	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
> +	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
> +	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
> +	test -f $BLOB_FILE &&
> +	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
> +	git gc --no-prune &&
> +	test 1 = $(git count-objects | sed "s/ .*//") &&
> +	test -f $BLOB_FILE &&
> +	git gc --prune=5002.days.ago &&
> +	test 1 = $(git count-objects | sed "s/ .*//") &&
> +	test -f $BLOB_FILE &&
> +	git gc --prune=5000.days.ago &&
> +	test 0 = $(git count-objects | sed "s/ .*//") &&
> +	test ! -f $BLOB_FILE
> +
> +'
> +

I'm not a huge fan of such contiguous blocks, as a failure is harder to pin to any of the several tests inside, especially with 'test' not showing any output when it fails.

Also, it does test the interaction between having gc.pruneExpire set, but passing --no-prune: the tests above leave gc.pruneExpire at "2.days.ago". This wasn't immediately obvious however, and you should probably set the config again just in case someone decides to insert a test with a different value.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Johannes Schindelin· Feb 14, 2009, 22:05 UTC · re: Thomas Rast · lore

Re: [PATCH] gc: make --prune useful again by accepting an optional parameter

Thanks for both of your sets of comments.
This is the interdiff.
---
 Documentation/git-gc.txt |    7 ++++---
 builtin-gc.c             |    2 +-
 t/t5304-prune.sh         |   24 +++++++++++++++++++++---
 3 files changed, 26 insertions(+), 7 deletions(-)
Show changes to 3 files +26 −7

Documentation/git-gc.txt, builtin-gc.c, t/t5304-prune.sh

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index fcef5fb..b292e98 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository
 
 SYNOPSIS
 --------
-'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]
+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune]
 
 DESCRIPTION
 -----------
@@ -60,8 +60,9 @@ are consolidated into a single pack by using the `-A` option of
 automatic consolidation of packs.
 
 --prune=<date>::
-	Prune loose objects older than date (default is 2 weeks ago).
-	This option is on by default.
+	Prune loose objects older than date (default is 2 weeks ago,
+	overrideable by the config variable `gc.pruneExpire`).  This
+	option is on by default.
 
 --no-prune::
 	Do not prune any loose objects.
diff --git a/builtin-gc.c b/builtin-gc.c
index 06bc83c..fc556ed 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -181,7 +181,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 
 	struct option builtin_gc_options[] = {
 		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
-			"prune unreferenced objects (deprecated)",
+			"prune unreferenced objects",
 			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
index 2e9c4a9..55ed7c7 100755
--- a/t/t5304-prune.sh
+++ b/t/t5304-prune.sh
@@ -112,7 +112,7 @@ test_expect_success 'prune: do not prune heads listed as an argument' '
 
 '
 
-test_expect_success 'gc --no-prune && gc --prune=<date>' '
+test_expect_success 'gc --no-prune' '
 
 	before=$(git count-objects | sed "s/ .*//") &&
 	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
@@ -120,14 +120,32 @@ test_expect_success 'gc --no-prune && gc --prune=<date>' '
 	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
 	test -f $BLOB_FILE &&
 	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git config gc.pruneExpire 2.days.ago &&
 	git gc --no-prune &&
 	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc respects gc.pruneExpire' '
+
+	git config gc.pruneExpire 5002.days.ago &&
+	git gc &&
 	test -f $BLOB_FILE &&
+	git config gc.pruneExpire 5000.days.ago &&
+	git gc &&
+	test ! -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc --prune=<date>' '
+
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
 	git gc --prune=5002.days.ago &&
-	test 1 = $(git count-objects | sed "s/ .*//") &&
 	test -f $BLOB_FILE &&
 	git gc --prune=5000.days.ago &&
-	test 0 = $(git count-objects | sed "s/ .*//") &&
 	test ! -f $BLOB_FILE
 
 '

-- 
1.6.2.rc0.367.g7f9a5
Johannes Schindelin· Feb 14, 2009, 22:10 UTC · re: Johannes Schindelin · lore

[PATCH v2] gc: make --prune useful again by accepting an optional parameter

With this patch, "git gc --no-prune" will not prune any loose (and dangling) object, and "git gc --prune=5.minutes.ago" will prune all loose objects older than 5 minutes.

This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Documentation/git-gc.txt |   10 +++++++++-
 builtin-gc.c             |   19 ++++++++++++-------
 t/t5304-prune.sh         |   38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 8 deletions(-)
Show changes to 3 files +59 −8

Documentation/git-gc.txt, builtin-gc.c, t/t5304-prune.sh

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 7086eea..b292e98 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository
 
 SYNOPSIS
 --------
-'git gc' [--aggressive] [--auto] [--quiet]
+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune]
 
 DESCRIPTION
 -----------
@@ -59,6 +59,14 @@ are consolidated into a single pack by using the `-A` option of
 'git-repack'. Setting `gc.autopacklimit` to 0 disables
 automatic consolidation of packs.
 
+--prune=<date>::
+	Prune loose objects older than date (default is 2 weeks ago,
+	overrideable by the config variable `gc.pruneExpire`).  This
+	option is on by default.
+
+--no-prune::
+	Do not prune any loose objects.
+
 --quiet::
 	Suppress all progress reports.
 
diff --git a/builtin-gc.c b/builtin-gc.c
index a201438..8d990ed 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -161,7 +161,8 @@ static int need_to_gc(void)
 	 */
 	if (too_many_packs())
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now") ?
+			      "-a" : "-A",
 			      MAX_ADD);
 	else if (!too_many_loose_objects())
 		return 0;
@@ -173,14 +174,15 @@ static int need_to_gc(void)
 
 int cmd_gc(int argc, const char **argv, const char *prefix)
 {
-	int prune = 0;
 	int aggressive = 0;
 	int auto_gc = 0;
 	int quiet = 0;
 	char buf[80];
 
 	struct option builtin_gc_options[] = {
-		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects (deprecated)"),
+		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
+			"prune unreferenced objects",
+			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
 		OPT_BOOLEAN('q', "quiet", &quiet, "suppress progress reports"),
@@ -218,7 +220,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 			"\"git help gc\" for more information.\n");
 	} else
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now")
+			      ? "-a" : "-A",
 			      MAX_ADD);
 
 	if (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))
@@ -230,9 +233,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	if (run_command_v_opt(argv_repack, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_repack[0]);
 
-	argv_prune[2] = prune_expire;
-	if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
-		return error(FAILED_RUN, argv_prune[0]);
+	if (prune_expire) {
+		argv_prune[2] = prune_expire;
+		if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
+			return error(FAILED_RUN, argv_prune[0]);
+	}
 
 	if (run_command_v_opt(argv_rerere, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_rerere[0]);
diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
index 771c0a0..55ed7c7 100755
--- a/t/t5304-prune.sh
+++ b/t/t5304-prune.sh
@@ -112,4 +112,42 @@ test_expect_success 'prune: do not prune heads listed as an argument' '
 
 '
 
+test_expect_success 'gc --no-prune' '
+
+	before=$(git count-objects | sed "s/ .*//") &&
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git config gc.pruneExpire 2.days.ago &&
+	git gc --no-prune &&
+	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc respects gc.pruneExpire' '
+
+	git config gc.pruneExpire 5002.days.ago &&
+	git gc &&
+	test -f $BLOB_FILE &&
+	git config gc.pruneExpire 5000.days.ago &&
+	git gc &&
+	test ! -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc --prune=<date>' '
+
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git gc --prune=5002.days.ago &&
+	test -f $BLOB_FILE &&
+	git gc --prune=5000.days.ago &&
+	test ! -f $BLOB_FILE
+
+'
+
 test_done

-- 
1.6.2.rc0.367.g7f9a5
Thomas Rast· Feb 14, 2009, 22:38 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2] gc: make --prune useful again by accepting an optional parameter

Johannes Schindelin wrote:
Show 12 quoted lines
> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.
> 
> This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  Documentation/git-gc.txt |   10 +++++++++-
>  builtin-gc.c             |   19 ++++++++++++-------
>  t/t5304-prune.sh         |   38 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 59 insertions(+), 8 deletions(-)
Ack.
-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Junio C Hamano· Feb 15, 2009, 19:33 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2] gc: make --prune useful again by accepting an optional parameter

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 7 quoted lines
> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.
>
> This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Thanks.

Being able to say --no-$opt to an OPT_STRING to NULLify a variable that has non-NULL default value is somewhat amusing ;-)

Jan Krüger· Feb 14, 2009, 21:33 UTC · re: Johannes Schindelin · lore

Re: [PATCH] gc: make --prune useful again by accepting an optional parameter

Hi,
> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.
I like it, so disregard my patch.
Show 11 quoted lines
> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
> index 7086eea..fcef5fb 100644
> --- a/Documentation/git-gc.txt
> +++ b/Documentation/git-gc.txt
> @@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local
> repository
>
>  SYNOPSIS
>  --------
> -'git gc' [--aggressive] [--auto] [--quiet]
> +'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]
Maybe add --no-prune too.
Show 11 quoted lines
>
>  DESCRIPTION
>  -----------
> @@ -59,6 +59,13 @@ are consolidated into a single pack by using the `-A`
> option of
>  'git-repack'. Setting `gc.autopacklimit` to 0 disables
>  automatic consolidation of packs.
>
> +--prune=<date>::
> +	Prune loose objects older than date (default is 2 weeks ago).
> +	This option is on by default.
Maybe refer to the config option at least briefly.
Show 27 quoted lines
> +
> +--no-prune::
> +	Do not prune any loose objects.
> +
>  --quiet::
>  	Suppress all progress reports.
>
> diff --git a/builtin-gc.c b/builtin-gc.c
> index a201438..a962f38 100644
> --- a/builtin-gc.c
> +++ b/builtin-gc.c
> [snip]
> @@ -173,14 +174,15 @@ static int need_to_gc(void)
>
>  int cmd_gc(int argc, const char **argv, const char *prefix)
>  {
> -	int prune = 0;
>  	int aggressive = 0;
>  	int auto_gc = 0;
>  	int quiet = 0;
>  	char buf[80];
>
>  	struct option builtin_gc_options[] = {
> -		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects
> (deprecated)"),
> +		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
> +			"prune unreferenced objects (deprecated)",

If you actually make it useful again shouldn't we stop marking it as deprecated?

-Jan

← back to recent threads