threads / patch / 3785

patchAdd git-clean command

Subject: [PATCH] Add git-clean command

## tl;dr

11 messages between Apr 3, 2006 and Apr 6, 2006. Diffs are folded; open one to read it.

replies: 10people: 6as markdown or json

Pavel Roskin· Apr 3, 2006, 22:18 UTC · lore

This command removes untracked files from the working tree. This implementation is based on cg-clean with some simplifications. The documentation is included.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---
 .gitignore                  |    1 +
 Documentation/git-clean.txt |   50 +++++++++++++++++++++++++++++++
 Makefile                    |    2 +
 git-clean.sh                |   70 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 122 insertions(+), 1 deletions(-)
Show changes to 4 files +122 −1

.gitignore, Documentation/git-clean.txt, Makefile, git-clean.sh

diff --git a/.gitignore b/.gitignore
index 75891c3..b5959d6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,6 +15,7 @@ git-checkout
 git-checkout-index
 git-cherry
 git-cherry-pick
+git-clean
 git-clone
 git-clone-pack
 git-commit
diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt
new file mode 100644
index 0000000..0c671e1
--- /dev/null
+++ b/Documentation/git-clean.txt
@@ -0,0 +1,50 @@
+git-clean(1)
+============
+
+NAME
+----
+git-clean - Remove untracked files from the working tree
+
+SYNOPSIS
+--------
+[verse]
+'git-clean' [-d | -D] [-n] [-q] [-x]
+
+DESCRIPTION
+-----------
+Removes files unknown to git.  This allows to clean the working tree
+from files that are not under version control.  If the '-x' option is
+specified, ignored files are also removed, allowing to remove all
+build products.
+
+OPTIONS
+-------
+-d::
+	Remove untracked directories in addition to untracked files.
+
+-D::
+	Same as above, but use chmod first.  This is useful for
+	read-only directories used in some testsuites.
+
+-n::
+	Don't actually remove anything, just show what would be done.
+
+-q::
+	Be quiet, only report errors, but not the files that are
+	successfully removed.
+
+-x::
+	Don't use the ignore rules.  This allows removing all untracked
+	files, including build products.  This can be used (possibly in
+	conjunction with gitlink:git-reset[1]) to create a pristine
+	working directory to test a clean build.
+
+
+Author
+------
+Written by Pavel Roskin <proski@gnu.org>
+
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index c79d646..221e59a 100644
--- a/Makefile
+++ b/Makefile
@@ -114,7 +114,7 @@ ### --- END CONFIGURATION SECTION ---
 
 SCRIPT_SH = \
 	git-add.sh git-bisect.sh git-branch.sh git-checkout.sh \
-	git-cherry.sh git-clone.sh git-commit.sh \
+	git-cherry.sh git-clean.sh git-clone.sh git-commit.sh \
 	git-count-objects.sh git-diff.sh git-fetch.sh \
 	git-format-patch.sh git-log.sh git-ls-remote.sh \
 	git-merge-one-file.sh git-parse-remote.sh \
diff --git a/git-clean.sh b/git-clean.sh
new file mode 100755
index 0000000..8470559
--- /dev/null
+++ b/git-clean.sh
@@ -0,0 +1,70 @@
+#!/bin/sh
+#
+# Copyright (c) 2005-2006 Pavel Roskin
+#
+
+USAGE="[-d | -D] [-n] [-q] [-x]"
+LONG_USAGE='Clean untracked files from the working directory
+	-d	remove directories as well
+	-D	remove directories, try harder (chmod first)
+	-n 	don'\''t remove anything, just show what would be done
+	-q	be quiet, only report errors
+	-x	remove ignored files as well'
+SUBDIRECTORY_OK=Yes
+. git-sh-setup
+
+noexclude=
+cleandir=
+cleandirhard=
+quiet=
+rmf="rm -f"
+rmrf="rm -rf"
+rm_refuse="echo Not removing"
+echo1="echo"
+
+for arg in "$@"; do
+	if [ "$arg" = "-d" ]; then
+		cleandir=1
+	elif [ "$arg" = "-D" ]; then
+		cleandir=1
+		cleandirhard=1
+	elif [ "$arg" = "-n" ]; then
+		quiet=1
+		rmf="echo Would remove"
+		rmrf="echo Would remove"
+		rm_refuse="echo Would not remove"
+		echo1=":"
+	elif [ "$arg" = "-q" ]; then
+		quiet=1
+	elif [ "$arg" = "-x" ]; then
+		noexclude=1
+	else
+		echo >&2 "Unknown option: \"$arg\""
+		exit 1
+	fi
+done
+
+excl1=
+excl2=
+if [ -z "$noexclude" ]; then
+	excl1="--exclude-per-directory=.gitignore"
+	if [ -f "$GIT_DIR/info/exclude" ]; then
+		excl2="--exclude-from=$GIT_DIR/info/exclude"
+	fi
+fi
+
+git-ls-files --others --directory "$excl1" "$excl2" |
+while read -r file; do
+	if [ -d "$file" -a ! -L "$file" ]; then
+		if [ -z "$cleandir" ]; then
+			$rm_refuse "$file"
+			continue
+		fi
+		$echo1 "Removing $file"
+		[ "$cleandirhard" ] && chmod -R 700 "$file"
+		$rmrf "$file"
+	else
+		$echo1 "Removing $file"
+		$rmf "$file"
+	fi
+done
Junio C Hamano· Apr 4, 2006, 00:06 UTC · re: Pavel Roskin · lore

Re: [PATCH] Add git-clean command

Pavel Roskin <proski@gnu.org> writes:
> This command removes untracked files from the working tree.  This
> implementation is based on cg-clean with some simplifications.  The
> documentation is included.

I am not opposed to the command in the sense that I do not want to forbid people from doing what they want to do, but on the other hand I do not see why people (apparently many people) want to have something like this. Are their "make clean" broken?

Having said that, just some nitpicks.
> diff --git a/.gitignore b/.gitignore
>...
> +git-clean

I appreciate the attention to the detail; very nice to have a .gitignore entry along with addition of a command.

> diff --git a/git-clean.sh b/git-clean.sh
>...
> +for arg in "$@"; do
	for arg
        do
        	...
> +	if [ "$arg" = "-d" ]; then
	case "$arg" in -d)...
Show 12 quoted lines
> +excl1=
> +excl2=
> +if [ -z "$noexclude" ]; then
> +	excl1="--exclude-per-directory=.gitignore"
> +	if [ -f "$GIT_DIR/info/exclude" ]; then
> +		excl2="--exclude-from=$GIT_DIR/info/exclude"
> +	fi
> +fi
> +
> +git-ls-files --others --directory "$excl1" "$excl2" |
> +while read -r file; do
> ...

The $noexclude case passes two empty strings to git-ls-files, which may happen to be harmless with the current implementation, but does not feel quite right.

Maybe better to read ls-files -z to be really pathname safe, I dunno.

> +		$echo1 "Removing $file"
> +		[ "$cleandirhard" ] && chmod -R 700 "$file"

I am not quite sure this chmod -R is a good idea. If we are trying really hard would we need to also make sure we can rmdir the "$file" by chmod'ing its parent directory? But once we start doing that where would we stop?

Sam Ravnborg· Apr 4, 2006, 10:58 UTC · re: Junio C Hamano · lore

Re: [PATCH] Add git-clean command

On Mon, Apr 03, 2006 at 05:06:36PM -0700, Junio C Hamano wrote:
Show 5 quoted lines
> 
> I am not opposed to the command in the sense that I do not want
> to forbid people from doing what they want to do, but on the
> other hand I do not see why people (apparently many people) want
> to have something like this.  Are their "make clean" broken?

No reason to waste time on make clean. git ls-files -o | xargs rm Does the same job nicely.

Other typical usecases for me: Remove temporaries that I created while trying out stuff. Often I have a bunch of files named 'x', 'xx', 'fisk' etc. around for no use. An easy way to remove these without breaking my 'allmodconfig' build is nice. It anyway > 1 hour to build and I like to get rid of the untracked stuff in an easy way.

So use cases goes like this:
- Remove everything not tracked by git (including .gitignore files)
- Remove everything except tracked by git or ignored
- Remove ignored files (replacement of make clean) (seldom)
Above should work both from top-level dir and in subdirectories.

That is my minimal expectations to git clean. What Pavel came up with cover everything except the make clean replacement part.

	Sam
Pavel Roskin· Apr 4, 2006, 15:52 UTC · re: Sam Ravnborg · lore

Re: [PATCH] Add git-clean command

On Tue, 2006-04-04 at 12:58 +0200, Sam Ravnborg wrote:
Show 6 quoted lines
> On Mon, Apr 03, 2006 at 05:06:36PM -0700, Junio C Hamano wrote:
> > 
> > I am not opposed to the command in the sense that I do not want
> > to forbid people from doing what they want to do, but on the
> > other hand I do not see why people (apparently many people) want
> > to have something like this.  Are their "make clean" broken?

Normally it is not, but if it is, they want to know it. If git-clean remove something after "make clean" then maybe the later is incomplete.

To put it another way, if I make changes and share them with others, I want to make sure that my changes will work for them. If the mechanism for sharing my changes is make-based, I rely on make to check them. In projects using Automake it's called "make distcheck". If I'm sharing my changes as a git patch, I want a git-based verification that the project still builds from scratch.

Also, the files cleaned by "make clean" are normally build products. Things that you _really_ don't want to be in the source tree at the release time are other files, such as backup files with changes that you decided to back out, sources that you may not share, firmware that cost your company $50000 and so on. "make clean" won't remove them.

> No reason to waste time on make clean.
> git ls-files -o | xargs rm
> Does the same job nicely.
This would remove ignored files.  It's not always what I want.
Show 11 quoted lines
> Other typical usecases for me:
> Remove temporaries that I created while trying out stuff.
> Often I have a bunch of files named 'x', 'xx', 'fisk' etc.
> around for no use. An easy way to remove these without breaking
> my 'allmodconfig' build is nice. It anyway > 1 hour to build and
> I like to get rid of the untracked stuff in an easy way.
> 
> So use cases goes like this:
> - Remove everything not tracked by git (including .gitignore files)
> - Remove everything except tracked by git or ignored
> - Remove ignored files (replacement of make clean) (seldom)
I'll think about the later.
Show 5 quoted lines
> Above should work both from top-level dir and in subdirectories.
> 
> That is my minimal expectations to git clean.
> What Pavel came up with cover everything except the make clean
> replacement part.

Exactly. The "make clean" replacement is actually the one I didn't implement.

-- 
Regards,
Pavel Roskin
Jim Meyering· Apr 5, 2006, 16:02 UTC · re: Pavel Roskin · lore

unchecked uses of strdup

There are pretty many uses of strdup in git's sources. Here's one that can cause trouble if it ever returns NULL:

    [from fsck-objects.c]
    static int fsck_head_link(void)
    {
            unsigned char sha1[20];
            const char *git_HEAD = strdup(git_path("HEAD"));
            const char *git_refs_heads_master = resolve_ref(git_HEAD, sha1, 1);

The problem is that resolve_ref does an unconditional `stat' on the parameter corresponding to the maybe-NULL git_HEAD.

One solution is to change such uses of strdup to uses of xstrdup.
Alex Riesen· Apr 6, 2006, 14:11 UTC · re: Jim Meyering · lore

Re: unchecked uses of strdup

On 4/5/06, Jim Meyering <jim@meyering.net> wrote:
Show 12 quoted lines
> There are pretty many uses of strdup in git's sources.
> Here's one that can cause trouble if it ever returns NULL:
>
>     [from fsck-objects.c]
>     static int fsck_head_link(void)
>     {
>             unsigned char sha1[20];
>             const char *git_HEAD = strdup(git_path("HEAD"));
>             const char *git_refs_heads_master = resolve_ref(git_HEAD, sha1, 1);
>
> The problem is that resolve_ref does an unconditional `stat'
> on the parameter corresponding to the maybe-NULL git_HEAD.

That's actually alright (aside a nice core file). Worse are the cases where a NULL would cause some "normal" behaviour, e.g. arguments, which have a meaning for NULL value.

Pavel Roskin· Apr 5, 2006, 06:00 UTC · re: Junio C Hamano · lore

Re: [PATCH] Add git-clean command

On Mon, 2006-04-03 at 17:06 -0700, Junio C Hamano wrote:
Show 7 quoted lines
> > diff --git a/git-clean.sh b/git-clean.sh
> >...
> > +for arg in "$@"; do
> 
> 	for arg
>         do
>         	...

I checked other git shell scripts and copied the "while" construct from one of them - if it's good for other commands, it's good for git-clean.

Show 20 quoted lines
> > +	if [ "$arg" = "-d" ]; then
> 
> 	case "$arg" in -d)...
> 
> > +excl1=
> > +excl2=
> > +if [ -z "$noexclude" ]; then
> > +	excl1="--exclude-per-directory=.gitignore"
> > +	if [ -f "$GIT_DIR/info/exclude" ]; then
> > +		excl2="--exclude-from=$GIT_DIR/info/exclude"
> > +	fi
> > +fi
> > +
> > +git-ls-files --others --directory "$excl1" "$excl2" |
> > +while read -r file; do
> > ...
> 
> The $noexclude case passes two empty strings to git-ls-files,
> which may happen to be harmless with the current implementation,
> but does not feel quite right.

Good catch. This is needed since $GIT_DIR can contain spaces. I believe ${excl2:+"$excl2"} would do the trick.

> Maybe better to read ls-files -z to be really pathname safe, I
> dunno.

I think "xargs -0" has its own problems (argument length limitations), and the other solution is to use perl. While at that, I'd rather rewrite the whole script in Perl, or maybe in even C.

I think this should eventually happen to all git scripts, but I have no intention to do it right now unless you really want me to.

Show 7 quoted lines
> > +		$echo1 "Removing $file"
> > +		[ "$cleandirhard" ] && chmod -R 700 "$file"
> 
> I am not quite sure this chmod -R is a good idea.  If we are
> trying really hard would we need to also make sure we can rmdir
> the "$file" by chmod'ing its parent directory?  But once we
> start doing that where would we stop?
OK, I was undecided on that.  I'm dropping this option.
-- 
Regards,
Pavel Roskin
Martin Waitz· Apr 4, 2006, 08:20 UTC · re: Pavel Roskin · lore

Re: [PATCH] Add git-clean command

hoi :)
Show 5 quoted lines
> +-x::
> +	Don't use the ignore rules.  This allows removing all untracked
> +	files, including build products.  This can be used (possibly in
> +	conjunction with gitlink:git-reset[1]) to create a pristine
> +	working directory to test a clean build.

as ignored files are generally generated files, doesn't it make sense to clean up the "ignored" files, and leave other untracked files alone? That way you don't loose files which you forgot to add to git.

What is the use case of cleaning up all untracked files without also cleaning ignored files?

-- 
Martin Waitz
Junio C Hamano· Apr 4, 2006, 09:08 UTC · re: Martin Waitz · lore

Re: [PATCH] Add git-clean command

Martin Waitz <tali@admingilde.org> writes:
Show 9 quoted lines
>> +-x::
>> +	Don't use the ignore rules.  This allows removing all untracked
>> +	files, including build products.  This can be used (possibly in
>> +	conjunction with gitlink:git-reset[1]) to create a pristine
>> +	working directory to test a clean build.
>
> as ignored files are generally generated files, doesn't it make sense
> to clean up the "ignored" files, and leave other untracked files
> alone?  That way you don't loose files which you forgot to add to git.

Sounds like a sane suggestion, but my previous comment about "make clean" broken for people who want this "git clean" feature applies here as well.

One justification I can think of for "git clean" without -x flag is to make a clean tree that has only the source and build products, removing editor backup files, throwaway test output files and friends, but as you pointed out, this risks losing newly created source files that you forgot to add, so I would need a bit of convincing before I use such a command myself.

Compared to that, removing only ignored files sounds like a much safer operation -- they are explicitly listed as expendable, so it is a lot less likely to lose anything important. But again, that is what "make clean" is there for...

Alex Riesen· Apr 4, 2006, 09:17 UTC · re: Martin Waitz · lore

Re: [PATCH] Add git-clean command

On 4/4/06, Martin Waitz <tali@admingilde.org> wrote:
> What is the use case of cleaning up all untracked files without also
> cleaning ignored files?

Thinks of git's .gitignore: it has config.mak in it. Are you sure you want "clean" your build customizations?

Pavel Roskin· Apr 4, 2006, 15:32 UTC · re: Alex Riesen · lore

Re: [PATCH] Add git-clean command

On Tue, 2006-04-04 at 11:17 +0200, Alex Riesen wrote:
Show 6 quoted lines
> On 4/4/06, Martin Waitz <tali@admingilde.org> wrote:
> > What is the use case of cleaning up all untracked files without also
> > cleaning ignored files?
> 
> Thinks of git's .gitignore: it has config.mak in it. Are you sure you want
> "clean" your build customizations?

In may case, I normally want to remove copies of the sources. For example, I take foo.c, make a clean copy of it, then I change and test it. If if doesn't work and I want to try another approach, I copy it to foo.c-bad or foo.c-approach1. I also make diffs between files to see what exactly I changed. I may also create files for output, valgrind logs and so on.

At some point, I'm satisfied with foo.c, so I commit it. Then I want to remove copies, diffs and other stuff. Yet I don't want to rebuild everything.

It's very rare that I add a new file, and I always remember to add it to the version control. But I'll consider adding an option to only remove ignored files.

-- 
Regards,
Pavel Roskin

← back to recent threads