threads / discuss / 12900

Truncating and cleaning a imported git repositary to make it more usable

Subject: Truncating and cleaning a imported git repositary to make it more usable

## tl;dr

14 messages between Mar 28, 2008 and Mar 31, 2008.

replies: 13people: 7as markdown or json

Alex Bennee· Mar 28, 2008, 12:35 UTC · lore
Hi,

I've succeeded in porting our whole CVS repository into git (with plenty of swap, cvsps is very memory hungry). Now we have a reference historical repository I need to think about creating one we actually use for day to day use (the .git is currently in the order of 650Mb which is a little heavy for pushing around DSL links willy nilly).

What I need to achieve is fairly simple.
1. Find the common ancestor point for all the product branches I want to
keep in the repo.

Can anyone point me to the magic invocation that would allow this? git-merge-base can find the common point for two commits, do I just have to iteratively use that?

I'm currently working over a terminal link so it would be dead handy if there was a visual way of showing what gitk --all does on the command line.

2. Create a new repo, starting at the common ancestor point
  2(a). I'd need to remove the orphaned old CVS modules which are still
around but not referenced any more.
3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
them

I'm not sure if I could pull directly into the new repo here. I could do it by doing a:

  old_repo: git-format-patch ANCESTOR..BRANCH_HEAD
  new_repo: git-am ../old_repo/0*
But I'm open to newer cleaner ways of dealing with this.

Has anyone done this before? Are there cleaner ways of dealing with this?

-- Alex, homepage: http://www.bennee.com/~alex/ When one knows women one pities men, but when one studies men, one excuses women. -- Horne Tooke

Avery Pennarun· Mar 28, 2008, 12:50 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On 3/28/08, Alex Bennee <kernel-hacker@bennee.com> wrote:
Show 5 quoted lines
>  I've succeeded in porting our whole CVS repository into git (with plenty
>  of swap, cvsps is very memory hungry). Now we have a reference
>  historical repository I need to think about creating one we actually use
>  for day to day use (the .git is currently in the order of 650Mb which is
>  a little heavy for pushing around DSL links willy nilly).

You used git-repack and git-gc, right? If not, the repo will be bigger than it should be.

Show 10 quoted lines
>  1. Find the common ancestor point for all the product branches I want to
>  keep in the repo.
>
>  Can anyone point me to the magic invocation that would allow this?
>  git-merge-base can find the common point for two commits, do I just have
>  to iteratively use that?
>
>  I'm currently working over a terminal link so it would be dead handy if
>  there was a visual way of showing what gitk --all does on the command
>  line.
"git show-branch -a" probably does what you want.
Show 7 quoted lines
>  2. Create a new repo, starting at the common ancestor point
>
>   2(a). I'd need to remove the orphaned old CVS modules which are still
>  around but not referenced any more.
>
>  3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
>  them

You probably want to look at git-filter-branch, particularly the --parent-filter option.

Note that 2(a) would be easiest if you just did it before importing from CVS at all, since it's easy to remove old modules from CVS just by moving the files out of the way.

Have fun,
Avery
Alex Bennee· Mar 28, 2008, 13:08 UTC · re: Avery Pennarun · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 12:50 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
Show 9 quoted lines
> On 3/28/08, Alex Bennee <kernel-hacker@bennee.com> wrote:
>  >  I've succeeded in porting our whole CVS repository into git (with plenty
>  >  of swap, cvsps is very memory hungry). Now we have a reference
>  >  historical repository I need to think about creating one we actually use
>  >  for day to day use (the .git is currently in the order of 650Mb which is
>  >  a little heavy for pushing around DSL links willy nilly).
>
>  You used git-repack and git-gc, right?  If not, the repo will be
>  bigger than it should be.

Oh yes. Unfortunately for 7 years of development of an embedded platform there is a lot of cruft in the CVS repo :-(.

However as it needs to be a drop in replacement for the current repo we can't just drop all the history and start from a fresh repo.

>  >  1. Find the common ancestor point for all the product branches I want to
>  >  keep in the repo.
>  "git show-branch -a" probably does what you want.
Cool. Or in fact git-show-branch --merge-base seems to be the baby.

I'm sure it's been mentioned before but the plethora of git commands can make it hard to find the commands you want even if your sure there is some way of doing it :-)

Show 14 quoted lines
>  >  2. Create a new repo, starting at the common ancestor point
>  >
>  >   2(a). I'd need to remove the orphaned old CVS modules which are still
>  >  around but not referenced any more.
>  >
>  >  3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
>  >  them
>
>  You probably want to look at git-filter-branch, particularly the
>  --parent-filter option.
>
>  Note that 2(a) would be easiest if you just did it before importing
>  from CVS at all, since it's easy to remove old modules from CVS just
>  by moving the files out of the way.

Trouble is looking at the current repo I'm not sure which modules are orphans (at the common anscestor point) and which have been added since.

Of course now I can easily go back in time (easier than in CVS) I can build up this list.

>  Have fun,
For certain values of fun ;-)
-- 
Alex, homepage: http://www.bennee.com/~alex/
Alex Bennee· Mar 28, 2008, 15:52 UTC · re: Avery Pennarun · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 12:50 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
Show 15 quoted lines
> On 3/28/08, Alex Bennee <kernel-hacker@bennee.com> wrote:
>  >  2. Create a new repo, starting at the common ancestor point
>  >
>  >   2(a). I'd need to remove the orphaned old CVS modules which are still
>  >  around but not referenced any more.
>  >
>  >  3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
>  >  them
>
>  You probably want to look at git-filter-branch, particularly the
>  --parent-filter option.
>
>  Note that 2(a) would be easiest if you just did it before importing
>  from CVS at all, since it's easy to remove old modules from CVS just
>  by moving the files out of the way.

Ahh, having investigated further it seems the repo contains two CVS modules which account for about 1/3 of the checkout and are not needed (as they are standalone and separate). When I do the final import from CVS I can skip them. Having said that I'll need to run that over the weekend as cvsps takes up so much memory and swaps a lot.

I tried what I think is the correct usage of git-filter-branch but I get:

ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' dev-branch warning: refname 'dev-branch' is ambiguous. Which ref do you want to rewrite?

Any ideas?
-- 
Alex, homepage: http://www.bennee.com/~alex/
Alex Bennee· Mar 28, 2008, 16:03 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 3:52 PM, Alex Bennee <kernel-hacker@bennee.com> wrote:
>  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
>  'rm -rf big_dira big_dirb' dev-branch
>  warning: refname 'dev-branch' is ambiguous.
>  Which ref do you want to rewrite?
Ignore me. Not sure why but:

git-checkout dev-branch git-filter-branch --tree-filter 'rm -rf big_dira big_dirb' HEAD

works
-- 
Alex, homepage: http://www.bennee.com/~alex/
Johannes Sixt· Mar 28, 2008, 16:43 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

Alex Bennee schrieb:
Show 11 quoted lines
> On Fri, Mar 28, 2008 at 3:52 PM, Alex Bennee <kernel-hacker@bennee.com> wrote:
> 
>>  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
>>  'rm -rf big_dira big_dirb' dev-branch
>>  warning: refname 'dev-branch' is ambiguous.
>>  Which ref do you want to rewrite?
> 
> Ignore me. Not sure why but:
> 
> git-checkout dev-branch
> git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD

You really shouldn't do it this way, unless you do it on a ramdisk. Better use an --index-filter. This is modeled after the last example in the man page (and, of course, untested):

git filter-branch --index-filter \
        'git ls-files -s |
	 grep -v "	big_dira" |
	 grep -v "	big_dirb" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
where the space before big_dir is actually a literal TAB!
-- Hannes
Alex Bennee· Mar 28, 2008, 18:07 UTC · re: Johannes Sixt · lore

[PATCH] Documentation: Another example for git-filter-branch

On Fri, 2008-03-28 at 17:43 +0100, Johannes Sixt wrote:
Show 7 quoted lines
> Alex Bennee schrieb:
> > git-checkout dev-branch
> > git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD
> 
> You really shouldn't do it this way, unless you do it on a ramdisk. Better
> use an --index-filter. This is modeled after the last example in the man
> page (and, of course, untested):
I missed the implications of the last example...
Show 9 quoted lines
> git filter-branch --index-filter \
>         'git ls-files -s |
> 	 grep -v "	big_dira" |
> 	 grep -v "	big_dirb" |
>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                         git update-index --index-info &&
>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
> 
> where the space before big_dir is actually a literal TAB!

Good example. Maybe we should add the following (because plenty of examples is always a good thing IMHO)?

>From 341b480bad4ed9f99a54dc66ba20b0cead4594b5 Mon Sep 17 00:00:00 2001
From: Alex Bennee <Alex.Bennee@cambridgebroadband.com>
Date: Fri, 28 Mar 2008 18:00:20 +0000
Subject: [PATCH] Update git filter-branch examples for cleaning directories
I had to do this to a tree I imported from CVS which had a lot of cruft in it
which was taking ages. Luckily Johannes Sixt submitted this scriplet on the list
before I finished which makes things clearer.
---
 Documentation/git-filter-branch.txt |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 543a1cf..0e4b581 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -184,6 +184,27 @@ git filter-branch --index-filter 'git update-index --remove filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+If you need to remove whole directories it is tempting to use "rm -rf"
+in the --tree-filter form, however this will generate a lot of disk IO
+on big trees. Hence it's more preferable to use the index filter. For
+example:
+
+--------------------------------------------------------------------------
+git filter-branch --index-filter \
+       'git ls-files -s |
+        grep -v "      big_dira" |
+	grep -v "      big_dirb" |
+		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
+                git update-index --index-info &&
+	mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
+--------------------------------------------------------------------------
+
+NB: The grep is matching literal TABs from the output of "git
+ls-files". Try <ctrl-v><tab> if your shell keeps giving you a directory
+listing every time you hit tab.
+
+
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:
-- 
1.5.5.rc0.6.gdeda.dirty





--
Alex, homepage: http://www.bennee.com/~alex/
We reject: kings, presidents, and voting. We believe in: rough consensus
and working code. -- Dave Clark
Mike Hommey· Mar 28, 2008, 20:44 UTC · re: Johannes Sixt · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 05:43:58PM +0100, Johannes Sixt wrote:
Show 24 quoted lines
> Alex Bennee schrieb:
> > On Fri, Mar 28, 2008 at 3:52 PM, Alex Bennee <kernel-hacker@bennee.com> wrote:
> > 
> >>  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
> >>  'rm -rf big_dira big_dirb' dev-branch
> >>  warning: refname 'dev-branch' is ambiguous.
> >>  Which ref do you want to rewrite?
> > 
> > Ignore me. Not sure why but:
> > 
> > git-checkout dev-branch
> > git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD
> 
> You really shouldn't do it this way, unless you do it on a ramdisk. Better
> use an --index-filter. This is modeled after the last example in the man
> page (and, of course, untested):
> 
> git filter-branch --index-filter \
>         'git ls-files -s |
> 	 grep -v "	big_dira" |
> 	 grep -v "	big_dirb" |
>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                         git update-index --index-info &&
>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

git filter-branch --index-filter 'git rm -r --cached big_dira big_dirb' should be working nowadays.

Mike
Alex Bennee· Mar 29, 2008, 13:47 UTC · re: Johannes Sixt · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 4:43 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:
Show 14 quoted lines
> Alex Bennee schrieb:
>  You really shouldn't do it this way, unless you do it on a ramdisk. Better
>  use an --index-filter. This is modeled after the last example in the man
>  page (and, of course, untested):
>
>  git filter-branch --index-filter \
>         'git ls-files -s |
>          grep -v "      big_dira" |
>          grep -v "      big_dirb" |
>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                         git update-index --index-info &&
>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
>
>  where the space before big_dir is actually a literal TAB!

Hmm thats odd. Despite having successfully run the filter my repo is still the same size. git-fsck --full shows loads of dangling commits but I'm guessing because the repo is packed they don't get dropped. Is it possible to repack the whole repo or do you have to manually unpack the packs and re-pack?

-- 
Alex, homepage: http://www.bennee.com/~alex/
Jakub Narebski· Mar 29, 2008, 14:30 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

Alex Bennee wrote:
Show 22 quoted lines
> On Fri, Mar 28, 2008 at 4:43 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:
>> Alex Bennee schrieb:
>>
>>  You really shouldn't do it this way, unless you do it on a ramdisk. Better
>>  use an --index-filter. This is modeled after the last example in the man
>>  page (and, of course, untested):
>>
>>  git filter-branch --index-filter \
>>         'git ls-files -s |
>>          grep -v "      big_dira" |
>>          grep -v "      big_dirb" |
>>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>>                         git update-index --index-info &&
>>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
>>
>>  where the space before big_dir is actually a literal TAB!
> 
> Hmm thats odd. Despite having successfully run the filter my repo is
> still the same size. git-fsck --full shows loads of dangling commits
> but I'm guessing because the repo is packed they don't get dropped. Is
> it possible to repack the whole repo or do you have to manually unpack
> the packs and re-pack?
Did you pruned the repo ("git gc --prune" or "git prune")?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Johannes Sixt· Mar 31, 2008, 06:17 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

Alex Bennee schrieb:
Show 5 quoted lines
> Hmm thats odd. Despite having successfully run the filter my repo is
> still the same size. git-fsck --full shows loads of dangling commits
> but I'm guessing because the repo is packed they don't get dropped. Is
> it possible to repack the whole repo or do you have to manually unpack
> the packs and re-pack?
Yes. Run git repack -a -d -f or create a local clone using the file protocol:
   git clone file://$(pwd)/this that

(If you don't use the file protocol, git clone just hard-links the repository, and you again don't get the reduced size.)

-- Hannes
Jeff King· Mar 28, 2008, 18:08 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 04:03:18PM +0000, Alex Bennee wrote:
> >  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
> >  'rm -rf big_dira big_dirb' dev-branch
> >  warning: refname 'dev-branch' is ambiguous.
> >  Which ref do you want to rewrite?

You have two refs called dev-branch. Probably both a tag and a branch (i.e., refs/heads/dev-branch and refs/tags/dev-branch). You can disambiguate by saying "heads/dev-branch").

> Ignore me. Not sure why but:
> 
> git-checkout dev-branch
> git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD

git-checkout prefers branches to tags[1], so there is no ambiguity. It puts the full refname (refs/heads/dev-branch) into your HEAD, so there is no ambiguity when saying "HEAD".

[1] Actually, the upcoming builtin-checkout no longer behaves this way, and will complain about the ambiguity.

-Peff
Jeff King· Mar 28, 2008, 18:19 UTC · re: Jeff King · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

On Fri, Mar 28, 2008 at 02:08:49PM -0400, Jeff King wrote:
> [1] Actually, the upcoming builtin-checkout no longer behaves this way,
> and will complain about the ambiguity.
To clarify: it is simply a warning; it will still switch to the branch.
-Peff
Michael Haggerty· Mar 29, 2008, 17:51 UTC · re: Alex Bennee · lore

Re: Truncating and cleaning a imported git repositary to make it more usable

Alex Bennee wrote:
> I've succeeded in porting our whole CVS repository into git (with plenty
> of swap, cvsps is very memory hungry). [...]

It sounds like you are doing a one-time conversion. If so, please consider using cvs2svn [1], which can also convert to git [2]. cvsps-based tools are known to have problems with most nontrivial CVS repositories (the contents of the output repository simply don't agree with those of the CVS original). IMO, the only reason to use a cvsps-based tool is if you really need incremental conversions *and* are willing to live with data loss.

Before you trust your new repository, please make some checks. As a minimum, make sure that the contents of each tag and branch tip revision is the same as the contents that you get if you check out of CVS. (I bet they don't!)

Michael

[1] http://cvs2svn.tigris.org [2] http://cvs2svn.tigris.org/cvs2git.html

← back to recent threads