threads / discuss / 34920

Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.

Subject: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.

## tl;dr

3 messages between Sep 12, 2013 and Sep 13, 2013.

replies: 2people: 2as markdown or json

John Gietzen· Sep 12, 2013, 22:01 UTC · lore

Background: Windows, git version 1.8.3.msysgit.0 bare repo, 54k commits after migration from HG git filter-branch --prune-empty -- --all

I'm trying to clean up our repository after migrating it from HG.  I'm running the filter-branch command listed above in an effort to clean up all of garbage commits that HG required ("closing branch" commits and their ilk).
>From my past experience, "git filter-branch" is extremely quick when using simple filters, like env-filter, since it doesn't have to touch the working dir.  However, in our case each revision is taking 1-3 seconds; our entire repo will take 30 hours to clean up at this rate.  Normally, this wouldn't be a problem, except that we are getting "sh.exe couldn't start" errors after anywhere between the 5000th and 6000th rewritten commit.  Filter-branch doesn't have support for picking up where it left off, so we are entirely unable to clean up our repo. 
All that being said, I have 3 questions:
  1.  Is there anything I can do to speed up the filter-branch command? (Alternatively, is there a way I can profile git-filter-branch.sh on msysgit?)
  2.  Any idea why sh.exe would fail?
  3.  Is there a way I can resume the filter-branch command when/if it fails?  (Alternatively, is there a way I can do the filter-branch in pieces and efficiently rebase... or something?)
I have already had to modify git-filter-branch.sh myself (to support the immense number of refs we are rewriting), so I'm comfortable with that.
Any help you can offer would be appreciated.
 
Thanks in advance,
John Gietzen
Felipe Contreras· Sep 13, 2013, 00:47 UTC · re: John Gietzen · lore

Re: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.

On Thu, Sep 12, 2013 at 5:01 PM, John Gietzen <jgietzen@woot.com> wrote:
Show 8 quoted lines
> Background:
> Windows, git version 1.8.3.msysgit.0
> bare repo, 54k commits after migration from HG
> git filter-branch --prune-empty -- --all
>
> I'm trying to clean up our repository after migrating it from HG.  I'm running the filter-branch command listed above in an effort to clean up all of garbage commits that HG required ("closing branch" commits and their ilk).
>
> From my past experience, "git filter-branch" is extremely quick when using simple filters, like env-filter, since it doesn't have to touch the working dir.  However, in our case each revision is taking 1-3 seconds; our entire repo will take 30 hours to clean up at this rate.  Normally, this wouldn't be a problem, except that we are getting "sh.exe couldn't start" errors after anywhere between the 5000th and 6000th rewritten commit.  Filter-branch doesn't have support for picking up where it left off, so we are entirely unable to clean up our repo.

Indeed, I remember writing my own simplified version of 'git filter-branch' that was much faster. If I recall correctly, the trick was avoiding 'git write-tree' which can be done if you are not using any tree filter, but 'git filter-branch' is not that smart.

If all you want to do is prune empty commits, it should be easy to write a script that simply does 'git commit-tree'. I might decide to do that based on my script if I have time today.

-- 
Felipe Contreras
Felipe Contreras· Sep 13, 2013, 09:15 UTC · re: Felipe Contreras · lore

Re: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.

On Thu, Sep 12, 2013 at 7:47 PM, Felipe Contreras <felipe.contreras@gmail.com> wrote:

Show 8 quoted lines
> Indeed, I remember writing my own simplified version of 'git
> filter-branch' that was much faster. If I recall correctly, the trick
> was avoiding 'git write-tree' which can be done if you are not using
> any tree filter, but 'git filter-branch' is not that smart.
>
> If all you want to do is prune empty commits, it should be easy to
> write a script that simply does 'git commit-tree'. I might decide to
> do that based on my script if I have time today.
Here it is, it's straightforward and should be easy to understand.
-- 
Felipe Contreras

← back to recent threads