# Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out. 3 messages from 2013-09-12 to 2013-09-13. Participants: John Gietzen, Felipe Contreras. Thread: https://gitlist.dev/t/34920 ## John Gietzen, 2013-09-12 22:01 Subject: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out. Message-ID: URL: https://gitlist.dev/e/D4C9DBA5F955904683C89B4A96DA7759EA78D60FDF%40synapsemail ``` Background: Windows, git version 1.8.3.msysgit.0 bare repo, 54k commits after migration from HG git filter-branch --prune-empty -- --all I'm trying to clean up our repository after migrating it from HG. I'm running the filter-branch command listed above in an effort to clean up all of garbage commits that HG required ("closing branch" commits and their ilk). >From my past experience, "git filter-branch" is extremely quick when using simple filters, like env-filter, since it doesn't have to touch the working dir. However, in our case each revision is taking 1-3 seconds; our entire repo will take 30 hours to clean up at this rate. Normally, this wouldn't be a problem, except that we are getting "sh.exe couldn't start" errors after anywhere between the 5000th and 6000th rewritten commit. Filter-branch doesn't have support for picking up where it left off, so we are entirely unable to clean up our repo. All that being said, I have 3 questions: 1. Is there anything I can do to speed up the filter-branch command? (Alternatively, is there a way I can profile git-filter-branch.sh on msysgit?) 2. Any idea why sh.exe would fail? 3. Is there a way I can resume the filter-branch command when/if it fails? (Alternatively, is there a way I can do the filter-branch in pieces and efficiently rebase... or something?) I have already had to modify git-filter-branch.sh myself (to support the immense number of refs we are rewriting), so I'm comfortable with that. Any help you can offer would be appreciated. Thanks in advance, John Gietzen ``` ## Felipe Contreras, 2013-09-13 00:47 Subject: Re: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out. Message-ID: URL: https://gitlist.dev/e/CAMP44s1cmcfSuB-TnFL%2B-57i5W%3DEA8v322rVDoUELCPF948jPw%40mail.gmail.com In-Reply-To: ``` On Thu, Sep 12, 2013 at 5:01 PM, John Gietzen wrote: > Background: > Windows, git version 1.8.3.msysgit.0 > bare repo, 54k commits after migration from HG > git filter-branch --prune-empty -- --all > > I'm trying to clean up our repository after migrating it from HG. I'm running the filter-branch command listed above in an effort to clean up all of garbage commits that HG required ("closing branch" commits and their ilk). > > From my past experience, "git filter-branch" is extremely quick when using simple filters, like env-filter, since it doesn't have to touch the working dir. However, in our case each revision is taking 1-3 seconds; our entire repo will take 30 hours to clean up at this rate. Normally, this wouldn't be a problem, except that we are getting "sh.exe couldn't start" errors after anywhere between the 5000th and 6000th rewritten commit. Filter-branch doesn't have support for picking up where it left off, so we are entirely unable to clean up our repo. Indeed, I remember writing my own simplified version of 'git filter-branch' that was much faster. If I recall correctly, the trick was avoiding 'git write-tree' which can be done if you are not using any tree filter, but 'git filter-branch' is not that smart. If all you want to do is prune empty commits, it should be easy to write a script that simply does 'git commit-tree'. I might decide to do that based on my script if I have time today. -- Felipe Contreras ``` ## Felipe Contreras, 2013-09-13 09:15 Subject: Re: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out. Message-ID: URL: https://gitlist.dev/e/CAMP44s0gHKTgw3xm6qRnvJk9S%3Dp7ndMU6cwt3Ju38pBWbO_Syg%40mail.gmail.com In-Reply-To: ``` On Thu, Sep 12, 2013 at 7:47 PM, Felipe Contreras wrote: > Indeed, I remember writing my own simplified version of 'git > filter-branch' that was much faster. If I recall correctly, the trick > was avoiding 'git write-tree' which can be done if you are not using > any tree filter, but 'git filter-branch' is not that smart. > > If all you want to do is prune empty commits, it should be easy to > write a script that simply does 'git commit-tree'. I might decide to > do that based on my script if I have time today. Here it is, it's straightforward and should be easy to understand. -- Felipe Contreras ```