{"thread":{"id":"34920","subject":"Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.","startedAt":"2013-09-12T22:01:02Z","lastAt":"2013-09-13T09:15:19Z","messageCount":3,"participants":["John Gietzen","Felipe Contreras"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"227585","messageId":"D4C9DBA5F955904683C89B4A96DA7759EA78D60FDF@synapsemail","threadId":"34920","inReplyTo":null,"subject":"Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.","fromName":"John Gietzen","fromEmail":"jgietzen@woot.com","sentAt":"2013-09-12T22:01:02Z","receivedAt":"2013-09-12T22:01:02Z","isPatch":false,"sender":{"key":"jgietzen@woot.com","avatar":null},"body":"Background:\nWindows, git version 1.8.3.msysgit.0\nbare repo, 54k commits after migration from HG\ngit filter-branch --prune-empty -- --all\n\nI'm trying to clean up our repository after migrating it from HG. I'm running the filter-branch command listed above in an effort to clean up all of garbage commits that HG required (\"closing branch\" commits and their ilk).\n\n>From my past experience, \"git filter-branch\" is extremely quick when using simple filters, like env-filter, since it doesn't have to touch the working dir. However, in our case each revision is taking 1-3 seconds; our entire repo will take 30 hours to clean up at this rate. Normally, this wouldn't be a problem, except that we are getting \"sh.exe couldn't start\" errors after anywhere between the 5000th and 6000th rewritten commit. Filter-branch doesn't have support for picking up where it left off, so we are entirely unable to clean up our repo. \n\nAll that being said, I have 3 questions:\n 1. Is there anything I can do to speed up the filter-branch command? (Alternatively, is there a way I can profile git-filter-branch.sh on msysgit?)\n 2. Any idea why sh.exe would fail?\n 3. Is there a way I can resume the filter-branch command when/if it fails? (Alternatively, is there a way I can do the filter-branch in pieces and efficiently rebase... or something?)\n\nI have already had to modify git-filter-branch.sh myself (to support the immense number of refs we are rewriting), so I'm comfortable with that.\n\nAny help you can offer would be appreciated.\n \nThanks in advance,\nJohn Gietzen\n"},{"id":"227596","messageId":"CAMP44s1cmcfSuB-TnFL+-57i5W=EA8v322rVDoUELCPF948jPw@mail.gmail.com","threadId":"34920","inReplyTo":"D4C9DBA5F955904683C89B4A96DA7759EA78D60FDF@synapsemail","subject":"Re: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-09-13T00:47:49Z","receivedAt":"2013-09-13T00:47:49Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Thu, Sep 12, 2013 at 5:01 PM, John Gietzen wrote:\n> Background:\n> Windows, git version 1.8.3.msysgit.0\n> bare repo, 54k commits after migration from HG\n> git filter-branch --prune-empty -- --all\n>\n> I'm trying to clean up our repository after migrating it from HG. I'm running the filter-branch command listed above in an effort to clean up all of garbage commits that HG required (\"closing branch\" commits and their ilk).\n>\n> From my past experience, \"git filter-branch\" is extremely quick when using simple filters, like env-filter, since it doesn't have to touch the working dir. However, in our case each revision is taking 1-3 seconds; our entire repo will take 30 hours to clean up at this rate. Normally, this wouldn't be a problem, except that we are getting \"sh.exe couldn't start\" errors after anywhere between the 5000th and 6000th rewritten commit. Filter-branch doesn't have support for picking up where it left off, so we are entirely unable to clean up our repo.\n\nIndeed, I remember writing my own simplified version of 'git\nfilter-branch' that was much faster. If I recall correctly, the trick\nwas avoiding 'git write-tree' which can be done if you are not using\nany tree filter, but 'git filter-branch' is not that smart.\n\nIf all you want to do is prune empty commits, it should be easy to\nwrite a script that simply does 'git commit-tree'. I might decide to\ndo that based on my script if I have time today.\n\n-- \nFelipe Contreras\n"},{"id":"227618","messageId":"CAMP44s0gHKTgw3xm6qRnvJk9S=p7ndMU6cwt3Ju38pBWbO_Syg@mail.gmail.com","threadId":"34920","inReplyTo":"CAMP44s1cmcfSuB-TnFL+-57i5W=EA8v322rVDoUELCPF948jPw@mail.gmail.com","subject":"Re: Converting repo from HG, `git filter-branch --prune-empty -- --all` is extremely slow and errors out.","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-09-13T09:15:19Z","receivedAt":"2013-09-13T09:15:19Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Thu, Sep 12, 2013 at 7:47 PM, Felipe Contreras\n wrote:\n\n> Indeed, I remember writing my own simplified version of 'git\n> filter-branch' that was much faster. If I recall correctly, the trick\n> was avoiding 'git write-tree' which can be done if you are not using\n> any tree filter, but 'git filter-branch' is not that smart.\n>\n> If all you want to do is prune empty commits, it should be easy to\n> write a script that simply does 'git commit-tree'. I might decide to\n> do that based on my script if I have time today.\n\nHere it is, it's straightforward and should be easy to understand.\n\n-- \nFelipe Contreras\n"}]}