git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Performance issue exposed by git-filter-branch

From
KBKen Brownfield <krb@irridia.com>
Date
Dec 17, 2010, 02:36 UTC
Message-ID
<9A686258-A504-4CBB-9993-048B45B5EE6A@irridia.com>
In-Reply-To
<201012170254.13005.trast@student.ethz.ch>
I had considered this approach (and the one mentioned by Jonathan) but there are no git tools to actually perform the filter I wanted on the export in this form.  I could (and will) parse fast-export and make an attempt a filtering files/directories... my concern is that I won't do it right, and will introduce subtle corruption.  But if there's no existing tool, I'll take a crack at it. :-)
Thanks for your suggestions so far,
Ken
PS: This was my exact first thought, since I was previously used to performing "svnadmin dump/svndumpfilter/svnadmin load" on this repository when it was in SVN.
On Dec 16, 2010, at 5:54 PM, Thomas Rast wrote:
Show 18 quoted lines
> Ken Brownfield wrote:
>> git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch -- bigdirtree stuff/a stuff/b stuff/c stuff/dir/{a,b,c}' --prune-empty --tag-name-filter cat -- --all
> [...]
>> Now that the same repository has grown, this same filter-branch
>> process now takes 6.5 *days* at 100% CPU on the same machine (2x4
>> Xeon, x86_64) on git-1.7.3.2.  There's no I/O, memory, or other
>> resource contention.
> 
> If all you do is an index-filter for deletion, I think it should be
> rather easy to achieve good results by filtering the fast-export
> stream to remove these files, and then piping that back to
> fast-import.
> 
> (It's just that AFAIK nobody has written that code yet.)
> 
> -- 
> Thomas Rast
> trast@{inf,student}.ethz.ch
Previous: Thomas RastNext: Jakub Narebski
Message 7 of 15 in “Performance issue exposed by git-filter-branch”
  1. Ken BrownfieldDec 17, 2010
  2. Jonathan NiederDec 17, 2010
  3. Ken BrownfieldDec 17, 2010
  4. Jonathan NiederDec 17, 2010
  5. Jonathan NiederDec 17, 2010
  6. Thomas RastDec 17, 2010
  7. Ken BrownfieldDec 17, 2010
  8. Jakub NarebskiDec 17, 2010
  9. Ken BrownfieldDec 21, 2010
  10. Jonathan NiederDec 17, 2010
  11. Elijah NewrenDec 17, 2010
  12. Ken BrownfieldFeb 4, 2011
  13. Elijah NewrenFeb 5, 2011
  14. Nguyen Thai Ngoc DuyDec 17, 2010
  15. Ken BrownfieldDec 21, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.