git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Performance issue exposed by git-filter-branch

From
KBKen Brownfield <krb@irridia.com>
Date
Dec 21, 2010, 04:49 UTC
Message-ID
<3800FCA0-3894-417D-847D-421693A53E75@irridia.com>
In-Reply-To
<m3lj4sll39.fsf@localhost.localdomain>
This does pretty much exactly what I want (and a lot more), but reposurgeon is now over three days into reading the fast-export stream at 100% CPU.  My guess is that it's about 30% done.
It does look like a great tool for smaller repositories.
Thanks for the suggestion, though!  It looks like git_fast_filter is my next stop.
Ken
On Dec 16, 2010, at 6:51 PM, Jakub Narebski wrote:
Show 34 quoted lines
> Please do not toppost.
> 
> Ken Brownfield <krb@irridia.com> writes:
> 
>> I had considered this approach (and the one mentioned by Jonathan)
>> but there are no git tools to actually perform the filter I wanted
>> on the export in this form.  I could (and will) parse fast-export
>> and make an attempt a filtering files/directories... my concern is
>> that I won't do it right, and will introduce subtle corruption.  But
>> if there's no existing tool, I'll take a crack at it. :-)
> 
> You can try ESR's reposurgeon:
> 
>  http://www.catb.org/~esr/reposurgeon/
> 
> It's limitation is that it loads structure of DAG of revisions (but
> not blobs i.e. contents of file) to memory.  IIRC.  It is not
> streaming, but "DOM" based, otherwise some commands would not work.
> 
> 
> By the way, git-filter-branch documentation recomments to use
> index-filter with git-update-index instead of tree-filter with git-rm,
> and if tree-filter is needed, to use some fast filesystem, e.g. RAM
> one.
> 
> But probably you know all that.
> -- 
> Jakub Narebski
> Poland
> ShadeHawk on #git
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous: Jakub NarebskiNext: Jonathan Nieder
Message 9 of 15 in “Performance issue exposed by git-filter-branch”
  1. Ken BrownfieldDec 17, 2010
  2. Jonathan NiederDec 17, 2010
  3. Ken BrownfieldDec 17, 2010
  4. Jonathan NiederDec 17, 2010
  5. Jonathan NiederDec 17, 2010
  6. Thomas RastDec 17, 2010
  7. Ken BrownfieldDec 17, 2010
  8. Jakub NarebskiDec 17, 2010
  9. Ken BrownfieldDec 21, 2010
  10. Jonathan NiederDec 17, 2010
  11. Elijah NewrenDec 17, 2010
  12. Ken BrownfieldFeb 4, 2011
  13. Elijah NewrenFeb 5, 2011
  14. Nguyen Thai Ngoc DuyDec 17, 2010
  15. Ken BrownfieldDec 21, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.