git/list[1] front-page[2] threads[3] people[4] search[5] about
 

filter-branch IO optimization

From
Enrico Weigelt <enrico.weigelt@vnc.biz>
Date
Oct 11, 2012, 15:39 UTC
Message-ID
<fa1e05a5-54b3-47ff-bd28-dc463ebbc4bd@zcs>
In-Reply-To
<7e000a0f-9e4e-4a4d-a8ce-5d017e17939c@zcs>
Hi folks,

for certain projects, I need to regularily run filter-branch on quite large repos (>10k commits), and that needs to be run multiple times, which takes several hours, so I'm looking for optimizations.

The main goal of this filtering is splitting out many modules from a large upstream repo into their own downstream repos. This process should be fully deterministic (IOW: running it twice at the same input, should produce exactly same output, so commit IDs stay the same after subsequent runs)

My current approach is most likely yet a bit too naive:
#1: forkoff new branch from current upstream
#2: run a tree-filter which:
    * removes all files not belonging to the wanted module
    * move the module directory under another subdir (./addons/)
    * fix author/comitter name/email if empty (because otherwise fails)
    * fix charater sets and indentions of source files
#3: loop through `git filter-branch --prune-empty` to get rid of empty
    merge nodes (which otherwise remain really a lot), until branch
    remains unchanged
#4: run plain rebase onto initial commit to linearize the history

All that is done is on per-module basis (for now only about 10, but soon can become much more).

One thing I haven't tried yet is using the -d option to move the .git-rewrite dir to an tmpfs (have to clarify some operating considerations first) ;-o

The next step I have in mind is using --subdirectory-filter, but open questsions are:

* does it suffer from the same problems w/ empty username/email like --tree-filter ?
** if yes: what can I do about it (have an additional pass for fixing that before
   running the --tree-filter ?
* can I somehow teach the --subdirectory filter to place the result under some
  somedir instead of directly to root ?
* can I use --tree-filter in combination with --subdireectory-filter ? 
  which one is executed first ?
thanks
-- 
Mit freundlichen Grüßen / Kind regards 

Enrico Weigelt 
VNC - Virtual Network Consult GmbH 
Head Of Development 

Pariser Platz 4a, D-10117 Berlin
Tel.: +49 (30) 3464615-20
Fax: +49 (30) 3464615-59

enrico.weigelt@vnc.biz; www.vnc.de 
Next: Johannes Sixt
Message 1 of 7 in “filter-branch IO optimization”
  1. Enrico WeigeltOct 11, 2012
  2. Johannes SixtOct 11, 2012
  3. Thomas RastOct 11, 2012
  4. Enrico WeigeltOct 12, 2012
  5. Enrico WeigeltOct 12, 2012
  6. Enrico WeigeltOct 12, 2012
  7. Jeff KingOct 12, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.