Re: Python extension commands in git - request for policy change

From: Joshua Jensen <jjensen@workspacewhiz.com>
Date: Dec 12, 2012, 05:11 UTC
Message-ID: <50C811ED.4000600@workspacewhiz.com>
In-Reply-To: <20121212033043.GA24937@thyrsus.com>

----- Original Message -----
From: Eric S. Raymond
Date: 12/11/2012 8:30 PM

Show 5 quoted lines

> It might be a good fit for extending git; I wouldn't be very surprised if
> that worked. However, I do have concerns about the "Oh, we'll just
> lash together a binding to C" attitude common among lua programmers; I
> foresee maintainability problems and the possibility of slow death by
> low-level details as that strategy tries to scale up.

I don't understand this statement: "Oh, we'll just lash together a binding to C" attitude.

??

> My sense is that git's use cases are better served by a glue language
> in the Python/Perl/Ruby class rather than an extension langage. But
> my mind is open on this issue.

I spend nearly 100% of my Git time on Windows.

Spawning new processes in Windows is dog slow. Using 'git rebase', arguably my favorite Git command, is time-waiting torture. I'm also on about as fast of a Windows machine as money can buy these days.

I have a Git add-on similar to git-media that uses the smudge and clean filters to read/write large binary files into a separate storage location. When checking out a workspace, Git shells out to run a filter for each file it needs to write to the workspace.

I can get a maximum of 100 processes per second with this technique, resulting in just 100 files being written to disk. However, I tend to see closer to 60 files written to disk.

So, I patched Git to allow the smudge/clean filters to load up a DLL that executes a Lua script. The Lua script properly retrieves+caches a file locally, or it puts the file on a network share.

The in-process DLL checkout ends up being every bit as fast as when we use Perforce to sync files to our local workspace. Git, then, can be a Perforce replacement for our needs.

(For those who don't know, Perforce handles large workspaces with massive binary files very efficiently.)

Anyway, my preference is to allow scripts to run in-process within Git, because it is far, far faster on Windows. I imagine it is faster than forking processes on non-Windows machines, too, but I have no statistics to back that up.

Python, Perl, or Ruby can be embedded, too, but Lua probably embeds the easiest and smallest out of those other 3 languages.

And shell scripts tend to be the slowest on Windows due to the excessive numbers of process invocations needed to get anything reasonable done.

-Josh