Re: Shell script cleanups/style changes?

Junio C Hamano <gitster@pobox.com> writes:

Well, since the work has already been done, I guess I might as well post it.

With regard to C rewrites: I would hazard a guess that git's performance might actually be improved by splitting some primitives into even smaller C building blocks and tying them all together with pipes (which makes shell scripts a natural container). As long as one designs the C chunks carefully enough that no bulk processing is done in the scripts themselves, this could actually lead to better schedulable pieces of software. I think that most index processing can be done in a list-merge style on sorted lists. That implies pipes and files on input and output, and a small memory footprint. With a good scheduler (the current Linux scheduler sucks at exploiting the asynchronicity of pipes; this should be better with CFS), this should make things work rather efficiently, be flexible for extension, and make good use of multi-core systems.

We have seen a recent example on this list: hand-chaining git-ls-files and a few other tools into a pipeline beat the pants off builtin-add.c.

Given that portability goes down the drain if we want to use similarly or more efficient constellations in C (multithreading and asynchronous I/O come to mind), I would not replace shell scripts (and the associated flexibility in extending functionality) lightly right now. As long as the main data flow is only managed rather than processed by the scripts, I think we would have more to gain by restructuring into pipelineable pieces. It will still be possible to ultimately tie those together in a single process image (with multiple threads presumably). But that immediately takes away a lot of flexibility.