Re: Summer of Code project ideas due this Friday

Comments on the "Better big-file support" section:

"While git can handle arbitrary-sized binary content [...]"

This is very much not true. Git tries at many places to load the complete file into memory and usually fails with "out of memory" if it can't. With the 32bit msysGit client this places the upper file size limit, from purely empirical observation, at 600-700 MByte. When a file is to large git fails late, adding and committing works (as long as there are no filters or other complications), but you can forget about pushing, rebasing or otherwise manipulating that commit. Even worse yet, commits consisting of smaller files but with a combined size over the limit will also cause out-of-memories.

Thus a main focus should be the memory problem, e.g. by using stream-like file handling everywhere, since not working at all is orders of magnitude worse than working slowly :)

"In some cases, this may be as simple as having a "large file" codepath that avoids pulling whole files into memory (e.g., during "git add")."

Ironically git add is one of the few things that work with large files, as mentioned above. Presumably the stream-oriented zlib enforced/encouraged a steam-like handling here :) Slow as hell though and of course it is usually not sensible to compress a 1.5 GByte file.

I'm very willing to work on this topic. Though I'm not a student and as a git code newbie I also don't have the skills for mentoring yet.