Re: serious performance issues with images, audio files, and other "non-code" data

From: Jeff King <peff@peff.net>
Date: May 25, 2010, 06:33 UTC
Message-ID: <20100525063354.GC10898@coredump.intra.peff.net>
In-Reply-To: <4BFA2451.6090507@puckerupgames.com>

On Mon, May 24, 2010 at 03:01:37AM -0400, John wrote:

Show 5 quoted lines

> Ok, fair enough. It's your project, and you are defining "source
> control" as that which git supports: non-binary, line-by-line text
> only, C, bash .. no images, documents, etc.
>
> I only wish that definition of "source" had been more clear from the get-go.

I think both Junio's and my responses were not "we can't do a better job with non-text sources" but rather "this is how we ended up with the current state".

I'll admit I have some reservations about trying to figure out a sane set of extensions for default .gitattributes, but that doesn't mean you can't try if you want to. :)

Show 6 quoted lines

> Perhaps a front and center blurb on the git home page or mission
> statement might clarify things for those of us who have different
> definitions of "source"?  That way, you wouldn't have to be bothered
> by folks trying to version all their project assets with git. For
> example, you could specify that non-text is out of scope for git, (or
> however you wish to define "source").

I don't know that we want to explicitly discourage such use. Obviously certain workflows don't work as well with randomly-changing binary blobs (e.g., reading format-patch output is next to useless, though it does still work as a transport if your project relies on emailed patch submissions).

In general, I think we are happy to take patches making binary storage more pleasant (e.g., textconv) as long as they don't somehow make the "normal" case of text worse. There are some things for which git is simply not well suited (single files in the gigabytes, for example), and those aren't likely to change because the some of the issues are fundamental to how git works (though there are often workarounds, like putting gigantic files in their own individual packs). But certainly 100M of jpgs does not seem like an unusable workload to me (as I mentioned, I have a several-gigabyte photo repository that git does just fine with).

-Peff