Re: [PATCH] xdiff-interface.c (buffer_is_binary): Remove buffer size limitation

On Tue, 4 Dec 2007, Dmitry V. Levin wrote:

Don't do "average" sizes. That's an almost totally meaningless number.

"Average" makes sense if you have some kind of gaussian distribution or similar. File sizes tend to be exponential distributions, and what makes much more sense is to look at the median. That doesn't show the effect of a few larger files, and also gives you a much better "half the files are smaller than x" idea.

And the median filesize for the kernel is just a few bytes over 4k.

Of the 23,000+ files in the current kernel, about 15,500 are less than 8kB. And 17,179 are smaller than the 10944 bytes you mention.

I'd argue that 8kB (or even 4kB) is probably a good number for things like that: it catches the bulk of all files in their entirety, but it *avoids* spending tons of time on the (few) really large files.

			Linus