Re: pack operation is thrashing my server

> Jakub Narebski <jnareb@gmail.com> wrote:
> > Nicolas Pitre <nico@cam.org> writes:
> > > On Tue, 12 Aug 2008, Geert Bosch wrote:
> > > > One nice optimization we could do for those pesky binary large
> > > > objects (like PDF, JPG and GZIP-ed data), is to detect such
> > > > files and revert to compression level 0. This should be
> > > > especially beneficial since already compressed data takes most
> > > > time to compress again.
> > >
> > > That would be a good thing indeed.
> >
> > Perhaps take a sample of some given size and calculate entropy in
> > it? Or just simply add gitattribute for per file compression
> > ratio...
>
> Estimating the entropy would make it "just magic".  Most of Git is
> "just magic" so that's a good direction to take.  I'm not familiar
> enough with the PDF/JPG/GZIP/ZIP stream formats to know what the
> first 4-8k looks like to know if it would give a good indication
> of being already compressed.
>
> Though I'd imagine looking at the first 4k should be sufficient
> for any compressed file.  Having a header composed of 4k of _text_
> before binary compressed data would be nuts.  Or a git-bundle with
> a large refs listing.  ;-)