From: demerphq Date: Sat, 01 Aug 2009 08:12:12 GMT Subject: Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots Message-ID: <9b18b3110908010112va6e2ceap727c4129a054ebda@mail.gmail.com> In-Reply-To: <4A739087.1090301@eaglescrag.net> 2009/8/1 J.H. : > Mark A Rada wrote: >> >>  >Note that for me the above results are not aligned in table. >>  >This is a cosmetic issue. >> >> The table formatting issue was due to a bad habit of mixing tabs and >> spaces, >> I decided to go with spaces this time :) >> >> >>  >One thing that would concern me greatly, is not so much the CPU time >> (though that's a *huge* change in comparison to gz) but the memory usage. >>  Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M. >>  >From a 'beefy' server perspective chewing up that much memory per snapshot >> for that long could be bad.  This is likely something that needs to have >> some sort of enable/disable switch if it's going to be included. >> >> True, and there are two solutions I can think of for this "problem". >> >>    1. My tests were at the default compression level, the XZ documentation >>    says that at lower levels you will get resource usage and compression >>    ratios that are comparable to BZip2. However, I'm not sure where you >>    would change the compression level variable for this (globally for the >>    system, somewhere in $GITWEB_CONFIG, a git config variable). Does >>    someone know the correct answer here? > > Well you can always call xz with -[1-9] to change the compression level > (same as gzip and bzip2) though I think a full disabling would be 'more' > preferable, though I'm not sure I like Jakub's suggestion of just deleting > it after the fact, it would work. > >>    2. Implement snapshot caching for Gitweb. > > I think it's slightly broken in my version (binary files don't work right > apparently, it's on my todo list to fix in my upcoming update) but both Lea > and I have done this long ago (caching layers in Gitweb), which would be an > acceptable workaround for this, create once and serve many - though this has > the downside of trading cpu for diskspace.  At least with xz there's less > diskspace used ;-) > > I think more my concern is more what's enabled by default, and since xz is > still new (as was pointed out) it's probably worth only enabling if the > admin selects it to be enabled. FWIW the perl project ripped out all the snapshot generation logic from gitweb, and replaced it with a tool that generates snapshots correctly for our requirements (if the build process needs additional files /currently/ git-archive does not support adding them), this includes a disk level cache for the snapshots since creating the tar, adding the additional files, then gziping is quite slow. If its interesting to people I can post it and the other changes here, although its not a "nice" change, as I literally ripped out the existing code. cheers, Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"