Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots
- From
- demerphq <demerphq@gmail.com>
- Date
- Aug 1, 2009, 08:12 UTC
- Message-ID
- <9b18b3110908010112va6e2ceap727c4129a054ebda@mail.gmail.com>
- In-Reply-To
- <4A739087.1090301@eaglescrag.net>
2009/8/1 J.H. <warthog19@eaglescrag.net>:
Show 43 quoted lines
> Mark A Rada wrote: >> >> >Note that for me the above results are not aligned in table. >> >This is a cosmetic issue. >> >> The table formatting issue was due to a bad habit of mixing tabs and >> spaces, >> I decided to go with spaces this time :) >> >> >> >One thing that would concern me greatly, is not so much the CPU time >> (though that's a *huge* change in comparison to gz) but the memory usage. >> Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M. >> >From a 'beefy' server perspective chewing up that much memory per snapshot >> for that long could be bad. This is likely something that needs to have >> some sort of enable/disable switch if it's going to be included. >> >> True, and there are two solutions I can think of for this "problem". >> >> 1. My tests were at the default compression level, the XZ documentation >> says that at lower levels you will get resource usage and compression >> ratios that are comparable to BZip2. However, I'm not sure where you >> would change the compression level variable for this (globally for the >> system, somewhere in $GITWEB_CONFIG, a git config variable). Does >> someone know the correct answer here? > > Well you can always call xz with -[1-9] to change the compression level > (same as gzip and bzip2) though I think a full disabling would be 'more' > preferable, though I'm not sure I like Jakub's suggestion of just deleting > it after the fact, it would work. > >> 2. Implement snapshot caching for Gitweb. > > I think it's slightly broken in my version (binary files don't work right > apparently, it's on my todo list to fix in my upcoming update) but both Lea > and I have done this long ago (caching layers in Gitweb), which would be an > acceptable workaround for this, create once and serve many - though this has > the downside of trading cpu for diskspace. At least with xz there's less > diskspace used ;-) > > I think more my concern is more what's enabled by default, and since xz is > still new (as was pointed out) it's probably worth only enabling if the > admin selects it to be enabled.
FWIW the perl project ripped out all the snapshot generation logic from gitweb, and replaced it with a tool that generates snapshots correctly for our requirements (if the build process needs additional files /currently/ git-archive does not support adding them), this includes a disk level cache for the snapshots since creating the tar, adding the additional files, then gziping is quite slow.
If its interesting to people I can post it and the other changes here, although its not a "nice" change, as I literally ripped out the existing code.
cheers, Yves
-- perl -Mre=debug -e "/just|another|perl|hacker/"