Re: git-daemon memory usage, disconnection.

From: Linus Torvalds <torvalds@osdl.org>
Date: Apr 19, 2006, 14:59 UTC
Message-ID: <Pine.LNX.4.64.0604190749160.3701@g5.osdl.org>
In-Reply-To: <1145452967.13200.92.camel@pmac.infradead.org>

On Wed, 19 Apr 2006, David Woodhouse wrote:

Show 15 quoted lines

>
> I'm running git-daemon from xinetd and it seems a little greedy...
> 
> Cpu(s):  2.7% us,  6.4% sy,  0.0% ni,  1.7% id, 87.7% wa,  1.4% hi,  0.0% si
> Mem:    253680k total,   250076k used,     3604k free,      568k buffers
> Swap:   500960k total,   500864k used,       96k free,    24696k cached
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 31232 nobody    18   0  155m  29m 7224 D  1.3 11.9   0:25.56 git-rev-list
> 30743 nobody    18   0  179m  29m 9480 D  0.7 11.9   0:42.60 git-rev-list
> 31277 nobody    18   0  147m  28m 7476 D  2.6 11.4   0:20.90 git-rev-list
> 30314 nobody    18   0  233m  26m 7696 D  0.0 10.6   1:20.24 git-rev-list
> 30612 nobody    18   0  204m  23m 7432 D  1.3  9.4   0:59.19 git-rev-list
> 30574 nobody    18   0  190m  20m 7608 D  0.3  8.3   0:50.77 git-rev-list
> 30208 nobody    18   0  140m  14m 7632 D  0.3  5.9   0:15.23 git-pack-object

Well, you've probably got two issues:

 - it looks like you aren't packing your archives (which explains why the 
   disk accesses are horrid, which in turn explains the "D" part).

   For a git server, you _really_ want all trees to be mostly packed, or 
   you want absolutely tons of memory (and 256kB is definitely not "tons"
   as far as git is concerned).

 - git-rev-list won't notice that there is nobody listening until it gets 
   a EPIPE, and it won't get an EPIPE until it actually outputs something, 
   and it won't output anything until it is largely done traversing the 
   tree..

> Now, this wouldn't be _so_ bad if there were only two of them running.
> The clients for the other four have actually given up and disconnected
> long ago, but git-daemon doesn't seem to have reacted to that.

Well, the way things work under UNIX, you normally don't notice that the other end isn't interested until you try to write, and you get a "nobody is listening". And sadly, the packing stuff does most (not all) of the heavy lifting before it can even start to write things out.

That said, I should probably take a look at git-rev-list --objects memory usage once again. It's neve rbeen exactly "lean" (and it can't really be: it does end up needing the total object list in memory for a full clone, and with something like the kernel, that's about 250 _thousand_ objects).

We should probably also make send-pack.c use the nice revision library, because right now it's doing that pipe to git-rev-list for no good reason.

		Linus