threads / discuss / 1031

git-local-pull

Subject: git-local-pull

## tl;dr

4 messages between Jun 27, 2005 and Jun 27, 2005.

replies: 3people: 3as markdown or json

David S. Miller· Jun 27, 2005, 03:05 UTC · lore

I tried to start using git-clone-script to clone repositories locally. It crunches on the disk for a couple of seconds, that's fine, but then I notice the disk activity stop and git-local-pull becomes cpu bound and grows to 80MB in size over the course of 5 minutes.

Is this a side effect of the new pack/unpack stuff?

Compared to what this thing is doing, manually symlinking the object database, copying over the HEAD, and building the index file is significantly faster.

Actually, git-clone-script didn't build an index file. So the compute time definitely came from something else entirely.

Linus Torvalds· Jun 27, 2005, 05:20 UTC · re: David S. Miller · lore

Re: git-local-pull

On Sun, 26 Jun 2005, David S. Miller wrote:
Show 6 quoted lines
> 
> I tried to start using git-clone-script to clone repositories
> locally.  It crunches on the disk for a couple of seconds,
> that's fine, but then I notice the disk activity stop and
> git-local-pull becomes cpu bound and grows to 80MB in size
> over the course of 5 minutes.

Try adding "-l" to the git-local-pull command line, to make it do hardlinks by default (and fall back to a copy if that fails).

> Is this a side effect of the new pack/unpack stuff?
Nope, nothing uses that yet. 

I think the 80MB is just because it built up the object relationship for the whole kernel db there. I don't know what the 5 minutes of CPU are, though, it should just be copying stuff (and at 250MB in and 250MB out, that's a lot of stuff to copy, but it shouldn't be five minutes worth, especially if it's all cached and you see a pegged CPU).

That copy is stupid, btw. Doing a "mmap + write" is a lot slower than doing a "while (data) { read(8kB) + write(8kB) }" loop. The read() will generally do better read-ahead and the small read/write will be better for the caches. But that still doesn't explain 5 minutes worth.

> Compared to what this thing is doing, manually symlinking
> the object database, copying over the HEAD, and building
> the index file is significantly faster.

Well, with "-l" that is what git-local-pull will do too. And if you want symlinks, use "-s". Or use both, in which case it will try a hardlink first, and then a symlink if it gets an EXDEV.

		Linus
Jeff Garzik· Jun 27, 2005, 05:31 UTC · re: Linus Torvalds · lore

Re: git-local-pull

Linus Torvalds wrote:
Show 24 quoted lines
> 
> On Sun, 26 Jun 2005, David S. Miller wrote:
> 
>>I tried to start using git-clone-script to clone repositories
>>locally.  It crunches on the disk for a couple of seconds,
>>that's fine, but then I notice the disk activity stop and
>>git-local-pull becomes cpu bound and grows to 80MB in size
>>over the course of 5 minutes.
> 
> 
> Try adding "-l" to the git-local-pull command line, to make it do 
> hardlinks by default (and fall back to a copy if that fails).
> 
> 
>>Is this a side effect of the new pack/unpack stuff?
> 
> 
> Nope, nothing uses that yet. 
> 
> I think the 80MB is just because it built up the object relationship for 
> the whole kernel db there. I don't know what the 5 minutes of CPU are, 
> though, it should just be copying stuff (and at 250MB in and 250MB out, 
> that's a lot of stuff to copy, but it shouldn't be five minutes worth, 
> especially if it's all cached and you see a pegged CPU).

FWIW, I notice that git-diff-tree uses >100MB of RAM, on a large git-pull-script run. It's generating the diffstat output, AFAICS.

	Jeff
Linus Torvalds· Jun 27, 2005, 18:46 UTC · re: Jeff Garzik · lore

Re: git-local-pull

On Mon, 27 Jun 2005, Jeff Garzik wrote:
> 
> FWIW, I notice that git-diff-tree uses >100MB of RAM, on a large 
> git-pull-script run.  It's generating the diffstat output, AFAICS.

Ahh, yes, I should make the diffstat be conditional, not everybody wants it, and it's often the most expensive part of the pull..

		Linus

← back to recent threads