threads / discuss / 12216

git svn forking an awful lot of "git config"s

Subject: git svn forking an awful lot of "git config"s

## tl;dr

5 messages between Feb 20, 2008 and Feb 21, 2008.

replies: 4people: 4as markdown or json

Johannes Schindelin· Feb 20, 2008, 15:27 UTC · lore

Hi,

we are approaching our first release of msysGit's installer with git-svn. However, I am experiencing a very bad performance, and an error:

$ git svn fetch trace: exec: 'git-svn' 'fetch' trace: built-in: git 'config' '--bool' '--get' 'svn.fetchall' trace: built-in: git 'config' '--get' 'svn.username' trace: built-in: git 'config' '--get' 'svn.repackflags' trace: built-in: git 'config' '--bool' '--get' 'svn.quiet' trace: built-in: git 'config' '--bool' '--get' 'svn.noauthcache' trace: built-in: git 'config' '--get' 'svn.revision' trace: built-in: git 'config' '--int' '--get' 'svn.repack' trace: built-in: git 'config' '--int' '--get' 'svn.logwindowsize' trace: built-in: git 'config' '--bool' '--get' 'svn.nocheckout' trace: built-in: git 'config' '--get' 'svn.configdir' trace: built-in: git 'config' '--bool' '--get' 'svn.noMetadata' trace: built-in: git 'config' '--bool' '--get' 'svn.useSvnsyncProps' trace: built-in: git 'config' '--bool' '--get' 'svn.followparent' trace: built-in: git 'config' '--get' 'svn.authorsfile' trace: built-in: git 'config' '--bool' '--get' 'svn.useSvmProps' trace: built-in: git 'config' '--bool' '--get' 'svn.uselogauthor' trace: built-in: git 'rev-parse' '--symbolic' '--all' trace: built-in: git 'config' '-l' trace: built-in: git 'config' '-l' trace: built-in: git 'config' '-l' trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.branches-maxRev' trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.tags-maxRev' trace: built-in: git 'config' '--get' 'svn-remote.svn.url' trace: built-in: git 'config' '--get' 'svn-remote.svn.uuid' trace: built-in: git 'config' 'svn-remote.svn.branches-maxRev' '8' could not lock config file config svn-remote.svn.branches-maxRev 8: command returned error: 255

I suspect that the locking problem is due to some strange anti-virus interaction, because issuing the same command on the command line succeeds.

However, did you notice the many calls to "git config"? Especially the three ones which list all values anyway?

I am not really sure if that is the single reason of the slowness -- remember, Windows is mightily spawn()-challenged -- but it sure would help to have git-svn read the config once at the beginning, probably with "-z", too, and then just read from the cached values, no?

Ciao, Dscho

P.S.: how far is the svn:external->submodule stuff?

Eric Wong· Feb 20, 2008, 22:39 UTC · re: Johannes Schindelin · lore

Re: git svn forking an awful lot of "git config"s

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 38 quoted lines

> Hi,
> 
> we are approaching our first release of msysGit's installer with git-svn.  
> However, I am experiencing a very bad performance, and an error:
> 
> $ git svn fetch
> trace: exec: 'git-svn' 'fetch'
> trace: built-in: git 'config' '--bool' '--get' 'svn.fetchall'
> trace: built-in: git 'config' '--get' 'svn.username'
> trace: built-in: git 'config' '--get' 'svn.repackflags'
> trace: built-in: git 'config' '--bool' '--get' 'svn.quiet'
> trace: built-in: git 'config' '--bool' '--get' 'svn.noauthcache'
> trace: built-in: git 'config' '--get' 'svn.revision'
> trace: built-in: git 'config' '--int' '--get' 'svn.repack'
> trace: built-in: git 'config' '--int' '--get' 'svn.logwindowsize'
> trace: built-in: git 'config' '--bool' '--get' 'svn.nocheckout'
> trace: built-in: git 'config' '--get' 'svn.configdir'
> trace: built-in: git 'config' '--bool' '--get' 'svn.noMetadata'
> trace: built-in: git 'config' '--bool' '--get' 'svn.useSvnsyncProps'
> trace: built-in: git 'config' '--bool' '--get' 'svn.followparent'
> trace: built-in: git 'config' '--get' 'svn.authorsfile'
> trace: built-in: git 'config' '--bool' '--get' 'svn.useSvmProps'
> trace: built-in: git 'config' '--bool' '--get' 'svn.uselogauthor'
> trace: built-in: git 'rev-parse' '--symbolic' '--all'
> trace: built-in: git 'config' '-l'
> trace: built-in: git 'config' '-l'
> trace: built-in: git 'config' '-l'
> trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.branches-maxRev'
> trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.tags-maxRev'
> trace: built-in: git 'config' '--get' 'svn-remote.svn.url'
> trace: built-in: git 'config' '--get' 'svn-remote.svn.uuid'
> trace: built-in: git 'config' 'svn-remote.svn.branches-maxRev' '8'
> could not lock config file
> config svn-remote.svn.branches-maxRev 8: command returned error: 255
> 
> I suspect that the locking problem is due to some strange anti-virus 
> interaction, because issuing the same command on the command line 
> succeeds.

I believe somebody on the list also had the same problem with file locking in Windows. Unfortunately, I have little idea as to what could be wrong. Could Windows not be releasing file locks properly?

It could be worth it to write a standalone Perl script that replicates the git-config calls without the rest of git-svn getting in the way...

Show 7 quoted lines

> However, did you notice the many calls to "git config"?  Especially the 
> three ones which list all values anyway?
> 
> I am not really sure if that is the single reason of the slowness -- 
> remember, Windows is mightily spawn()-challenged -- but it sure would help 
> to have git-svn read the config once at the beginning, probably with "-z", 
> too, and then just read from the cached values, no?

Many months ago, I thought about implementing a transparent caching layer in Git.pm to work with git configs. Of course, that requires cooperation from all readers/writers within the process... Done correctly, it would help more than just git-svn. too.

I think I had this idea around the time we made git-config output Perl hashes and arrays.

> Ciao,
> Dscho
> 
> P.S.: how far is the svn:external->submodule stuff?

Yikes. I've let other work pile up on my ever-growing todo-list :/ I'll see if I can dig it out and wrap it up this weekend or next...

-- 
Eric Wong

Jakub Narebski· Feb 20, 2008, 23:31 UTC · re: Eric Wong · lore

Re: git svn forking an awful lot of "git config"s

Eric Wong <normalperson@yhbt.net> writes:

> Many months ago, I thought about implementing a transparent caching layer
> in Git.pm to work with git configs.  Of course, that requires
> cooperation from all readers/writers within the process...  Done
> correctly, it would help more than just git-svn. too.

Take a look how gitweb uses "git config -z -l" to read all config
in one go, and save it to hash for later use, lazily.

> I think I had this idea around the time we made git-config output Perl
> hashes and arrays.

This is an alternate solution, better for simple scripts and one-off scripts (you don't need to write "git config -z -l" parser), but I think that additional eval might be not good for performance.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Nagy Balázs· Feb 20, 2008, 23:57 UTC · re: Jakub Narebski · lore

Re: git svn forking an awful lot of "git config"s

Jakub Narebski wrote:

> Take a look how gitweb uses "git config -z -l" to read all config
> in one go, and save it to hash for later use, lazily.

...

> This is an alternate solution, better for simple scripts and one-off
> scripts (you don't need to write "git config -z -l" parser), but
> I think that additional eval might be not good for performance.
>

Maybe I'll finish my git-depot proof of concept script someday. Here's the configuration reader method I use (please note git-depot doesn't use repositories -- yet):

sub readConfig
{
	my $self = shift;
	my ($cfg, @cfg, $cat, $key, $val);

	$cfg = Git::command(qw/config -z -l/);
	@cfg = split(/\000/m, $cfg);
	%{$self->{cfg}} = ();
	foreach (@cfg) {
		if (($cat, $key, $val) = (/\A(\S+)\.(\S+)[\n\r]+^(.+)\Z/ms)) {
			next unless $cat eq 'depot';
			$self->{cfg}->{$key} = $val;
		}
	}
	return 1;
}

Works like a charm.

-- 
-jul-

Johannes Schindelin· Feb 21, 2008, 00:18 UTC · re: Eric Wong · lore

Re: git svn forking an awful lot of "git config"s

Him

On Wed, 20 Feb 2008, Eric Wong wrote:

Show 43 quoted lines

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> 
> > we are approaching our first release of msysGit's installer with 
> > git-svn.  However, I am experiencing a very bad performance, and an 
> > error:
> > 
> > $ git svn fetch
> > trace: exec: 'git-svn' 'fetch'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.fetchall'
> > trace: built-in: git 'config' '--get' 'svn.username'
> > trace: built-in: git 'config' '--get' 'svn.repackflags'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.quiet'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.noauthcache'
> > trace: built-in: git 'config' '--get' 'svn.revision'
> > trace: built-in: git 'config' '--int' '--get' 'svn.repack'
> > trace: built-in: git 'config' '--int' '--get' 'svn.logwindowsize'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.nocheckout'
> > trace: built-in: git 'config' '--get' 'svn.configdir'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.noMetadata'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.useSvnsyncProps'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.followparent'
> > trace: built-in: git 'config' '--get' 'svn.authorsfile'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.useSvmProps'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.uselogauthor'
> > trace: built-in: git 'rev-parse' '--symbolic' '--all'
> > trace: built-in: git 'config' '-l'
> > trace: built-in: git 'config' '-l'
> > trace: built-in: git 'config' '-l'
> > trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.branches-maxRev'
> > trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.tags-maxRev'
> > trace: built-in: git 'config' '--get' 'svn-remote.svn.url'
> > trace: built-in: git 'config' '--get' 'svn-remote.svn.uuid'
> > trace: built-in: git 'config' 'svn-remote.svn.branches-maxRev' '8'
> > could not lock config file
> > config svn-remote.svn.branches-maxRev 8: command returned error: 255
> > 
> > I suspect that the locking problem is due to some strange anti-virus 
> > interaction, because issuing the same command on the command line 
> > succeeds.
> 
> I believe somebody on the list also had the same problem with file 
> locking in Windows.  Unfortunately, I have little idea as to what could 
> be wrong.  Could Windows not be releasing file locks properly?

I am not half as concerned with locks on Windows (since they occur regularly, thanks to _two_ Antiviruses, go figure), as with unnecessary config file reading.

Show 12 quoted lines

> > However, did you notice the many calls to "git config"?  Especially 
> > the three ones which list all values anyway?
> > 
> > I am not really sure if that is the single reason of the slowness -- 
> > remember, Windows is mightily spawn()-challenged -- but it sure would 
> > help to have git-svn read the config once at the beginning, probably 
> > with "-z", too, and then just read from the cached values, no?
> 
> Many months ago, I thought about implementing a transparent caching 
> layer in Git.pm to work with git configs.  Of course, that requires 
> cooperation from all readers/writers within the process...  Done 
> correctly, it would help more than just git-svn. too.

Yes, but I am really scared there. Last time, Pasky tried to put a _lot_ into Git.pm, and it did not work here. As a consequence, I put aside a _lot_ of my time to turn the most important (to me!) perl scripts into builtins.

> I think I had this idea around the time we made git-config output Perl 
> hashes and arrays.

You mean -z?

> > P.S.: how far is the svn:external->submodule stuff?
> 
> Yikes.  I've let other work pile up on my ever-growing todo-list :/ I'll 
> see if I can dig it out and wrap it up this weekend or next...

Actually, I can understand what you're talking about. But just let me tell this: I (amongst others) are _excited_ to wait until you have that particular feature working...

Ciao, Dscho