patchgit-mv is not able to handle big directories

When moving directory with large number of files git-mv says:

Can't exec "git-update-index": Argument list too long at /usr/local/bin/git-mv line 193. git-update-index failed to add new names with code -1

This patch fixes this by building list of files with limited len (currently 
5000) and executing git-update-index few times until all files will be 
processed. I don't know how to determinate limit of command line but 5000 
seems safe enougth to me.

--- git-mv.perl.orig 2005-11-23 11:24:10.000000000 +0600 +++ git-mv.perl 2005-11-23 11:33:31.000000000 +0600

Alexander Litvinov <lan@ac-sw.com> writes:

Two comments.

(1) the argument limit is enforced by the operating system in
    bytes (including environment size unfortunately) so we might
    want to count bytes not number of paths.  I heard GNU xargs
    uses 131072 as the default limit.

(2) I wonder if we can detect this particular failure case and
    then fall back on splitting the arguments dynamically, maybe
    something like this:

	sub xargs_system {
        	my ($cmd, @args) = @_;
                my $rc = system(@$cmd, @args);
                if ($rc == 'argument list too long error') {
                	my (@args0) = splice(@args, 0, @args/2);
			$rc = xargs_system($cmd, @args0);
                        return $c if ($rc);
                        return xargs_system($cmd, @args);
		}
		return $rc;
	}

    and:

	$rc = xargs_system([qw(git-update-index --)], @changedfiles);
	$rc = xargs_system([qw(git-update-index --add --)], @addedfiles);
	...

Junio C Hamano <junkio@cox.net> writes:

(3) Even better, 'git-update-index -z --stdin'

	if (@changedfiles) {
            open my $oh, qw(|- git-update-index -z --stdin)
                    or die "oops";
            for (@changedfiles) {
                print $oh "$_\0";
	    }
	    close $oh;
	}

	JC "added too many features that myself cannot remember" Hamano

Junio> open my $oh, qw(|- git-update-index -z --stdin) Junio> or die "oops";

This is Perl 5.6 or later. Breaks on Perl 5.5, which is still in use in some places.

To be compatible with 5.5, you have to create a handle explicitly:

        require IO::Handle;
        my $oh = IO::Handle->new;
        open $oh, qw(...) ...;

That works all the way back to 5.4, which is the earliest Perl supported by the core team.

merlyn@stonehenge.com (Randal L. Schwartz) writes:

I should have known better than posting Perl code to the list where a real guru is lurking and making afool of myself ;-).

Thanks for the advice. How much do we care about 5.5? IOW, is "in some places" wide enough to matter?

git-cvsimport, git-svnimport and git-shortlog share the same problem (svnimport declares that it wants 5.8). perl58delta.pod does say list form of open for pipes is new in that version. So what I wrote above requires 5.8 or better, perhaps? cvsimport uses that same structure. That is doubly Ouch.

On Wed, Nov 23, 2005 at 12:37:59AM -0800, Junio C Hamano wrote:

No, you're not using the list form for pipes. I use that in git-graft-ripple:

	open(P,"-|","git-rev-list","--parents","--merge-order",$range)

For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's really only the commercial Unixes where requiring 5.8 would be annoying.

Randal, is my guess even remotely accurate?

Ryan> For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's Ryan> really only the commercial Unixes where requiring 5.8 would be annoying.

Ryan> Randal, is my guess even remotely accurate?

I'd say that 50% of the Perl-using population is at 5.6, with 25% each at 5.5 and 5.8. Those on 5.5 are generally unable to upgrade Perl for corporate reasons.

Targetting Perl 5.6 would assist broad acceptance of git for the typical commercial end user. Targetting 5.5 where possible would ensure practical success for everyone.

However, I have not seen the "target market" of git discussed yet (I came late to the party), so if support for 5.6 (or 5.5) is not chosen, it merely limits the market.

If you'd like, I can review all the Perl code with a tool that determines the minimum Perl version, and provide patches to bring the code to 5.5 level. But if it's likely that someone will say "this is not important to us", I'd rather not waste my time. :)

merlyn@stonehenge.com (Randal L. Schwartz) writes:

I think the target market is people who use whatever POSIXy systems for their own work. To them, git and its dependencies are merely tools to get their job done, and while they might be capable and even willing to match the version requirements if it is absolutely necessary, they'd rather spend their time on what they do best (i.e. write software or documentation or whatever, whose changes are to be managed by git), than installing or upgrading base software just to satisfy git. And I'd like to see their time spent on what they do best, instead of caring and feeding of git, too.

So in that sense, excluding 75% of people with "5.8 only" is not acceptable at all, and covering 75% with "5.6 or better" might be good enough, at least as a starter.

If it is not too much trouble, I'd appreciate it, at least the first half that can be done without taking too much of your time. Is it a mechanical "lint" like thing, that says "line 47 you have list form of pipe open -- that's 5.8 and better"?

Junio> I think the target market is people who use whatever POSIXy Junio> systems for their own work. To them, git and its dependencies Junio> are merely tools to get their job done, and while they might be Junio> capable and even willing to match the version requirements if it Junio> is absolutely necessary, they'd rather spend their time on what Junio> they do best (i.e. write software or documentation or whatever, Junio> whose changes are to be managed by git), than installing or Junio> upgrading base software just to satisfy git. And I'd like to Junio> see their time spent on what they do best, instead of caring and Junio> feeding of git, too.

Yes, this is why I keep pointing out GNU tools being used instead of POSIXy versions. :) Mostly because they break on either my Darwin laptop or my OpenBSD host server.

Junio> So in that sense, excluding 75% of people with "5.8 only" is not Junio> acceptable at all, and covering 75% with "5.6 or better" might Junio> be good enough, at least as a starter.

I think we're in agreement.

Junio> If it is not too much trouble, I'd appreciate it, at least the Junio> first half that can be done without taking too much of your time. Junio> Is it a mechanical "lint" like thing, that says "line 47 you Junio> have list form of pipe open -- that's 5.8 and better"?

The "perlver" tool installed with the "Perl::MinimumVersion" module from the CPAN will do about 80% of it. The other 20% are things that I can use my brain for.

Randal L. Schwartz wrote:

There are a lot of Perl modules we use, so limiting it to 5.5 is probably a showstopper.

I'm very surprised you say that 5.6 is more prevalent than 5.8.

	-hpa

H> There are a lot of Perl modules we use, so limiting it to 5.5 is H> probably a showstopper.

Hmm. I should look at that then. Most Perl Modules are 5.5 compatible, unless they've been written by naive people recently. :)

H> I'm very surprised you say that 5.6 is more prevalent than 5.8.

For individual early adopters, 5.8 is nearly universal. But a lot of my corporate clients upgrade *very* slowly, and are on 5.6 for now.

You are overestimating the update-eagerness of the world.

SunOS 5.8 seems to come with 5.005_03 SunOS 5.9 seems to come with v5.6.1.

("Seems" because I am only 99% sure no-one around here has mucked with the perl in /usr/bin/.)

M.

Randal L. Schwartz wrote:

Well, I think the general target market[1] for Linux probably applies here, as well. On the other hand, the users limited to Perl 5.5 for corporate reasons are probably also prevented from installing new tools for general use, so the difference may not matter very much.

[1] - "World domination", of course.

Ryan Anderson wrote:

Servers with software installation restrictions are rarely used for development either and the server side part of git is all C. So long as that's true I don't think we need to bother very much with staying backwards compatible. It's nice if we manage it, but we shouldn't, imho, bend over backwards for the sake of perl 5.5 (or even 5.6).

Just my two öre. :)

Ryan Anderson <ryan@michonline.com> writes:

Oh, I didn't?

Junio C Hamano wrote:

Sorry, I read after waking up and totally didn't see the "qw" in there.
   You did, my bad.

I have found one error during directory movig: If I move directory with one file somewhere in it this script will try to add target directory instead of file. Commenting lines starting from 190 solve this error. But I don't understand what is the logic behind this case ? Why do target directory checked instead of target file ? Should we replace $dst my $destfiles[0] ?

at line 190 in git-mv:
    if (scalar @srcfiles == 1) {
	if ($overwritten{$dst} ==1) {
	    push @changedfiles, $dst;
	} else {
	    push @addedfiles, $dst;
	}
    }
    else {
	push @addedfiles, @dstfiles;
    }

Alexander Litvinov wrote:

Are you saying this setup
	foodir/somefile.c     <--file
	newdir/               <--directory

with this command
	git-mv foodir/ newdir

tries to create
	newdir/foodir/somefile.c  <-- directory

or does it create
	newdir/somefile.c         <-- file

?

It should create
	newdir/foodir/somefile.c  <-- file

Otherwise it's misbehaving.

Try running it with the -v switch to make it shout out loud what it's trying to do, and then paste the output here.

This is broken. It only checks if there's just one source-file regardless of whether or not it resided in a subdirectory. I'm not exactly fluent in perl so I can't submit a patch, but the src option needs to be directory aware, traverse all source directories and then move the files axing everything but the bottom-most dirname to the destination directory.

Any takers?

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:

It is calling git-update-index --add newdir but actiual file structire is correct: newdir/somefile.c - is a file

On Wednesday 23 November 2005 13:57, Andreas Ericsson wrote:

I still does not understand what this part should do. I know perl enought to fix it but I don't understand the logic.

On Wednesday 23 November 2005 11:21, Alexander Litvinov wrote:

Yes. For git-update-index we have to use the file inside the directory.

I just sent a patch for this.

Josef

On Wednesday 23 November 2005 08:26, Alexander Litvinov wrote:

$src/$dst pairs match the arguments given on the command line of git-mv, And git-mv (as was git-rename) is able to move directory trees around.

It is simplier to keep the granularity at directories, as multiple moves of directories can not overlap each other. If you do it at file level, you have to remote and create directories yourself.

Of course, for git it is done at file level (that are the @srcfiles/@dstfiles arrays), but this is quite trivial as git does not work with directories, but only with files.

Yes, that was part of my patch.

Josef

Alexander> I have made this change. I also belive it will work on earlier perl but I Alexander> can't test this.

This patch looks good back to 5.5. I didn't execute it either, but the Perl version installed in my head gave it a thumbs-up. :)