Re: Restart submodule update --recursive

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Oct 29, 2010, 16:40 UTC
Message-ID: <20101029164055.GA28934@burratino>
In-Reply-To: <4CCAB20A.1000408@prevac.pl>

Hi again,

Lukasz Palczewski wrote:

> Jonathan Nieder wrote:

>> [side note: if you use a newsreader, please reply by mail instead of
>> through gmane so the Cc: list can be preserved.]
>
> Sorry, but I do not use it. I will send You a normal email.

Thanks.

Show 10 quoted lines

> I have added two options to git submodule update:
> --save-progress
> --resume-update
> First one You can use, to save the ./.git/SubmoduleUpdateLog.log the
> progress of updating:
> git submodule update --recursive --save-progress
> The second one is used for skipping updated submodules:
> git submodule update --recursive --resume-update
> 
> The downside of it is, that u can use only one --resume-update.

Does the combination "git submodule update --recursive --save-progress --resume-update" work?

> Let me know if this change will be in next version of git.

I will leave that question to the submodule experts. But note that there is no need to wait: if you change the ". git-sh-setup" line to

	. "$(git --exec-path)/git-sh-setup"

then if you install the modified script as git-mysubmodule anywhere on your $PATH, it will be used to implement

	git mysubmodule update --recursive --save-progress

.  Of course you can run it directly as "git-mysubmodule", too.

diffs are easier to review. If your change is for inclusion, it will also need a Signed-off-by: line; see Documentation/SubmittingPatches.

> cmd_update()
> {

[...]

Show 7 quoted lines

> 		--save-progress)
> 			shift
> 			save_progress=1
> 			if test -e ./.git/SubmoduleUpdateLog.log
> 			then
> 				rm ./.git/SubmoduleUpdateLog.log
> 			fi

Some current temporary files used by git:

	.git/rr-cache/*
	.git/gitk.cache
	.git/rebase-merge/*
	.git/rebase-apply/*

The
	if test -e $foo
	then
		rm $foo
	fi

idiom can be written more simply as

	rm -f $foo

but are you sure that is what you want? Maybe it would be better to

	1) destroy progress when the update is completed
	2) always save updated progress when resuming
	3) complain when --save-progress would require clobbering
	   some existing progress
	4) provide an additional --reset-progress option to start
	   over

FWIW "git am" and "git rebase" already have the ability to resume where they left off, using options with these names:

	git <foo> --continue
	git <foo> --skip
	git <foo> --abort

The --abort options undo the effect of the earlier runs. "git bisect" also keeps some state on progress.

	git bisect (good|bad); # continue
	git bisect skip
	git bisect reset

Show 11 quoted lines

> 	module_list "$@" |
> 	while read mode sha1 stage path
> 	do		
> 		name=$(module_name "$path") || exit
> 		url=$(git config submodule."$name".url)
> 		update_module=$(git config submodule."$name".update)
> 		
> 		if test -n "$submodule_resume"
> 		then
> 			SubmodulePathResume=`pwd`
> 			SubmodulePathResume=$SubmodulePathResume/"$name"

It might be more idiomatic to say

	module_list "$@" |
	...
			(
				prefix="$prefix$path/"

Three benefits:

 - local variables tend to get short, lowercase names
 - this variable looks like a path, but using $name makes it not
   one (if foo/bar is a submodule simple named "bar", for example)
 - using relative paths allows an update to be resumed after moving
   a repository or just accessing it via a different path.

The subshell ensures the old prefix is restored when cmd_update returns. The same effect could also be achieved by saving the old prefix (in case it is important to save a few processes).

> 			
> 			#say $SubmodulePathResume
> 			if test -e ./.git/SubmoduleUpdateLog.log

What if I try to resume and there is no progress information? Would it make sense to keep the progress information at the toplevel, so one could run commands like "submodule update --recursive --resumable" in submodules without affecting the superproject state?

Show 5 quoted lines

> 			then
> 				AFile=`cat "./.git/SubmoduleUpdateLog.log" | grep "$SubmodulePathResume"`
> 				
> 				#say $AFile;			
> 				if test -n "$AFile"

The following could work if each submodule is written to submodule-update.log only if all its submodules have been updated (as in your current code):

		if git grep --no-index -q -F "$prefix" "$GIT_DIR/submodule-update.log"
		then
			say "Skip...
			continue
		fi

I'm not sure fgrep is portable, which is why I use "git grep" here.

This is slower than it could be. Ideally one would only keep the last updated submodule and chop the module_list output there, instead of re-reading the list with each iteration (which is O(size of directory squared)).

Hope that helps, Jonathan