RFC patch, 3 partsfilter-branch: optionally dump all mappings at the end

Jeffrey Phillips Freeman aka tty1 asked on IRC about a way to fix the super-project after rewriting submodules. This quick series provides some extra tricks in the filter-branch toolbox that can be plumbed together to achieve this (see the last commit's message), and should hopefully be general enough to be of other use too.

Comes without docs and tests at this point, hence RFC. It's also only lightly tested, but since Jeffrey has a use-case, I'll leave it up to him to bend it a bit and see if anything breaks.

Thomas Rast (3):
  filter-branch: optionally dump all mappings at the end
  filter-branch: optionally load existing mappings prior to filtering
  filter-branch: support --submodule-filter

 git-filter-branch.sh |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 64 insertions(+), 0 deletions(-)

In some cases, like the sub/super-project filtering attempted in this series, it may be necessary to carry over the mappings from one or more filter-branch run to another.

As the first part to this, make a --dump-map option that dumps a flat text file with lines of the form

  $sha1 $(map $sha1)

after it has established all mappings.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
 git-filter-branch.sh |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

On Freitag, 31. Dezember 2010, Thomas Rast wrote:

Despite the alternative implementation that I suggest below, the commit message (and also the documentation to be written) should state the output format in this way.

Please use

	if is_absolute_path "$OPTARG"
	then
	...

There must be a more efficient way than to spawn an additional process for each mapped SHA1. Since this excercises only one branch in the map() function anyway, how about this:

	( cd "$workdir"/../map/ && ls -1 ) |
	while read sha1; do
		read mapped << $sha1 &&
		echo $sha1 $mapped
	done > "$dump_map"

Though, there is some mechanism that maps one SHA1s to more than one, and my approach might not cover that case.

-- Hannes

In the previous commit filter-branch learned to dump its mappings. Introduce a --load-map option that lets it load map files in the same format.

This option does not accumulate; only the last one counts. Letting it accumulate would require two argument-parsing passes since we need to know the $tempdir before we can establish any maps.

Also, the maps add to all filtered maps. This is a bit of a tradeoff. As it stands, it could be useful to remap refs in another repository using the map file from elsewere. On the other hand, it does mean that it may rewrite more refs than the user intended.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
 git-filter-branch.sh |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

On Freitag, 31. Dezember 2010, Thomas Rast wrote:

Please use is_absolute_path here as well.

-- Hannes

This new filter gets each submodule's current sha1 and path on stdin, separated by a tab. It can then emit a new submodule sha1 and/or path, and filter-branch will:

* if the path differs, remove the submodule at the old path;

* add/replace the new sha1 at the new path.

Additionally, returning an empty new sha1 deletes the submodule.

You can tie this together with the last two commits to filter the super-project after a subproject filtering as follows:

  ( cd sub1 && git filter-branch --dump-map map <filters|args> )
  ( cd sub2 && git filter-branch --dump-map map <filters|args> )
  cat sub1/map sub2/map > map
  git filter-branch --load-map map \
  	--submodule-filter "map $(cut -f1)" \
	<args>

Other use-cases should also be covered since we also pass through the path.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
 git-filter-branch.sh |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)

On Freitag, 31. Dezember 2010, Thomas Rast wrote:

As I said, I'm not particularly fond of a new --submodule-filter because it is just a special --index-filter.

Your implementation is highly problematic:

This 'die' will not do anything useful except to write an error message.

This 'read' is part of a pipe, and as such many shells run it in a sub-shell; the values that it reads do not survive the pipe, hence, the subsequent code does not do what you intend it.

In this case, you can put everything from 'read' to the last 'fi' below inside a block { } because there are no process states that need to survive the pipe.

The whole if-branch is a pipe, and it's parts are run in a sub-shell (although shells are allowed to optimize this). This means that the 'die' calls will only exit the pipe, but not terminate filter-branch. You at least need to have '|| exit' behind the last 'fi' and &&-join the if-statements.

-- Hannes

So im kinda new with all this so bare with me guys. I wanted to figure out how to apply these patches, now i know i can use git to do this with its patch command and such. However i was curious does this exist as a branch somewhere official or semi-official? I currently seem to be using --split-submodule which is itself in a git repo i have (which i want to also find its source repo so i can keep up to date with it). So applying the patch itself might be somewhat troubling due to conflicts, therefore id like to actually merge in a remote branch to keep things easy. So can you guys point me to which repo would be best for me to keep up to date with this would be?

On 12/31/2010 10:29 AM, Thomas Rast wrote:

Please don't top-post. (There's nothing wrong with snipping the whole message if it does not really relate, as in this case.)

Ccs for the patch at the end; I don't really care much but I could roll all of it into a sort of "submodule tools" series for g-f-b, so if you like it, speak up.

Jeffrey Phillips Freeman wrote:

Not really.

You can dig through the mailing list archives and also e.g. gitworkflows to see how Junio handles incoming patches, but for series like this you usually have to apply them yourself. I deliberately flagged it RFC because I wanted to get some feedback ... and because I would have had to spend more time on it for docs&tests.

For others reading this, I wrote --split-submodule also based on an IRC request from Jeffrey. The patch is at the end. But it's way less thought through than the --{dump,load}-map feature. In particular I've been wondering whether it's possible to use the latter to implement --split-submodule as a fairly concise index filter.

You'll get the same conflicts from merging two little branches with the features on them as with a 'git am -3'.

Many patches are never pushed to a public repo (there are some notable exceptions in longer-running work). The hassle of sending and applying email is far outweighed by the ease of review. Many reviews happen without applying the series at all. That I pushed both of them to a public repo was an exception for your convenience.

--- 8< ---
Subject: TOY PATCH: filter-branch --split-submodule

Sometimes it makes sense to split out a path not as a subdirectory (that would be merged by subtree-merge), but as a submodule. Since git objects are just shaped in the right way, this is actually quite easy to do in a way that maintains the correct history relations:

When splitting out DIR in commit C, the submodule commit has tree C:DIR and the rewritten superproject commit C' gets a submodule entry at C:DIR instead.

Parent rewriting is done in the obvious way: submodule commits have their corresponding submodule-changing ancestors as parents. These are easy to fetch since we can basically use $(map C^):DIR.

This is a toy patch because of its terrible interface: you will still have to put the submodule in place. As a start, you can

  git clone . DIR
  ( cd DIR && git reset --hard $(git rev-parse :DIR) )

to get a sub-repo set to the correct commit.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>

Thomas Rast <trast@student.ethz.ch> writes:

The patch from a cursory look feels sane.

Just a style, but I find

	if test -n "$split_submodule"
        then
        	sub_differs= sub_parents= sub_commit=

easier to read. Not a biggie, as the neighbourhood in the script already is infested in the other style, but I thought I'd mention it.

Do we need double quotes around it?

	if test -z "$parents" && test -n "$submodule"
        then
        	sub_differs=t
	fi

If even one of the parents is different, we say "differs"...

... so we can just pick from the first parent and know all of them are the same (could be empty which also is fine). Good.

On Freitag, 31. Dezember 2010, Thomas Rast wrote:

I like the --dump-map and --load-map options. But I think that a --submodule-filter is not necessary; it is just a particular --index-filter, and IMO should be part of the documentation in the form of an example that illustrates how to use the new options.

-- Hannes