[SCRIPT/RFC 0/3] git-commit --onto-parent (three-way merge, no working tree file changes)

Hi to all,

Here`s my humble attempt on (once more) scratching a use case which seems to be desired occasionally through the years, obviously not enough to be actually implemented yet, but might be worth some more attention... :)

For the reference, some past mentions of (more or less) similar ideas are listed at the end of this e-mail[*1*], where I`m also Cc-ing people previously involved, might be still sharing interest - as this is my first "new thread" list message, I`m sorry if this is not an encouraged practice, please let me know.

Approach discussed here could have a few more useful applications, but one seems to be standing out the most - in case where multiple topic branches are temporarily merged for integration testing, it could be very useful to be able to post "hotfix" commits to merged branches directly, _without actually switching to them_ (and thus without touching working tree files), and still keeping everything merged, in one go.

Example starting point is "master" branch with 3 topic branches (A, B, C), to be (throwaway) merged for integration testing inside temporary "test" branch:

(1)        o---o---A (topicA)
          /
         /
        /
    ---o---o---M (master, test, HEAD)
        \   \
         \   o---B (topicB)
          \
           o---o---C (topicC)

This is what we end up with once "master" and topic branches are merged in merge commit M1 inside temporary "test" branch for further integration testing:

(2)        o---o---A (topicA)
          /         \
         /           M1 (test, HEAD)
        /           /||
    ---o---o---M---/ || (master)
        \   \       / |
         \   o---B-/  | (topicB)
          \           |
           o---o---C--/ (topicC)

Upon discovery of a fix needed inside "topicA", hotfix changes X should be committed to "topicA" branch and re-merged inside merge commit M2 on temporary integration "test" branch (previous temporary merge commit M1 is thrown away as uninteresting):

(3)        o---o---A---X (topicA)
          /             \
         /               M2 (test, HEAD)
        /               /||
    ---o---o---M-------/ || (master)
        \   \           / |
         \   o---B-----/  | (topicB)
          \              /
           o---o---C----/ (topicC)

Now, usually advised approach to get from (2) to (3), where one is expected to switch to desired topic branch, commit the change, and then re-merge everything again into integration testing branch, is arguably a bit tiresome, but where it actually fails short is the fact that branch switching back and forth (for each hotfix commit) could possibly keep changing a lot of files otherwise untouched by the hotfix changes we`re committing (but different between branches), causing build systems to needlessly waste what could be significant time compiling them again and again.

Example script proposes using something like this instead:

(4) git commit --onto-parent topicA
     
... in above-mentioned case (2) to commit X onto "topicA" branch 
directly, re-merging all previously merged integration testing topic 
branches at the same time, reaching state (3) without any 
intermediate branch switching (and without touching working tree, 
thus without needless recompilation).

Once integration tests pass, integration test branch will be thrown away and new commits on each topic branch should still be properly tested - we`re just deferring it not to do it in the middle of integration testing, saving some (or a lot) needless rebuild cycles at the same time (or in the first place, even).

Scripts in series:
  [1/3]: setup.sh
  [2/3]: git-merge-one-file--cached
  [3/3]: git-commit--onto-parent.sh

Regards, Buga

[*1*] Some previous list mentions of similar functionality, in order 
 of appearance, latest on top (I kind of remember seeing more, but 
 can`t find them now, please feel free to add here, or notify more 
 people interested in the past):

 - [PATCH/RFC] git-post: the opposite of git-cherry-pick (2017-10-05)
   https://public-inbox.org/git/c6b52120-98bf-d685-6dc0-3c83e9e80d30@kdbg.org/

 - "groups of files" in Git? (2017-07-11)
   https://public-inbox.org/git/CAEcERAz3vYekvJ8SM1FfdAVsP3LMVqA1O3yoJVThvg-0fPtVCg@mail.gmail.com/

 - Making a (quick) commit to another branch (2013-04-27)
   https://public-inbox.org/git/517BDB6D.8040809@cedarsoft.com/

 - Commit to other branch (2010-05-31)
   https://public-inbox.org/git/4C03D9C1.1060404@cedarsoft.com/

 - [RFC] git commit --branch (2006-05-29)
   https://public-inbox.org/git/20060529202851.GE14325@admingilde.org/

 - n-heads and patch dependency chains (2006-04-03)
   https://public-inbox.org/git/4430D352.4010707@vilain.net/

 - Multi-headed branches (hydra? :)) for basic patch calculus (2006-04-02)
   https://public-inbox.org/git/1143950852.21233.23.camel@localhost.localdomain/

On 26/11/2017 23:35, Igor Djordjevic wrote:

To begin with, you can use provided "setup.sh"[*1*] script, putting you straight into position shown on graph (2) above, with addition of tag "A" and remote branch "origin/topicA" so you could try using these as "--onto-parent" values, too.

As seen in there, change "X" is already made and staged, so you can now just run something like:

	git-commit--onto-parent.sh --onto-parent topicA

... to see the logic in action.
 
Instead of "topicA", you may try providing tag "A", remote branch 
"origin/topicA" or even plain commit hash. It`s interesting to add 
"--amend" option into the mix, too, and see what happens. Also, you 
can try using "topicB" and see the commit fail (as it doesn`t merge 
cleanly).

All this while "test.txt" file doesn`t get modified on disk (nor would any other file) - being desired behaviour, as we didn`t actually change anything inside the working tree, but just amended history of how we got here, so recompilation isn`t needlessly triggered :)

p.s. Note these two lines near the end:

	sed -i '4iX1' test.txt  # works with simple patch apply
	sed -i '17iX2' test.txt # needs three-way file merge

You can play with it, commenting out one or the other and observing how it influences "git commit --onto-parent" in regards of the parent provided.

Regards, Buga

[*1*] "setup.sh", can clean previous setup run as well, but commented 
 out here for safety, not to unexpectedly delete something for unwary 
 user.
--- 8< ---
#!/bin/sh

#rm -rf ./.git #rm -f ./test.txt

git init

touch ./test.txt git add -- test.txt

for i in {1..10}
do
	echo $i >>test.txt	
	git commit -am "$i"
done

echo M >>test.txt git commit -am "M"

git checkout -b topicA HEAD~2

for i in 1 2
do
	sed -i "${i}iA${i}" test.txt
	git commit -am "A$i"
done
sed -i '3iA' test.txt
git commit -am "A"
git tag A

# simulate remote branch mkdir -p ./.git/refs/remotes/origin && echo $(git rev-parse HEAD^0) >$_/topicA

git checkout -b topicB master^

sed -i '4iB1' test.txt git commit -am "B1" sed -i '5iB' test.txt git commit -am "B"

git checkout -b topicC master~2

for i in 1 2
do
	j=`expr "$i" + 5`
	sed -i "${j}iC${i}" test.txt
	git commit -am "C$i"
done
sed -i "8iC" test.txt
git commit -am "C"

git checkout -b test master git merge --no-edit topicA topicB topicC

sed -i '4iX1' test.txt # works with simple patch apply sed -i '17iX2' test.txt # needs three-way file merge git add -- test.txt

echo git log --all --decorate --oneline --graph echo git diff --cached

Finally, "git-commit--onto-parent.sh"[*1*] shows an initial script version for you to examine, test out and hopefully comment on :)

Especially interesting part might be index-only three-way file merge, through usage of "git-merge-one-file--cached" script. Of course, this still only works for some trivial resolutions, where in case of more complex ones, involving unresolved conflicts, we back-out and fail. Still, it should be more powerful than `git-apply`.

Consider this proof of concept and work in progress, an idea where I`d like feedback on everything you come up with or find interesting, even parameter name possibly used instead of "--onto-parent" (and its short version), or approach in general.

For example, it might make sense to separate commit creation (on current HEAD`s parent) and its actual re-merging into integration test branch, where "--remerge" (or something) parameter would be used on top of "--onto-parent" to trigger both, if/when desired.

Another direction to think in might be introducing more general "--onto" parameter, too (or instead), without "parent" restriction, allowing to record a commit on top of any arbitrary commit (other than HEAD). This could even be defaulted to "git commit <commit-ish>" (no option needed), where current "git commit" behaviour would then just be a special case of omitted <commit-ish> defaulting to HEAD, aligning well with other Git commands sharing the same behaviour.

Alas, rewind to present...

Please do note that I`m still relatively new to Git, and pretty new to both Linux and scripting in general (on Windows as well), and the whole concept of open-source software contributing, even, so please bare with me (or at least don`t get upset too much, lol), and do feel free to share your thoughts and remarks, even the trivial or harsh ones -- I`m grateful to learn and expand my knowledge, hopefully producing something useful in return :) Heck, might be I`m totally off-track here as well.

p.s. For some context - nowadays I mostly work in Delphi, and occasionally in C#, though through last 20 years I`ve been involved with C, Pascal, Basic, but also PHP, JavaScript, and whatnot - even good old assembly from time to time, when needed :)

Regards, Buga

[*1*] "git-commit--onto-parent.sh", probably too heavily commented in 
 the first place, but as I`m new to everything here I kind of feel the 
 plain words might unfortunately describe my intention a bit better 
 than my code, for now at least.
--- 8< ---
#!/bin/sh
#
# Copyright (c) 2017 Igor Djordjevic

i=$#
while test $i != 0
do
	#
	# Parameter parsing might be uninteresting here, as the whole
	# script is currently just a wrapper around `git commit`, for a
	# functionality that conceptually belongs there directly.
	#
	case "$1" in
	--onto-parent=*)
		onto_parent="${1#*=}"

		shift && i=$(expr $i - 1)
		;;
	--onto-parent)
		shift && i=$(expr $i - 1)
		onto_parent="$1"

		shift && i=$(expr $i - 1)
		;;
	-a|--a|--al|--all)
		all=t

		#
		# For now, `git commit` "--all" option is special-cased in
		# terms of being stripped out of the original command line
		# (to be passed to `git commit`) and processed manually, as
		# once commit is to be made, due to states of index and
		# working tree, "--all" is most probably NOT what the user
		# wants nor expects ;)
		#
		shift && i=$(expr $i - 1)
		;;
	*)
		# parameters to pass down to `git commit`
		set -- "$@" "$1"
		shift && i=$(expr $i - 1)
		;;
	esac
done

main () {
	#
	# Store current HEAD (ref or commit) and verify that
	# --onto-parent is valid and amongst its parents.
	#
	head="$(git symbolic-ref --short --quiet HEAD)" ||
	head="$(git rev-parse HEAD^0)" &&
	verify_onto_parent "$head" "$onto_parent" || exit 1

	#
	# As both HEAD and "--onto-parent" could be refs, where underlying
	# commits could change, store original commits for later parents
	# processing, getting updated parent list for new/updated
	# merge commit.
	#
	head_commit="$(git rev-parse "$head"^0)" &&
	onto_parent_commit="$(git rev-parse "$onto_parent"^0)" || exit 1

	#
	# Custom processing of stripped "--all" parameter - if we were to
	# just pass it to `git commit`, "--all" would most probably yield
	# an unexpected result in the eyes of the user, as it would include
	# _all changes from all the other merge commit parents as well_,
	# not just the changes we may actually wanted to "push down"
	# (commit) onto specified parent (what would `git diff` show),
	# due to state of index and working tree at the time of commit.
	#
	if test -n "$all"
	then
		git add --update
	fi

	#
	# Abort if no cached changes, nothing to be committed.
	#
	git diff --cached --quiet
	if test $? -eq 0
	then
		printf >&2 '%s\n' "error: no changes added to commit"
		exit 1
	fi

	#
	# Backup current index to be restored (and committed) in the end.
	#
	merge_index="$(git write-tree)" || exit 1

	#
	# Reset index to destination parent (without touching working
	# tree), and try applying cached changes.
	#
	# In case changes do not apply cleanly onto desired parent, abort.
	#
	apply_changes "$head_commit" "$onto_parent_commit" "$merge_index" "$onto_parent" || exit 1

	#
	# Move HEAD to specified parent (without touching working tree)
	# to prepare to record a commit there, and make the commit,
	# passing parameters through to `git commit`.
	#
	# In case of error, or when `git commit` didn`t actually make
	# the commit, like when --dry-run parameter is provided (for
	# example), abort, as there is nothing to do - no new commit, no
	# need to produce the "updated" merge commit, either.
	#
	# Note that we don`t abort right away, as restoring original
	# index and HEAD position is needed all the same, so we
	# potentially abort only once that is done, a bit further below.
	#
	# [ This is something that could be thought of a bit more,
	# might be forbidding passing through of some `git commit`
	# parameters in the first place, like --dry-run...? ]
	#
	move_head "$onto_parent" &&
	git commit "$@"
	if test $? -ne 0 || {
		new_parent_commit="$(git rev-parse HEAD^0)"
		test "$new_parent_commit" = "$onto_parent_commit"
	}
	then
		no_commit=t
	fi

	#
	# Remove entry from HEAD reflog, not to pollute it with
	# uninteresting in-between steps we take, leaking implementation
	# details to end user.
	#
	# We do left it inside corresponding branch reflog where commit
	# is made (if $onto_parent was a branch), though, as that`s where
	# it still matters.
	#
	git reflog delete HEAD@{0}

	#
	# Restore original index state and move HEAD to original position,
	# (still not touching working tree), aborting if previously
	# signalled.
	#
	git read-tree --reset "$merge_index" &&
	move_head "$head" &&
	test -z "$no_commit" || exit 1

	#
	# Drop original HEAD merge commit to have it replaced by
	# upcoming "updated" merge commit.
	#
	# This step is needed for eventually getting an expected merge
	# message out of "git fmt-merge-msg", as it seems HEAD dependent
	# as well, beside being input format picky already...?
	#
	#git update-ref --create-reflog -m "reset: moving to HEAD^" HEAD HEAD^ || exit 1
	reflog_ref="$(git symbolic-ref --short --quiet HEAD)"
	git update-ref HEAD HEAD^ || exit 1

	#
	# Remove both HEAD and underlying reference reflog entries this
	# time, as here we really want to mask previous step completely,
	# being taken just to satisfy "git fmt-merge-msg" expectations.
	#
	if test -n "$reflog_ref"
	then
		git reflog delete "$reflog_ref"@{0}
	fi
	git reflog delete HEAD@{0}

	#
	# Prepare "updated" merge commit message and parent list.
	#
	if test -n "$(git rev-parse --verify --quiet $head_commit^2^{commit})"
	then
		merge_parents="$(get_merge_parents "$head_commit" "$onto_parent" "$onto_parent_commit" "$new_parent_commit")" &&
		merge_message="$(get_merge_message "$merge_parents")" || exit 1
	else
		#
		# As we`re actually selling the option as "--onto-parent" and
		# not "--onto-MERGE-parent", we might as well properly support
		# a special case where HEAD commit is not a merge.
		#
		# Existing HEAD commit will come after commit to be made,
		# basically being kind of rebased onto new commit (but still
		# not touching working tree), where we can then also reuse
		# original HEAD commit authorship, and message, too (instead
		# of building a merge one).
		#
		merge_parents="$new_parent_commit" &&
		merge_message="$(git show -s --format=%B "$head_commit")" &&
		reuse_authorship $head_commit || exit 1
	fi
	merge_parent_commits="$(get_merge_parent_commits "$merge_parents")" &&

	#
	# Do the actual commit, updating HEAD accordingly.
	#
	merge_commit="$(printf '%s\n' "$merge_message" |
	git commit-tree "$merge_index" $merge_parent_commits)" &&
	git update-ref --create-reflog -m "$merge_message" HEAD "$merge_commit" || exit 1
}

verify_onto_parent () {
	#
	# $1 starting point head (ref or commit)
	# $2 parent of $1 to commit onto (ref or commit)
	#
	local head="$1"
	local onto_parent="$2"

	if test -z "$onto_parent"
	then
		printf >&2 '%s\n' "error: no parent provided"
		printf >&2 '%s\n' "(use \"--onto-parent <commit-ish>\""
		return 1;
	fi

	if test -z "$(git rev-parse --verify --quiet "$onto_parent"^{commit})"
	then
		printf >&2 '%s\n' "error: '$onto_parent' not valid commit object"
		return 1
	fi

	local onto_parent_commit="$(git rev-parse "$onto_parent"^0)"
	for parent_commit in $(git rev-parse $head^@)
	do
		if test "$parent_commit" = "$onto_parent_commit"
		then
			return 0
		fi
	done

	printf >&2 '%s\n' "error: '$onto_parent' not parent of '$head'"
	return 1
}

apply_changes () {
	#
	# $1 original/starting point HEAD commit
	# $2 parent commit of $1 to apply changes to
	# $3 index with changes on top of $1 to apply/merge onto $2
	# $4 original parameter value of $2 (ref or commit), used for
	#    prettier message only
	#
	local head_commit="$1"
	local onto_parent_commit="$2"
	local merge_index="$3"
	local onto_parent="$4"

	git read-tree --reset $onto_parent_commit &&

	#
	# Attempt simple patching first - take differences between
	# $head_commit and $merge_index and try applying to current index
	# (previously reset to $onto_parent_commit).
	#
	git diff-tree --binary --patch --find-renames --find-copies $head_commit $merge_index |
	git apply --cached 2>/dev/null &&
	return 0

	printf '%s\n' "Unable to apply cleanly onto '$onto_parent', trying simple merge"

	#
	# A bit more aggressive approach - try merging with resolving
	# trivial conflicts on tree level only (involving file as a whole,
	# no conflicts inside file itself).
	#
	# Note that we take $head_commit as merge-base, producing such
	# three-way merge result that basically all changes between
	# $onto_parent_commit and $head_commit are reversed, as they`re
	# also included inside $merge_index, where only differences
	# between $head_commit and $merge_index are applied (in a
	# three-way merge manner) to $onto_parent_commit, being exactly
	# what we want here.
	#
	git read-tree -i -m --aggressive $head_commit $onto_parent_commit $merge_index || exit 1
	git write-tree >/dev/null 2>&1 &&
	return 0

	printf '%s\n' "Simple merge did not work, trying automatic merge"

	#
	# Final attempt - try merging with resolving trivial conflicts on
	# file level, too (conflicts inside file itself).
	#
	# Notice usage of "git-merge-one-file--cached" script here, being
	# a slightly tweaked version of original "git-merge-one-file",
	# not touching working tree but stuffing trivial three-way
	# file merge resolution back into index directly.
	#
	# If still left with conflicts that need to be resolved manually,
	# abort... and go home, you`re drunk.
	#
	if ! git merge-index -o git-merge-one-file--cached -a
	then
		# abort, cleanup
		git read-tree --reset $merge_index
		exit 1
	fi
}

move_head () {
	#
	# $1 destination ref or commit
	#
	# Move HEAD to $1 without touching the working tree.
	#
	# Kind of "soft checkout", where original "git checkout" touches
	# the working tree, and "git reset --soft" does not move HEAD,
	# both undesired here.
	#
	local destination="$1"

	local destination_commit="$(git rev-parse --verify --quiet $destination^0)" ||
	{
		printf >&2 '%s\n' "fatal: invalid reference: $destination"
		return 1
	}
	#local reflog_message="$(get_checkout_reflog_message $destination)"
	local destination_ref="$(git rev-parse --symbolic-full-name $destination)"

	case "$destination_ref" in
	refs/heads/*)
		# can`t use "update-ref --no-deref" as it writes commit only,
		# instead of ref, essentially detaching HEAD to that commit
		#git symbolic-ref -m "$reflog_message" HEAD "$destination_ref"
		git symbolic-ref HEAD "$destination_ref"
		;;
	refs/tags/*|\
	refs/remotes/*|\
	"")
		# can`t use "symbolic-ref" as it refuses to write commit only,
		# expecting a valid ref instead (value inside "refs/")
		#git update-ref --create-reflog -m "$reflog_message" --no-deref HEAD "$destination_commit"
		git update-ref --no-deref HEAD "$destination_commit"

		# mask this step as end-user uninteresting implementation detail
		git reflog delete HEAD@{0}
		;;
	*)
		printf >&2 '%s\n' "fatal: invalid reference: $destination_ref"
		return 1
		;;
	esac

	return 0
}

get_merge_parents () {
	#
	# $1 original/starting point HEAD commit
	# $2 parent of $1 to commit onto (ref or commit)
	# $3 original commit of $2 (if $2 is ref, otherwise equals $2)
	# $4 new commit (onto $2, to be new merge parent)
	#
	# Walk original merge commit parents to find the one we`re posting
	# onto (or amending, even), and update/replace it accordingly with
	# new commit, becoming a new parent of upcoming new/updated merge
	# commit.
	#
	# Where possible, prefer taking ref over commit, making for a
	# prettier merge commit message.
	#
	local head_commit="$1"
	local onto_parent="$2"
	local onto_parent_old_commit="$3"
	local new_parent_commit="$4"
	local merge_parents=

	local onto_parent_new_commit="$(git rev-parse $onto_parent^0)"

	for parent_commit in $(git rev-parse $head_commit^@)
	do
		local merge_parent=

		if test "$parent_commit" = "$onto_parent_old_commit"
		then
			if test "$onto_parent_new_commit" = "$new_parent_commit"
			then
				# $onto_parent is a branch (updateable ref)
				merge_parent="$onto_parent"
			else
				merge_parent="$new_parent_commit"
			fi
		else
			parent_ref="$(git for-each-ref --points-at $parent_commit --count=1 --format="%(refname)")"
			if test -n "$parent_ref"
			then
				merge_parent="$parent_ref"
			else
				merge_parent="$parent_commit"
			fi
		fi

		# echo to flatten whitespace
		merge_parents="$(echo $merge_parents $merge_parent)"
	done

	if test -n "$merge_parents"
	then
		printf '%s\n' "$merge_parents"
		return 0
	else
		return 1
	fi
}

get_merge_message () {
	#
	# $@ merge_parents
	#
	# Provide to-be merge commit message using
	# existing `git fmt-merge-msg` machinery.
	#
	local merge_heads="$(get_merge_heads $@)" &&
	local merge_message="$(printf "$merge_heads" | git fmt-merge-msg)"

	if test -n "$merge_message"
	then
		printf '%s\n' "$merge_message"
		return 0
	else
		return 1
	fi
}

get_merge_heads () {
	#
	# $@ merge_parents
	#
	# Provide input for `git fmt-merge-msg` to get
	# nicely formatted merge commit message.
	#
	# Final result loosely mimics FETCH_HEAD file layout.
	#
	local merge_heads=
	local merge_head=

	#
	# Skip the first parent, as that is the original merge
	# destination, where we`re only interested in parents to be
	# merged into it.
	#
	shift

	for merge_parent in $@
	do
		local merge_parent_ref="$(git rev-parse --symbolic-full-name $merge_parent)"
		local merge_parent_commit="$(git rev-parse $merge_parent)"

		case "$merge_parent_ref" in
		refs/heads/*)
			merge_head="$merge_parent_commit\t\tbranch '${merge_parent_ref#refs/heads/}' of ."
			;;
		refs/tags/*)
			merge_head="$merge_parent_commit\t\ttag '${merge_parent_ref#refs/tags/}' of ."
			;;
		refs/remotes/*)
			merge_head="$merge_parent_commit\t\tremote-tracking branch '${merge_parent_ref#refs/remotes/}' of ."
			;;
		*)
			merge_head="$merge_parent_commit\t\t'$(git rev-parse --short $merge_parent_commit)' of ."
			;;
		esac

		merge_heads="$merge_heads$merge_head\n"
	done

	if test -n "$merge_heads"
	then
		# '\n' already appended
		printf '%s' "$merge_heads"
		return 0
	else
		return 1
	fi
}

reuse_authorship () {
	#
	# $1 commit to reuse authorship from
	#
	local commit="$1"

	GIT_AUTHOR_NAME="$(git show -s --format=%an $commit)"
	GIT_AUTHOR_EMAIL="$(git show -s --format=%ae $commit)"
	GIT_AUTHOR_DATE="$(git show -s --format=%at $commit)"

	export GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_AUTHOR_DATE
}

get_merge_parent_commits () {
	#
	# $@ merge_parents (might contain ref)
	#
	# Provide to-be merge commit parent parameters
	# in format suitable for `git commit-tree`.
	#
	local merge_parent_commits=

	for merge_parent in $@
	do
		merge_parent_commits="$(printf '%s\n' "$merge_parent_commits -p $(git rev-parse $merge_parent^0)")"
	done

	if test -n "$merge_parent_commits"
	then
		printf '%s\n' "$merge_parent_commits"
		return 0
	else
		return 1
	fi
}

get_checkout_reflog_message () {
	#
	# $1 destination ref or commit
	#
	local destination="$1"
	local source=

	source="$(git symbolic-ref --short --quiet HEAD)" ||
	source="$(git rev-parse HEAD^0)"

	printf '%s' "checkout: moving from $source to $destination"
}

main "$@"

Original "git-merge-one-file" script is slightly tweaked here into "git-merge-one-file--cached"[*1*] to allow (still trivial) _index-only_ three-way file merge, not touching files inside working tree.

Proof of concept, not thoroughly tested, original content left in, commented out. Feel free to comment/polish.

To make it available, I guess it would be best placed beside existing "git-merge-one-file" script...?

Regards, Buga

[*1*] "git-merge-one-file--cached" --- 8< --- #!/bin/sh # # Copyright (c) Linus Torvalds, 2005 # # --- # Original "git-merge-one-file" script slightly tweaked into # "git-merge-one-file--cached" to allow (still trivial) index-only # three-way file merge, not touching files inside working tree. # # Proof of concept, not thoroughly tested, original content left in, # commented out. # --- # # This is the git per-file merge script, called with # # $1 - original file SHA1 (or empty) # $2 - file in branch1 SHA1 (or empty) # $3 - file in branch2 SHA1 (or empty) # $4 - pathname in repository # $5 - original file mode (or empty) # $6 - file in branch1 mode (or empty) # $7 - file in branch2 mode (or empty) # # Handle some trivial cases.. The _really_ trivial cases have # been handled already by git read-tree, but that one doesn't # do any merges that might change the tree layout.

USAGE='<orig blob> <our blob> <their blob> <path>' USAGE="$USAGE <orig mode> <our mode> <their mode>" LONG_USAGE="usage: git merge-one-file $USAGE

Blob ids and modes should be empty for missing files."

SUBDIRECTORY_OK=Yes . git-sh-setup cd_to_toplevel require_work_tree

if test $# != 7
then
	echo "$LONG_USAGE"
	exit 1
fi

case "${1:-.}${2:-.}${3:-.}" in
#
# Deleted in both or deleted in one and unchanged in the other
#
"$1.." | "$1.$1" | "$1$1.")
	if { test -z "$6" && test "$5" != "$7"; } ||
	   { test -z "$7" && test "$5" != "$6"; }
	then
		echo "ERROR: File $4 deleted on one branch but had its" >&2
		echo "ERROR: permissions changed on the other." >&2
		exit 1
	fi

	if test -n "$2"
	then
		echo "Removing $4"
	# else
		# read-tree checked that index matches HEAD already,
		# so we know we do not have this path tracked.
		# there may be an unrelated working tree file here,
		# which we should just leave unmolested.  Make sure
		# we do not have it in the index, though.
		# exec git update-index --remove -- "$4"
	fi
	# if test -f "$4"
	# then
		# rm -f -- "$4" &&
		# rmdir -p "$(expr "z$4" : 'z\(.*\)/')" 2>/dev/null || :
	# fi &&
		exec git update-index --remove --cacheinfo "$6","$2","$4"
	;;

#
# Added in one.
#
".$2.")
	# the other side did not add and we added so there is nothing
	# to be done, except making the path merged.
	exec git update-index --add --cacheinfo "$6","$2","$4"
	;;
"..$3")
	echo "Adding $4"
	if test -f "$4"
	then
		echo "ERROR: untracked $4 is overwritten by the merge." >&2
		exit 1
	fi
	git update-index --add --cacheinfo "$7","$3","$4" # &&
		# exec git checkout-index -u -f -- "$4"
	;;

#
# Added in both, identically (check for same permissions).
#
".$3$2")
	if test "$6" != "$7"
	then
		echo "ERROR: File $4 added identically in both branches," >&2
		echo "ERROR: but permissions conflict $6->$7." >&2
		exit 1
	fi
	echo "Adding $4"
	git update-index --add --cacheinfo "$6","$2","$4" # &&
		# exec git checkout-index -u -f -- "$4"
	;;

# # Modified in both, but differently. # "$1$2$3" | ".$2$3")

	case ",$6,$7," in
	*,120000,*)
		echo "ERROR: $4: Not merging symbolic link changes." >&2
		exit 1
		;;
	*,160000,*)
		echo "ERROR: $4: Not merging conflicting submodule changes." >&2
		exit 1
		;;
	esac

	src1=$(git unpack-file $2)
	src2=$(git unpack-file $3)
	case "$1" in
	'')
		echo "Added $4 in both, but differently."
		orig=$(git unpack-file e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
		;;
	*)
		echo "Auto-merging $4"
		orig=$(git unpack-file $1)
		;;
	esac

	git merge-file "$src1" "$orig" "$src2"
	ret=$?
	msg=
	if test $ret != 0 || test -z "$1"
	then
		msg='content conflict'
		ret=1
	fi

	# Create the working tree file, using "our tree" version from the
	# index, and then store the result of the merge.
	# git checkout-index -f --stage=2 -- "$4" && cat "$src1" >"$4" || exit 1
	# rm -f -- "$orig" "$src1" "$src2"

	if test "$6" != "$7"
	then
		if test -n "$msg"
		then
			msg="$msg, "
		fi
		msg="${msg}permissions conflict: $5->$6,$7"
		ret=1
	fi

	if test $ret = 0
	then
		merge_blob=$(git hash-object -w --path="$4" -- "$src1")
	fi
	rm -f -- "$orig" "$src1" "$src2"

	if test $ret != 0
	then
		echo "ERROR: $msg in $4" >&2
		exit 1
	fi
	# exec git update-index -- "$4"
	exec git update-index --cacheinfo "$6","$merge_blob","$4"
	;;

*)
	echo "ERROR: $4: Not handling case $1 -> $2 -> $3" >&2
	;;
esac
exit 1

Am 26.11.2017 um 23:35 schrieb Igor Djordjevic:

I my opinion, putting the focus on integration merge commits and the desire to automate the re-merge step brings in a LOT of complexity in the implementation for a very specific use-case that does not necessarily help other cases.

For example, in my daily work, I have encountered situations where, while working on one topic, I made a hot-fix for a different topic. There is no demand for a merge step in this scenario.

In your scenario above, it would certainly not be too bad if you forgo the automatic merge and have the user issue a merge command manually. The resulting history could look like this:

(3)         o---o---A---X    (topicA)
            /         \   \
           /           M1--M2 (test, HEAD)
          /           /||
      ---o---o---M---' ||     (master)
          \   \       / |
           \   o-----B /      (topicB)
            \         /
             o---o---C        (topicC)

I.e., commit --onto-parent A produced commit X, but M2 was then a regular manual merge. (Of course, I am assuming that the merge commits are dispensible, and only the resulting tree is of interest.)

Moreover, you seem to assume that an integration branch is an octopus merge, that can be re-created easily. I would say that this a very, very exceptional situation.

----

At this point, I spent five minutes thinking of how I would use commit --onto-parent if I did not have git-post.

While on the integration branch, I typically make separate commits for each fix, mostly because the bugs are discovered and fixed not simultaneously, but over time. So, I have a small number of commits that I distribute later using my git-post script. But that does not have to be so. I think I could work with a git commit --onto-parent feature as long as it does not attempt to make a merge commit for me. (I would hate that.)

Sometimes, however I have two bug fixes in the worktree, ready to be committed. Then the ability to pass pathspec to git commit is useful. Does your implementation support this use case (partially staged worktree changes)?

Thanks, -- Hannes

Hi Johannes,

On 27/11/2017 22:54, Johannes Sixt wrote:

It might seem more complex than it is, until you examine the guts to see how it really works :)

Basic concept is pretty simple, as I actually don`t automate anything, at least not in terms of what would manual user steps look like - for example, there`s no real re-merge step in terms of actually redoing the merge, I just reuse what was already there in a very clean way, I would think (supported by my current, humble knowledge, still).

The only merge that could possibly ever happen is upon committing desired subset of changes onto parent, and that shouldn`t be too complex by definition, otherwise that commit doesn`t really belong there in the first place, if it can`t be meaningfully applied where we want it (for now, at least).

That said, the whole operation of "posting on parent and re-merging everything", the way it looks like from the outside, could end just with a simple diff-apply-commit-commit internally, no merges at all. Only if simple `git apply` fails, we try some trivial merging - and all that inside separate (parent) index only, not touching original HEAD index nor working tree, staying pristine for the whole process, untouched.

Once done, you should be in the very same situation you started from, nothing changed, just having your history tweaked a bit to tell a different story on how you got there (now including a commit you posted on your HEAD`s parent).

Otherwise, I agree that explained use case might be a bit specific, but that is only because I recognized that one to be the most interesting to initially present (not to say one of more complex cases) - to me, at least, but it is certainly not the only one.

Don`t let "usual/preferred/recommended" Git workflow distract you too much - one of the reasons I made this is because it also allows _kind of_ "vanilla Git" patch queue, where you can quickly work on top of the merge head, pushing commits onto parents below, being tips of your "queues", putting you up to speed without a need to ever switch a branch (hypothetically), until satisfied with what you have, where you can slow down and polish each branch separately, as usual.

Like working on multiple branches at the same time, in the manner similar to what `git add --patch` allows in regards to working on multiple commits at the same time. This just takes it on yet another level... hopefully :)

I see - and what you`re asking for is what I already envisioned and hoped to get some more feedback about, here`s excerpt from [SCRIPT/RFC 3/3] git-commit--onto-parent.sh[1] (I guess you didn`t have time to checked that one yet?):

  For example, it might make sense to separate commit creation (on 
  current HEAD`s parent) and its actual re-merging into integration 
  test branch, where "--remerge" (or something) parameter would be used 
  on top of "--onto-parent" to trigger both, if/when desired.
  
  Another direction to think in might be introducing more general 
  "--onto" parameter, too (or instead), without "parent" restriction, 
  allowing to record a commit on top of any arbitrary commit (other 
  than HEAD). This could even be defaulted to "git commit <commit-ish>" 
  (no option needed), where current "git commit" behaviour would then 
  just be a special case of omitted <commit-ish> defaulting to HEAD, 
  aligning well with other Git commands sharing the same behaviour.

So I definitely look forward decoupling these two ((1) commit to parent and (2) remerge), with enough discussion flowing :)

Heck, even "to parent" is an artificial/imposed restriction now, in reality you could commit on top of any other commit you want (without switching branches)... but let`s take one step at a time.

Just note that omitting the remerge step is what actually makes the logic more complex, as we now need to change the original situation, too, both HEAD index and working tree, to remove changes which we committed elsewhere (without "merging" back in).

But it is interesting case to look into ;)

Actually, I make no assumptions - head of "integration branch" doesn`t even have to be a merge commit. And as we are not really "remerging" anything, there is no need to recreate anything ;) (what I tried to explain at the beginning of this e-mail).

The sole point is that your current situation inside HEAD doesn`t change at all, no matter how complex it is, we just rewrite history on how we got there (to include new commit onto desired parent).

I would be interested in looking into this, thanks for your feedback.

But, thinking about it more, you do realize that, without "updated merge", that would be removing committed changes from where you`re currently working at (once committed where you want it), is that what you expect?

Because it seems a bit strange, as you loose overview on how all those commits work together, and what you did so far, even, so some future merge might yield unexpected conflicts, which could have been avoided.

From what I understand, your `git-post` makes a commit where you are _and copies_ it over to destination, so you end up with same changes in two places (two commits). More, you seem to keep working on top of your previous commit(s), which seems logical, later "posting" it where desired - so after the fact. It also means you do still have all your "fix" commits in HEAD while you work on them (and until you git-post and abandon them, discarding the integration branch).

But `git commit --onto-parent` proposed here does the commit on destination only, where later "remerge" is crucial part of still keeping that commit changes in the current HEAD working tree as well, so you can still keep working on top of them.

Could it be that you`re looking at `git commit --onto-parent` from an unexpected perspective (through your `git-post`, being conceptually different)...?

Though your use case graph (3) above should mean you`re very well aware of implications, I guess, so I don`t know... :)

From what the idea is, it should work with anything that already works with plain `git commit`, as script is currently just a wrapper around it, providing additional functionality to speed things up.

So yes, `--onto-parent` works with `--patch`, `--amend`, etc :) If not, that`s a bug which I`d appreciate being reported.

Might be some `git commit` options don`t have sense in combination with `--onto-parent` as well, so those could be restricted, once recognized.

... aaand that said, I`ve just noticed that pathspec doesn`t actually work, for no good reason. If you add path through `git add` first, it will work, but not through `git commit --onto-parent -- <pathspec>` directly, which I need to address. Currently, it complains about no changes added to commit. Thanks for spotting! ;)

Thank you for your time and valuable feedback, and one from a real life use-case perspective.

Regards, Buga

[1] https://public-inbox.org/git/d5f243a5-6e35-f3fc-4daf-6e1376bef897@kdbg.org/T/#m72f45ad7a8f1c733266a875bca087ee82cc781e7

Am 28.11.2017 um 02:15 schrieb Igor Djordjevic:

Ok, then please explain, how this process should work in my workflow and with the `commit --onto-parent` feature that you have in mind. I have an integration branch (which is a throw-away type, so you can mangle it in any way you want); it is a series of merges:

  ...A    ...C          <- topics A, C
      \       \
    ---o---o---o---o    <- integration
          /       /
      ...B    ...D      <- topics B, D

Now I find a bug in topic B. Assume that the merges of C and D have textual conflicts with the integration branch (but not with B) and/or may be evil. What should I do?

With git-post, I make a fixup commit commit on the integration branch, then `git post B && git merge B`:

  ...A    ...C                  <- topics A, C
      \       \
    ---o---o---o---o---f---F    <- integration
          /       /       /
      ...B    ...D       /      <- topic D
          \             /
           f'----------'        <- topic B

The merge F does not introduce any changes on the integration branch, so I do not need it, but it helps keep topic B off radar when I ask `git branch --no-merged` later.

'kay, I'm not eagerly waiting for this particular next level (I prefer to keep things plain and simple), but I would never say this were a broken workflow. ;)

I did have a brief look, but I stopped when I saw

	# Remove entry from HEAD reflog, not to pollute it with
	# uninteresting in-between steps we take, leaking implementation
	# details to end user.

It's a clear sign for me that's something wrong. It is not just reflogs that can become stale, but all operations that follow the `git commit` can fail. How do you clean up such a mess?

The "complex situation" I had in mind was not so much about the user's view, but how the implementation has to work when there are errors.

Just to clarify: git-post only copies existing commits, but does not make the original itself.

That may be the case. I'm very interested in your answer to my challenge above.

-- Hannes

Hi Hannes,

On 29/11/2017 20:11, Johannes Sixt wrote:

But you`re not committing (posting) on your HEAD`s (direct) parent in the first place (topic B), so `commit --onto-parent` isn`t right tool for the job... yet :)

To make it easier to explain, I marked your integration branch initial head with "I" in the quote above (commit merging-in branch D), and marked commit merging-in branch C with "E".

HEAD being currently on commit "I", you can only use `--onto-parent` option to commit onto "E" or "D", being parents of "I".

To work with `--onto-parent` and be able to commit on top of any of the topic branches, you would need a situation like this instead:

 (1)  ...C      <- topic C
         |
    ...A |      <- topic A
        \|
      ...o I    <- integration
        /|
    ...B |      <- topic B
         |
      ...D      <- topic D

With `commit --onto-parent` you would skip `git post B && git merge B` steps, where "fixup commit" would be done with `--onto-parent B`, So you end up in situation like this:

 (2)      ...C      <- topic C
             |
        ...A |      <- topic A
            \|
          ...o I'   <- integration
            /|
    ...B---f |      <- topic B
             |
          ...D      <- topic D

State of index and working tree files in your F and my I' commit is exactly the same (I' = I + f), where in my case (2) history looks like "f" was part of topic B from the start, before integration merge happened.

BUT, all this said, I understand that your starting position, where not all topic branches are merged at the same time (possibly to keep conflict resolution sane), is probably more often to be encountered in real-life work... and as existing `--onto-parent` machinery should be able to support it already, I`m looking forward to make it happen :)

Once there, starting from your initial position:

... and doing something like `git commit --onto B --merge` would yield:
 
 (3) ...A    ...C            <- topics A, C
         \       \ E
       ---o---o---o---o I'   <- integration
             /       /|
         ...B    ...D |      <- topic D
             \        |
              f-------'      <- topic B

... where (I' = I + f) is still true. If that`s preferred in some cases, it could even look like this instead:

 (4) ...A    ...C             <- topics A, C
         \       \ E  I
       ---o---o---o---o---F   <- integration
             /       /   /
         ...B    ...D   /     <- topic D
             \         /
              f-------'       <- topic B

... where F(4) = I'(3), so similar situation, just that we don`t discard I but post F on top of it.

Good thing is all necessary logic should already be in place, I just need to think a bit about the most sane user interface, and get back to you. Thanks for invaluable input so far :)

Of course, do feel free to drop any ideas you come up with as well, on how `git commit` user interface/options leading to (3) or (4) should look like (they could both be supported).

Hehe, thanks, I guess :) Simplicity, but user-oriented, is the major point here, where you can work on unrelated patch series at the same time (patch queues?), without a need to switch branches, while you still make "per series" history as you go, without a need to do it after the fact.

It`s a matter of possibilities (where Git usually excels), so one can choose for himself, depending on the situation at hand.

I would respectfully disagree about it being wrong per se, but that said, "cleaning up" reflogs was a last minute design decision so we don`t show uninteresting states - from user`s point of view, all that happened is commit on top of --onto-parent, and merging that commit into where he already was (instead of the commit he was on).

Implementation details on how we got there should be irrelevant, even more if we failed to do so, cluttering reflog with unimportant leftovers.

Commands are chained, and in case _anything_ goes wrong, we just bail out and restore original index and HEAD position. As working tree is not touched at all, no cleanup needed there, so everything is pretty simple. Of course, valid tests still should/need to be made, to make sure everything is covered, but there really isn`t much to cover in the first place.

But this is open for discussion, of course, and code for ref-logging all steps is still inside the script, just commented out. Heck, there`s even get_checkout_reflog_message() function at the end of the script, used to mimic real checkout reflog message, that`s currently unused ;)

Simple - abort. Restore backed up index and HEAD state is all that`s to it, and it looks like we never issued the command to begin with.

Yes, I understand that (hopefully communicated from the second part of the quote), but I see my wording of that first sentence was a bit unfortunate there (to say the least, lol), thanks for clarifying.

From everything said above, it seems so - at the moment, but I`m looking forward making it work for you, too, as it does make sense "out in the wild", in real-life scenarios.

Thank you again, for inspiring discussion.

Regards, Buga

Am 30.11.2017 um 00:10 schrieb Igor Djordjevic:

Ohh..kay.

This is a very, VERY exotic workflow, I would say. How would you construct commit I when three or more topics have conflicts? merge-octopus does not support this use-case.

I am not used to this picture. I would not think that it is totally unacceptable, but it still has a hmm-factor.

This is very acceptable.

Nevertheless, IMO, it is not the task of git-commit to re-compute a merge commit. It would be OK that it commits changes on top of a branch that is not checked out. Perhaps it would even be OK to remove the change from the current workspace (index and worktree), because it will return in the form of a merge later, but producing that merge is certainly not the task of git-commit.

-- Hannes

Hi Hannes,

On 01/12/2017 18:23, Johannes Sixt wrote:

But I`m not interested in constructing commit I in the first place, only help working with it once it`s already there (which shouldn`t be too uncommon in case of unrelated, non-conflicting topics) - but I already agreed my example could have been a bit too esoteric, distracting attention from the important part :)

I`ll continue on this further below, where you commented on those more common scenarios.

Here, let`s try to look at the whole situation from a "patch queue" perspective instead, starting from something like this:

(5) ---O---I  <- integration <- HEAD

... and then making progress like this - first, commit A1 onto O, starting "topicA" branch at the same time, even, maybe using syntax like `git commit --onto-parent O -b topicA`:

(6) ---O---J  <- integration <- HEAD               [ J = I + A1 ]
        \ /
         A1   <- topicA

..., then commit B1 onto O to start "topicB" branch:

(7)      B1   <- topicB
        / \
    ---O---K  <- integration <- HEAD               [ K = J + B1 ]
        \ /
         A1   <- topicA

..., then add one more commit (patch) onto B1:

(8)      B1---B2   <- topicB
        /      \
    ---O--------L  <- integration <- HEAD          [ L = K + B2 ]
        \      /
         A1---/    <- topicA

..., and one more, B3:

(9)      B1---B2---B3   <- topicB
        /           \
    ---O-------------M  <- integration <- HEAD     [ M = L + B3 ]
        \           /
         A1--------/    <- topicA

We can also start a new topic branch (queue), commit C1:

(10)     B1---B2---B3   <- topicB
        /           \
    ---O-------------N  <- integration <- HEAD     [ N = M + C1 ]
       |\           /|
       | A1--------/ /  <- topicA
       \            /
        C1---------/    <- topicC

And lets make one more "topicA" related commit A2:

(11)     B1---B2---B3   <- topicB
        /           \
    ---O-------------P  <- integration <- HEAD     [ P = N + A2 ]
       |\           /|
       | A1---A2---/ /  <- topicA
       \            /
        C1---------/    <- topicC

Notice how HEAD never leaves "integration" branch, and underlying commit is recreated each time? It`s like a live branch we`re working on, but we`re not actually committing to.

No branch switching (and no working tree file changes caused by it), and we`re working on multiple topics/branches simultaneously, being able to instantly test their mutual interaction as we go, but also creating their separate (and "clean") histories at the same time.

I guess this would make most sense with simple topics we _could_ practically work on at the same time without making our life too complicated - what stands for "git add --patch", too, when working on multiple commits at the same time.

Once satisfied, of course each topic branch would need to be tested separately, and each commit, even - all the same as with "git add --patch" commits.

And "git add --patch" can still be used here, too, to distribute partial changes, currently existing together inside the working tree, to different topic branches, at the time of making the commit itself.

Does this approach make more sense in regards to "git commit --onto-parent" functionality I`m having in mind? Or I`m dreaming too much here...? :)

Main idea is not to pile up uninteresting merge commits inside (throwaway) integration branch, needlessly complicating history, but pretend as we just made the integration merge, where fixup commit f was already existing in its topic branch prior the merge.

And I can see logic behind this case, too (alongside that other one, where they can both be supported, for different scenarios).

Just to make sure we`re on the same page - you mean conceptually, or practically? Or both...? :)

Because practically, we don`t do any merge operations/computations (as can be seen inside the script), so we really don`t produce any merges :)

Having a merge commit being updated as an outcome is just a consequence of altering our history to pretend fixup commit already existed elsewhere, being brought into our current state from there - and that "current state" is updated accordingly to reflect the change we made in the history that led to it.

Let`s look at a different example, no merges involved:

(12) ---A---B---C---D  <- HEAD

Here, we can do `git commit --onto-parent C` to make the history look like this:

(13) ---A---B---C---E---D'  <- HEAD

..., where we practically inserted commit E between our HEAD and commit C, causing D to be updated to D' accordingly (rebased, practically).

Idea is the same with merges involved, updating HEAD commit to reflect altered history.

In situation (12), we could also do `git commit --onto-parent C --amend`, ending up with history like this instead:

(14) ---A---B---F---D'  <- HEAD

..., where F is amended C, E being added to it, instead of being a separate commit.

So there are quite some possibilities here.

But as said, I can understand a wish to have a commit elsewhere without "merging" it in, too (changes removed from current workspace), or "merging" it inside a separate commit (not updating existing HEAD one).

And saying that, making the merge commit at the same time could then be observed as no more than a shortcut, too, like being able to create a new branch on checkout using "-b" option.

So even conceptually, it would make sense to wish for something like this:

(15) git commit --onto B --merge

..., producing that graph (4) shown in the quote above, I would think?

But even in that case, do note no merge is really happening, we just alter history of our current state, where we already are.

Splitting changes out of HEAD to a commit directly preceding it, on a different branch, and still keeping those changes in HEAD, naturally causes our current HEAD state to have multiple parents, what is perceived as a merge commit, even without a merge operation leading to it.

Thus, `git commit` seems it should be up to the task, whichever approach user opts for :)

Regards, Buga

I am sorry for not responding in detail. I think we've reached a mutual understanding of our workflows.

Though, from the ideas you tossed around most recently, you seem to want to make git-commit into a kitchen-sink for everything. I have my doubts that this will be a welcome change. Just because new commits are created does not mean that the feature must live in git-commit.

-- Hannes

On Wed, Dec 6, 2017 at 10:34 AM, Johannes Sixt <j6t@kdbg.org> wrote:

Nicely put.

On 06/12/2017 19:34, Johannes Sixt wrote:

No problem, thanks for your time so far.

There might be one more thing I should address, possibly left unclear from my previous message, but I`ll leave that for a follow-up e-mail, not being that important at the moment for the topic itself.

On 06/12/2017 19:40, Junio C Hamano wrote:

Yeah, I understand that might have felt cluttering, besides also being out of scope of the original topic idea. Thanks for the reality check (to both).

To get back on track, and regarding what`s already been said, would having something like this(1) feel useful?

(1) git commit --onto <commit>

So in previously mentioned situation:

(2) ...A    ...C            <- topics A, C
        \       \
      ---o---o---o---o I    <- integration <- HEAD
            /       /
        ...B    ...D        <- topics B, D

... it would allow committing changes F inside HEAD on top of B directly, no checkout / branch switching needed, getting to:

(3) ...A    ...C            <- topics A, C
        \       \
      ---o---o---o---o I    <- integration <- HEAD
            /       /
        ...B    ...D        <- topic D
            \
             F              <- topic B

So the most conservative approach, where changes F are removed from HEAD index and working tree, leaving it up to the user to decide if he will then merge them back in (or do something else).

I stress the major selling point here still being avoiding branch switching back and forth in order to commit a fixup on a different branch, which could otherwise trigger needless rebuilds, being significant in large projects.

And thanks to that `git-merge-one-file--cached`[1] script, we are also able to resolve some more of trivial conflicts when applying F onto B, using three-way file merge when needed, but still not touching working tree (contrary to original `git-merge-one-file`).

Regards, Buga

[1] https://public-inbox.org/git/CAPc5daWupO6DMOMFGn=XjUCG-JMYc4eyo8+TmAsdWcAOHXzwWg@mail.gmail.com/T/#mcb3953542dc265516e3ab1bff006ff1b5b85126a

Igor Djordjevic <igor.d.djordjevic@gmail.com> writes:

Are you asking me if _I_ find it useful? It is not a very useful question to ask, as I've taken things that I do not find useful myself.

Having said that, I do not see me personally using it. You keep claiming that committing without ever materializing the exact state that is committed in the working tree is a good thing.

I do not subscribe to that view.  

I'd rather do a quick fix-up on top (which ensures that at least the fix-up works in the context of the tip), and then "rebase -i" to move it a more appropriate place in the history (during which I have a chance to ensure that the fix-up works in the context it is intended to apply to).

I know that every time I say this, people who prefer to commit things that never existed in the working tree will say "but we'll test it later after we make these commit without having their state in the working tree". But I also know better that "later" often do not come, ever, at least for people like me ;-).

The amount of work _required_ to record the fix-up at its final resting place deeper in the history would be larger with "rebase -i" approach, simply because approaches like "commit --onto" and "git post" that throw a new commit deep in the history would not require ever materializing it in the working tree. But because I care about what I am actually committing, and because I am just lazy as any other human (if not more), I'd prefer an apporach that _forces_ me to have a checkout of the exact state that I'd be committing. That would prod me to actually looking at and testing the state after the change in the context it is meant to go.

On 08/12/2017 17:24, Junio C Hamano wrote:

It was also (kind of shy and subtle) "would you take it?", indeed, but I do value your personal opinion here, too, being a recognized developer, and one really knowing the Git (mailing list) community on top of it, so I appreciate you addressed both sides of the question.

And it was partly addressed to Hannes, but more for a confirmation, I guess, him being the one to favor such a flow in the first place, over what I initially suggested.

No - and I find it an important difference to note - just that it might be acceptable / more preferable _in certain situations_, where the only alternative seems to be wasting (significant) amount of time on needless rebuilds of many files (just because of branch switching, otherwise untouched by the changes we`re interested in).

If this is perceived a too uncommon/exotic case to worth addressing is a different matter, though.

Chris reported in this very topic[1] that sometimes, due to conflicts with later commits, "checkout > commit > [checkout >] rebase --onto" is "much easier to do", where "commit --fixup > rebase -i" "breaks" (does not apply cleanly).

No comment here ;)

All that I agree with, too.

But that said, I do find `git add --patch` invaluable (for example), where one can still opt to commit right away (and test later ;)), or do a proper `git stash push --keep-index` first in order to actually check/test the exact state/context before committing.

One of the biggest advantages I see in using Git is that it provides so many possibilities, where there is not necessarily a single "correct" way to do something - depending on the (sub)context, the decision on "_the_ correct" way can be deferred to the user himself.

Git (usually) does not judge, except in cases where something is considered "plain wrong" - still different than "might not be the best approach", but not necessarily a wrong one, either.

But I do realize it also means more chances for beginner users to shoot themselves in the foot, blaming it on Git, so even if just a matter of personal taste, a more restrictive preference from the Git maintainer is understandable :)

Regards, Buga

[1] https://public-inbox.org/git/CAPc5daWupO6DMOMFGn=XjUCG-JMYc4eyo8+TmAsdWcAOHXzwWg@mail.gmail.com/T/#m989306ab9327e15f14027cfd74ae8c5bf487affb

On Dec 9, 2017, at 01:54, Igor Djordjevic wrote:

It was more of a rant about conflict resolution by rebase rather than a concern about modification time of files. While I'd prefer git to not touch the source tree unnecessarily, it's not really a big deal for me if it does and some parts of the project need to be rebuilt.

The situations which tend to cause conflicts for me usually look like this.

  ---A---B

Say, a file has this line added somewhere in commit A:

  +int some_function(); 

Then commit B adds another line:

   int some_function(); 
  +int another_function();

Then I realize that I would like to have commit A to introduce some other_function() as well, so I add a fixup:

  ---A---B---f!A

with the following diff:

   int some_function(); 
  +int other_function();
   int another_function();

And then I often find that "rebase -i --autosquash" fails to apply the commit B because it expects slightly different context around the changed lines.

However, sometimes I see that if I do this:

  ---A---B
      \
       f!A

then "rebase --onto f!A A B" succeeds, nicely resolving the conflict without bothering me. No idea why. I kinda hoped that you may know this magic and incorporate it into "commit --onto" which will allow to immediately get to the result of the rebase:

  ---A---f!A---B'

without spelling it all manually.

(And yeah, I'm actually Alexei, not Chris. That was my MUA being dumb and using an old pseudonym than Google insists I'm called by.)

Hi Alexei,

On 09/12/2017 03:18, Alexei Lozovsky wrote:

Nevertheless, I found it valuable in supporting the case where "commit --fixup > rebase -i" seems to require even more work than otherwise necessary :)

But thanks for clarifying, anyway, it does feel like `git rebase -i --autosquash` could be smarter in this regards, if `git rebase --onto` does it better...?

Even though your explanation seems clear, having a real, easily reproducible case would help as well, I guess.

If you mind enough to be bothered testing it out, might be even existing/initial state of originally proposed `git commit --onto-parent` script would work for you, as it does incorporate some trivial three-way merge resolution.

In your starting situation:

    ---A---B

... you would just do something like:

    git commit --onto-parent A

... hopefully ending up in the desired state (hopefully = conflicts automatically resolved):

    ---A---C---B'

You could even do this instead:

    git commit --onto-parent A --amend

... ending up with:

    ---A'---B'

... as that is basically what you wanted in the first place ;)

Ah, sorry for the confusion :)

Regards, Buga

Hi Igor

On 09/12/17 03:03, Igor Djordjevic wrote:

Creating the fixup directly on A rather than on top of B avoids the conflicting merge B f!A A. Creating the fixup on top of B and then using git commit --onto A would suffer from the same conflicts as rebase does. I don't think there is any way for 'git rebase --autosquash' to avoid the conflicts unless it used a special fixup merge strategy that somehow took advantage of the DAG to resolve the conflicts by realizing they come from a later commit. However I don't think that could be implemented reliably as sometimes one wants those conflicting lines from the later commit to be moved to the earlier commit with the fixup.

Best Wishes

Phillip

Hi Philip,

On 09/12/2017 20:01, Phillip Wood wrote:

I`m a bit confused here, as you`re replying to the part where we strictly discussed `rebase --autosquash` versus `rebase --onto`, having the latter succeed where the former fails - but you`re mentioning `git _commit_ --onto` instead, comparing it with `rebase`... and which one of the two ("--autosquash", I assume)?

Even further, while I do seem to understand (and agree with) what 
you`re talking about with `commit --onto` and `rebase --autosquah` 
suffering from the same conflicts in attempt to take f!A, originally 
created on top of B, and apply it on top of A - the thing is that 
Alexei actually pointed to B being the problematic one, failing to 
rebase on top of already (successfully) autosquashed A' (where A' = A 
+ f!A, fixup applied through --autosquash), while it doesn`t fail 
rebasing --onto f!A when f!A is being committed on top of A directly 
(and not through --autosquash).

In that (very?) specific case, proposed `git commit --onto-parent`[1] doesn`t suffer from this, as once f!A is successfully applied onto A (either squashed in with --amend, or on top of it), we take original f!A _snapshot_ (not patch!) made on top of B, and just "declare" it B` (being equal to B + f!A, which we already know, and being correct), without a need to (try to) apply B patch on top of fixed-up A to create B', as `rebase` does (and fails).

I think I agree on this part being tricky (if possible at all), but I also think this is not what Alexei was complaining about, nor what we were discussing (as I tried to explain above) - but please do correct me if I misunderstood you.

That said, and what I mentioned already, we might really benefit from simple test case(s), showing "rebase --autosquash" failing where "rebase --onto" works, as Alexei explained, giving some more (and firm) context to the discussion.

I *think* I`ve experienced this in the past myself, but now I can`t seem to wrap my head around a reproducible example just yet... :$

Regards, Buga

[1] https://public-inbox.org/git/4a92e34c-d713-25d3-e1ac-100525011d3f@talktalk.net/T/#m72f45ad7a8f1c733266a875bca087ee82cc781e7

On 10/12/17 01:20, Igor Djordjevic wrote:

Sorry I should have been clearer. The point I was somewhat obliquely making was that 'rebase --onto' succeeds where 'rebase --autosquash' fails not because it is smarter but because it is doing something different. Specifically it avoids the conflicting merge to create A' as the user has already created that commit in the temporary branch

Yes because in an earlier message you said

and I was pointing out that this would involve performing the same merge as 'rebase --autosquash' which has conflicts

I understood Alexei to mean that it was merging the f!A into A that caused conflicts due to the fact that f!A has conflicting context that was introduced in B. After all B' the rebased B is merge A A' B whether it is created by 'rebase --autosquash' or 'rebase --onto'. A' must be the same in both cases or one is applying a different fix.

I've found conflicts arising from moving fixups can be quite common, so these days I tend to edit the commit to be fixed up directly. I have a script git-amend that does something like

target=$(git rev-parse --verify "$1") && GIT_SEQUENCE_EDITOR="sed -i s/^pick $target/edit $target/" rebase -ik $target^

so I can just type 'git amend <commit>' to make this easier

Ah I understand, but that only works when you're fixing up HEAD~1. If you had A-B-C-f!A you have to recreate B with a merge.

No, I don't think Alexei was complaining about that directly, but if such a solution existed he (and everyone else) wouldn't have to bother with the --onto approach in the case where merging the fixup creates conflicts.

Best Wishes

Phillip

Hi Philip,

On 10/12/2017 13:22, Phillip Wood wrote:

No problem, and thanks for clarifying, I understand and agree to all that with you. I was just pointing that it wasn`t something I was commenting to (nor specially interested in), because of what Alexei actually wrote - here`s his quote (emphasis mine):

  "And then I often find that "rebase -i --autosquash" _fails to apply
  the commit B_ because it expects slightly different context around
  the changed lines."

From there, it seemed pretty clear he perceived the failure not coming from creating A', but applying B on top of it, and that is what got my attention. But, read below...

Yeah, what I assumed (and agreed to), thanks for confirmation. What made me a bit uncertain was that you left that part of my earlier message quoted _after_ your inline reply to it, thus making overall context a bit difficult to be exactly sure in :P

Yes, I understand and agree you might be right, what you are talking about being what he actually _meant_, but because that is not what he _wrote_, I wanted to see an example of it, (still?) hoping that he really did mean what he wrote (commit B being the problematic one), as then there would be a possibility for improvement.

And your analysis seems correct, and that`s what I was afraid of as well - but wasn`t really sure, especially as I seem to remember something similar from my own (humble) experience, thus leaving a possibility for an example to prove differently.

But if that is absolutely impossible, as you claim, like not even due to some commit squashing, some edge case, or something - and I don`t feel like I have enough knowledge/experience to judge that myself at the moment - then you have to be right, and what he wrote is really not what he meant... nor what I thought I remembered from my own past experience, either :/ Nor there is any chance for improvement here, unfortunately, I guess.

Still, I hope for that example...! :D

This is useful, thanks. I have something like `git commit --amend <commit>` on my wish list for quite some time :) Still not getting to look into it, though.

Yes, and thus the notion of what he mentioned as being a "(very?) specific case" ;) That initial/draft version of "git commit --onto-parent" script I sent to the list[1] operates on the first parent commit only, indeed, though its main point/purpose had nothing to do with smarter merges, but just not touching the working tree while at it, if possible.

Yes, I think we understand each other now (unfortunately, I guess, as that also means there is nothing more to add to it, in terms of improving existing situation). Thank you for your thoughts :)

Regards, Buga

[1] https://public-inbox.org/git/4a92e34c-d713-25d3-e1ac-100525011d3f@talktalk.net/T/#m72f45ad7a8f1c733266a875bca087ee82cc781e7

On Dec 11, 2017, at 01:17, Igor Djordjevic wrote:

I'm not really good at remembering the exact details, so if you ask for a testimony then I'm not sure whether it's the conflicts in the fixups or the later commits that I was annoyed by :) I'm also not really versed in the technical details of rebasing, so I cannot give an educated guess on which one is more likely to cause conflicts.

I keep this thread pinned, so I hope to provide a more concrete example as soon as I encounter the conflicting situation again in the wild. I'm not sure that I am able to construct a relevant example artificially.

On Dec 10, 2017, at 14:22, Phillip Wood wrote:

Hm... I just realized that using "edit" command during interactive rebase should probably be the same as the strategy with a temporary branch and rebase --onto I described earlier. I should fix my habits, I guess.

Hi Igor

On 09/12/17 03:03, Igor Djordjevic wrote:

Creating the fixup directly on A rather than on top of B avoids the conflicting merge B f!A A. Creating the fixup on top of B and then using git commit --onto A would suffer from the same conflicts as rebase does. I don't think there is any way for 'git rebase --autosquash' to avoid the conflicts unless it used a special fixup merge strategy that somehow took advantage of the DAG to resolve the conflicts by realizing they come from a later commit. However I don't think that could be implemented reliably as sometimes one wants those conflicting lines from the later commit to be moved to the earlier commit with the fixup.

Best Wishes

Phillip

On Nov 27, 2017, at 00:35, Igor Djordjevic <igor.d.djordjevic@gmail.com> wrote:

I'm actually doing the described workflow quite often with git rebase when working on a topic. Given the following structure:

  ---o               (master)
      \
       o---A---B---C (topic)

When I want to make changes to commit A one option is to make them directly on topic, then do "git commit --fixup A", and then eventual interactive rebase onto master will clean them up:

  ---o                     (master)
      \
       o---A---B---C---f!A (topic)

However, sometimes this breaks when changes in B or C conflict somehow with A (which may happen quite a lot during development of a topic), so the rebase will not apply cleanly. So sometimes I make a temporary branch from A, commit the fixup there:

  ---o               (master)
      \
       o---A---B---C (topic)
            \
             f!A     (temp)

and then use "git rebase --onto temp A topic" to move the topic back on track:

  ---o                     (master)
      \
       o---A---f!A         (temp)
                  \
                   B'---C' (topic)

after which the final cleanup rebase is much easier to do.

Obviously, all the branch switching and rebasing does take its tall on file modifications.

Hi Chris,

On 30/11/2017 23:40, Chris Nerwert wrote:

From use case you described (and which I often experience myself), it seems plain "git commit --onto A" would be of help here, committing fixup onto A directly, without a need to switch to it (branch or not), a case I`m discussing with Hannes in that other sub-thread[1] of this e-mail, too.

But from there, your flow takes a different direction, using rebase, while this whole thread started around some merge-like functionality.

I can imagine a user interface doing what you (and I) would like, something like:

(1) git commit --onto A --rebase

..., where your changes would first be committed onto commit A, and then commits from A (excluded) to HEAD (included) rebased onto this new commit.

BUT, as far as it seems to me, rebase currently touches working tree for each operation (am I wrong here?), so once the rebase sequence is initiated, it would internally still need to checkout to your new fixup commit (on top of A), and then proceed applying changes and changing working tree with each commit being rebased, overall failing to address your main concern - needless (untouched) file modifications, even in case of no conflicts.

I find this scenario quite interesting as well, but I`m afraid it may currently be out of scope of what I`m trying to accomplish with "git commit --onto[-parent]", for the most part because it looks like it would need "index only rebase" first (not touching working tree, that is)...?

If we had that, it would/should be pretty easy to add it into the mix with "git commit --onto" here, ending up with something as imagined in command line (1) above :) I`ll make a note of it, thanks.

Any further help appreciated, of course :)

Regards, Buga

[1] https://public-inbox.org/git/0F30BEF8-A9F7-43B9-BC89-4B9CD7AF3E16@gmail.com/T/#me830a80d745df60ae8bd6a2e67eee4bd4dabf56c