threads / patch / 48716

patchunpack-trees: do not fail reset because of unmerged skipped entry

Subject: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

## tl;dr

9 messages between Jun 15, 2018 and Jul 11, 2018. Diffs are folded; open one to read it.

replies: 8people: 3as markdown or json

Max Kirillov· Jun 15, 2018, 04:42 UTC · lore

After modify/delete merge conflict happens in a file skipped by sparse checkout, "git reset --merge", which implements the "--abort" actions, and "git reset --hard" fail with message "Entry * not uptodate. Cannot update sparse checkout." The reason is that the entry is verified in apply_sparse_checkout() for being up-to-date even when it has a conflict. Checking conflicted entry for being up-to-date is not performed in other cases. One obvious reason to not check it is that it is already modified by inserting conflict marks.

Fix by not checking conflicted entries before performing reset. Also, add test case which verifies the issue is fixed.

Signed-off-by: Max Kirillov <max@max630.net>
---
I have tried to use sparse-checkout for merging and cherrypicking, to save on IO
and disk space. It works, mostly, but there are issues here and there.
This one was low hanging, and also pretty annoying.
 t/t3035-merge-sparse.sh | 46 +++++++++++++++++++++++++++++++++++++++++
 unpack-trees.c          |  2 +-
 2 files changed, 47 insertions(+), 1 deletion(-)
 create mode 100755 t/t3035-merge-sparse.sh
Show changes to 2 files +47 −1

t/t3035-merge-sparse.sh, unpack-trees.c

diff --git a/t/t3035-merge-sparse.sh b/t/t3035-merge-sparse.sh
new file mode 100755
index 0000000000..c6b2b0b82a
--- /dev/null
+++ b/t/t3035-merge-sparse.sh
@@ -0,0 +1,46 @@
+#!/bin/sh
+
+test_description='merge with sparse files'
+
+. ./test-lib.sh
+
+# test_file $filename $content
+test_file () {
+	echo "$2" > "$1" &&
+	git add "$1"
+}
+
+# test_commit_this $message_and_tag
+test_commit_this () {
+	git commit -m "$1" &&
+	git tag "$1"
+}
+
+test_expect_success 'setup' '
+	test_file checked-out init &&
+	test_file modify_delete modify_delete_init &&
+	test_commit_this init &&
+	test_file modify_delete modify_delete_theirs &&
+	test_commit_this theirs &&
+	git reset --hard init &&
+	git rm modify_delete &&
+	test_commit_this ours &&
+	git config core.sparseCheckout true &&
+	echo "/checked-out" >.git/info/sparse-checkout &&
+	git reset --hard &&
+	! git merge theirs
+'
+
+test_expect_success 'reset --hard works after the conflict' '
+	git reset --hard
+'
+
+test_expect_success 'setup: conflict back' '
+	! git merge theirs
+'
+
+test_expect_success 'Merge abort works after the conflict' '
+	git merge --abort
+'
+
+test_done
diff --git a/unpack-trees.c b/unpack-trees.c
index e73745051e..65ae0721a6 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -468,7 +468,7 @@ static int apply_sparse_checkout(struct index_state *istate,
 		 * also stat info may have lost after merged_entry() so calling
 		 * verify_uptodate() again may fail
 		 */
-		if (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))
+		if (!(ce->ce_flags & CE_UPDATE) && !(ce->ce_flags & CE_CONFLICTED) && verify_uptodate_sparse(ce, o))
 			return -1;
 		ce->ce_flags |= CE_WT_REMOVE;
 		ce->ce_flags &= ~CE_UPDATE;
-- 
2.17.0.1185.g782057d875
Junio C Hamano· Jun 15, 2018, 19:58 UTC · re: Max Kirillov · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

Max Kirillov <max@max630.net> writes:
Show 11 quoted lines
> After modify/delete merge conflict happens in a file skipped by sparse
> checkout, "git reset --merge", which implements the "--abort" actions, and
> "git reset --hard" fail with message "Entry * not uptodate. Cannot update
> sparse checkout." The reason is that the entry is verified in
> apply_sparse_checkout() for being up-to-date even when it has a conflict.
> Checking conflicted entry for being up-to-date is not performed in other
> cases. One obvious reason to not check it is that it is already modified
> by inserting conflict marks.
>
> Fix by not checking conflicted entries before performing reset.
> Also, add test case which verifies the issue is fixed.

I do not know offhand if "reset --merge" should force succeeding in such a case, but I agree that it is criminal to stop "reset --hard" with "not uptodate", as the whole point of "hard reset" is to get rid of the 'not up-to-date' modification.

I guess not may people make serious use of sparsely checked-out working tree and that is why such a failure is reported this late after the feature was introduced?

> +test_expect_success 'reset --hard works after the conflict' '
> +	git reset --hard
> +'

Do we want to verify the state after the 'hard' reset succeeds as well? Things like

 - all paths in the HEAD and all paths in the index are identical;
 - paths that do exist in the working tree are all identical to HEAD
   version; and
 - paths that do not exist in the working tree are missing due to
   the sparse checkout setting (iow, it is a bug if a path that is
   outside the "sparse" setting is missing from the working tree).
Show 7 quoted lines
> +test_expect_success 'setup: conflict back' '
> +	! git merge theirs
> +'
> +
> +test_expect_success 'Merge abort works after the conflict' '
> +	git merge --abort
> +'
Likewise here.
Show 14 quoted lines
> +test_done
> diff --git a/unpack-trees.c b/unpack-trees.c
> index e73745051e..65ae0721a6 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -468,7 +468,7 @@ static int apply_sparse_checkout(struct index_state *istate,
>  		 * also stat info may have lost after merged_entry() so calling
>  		 * verify_uptodate() again may fail
>  		 */
> -		if (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))
> +		if (!(ce->ce_flags & CE_UPDATE) && !(ce->ce_flags & CE_CONFLICTED) && verify_uptodate_sparse(ce, o))
>  			return -1;
>  		ce->ce_flags |= CE_WT_REMOVE;
>  		ce->ce_flags &= ~CE_UPDATE;
Thanks.
Max Kirillov· Jun 16, 2018, 08:22 UTC · re: Junio C Hamano · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

> I do not know offhand if "reset --merge" should force succeeding in

such a case, but I agree that it is criminal to stop "reset --hard" with "not uptodate", as the whole point of "hard reset" is to get rid of the 'not up-to-date' modification.

I originally had a fix just for "reset --hard". It was in verify_uptodate_1(), to move check for "->reset" earlier. But then I found that "merge --abort" does not use "reset --hard", but rather --merge, so I fixed that. Because --merge should work also, shouldn't it?

Actually, I think that fix in verify_uptodate_1() was right, I just did not find what it affects, after the other fix

Max Kirillov· Jul 10, 2018, 19:23 UTC · re: Junio C Hamano · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

On Fri, Jun 15, 2018 at 12:58:40PM -0700, Junio C Hamano wrote:
Show 11 quoted lines
> Do we want to verify the state after the 'hard' reset succeeds as
> well?  Things like 
> 
>  - all paths in the HEAD and all paths in the index are identical;
> 
>  - paths that do exist in the working tree are all identical to HEAD
>    version; and
> 
>  - paths that do not exist in the working tree are missing due to
>    the sparse checkout setting (iow, it is a bug if a path that is
>    outside the "sparse" setting is missing from the working tree).

I implemented the additional check, it is a bit different literally, but should be equivalent for this case

Duy Nguyen· Jun 16, 2018, 05:14 UTC · re: Max Kirillov · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

On Fri, Jun 15, 2018 at 07:42:51AM +0300, Max Kirillov wrote:
Show 5 quoted lines
> After modify/delete merge conflict happens in a file skipped by sparse
> checkout, "git reset --merge", which implements the "--abort" actions, and
> "git reset --hard" fail with message "Entry * not uptodate. Cannot update
> sparse checkout." The reason is that the entry is verified in
> apply_sparse_checkout() for being up-to-date even when it has a conflict.

Conflicted entries should not be skipped by design. Even if you specify sparse patterns to ignore them, they must be checked out. When a conflicted entry appears in apply_sparse_checkout() something else is already wrong.

I think this is a better fix along that line. As you can see we already un-skip staged entries. But I think I forgot (or did not know) about CE_CONFLICTED. This change passes your new tests, but I didn't try to run the whole test suite to see if I broke anything else.

-- 8< --
Show changes to unpack-trees.c +1 −2
diff --git a/unpack-trees.c b/unpack-trees.c
index 3a85a02a77..eb544ee1b3 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1246,7 +1246,7 @@ static void mark_new_skip_worktree(struct exclude_list *el,
 		if (select_flag && !(ce->ce_flags & select_flag))
 			continue;
 
-		if (!ce_stage(ce))
+		if (!ce_stage(ce) && !(ce->ce_flags & CE_CONFLICTED))
 			ce->ce_flags |= skip_wt_flag;
 		else
 			ce->ce_flags &= ~skip_wt_flag;
-- 8< --

> Checking conflicted entry for being up-to-date is not performed in other
> cases. One obvious reason to not check it is that it is already modified
> by inserting conflict marks.
> 
> Fix by not checking conflicted entries before performing reset.
> Also, add test case which verifies the issue is fixed.
> 
> Signed-off-by: Max Kirillov <max@max630.net>
> ---
> I have tried to use sparse-checkout for merging and cherrypicking, to save on IO
> and disk space. It works, mostly, but there are issues here and there.
> This one was low hanging, and also pretty annoying.
> 
>  t/t3035-merge-sparse.sh | 46 +++++++++++++++++++++++++++++++++++++++++
>  unpack-trees.c          |  2 +-
>  2 files changed, 47 insertions(+), 1 deletion(-)
>  create mode 100755 t/t3035-merge-sparse.sh
> 
> diff --git a/t/t3035-merge-sparse.sh b/t/t3035-merge-sparse.sh
> new file mode 100755
> index 0000000000..c6b2b0b82a
> --- /dev/null
> +++ b/t/t3035-merge-sparse.sh
> @@ -0,0 +1,46 @@
> +#!/bin/sh
> +
> +test_description='merge with sparse files'
> +
> +. ./test-lib.sh
> +
> +# test_file $filename $content
> +test_file () {
> +	echo "$2" > "$1" &&
> +	git add "$1"
> +}
> +
> +# test_commit_this $message_and_tag
> +test_commit_this () {
> +	git commit -m "$1" &&
> +	git tag "$1"
> +}
> +
> +test_expect_success 'setup' '
> +	test_file checked-out init &&
> +	test_file modify_delete modify_delete_init &&
> +	test_commit_this init &&
> +	test_file modify_delete modify_delete_theirs &&
> +	test_commit_this theirs &&
> +	git reset --hard init &&
> +	git rm modify_delete &&
> +	test_commit_this ours &&
> +	git config core.sparseCheckout true &&
> +	echo "/checked-out" >.git/info/sparse-checkout &&
> +	git reset --hard &&
> +	! git merge theirs
> +'
> +
> +test_expect_success 'reset --hard works after the conflict' '
> +	git reset --hard
> +'
> +
> +test_expect_success 'setup: conflict back' '
> +	! git merge theirs
> +'
> +
> +test_expect_success 'Merge abort works after the conflict' '
> +	git merge --abort
> +'
> +
> +test_done
> diff --git a/unpack-trees.c b/unpack-trees.c
> index e73745051e..65ae0721a6 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -468,7 +468,7 @@ static int apply_sparse_checkout(struct index_state *istate,
>  		 * also stat info may have lost after merged_entry() so calling
>  		 * verify_uptodate() again may fail
>  		 */
> -		if (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))
> +		if (!(ce->ce_flags & CE_UPDATE) && !(ce->ce_flags & CE_CONFLICTED) && verify_uptodate_sparse(ce, o))
>  			return -1;
>  		ce->ce_flags |= CE_WT_REMOVE;
>  		ce->ce_flags &= ~CE_UPDATE;
> -- 
> 2.17.0.1185.g782057d875
> 
Max Kirillov· Jul 10, 2018, 19:21 UTC · re: Duy Nguyen · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

On Sat, Jun 16, 2018 at 07:14:44AM +0200, Duy Nguyen wrote:
Show 15 quoted lines
> -- 8< --
> diff --git a/unpack-trees.c b/unpack-trees.c
> index 3a85a02a77..eb544ee1b3 100644
> --- a/unpack-trees.c
> +++ b/unpack-trees.c
> @@ -1246,7 +1246,7 @@ static void mark_new_skip_worktree(struct exclude_list *el,
>  		if (select_flag && !(ce->ce_flags & select_flag))
>  			continue;
>  
> -		if (!ce_stage(ce))
> +		if (!ce_stage(ce) && !(ce->ce_flags & CE_CONFLICTED))
>  			ce->ce_flags |= skip_wt_flag;
>  		else
>  			ce->ce_flags &= ~skip_wt_flag;
> -- 8< --
I tried your fix and it is working. I put it instead of my original fix. Would you sign it off?
Duy Nguyen· Jul 11, 2018, 15:25 UTC · re: Max Kirillov · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

On Tue, Jul 10, 2018 at 9:22 PM Max Kirillov <max@max630.net> wrote:
Show 19 quoted lines
>
> On Sat, Jun 16, 2018 at 07:14:44AM +0200, Duy Nguyen wrote:
> > -- 8< --
> > diff --git a/unpack-trees.c b/unpack-trees.c
> > index 3a85a02a77..eb544ee1b3 100644
> > --- a/unpack-trees.c
> > +++ b/unpack-trees.c
> > @@ -1246,7 +1246,7 @@ static void mark_new_skip_worktree(struct exclude_list *el,
> >               if (select_flag && !(ce->ce_flags & select_flag))
> >                       continue;
> >
> > -             if (!ce_stage(ce))
> > +             if (!ce_stage(ce) && !(ce->ce_flags & CE_CONFLICTED))
> >                       ce->ce_flags |= skip_wt_flag;
> >               else
> >                       ce->ce_flags &= ~skip_wt_flag;
> > -- 8< --
>
> I tried your fix and it is working. I put it instead of my original fix. Would you sign it off?
Signed-off-by: Duy Nguyen <pclouds@gmail.com>
-- 
Duy
Junio C Hamano· Jul 11, 2018, 16:35 UTC · re: Duy Nguyen · lore

Re: [PATCH] unpack-trees: do not fail reset because of unmerged skipped entry

Duy Nguyen <pclouds@gmail.com> writes:
Show 22 quoted lines
> On Tue, Jul 10, 2018 at 9:22 PM Max Kirillov <max@max630.net> wrote:
>>
>> On Sat, Jun 16, 2018 at 07:14:44AM +0200, Duy Nguyen wrote:
>> > -- 8< --
>> > diff --git a/unpack-trees.c b/unpack-trees.c
>> > index 3a85a02a77..eb544ee1b3 100644
>> > --- a/unpack-trees.c
>> > +++ b/unpack-trees.c
>> > @@ -1246,7 +1246,7 @@ static void mark_new_skip_worktree(struct exclude_list *el,
>> >               if (select_flag && !(ce->ce_flags & select_flag))
>> >                       continue;
>> >
>> > -             if (!ce_stage(ce))
>> > +             if (!ce_stage(ce) && !(ce->ce_flags & CE_CONFLICTED))
>> >                       ce->ce_flags |= skip_wt_flag;
>> >               else
>> >                       ce->ce_flags &= ~skip_wt_flag;
>> > -- 8< --
>>
>> I tried your fix and it is working. I put it instead of my original fix. Would you sign it off?
>
> Signed-off-by: Duy Nguyen <pclouds@gmail.com>
Thanks, both.
Max Kirillov· Jul 10, 2018, 19:17 UTC · re: Max Kirillov · lore

[PATCH v2] unpack-trees: do not fail reset because of unmerged skipped entry

After modify/delete merge conflict happens in a file skipped by sparse checkout, "git reset --merge", which implements the "--abort" actions, and "git reset --hard" fail with message "Entry * not uptodate. Cannot update sparse checkout."

As explained in [1], the up-to-date checker mistakenly treats conflicted entry which does not exist in HEAD as still skipped by sparse checkout.

Use the fix suggested in [1]. Also, add test case which verifies the issue is fixed.

[1] https://public-inbox.org/git/20180616051444.GA29754@duynguyen.home/
Fix-authored-by: Duy Nguyen <pclouds@gmail.com>
Signed-off-by: Max Kirillov <max@max630.net>
---
* used the right fix. Probably there should be sign off?
* add extra check that state is correct after the reset. it is literally
  different from the one described by Junio but should be equivalent for this
  case
 t/t3035-merge-sparse.sh | 59 +++++++++++++++++++++++++++++++++++++++++++++++++
 unpack-trees.c          |  2 +-
 2 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100755 t/t3035-merge-sparse.sh
Show changes to 2 files +60 −1

t/t3035-merge-sparse.sh, unpack-trees.c

diff --git a/t/t3035-merge-sparse.sh b/t/t3035-merge-sparse.sh
new file mode 100755
index 0000000..0c0b433
--- /dev/null
+++ b/t/t3035-merge-sparse.sh
@@ -0,0 +1,59 @@
+#!/bin/sh
+
+test_description='merge with sparse files'
+
+. ./test-lib.sh
+
+# test_file $filename $content
+test_file () {
+	echo "$2" > "$1" &&
+	git add "$1"
+}
+
+# test_commit_this $message_and_tag
+test_commit_this () {
+	git commit -m "$1" &&
+	git tag "$1"
+}
+
+test_expect_success 'setup' '
+	: >empty &&
+	test_file checked-out init &&
+	test_file modify_delete modify_delete_init &&
+	test_commit_this init &&
+	test_file modify_delete modify_delete_theirs &&
+	test_commit_this theirs &&
+	git reset --hard init &&
+	git rm modify_delete &&
+	test_commit_this ours &&
+	git config core.sparseCheckout true &&
+	echo "/checked-out" >.git/info/sparse-checkout &&
+	git reset --hard &&
+	! git merge theirs
+'
+
+test_expect_success 'reset --hard works after the conflict' '
+	git reset --hard
+'
+
+test_expect_success 'is reset properly' '
+	git status --porcelain -- modify_delete >out &&
+	test_cmp empty out &&
+	test_path_is_missing modify_delete
+'
+
+test_expect_success 'setup: conflict back' '
+	! git merge theirs
+'
+
+test_expect_success 'Merge abort works after the conflict' '
+	git merge --abort
+'
+
+test_expect_success 'is aborted properly' '
+	git status --porcelain -- modify_delete >out &&
+	test_cmp empty out &&
+	test_path_is_missing modify_delete
+'
+
+test_done
diff --git a/unpack-trees.c b/unpack-trees.c
index 3a85a02..eb544ee 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1246,7 +1246,7 @@ static void mark_new_skip_worktree(struct exclude_list *el,
 		if (select_flag && !(ce->ce_flags & select_flag))
 			continue;
 
-		if (!ce_stage(ce))
+		if (!ce_stage(ce) && !(ce->ce_flags & CE_CONFLICTED))
 			ce->ce_flags |= skip_wt_flag;
 		else
 			ce->ce_flags &= ~skip_wt_flag;
-- 
2.0.0

← back to recent threads