{"thread":{"id":"56534","subject":"[PATCH 1/6] t2500: add various tests for nuking untracked files","startedAt":"2021-09-18T23:15:33Z","lastAt":"2021-12-13T20:10:46Z","messageCount":82,"participants":["Elijah Newren via GitGitGadget","Philip Oakley","Ævar Arnfjörð Bjarmason","Phillip Wood","Elijah Newren","Luke Diamand","Junio C Hamano","Eric Sunshine","Jack O'Connor"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"436328","messageId":"b634136a74b37a6f73bc3038f79031e040423883.1632006923.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH 1/6] t2500: add various tests for nuking untracked files","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:18Z","receivedAt":"2021-09-18T23:15:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nNoting that unpack_trees treats reset=1 & update=1 as license to nuke\nuntracked files, I looked for code paths that use this combination and\ntried to generate testcases which demonstrated unintentional loss of\nuntracked files and directories. I found several.\n\nI also include testcases for `git reset --{hard,merge,keep}`. A hard\nreset is perhaps the most direct test of unpack_tree's reset=1 behavior,\nbut we cannot make `git reset --hard` preserve untracked files without\nsome migration work.\n\nAlso, the two commands `checkout --force` (because of the --force) and\n`read-tree --reset` (because it's plumbing and we need to keep it\nbackward compatible) were left out as we expect those to continue\nremoving untracked files and directories.\n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n 1 file changed, 244 insertions(+)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nnew file mode 100755\nindex 00000000000..a1a6dfa671e\n--- /dev/null\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -0,0 +1,244 @@\n+#!/bin/sh\n+\n+test_description='Test handling of overwriting untracked files'\n+\n+. ./test-lib.sh\n+\n+test_setup_reset () {\n+\ttest_create_repo reset_$1 &&\n+\t(\n+\t\tcd reset_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\tgit branch stable &&\n+\t\tgit branch work &&\n+\n+\t\tgit checkout work &&\n+\t\ttest_commit foo &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_success 'reset --hard will nuke untracked files/dirs' '\n+\ttest_setup_reset hard &&\n+\t(\n+\t\tcd reset_hard &&\n+\t\tgit ls-tree -r stable &&\n+\t\tgit log --all --name-status --oneline &&\n+\t\tgit ls-tree -r work &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\techo foo >expect &&\n+\n+\t\tgit reset --hard work &&\n+\n+\t\t# check that untracked directory foo.t/ was nuked\n+\t\ttest_path_is_file foo.t &&\n+\t\ttest_cmp expect foo.t\n+\t)\n+'\n+\n+test_expect_success 'reset --merge will preserve untracked files/dirs' '\n+\ttest_setup_reset merge &&\n+\t(\n+\t\tcd reset_merge &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n+\t)\n+'\n+\n+test_expect_success 'reset --keep will preserve untracked files/dirs' '\n+\ttest_setup_reset keep &&\n+\t(\n+\t\tcd reset_keep &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n+\t)\n+'\n+\n+test_setup_checkout_m () {\n+\ttest_create_repo checkout &&\n+\t(\n+\t\tcd checkout &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines file has some >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit branch stable &&\n+\n+\t\tgit switch -c work &&\n+\t\techo stuff >notes.txt &&\n+\t\ttest_write_lines file has some words >filler &&\n+\t\tgit add notes.txt filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_failure 'checkout -m does not nuke untracked file' '\n+\ttest_setup_checkout_m &&\n+\t(\n+\t\tcd checkout &&\n+\n+\t\t# Tweak filler\n+\t\ttest_write_lines this file has some >filler &&\n+\t\t# Make an untracked file, save its contents in \"expect\"\n+\t\techo precious >notes.txt &&\n+\t\tcp notes.txt expect &&\n+\n+\t\ttest_must_fail git checkout -m work &&\n+\t\ttest_cmp expect notes.txt\n+\t)\n+'\n+\n+test_setup_sequencing () {\n+\ttest_create_repo sequencing_$1 &&\n+\t(\n+\t\tcd sequencing_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines this file has some words >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tmkdir -p foo/bar &&\n+\t\ttest_commit foo/bar/baz &&\n+\n+\t\tgit branch simple &&\n+\t\tgit branch fooey &&\n+\n+\t\tgit checkout fooey &&\n+\t\tgit rm foo/bar/baz.t &&\n+\t\techo stuff >>filler &&\n+\t\tgit add -u &&\n+\t\tgit commit -m \"changes\" &&\n+\n+\t\tgit checkout simple &&\n+\t\techo items >>filler &&\n+\t\techo newstuff >>newfile &&\n+\t\tgit add filler newfile &&\n+\t\tgit commit -m another\n+\t)\n+}\n+\n+test_expect_failure 'git rebase --abort and untracked files' '\n+\ttest_setup_sequencing rebase_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_abort_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\ttest_must_fail git rebase simple &&\n+\n+\t\tcat init.t &&\n+\t\tgit rm init.t &&\n+\t\techo precious >init.t &&\n+\t\tcp init.t expect &&\n+\t\tgit status --porcelain &&\n+\t\ttest_must_fail git rebase --abort &&\n+\t\ttest_cmp expect init.t\n+\t)\n+'\n+\n+test_expect_failure 'git rebase fast forwarding and untracked files' '\n+\ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_fast_forward_and_untracked &&\n+\t\tgit checkout init &&\n+\t\techo precious >filler &&\n+\t\tcp filler expect &&\n+\t\ttest_must_fail git rebase init simple &&\n+\t\ttest_cmp expect filler\n+\t)\n+'\n+\n+test_expect_failure 'git rebase --autostash and untracked files' '\n+\ttest_setup_sequencing rebase_autostash_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_autostash_and_untracked &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit rebase --autostash init &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git stash and untracked files' '\n+\ttest_setup_sequencing stash_and_untracked_files &&\n+\t(\n+\t\tcd sequencing_stash_and_untracked_files &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit status --porcelain &&\n+\t\tgit stash push &&\n+\t\tgit status --porcelain &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+\ttest_setup_sequencing am_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_abort_and_untracked &&\n+\t\tgit format-patch -1 --stdout fooey >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete the conflicted file; we will stage and commit it later\n+\t\trm filler &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir filler &&\n+\t\techo foo >filler/file1 &&\n+\t\techo bar >filler/file2 &&\n+\n+\t\ttest_must_fail git am --abort 2>errors &&\n+\t\ttest_path_is_dir filler &&\n+\t\tgrep \"Updating .filler. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+\ttest_setup_sequencing am_skip_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_skip_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\tgit format-patch -1 --stdout simple >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete newfile\n+\t\trm newfile &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir newfile &&\n+\t\techo foo >newfile/file1 &&\n+\t\techo bar >newfile/file2 &&\n+\n+\t\t# Change our mind about resolutions, just skip this patch\n+\t\ttest_must_fail git am --skip 2>errors &&\n+\t\ttest_path_is_dir newfile &&\n+\t\tgrep \"Updating .newfile. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"436329","messageId":"pull.1036.git.1632006923.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":null,"subject":"[PATCH 0/6] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:17Z","receivedAt":"2021-09-18T23:15:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on en/am-abort-fix.\n\nWe have multiple codepaths that delete untracked files/directories but\nshouldn't. There are also some codepaths where we delete untracked\nfiles/directories intentionally (based on mailing list discussion), but\nwhere that intent is not documented. Fix the documentation, add several new\n(mostly failing) testcases, fix some of the new testcases, and add comments\nabout some potential remaining problems. (I found these as a side-effect of\nlooking at [1], though [2] pointed out one explicitly while I was working on\nit.)\n\nNote that I'm using Junio's declaration about checkout -f and reset --hard\n(and also presuming that since read-tree --reset is porcelain that its\nbehavior should be left alone)[3] in this series.\n\nSIDENOTE about treating (some) ignored files as precious:\n\nThere's another related topic here that came up in the mailing list threads\nthat is separate even if similar: namely, treating ignored files as precious\ninstead of deleting them. I do not try to handle that here, but I believe\nthat would actually be relatively easy to handle. If you leave\nunpack_trees_options->dir as NULL, then ignored files are treated as\nprecious (my original patch 2 made that mistake). There's a few other\nlocations that already optionally set up unpack_trees_options->dir (a quick\nsearch for \"overwrite_ignore\" and \"overwrite-ignore\" will find them), so\nwe'd just need to implement that option flag in more places corresponding to\nthe new callsites (and perhaps make a global core.overwrite_ignored config\noption to affect all of these). Of course, doing so would globally treat\nignored files as precious rather than allowing them to be configured on a\nper-path basis, but honestly I think the idea of configuring ignored files\nas precious on a per-path basis sounds like insanity. (We have enough bugs\nwith untracked and ignored files without adding yet another type. Also,\ntla/baz was excessively confusing to me due in part to the number of types\nof files and I'd rather not see such ideas ported to git. And, of course,\nconfiguring per-path rules sounds like lots of work for end users to\nconfigure. There may be additional reasons against it.) So, if someone wants\nto pursue the precious-ignored concept then I'd much rather see it done as a\nglobal setting. Just my $0.02.\n\n[1] https://lore.kernel.org/git/xmqqv93n7q1v.fsf@gitster.g/ [2]\nhttps://lore.kernel.org/git/C357A648-8B13-45C3-9388-C0C7F7D40DAE@gmail.com/\n[3] https://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/\n\nElijah Newren (6):\n t2500: add various tests for nuking untracked files\n Split unpack_trees 'reset' flag into two for untracked handling\n unpack-trees: avoid nuking untracked dir in way of unmerged file\n unpack-trees: avoid nuking untracked dir in way of locally deleted\n file\n Comment important codepaths regarding nuking untracked files/dirs\n Documentation: call out commands that nuke untracked files/directories\n\n Documentation/git-checkout.txt | 5 +-\n Documentation/git-read-tree.txt | 5 +-\n Documentation/git-reset.txt | 3 +-\n builtin/am.c | 6 +-\n builtin/checkout.c | 10 +-\n builtin/read-tree.c | 11 +-\n builtin/reset.c | 15 +-\n builtin/stash.c | 3 +-\n builtin/submodule--helper.c | 4 +\n builtin/worktree.c | 5 +\n contrib/rerere-train.sh | 2 +-\n reset.c | 9 +-\n submodule.c | 1 +\n t/t1013-read-tree-submodule.sh | 4 +-\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n unpack-trees.c | 56 +++++--\n unpack-trees.h | 4 +-\n 17 files changed, 359 insertions(+), 28 deletions(-)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\n\nbase-commit: c5ead19ea282a288e01d86536349a4ae4a093e4b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1036%2Fnewren%2Funtracked_removal-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1036/newren/untracked_removal-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1036\n-- \ngitgitgadget\n"},{"id":"436330","messageId":"01bf850bb0f0796564e9363d7faeb792a594f684.1632006924.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH 4/6] unpack-trees: avoid nuking untracked dir in way of locally deleted file","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:21Z","receivedAt":"2021-09-18T23:15:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 4 ++++\n 2 files changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 017946a494f..d4d9dc928aa 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -218,7 +218,7 @@ test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \t)\n '\n \n-test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+test_expect_success 'git am --skip and untracked dir vs deleted file' '\n \ttest_setup_sequencing am_skip_and_untracked &&\n \t(\n \t\tcd sequencing_am_skip_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3b3d1c0ff40..858595a13f1 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2395,7 +2395,11 @@ static int deleted_entry(const struct cache_entry *ce,\n \t\tif (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n \t\t\treturn -1;\n \t\treturn 0;\n+\t} else {\n+\t\tif (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n+\t\t\treturn -1;\n \t}\n+\n \tif (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))\n \t\treturn -1;\n \tadd_entry(o, ce, CE_REMOVE, 0);\n-- \ngitgitgadget\n\n"},{"id":"436331","messageId":"a69117a1c9e6dae791e07e0df6701f581623f826.1632006923.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH 3/6] unpack-trees: avoid nuking untracked dir in way of unmerged file","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:20Z","receivedAt":"2021-09-18T23:15:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 35 ++++++++++++++++++++++++++++----\n 2 files changed, 32 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 786ec33d63a..017946a494f 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -197,7 +197,7 @@ test_expect_failure 'git stash and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \ttest_setup_sequencing am_abort_and_untracked &&\n \t(\n \t\tcd sequencing_am_abort_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d952eebe96a..3b3d1c0ff40 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2163,9 +2163,15 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n+enum absent_checking_type {\n+\tCOMPLETELY_ABSENT,\n+\tABSENT_ANY_DIRECTORY\n+};\n+\n static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\t\t const struct cache_entry *ce, struct stat *st,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tconst struct cache_entry *result;\n@@ -2200,6 +2206,10 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \t}\n \n+\t/* If we only care about directories, then we can remove */\n+\tif (absent_type == ABSENT_ANY_DIRECTORY)\n+\t\treturn 0;\n+\n \t/*\n \t * The previous round may already have decided to\n \t * delete this path, which is in a subdirectory that\n@@ -2220,6 +2230,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n */\n static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tint len;\n@@ -2245,7 +2256,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t\t\t\t\t\tNULL, o);\n \t\t\telse\n \t\t\t\tret = check_ok_to_remove(path, len, DT_UNKNOWN, NULL,\n-\t\t\t\t\t\t\t &st, error_type, o);\n+\t\t\t\t\t\t\t &st, error_type,\n+\t\t\t\t\t\t\t absent_type, o);\n \t\t}\n \t\tfree(path);\n \t\treturn ret;\n@@ -2260,7 +2272,7 @@ static int verify_absent_1(const struct cache_entry *ce,\n \n \t\treturn check_ok_to_remove(ce->name, ce_namelen(ce),\n \t\t\t\t\t ce_to_dtype(ce), ce, &st,\n-\t\t\t\t\t error_type, o);\n+\t\t\t\t\t error_type, absent_type, o);\n \t}\n }\n \n@@ -2270,14 +2282,23 @@ static int verify_absent(const struct cache_entry *ce,\n {\n \tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n+}\n+\n+static int verify_absent_if_directory(const struct cache_entry *ce,\n+\t\t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t\t struct unpack_trees_options *o)\n+{\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\t\treturn 0;\n+\treturn verify_absent_1(ce, error_type, ABSENT_ANY_DIRECTORY, o);\n }\n \n static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types error_type,\n \t\t\t\tstruct unpack_trees_options *o)\n {\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n }\n \n static int merged_entry(const struct cache_entry *ce,\n@@ -2351,6 +2372,12 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t * Previously unmerged entry left as an existence\n \t\t * marker by read_index_unmerged();\n \t\t */\n+\t\tif (verify_absent_if_directory(merge,\n+\t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n+\t\t\tdiscard_cache_entry(merge);\n+\t\t\treturn -1;\n+\t\t}\n+\n \t\tinvalidate_ce_path(old, o);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"436332","messageId":"45bd05a945f034d03555f04a1ba85835482dc591.1632006923.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:19Z","receivedAt":"2021-09-18T23:15:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nTraditionally, unpack_trees_options->reset was used to signal that it\nwas okay to delete any untracked files in the way. This was used by\n`git read-tree --reset`, but then started appearing in other places as\nwell. However, many of the other uses should not be deleting untracked\nfiles in the way. Split this into two separate fields:\n reset_nuke_untracked\n reset_keep_untracked\nand, since many code paths in unpack_trees need to be followed for both\nof these flags, introduce a third one for convenience:\n reset_either\nwhich is simply an or-ing of the other two.\n\nModify existing callers so that\n read-tree --reset\n reset --hard\n checkout --force\ncontinue using reset_nuke_untracked, but so that other callers,\nincluding\n am\n checkout without --force\n stash (though currently dead code; reset always had a value of 0)\n numerous callers from rebase/sequencer to reset_head()\nwill use the new reset_keep_untracked field.\n\nSigned-off-by: Elijah Newren \n---\n builtin/am.c | 6 +++++-\n builtin/checkout.c | 10 +++++++++-\n builtin/read-tree.c | 11 ++++++++---\n builtin/reset.c | 15 +++++++++++++--\n builtin/stash.c | 2 +-\n reset.c | 9 +++++++--\n t/t1013-read-tree-submodule.sh | 4 ++--\n t/t2500-untracked-overwriting.sh | 6 +++---\n unpack-trees.c | 17 ++++++++++++-----\n unpack-trees.h | 4 +++-\n 10 files changed, 63 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex c79e0167e98..dbe6cbe6a33 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1918,8 +1918,12 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.dst_index = &the_index;\n \topts.update = 1;\n \topts.merge = 1;\n-\topts.reset = reset;\n+\topts.reset_keep_untracked = reset;\n \topts.fn = twoway_merge;\n+\t/* Setup opts.dir so that ignored files in the way get overwritten */\n+\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\tsetup_standard_excludes(opts.dir);\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\n \ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b5d477919a7..ab0bb4d94f0 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -646,12 +646,20 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n \topts.head_idx = -1;\n \topts.update = worktree;\n \topts.skip_unmerged = !worktree;\n-\topts.reset = 1;\n+\tif (o->force)\n+\t\topts.reset_nuke_untracked = 1;\n+\telse\n+\t\topts.reset_keep_untracked = 1;\n \topts.merge = 1;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n+\tif (o->overwrite_ignore) {\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n+\t}\n \tinit_checkout_metadata(&opts.meta, info->refname,\n \t\t\t info->commit ? &info->commit->object.oid : null_oid(),\n \t\t\t NULL);\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 485e7b04794..8b94e1aa261 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -133,7 +133,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\t N_(\"3-way merge if no file level merging required\")),\n \t\tOPT_BOOL(0, \"aggressive\", &opts.aggressive,\n \t\t\t N_(\"3-way merge in presence of adds and removes\")),\n-\t\tOPT_BOOL(0, \"reset\", &opts.reset,\n+\t\tOPT_BOOL(0, \"reset\", &opts.reset_keep_untracked,\n \t\t\t N_(\"same as -m, but discard unmerged entries\")),\n \t\t{ OPTION_STRING, 0, \"prefix\", &opts.prefix, N_(\"/\"),\n \t\t N_(\"read the tree into the index under /\"),\n@@ -162,6 +162,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \topts.head_idx = -1;\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n+\tif (opts.reset_keep_untracked) {\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n+\t}\n \n \tgit_config(git_read_tree_config, NULL);\n \n@@ -171,7 +176,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n \n \tprefix_set = opts.prefix ? 1 : 0;\n-\tif (1 < opts.merge + opts.reset + prefix_set)\n+\tif (1 < opts.merge + opts.reset_keep_untracked + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n \t/*\n@@ -183,7 +188,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t * mode.\n \t */\n \n-\tif (opts.reset || opts.merge || opts.prefix) {\n+\tif (opts.reset_keep_untracked || opts.merge || opts.prefix) {\n \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n \t\t\tdie(_(\"You need to resolve your current index first\"));\n \t\tstage = opts.merge = 1;\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 43e855cb887..ba39c4882a6 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -10,6 +10,7 @@\n #define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"tag.h\"\n #include \"object.h\"\n@@ -70,9 +71,19 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\n-\t\t/* fallthrough */\n+\t\topts.reset_nuke_untracked = 1;\n+\t\tbreak;\n+\tcase MIXED:\n+\t\topts.reset_keep_untracked = 1; /* but opts.update=0, so untracked left alone */\n+\t\tbreak;\n \tdefault:\n-\t\topts.reset = 1;\n+\t\tBUG(\"invalid reset_type passed to reset_index\");\n+\t}\n+\tif (opts.reset_keep_untracked) {\n+\t\t/* Setup opts.dir so we can overwrite ignored files */\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n \t}\n \n \tread_cache_unmerged();\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..4ceb3581b47 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -256,7 +256,7 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n \topts.merge = 1;\n-\topts.reset = reset;\n+\topts.reset_keep_untracked = reset;\n \topts.update = update;\n \topts.fn = oneway_merge;\n \ndiff --git a/reset.c b/reset.c\nindex 79310ae071b..0880c76aef9 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -1,5 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"cache-tree.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"refs.h\"\n #include \"reset.h\"\n@@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.update = 1;\n \tunpack_tree_opts.merge = 1;\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n-\tif (!detach_head)\n-\t\tunpack_tree_opts.reset = 1;\n+\tif (!detach_head) {\n+\t\tunpack_tree_opts.reset_keep_untracked = 1;\n+\t\tunpack_tree_opts.dir = xcalloc(1, sizeof(*unpack_tree_opts.dir));\n+\t\tunpack_tree_opts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(unpack_tree_opts.dir);\n+\t}\n \n \tif (repo_read_index_unmerged(r) < 0) {\n \t\tret = error(_(\"could not read index\"));\ndiff --git a/t/t1013-read-tree-submodule.sh b/t/t1013-read-tree-submodule.sh\nindex b6df7444c05..4e485c223ad 100755\n--- a/t/t1013-read-tree-submodule.sh\n+++ b/t/t1013-read-tree-submodule.sh\n@@ -10,10 +10,10 @@ KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1\n \n test_submodule_switch_recursing_with_args \"read-tree -u -m\"\n \n-test_submodule_forced_switch_recursing_with_args \"read-tree -u --reset\"\n+test_submodule_switch_recursing_with_args \"read-tree -u --reset\"\n \n test_submodule_switch \"read-tree -u -m\"\n \n-test_submodule_forced_switch \"read-tree -u --reset\"\n+test_submodule_switch \"read-tree -u --reset\"\n \n test_done\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex a1a6dfa671e..786ec33d63a 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -92,7 +92,7 @@ test_setup_checkout_m () {\n \t)\n }\n \n-test_expect_failure 'checkout -m does not nuke untracked file' '\n+test_expect_success 'checkout -m does not nuke untracked file' '\n \ttest_setup_checkout_m &&\n \t(\n \t\tcd checkout &&\n@@ -138,7 +138,7 @@ test_setup_sequencing () {\n \t)\n }\n \n-test_expect_failure 'git rebase --abort and untracked files' '\n+test_expect_success 'git rebase --abort and untracked files' '\n \ttest_setup_sequencing rebase_abort_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_abort_and_untracked &&\n@@ -155,7 +155,7 @@ test_expect_failure 'git rebase --abort and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git rebase fast forwarding and untracked files' '\n+test_expect_success 'git rebase fast forwarding and untracked files' '\n \ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_fast_forward_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5786645f315..d952eebe96a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -301,7 +301,7 @@ static int check_submodule_move_head(const struct cache_entry *ce,\n \tif (!sub)\n \t\treturn 0;\n \n-\tif (o->reset)\n+\tif (o->reset_nuke_untracked)\n \t\tflags |= SUBMODULE_MOVE_HEAD_FORCE;\n \n \tif (submodule_move_head(ce->name, old_id, new_id, flags))\n@@ -1696,6 +1696,13 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n+\tif (o->reset_nuke_untracked && o->reset_keep_untracked)\n+\t\tBUG(\"reset_nuke_untracked and reset_keep_untracked are incompatible\");\n+\n+\to->reset_either = 0;\n+\tif (o->reset_nuke_untracked || o->reset_keep_untracked)\n+\t\to->reset_either = 1;\n+\n \ttrace_performance_enter();\n \ttrace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n \n@@ -1989,7 +1996,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n \t */\n \tif ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))\n \t\t; /* keep checking */\n-\telse if (o->reset || ce_uptodate(ce))\n+\telse if (o->reset_either || ce_uptodate(ce))\n \t\treturn 0;\n \n \tif (!lstat(ce->name, &st)) {\n@@ -2218,7 +2225,7 @@ static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n \n-\tif (o->index_only || o->reset || !o->update)\n+\tif (o->index_only || o->reset_nuke_untracked || !o->update)\n \t\treturn 0;\n \n \tlen = check_leading_path(ce->name, ce_namelen(ce), 0);\n@@ -2585,7 +2592,7 @@ int twoway_merge(const struct cache_entry * const *src,\n \n \tif (current) {\n \t\tif (current->ce_flags & CE_CONFLICTED) {\n-\t\t\tif (same(oldtree, newtree) || o->reset) {\n+\t\t\tif (same(oldtree, newtree) || o->reset_either) {\n \t\t\t\tif (!newtree)\n \t\t\t\t\treturn deleted_entry(current, current, o);\n \t\t\t\telse\n@@ -2683,7 +2690,7 @@ int oneway_merge(const struct cache_entry * const *src,\n \n \tif (old && same(old, a)) {\n \t\tint update = 0;\n-\t\tif (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old) &&\n+\t\tif (o->reset_either && o->update && !ce_uptodate(old) && !ce_skip_worktree(old) &&\n \t\t\t!(old->ce_flags & CE_FSMONITOR_VALID)) {\n \t\t\tstruct stat st;\n \t\t\tif (lstat(old->name, &st) ||\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 2d88b19dca7..c419bf8b1f9 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -46,7 +46,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n \n struct unpack_trees_options {\n-\tunsigned int reset,\n+\tunsigned int reset_nuke_untracked,\n+\t\t reset_keep_untracked,\n+\t\t reset_either, /* internal use only */\n \t\t merge,\n \t\t update,\n \t\t clone,\n-- \ngitgitgadget\n\n"},{"id":"436333","messageId":"6ea23d165cf1f9433831cb4a01abf6f483a8bc29.1632006924.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH 6/6] Documentation: call out commands that nuke untracked files/directories","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:23Z","receivedAt":"2021-09-18T23:15:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSome commands have traditionally also removed untracked files (or\ndirectories) that were in the way of a tracked file we needed. Document\nthese cases.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-checkout.txt | 5 +++--\n Documentation/git-read-tree.txt | 5 +++--\n Documentation/git-reset.txt | 3 ++-\n 3 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\nindex b1a6fe44997..d473c9bf387 100644\n--- a/Documentation/git-checkout.txt\n+++ b/Documentation/git-checkout.txt\n@@ -118,8 +118,9 @@ OPTIONS\n -f::\n --force::\n \tWhen switching branches, proceed even if the index or the\n-\tworking tree differs from `HEAD`. This is used to throw away\n-\tlocal changes.\n+\tworking tree differs from `HEAD`, and even if there are untracked\n+\tfiles in the way. This is used to throw away local changes and\n+\tany untracked files or directories that are in the way.\n +\n When checking out paths from the index, do not fail upon unmerged\n entries; instead, unmerged entries are ignored.\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 5fa8bab64c2..4731ec3283f 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -39,8 +39,9 @@ OPTIONS\n \n --reset::\n \tSame as -m, except that unmerged entries are discarded instead\n-\tof failing. When used with `-u`, updates leading to loss of\n-\tworking tree changes will not abort the operation.\n+\tof failing. When used with `-u`, updates leading to loss of\n+\tworking tree changes or untracked files or directories will not\n+\tabort the operation.\n \n -u::\n \tAfter a successful merge, update the files in the work\ndiff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\nindex 252e2d4e47d..6f7685f53d5 100644\n--- a/Documentation/git-reset.txt\n+++ b/Documentation/git-reset.txt\n@@ -69,7 +69,8 @@ linkgit:git-add[1]).\n \n --hard::\n \tResets the index and working tree. Any changes to tracked files in the\n-\tworking tree since `` are discarded.\n+\tworking tree since `` are discarded. Any untracked files or\n+\tdirectories in the way of writing any tracked files are simply deleted.\n \n --merge::\n \tResets the index and updates the files in the working tree that are\n-- \ngitgitgadget\n"},{"id":"436334","messageId":"60c5d6b4615a6ac4179ec6c10e17cca480bc147a.1632006924.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH 5/6] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-18T23:15:22Z","receivedAt":"2021-09-18T23:15:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last few commits we focused on code in unpack-trees.c that\nmistakenly removed untracked files or directories. There may be more of\nthose, but in this commit we change our focus: callers of toplevel\ncommands that are expected to remove untracked files or directories.\n\nAs noted previously, we have toplevel commands that are expected to\ndelete untracked files such as 'read-tree --reset', 'reset --hard', and\n'checkout --force'. However, that does not mean that other highlevel\ncommands that happen to call these other commands thought about or\nconveyed to users the possibility that untracked files could be removed.\nAudit the code for such callsites, and add comments near existing\ncallsites to mention whether these are safe or not.\n\nMy auditing is somewhat incomplete, though; it skipped several cases:\n * git-rebase--preserve-merges.sh: is in the process of being\n deprecated/removed, so I won't leave a note that there are\n likely more bugs in that script.\n * contrib/git-new-workdir: why is the -f flag being used in a new\n empty directory?? It shouldn't hurt, but it seems useless.\n * git-p4.py: Don't see why -f is needed for a new dir (maybe it's\n not and is just superfluous), but I'm not at all familiar with\n the p4 stuff\n * git-archimport.perl: Don't care; arch is long since dead\n * git-cvs*.perl: Don't care; cvs is long since dead\n\nSigned-off-by: Elijah Newren \n---\n builtin/stash.c | 1 +\n builtin/submodule--helper.c | 4 ++++\n builtin/worktree.c | 5 +++++\n contrib/rerere-train.sh | 2 +-\n submodule.c | 1 +\n 5 files changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 4ceb3581b47..ce5e0364c68 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1519,6 +1519,7 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\t} else {\n \t\t\tstruct child_process cp = CHILD_PROCESS_INIT;\n \t\t\tcp.git_cmd = 1;\n+\t\t\t/* BUG: this nukes untracked files in the way */\n \t\t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"-q\",\n \t\t\t\t \"--no-recurse-submodules\", NULL);\n \t\t\tif (run_command(&cp)) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex ef2776a9e45..a49242d15ae 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2864,6 +2864,10 @@ static int add_submodule(const struct add_data *add_data)\n \t\tprepare_submodule_repo_env(&cp.env_array);\n \t\tcp.git_cmd = 1;\n \t\tcp.dir = add_data->sm_path;\n+\t\t/*\n+\t\t * NOTE: we only get here if add_data->force is true, so\n+\t\t * passing --force to checkout is reasonable.\n+\t\t */\n \t\tstrvec_pushl(&cp.args, \"checkout\", \"-f\", \"-q\", NULL);\n \n \t\tif (add_data->branch) {\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 0d0a80da61f..383947ff54f 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -356,6 +356,11 @@ static int add_worktree(const char *path, const char *refname,\n \tif (opts->checkout) {\n \t\tcp.argv = NULL;\n \t\tstrvec_clear(&cp.args);\n+\t\t/*\n+\t\t * NOTE: reset --hard is okay here, because 'worktree add'\n+\t\t * refuses to work in an extant non-empty directory, so there\n+\t\t * is no risk of deleting untracked files.\n+\t\t */\n \t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"--no-recurse-submodules\", NULL);\n \t\tif (opts->quiet)\n \t\t\tstrvec_push(&cp.args, \"--quiet\");\ndiff --git a/contrib/rerere-train.sh b/contrib/rerere-train.sh\nindex eeee45dd341..75125d6ae00 100755\n--- a/contrib/rerere-train.sh\n+++ b/contrib/rerere-train.sh\n@@ -91,7 +91,7 @@ do\n \t\tgit checkout -q $commit -- .\n \t\tgit rerere\n \tfi\n-\tgit reset -q --hard\n+\tgit reset -q --hard # Might nuke untracked files...\n done\n \n if test -z \"$branch\"\ndiff --git a/submodule.c b/submodule.c\nindex 8e611fe1dbf..a9b71d585cf 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1866,6 +1866,7 @@ static void submodule_reset_index(const char *path)\n \n \tstrvec_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t get_super_prefix_or_empty(), path);\n+\t/* TODO: determine if this might overwright untracked files */\n \tstrvec_pushl(&cp.args, \"read-tree\", \"-u\", \"--reset\", NULL);\n \n \tstrvec_push(&cp.args, empty_tree_oid_hex());\n-- \ngitgitgadget\n\n"},{"id":"436365","messageId":"05be2442-d4e7-6007-a3ed-5f49f1b3d806@iee.email","threadId":"56534","inReplyTo":"6ea23d165cf1f9433831cb4a01abf6f483a8bc29.1632006924.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/6] Documentation: call out commands that nuke untracked files/directories","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-09-19T10:52:53Z","receivedAt":"2021-09-19T11:09:50Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"truly minor nit.\nOn 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n>\n> Some commands have traditionally also removed untracked files (or\n> directories) that were in the way of a tracked file we needed. Document\n> these cases.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> Documentation/git-checkout.txt | 5 +++--\n> Documentation/git-read-tree.txt | 5 +++--\n> Documentation/git-reset.txt | 3 ++-\n> 3 files changed, 8 insertions(+), 5 deletions(-)\n>\n> diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\n> index b1a6fe44997..d473c9bf387 100644\n> --- a/Documentation/git-checkout.txt\n> +++ b/Documentation/git-checkout.txt\n> @@ -118,8 +118,9 @@ OPTIONS\n> -f::\n> --force::\n> \tWhen switching branches, proceed even if the index or the\n> -\tworking tree differs from `HEAD`. This is used to throw away\n> -\tlocal changes.\n> +\tworking tree differs from `HEAD`, and even if there are untracked\n> +\tfiles in the way. This is used to throw away local changes and\ndouble space after full stop?\n> +\tany untracked files or directories that are in the way.\n> +\n> When checking out paths from the index, do not fail upon unmerged\n> entries; instead, unmerged entries are ignored.\n> diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\n> index 5fa8bab64c2..4731ec3283f 100644\n> --- a/Documentation/git-read-tree.txt\n> +++ b/Documentation/git-read-tree.txt\n> @@ -39,8 +39,9 @@ OPTIONS\n> \n> --reset::\n> \tSame as -m, except that unmerged entries are discarded instead\n> -\tof failing. When used with `-u`, updates leading to loss of\n> -\tworking tree changes will not abort the operation.\n> +\tof failing. When used with `-u`, updates leading to loss of\nIs the single space to double space change desired?\nI had the impression that the project had decided on single spaces, but\nI can't see anything in SubmittingPatches or CodingGuidelines. I don't\nthink there are DocumentationGuidelines.\n> +\tworking tree changes or untracked files or directories will not\n> +\tabort the operation.\n> \n> -u::\n> \tAfter a successful merge, update the files in the work\n> diff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\n> index 252e2d4e47d..6f7685f53d5 100644\n> --- a/Documentation/git-reset.txt\n> +++ b/Documentation/git-reset.txt\n> @@ -69,7 +69,8 @@ linkgit:git-add[1]).\n> \n> --hard::\n> \tResets the index and working tree. Any changes to tracked files in the\n> -\tworking tree since `` are discarded.\n> +\tworking tree since `` are discarded. Any untracked files or\nas above,  s /.  /. /  ?\n> +\tdirectories in the way of writing any tracked files are simply deleted.\n> \n> --merge::\n> \tResets the index and updates the files in the working tree that are\n--\nPhilip\n"},{"id":"436368","messageId":"f5b88c2f-b265-9342-4227-886049ad7bce@iee.email","threadId":"56534","inReplyTo":"05be2442-d4e7-6007-a3ed-5f49f1b3d806@iee.email","subject":"Re: [PATCH 6/6] Documentation: call out commands that nuke untracked files/directories","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-09-19T13:36:22Z","receivedAt":"2021-09-19T13:36:25Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 19/09/2021 11:52, Philip Oakley wrote:\n> truly minor nit.\n> On 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n>> From: Elijah Newren \n>>\n>> Some commands have traditionally also removed untracked files (or\n>> directories) that were in the way of a tracked file we needed. Document\n>> these cases.\n>>\n>> Signed-off-by: Elijah Newren \n>> ---\n>> Documentation/git-checkout.txt | 5 +++--\n>> Documentation/git-read-tree.txt | 5 +++--\n>> Documentation/git-reset.txt | 3 ++-\n>> 3 files changed, 8 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\n>> index b1a6fe44997..d473c9bf387 100644\n>> --- a/Documentation/git-checkout.txt\n>> +++ b/Documentation/git-checkout.txt\n>> @@ -118,8 +118,9 @@ OPTIONS\n>> -f::\n>> --force::\n>> \tWhen switching branches, proceed even if the index or the\n>> -\tworking tree differs from `HEAD`. This is used to throw away\n>> -\tlocal changes.\n>> +\tworking tree differs from `HEAD`, and even if there are untracked\n>> +\tfiles in the way. This is used to throw away local changes and\n> double space after full stop?\n>> +\tany untracked files or directories that are in the way.\n>> +\n>> When checking out paths from the index, do not fail upon unmerged\n>> entries; instead, unmerged entries are ignored.\n>> diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\n>> index 5fa8bab64c2..4731ec3283f 100644\n>> --- a/Documentation/git-read-tree.txt\n>> +++ b/Documentation/git-read-tree.txt\n>> @@ -39,8 +39,9 @@ OPTIONS\n>> \n>> --reset::\n>> \tSame as -m, except that unmerged entries are discarded instead\n>> -\tof failing. When used with `-u`, updates leading to loss of\n>> -\tworking tree changes will not abort the operation.\n>> +\tof failing. When used with `-u`, updates leading to loss of\n> Is the single space to double space change desired?\n> I had the impression that the project had decided on single spaces, but\n> I can't see anything in SubmittingPatches or CodingGuidelines. I don't\n> think there are DocumentationGuidelines.\n\nI may have been mistaken about any project decision. I had a look around\nthe archives and only came up with a 2008 post [1] by Junio that, at the\ntime, was looking for two spaces after the full stop.\n\nIt's not clear if we consider the man pages to be 'typeset' such that a\nsingle space would be the norm, or mono-spaced 'typewriter' style (two\nspaces). There's much commentary in the Wikipedia article [2].\n\nSo still minor.\n--\nPhilip\n\n>> +\tworking tree changes or untracked files or directories will not\n>> +\tabort the operation.\n>> \n>> -u::\n>> \tAfter a successful merge, update the files in the work\n>> diff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\n>> index 252e2d4e47d..6f7685f53d5 100644\n>> --- a/Documentation/git-reset.txt\n>> +++ b/Documentation/git-reset.txt\n>> @@ -69,7 +69,8 @@ linkgit:git-add[1]).\n>> \n>> --hard::\n>> \tResets the index and working tree. Any changes to tracked files in the\n>> -\tworking tree since `` are discarded.\n>> +\tworking tree since `` are discarded. Any untracked files or\n> as above,  s /.  /. /  ?\n>> +\tdirectories in the way of writing any tracked files are simply deleted.\n>> \n>> --merge::\n>> \tResets the index and updates the files in the working tree that are\n>>\n[1] https://lore.kernel.org/git/7vfxtu3fku.fsf@gitster.siamese.dyndns.org/\n[2] https://en.wikipedia.org/wiki/Sentence_spacing\n"},{"id":"436369","messageId":"871r5kd6dg.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"b634136a74b37a6f73bc3038f79031e040423883.1632006923.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/6] t2500: add various tests for nuking untracked files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-19T13:44:55Z","receivedAt":"2021-09-19T13:48:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 18 2021, Elijah Newren via GitGitGadget wrote:\n\n> +\ttest_create_repo reset_$1 &&\n\ns/test_create_repo/git init/ these days (also for the rest).\n\n> +\t\tmkdir foo.t &&\n> +\t\techo precious >foo.t/file &&\n> +\t\tcp foo.t/file expect &&\n> +\n> +\t\ttest_must_fail git reset --merge work 2>error &&\n> +\t\ttest_cmp expect foo.t/file &&\n> +\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n\nThe test is ambiguous about whether we complain about foo.t/file, or\nfoo.t, if there was foo.t{file,file-two} would we complain just once or\ntwice?\n\nI think it's just the directory, but probably worthwhile for the test to\nmake the distinction. If it's a \"a/sub/dir/file\" do we complain about\n\"a/\" or \"a/sub/dir/\" ?\n"},{"id":"436370","messageId":"87wnncbrm4.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"45bd05a945f034d03555f04a1ba85835482dc591.1632006923.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-19T13:48:49Z","receivedAt":"2021-09-19T13:51:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 18 2021, Elijah Newren via GitGitGadget wrote:\n\n> -\topts.reset = reset;\n> +\topts.reset_keep_untracked = reset;\n> \topts.fn = twoway_merge;\n> +\t/* Setup opts.dir so that ignored files in the way get overwritten */\n> +\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\tsetup_standard_excludes(opts.dir);\n\nIs the \"opts.dir\" free'd later somehow?\n\n> \topts.head_idx = -1;\n> \topts.update = worktree;\n> \topts.skip_unmerged = !worktree;\n> -\topts.reset = 1;\n> +\tif (o->force)\n> +\t\topts.reset_nuke_untracked = 1;\n> +\telse\n> +\t\topts.reset_keep_untracked = 1;\n\nIn both cases opts.reset_keep_untracked is set to 1, I assume it's a\nmistake, aside from that perhaps betteras:\n\n opts.reset_keep_untracked = o->force; /* or !o->force, depending... */\n\n> \topts.merge = 1;\n> \topts.fn = oneway_merge;\n> \topts.verbose_update = o->show_progress;\n> \topts.src_index = &the_index;\n> \topts.dst_index = &the_index;\n> +\tif (o->overwrite_ignore) {\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n\nditto potential leak.\n\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(opts.dir);\n> +\t}\n\n\nditto (also more omitted).\n"},{"id":"436371","messageId":"87sfy0brk5.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"01bf850bb0f0796564e9363d7faeb792a594f684.1632006924.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/6] unpack-trees: avoid nuking untracked dir in way of locally deleted file","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-19T13:52:28Z","receivedAt":"2021-09-19T13:53:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 18 2021, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren \n>\n> Signed-off-by: Elijah Newren \n> ---\n> t/t2500-untracked-overwriting.sh | 2 +-\n> unpack-trees.c | 4 ++++\n> 2 files changed, 5 insertions(+), 1 deletion(-)\n>\n> diff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\n> index 017946a494f..d4d9dc928aa 100755\n> --- a/t/t2500-untracked-overwriting.sh\n> +++ b/t/t2500-untracked-overwriting.sh\n> @@ -218,7 +218,7 @@ test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n> \t)\n> '\n> \n> -test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n> +test_expect_success 'git am --skip and untracked dir vs deleted file' '\n> \ttest_setup_sequencing am_skip_and_untracked &&\n> \t(\n> \t\tcd sequencing_am_skip_and_untracked &&\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 3b3d1c0ff40..858595a13f1 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -2395,7 +2395,11 @@ static int deleted_entry(const struct cache_entry *ce,\n> \t\tif (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n> \t\t\treturn -1;\n> \t\treturn 0;\n> +\t} else {\n> +\t\tif (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n> +\t\t\treturn -1;\n> \t}\n\nMaybe just \"else if\" ?\n\n[...]\n\n> +\n\nStray whitespace change\n"},{"id":"436406","messageId":"acef3628-9542-d777-2534-577de9707e15@gmail.com","threadId":"56534","inReplyTo":"45bd05a945f034d03555f04a1ba85835482dc591.1632006923.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-09-20T10:19:07Z","receivedAt":"2021-09-20T10:19:12Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Traditionally, unpack_trees_options->reset was used to signal that it\n> was okay to delete any untracked files in the way. This was used by\n> `git read-tree --reset`, but then started appearing in other places as\n> well. However, many of the other uses should not be deleting untracked\n> files in the way. Split this into two separate fields:\n> reset_nuke_untracked\n> reset_keep_untracked\n> and, since many code paths in unpack_trees need to be followed for both\n> of these flags, introduce a third one for convenience:\n> reset_either\n> which is simply an or-ing of the other two.\n\nSee [1] for an alternative approach that used an enum instead of adding \nmutually exclusive flags.\n\n> Modify existing callers so that\n> read-tree --reset\n\nit would be nice if read-tree callers could choose whether they want to \nremove untracked files or not - that could always be added later. This \npatch changes the behavior of 'git read-tree -m -u' (and other commands) \nso that they will overwrite ignored files - I'm in favor of that change \nbut it would be good to spell out the change in the commit message.\n\n> reset --hard\n> checkout --force\n\nI often use checkout --force to clear unwanted changes when I'm \nswitching branches, I'd prefer it if it did not remove untracked files.\n\n> continue using reset_nuke_untracked, but so that other callers,\n> including\n> am\n> checkout without --force\n> stash (though currently dead code; reset always had a value of 0)\n> numerous callers from rebase/sequencer to reset_head()\n> will use the new reset_keep_untracked field.\n\nThis is great. In the discussion around [1] there is a mention of 'git \ncheckout ' which also overwrites untracked files. It does not \nuse unpack_trees() so is arguably outside the scope of what you're doing \nhere but it might be worth mentioning.\n\n> [...]\n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index 485e7b04794..8b94e1aa261 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -133,7 +133,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> \t\t\t N_(\"3-way merge if no file level merging required\")),\n> \t\tOPT_BOOL(0, \"aggressive\", &opts.aggressive,\n> \t\t\t N_(\"3-way merge in presence of adds and removes\")),\n> -\t\tOPT_BOOL(0, \"reset\", &opts.reset,\n> +\t\tOPT_BOOL(0, \"reset\", &opts.reset_keep_untracked,\n> \t\t\t N_(\"same as -m, but discard unmerged entries\")),\n> \t\t{ OPTION_STRING, 0, \"prefix\", &opts.prefix, N_(\"/\"),\n> \t\t N_(\"read the tree into the index under /\"),\n> @@ -162,6 +162,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> \topts.head_idx = -1;\n> \topts.src_index = &the_index;\n> \topts.dst_index = &the_index;\n> +\tif (opts.reset_keep_untracked) {\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(opts.dir);\n> +\t}\n\nDoes this clobber any excludes added by --exclude-per-directory?\n\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 43e855cb887..ba39c4882a6 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -10,6 +10,7 @@\n> #define USE_THE_INDEX_COMPATIBILITY_MACROS\n> #include \"builtin.h\"\n> #include \"config.h\"\n> +#include \"dir.h\"\n> #include \"lockfile.h\"\n> #include \"tag.h\"\n> #include \"object.h\"\n> @@ -70,9 +71,19 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> \t\tbreak;\n> \tcase HARD:\n> \t\topts.update = 1;\n> -\t\t/* fallthrough */\n> +\t\topts.reset_nuke_untracked = 1;\n> +\t\tbreak;\n> +\tcase MIXED:\n> +\t\topts.reset_keep_untracked = 1; /* but opts.update=0, so untracked left alone */\n> +\t\tbreak;\n> \tdefault:\n> -\t\topts.reset = 1;\n> +\t\tBUG(\"invalid reset_type passed to reset_index\");\n\nThere is no case SOFT: but in that case we don't call reset_index() so \nwe're OK.\n\n> diff --git a/reset.c b/reset.c\n> index 79310ae071b..0880c76aef9 100644\n> --- a/reset.c\n> +++ b/reset.c\n> @@ -1,5 +1,6 @@\n> #include \"git-compat-util.h\"\n> #include \"cache-tree.h\"\n> +#include \"dir.h\"\n> #include \"lockfile.h\"\n> #include \"refs.h\"\n> #include \"reset.h\"\n> @@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n> \tunpack_tree_opts.update = 1;\n> \tunpack_tree_opts.merge = 1;\n> \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n> -\tif (!detach_head)\n> -\t\tunpack_tree_opts.reset = 1;\n\nUnrelated to this patch but this looks dodgy to me. For 'git rebase \n ' where is ahead of we skip the \nrebase and use reset_head() to checkout without 'detach_head' \nset. I think this should be checking 'reset_hard' instead of 'detach_head'\n\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 5786645f315..d952eebe96a 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -301,7 +301,7 @@ static int check_submodule_move_head(const struct cache_entry *ce,\n> \tif (!sub)\n> \t\treturn 0;\n> \n> -\tif (o->reset)\n> +\tif (o->reset_nuke_untracked)\n> \t\tflags |= SUBMODULE_MOVE_HEAD_FORCE;\n> \n> \tif (submodule_move_head(ce->name, old_id, new_id, flags))\n> @@ -1696,6 +1696,13 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \tif (len > MAX_UNPACK_TREES)\n> \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n> \n> +\tif (o->reset_nuke_untracked && o->reset_keep_untracked)\n> +\t\tBUG(\"reset_nuke_untracked and reset_keep_untracked are incompatible\");\n> +\n> +\to->reset_either = 0;\n> +\tif (o->reset_nuke_untracked || o->reset_keep_untracked)\n> +\t\to->reset_either = 1;\n\n\no->reset_either = o->reset_nuke_untracked | o->reset_keep_untracked\n\n\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index 2d88b19dca7..c419bf8b1f9 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -46,7 +46,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n> void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n> \n> struct unpack_trees_options {\n> -\tunsigned int reset,\n> +\tunsigned int reset_nuke_untracked,\n> +\t\t reset_keep_untracked,\n> +\t\t reset_either, /* internal use only */\n\nI think I prefer the enum approach in [1] but I'm biased and I'm not \nsure it's worth getting excited about. Thanks for working on this it \nwill be great to have git stop overwriting untracked files so often.\n\nBest Wishes\n\nPhillip\n\n[1] \nhttps://lore.kernel.org/git/20190501101403.20294-1-phillip.wood123@gmail.com/\n\n\n> \t\t merge,\n> \t\t update,\n> \t\t clone,\n> \n\n"},{"id":"436418","messageId":"CABPp-BEqiYKAqP1QvMB1rqqQjiP7Yn0tbhaHb7OdVfcpaE3hjw@mail.gmail.com","threadId":"56534","inReplyTo":"871r5kd6dg.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/6] t2500: add various tests for nuking untracked files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-20T14:48:28Z","receivedAt":"2021-09-20T14:48:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 19, 2021 at 6:47 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Sat, Sep 18 2021, Elijah Newren via GitGitGadget wrote:\n>\n> > + test_create_repo reset_$1 &&\n>\n> s/test_create_repo/git init/ these days (also for the rest).\n\nAh, from your f0d4d398e2 (\"test-lib: split up and deprecate\ntest_create_repo()\", 2021-05-10). Interesting history; I'll\nswitchover to git init.\n\n> > + mkdir foo.t &&\n> > + echo precious >foo.t/file &&\n> > + cp foo.t/file expect &&\n> > +\n> > + test_must_fail git reset --merge work 2>error &&\n> > + test_cmp expect foo.t/file &&\n> > + grep \"Updating.*foo.t.*would lose untracked files\" error\n>\n> The test is ambiguous about whether we complain about foo.t/file, or\n> foo.t, if there was foo.t{file,file-two} would we complain just once or\n> twice?\n>\n> I think it's just the directory, but probably worthwhile for the test to\n> make the distinction. If it's a \"a/sub/dir/file\" do we complain about\n> \"a/\" or \"a/sub/dir/\" ?\n\nYeah, I'll switch it to grep \"Updating .foo.t. would lose untracked files\" error\n\nto make it clearer (where I'm using '.' instead of attempting to\nescape single quote characters appropriately).\n"},{"id":"436423","messageId":"CABPp-BFiKuTi_uSmWXc4FF-zYzODhtVdv8AQbz_--bz6XZjp6A@mail.gmail.com","threadId":"56534","inReplyTo":"87wnncbrm4.fsf@evledraar.gmail.com","subject":"Re: [PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-20T15:20:59Z","receivedAt":"2021-09-20T15:21:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 19, 2021 at 6:51 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Sat, Sep 18 2021, Elijah Newren via GitGitGadget wrote:\n>\n> > - opts.reset = reset;\n> > + opts.reset_keep_untracked = reset;\n> > opts.fn = twoway_merge;\n> > + /* Setup opts.dir so that ignored files in the way get overwritten */\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n>\n> Is the \"opts.dir\" free'd later somehow?\n\nNo, much like other callsites that set this up (e.g.\nbuiltin/checkout.c), there isn't a place that frees it. In copying\nhow they worked, I also copied their bugs. ;-)\n\nI'm tempted to move this code into setup_unpack_trees_porcelain() and\nthen free it in clear_unpack_trees_porcelain()...though not all\ncallers make use of those functions. Hmm...\n\n> > opts.head_idx = -1;\n> > opts.update = worktree;\n> > opts.skip_unmerged = !worktree;\n> > - opts.reset = 1;\n> > + if (o->force)\n> > + opts.reset_nuke_untracked = 1;\n> > + else\n> > + opts.reset_keep_untracked = 1;\n>\n> In both cases opts.reset_keep_untracked is set to 1, I assume it's a\n> mistake\n\nNo, it only sets opts.reset_keep_untracked to 1 when o->force is\nfalse. If o->force is true, it instead sets opts.reset_nuke_untracked\nto 1. There's no mistake there.\n\n> > opts.merge = 1;\n> > opts.fn = oneway_merge;\n> > opts.verbose_update = o->show_progress;\n> > opts.src_index = &the_index;\n> > opts.dst_index = &the_index;\n> > + if (o->overwrite_ignore) {\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n>\n> ditto potential leak.\n>\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n> > + }\n>\n>\n> ditto (also more omitted).\n\nYep.\n"},{"id":"436435","messageId":"CABPp-BGnWeFiJ11x8j1J+yjgVB9r858S47y40h8cFQYF4TR1HA@mail.gmail.com","threadId":"56534","inReplyTo":"acef3628-9542-d777-2534-577de9707e15@gmail.com","subject":"Re: [PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-20T16:05:07Z","receivedAt":"2021-09-20T16:05:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Sep 20, 2021 at 3:19 AM Phillip Wood wrote:\n>\n> On 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Traditionally, unpack_trees_options->reset was used to signal that it\n> > was okay to delete any untracked files in the way. This was used by\n> > `git read-tree --reset`, but then started appearing in other places as\n> > well. However, many of the other uses should not be deleting untracked\n> > files in the way. Split this into two separate fields:\n> > reset_nuke_untracked\n> > reset_keep_untracked\n> > and, since many code paths in unpack_trees need to be followed for both\n> > of these flags, introduce a third one for convenience:\n> > reset_either\n> > which is simply an or-ing of the other two.\n>\n> See [1] for an alternative approach that used an enum instead of adding\n> mutually exclusive flags.\n\nOh, interesting. Any reason you didn't pursue that old series further?\n\n> > Modify existing callers so that\n> > read-tree --reset\n>\n> it would be nice if read-tree callers could choose whether they want to\n> remove untracked files or not - that could always be added later. This\n> patch changes the behavior of 'git read-tree -m -u' (and other commands)\n> so that they will overwrite ignored files - I'm in favor of that change\n> but it would be good to spell out the change in the commit message.\n\nThose commands made no distinction between untracked and ignored files\npreviously, and overwrote all of them. This patch changes those\ncommands so that they stop overwriting untracked files, unless those\nfiles are ignored. So, there's no change in behavior for ignored\nfiles, only for non-ignored untracked files.\n\nYour suggestion to point out the behavior relative to ignored files in\nthe commit message, though, is probably a good idea. I should mention\nthat ignored files will continue to be removed by these commands.\n\n> > reset --hard\n> > checkout --force\n>\n> I often use checkout --force to clear unwanted changes when I'm\n> switching branches, I'd prefer it if it did not remove untracked files.\n\nI originally started down that path to see what it looked like, but\nJunio weighed in and explicitly called out checkout --force as being a\ncommand that should remove untracked files in the way. See\nhttps://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/. Seems you\nalso felt that way previously, at\nhttps://lore.kernel.org/git/d4c36a24-b40c-a6ca-7a05-572ab93a0101@gmail.com/\n-- any reason for your change of opinion?\n\n> > continue using reset_nuke_untracked, but so that other callers,\n> > including\n> > am\n> > checkout without --force\n> > stash (though currently dead code; reset always had a value of 0)\n> > numerous callers from rebase/sequencer to reset_head()\n> > will use the new reset_keep_untracked field.\n>\n> This is great. In the discussion around [1] there is a mention of 'git\n> checkout ' which also overwrites untracked files. It does not\n> use unpack_trees() so is arguably outside the scope of what you're doing\n> here but it might be worth mentioning.\n\nOh, that's interesting. Yeah, that's worth mentioning and perhaps digging into.\n\n>\n> > [...]\n> > diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> > index 485e7b04794..8b94e1aa261 100644\n> > --- a/builtin/read-tree.c\n> > +++ b/builtin/read-tree.c\n> > @@ -133,7 +133,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> > N_(\"3-way merge if no file level merging required\")),\n> > OPT_BOOL(0, \"aggressive\", &opts.aggressive,\n> > N_(\"3-way merge in presence of adds and removes\")),\n> > - OPT_BOOL(0, \"reset\", &opts.reset,\n> > + OPT_BOOL(0, \"reset\", &opts.reset_keep_untracked,\n> > N_(\"same as -m, but discard unmerged entries\")),\n> > { OPTION_STRING, 0, \"prefix\", &opts.prefix, N_(\"/\"),\n> > N_(\"read the tree into the index under /\"),\n> > @@ -162,6 +162,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> > opts.head_idx = -1;\n> > opts.src_index = &the_index;\n> > opts.dst_index = &the_index;\n> > + if (opts.reset_keep_untracked) {\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n> > + }\n>\n> Does this clobber any excludes added by --exclude-per-directory?\n\nOh, um...I've basically implemented a --exclude-standard and assumed\nit was passed, ignoring whatever setting of opts.dir was already set\nup by exclude-per-directory. Oops.\n\n> > diff --git a/builtin/reset.c b/builtin/reset.c\n> > index 43e855cb887..ba39c4882a6 100644\n> > --- a/builtin/reset.c\n> > +++ b/builtin/reset.c\n> > @@ -10,6 +10,7 @@\n> > #define USE_THE_INDEX_COMPATIBILITY_MACROS\n> > #include \"builtin.h\"\n> > #include \"config.h\"\n> > +#include \"dir.h\"\n> > #include \"lockfile.h\"\n> > #include \"tag.h\"\n> > #include \"object.h\"\n> > @@ -70,9 +71,19 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> > break;\n> > case HARD:\n> > opts.update = 1;\n> > - /* fallthrough */\n> > + opts.reset_nuke_untracked = 1;\n> > + break;\n> > + case MIXED:\n> > + opts.reset_keep_untracked = 1; /* but opts.update=0, so untracked left alone */\n> > + break;\n> > default:\n> > - opts.reset = 1;\n> > + BUG(\"invalid reset_type passed to reset_index\");\n>\n> There is no case SOFT: but in that case we don't call reset_index() so\n> we're OK.\n>\n> > diff --git a/reset.c b/reset.c\n> > index 79310ae071b..0880c76aef9 100644\n> > --- a/reset.c\n> > +++ b/reset.c\n> > @@ -1,5 +1,6 @@\n> > #include \"git-compat-util.h\"\n> > #include \"cache-tree.h\"\n> > +#include \"dir.h\"\n> > #include \"lockfile.h\"\n> > #include \"refs.h\"\n> > #include \"reset.h\"\n> > @@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n> > unpack_tree_opts.update = 1;\n> > unpack_tree_opts.merge = 1;\n> > init_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n> > - if (!detach_head)\n> > - unpack_tree_opts.reset = 1;\n>\n> Unrelated to this patch but this looks dodgy to me. For 'git rebase\n> ' where is ahead of we skip the\n> rebase and use reset_head() to checkout without 'detach_head'\n> set. I think this should be checking 'reset_hard' instead of 'detach_head'\n>\n> > diff --git a/unpack-trees.c b/unpack-trees.c\n> > index 5786645f315..d952eebe96a 100644\n> > --- a/unpack-trees.c\n> > +++ b/unpack-trees.c\n> > @@ -301,7 +301,7 @@ static int check_submodule_move_head(const struct cache_entry *ce,\n> > if (!sub)\n> > return 0;\n> >\n> > - if (o->reset)\n> > + if (o->reset_nuke_untracked)\n> > flags |= SUBMODULE_MOVE_HEAD_FORCE;\n> >\n> > if (submodule_move_head(ce->name, old_id, new_id, flags))\n> > @@ -1696,6 +1696,13 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> > if (len > MAX_UNPACK_TREES)\n> > die(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n> >\n> > + if (o->reset_nuke_untracked && o->reset_keep_untracked)\n> > + BUG(\"reset_nuke_untracked and reset_keep_untracked are incompatible\");\n> > +\n> > + o->reset_either = 0;\n> > + if (o->reset_nuke_untracked || o->reset_keep_untracked)\n> > + o->reset_either = 1;\n>\n> \n> o->reset_either = o->reset_nuke_untracked | o->reset_keep_untracked\n> \n\nGoes away entirely if we adopt your enum suggestion.\n\n> > diff --git a/unpack-trees.h b/unpack-trees.h\n> > index 2d88b19dca7..c419bf8b1f9 100644\n> > --- a/unpack-trees.h\n> > +++ b/unpack-trees.h\n> > @@ -46,7 +46,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n> > void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n> >\n> > struct unpack_trees_options {\n> > - unsigned int reset,\n> > + unsigned int reset_nuke_untracked,\n> > + reset_keep_untracked,\n> > + reset_either, /* internal use only */\n>\n> I think I prefer the enum approach in [1] but I'm biased and I'm not\n> sure it's worth getting excited about. Thanks for working on this it\n> will be great to have git stop overwriting untracked files so often.\n\nI think the enum approach makes sense; I'll try it out.\n"},{"id":"436437","messageId":"CABPp-BEk5zQNwzUdFR=iG6d-d5u8pUEgLuiF7+hs7HT7p5BCqQ@mail.gmail.com","threadId":"56534","inReplyTo":"87sfy0brk5.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/6] unpack-trees: avoid nuking untracked dir in way of locally deleted file","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-20T16:12:04Z","receivedAt":"2021-09-20T16:13:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 19, 2021 at 6:52 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Sat, Sep 18 2021, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren \n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > t/t2500-untracked-overwriting.sh | 2 +-\n> > unpack-trees.c | 4 ++++\n> > 2 files changed, 5 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\n> > index 017946a494f..d4d9dc928aa 100755\n> > --- a/t/t2500-untracked-overwriting.sh\n> > +++ b/t/t2500-untracked-overwriting.sh\n> > @@ -218,7 +218,7 @@ test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n> > )\n> > '\n> >\n> > -test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n> > +test_expect_success 'git am --skip and untracked dir vs deleted file' '\n> > test_setup_sequencing am_skip_and_untracked &&\n> > (\n> > cd sequencing_am_skip_and_untracked &&\n> > diff --git a/unpack-trees.c b/unpack-trees.c\n> > index 3b3d1c0ff40..858595a13f1 100644\n> > --- a/unpack-trees.c\n> > +++ b/unpack-trees.c\n> > @@ -2395,7 +2395,11 @@ static int deleted_entry(const struct cache_entry *ce,\n> > if (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n> > return -1;\n> > return 0;\n> > + } else {\n> > + if (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n> > + return -1;\n> > }\n>\n> Maybe just \"else if\" ?\n\nYeah, that makes sense.\n\n> [...]\n\nThat's kind of misleading. ;-) You trimmed out a single line here,\nand in particular one that only contained a trailing curly brace.\nThus, your \"trimming\" here actually made things longer.\n\n>\n> > +\n>\n> Stray whitespace change\n\nNo, the whitespace addition was after making the if-block above it\nmore complex with the extra else block. That if-block is now\napproximately 2/3 of the length of the function, and is the part that\nis relevant to the comment above it. Since the code that follows the\nif-block is separate from the comment above and the if-block became\nmore complex, it felt natural to add a bit of spacing. So, it wasn't\nstray, but intentional and related to the changes above.\n"},{"id":"436440","messageId":"CABPp-BEVDs4rRGV3WzeK929CwhefhwwyodSs6=5iP0-dt3cUKg@mail.gmail.com","threadId":"56534","inReplyTo":"f5b88c2f-b265-9342-4227-886049ad7bce@iee.email","subject":"Re: [PATCH 6/6] Documentation: call out commands that nuke untracked files/directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-20T16:29:36Z","receivedAt":"2021-09-20T16:29:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 19, 2021 at 6:36 AM Philip Oakley wrote:\n>\n> On 19/09/2021 11:52, Philip Oakley wrote:\n> > truly minor nit.\n> > On 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n> >> From: Elijah Newren \n> >>\n> >> Some commands have traditionally also removed untracked files (or\n> >> directories) that were in the way of a tracked file we needed. Document\n> >> these cases.\n> >>\n> >> Signed-off-by: Elijah Newren \n> >> ---\n> >> Documentation/git-checkout.txt | 5 +++--\n> >> Documentation/git-read-tree.txt | 5 +++--\n> >> Documentation/git-reset.txt | 3 ++-\n> >> 3 files changed, 8 insertions(+), 5 deletions(-)\n> >>\n> >> diff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\n> >> index b1a6fe44997..d473c9bf387 100644\n> >> --- a/Documentation/git-checkout.txt\n> >> +++ b/Documentation/git-checkout.txt\n> >> @@ -118,8 +118,9 @@ OPTIONS\n> >> -f::\n> >> --force::\n> >> When switching branches, proceed even if the index or the\n> >> - working tree differs from `HEAD`. This is used to throw away\n> >> - local changes.\n> >> + working tree differs from `HEAD`, and even if there are untracked\n> >> + files in the way. This is used to throw away local changes and\n> > double space after full stop?\n\nNote that the original also had a double space after full stop\n(looking at the previous sentence).\n\n> >> + any untracked files or directories that are in the way.\n> >> +\n> >> When checking out paths from the index, do not fail upon unmerged\n> >> entries; instead, unmerged entries are ignored.\n> >> diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\n> >> index 5fa8bab64c2..4731ec3283f 100644\n> >> --- a/Documentation/git-read-tree.txt\n> >> +++ b/Documentation/git-read-tree.txt\n> >> @@ -39,8 +39,9 @@ OPTIONS\n> >>\n> >> --reset::\n> >> Same as -m, except that unmerged entries are discarded instead\n> >> - of failing. When used with `-u`, updates leading to loss of\n> >> - working tree changes will not abort the operation.\n> >> + of failing. When used with `-u`, updates leading to loss of\n> > Is the single space to double space change desired?\n> > I had the impression that the project had decided on single spaces, but\n> > I can't see anything in SubmittingPatches or CodingGuidelines. I don't\n> > think there are DocumentationGuidelines.\n\nDouble space is better as per Junio's declaration here:\nhttps://lore.kernel.org/git/xmqqftchkext.fsf@gitster.c.googlers.com/\n\nHowever, it's so minor that I wouldn't normally bother to change it\nspecifically. In this case, I originally was tweaking the sentence\nbefore as well, and when modifying both sentences I just naturally put\ntwo spaces after the full stop between them. But then I re-read and\ndecided to reword and ended up restoring the original first sentence\nand didn't even notice that resulted in a change of spacing at the end\nof the sentence.\n\n> I may have been mistaken about any project decision. I had a look around\n> the archives and only came up with a 2008 post [1] by Junio that, at the\n> time, was looking for two spaces after the full stop.\n>\n> It's not clear if we consider the man pages to be 'typeset' such that a\n> single space would be the norm, or mono-spaced 'typewriter' style (two\n> spaces). There's much commentary in the Wikipedia article [2].\n>\n> So still minor.\n\nYes, the double space is \"more correct\" in the sources as per Junio's\ndeclaration in the link I provided above, but I agree it's pretty\nminor.\n\n> [1] https://lore.kernel.org/git/7vfxtu3fku.fsf@gitster.siamese.dyndns.org/\n> [2] https://en.wikipedia.org/wiki/Sentence_spacing\n"},{"id":"436526","messageId":"a3433f37-4a22-faa1-16a7-cbc01a24af6e@gmail.com","threadId":"56534","inReplyTo":"CABPp-BGnWeFiJ11x8j1J+yjgVB9r858S47y40h8cFQYF4TR1HA@mail.gmail.com","subject":"Re: [PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-09-20T18:11:32Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 20/09/2021 17:05, Elijah Newren wrote:\n> On Mon, Sep 20, 2021 at 3:19 AM Phillip Wood wrote:\n>>\n>> On 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n>>>\n>>> Traditionally, unpack_trees_options->reset was used to signal that it\n>>> was okay to delete any untracked files in the way. This was used by\n>>> `git read-tree --reset`, but then started appearing in other places as\n>>> well. However, many of the other uses should not be deleting untracked\n>>> files in the way. Split this into two separate fields:\n>>> reset_nuke_untracked\n>>> reset_keep_untracked\n>>> and, since many code paths in unpack_trees need to be followed for both\n>>> of these flags, introduce a third one for convenience:\n>>> reset_either\n>>> which is simply an or-ing of the other two.\n>>\n>> See [1] for an alternative approach that used an enum instead of adding\n>> mutually exclusive flags.\n> \n> Oh, interesting. Any reason you didn't pursue that old series further?\n\nMainly lack of time/distracted by other things. I was also not that \nconfident about modifying the unpack_trees() code. Duy was very helpful \nbut then moved on quite soon after I posted that series I think and \nthere didn't seem to be much interest from others.\n\n>>> Modify existing callers so that\n>>> read-tree --reset\n>>\n>> it would be nice if read-tree callers could choose whether they want to\n>> remove untracked files or not - that could always be added later. This\n>> patch changes the behavior of 'git read-tree -m -u' (and other commands)\n>> so that they will overwrite ignored files - I'm in favor of that change\n>> but it would be good to spell out the change in the commit message.\n> \n> Those commands made no distinction between untracked and ignored files\n> previously, and overwrote all of them.\n\nAre you sure, I thought 'read-tree -m -u' unlike 'read-tree --reset -u' \nrefused to overwrite untracked and ignored files currently.\n\n> This patch changes those\n> commands so that they stop overwriting untracked files, unless those\n> files are ignored. So, there's no change in behavior for ignored\n> files, only for non-ignored untracked files.\n> \n> Your suggestion to point out the behavior relative to ignored files in\n> the commit message, though, is probably a good idea. I should mention\n> that ignored files will continue to be removed by these commands.\n> \n>>> reset --hard\n>>> checkout --force\n>>\n>> I often use checkout --force to clear unwanted changes when I'm\n>> switching branches, I'd prefer it if it did not remove untracked files.\n> \n> I originally started down that path to see what it looked like, but\n> Junio weighed in and explicitly called out checkout --force as being a\n> command that should remove untracked files in the way. See\n> https://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/. Seems you\n> also felt that way previously, at\n> https://lore.kernel.org/git/d4c36a24-b40c-a6ca-7a05-572ab93a0101@gmail.com/\n> -- any reason for your change of opinion?\n\nI've no recollection of writing that email! When I was writing today I \nthought that 'checkout -f' and 'switch --discard-changes' behaved the \nsame way but it appears from that other message that they do not so \nmaybe it is OK for 'checkout -f' to nuke everything if there is a safe \nalternative available in the form of 'switch --discard-changes'\n\n>>> continue using reset_nuke_untracked, but so that other callers,\n>>> including\n>>> am\n>>> checkout without --force\n>>> stash (though currently dead code; reset always had a value of 0)\n>>> numerous callers from rebase/sequencer to reset_head()\n>>> will use the new reset_keep_untracked field.\n>>\n>> This is great. In the discussion around [1] there is a mention of 'git\n>> checkout ' which also overwrites untracked files. It does not\n>> use unpack_trees() so is arguably outside the scope of what you're doing\n>> here but it might be worth mentioning.\n> \n> Oh, that's interesting. Yeah, that's worth mentioning and perhaps digging into.\n\nIt'd be fantastic to fix that if you have the time and inclination to \ndig into it.\n\nBest Wishes\n\nPhillip\n\n>>> [...]\n>>> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>>> index 485e7b04794..8b94e1aa261 100644\n>>> --- a/builtin/read-tree.c\n>>> +++ b/builtin/read-tree.c\n>>> @@ -133,7 +133,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>>> N_(\"3-way merge if no file level merging required\")),\n>>> OPT_BOOL(0, \"aggressive\", &opts.aggressive,\n>>> N_(\"3-way merge in presence of adds and removes\")),\n>>> - OPT_BOOL(0, \"reset\", &opts.reset,\n>>> + OPT_BOOL(0, \"reset\", &opts.reset_keep_untracked,\n>>> N_(\"same as -m, but discard unmerged entries\")),\n>>> { OPTION_STRING, 0, \"prefix\", &opts.prefix, N_(\"/\"),\n>>> N_(\"read the tree into the index under /\"),\n>>> @@ -162,6 +162,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>>> opts.head_idx = -1;\n>>> opts.src_index = &the_index;\n>>> opts.dst_index = &the_index;\n>>> + if (opts.reset_keep_untracked) {\n>>> + opts.dir = xcalloc(1, sizeof(*opts.dir));\n>>> + opts.dir->flags |= DIR_SHOW_IGNORED;\n>>> + setup_standard_excludes(opts.dir);\n>>> + }\n>>\n>> Does this clobber any excludes added by --exclude-per-directory?\n> \n> Oh, um...I've basically implemented a --exclude-standard and assumed\n> it was passed, ignoring whatever setting of opts.dir was already set\n> up by exclude-per-directory. Oops.\n> \n>>> diff --git a/builtin/reset.c b/builtin/reset.c\n>>> index 43e855cb887..ba39c4882a6 100644\n>>> --- a/builtin/reset.c\n>>> +++ b/builtin/reset.c\n>>> @@ -10,6 +10,7 @@\n>>> #define USE_THE_INDEX_COMPATIBILITY_MACROS\n>>> #include \"builtin.h\"\n>>> #include \"config.h\"\n>>> +#include \"dir.h\"\n>>> #include \"lockfile.h\"\n>>> #include \"tag.h\"\n>>> #include \"object.h\"\n>>> @@ -70,9 +71,19 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n>>> break;\n>>> case HARD:\n>>> opts.update = 1;\n>>> - /* fallthrough */\n>>> + opts.reset_nuke_untracked = 1;\n>>> + break;\n>>> + case MIXED:\n>>> + opts.reset_keep_untracked = 1; /* but opts.update=0, so untracked left alone */\n>>> + break;\n>>> default:\n>>> - opts.reset = 1;\n>>> + BUG(\"invalid reset_type passed to reset_index\");\n>>\n>> There is no case SOFT: but in that case we don't call reset_index() so\n>> we're OK.\n>>\n>>> diff --git a/reset.c b/reset.c\n>>> index 79310ae071b..0880c76aef9 100644\n>>> --- a/reset.c\n>>> +++ b/reset.c\n>>> @@ -1,5 +1,6 @@\n>>> #include \"git-compat-util.h\"\n>>> #include \"cache-tree.h\"\n>>> +#include \"dir.h\"\n>>> #include \"lockfile.h\"\n>>> #include \"refs.h\"\n>>> #include \"reset.h\"\n>>> @@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n>>> unpack_tree_opts.update = 1;\n>>> unpack_tree_opts.merge = 1;\n>>> init_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n>>> - if (!detach_head)\n>>> - unpack_tree_opts.reset = 1;\n>>\n>> Unrelated to this patch but this looks dodgy to me. For 'git rebase\n>> ' where is ahead of we skip the\n>> rebase and use reset_head() to checkout without 'detach_head'\n>> set. I think this should be checking 'reset_hard' instead of 'detach_head'\n>>\n>>> diff --git a/unpack-trees.c b/unpack-trees.c\n>>> index 5786645f315..d952eebe96a 100644\n>>> --- a/unpack-trees.c\n>>> +++ b/unpack-trees.c\n>>> @@ -301,7 +301,7 @@ static int check_submodule_move_head(const struct cache_entry *ce,\n>>> if (!sub)\n>>> return 0;\n>>>\n>>> - if (o->reset)\n>>> + if (o->reset_nuke_untracked)\n>>> flags |= SUBMODULE_MOVE_HEAD_FORCE;\n>>>\n>>> if (submodule_move_head(ce->name, old_id, new_id, flags))\n>>> @@ -1696,6 +1696,13 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>>> if (len > MAX_UNPACK_TREES)\n>>> die(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n>>>\n>>> + if (o->reset_nuke_untracked && o->reset_keep_untracked)\n>>> + BUG(\"reset_nuke_untracked and reset_keep_untracked are incompatible\");\n>>> +\n>>> + o->reset_either = 0;\n>>> + if (o->reset_nuke_untracked || o->reset_keep_untracked)\n>>> + o->reset_either = 1;\n>>\n>> \n>> o->reset_either = o->reset_nuke_untracked | o->reset_keep_untracked\n>> \n> \n> Goes away entirely if we adopt your enum suggestion.\n> \n>>> diff --git a/unpack-trees.h b/unpack-trees.h\n>>> index 2d88b19dca7..c419bf8b1f9 100644\n>>> --- a/unpack-trees.h\n>>> +++ b/unpack-trees.h\n>>> @@ -46,7 +46,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n>>> void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n>>>\n>>> struct unpack_trees_options {\n>>> - unsigned int reset,\n>>> + unsigned int reset_nuke_untracked,\n>>> + reset_keep_untracked,\n>>> + reset_either, /* internal use only */\n>>\n>> I think I prefer the enum approach in [1] but I'm biased and I'm not\n>> sure it's worth getting excited about. Thanks for working on this it\n>> will be great to have git stop overwriting untracked files so often.\n> \n> I think the enum approach makes sense; I'll try it out.\n> \n\n"},{"id":"436915","messageId":"CABPp-BGBX=QR0FVML1iukOSUir5d_st1jbuMjqsb6aEJ96wHhw@mail.gmail.com","threadId":"56534","inReplyTo":"a3433f37-4a22-faa1-16a7-cbc01a24af6e@gmail.com","subject":"Re: [PATCH 2/6] Split unpack_trees 'reset' flag into two for untracked handling","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-24T02:27:35Z","receivedAt":"2021-09-24T02:27:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Sep 20, 2021 at 11:11 AM Phillip Wood wrote:\n>\n> On 20/09/2021 17:05, Elijah Newren wrote:\n> > On Mon, Sep 20, 2021 at 3:19 AM Phillip Wood wrote:\n> >>\n> >> On 19/09/2021 00:15, Elijah Newren via GitGitGadget wrote:\n> >>> From: Elijah Newren \n> >>>\n> >>> Traditionally, unpack_trees_options->reset was used to signal that it\n> >>> was okay to delete any untracked files in the way. This was used by\n> >>> `git read-tree --reset`, but then started appearing in other places as\n> >>> well. However, many of the other uses should not be deleting untracked\n> >>> files in the way. Split this into two separate fields:\n> >>> reset_nuke_untracked\n> >>> reset_keep_untracked\n> >>> and, since many code paths in unpack_trees need to be followed for both\n> >>> of these flags, introduce a third one for convenience:\n> >>> reset_either\n> >>> which is simply an or-ing of the other two.\n> >>\n> >> See [1] for an alternative approach that used an enum instead of adding\n> >> mutually exclusive flags.\n> >\n> > Oh, interesting. Any reason you didn't pursue that old series further?\n>\n> Mainly lack of time/distracted by other things. I was also not that\n> confident about modifying the unpack_trees() code. Duy was very helpful\n> but then moved on quite soon after I posted that series I think and\n> there didn't seem to be much interest from others.\n>\n> >>> Modify existing callers so that\n> >>> read-tree --reset\n> >>\n> >> it would be nice if read-tree callers could choose whether they want to\n> >> remove untracked files or not - that could always be added later. This\n> >> patch changes the behavior of 'git read-tree -m -u' (and other commands)\n> >> so that they will overwrite ignored files - I'm in favor of that change\n> >> but it would be good to spell out the change in the commit message.\n> >\n> > Those commands made no distinction between untracked and ignored files\n> > previously, and overwrote all of them.\n>\n> Are you sure, I thought 'read-tree -m -u' unlike 'read-tree --reset -u'\n> refused to overwrite untracked and ignored files currently.\n\nDoh, I was thinking of read-tree --reset -u rather than read-tree -m\n-u, despite the fact that you explicitly called out (and I even quoted\nyou on) the latter. You are right.\n\n> > This patch changes those\n> > commands so that they stop overwriting untracked files, unless those\n> > files are ignored. So, there's no change in behavior for ignored\n> > files, only for non-ignored untracked files.\n> >\n> > Your suggestion to point out the behavior relative to ignored files in\n> > the commit message, though, is probably a good idea. I should mention\n> > that ignored files will continue to be removed by these commands.\n> >\n> >>> reset --hard\n> >>> checkout --force\n> >>\n> >> I often use checkout --force to clear unwanted changes when I'm\n> >> switching branches, I'd prefer it if it did not remove untracked files.\n> >\n> > I originally started down that path to see what it looked like, but\n> > Junio weighed in and explicitly called out checkout --force as being a\n> > command that should remove untracked files in the way. See\n> > https://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/. Seems you\n> > also felt that way previously, at\n> > https://lore.kernel.org/git/d4c36a24-b40c-a6ca-7a05-572ab93a0101@gmail.com/\n> > -- any reason for your change of opinion?\n>\n> I've no recollection of writing that email! When I was writing today I\n> thought that 'checkout -f' and 'switch --discard-changes' behaved the\n> same way but it appears from that other message that they do not so\n> maybe it is OK for 'checkout -f' to nuke everything if there is a safe\n> alternative available in the form of 'switch --discard-changes'\n>\n> >>> continue using reset_nuke_untracked, but so that other callers,\n> >>> including\n> >>> am\n> >>> checkout without --force\n> >>> stash (though currently dead code; reset always had a value of 0)\n> >>> numerous callers from rebase/sequencer to reset_head()\n> >>> will use the new reset_keep_untracked field.\n> >>\n> >> This is great. In the discussion around [1] there is a mention of 'git\n> >> checkout ' which also overwrites untracked files. It does not\n> >> use unpack_trees() so is arguably outside the scope of what you're doing\n> >> here but it might be worth mentioning.\n> >\n> > Oh, that's interesting. Yeah, that's worth mentioning and perhaps digging into.\n>\n> It'd be fantastic to fix that if you have the time and inclination to\n> dig into it.\n\nI won't include it in this series, but I'll throw it on my (long) pile\nof things to perhaps look at later.\n\n\nThanks for the suggestions and pointers in your reviews!\n"},{"id":"436926","messageId":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.git.1632006923.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:03Z","receivedAt":"2021-09-24T06:37:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on en/am-abort-fix.\n\nWe have multiple codepaths that delete untracked files/directories but\nshouldn't. There are also some codepaths where we delete untracked\nfiles/directories intentionally (based on mailing list discussion), but\nwhere that intent is not documented. Fix the documentation, add several new\n(mostly failing) testcases, fix some of the new testcases, and add comments\nabout some potential remaining problems. (I found these as a side-effect of\nlooking at [1], though [2] pointed out one explicitly while I was working on\nit.)\n\nNote that I'm using Junio's declaration about checkout -f and reset --hard\n(and also presuming that since read-tree --reset is porcelain that its\nbehavior should be left alone)[3] in this series.\n\nChanges since v1:\n\n * Various small cleanups (suggested by Ævar)\n * Fixed memory leaks of unpack_trees_opts->dir (also suggested by Ævar)\n * Use an enum for unpack_trees_options->reset, instead of multiple fields\n (suggested by Phillip)\n * Avoid changing behavior for cases not setting unpack_trees_options.reset\n > 0 (even if it may make sense to nuke ignored files when running either\n read-tree -m -u or the various reset flavors run internally by\n rebase/sequencer); we can revisit that later.\n\nSIDENOTE about treating (some) ignored files as precious:\n\nThere's another related topic here that came up in the mailing list threads\nthat is separate even if similar: namely, treating ignored files as precious\ninstead of deleting them. I do not try to handle that here, but I believe\nthat would actually be relatively easy to handle. If you leave\nunpack_trees_options->dir as NULL, then ignored files are treated as\nprecious (my original patch 2 made that mistake). There's a few other\nlocations that already optionally set up unpack_trees_options->dir (a quick\nsearch for \"overwrite_ignore\" and \"overwrite-ignore\" will find them), so\nwe'd just need to implement that option flag in more places corresponding to\nthe new callsites (and perhaps make a global core.overwrite_ignored config\noption to affect all of these). Of course, doing so would globally treat\nignored files as precious rather than allowing them to be configured on a\nper-path basis, but honestly I think the idea of configuring ignored files\nas precious on a per-path basis sounds like insanity. (We have enough bugs\nwith untracked and ignored files without adding yet another type. Also,\ntla/baz was excessively confusing to me due in part to the number of types\nof files and I'd rather not see such ideas ported to git. And, of course,\nconfiguring per-path rules sounds like lots of work for end users to\nconfigure. There may be additional reasons against it.) So, if someone wants\nto pursue the precious-ignored concept then I'd much rather see it done as a\nglobal setting. Just my $0.02.\n\n[1] https://lore.kernel.org/git/xmqqv93n7q1v.fsf@gitster.g/ [2]\nhttps://lore.kernel.org/git/C357A648-8B13-45C3-9388-C0C7F7D40DAE@gmail.com/\n[3] https://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/\n\nElijah Newren (6):\n t2500: add various tests for nuking untracked files\n Change unpack_trees' 'reset' flag into an enum\n unpack-trees: avoid nuking untracked dir in way of unmerged file\n unpack-trees: avoid nuking untracked dir in way of locally deleted\n file\n Comment important codepaths regarding nuking untracked files/dirs\n Documentation: call out commands that nuke untracked files/directories\n\n Documentation/git-checkout.txt | 5 +-\n Documentation/git-read-tree.txt | 5 +-\n Documentation/git-reset.txt | 3 +-\n builtin/am.c | 13 +-\n builtin/checkout.c | 18 ++-\n builtin/read-tree.c | 3 +\n builtin/reset.c | 20 ++-\n builtin/stash.c | 18 ++-\n builtin/submodule--helper.c | 4 +\n builtin/worktree.c | 5 +\n contrib/rerere-train.sh | 2 +-\n reset.c | 13 +-\n submodule.c | 1 +\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n unpack-trees.c | 44 +++++-\n unpack-trees.h | 11 +-\n 16 files changed, 387 insertions(+), 22 deletions(-)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\n\nbase-commit: c5ead19ea282a288e01d86536349a4ae4a093e4b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1036%2Fnewren%2Funtracked_removal-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1036/newren/untracked_removal-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1036\n\nRange-diff vs v1:\n\n 1: b634136a74b ! 1: 9460a49c7ed t2500: add various tests for nuking untracked files\n @@ t/t2500-untracked-overwriting.sh (new)\n +. ./test-lib.sh\n +\n +test_setup_reset () {\n -+\ttest_create_repo reset_$1 &&\n ++\tgit init reset_$1 &&\n +\t(\n +\t\tcd reset_$1 &&\n +\t\ttest_commit init &&\n @@ t/t2500-untracked-overwriting.sh (new)\n +\n +\t\ttest_must_fail git reset --merge work 2>error &&\n +\t\ttest_cmp expect foo.t/file &&\n -+\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n ++\t\tgrep \"Updating .foo.t. would lose untracked files\" error\n +\t)\n +'\n +\n @@ t/t2500-untracked-overwriting.sh (new)\n +'\n +\n +test_setup_checkout_m () {\n -+\ttest_create_repo checkout &&\n ++\tgit init checkout &&\n +\t(\n +\t\tcd checkout &&\n +\t\ttest_commit init &&\n @@ t/t2500-untracked-overwriting.sh (new)\n +'\n +\n +test_setup_sequencing () {\n -+\ttest_create_repo sequencing_$1 &&\n ++\tgit init sequencing_$1 &&\n +\t(\n +\t\tcd sequencing_$1 &&\n +\t\ttest_commit init &&\n 2: 45bd05a945f ! 2: b77692b8f49 Split unpack_trees 'reset' flag into two for untracked handling\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - Split unpack_trees 'reset' flag into two for untracked handling\n + Change unpack_trees' 'reset' flag into an enum\n \n Traditionally, unpack_trees_options->reset was used to signal that it\n was okay to delete any untracked files in the way. This was used by\n `git read-tree --reset`, but then started appearing in other places as\n well. However, many of the other uses should not be deleting untracked\n - files in the way. Split this into two separate fields:\n - reset_nuke_untracked\n - reset_keep_untracked\n - and, since many code paths in unpack_trees need to be followed for both\n - of these flags, introduce a third one for convenience:\n - reset_either\n - which is simply an or-ing of the other two.\n + files in the way. Change this value to an enum so that a value of 1\n + (i.e. \"true\") can be split into two:\n + UNPACK_RESET_PROTECT_UNTRACKED,\n + UNPACK_RESET_OVERWRITE_UNTRACKED\n + In order to catch accidental misuses, define with the enum a special\n + value of\n + UNPACK_RESET_INVALID = 1\n + which will trigger a BUG().\n \n Modify existing callers so that\n read-tree --reset\n reset --hard\n checkout --force\n - continue using reset_nuke_untracked, but so that other callers,\n - including\n + continue using the UNPACK_RESET_OVERWRITE_UNTRACKED logic, while other\n + callers, including\n am\n checkout without --force\n stash (though currently dead code; reset always had a value of 0)\n numerous callers from rebase/sequencer to reset_head()\n - will use the new reset_keep_untracked field.\n + will use the new UNPACK_RESET_PROTECT_UNTRACKED value.\n +\n + In order to protect untracked files but still allow deleting of ignored\n + files, we also have to setup unpack_trees_opt.dir. It may make sense to\n + set up unpack_trees_opt.dir in more cases, but here I tried to only do\n + so in cases where we switched from deleting all untracked files to\n + avoiding doing so (i.e. where we now use\n + UNPACK_RESET_PROTECT_UNTRACKED).\n +\n + Also, note that 'git checkout ' currently also allows\n + overwriting untracked files. That case should also be fixed, but it\n + does not use unpack_trees() and thus is outside the scope of the current\n + changes.\n \n Signed-off-by: Elijah Newren \n \n @@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote,\n \topts.update = 1;\n \topts.merge = 1;\n -\topts.reset = reset;\n -+\topts.reset_keep_untracked = reset;\n ++\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n \topts.fn = twoway_merge;\n -+\t/* Setup opts.dir so that ignored files in the way get overwritten */\n -+\topts.dir = xcalloc(1, sizeof(*opts.dir));\n -+\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\tsetup_standard_excludes(opts.dir);\n ++\tif (opts.reset) {\n ++\t\t/* Allow ignored files in the way to get overwritten */\n ++\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n ++\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n ++\t\tsetup_standard_excludes(opts.dir);\n ++\t}\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\n \n +@@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n + \t\treturn -1;\n + \t}\n + \n ++\tif (opts.reset) {\n ++\t\tdir_clear(opts.dir);\n ++\t\tFREE_AND_NULL(opts.dir);\n ++\t}\n ++\n + \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n + \t\tdie(_(\"unable to write new index file\"));\n + \n \n ## builtin/checkout.c ##\n @@ builtin/checkout.c: static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n + {\n + \tstruct unpack_trees_options opts;\n + \tstruct tree_desc tree_desc;\n ++\tint unpack_trees_ret;\n + \n + \tmemset(&opts, 0, sizeof(opts));\n \topts.head_idx = -1;\n \topts.update = worktree;\n \topts.skip_unmerged = !worktree;\n -\topts.reset = 1;\n -+\tif (o->force)\n -+\t\topts.reset_nuke_untracked = 1;\n -+\telse\n -+\t\topts.reset_keep_untracked = 1;\n ++\topts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n ++\t\t\t\tUNPACK_RESET_PROTECT_UNTRACKED;\n \topts.merge = 1;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n @@ builtin/checkout.c: static int reset_tree(struct tree *tree, const struct checko\n \tinit_checkout_metadata(&opts.meta, info->refname,\n \t\t\t info->commit ? &info->commit->object.oid : null_oid(),\n \t\t\t NULL);\n + \tparse_tree(tree);\n + \tinit_tree_desc(&tree_desc, tree->buffer, tree->size);\n +-\tswitch (unpack_trees(1, &tree_desc, &opts)) {\n ++\tunpack_trees_ret = unpack_trees(1, &tree_desc, &opts);\n ++\n ++\tif (o->overwrite_ignore) {\n ++\t\tdir_clear(opts.dir);\n ++\t\tFREE_AND_NULL(opts.dir);\n ++\t}\n ++\n ++\tswitch (unpack_trees_ret) {\n + \tcase -2:\n + \t\t*writeout_error = 1;\n + \t\t/*\n \n ## builtin/read-tree.c ##\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n - \t\t\t N_(\"3-way merge if no file level merging required\")),\n - \t\tOPT_BOOL(0, \"aggressive\", &opts.aggressive,\n - \t\t\t N_(\"3-way merge in presence of adds and removes\")),\n --\t\tOPT_BOOL(0, \"reset\", &opts.reset,\n -+\t\tOPT_BOOL(0, \"reset\", &opts.reset_keep_untracked,\n - \t\t\t N_(\"same as -m, but discard unmerged entries\")),\n - \t\t{ OPTION_STRING, 0, \"prefix\", &opts.prefix, N_(\"/\"),\n - \t\t N_(\"read the tree into the index under /\"),\n -@@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n - \topts.head_idx = -1;\n - \topts.src_index = &the_index;\n - \topts.dst_index = &the_index;\n -+\tif (opts.reset_keep_untracked) {\n -+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n -+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opts.dir);\n -+\t}\n - \n - \tgit_config(git_read_tree_config, NULL);\n - \n -@@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n - \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n - \n - \tprefix_set = opts.prefix ? 1 : 0;\n --\tif (1 < opts.merge + opts.reset + prefix_set)\n -+\tif (1 < opts.merge + opts.reset_keep_untracked + prefix_set)\n + \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n ++\tif (opts.reset)\n ++\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n ++\n \t/*\n -@@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n - \t * mode.\n - \t */\n - \n --\tif (opts.reset || opts.merge || opts.prefix) {\n -+\tif (opts.reset_keep_untracked || opts.merge || opts.prefix) {\n - \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n - \t\t\tdie(_(\"You need to resolve your current index first\"));\n - \t\tstage = opts.merge = 1;\n + \t * NEEDSWORK\n + \t *\n \n ## builtin/reset.c ##\n @@\n @@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id\n \tcase HARD:\n \t\topts.update = 1;\n -\t\t/* fallthrough */\n -+\t\topts.reset_nuke_untracked = 1;\n ++\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n +\t\tbreak;\n +\tcase MIXED:\n -+\t\topts.reset_keep_untracked = 1; /* but opts.update=0, so untracked left alone */\n ++\t\topts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n ++\t\t/* but opts.update=0, so working tree not updated */\n +\t\tbreak;\n \tdefault:\n -\t\topts.reset = 1;\n +\t\tBUG(\"invalid reset_type passed to reset_index\");\n +\t}\n -+\tif (opts.reset_keep_untracked) {\n ++\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n +\t\t/* Setup opts.dir so we can overwrite ignored files */\n +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n @@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id\n \t}\n \n \tread_cache_unmerged();\n +@@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n + \tret = 0;\n + \n + out:\n ++\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n ++\t\tdir_clear(opts.dir);\n ++\t\tFREE_AND_NULL(opts.dir);\n ++\t}\n + \tfor (i = 0; i < nr; i++)\n + \t\tfree((void *)desc[i].buffer);\n + \treturn ret;\n \n ## builtin/stash.c ##\n +@@ builtin/stash.c: static int reset_tree(struct object_id *i_tree, int update, int reset)\n + \tstruct tree_desc t[MAX_UNPACK_TREES];\n + \tstruct tree *tree;\n + \tstruct lock_file lock_file = LOCK_INIT;\n ++\tint unpack_trees_ret;\n + \n + \tread_cache_preload(NULL);\n + \tif (refresh_cache(REFRESH_QUIET))\n @@ builtin/stash.c: static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n \topts.merge = 1;\n -\topts.reset = reset;\n -+\topts.reset_keep_untracked = reset;\n ++\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n ++\tif (opts.reset) {\n ++\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n ++\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n ++\t\tsetup_standard_excludes(opts.dir);\n ++\t}\n \topts.update = update;\n \topts.fn = oneway_merge;\n \n +-\tif (unpack_trees(nr_trees, t, &opts))\n ++\tunpack_trees_ret = unpack_trees(nr_trees, t, &opts);\n ++\n ++\tif (opts.reset) {\n ++\t\tdir_clear(opts.dir);\n ++\t\tFREE_AND_NULL(opts.dir);\n ++\t}\n ++\n ++\tif (unpack_trees_ret)\n + \t\treturn -1;\n + \n + \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \n ## reset.c ##\n @@\n @@ reset.c: int reset_head(struct repository *r, struct object_id *oid, const char\n -\tif (!detach_head)\n -\t\tunpack_tree_opts.reset = 1;\n +\tif (!detach_head) {\n -+\t\tunpack_tree_opts.reset_keep_untracked = 1;\n ++\t\tunpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n +\t\tunpack_tree_opts.dir = xcalloc(1, sizeof(*unpack_tree_opts.dir));\n +\t\tunpack_tree_opts.dir->flags |= DIR_SHOW_IGNORED;\n +\t\tsetup_standard_excludes(unpack_tree_opts.dir);\n @@ reset.c: int reset_head(struct repository *r, struct object_id *oid, const char\n \n \tif (repo_read_index_unmerged(r) < 0) {\n \t\tret = error(_(\"could not read index\"));\n -\n - ## t/t1013-read-tree-submodule.sh ##\n -@@ t/t1013-read-tree-submodule.sh: KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1\n - \n - test_submodule_switch_recursing_with_args \"read-tree -u -m\"\n - \n --test_submodule_forced_switch_recursing_with_args \"read-tree -u --reset\"\n -+test_submodule_switch_recursing_with_args \"read-tree -u --reset\"\n +@@ reset.c: reset_head_refs:\n + \t\t\t oid_to_hex(oid), \"1\", NULL);\n \n - test_submodule_switch \"read-tree -u -m\"\n - \n --test_submodule_forced_switch \"read-tree -u --reset\"\n -+test_submodule_switch \"read-tree -u --reset\"\n - \n - test_done\n + leave_reset_head:\n ++\tif (unpack_tree_opts.dir) {\n ++\t\tdir_clear(unpack_tree_opts.dir);\n ++\t\tFREE_AND_NULL(unpack_tree_opts.dir);\n ++\t}\n + \tstrbuf_release(&msg);\n + \trollback_lock_file(&lock);\n + \tclear_unpack_trees_porcelain(&unpack_tree_opts);\n \n ## t/t2500-untracked-overwriting.sh ##\n @@ t/t2500-untracked-overwriting.sh: test_setup_checkout_m () {\n @@ t/t2500-untracked-overwriting.sh: test_expect_failure 'git rebase --abort and un\n \t\tcd sequencing_rebase_fast_forward_and_untracked &&\n \n ## unpack-trees.c ##\n -@@ unpack-trees.c: static int check_submodule_move_head(const struct cache_entry *ce,\n - \tif (!sub)\n - \t\treturn 0;\n - \n --\tif (o->reset)\n -+\tif (o->reset_nuke_untracked)\n - \t\tflags |= SUBMODULE_MOVE_HEAD_FORCE;\n - \n - \tif (submodule_move_head(ce->name, old_id, new_id, flags))\n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n - \tif (len > MAX_UNPACK_TREES)\n - \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n + \tstruct pattern_list pl;\n + \tint free_pattern_list = 0;\n \n -+\tif (o->reset_nuke_untracked && o->reset_keep_untracked)\n -+\t\tBUG(\"reset_nuke_untracked and reset_keep_untracked are incompatible\");\n ++\tif (o->reset == UNPACK_RESET_INVALID)\n ++\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n +\n -+\to->reset_either = 0;\n -+\tif (o->reset_nuke_untracked || o->reset_keep_untracked)\n -+\t\to->reset_either = 1;\n -+\n - \ttrace_performance_enter();\n - \ttrace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n - \n -@@ unpack-trees.c: static int verify_uptodate_1(const struct cache_entry *ce,\n - \t */\n - \tif ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))\n - \t\t; /* keep checking */\n --\telse if (o->reset || ce_uptodate(ce))\n -+\telse if (o->reset_either || ce_uptodate(ce))\n - \t\treturn 0;\n + \tif (len > MAX_UNPACK_TREES)\n + \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n - \tif (!lstat(ce->name, &st)) {\n @@ unpack-trees.c: static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n \n -\tif (o->index_only || o->reset || !o->update)\n -+\tif (o->index_only || o->reset_nuke_untracked || !o->update)\n ++\tif (o->index_only || !o->update ||\n ++\t o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED)\n \t\treturn 0;\n \n \tlen = check_leading_path(ce->name, ce_namelen(ce), 0);\n -@@ unpack-trees.c: int twoway_merge(const struct cache_entry * const *src,\n - \n - \tif (current) {\n - \t\tif (current->ce_flags & CE_CONFLICTED) {\n --\t\t\tif (same(oldtree, newtree) || o->reset) {\n -+\t\t\tif (same(oldtree, newtree) || o->reset_either) {\n - \t\t\t\tif (!newtree)\n - \t\t\t\t\treturn deleted_entry(current, current, o);\n - \t\t\t\telse\n -@@ unpack-trees.c: int oneway_merge(const struct cache_entry * const *src,\n - \n - \tif (old && same(old, a)) {\n - \t\tint update = 0;\n --\t\tif (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old) &&\n -+\t\tif (o->reset_either && o->update && !ce_uptodate(old) && !ce_skip_worktree(old) &&\n - \t\t\t!(old->ce_flags & CE_FSMONITOR_VALID)) {\n - \t\t\tstruct stat st;\n - \t\t\tif (lstat(old->name, &st) ||\n \n ## unpack-trees.h ##\n @@ unpack-trees.h: void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n + */\n void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n \n ++enum unpack_trees_reset_type {\n ++\tUNPACK_RESET_NONE = 0, /* traditional \"false\" value; still valid */\n ++\tUNPACK_RESET_INVALID = 1, /* \"true\" no longer valid; use below values */\n ++\tUNPACK_RESET_PROTECT_UNTRACKED,\n ++\tUNPACK_RESET_OVERWRITE_UNTRACKED\n ++};\n ++\n struct unpack_trees_options {\n -\tunsigned int reset,\n -+\tunsigned int reset_nuke_untracked,\n -+\t\t reset_keep_untracked,\n -+\t\t reset_either, /* internal use only */\n - \t\t merge,\n +-\t\t merge,\n ++\tunsigned int merge,\n \t\t update,\n \t\t clone,\n + \t\t index_only,\n +@@ unpack-trees.h: struct unpack_trees_options {\n + \t\t exiting_early,\n + \t\t show_all_errors,\n + \t\t dry_run;\n ++\tenum unpack_trees_reset_type reset;\n + \tconst char *prefix;\n + \tint cache_bottom;\n + \tstruct dir_struct *dir;\n 3: a69117a1c9e = 3: 208f3b3ebe5 unpack-trees: avoid nuking untracked dir in way of unmerged file\n 4: 01bf850bb0f ! 4: 0a0997d081b unpack-trees: avoid nuking untracked dir in way of locally deleted file\n @@ unpack-trees.c: static int deleted_entry(const struct cache_entry *ce,\n \t\tif (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n \t\t\treturn -1;\n \t\treturn 0;\n -+\t} else {\n -+\t\tif (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n -+\t\t\treturn -1;\n ++\t} else if (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o)) {\n ++\t\treturn -1;\n \t}\n +\n \tif (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))\n 5: 60c5d6b4615 = 5: 4b78a526d2a Comment important codepaths regarding nuking untracked files/dirs\n 6: 6ea23d165cf = 6: 993451a8036 Documentation: call out commands that nuke untracked files/directories\n\n-- \ngitgitgadget\n"},{"id":"436927","messageId":"b77692b8f49e6551de201104d0e3527f3379b017.1632465429.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] Change unpack_trees' 'reset' flag into an enum","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:05Z","receivedAt":"2021-09-24T06:37:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nTraditionally, unpack_trees_options->reset was used to signal that it\nwas okay to delete any untracked files in the way. This was used by\n`git read-tree --reset`, but then started appearing in other places as\nwell. However, many of the other uses should not be deleting untracked\nfiles in the way. Change this value to an enum so that a value of 1\n(i.e. \"true\") can be split into two:\n UNPACK_RESET_PROTECT_UNTRACKED,\n UNPACK_RESET_OVERWRITE_UNTRACKED\nIn order to catch accidental misuses, define with the enum a special\nvalue of\n UNPACK_RESET_INVALID = 1\nwhich will trigger a BUG().\n\nModify existing callers so that\n read-tree --reset\n reset --hard\n checkout --force\ncontinue using the UNPACK_RESET_OVERWRITE_UNTRACKED logic, while other\ncallers, including\n am\n checkout without --force\n stash (though currently dead code; reset always had a value of 0)\n numerous callers from rebase/sequencer to reset_head()\nwill use the new UNPACK_RESET_PROTECT_UNTRACKED value.\n\nIn order to protect untracked files but still allow deleting of ignored\nfiles, we also have to setup unpack_trees_opt.dir. It may make sense to\nset up unpack_trees_opt.dir in more cases, but here I tried to only do\nso in cases where we switched from deleting all untracked files to\navoiding doing so (i.e. where we now use\nUNPACK_RESET_PROTECT_UNTRACKED).\n\nAlso, note that 'git checkout ' currently also allows\noverwriting untracked files. That case should also be fixed, but it\ndoes not use unpack_trees() and thus is outside the scope of the current\nchanges.\n\nSigned-off-by: Elijah Newren \n---\n builtin/am.c | 13 ++++++++++++-\n builtin/checkout.c | 18 ++++++++++++++++--\n builtin/read-tree.c | 3 +++\n builtin/reset.c | 20 ++++++++++++++++++--\n builtin/stash.c | 17 +++++++++++++++--\n reset.c | 13 +++++++++++--\n t/t2500-untracked-overwriting.sh | 6 +++---\n unpack-trees.c | 6 +++++-\n unpack-trees.h | 11 +++++++++--\n 9 files changed, 92 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex c79e0167e98..b17baa67ad8 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1918,8 +1918,14 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.dst_index = &the_index;\n \topts.update = 1;\n \topts.merge = 1;\n-\topts.reset = reset;\n+\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n \topts.fn = twoway_merge;\n+\tif (opts.reset) {\n+\t\t/* Allow ignored files in the way to get overwritten */\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n+\t}\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\n \n@@ -1928,6 +1934,11 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \t\treturn -1;\n \t}\n \n+\tif (opts.reset) {\n+\t\tdir_clear(opts.dir);\n+\t\tFREE_AND_NULL(opts.dir);\n+\t}\n+\n \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\n \ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b5d477919a7..52826e0d145 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -641,23 +641,37 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n {\n \tstruct unpack_trees_options opts;\n \tstruct tree_desc tree_desc;\n+\tint unpack_trees_ret;\n \n \tmemset(&opts, 0, sizeof(opts));\n \topts.head_idx = -1;\n \topts.update = worktree;\n \topts.skip_unmerged = !worktree;\n-\topts.reset = 1;\n+\topts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n+\t\t\t\tUNPACK_RESET_PROTECT_UNTRACKED;\n \topts.merge = 1;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n+\tif (o->overwrite_ignore) {\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n+\t}\n \tinit_checkout_metadata(&opts.meta, info->refname,\n \t\t\t info->commit ? &info->commit->object.oid : null_oid(),\n \t\t\t NULL);\n \tparse_tree(tree);\n \tinit_tree_desc(&tree_desc, tree->buffer, tree->size);\n-\tswitch (unpack_trees(1, &tree_desc, &opts)) {\n+\tunpack_trees_ret = unpack_trees(1, &tree_desc, &opts);\n+\n+\tif (o->overwrite_ignore) {\n+\t\tdir_clear(opts.dir);\n+\t\tFREE_AND_NULL(opts.dir);\n+\t}\n+\n+\tswitch (unpack_trees_ret) {\n \tcase -2:\n \t\t*writeout_error = 1;\n \t\t/*\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 485e7b04794..740fc0335af 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -174,6 +174,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n+\tif (opts.reset)\n+\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n+\n \t/*\n \t * NEEDSWORK\n \t *\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 43e855cb887..a12ee986e9f 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -10,6 +10,7 @@\n #define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"tag.h\"\n #include \"object.h\"\n@@ -70,9 +71,20 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\n-\t\t/* fallthrough */\n+\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n+\t\tbreak;\n+\tcase MIXED:\n+\t\topts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n+\t\t/* but opts.update=0, so working tree not updated */\n+\t\tbreak;\n \tdefault:\n-\t\topts.reset = 1;\n+\t\tBUG(\"invalid reset_type passed to reset_index\");\n+\t}\n+\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n+\t\t/* Setup opts.dir so we can overwrite ignored files */\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n \t}\n \n \tread_cache_unmerged();\n@@ -104,6 +116,10 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \tret = 0;\n \n out:\n+\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n+\t\tdir_clear(opts.dir);\n+\t\tFREE_AND_NULL(opts.dir);\n+\t}\n \tfor (i = 0; i < nr; i++)\n \t\tfree((void *)desc[i].buffer);\n \treturn ret;\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..563f590afbd 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -237,6 +237,7 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \tstruct tree_desc t[MAX_UNPACK_TREES];\n \tstruct tree *tree;\n \tstruct lock_file lock_file = LOCK_INIT;\n+\tint unpack_trees_ret;\n \n \tread_cache_preload(NULL);\n \tif (refresh_cache(REFRESH_QUIET))\n@@ -256,11 +257,23 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n \topts.merge = 1;\n-\topts.reset = reset;\n+\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n+\tif (opts.reset) {\n+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n+\t}\n \topts.update = update;\n \topts.fn = oneway_merge;\n \n-\tif (unpack_trees(nr_trees, t, &opts))\n+\tunpack_trees_ret = unpack_trees(nr_trees, t, &opts);\n+\n+\tif (opts.reset) {\n+\t\tdir_clear(opts.dir);\n+\t\tFREE_AND_NULL(opts.dir);\n+\t}\n+\n+\tif (unpack_trees_ret)\n \t\treturn -1;\n \n \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\ndiff --git a/reset.c b/reset.c\nindex 79310ae071b..1695f3828c5 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -1,5 +1,6 @@\n #include \"git-compat-util.h\"\n #include \"cache-tree.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"refs.h\"\n #include \"reset.h\"\n@@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.update = 1;\n \tunpack_tree_opts.merge = 1;\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n-\tif (!detach_head)\n-\t\tunpack_tree_opts.reset = 1;\n+\tif (!detach_head) {\n+\t\tunpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n+\t\tunpack_tree_opts.dir = xcalloc(1, sizeof(*unpack_tree_opts.dir));\n+\t\tunpack_tree_opts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(unpack_tree_opts.dir);\n+\t}\n \n \tif (repo_read_index_unmerged(r) < 0) {\n \t\tret = error(_(\"could not read index\"));\n@@ -131,6 +136,10 @@ reset_head_refs:\n \t\t\t oid_to_hex(oid), \"1\", NULL);\n \n leave_reset_head:\n+\tif (unpack_tree_opts.dir) {\n+\t\tdir_clear(unpack_tree_opts.dir);\n+\t\tFREE_AND_NULL(unpack_tree_opts.dir);\n+\t}\n \tstrbuf_release(&msg);\n \trollback_lock_file(&lock);\n \tclear_unpack_trees_porcelain(&unpack_tree_opts);\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 2412d121ea8..18604360df8 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -92,7 +92,7 @@ test_setup_checkout_m () {\n \t)\n }\n \n-test_expect_failure 'checkout -m does not nuke untracked file' '\n+test_expect_success 'checkout -m does not nuke untracked file' '\n \ttest_setup_checkout_m &&\n \t(\n \t\tcd checkout &&\n@@ -138,7 +138,7 @@ test_setup_sequencing () {\n \t)\n }\n \n-test_expect_failure 'git rebase --abort and untracked files' '\n+test_expect_success 'git rebase --abort and untracked files' '\n \ttest_setup_sequencing rebase_abort_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_abort_and_untracked &&\n@@ -155,7 +155,7 @@ test_expect_failure 'git rebase --abort and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git rebase fast forwarding and untracked files' '\n+test_expect_success 'git rebase fast forwarding and untracked files' '\n \ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_fast_forward_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5786645f315..fcbe63bbed9 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1693,6 +1693,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n \n+\tif (o->reset == UNPACK_RESET_INVALID)\n+\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n+\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n@@ -2218,7 +2221,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n \n-\tif (o->index_only || o->reset || !o->update)\n+\tif (o->index_only || !o->update ||\n+\t o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED)\n \t\treturn 0;\n \n \tlen = check_leading_path(ce->name, ce_namelen(ce), 0);\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 2d88b19dca7..1f386fb16cc 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -45,9 +45,15 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n */\n void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n \n+enum unpack_trees_reset_type {\n+\tUNPACK_RESET_NONE = 0, /* traditional \"false\" value; still valid */\n+\tUNPACK_RESET_INVALID = 1, /* \"true\" no longer valid; use below values */\n+\tUNPACK_RESET_PROTECT_UNTRACKED,\n+\tUNPACK_RESET_OVERWRITE_UNTRACKED\n+};\n+\n struct unpack_trees_options {\n-\tunsigned int reset,\n-\t\t merge,\n+\tunsigned int merge,\n \t\t update,\n \t\t clone,\n \t\t index_only,\n@@ -64,6 +70,7 @@ struct unpack_trees_options {\n \t\t exiting_early,\n \t\t show_all_errors,\n \t\t dry_run;\n+\tenum unpack_trees_reset_type reset;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"436928","messageId":"9460a49c7edfd4ebcace2ef66b265dc1bef67769.1632465429.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] t2500: add various tests for nuking untracked files","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:04Z","receivedAt":"2021-09-24T06:37:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nNoting that unpack_trees treats reset=1 & update=1 as license to nuke\nuntracked files, I looked for code paths that use this combination and\ntried to generate testcases which demonstrated unintentional loss of\nuntracked files and directories. I found several.\n\nI also include testcases for `git reset --{hard,merge,keep}`. A hard\nreset is perhaps the most direct test of unpack_tree's reset=1 behavior,\nbut we cannot make `git reset --hard` preserve untracked files without\nsome migration work.\n\nAlso, the two commands `checkout --force` (because of the --force) and\n`read-tree --reset` (because it's plumbing and we need to keep it\nbackward compatible) were left out as we expect those to continue\nremoving untracked files and directories.\n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n 1 file changed, 244 insertions(+)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nnew file mode 100755\nindex 00000000000..2412d121ea8\n--- /dev/null\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -0,0 +1,244 @@\n+#!/bin/sh\n+\n+test_description='Test handling of overwriting untracked files'\n+\n+. ./test-lib.sh\n+\n+test_setup_reset () {\n+\tgit init reset_$1 &&\n+\t(\n+\t\tcd reset_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\tgit branch stable &&\n+\t\tgit branch work &&\n+\n+\t\tgit checkout work &&\n+\t\ttest_commit foo &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_success 'reset --hard will nuke untracked files/dirs' '\n+\ttest_setup_reset hard &&\n+\t(\n+\t\tcd reset_hard &&\n+\t\tgit ls-tree -r stable &&\n+\t\tgit log --all --name-status --oneline &&\n+\t\tgit ls-tree -r work &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\techo foo >expect &&\n+\n+\t\tgit reset --hard work &&\n+\n+\t\t# check that untracked directory foo.t/ was nuked\n+\t\ttest_path_is_file foo.t &&\n+\t\ttest_cmp expect foo.t\n+\t)\n+'\n+\n+test_expect_success 'reset --merge will preserve untracked files/dirs' '\n+\ttest_setup_reset merge &&\n+\t(\n+\t\tcd reset_merge &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating .foo.t. would lose untracked files\" error\n+\t)\n+'\n+\n+test_expect_success 'reset --keep will preserve untracked files/dirs' '\n+\ttest_setup_reset keep &&\n+\t(\n+\t\tcd reset_keep &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n+\t)\n+'\n+\n+test_setup_checkout_m () {\n+\tgit init checkout &&\n+\t(\n+\t\tcd checkout &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines file has some >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit branch stable &&\n+\n+\t\tgit switch -c work &&\n+\t\techo stuff >notes.txt &&\n+\t\ttest_write_lines file has some words >filler &&\n+\t\tgit add notes.txt filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_failure 'checkout -m does not nuke untracked file' '\n+\ttest_setup_checkout_m &&\n+\t(\n+\t\tcd checkout &&\n+\n+\t\t# Tweak filler\n+\t\ttest_write_lines this file has some >filler &&\n+\t\t# Make an untracked file, save its contents in \"expect\"\n+\t\techo precious >notes.txt &&\n+\t\tcp notes.txt expect &&\n+\n+\t\ttest_must_fail git checkout -m work &&\n+\t\ttest_cmp expect notes.txt\n+\t)\n+'\n+\n+test_setup_sequencing () {\n+\tgit init sequencing_$1 &&\n+\t(\n+\t\tcd sequencing_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines this file has some words >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tmkdir -p foo/bar &&\n+\t\ttest_commit foo/bar/baz &&\n+\n+\t\tgit branch simple &&\n+\t\tgit branch fooey &&\n+\n+\t\tgit checkout fooey &&\n+\t\tgit rm foo/bar/baz.t &&\n+\t\techo stuff >>filler &&\n+\t\tgit add -u &&\n+\t\tgit commit -m \"changes\" &&\n+\n+\t\tgit checkout simple &&\n+\t\techo items >>filler &&\n+\t\techo newstuff >>newfile &&\n+\t\tgit add filler newfile &&\n+\t\tgit commit -m another\n+\t)\n+}\n+\n+test_expect_failure 'git rebase --abort and untracked files' '\n+\ttest_setup_sequencing rebase_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_abort_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\ttest_must_fail git rebase simple &&\n+\n+\t\tcat init.t &&\n+\t\tgit rm init.t &&\n+\t\techo precious >init.t &&\n+\t\tcp init.t expect &&\n+\t\tgit status --porcelain &&\n+\t\ttest_must_fail git rebase --abort &&\n+\t\ttest_cmp expect init.t\n+\t)\n+'\n+\n+test_expect_failure 'git rebase fast forwarding and untracked files' '\n+\ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_fast_forward_and_untracked &&\n+\t\tgit checkout init &&\n+\t\techo precious >filler &&\n+\t\tcp filler expect &&\n+\t\ttest_must_fail git rebase init simple &&\n+\t\ttest_cmp expect filler\n+\t)\n+'\n+\n+test_expect_failure 'git rebase --autostash and untracked files' '\n+\ttest_setup_sequencing rebase_autostash_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_autostash_and_untracked &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit rebase --autostash init &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git stash and untracked files' '\n+\ttest_setup_sequencing stash_and_untracked_files &&\n+\t(\n+\t\tcd sequencing_stash_and_untracked_files &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit status --porcelain &&\n+\t\tgit stash push &&\n+\t\tgit status --porcelain &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+\ttest_setup_sequencing am_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_abort_and_untracked &&\n+\t\tgit format-patch -1 --stdout fooey >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete the conflicted file; we will stage and commit it later\n+\t\trm filler &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir filler &&\n+\t\techo foo >filler/file1 &&\n+\t\techo bar >filler/file2 &&\n+\n+\t\ttest_must_fail git am --abort 2>errors &&\n+\t\ttest_path_is_dir filler &&\n+\t\tgrep \"Updating .filler. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+\ttest_setup_sequencing am_skip_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_skip_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\tgit format-patch -1 --stdout simple >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete newfile\n+\t\trm newfile &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir newfile &&\n+\t\techo foo >newfile/file1 &&\n+\t\techo bar >newfile/file2 &&\n+\n+\t\t# Change our mind about resolutions, just skip this patch\n+\t\ttest_must_fail git am --skip 2>errors &&\n+\t\ttest_path_is_dir newfile &&\n+\t\tgrep \"Updating .newfile. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"436929","messageId":"208f3b3ebe5898150948155204724c68b46e2fb0.1632465429.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] unpack-trees: avoid nuking untracked dir in way of unmerged file","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:06Z","receivedAt":"2021-09-24T06:37:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 35 ++++++++++++++++++++++++++++----\n 2 files changed, 32 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 18604360df8..5ec66058cfc 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -197,7 +197,7 @@ test_expect_failure 'git stash and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \ttest_setup_sequencing am_abort_and_untracked &&\n \t(\n \t\tcd sequencing_am_abort_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex fcbe63bbed9..f7d0088a4fd 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2159,9 +2159,15 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n+enum absent_checking_type {\n+\tCOMPLETELY_ABSENT,\n+\tABSENT_ANY_DIRECTORY\n+};\n+\n static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\t\t const struct cache_entry *ce, struct stat *st,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tconst struct cache_entry *result;\n@@ -2196,6 +2202,10 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \t}\n \n+\t/* If we only care about directories, then we can remove */\n+\tif (absent_type == ABSENT_ANY_DIRECTORY)\n+\t\treturn 0;\n+\n \t/*\n \t * The previous round may already have decided to\n \t * delete this path, which is in a subdirectory that\n@@ -2216,6 +2226,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n */\n static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tint len;\n@@ -2242,7 +2253,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t\t\t\t\t\tNULL, o);\n \t\t\telse\n \t\t\t\tret = check_ok_to_remove(path, len, DT_UNKNOWN, NULL,\n-\t\t\t\t\t\t\t &st, error_type, o);\n+\t\t\t\t\t\t\t &st, error_type,\n+\t\t\t\t\t\t\t absent_type, o);\n \t\t}\n \t\tfree(path);\n \t\treturn ret;\n@@ -2257,7 +2269,7 @@ static int verify_absent_1(const struct cache_entry *ce,\n \n \t\treturn check_ok_to_remove(ce->name, ce_namelen(ce),\n \t\t\t\t\t ce_to_dtype(ce), ce, &st,\n-\t\t\t\t\t error_type, o);\n+\t\t\t\t\t error_type, absent_type, o);\n \t}\n }\n \n@@ -2267,14 +2279,23 @@ static int verify_absent(const struct cache_entry *ce,\n {\n \tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n+}\n+\n+static int verify_absent_if_directory(const struct cache_entry *ce,\n+\t\t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t\t struct unpack_trees_options *o)\n+{\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\t\treturn 0;\n+\treturn verify_absent_1(ce, error_type, ABSENT_ANY_DIRECTORY, o);\n }\n \n static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types error_type,\n \t\t\t\tstruct unpack_trees_options *o)\n {\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n }\n \n static int merged_entry(const struct cache_entry *ce,\n@@ -2348,6 +2369,12 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t * Previously unmerged entry left as an existence\n \t\t * marker by read_index_unmerged();\n \t\t */\n+\t\tif (verify_absent_if_directory(merge,\n+\t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n+\t\t\tdiscard_cache_entry(merge);\n+\t\t\treturn -1;\n+\t\t}\n+\n \t\tinvalidate_ce_path(old, o);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"436930","messageId":"0a0997d081b5e85f1a3d94b204c073f81a469440.1632465429.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] unpack-trees: avoid nuking untracked dir in way of locally deleted file","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:07Z","receivedAt":"2021-09-24T06:37:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 3 +++\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 5ec66058cfc..5c0bf4d21fc 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -218,7 +218,7 @@ test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \t)\n '\n \n-test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+test_expect_success 'git am --skip and untracked dir vs deleted file' '\n \ttest_setup_sequencing am_skip_and_untracked &&\n \t(\n \t\tcd sequencing_am_skip_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f7d0088a4fd..b1e7ee9dfc0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2392,7 +2392,10 @@ static int deleted_entry(const struct cache_entry *ce,\n \t\tif (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n \t\t\treturn -1;\n \t\treturn 0;\n+\t} else if (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o)) {\n+\t\treturn -1;\n \t}\n+\n \tif (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))\n \t\treturn -1;\n \tadd_entry(o, ce, CE_REMOVE, 0);\n-- \ngitgitgadget\n\n"},{"id":"436931","messageId":"4b78a526d2acfbcd85d6e4cb94894725ea0350aa.1632465429.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:08Z","receivedAt":"2021-09-24T06:37:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last few commits we focused on code in unpack-trees.c that\nmistakenly removed untracked files or directories. There may be more of\nthose, but in this commit we change our focus: callers of toplevel\ncommands that are expected to remove untracked files or directories.\n\nAs noted previously, we have toplevel commands that are expected to\ndelete untracked files such as 'read-tree --reset', 'reset --hard', and\n'checkout --force'. However, that does not mean that other highlevel\ncommands that happen to call these other commands thought about or\nconveyed to users the possibility that untracked files could be removed.\nAudit the code for such callsites, and add comments near existing\ncallsites to mention whether these are safe or not.\n\nMy auditing is somewhat incomplete, though; it skipped several cases:\n * git-rebase--preserve-merges.sh: is in the process of being\n deprecated/removed, so I won't leave a note that there are\n likely more bugs in that script.\n * contrib/git-new-workdir: why is the -f flag being used in a new\n empty directory?? It shouldn't hurt, but it seems useless.\n * git-p4.py: Don't see why -f is needed for a new dir (maybe it's\n not and is just superfluous), but I'm not at all familiar with\n the p4 stuff\n * git-archimport.perl: Don't care; arch is long since dead\n * git-cvs*.perl: Don't care; cvs is long since dead\n\nSigned-off-by: Elijah Newren \n---\n builtin/stash.c | 1 +\n builtin/submodule--helper.c | 4 ++++\n builtin/worktree.c | 5 +++++\n contrib/rerere-train.sh | 2 +-\n submodule.c | 1 +\n 5 files changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 563f590afbd..0d071183eba 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1532,6 +1532,7 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\t} else {\n \t\t\tstruct child_process cp = CHILD_PROCESS_INIT;\n \t\t\tcp.git_cmd = 1;\n+\t\t\t/* BUG: this nukes untracked files in the way */\n \t\t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"-q\",\n \t\t\t\t \"--no-recurse-submodules\", NULL);\n \t\t\tif (run_command(&cp)) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex ef2776a9e45..a49242d15ae 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -2864,6 +2864,10 @@ static int add_submodule(const struct add_data *add_data)\n \t\tprepare_submodule_repo_env(&cp.env_array);\n \t\tcp.git_cmd = 1;\n \t\tcp.dir = add_data->sm_path;\n+\t\t/*\n+\t\t * NOTE: we only get here if add_data->force is true, so\n+\t\t * passing --force to checkout is reasonable.\n+\t\t */\n \t\tstrvec_pushl(&cp.args, \"checkout\", \"-f\", \"-q\", NULL);\n \n \t\tif (add_data->branch) {\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 0d0a80da61f..383947ff54f 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -356,6 +356,11 @@ static int add_worktree(const char *path, const char *refname,\n \tif (opts->checkout) {\n \t\tcp.argv = NULL;\n \t\tstrvec_clear(&cp.args);\n+\t\t/*\n+\t\t * NOTE: reset --hard is okay here, because 'worktree add'\n+\t\t * refuses to work in an extant non-empty directory, so there\n+\t\t * is no risk of deleting untracked files.\n+\t\t */\n \t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"--no-recurse-submodules\", NULL);\n \t\tif (opts->quiet)\n \t\t\tstrvec_push(&cp.args, \"--quiet\");\ndiff --git a/contrib/rerere-train.sh b/contrib/rerere-train.sh\nindex eeee45dd341..75125d6ae00 100755\n--- a/contrib/rerere-train.sh\n+++ b/contrib/rerere-train.sh\n@@ -91,7 +91,7 @@ do\n \t\tgit checkout -q $commit -- .\n \t\tgit rerere\n \tfi\n-\tgit reset -q --hard\n+\tgit reset -q --hard # Might nuke untracked files...\n done\n \n if test -z \"$branch\"\ndiff --git a/submodule.c b/submodule.c\nindex 8e611fe1dbf..a9b71d585cf 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1866,6 +1866,7 @@ static void submodule_reset_index(const char *path)\n \n \tstrvec_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t get_super_prefix_or_empty(), path);\n+\t/* TODO: determine if this might overwright untracked files */\n \tstrvec_pushl(&cp.args, \"read-tree\", \"-u\", \"--reset\", NULL);\n \n \tstrvec_push(&cp.args, empty_tree_oid_hex());\n-- \ngitgitgadget\n\n"},{"id":"436932","messageId":"993451a8036165930fb6ada72156695c24940c6c.1632465429.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] Documentation: call out commands that nuke untracked files/directories","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T06:37:09Z","receivedAt":"2021-09-24T06:37:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSome commands have traditionally also removed untracked files (or\ndirectories) that were in the way of a tracked file we needed. Document\nthese cases.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-checkout.txt | 5 +++--\n Documentation/git-read-tree.txt | 5 +++--\n Documentation/git-reset.txt | 3 ++-\n 3 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\nindex b1a6fe44997..d473c9bf387 100644\n--- a/Documentation/git-checkout.txt\n+++ b/Documentation/git-checkout.txt\n@@ -118,8 +118,9 @@ OPTIONS\n -f::\n --force::\n \tWhen switching branches, proceed even if the index or the\n-\tworking tree differs from `HEAD`. This is used to throw away\n-\tlocal changes.\n+\tworking tree differs from `HEAD`, and even if there are untracked\n+\tfiles in the way. This is used to throw away local changes and\n+\tany untracked files or directories that are in the way.\n +\n When checking out paths from the index, do not fail upon unmerged\n entries; instead, unmerged entries are ignored.\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 5fa8bab64c2..4731ec3283f 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -39,8 +39,9 @@ OPTIONS\n \n --reset::\n \tSame as -m, except that unmerged entries are discarded instead\n-\tof failing. When used with `-u`, updates leading to loss of\n-\tworking tree changes will not abort the operation.\n+\tof failing. When used with `-u`, updates leading to loss of\n+\tworking tree changes or untracked files or directories will not\n+\tabort the operation.\n \n -u::\n \tAfter a successful merge, update the files in the work\ndiff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\nindex 252e2d4e47d..6f7685f53d5 100644\n--- a/Documentation/git-reset.txt\n+++ b/Documentation/git-reset.txt\n@@ -69,7 +69,8 @@ linkgit:git-add[1]).\n \n --hard::\n \tResets the index and working tree. Any changes to tracked files in the\n-\tworking tree since `` are discarded.\n+\tworking tree since `` are discarded. Any untracked files or\n+\tdirectories in the way of writing any tracked files are simply deleted.\n \n --merge::\n \tResets the index and updates the files in the working tree that are\n-- \ngitgitgadget\n"},{"id":"436942","messageId":"CAE5ih79bzExNrp__GAUaVZWvMucYDrvSHSJe_+dNyeVR3VXynw@mail.gmail.com","threadId":"56534","inReplyTo":"60c5d6b4615a6ac4179ec6c10e17cca480bc147a.1632006924.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2021-09-24T11:47:43Z","receivedAt":"2021-09-24T11:48:08Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On Sun, 19 Sept 2021 at 00:15, Elijah Newren via GitGitGadget\n wrote:\n>\n> From: Elijah Newren \n>\n> In the last few commits we focused on code in unpack-trees.c that\n> mistakenly removed untracked files or directories. There may be more of\n> those, but in this commit we change our focus: callers of toplevel\n> commands that are expected to remove untracked files or directories.\n>\n> As noted previously, we have toplevel commands that are expected to\n> delete untracked files such as 'read-tree --reset', 'reset --hard', and\n> 'checkout --force'. However, that does not mean that other highlevel\n> commands that happen to call these other commands thought about or\n> conveyed to users the possibility that untracked files could be removed.\n> Audit the code for such callsites, and add comments near existing\n> callsites to mention whether these are safe or not.\n>\n> My auditing is somewhat incomplete, though; it skipped several cases:\n> * git-rebase--preserve-merges.sh: is in the process of being\n> deprecated/removed, so I won't leave a note that there are\n> likely more bugs in that script.\n> * contrib/git-new-workdir: why is the -f flag being used in a new\n> empty directory?? It shouldn't hurt, but it seems useless.\n> * git-p4.py: Don't see why -f is needed for a new dir (maybe it's\n> not and is just superfluous), but I'm not at all familiar with\n> the p4 stuff\n\nAssuming you're talking about this code in git-p4.py:\n\n print(\"Synchronizing p4 checkout...\")\n if new_client_dir:\n # old one was destroyed, and maybe nobody told p4\n p4_sync(\"...\", \"-f\")\n else:\n p4_sync(\"...\")\n\nThis is doing a Perforce sync in the P4 repo, not the git repo.\n\nIn the usual/happy case, this directory already exists, the Perforce\nserver knows about its state, and a normal \"p4 sync ...\" will bring it\nup to date.\n\nBut, if someone manually deleted the directory then \"p4 sync ...\" will\nonly update modified files, and all sorts of things will then go wrong\n(e.g. the files we updated in the git view won't be present, and\ngit-p4 will fall flat on its face).\n\nSo in this case, do a forced sync, which syncs everything ignoring the\nP4 server's idea of what files are/not present.\n\nLuke\n"},{"id":"436949","messageId":"CABPp-BEYH5HzQrRvMYw3vKMPQQ9b7XXfz5so5Wag7cbyyt=q=w@mail.gmail.com","threadId":"56534","inReplyTo":"CAE5ih79bzExNrp__GAUaVZWvMucYDrvSHSJe_+dNyeVR3VXynw@mail.gmail.com","subject":"Re: [PATCH 5/6] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-24T13:41:37Z","receivedAt":"2021-09-24T13:41:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Sep 24, 2021 at 4:47 AM Luke Diamand wrote:\n>\n> On Sun, 19 Sept 2021 at 00:15, Elijah Newren via GitGitGadget\n> wrote:\n> >\n> > From: Elijah Newren \n> >\n> > In the last few commits we focused on code in unpack-trees.c that\n> > mistakenly removed untracked files or directories. There may be more of\n> > those, but in this commit we change our focus: callers of toplevel\n> > commands that are expected to remove untracked files or directories.\n> >\n> > As noted previously, we have toplevel commands that are expected to\n> > delete untracked files such as 'read-tree --reset', 'reset --hard', and\n> > 'checkout --force'. However, that does not mean that other highlevel\n> > commands that happen to call these other commands thought about or\n> > conveyed to users the possibility that untracked files could be removed.\n> > Audit the code for such callsites, and add comments near existing\n> > callsites to mention whether these are safe or not.\n> >\n> > My auditing is somewhat incomplete, though; it skipped several cases:\n> > * git-rebase--preserve-merges.sh: is in the process of being\n> > deprecated/removed, so I won't leave a note that there are\n> > likely more bugs in that script.\n> > * contrib/git-new-workdir: why is the -f flag being used in a new\n> > empty directory?? It shouldn't hurt, but it seems useless.\n> > * git-p4.py: Don't see why -f is needed for a new dir (maybe it's\n> > not and is just superfluous), but I'm not at all familiar with\n> > the p4 stuff\n>\n> Assuming you're talking about this code in git-p4.py:\n>\n> print(\"Synchronizing p4 checkout...\")\n> if new_client_dir:\n> # old one was destroyed, and maybe nobody told p4\n> p4_sync(\"...\", \"-f\")\n> else:\n> p4_sync(\"...\")\n\nNo, I was talking about this code:\n\n system([ \"git\", \"checkout\", \"-f\" ])\n"},{"id":"436984","messageId":"xmqq7df5x4f1.fsf@gitster.g","threadId":"56534","inReplyTo":"b77692b8f49e6551de201104d0e3527f3379b017.1632465429.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/6] Change unpack_trees' 'reset' flag into an enum","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-24T17:35:30Z","receivedAt":"2021-09-24T17:35:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> Also, note that 'git checkout ' currently also allows\n> overwriting untracked files. That case should also be fixed, ...\n\nI wasted a few minutes wondering about the example. Please make it\nclear that you are checking out of a tree-ish that is different from\nHEAD, as there will by definition no \"overwriting untracked\" if you\nare checking out of the index.\n\nE.g. \"git checkout -- \".\n\nWith this command line:\n\n $ git checkout HEAD~24 -- path\n\nwhere path used to be there as late as 24 revisions ago, but since\nthen we removed, and the user wants to materialize the file out of\nthe old version, path, be it tracked, untracked, or even a\ndirectory, should be made identical to the copy from the given\nversion, no? Where does the \"should also be fixed\" come from?\n\n> diff --git a/builtin/am.c b/builtin/am.c\n> index c79e0167e98..b17baa67ad8 100644\n> --- a/builtin/am.c\n> +++ b/builtin/am.c\n> @@ -1918,8 +1918,14 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n> \topts.dst_index = &the_index;\n> \topts.update = 1;\n> \topts.merge = 1;\n> -\topts.reset = reset;\n> +\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n> \topts.fn = twoway_merge;\n> +\tif (opts.reset) {\n> +\t\t/* Allow ignored files in the way to get overwritten */\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(opts.dir);\n\nDo these three lines make a recurring pattern when opts.reset is set?\nI am wondering if this can be done more centrally by the unpack-trees\nmachinery (i.e. \"gee this one has o->reset set to X, so let's set up\nthe o->dir before doing anything\").\n\n> diff --git a/builtin/checkout.c b/builtin/checkout.c\n> index b5d477919a7..52826e0d145 100644\n> --- a/builtin/checkout.c\n> +++ b/builtin/checkout.c\n> @@ -641,23 +641,37 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n> {\n> \tstruct unpack_trees_options opts;\n> \tstruct tree_desc tree_desc;\n> +\tint unpack_trees_ret;\n> \n> \tmemset(&opts, 0, sizeof(opts));\n> \topts.head_idx = -1;\n> \topts.update = worktree;\n> \topts.skip_unmerged = !worktree;\n> -\topts.reset = 1;\n> +\topts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n> +\t\t\t\tUNPACK_RESET_PROTECT_UNTRACKED;\n> \topts.merge = 1;\n> \topts.fn = oneway_merge;\n> \topts.verbose_update = o->show_progress;\n> \topts.src_index = &the_index;\n> \topts.dst_index = &the_index;\n> +\tif (o->overwrite_ignore) {\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(opts.dir);\n> +\t}\n\nIf our longer term goal is to decide classification of files not in\nthe index (currently, \"ignored\" and \"untracked\", but we may want to\nadd a new \"precious\" class) and (across various commands that build\non the unpack-trees infrastructure) to protect the \"untracked\" and\n\"precious\" ones, with --[no-]overwrite-{ignore,untracked} options as\nescape hatches, uniformly, perhaps the --[no-]-overwrite-ignore\noption may be stolen from here and shifted to unpack_tree_options to\nhelp us going in that direction? This is just an observation for\nlonger term, not a suggestion to include the first step for such a\nmove in this series.\n\n> \tinit_checkout_metadata(&opts.meta, info->refname,\n> \t\t\t info->commit ? &info->commit->object.oid : null_oid(),\n> \t\t\t NULL);\n> \tparse_tree(tree);\n> \tinit_tree_desc(&tree_desc, tree->buffer, tree->size);\n> -\tswitch (unpack_trees(1, &tree_desc, &opts)) {\n> +\tunpack_trees_ret = unpack_trees(1, &tree_desc, &opts);\n> +\n> +\tif (o->overwrite_ignore) {\n> +\t\tdir_clear(opts.dir);\n> +\t\tFREE_AND_NULL(opts.dir);\n\nThis dir_clear() is also a recurring theme. See below.\n\n> +\t}\n> +\n> +\tswitch (unpack_trees_ret) {\n> \tcase -2:\n> \t\t*writeout_error = 1;\n> \t\t/*\n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index 485e7b04794..740fc0335af 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -174,6 +174,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> \tif (1 < opts.merge + opts.reset + prefix_set)\n> \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n> \n> +\tif (opts.reset)\n> +\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n> +\n\nWe do not do anything about opts.dir here by default, which means we\nby default do not overwrite ignored, but that's OK, because this old\ncommand explicitly takes --exclude-per-directory to tell it what the\ncommand should consider \"ignored\" and with the option given we do\nprepare opts.dir just fine.\n\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 43e855cb887..a12ee986e9f 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -10,6 +10,7 @@\n> #define USE_THE_INDEX_COMPATIBILITY_MACROS\n> #include \"builtin.h\"\n> #include \"config.h\"\n> +#include \"dir.h\"\n> #include \"lockfile.h\"\n> #include \"tag.h\"\n> #include \"object.h\"\n> @@ -70,9 +71,20 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> \t\tbreak;\n> \tcase HARD:\n> \t\topts.update = 1;\n> -\t\t/* fallthrough */\n> +\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n> +\t\tbreak;\n> +\tcase MIXED:\n> +\t\topts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n> +\t\t/* but opts.update=0, so working tree not updated */\n> +\t\tbreak;\n> \tdefault:\n> -\t\topts.reset = 1;\n> +\t\tBUG(\"invalid reset_type passed to reset_index\");\n> +\t}\n> +\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n\nUnlike the one in \"am\", this cares .reset being a particular value,\nnot just being non-zero. Puzzling.\n\nIt is a bit counter-intuitive in that we do not allow overwrite\nignored (which is currently a synonym for \"expendable\") when .reset\nis set to allow us to ovewrite untracked.\n\n> +\t\t/* Setup opts.dir so we can overwrite ignored files */\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(opts.dir);\n\n> @@ -104,6 +116,10 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> \tret = 0;\n> \n> out:\n> +\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n> +\t\tdir_clear(opts.dir);\n> +\t\tFREE_AND_NULL(opts.dir);\n\nThis dir_clear() is also a recurring theme. See below.\n\n> diff --git a/builtin/stash.c b/builtin/stash.c\n> index 8f42360ca91..563f590afbd 100644\n> --- a/builtin/stash.c\n> +++ b/builtin/stash.c\n> @@ -237,6 +237,7 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n> \tstruct tree_desc t[MAX_UNPACK_TREES];\n> \tstruct tree *tree;\n> \tstruct lock_file lock_file = LOCK_INIT;\n> +\tint unpack_trees_ret;\n> \n> \tread_cache_preload(NULL);\n> \tif (refresh_cache(REFRESH_QUIET))\n> @@ -256,11 +257,23 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n> \topts.src_index = &the_index;\n> \topts.dst_index = &the_index;\n> \topts.merge = 1;\n> -\topts.reset = reset;\n> +\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n> +\tif (opts.reset) {\n> +\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n> +\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(opts.dir);\n> +\t}\n> \topts.update = update;\n> \topts.fn = oneway_merge;\n> \n> -\tif (unpack_trees(nr_trees, t, &opts))\n> +\tunpack_trees_ret = unpack_trees(nr_trees, t, &opts);\n> +\n> +\tif (opts.reset) {\n> +\t\tdir_clear(opts.dir);\n> +\t\tFREE_AND_NULL(opts.dir);\n\nThis dir_clear() is also a recurring theme. Why aren't their guards\nuniformly \"if (opts.dir)\"? The logic to decide if we set up opts.dir\nor not may be far from here and may be different from code path to\ncode path, but the need to clear opts.dir should not have to care\nwhy opts.dir was populated, no?\n\n\n> +\t}\n> +\n> +\tif (unpack_trees_ret)\n> \t\treturn -1;\n> \n> \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n> diff --git a/reset.c b/reset.c\n> index 79310ae071b..1695f3828c5 100644\n> --- a/reset.c\n> +++ b/reset.c\n> @@ -1,5 +1,6 @@\n> #include \"git-compat-util.h\"\n> #include \"cache-tree.h\"\n> +#include \"dir.h\"\n> #include \"lockfile.h\"\n> #include \"refs.h\"\n> #include \"reset.h\"\n> @@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n> \tunpack_tree_opts.update = 1;\n> \tunpack_tree_opts.merge = 1;\n> \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n> -\tif (!detach_head)\n> -\t\tunpack_tree_opts.reset = 1;\n> +\tif (!detach_head) {\n> +\t\tunpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n> +\t\tunpack_tree_opts.dir = xcalloc(1, sizeof(*unpack_tree_opts.dir));\n> +\t\tunpack_tree_opts.dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(unpack_tree_opts.dir);\n> +\t}\n> \n> \tif (repo_read_index_unmerged(r) < 0) {\n> \t\tret = error(_(\"could not read index\"));\n> @@ -131,6 +136,10 @@ reset_head_refs:\n> \t\t\t oid_to_hex(oid), \"1\", NULL);\n> \n> leave_reset_head:\n> +\tif (unpack_tree_opts.dir) {\n> +\t\tdir_clear(unpack_tree_opts.dir);\n> +\t\tFREE_AND_NULL(unpack_tree_opts.dir);\n\nYes, I think this is the right way to decide if we call dir_clear(),\nand all other hunks in this patch should do the same.\n\n> +\t}\n> \tstrbuf_release(&msg);\n> \trollback_lock_file(&lock);\n> \tclear_unpack_trees_porcelain(&unpack_tree_opts);\n> diff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\n> index 2412d121ea8..18604360df8 100755\n> --- a/t/t2500-untracked-overwriting.sh\n> +++ b/t/t2500-untracked-overwriting.sh\n> @@ -92,7 +92,7 @@ test_setup_checkout_m () {\n> \t)\n> }\n> \n> -test_expect_failure 'checkout -m does not nuke untracked file' '\n> +test_expect_success 'checkout -m does not nuke untracked file' '\n> \ttest_setup_checkout_m &&\n> \t(\n> \t\tcd checkout &&\n> @@ -138,7 +138,7 @@ test_setup_sequencing () {\n> \t)\n> }\n> \n> -test_expect_failure 'git rebase --abort and untracked files' '\n> +test_expect_success 'git rebase --abort and untracked files' '\n> \ttest_setup_sequencing rebase_abort_and_untracked &&\n> \t(\n> \t\tcd sequencing_rebase_abort_and_untracked &&\n> @@ -155,7 +155,7 @@ test_expect_failure 'git rebase --abort and untracked files' '\n> \t)\n> '\n> \n> -test_expect_failure 'git rebase fast forwarding and untracked files' '\n> +test_expect_success 'git rebase fast forwarding and untracked files' '\n> \ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n> \t(\n> \t\tcd sequencing_rebase_fast_forward_and_untracked &&\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 5786645f315..fcbe63bbed9 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1693,6 +1693,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \tstruct pattern_list pl;\n> \tint free_pattern_list = 0;\n> \n> +\tif (o->reset == UNPACK_RESET_INVALID)\n> +\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n> +\n> \tif (len > MAX_UNPACK_TREES)\n> \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n> \n> @@ -2218,7 +2221,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n> \tint len;\n> \tstruct stat st;\n> \n> -\tif (o->index_only || o->reset || !o->update)\n> +\tif (o->index_only || !o->update ||\n> +\t o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED)\n> \t\treturn 0;\n> \n> \tlen = check_leading_path(ce->name, ce_namelen(ce), 0);\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index 2d88b19dca7..1f386fb16cc 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -45,9 +45,15 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n> */\n> void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n> \n> +enum unpack_trees_reset_type {\n> +\tUNPACK_RESET_NONE = 0, /* traditional \"false\" value; still valid */\n> +\tUNPACK_RESET_INVALID = 1, /* \"true\" no longer valid; use below values */\n> +\tUNPACK_RESET_PROTECT_UNTRACKED,\n> +\tUNPACK_RESET_OVERWRITE_UNTRACKED\n> +};\n> +\n> struct unpack_trees_options {\n> -\tunsigned int reset,\n> -\t\t merge,\n> +\tunsigned int merge,\n> \t\t update,\n> \t\t clone,\n> \t\t index_only,\n> @@ -64,6 +70,7 @@ struct unpack_trees_options {\n> \t\t exiting_early,\n> \t\t show_all_errors,\n> \t\t dry_run;\n> +\tenum unpack_trees_reset_type reset;\n> \tconst char *prefix;\n> \tint cache_bottom;\n> \tstruct dir_struct *dir;\n"},{"id":"436985","messageId":"CAPig+cRCtuPtXesh8494W=HOQJishCY+vSX1Fr_mF86qvnEP9g@mail.gmail.com","threadId":"56534","inReplyTo":"4b78a526d2acfbcd85d6e4cb94894725ea0350aa.1632465429.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/6] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-09-24T17:50:07Z","receivedAt":"2021-09-24T17:50:21Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Sep 24, 2021 at 2:37 AM Elijah Newren via GitGitGadget\n wrote:\n> In the last few commits we focused on code in unpack-trees.c that\n> mistakenly removed untracked files or directories. There may be more of\n> those, but in this commit we change our focus: callers of toplevel\n> commands that are expected to remove untracked files or directories.\n>\n> As noted previously, we have toplevel commands that are expected to\n> delete untracked files such as 'read-tree --reset', 'reset --hard', and\n> 'checkout --force'. However, that does not mean that other highlevel\n> commands that happen to call these other commands thought about or\n> conveyed to users the possibility that untracked files could be removed.\n> Audit the code for such callsites, and add comments near existing\n> callsites to mention whether these are safe or not.\n> [...]\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/builtin/worktree.c b/builtin/worktree.c\n> @@ -356,6 +356,11 @@ static int add_worktree(const char *path, const char *refname,\n> + /*\n> + * NOTE: reset --hard is okay here, because 'worktree add'\n> + * refuses to work in an extant non-empty directory, so there\n> + * is no risk of deleting untracked files.\n> + */\n> strvec_pushl(&cp.args, \"reset\", \"--hard\", \"--no-recurse-submodules\", NULL);\n\nI understand that this comment helps you or some other person auditing\nsimilar cases in the future, however, as a standalone comment for a\nreader who isn't aware of the intention, it seems more confusing than\nilluminating. It also detracts from the important purpose of `--hard`\nhere, which is that it is necessary in order to get `git reset` to\nactually \"checkout\" the files into the empty directory, so use of\n`--hard` is not an accident or carelessness.\n\nThese days, we'd probably just use:\n\n git restore --no-recurse-submodules .\n\ninstead (including the final `.`) to achieve the same, and that\nwouldn't need any sort of cuationary comment like the one being added\nby this patch. So, perhaps that's a better way to go, or maybe it's\noutside the scope of this series...\n"},{"id":"437076","messageId":"CABPp-BFMjQK+14A90agS5wUG8Tpug68dUSPcz0RRZNkiaMzCug@mail.gmail.com","threadId":"56534","inReplyTo":"CAPig+cRCtuPtXesh8494W=HOQJishCY+vSX1Fr_mF86qvnEP9g@mail.gmail.com","subject":"Re: [PATCH v2 5/6] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-26T06:35:16Z","receivedAt":"2021-09-26T06:35:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Sep 24, 2021 at 10:50 AM Eric Sunshine wrote:\n>\n> On Fri, Sep 24, 2021 at 2:37 AM Elijah Newren via GitGitGadget\n> wrote:\n> > In the last few commits we focused on code in unpack-trees.c that\n> > mistakenly removed untracked files or directories. There may be more of\n> > those, but in this commit we change our focus: callers of toplevel\n> > commands that are expected to remove untracked files or directories.\n> >\n> > As noted previously, we have toplevel commands that are expected to\n> > delete untracked files such as 'read-tree --reset', 'reset --hard', and\n> > 'checkout --force'. However, that does not mean that other highlevel\n> > commands that happen to call these other commands thought about or\n> > conveyed to users the possibility that untracked files could be removed.\n> > Audit the code for such callsites, and add comments near existing\n> > callsites to mention whether these are safe or not.\n> > [...]\n> > Signed-off-by: Elijah Newren \n> > ---\n> > diff --git a/builtin/worktree.c b/builtin/worktree.c\n> > @@ -356,6 +356,11 @@ static int add_worktree(const char *path, const char *refname,\n> > + /*\n> > + * NOTE: reset --hard is okay here, because 'worktree add'\n> > + * refuses to work in an extant non-empty directory, so there\n> > + * is no risk of deleting untracked files.\n> > + */\n> > strvec_pushl(&cp.args, \"reset\", \"--hard\", \"--no-recurse-submodules\", NULL);\n>\n> I understand that this comment helps you or some other person auditing\n> similar cases in the future, however, as a standalone comment for a\n> reader who isn't aware of the intention, it seems more confusing than\n> illuminating. It also detracts from the important purpose of `--hard`\n> here, which is that it is necessary in order to get `git reset` to\n> actually \"checkout\" the files into the empty directory, so use of\n> `--hard` is not an accident or carelessness.\n\nFair enough; I'll strike it.\n\n> These days, we'd probably just use:\n>\n> git restore --no-recurse-submodules .\n>\n> instead (including the final `.`) to achieve the same, and that\n> wouldn't need any sort of cuationary comment like the one being added\n> by this patch. So, perhaps that's a better way to go, or maybe it's\n> outside the scope of this series...\n\nYeah, that'd make sense. Though it'd make even more sense to get rid\nof the subprocess forking. Definitely something for a different\nseries, though.\n"},{"id":"437077","messageId":"CABPp-BHQif-tRkG8tB7V8B41jFesVbnCK1tsrTZNXg0_WaOcbg@mail.gmail.com","threadId":"56534","inReplyTo":"xmqq7df5x4f1.fsf@gitster.g","subject":"Re: [PATCH v2 2/6] Change unpack_trees' 'reset' flag into an enum","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-26T06:50:48Z","receivedAt":"2021-09-26T06:51:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Sep 24, 2021 at 10:35 AM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > Also, note that 'git checkout ' currently also allows\n> > overwriting untracked files. That case should also be fixed, ...\n>\n> I wasted a few minutes wondering about the example. Please make it\n> clear that you are checking out of a tree-ish that is different from\n> HEAD, as there will by definition no \"overwriting untracked\" if you\n> are checking out of the index.\n>\n> E.g. \"git checkout -- \".\n>\n> With this command line:\n>\n> $ git checkout HEAD~24 -- path\n>\n> where path used to be there as late as 24 revisions ago, but since\n> then we removed, and the user wants to materialize the file out of\n> the old version, path, be it tracked, untracked, or even a\n> directory, should be made identical to the copy from the given\n> version, no? Where does the \"should also be fixed\" come from?\n\nSorry, I should have been more careful. Phillip noted this other case\nthat overwrote untracked files and suggested mentioning it; see\nhttps://lore.kernel.org/git/acef3628-9542-d777-2534-577de9707e15@gmail.com/\nand the links from there. Perhaps I would have been better off\nmentioning that link and the other links in it.\n\n> > diff --git a/builtin/am.c b/builtin/am.c\n> > index c79e0167e98..b17baa67ad8 100644\n> > --- a/builtin/am.c\n> > +++ b/builtin/am.c\n> > @@ -1918,8 +1918,14 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n> > opts.dst_index = &the_index;\n> > opts.update = 1;\n> > opts.merge = 1;\n> > - opts.reset = reset;\n> > + opts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n> > opts.fn = twoway_merge;\n> > + if (opts.reset) {\n> > + /* Allow ignored files in the way to get overwritten */\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n>\n> Do these three lines make a recurring pattern when opts.reset is set?\n> I am wondering if this can be done more centrally by the unpack-trees\n> machinery (i.e. \"gee this one has o->reset set to X, so let's set up\n> the o->dir before doing anything\").\n\nIt's a very common pattern...but not universal due to some prevailing\nweirdness and differences in how commands handle ignored files. But,\nit looks like that can be fixed up, and then this can be centralized.\nI've got some patches doing this.\n\n> > diff --git a/builtin/checkout.c b/builtin/checkout.c\n> > index b5d477919a7..52826e0d145 100644\n> > --- a/builtin/checkout.c\n> > +++ b/builtin/checkout.c\n> > @@ -641,23 +641,37 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n> > {\n> > struct unpack_trees_options opts;\n> > struct tree_desc tree_desc;\n> > + int unpack_trees_ret;\n> >\n> > memset(&opts, 0, sizeof(opts));\n> > opts.head_idx = -1;\n> > opts.update = worktree;\n> > opts.skip_unmerged = !worktree;\n> > - opts.reset = 1;\n> > + opts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n> > + UNPACK_RESET_PROTECT_UNTRACKED;\n> > opts.merge = 1;\n> > opts.fn = oneway_merge;\n> > opts.verbose_update = o->show_progress;\n> > opts.src_index = &the_index;\n> > opts.dst_index = &the_index;\n> > + if (o->overwrite_ignore) {\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n> > + }\n>\n> If our longer term goal is to decide classification of files not in\n> the index (currently, \"ignored\" and \"untracked\", but we may want to\n> add a new \"precious\" class) and (across various commands that build\n> on the unpack-trees infrastructure) to protect the \"untracked\" and\n> \"precious\" ones, with --[no-]overwrite-{ignore,untracked} options as\n> escape hatches, uniformly, perhaps the --[no-]-overwrite-ignore\n> option may be stolen from here and shifted to unpack_tree_options to\n> help us going in that direction? This is just an observation for\n> longer term, not a suggestion to include the first step for such a\n> move in this series.\n\nI had thought about that, but I was already pretty deep down the\nrabbit hole of trying to avoid removing the current working directory\nas a side effect of other commands. I noted in my cover letter the\n--no-overwrite-ignore options and how they could be used building on\nthis series, and now you bring it up too. And your previous comment\nabout the common pattern for unpack_trees_options.dir forces me to go\nfurther down this hole a bit. So, I've got patches that implement a\nfair amount of this, but doesn't actually add the new\n--no-overwrite-ignore options or plumb them through. They do,\nhowever, add a FIXME comment where a single boolean value could be set\nin order to make such options work.\n\n>\n> > init_checkout_metadata(&opts.meta, info->refname,\n> > info->commit ? &info->commit->object.oid : null_oid(),\n> > NULL);\n> > parse_tree(tree);\n> > init_tree_desc(&tree_desc, tree->buffer, tree->size);\n> > - switch (unpack_trees(1, &tree_desc, &opts)) {\n> > + unpack_trees_ret = unpack_trees(1, &tree_desc, &opts);\n> > +\n> > + if (o->overwrite_ignore) {\n> > + dir_clear(opts.dir);\n> > + FREE_AND_NULL(opts.dir);\n>\n> This dir_clear() is also a recurring theme. See below.\n>\n> > + }\n> > +\n> > + switch (unpack_trees_ret) {\n> > case -2:\n> > *writeout_error = 1;\n> > /*\n> > diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> > index 485e7b04794..740fc0335af 100644\n> > --- a/builtin/read-tree.c\n> > +++ b/builtin/read-tree.c\n> > @@ -174,6 +174,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> > if (1 < opts.merge + opts.reset + prefix_set)\n> > die(\"Which one? -m, --reset, or --prefix?\");\n> >\n> > + if (opts.reset)\n> > + opts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n> > +\n>\n> We do not do anything about opts.dir here by default, which means we\n> by default do not overwrite ignored, but that's OK, because this old\n> command explicitly takes --exclude-per-directory to tell it what the\n> command should consider \"ignored\" and with the option given we do\n> prepare opts.dir just fine.\n>\n> > diff --git a/builtin/reset.c b/builtin/reset.c\n> > index 43e855cb887..a12ee986e9f 100644\n> > --- a/builtin/reset.c\n> > +++ b/builtin/reset.c\n> > @@ -10,6 +10,7 @@\n> > #define USE_THE_INDEX_COMPATIBILITY_MACROS\n> > #include \"builtin.h\"\n> > #include \"config.h\"\n> > +#include \"dir.h\"\n> > #include \"lockfile.h\"\n> > #include \"tag.h\"\n> > #include \"object.h\"\n> > @@ -70,9 +71,20 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> > break;\n> > case HARD:\n> > opts.update = 1;\n> > - /* fallthrough */\n> > + opts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n> > + break;\n> > + case MIXED:\n> > + opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n> > + /* but opts.update=0, so working tree not updated */\n> > + break;\n> > default:\n> > - opts.reset = 1;\n> > + BUG(\"invalid reset_type passed to reset_index\");\n> > + }\n> > + if (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n>\n> Unlike the one in \"am\", this cares .reset being a particular value,\n> not just being non-zero. Puzzling.\n>\n> It is a bit counter-intuitive in that we do not allow overwrite\n> ignored (which is currently a synonym for \"expendable\") when .reset\n> is set to allow us to ovewrite untracked.\n\nNo, when opts.reset == UNPACK_RESET_NUKE_UNTRACKED, unpack_trees() has\nno way to differentiate between untracked and ignored and thus just\ndeletes them all. Setting up unpack_trees_options.dir so that we can\ndifferentiate between untracked and ignored files when we are just\ngoing to treat them the same (delete them both) is wasted effort. So\nthe check for a certain type of reset value was merely an\noptimization. (And that optimization didn't apply in the am case,\nsince it doesn't use UNPACK_RESET_NUKE_UNTRACKED.)\n\nHowever, consolidating the unpack_trees_options.dir handling into\nunpack_trees() allows me to better document the optimization and only\nworry about it in one place, which seems to make the code much\nclearer.\n\n>\n> > + /* Setup opts.dir so we can overwrite ignored files */\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n>\n> > @@ -104,6 +116,10 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> > ret = 0;\n> >\n> > out:\n> > + if (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n> > + dir_clear(opts.dir);\n> > + FREE_AND_NULL(opts.dir);\n>\n> This dir_clear() is also a recurring theme. See below.\n>\n> > diff --git a/builtin/stash.c b/builtin/stash.c\n> > index 8f42360ca91..563f590afbd 100644\n> > --- a/builtin/stash.c\n> > +++ b/builtin/stash.c\n> > @@ -237,6 +237,7 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n> > struct tree_desc t[MAX_UNPACK_TREES];\n> > struct tree *tree;\n> > struct lock_file lock_file = LOCK_INIT;\n> > + int unpack_trees_ret;\n> >\n> > read_cache_preload(NULL);\n> > if (refresh_cache(REFRESH_QUIET))\n> > @@ -256,11 +257,23 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n> > opts.src_index = &the_index;\n> > opts.dst_index = &the_index;\n> > opts.merge = 1;\n> > - opts.reset = reset;\n> > + opts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n> > + if (opts.reset) {\n> > + opts.dir = xcalloc(1, sizeof(*opts.dir));\n> > + opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(opts.dir);\n> > + }\n> > opts.update = update;\n> > opts.fn = oneway_merge;\n> >\n> > - if (unpack_trees(nr_trees, t, &opts))\n> > + unpack_trees_ret = unpack_trees(nr_trees, t, &opts);\n> > +\n> > + if (opts.reset) {\n> > + dir_clear(opts.dir);\n> > + FREE_AND_NULL(opts.dir);\n>\n> This dir_clear() is also a recurring theme. Why aren't their guards\n> uniformly \"if (opts.dir)\"? The logic to decide if we set up opts.dir\n> or not may be far from here and may be different from code path to\n> code path, but the need to clear opts.dir should not have to care\n> why opts.dir was populated, no?\n\nYeah, you're right; I should have used \"if (opts.dir)\" for all these cases.\n\n> > + }\n> > +\n> > + if (unpack_trees_ret)\n> > return -1;\n> >\n> > if (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n> > diff --git a/reset.c b/reset.c\n> > index 79310ae071b..1695f3828c5 100644\n> > --- a/reset.c\n> > +++ b/reset.c\n> > @@ -1,5 +1,6 @@\n> > #include \"git-compat-util.h\"\n> > #include \"cache-tree.h\"\n> > +#include \"dir.h\"\n> > #include \"lockfile.h\"\n> > #include \"refs.h\"\n> > #include \"reset.h\"\n> > @@ -57,8 +58,12 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n> > unpack_tree_opts.update = 1;\n> > unpack_tree_opts.merge = 1;\n> > init_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n> > - if (!detach_head)\n> > - unpack_tree_opts.reset = 1;\n> > + if (!detach_head) {\n> > + unpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n> > + unpack_tree_opts.dir = xcalloc(1, sizeof(*unpack_tree_opts.dir));\n> > + unpack_tree_opts.dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(unpack_tree_opts.dir);\n> > + }\n> >\n> > if (repo_read_index_unmerged(r) < 0) {\n> > ret = error(_(\"could not read index\"));\n> > @@ -131,6 +136,10 @@ reset_head_refs:\n> > oid_to_hex(oid), \"1\", NULL);\n> >\n> > leave_reset_head:\n> > + if (unpack_tree_opts.dir) {\n> > + dir_clear(unpack_tree_opts.dir);\n> > + FREE_AND_NULL(unpack_tree_opts.dir);\n>\n> Yes, I think this is the right way to decide if we call dir_clear(),\n> and all other hunks in this patch should do the same.\n\nWill fix, though the consolidation ends up making it just one place anyway.\n"},{"id":"437151","messageId":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v2.git.1632465429.gitgitgadget@gmail.com","subject":"[PATCH v3 00/11] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:37Z","receivedAt":"2021-09-27T16:34:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"We have multiple codepaths that delete untracked files/directories but\nshouldn't. There are also some codepaths where we delete untracked\nfiles/directories intentionally (based on mailing list discussion), but\nwhere that intent is not documented. We also have some codepaths that\npreserve ignored files, which shouldn't. Fix the documentation, add several\nnew (mostly failing) testcases, fix some of the new testcases, and add\ncomments about some potential remaining problems. (I found these as a\nside-effect of looking at [1], though [2] pointed out one explicitly while I\nwas working on it.)\n\nNote that I'm using Junio's declaration about checkout -f and reset --hard\n(and also presuming that since read-tree --reset is porcelain that its\nbehavior should be left alone)[3] in this series.\n\nChanges since v2 (all due to Junio's request to consolidate\nunpack_trees_options.dir handling):\n\n * fix some (pre-existing) memory leaks, in preparation for consolidating\n some common code (new patch 2)\n * New patches (3 & 6) to make a few more commands remove ignored files by\n default -- which also fixes an existing testcase\n * New patches (4 & 5) to consolidate the various places handling\n unpack_trees_options.dir and default to treating ignored files as\n expendable within unpack_trees(). These change also make it very easy to\n add --no-overwrite-ignore options in the future to additional commands\n (checkout and merge already have such an option, though merge only passes\n that along to the fast-forwarding backend)\n\nChanges since v1:\n\n * Various small cleanups (suggested by Ævar)\n * Fixed memory leaks of unpack_trees_opts->dir (also suggested by Ævar)\n * Use an enum for unpack_trees_options->reset, instead of multiple fields\n (suggested by Phillip)\n * Avoid changing behavior for cases not setting unpack_trees_options.reset\n > 0 (even if it may make sense to nuke ignored files when running either\n read-tree -m -u or the various reset flavors run internally by\n rebase/sequencer); we can revisit that later.\n\nSIDENOTE about treating ignored files as precious:\n\nThe patches are now getting pretty close to being able to handle ignored\nfiles as precious. The only things left would be making merge pass the\n--no-overwrite-ignore option along to more backends, and adding the\n--no-overwrite-ignore option that both checkout and merge take to more\ncommands. There's already comments in the code about what boolean would need\nto be set by that flag. And then perhaps also make a global\ncore.overwrite_ignored config option to affect all of these. Granted, doing\nthis would globally treat ignored files as precious rather than allowing\nthem to be configured on a per-path basis, but honestly I think the idea of\nconfiguring ignored files as precious on a per-path basis sounds like\ninsanity. (We have enough bugs with untracked and ignored files without\nadding yet another type. And, of course, configuring per-path rules sounds\nlike lots of work for end users to configure. There may be additional\nreasons against it.) So, if someone wants to pursue the precious-ignored\nconcept then I'd much rather see it done as a global setting. Just my $0.02.\n\n[1] https://lore.kernel.org/git/xmqqv93n7q1v.fsf@gitster.g/ [2]\nhttps://lore.kernel.org/git/C357A648-8B13-45C3-9388-C0C7F7D40DAE@gmail.com/\n[3] https://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/\n\nElijah Newren (11):\n t2500: add various tests for nuking untracked files\n checkout, read-tree: fix leak of unpack_trees_options.dir\n read-tree, merge-recursive: overwrite ignored files by default\n unpack-trees: introduce preserve_ignored to unpack_trees_options\n unpack-trees: make dir an internal-only struct\n Remove ignored files by default when they are in the way\n Change unpack_trees' 'reset' flag into an enum\n unpack-trees: avoid nuking untracked dir in way of unmerged file\n unpack-trees: avoid nuking untracked dir in way of locally deleted\n file\n Comment important codepaths regarding nuking untracked files/dirs\n Documentation: call out commands that nuke untracked files/directories\n\n Documentation/git-checkout.txt | 5 +-\n Documentation/git-read-tree.txt | 23 +--\n Documentation/git-reset.txt | 3 +-\n builtin/am.c | 3 +-\n builtin/checkout.c | 10 +-\n builtin/clone.c | 1 +\n builtin/merge.c | 1 +\n builtin/read-tree.c | 26 ++--\n builtin/reset.c | 10 +-\n builtin/stash.c | 5 +-\n builtin/submodule--helper.c | 4 +\n contrib/rerere-train.sh | 2 +-\n merge-ort.c | 8 +-\n merge-recursive.c | 5 +-\n merge.c | 8 +-\n reset.c | 3 +-\n sequencer.c | 1 +\n submodule.c | 1 +\n t/t1013-read-tree-submodule.sh | 1 -\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n t/t7112-reset-submodule.sh | 1 -\n unpack-trees.c | 61 +++++++-\n unpack-trees.h | 14 +-\n 23 files changed, 366 insertions(+), 74 deletions(-)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\n\nbase-commit: ddb1055343948e0d0bc81f8d20245f1ada6430a0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1036%2Fnewren%2Funtracked_removal-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1036/newren/untracked_removal-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1036\n\nRange-diff vs v2:\n\n 1: 9460a49c7ed = 1: 66270ffc74e t2500: add various tests for nuking untracked files\n -: ----------- > 2: 0c74285b253 checkout, read-tree: fix leak of unpack_trees_options.dir\n -: ----------- > 3: 2501a0c552a read-tree, merge-recursive: overwrite ignored files by default\n -: ----------- > 4: f1a0700e598 unpack-trees: introduce preserve_ignored to unpack_trees_options\n -: ----------- > 5: 0d119142778 unpack-trees: make dir an internal-only struct\n -: ----------- > 6: b7fe354efff Remove ignored files by default when they are in the way\n 2: b77692b8f49 ! 7: 9eb20121fc3 Change unpack_trees' 'reset' flag into an enum\n @@ Commit message\n (i.e. \"true\") can be split into two:\n UNPACK_RESET_PROTECT_UNTRACKED,\n UNPACK_RESET_OVERWRITE_UNTRACKED\n - In order to catch accidental misuses, define with the enum a special\n - value of\n + In order to catch accidental misuses (i.e. where folks call it the way\n + they traditionally used to), define the special enum value of\n UNPACK_RESET_INVALID = 1\n which will trigger a BUG().\n \n @@ Commit message\n numerous callers from rebase/sequencer to reset_head()\n will use the new UNPACK_RESET_PROTECT_UNTRACKED value.\n \n - In order to protect untracked files but still allow deleting of ignored\n - files, we also have to setup unpack_trees_opt.dir. It may make sense to\n - set up unpack_trees_opt.dir in more cases, but here I tried to only do\n - so in cases where we switched from deleting all untracked files to\n - avoiding doing so (i.e. where we now use\n - UNPACK_RESET_PROTECT_UNTRACKED).\n + Also, note that it has been reported that 'git checkout \n + ' currently also allows overwriting untracked files[1]. That\n + case should also be fixed, but it does not use unpack_trees() and thus\n + is outside the scope of the current changes.\n \n - Also, note that 'git checkout ' currently also allows\n - overwriting untracked files. That case should also be fixed, but it\n - does not use unpack_trees() and thus is outside the scope of the current\n - changes.\n + [1] https://lore.kernel.org/git/15dad590-087e-5a48-9238-5d2826950506@gmail.com/\n \n Signed-off-by: Elijah Newren \n \n @@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote,\n \topts.update = 1;\n \topts.merge = 1;\n -\topts.reset = reset;\n +-\tif (!reset)\n +-\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n +\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n ++\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = twoway_merge;\n -+\tif (opts.reset) {\n -+\t\t/* Allow ignored files in the way to get overwritten */\n -+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n -+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opts.dir);\n -+\t}\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\n - \n -@@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n - \t\treturn -1;\n - \t}\n - \n -+\tif (opts.reset) {\n -+\t\tdir_clear(opts.dir);\n -+\t\tFREE_AND_NULL(opts.dir);\n -+\t}\n -+\n - \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n - \t\tdie(_(\"unable to write new index file\"));\n - \n \n ## builtin/checkout.c ##\n @@ builtin/checkout.c: static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n - {\n - \tstruct unpack_trees_options opts;\n - \tstruct tree_desc tree_desc;\n -+\tint unpack_trees_ret;\n - \n - \tmemset(&opts, 0, sizeof(opts));\n \topts.head_idx = -1;\n \topts.update = worktree;\n \topts.skip_unmerged = !worktree;\n -\topts.reset = 1;\n +\topts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n +\t\t\t\tUNPACK_RESET_PROTECT_UNTRACKED;\n ++\topts.preserve_ignored = (!o->force && !o->overwrite_ignore);\n \topts.merge = 1;\n +-\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\n - \topts.dst_index = &the_index;\n -+\tif (o->overwrite_ignore) {\n -+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n -+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opts.dir);\n -+\t}\n - \tinit_checkout_metadata(&opts.meta, info->refname,\n - \t\t\t info->commit ? &info->commit->object.oid : null_oid(),\n - \t\t\t NULL);\n - \tparse_tree(tree);\n - \tinit_tree_desc(&tree_desc, tree->buffer, tree->size);\n --\tswitch (unpack_trees(1, &tree_desc, &opts)) {\n -+\tunpack_trees_ret = unpack_trees(1, &tree_desc, &opts);\n -+\n -+\tif (o->overwrite_ignore) {\n -+\t\tdir_clear(opts.dir);\n -+\t\tFREE_AND_NULL(opts.dir);\n -+\t}\n -+\n -+\tswitch (unpack_trees_ret) {\n - \tcase -2:\n - \t\t*writeout_error = 1;\n - \t\t/*\n \n ## builtin/read-tree.c ##\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n \t *\n \n ## builtin/reset.c ##\n -@@\n - #define USE_THE_INDEX_COMPATIBILITY_MACROS\n - #include \"builtin.h\"\n - #include \"config.h\"\n -+#include \"dir.h\"\n - #include \"lockfile.h\"\n - #include \"tag.h\"\n - #include \"object.h\"\n @@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \t\tbreak;\n \tcase HARD:\n @@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id\n \tdefault:\n -\t\topts.reset = 1;\n +\t\tBUG(\"invalid reset_type passed to reset_index\");\n -+\t}\n -+\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n -+\t\t/* Setup opts.dir so we can overwrite ignored files */\n -+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n -+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opts.dir);\n \t}\n \n \tread_cache_unmerged();\n -@@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n - \tret = 0;\n - \n - out:\n -+\tif (opts.reset == UNPACK_RESET_PROTECT_UNTRACKED) {\n -+\t\tdir_clear(opts.dir);\n -+\t\tFREE_AND_NULL(opts.dir);\n -+\t}\n - \tfor (i = 0; i < nr; i++)\n - \t\tfree((void *)desc[i].buffer);\n - \treturn ret;\n \n ## builtin/stash.c ##\n -@@ builtin/stash.c: static int reset_tree(struct object_id *i_tree, int update, int reset)\n - \tstruct tree_desc t[MAX_UNPACK_TREES];\n - \tstruct tree *tree;\n - \tstruct lock_file lock_file = LOCK_INIT;\n -+\tint unpack_trees_ret;\n - \n - \tread_cache_preload(NULL);\n - \tif (refresh_cache(REFRESH_QUIET))\n @@ builtin/stash.c: static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n \topts.merge = 1;\n -\topts.reset = reset;\n +\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n -+\tif (opts.reset) {\n -+\t\topts.dir = xcalloc(1, sizeof(*opts.dir));\n -+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opts.dir);\n -+\t}\n \topts.update = update;\n +-\tif (update && !reset)\n ++\tif (update)\n + \t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = oneway_merge;\n \n --\tif (unpack_trees(nr_trees, t, &opts))\n -+\tunpack_trees_ret = unpack_trees(nr_trees, t, &opts);\n -+\n -+\tif (opts.reset) {\n -+\t\tdir_clear(opts.dir);\n -+\t\tFREE_AND_NULL(opts.dir);\n -+\t}\n -+\n -+\tif (unpack_trees_ret)\n - \t\treturn -1;\n - \n - \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \n ## reset.c ##\n -@@\n - #include \"git-compat-util.h\"\n - #include \"cache-tree.h\"\n -+#include \"dir.h\"\n - #include \"lockfile.h\"\n - #include \"refs.h\"\n - #include \"reset.h\"\n @@ reset.c: int reset_head(struct repository *r, struct object_id *oid, const char *action,\n - \tunpack_tree_opts.update = 1;\n - \tunpack_tree_opts.merge = 1;\n + \tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n --\tif (!detach_head)\n + \tif (!detach_head)\n -\t\tunpack_tree_opts.reset = 1;\n -+\tif (!detach_head) {\n +\t\tunpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n -+\t\tunpack_tree_opts.dir = xcalloc(1, sizeof(*unpack_tree_opts.dir));\n -+\t\tunpack_tree_opts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(unpack_tree_opts.dir);\n -+\t}\n \n \tif (repo_read_index_unmerged(r) < 0) {\n \t\tret = error(_(\"could not read index\"));\n -@@ reset.c: reset_head_refs:\n - \t\t\t oid_to_hex(oid), \"1\", NULL);\n - \n - leave_reset_head:\n -+\tif (unpack_tree_opts.dir) {\n -+\t\tdir_clear(unpack_tree_opts.dir);\n -+\t\tFREE_AND_NULL(unpack_tree_opts.dir);\n -+\t}\n - \tstrbuf_release(&msg);\n - \trollback_lock_file(&lock);\n - \tclear_unpack_trees_porcelain(&unpack_tree_opts);\n \n ## t/t2500-untracked-overwriting.sh ##\n @@ t/t2500-untracked-overwriting.sh: test_setup_checkout_m () {\n @@ t/t2500-untracked-overwriting.sh: test_expect_failure 'git rebase --abort and un\n \n ## unpack-trees.c ##\n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n - \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n + \tstruct dir_struct dir = DIR_INIT;\n \n +\tif (o->reset == UNPACK_RESET_INVALID)\n +\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n +\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n + \tif (o->dir)\n +@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n + \t\tensure_full_index(o->dst_index);\n + \t}\n \n ++\tif (o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED &&\n ++\t o->preserve_ignored)\n ++\t\tBUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n ++\n + \tif (!o->preserve_ignored) {\n + \t\to->dir = &dir;\n + \t\to->dir->flags |= DIR_SHOW_IGNORED;\n @@ unpack-trees.c: static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n @@ unpack-trees.h: void setup_unpack_trees_porcelain(struct unpack_trees_options *o\n -\t\t merge,\n +\tunsigned int merge,\n \t\t update,\n + \t\t preserve_ignored,\n \t\t clone,\n - \t\t index_only,\n @@ unpack-trees.h: struct unpack_trees_options {\n \t\t exiting_early,\n \t\t show_all_errors,\n @@ unpack-trees.h: struct unpack_trees_options {\n +\tenum unpack_trees_reset_type reset;\n \tconst char *prefix;\n \tint cache_bottom;\n - \tstruct dir_struct *dir;\n + \tstruct pathspec *pathspec;\n 3: 208f3b3ebe5 = 8: e4c42d43b09 unpack-trees: avoid nuking untracked dir in way of unmerged file\n 4: 0a0997d081b = 9: 1a770681704 unpack-trees: avoid nuking untracked dir in way of locally deleted file\n 5: 4b78a526d2a ! 10: 6b42a80bf3d Comment important codepaths regarding nuking untracked files/dirs\n @@ Commit message\n * git-archimport.perl: Don't care; arch is long since dead\n * git-cvs*.perl: Don't care; cvs is long since dead\n \n + Also, the reset --hard in builtin/worktree.c looks safe, due to only\n + running in an empty directory.\n +\n Signed-off-by: Elijah Newren \n \n ## builtin/stash.c ##\n @@ builtin/submodule--helper.c: static int add_submodule(const struct add_data *add\n \n \t\tif (add_data->branch) {\n \n - ## builtin/worktree.c ##\n -@@ builtin/worktree.c: static int add_worktree(const char *path, const char *refname,\n - \tif (opts->checkout) {\n - \t\tcp.argv = NULL;\n - \t\tstrvec_clear(&cp.args);\n -+\t\t/*\n -+\t\t * NOTE: reset --hard is okay here, because 'worktree add'\n -+\t\t * refuses to work in an extant non-empty directory, so there\n -+\t\t * is no risk of deleting untracked files.\n -+\t\t */\n - \t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"--no-recurse-submodules\", NULL);\n - \t\tif (opts->quiet)\n - \t\t\tstrvec_push(&cp.args, \"--quiet\");\n -\n ## contrib/rerere-train.sh ##\n @@ contrib/rerere-train.sh: do\n \t\tgit checkout -q $commit -- .\n 6: 993451a8036 = 11: de416f887d7 Documentation: call out commands that nuke untracked files/directories\n\n-- \ngitgitgadget\n"},{"id":"437152","messageId":"0c74285b25311c83bb158cf89a551160a0f3a5d3.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 02/11] checkout, read-tree: fix leak of unpack_trees_options.dir","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:39Z","receivedAt":"2021-09-27T16:34:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n builtin/checkout.c | 4 ++++\n builtin/read-tree.c | 5 +++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 8c69dcdf72a..5335435d616 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -760,6 +760,10 @@ static int merge_working_tree(const struct checkout_opts *opts,\n \t\tinit_tree_desc(&trees[1], tree->buffer, tree->size);\n \n \t\tret = unpack_trees(2, trees, &topts);\n+\t\tif (topts.dir) {\n+\t\t\tdir_clear(topts.dir);\n+\t\t\tFREE_AND_NULL(topts.dir);\n+\t\t}\n \t\tclear_unpack_trees_porcelain(&topts);\n \t\tif (ret == -1) {\n \t\t\t/*\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 485e7b04794..96102c222bf 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -250,6 +250,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (unpack_trees(nr_trees, t, &opts))\n \t\treturn 128;\n \n+\tif (opts.dir) {\n+\t\tdir_clear(opts.dir);\n+\t\tFREE_AND_NULL(opts.dir);\n+\t}\n+\n \tif (opts.debug_unpack || opts.dry_run)\n \t\treturn 0; /* do not write the index out */\n \n-- \ngitgitgadget\n\n"},{"id":"437153","messageId":"2501a0c552ad5147f61a96b9ebe45c5199e1dbfd.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 03/11] read-tree, merge-recursive: overwrite ignored files by default","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:40Z","receivedAt":"2021-09-27T16:34:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis fixes a long-standing patchwork of ignored files handling in\nread-tree and merge-recursive, called out and suggested by Junio long\nago. Quoting from commit dcf0c16ef1 (\"core.excludesfile clean-up\"\n2007-11-16):\n\n git-read-tree takes --exclude-per-directory=,\n not because the flexibility was needed. Again, this was\n because the option predates the standardization of the ignore\n files.\n\n ...\n\n On the other hand, I think it makes perfect sense to fix\n git-read-tree, git-merge-recursive and git-clean to follow the\n same rule as other commands. I do not think of a valid use case\n to give an exclude-per-directory that is nonstandard to\n read-tree command, outside a \"negative\" test in the t1004 test\n script.\n\n This patch is the first step to untangle this mess.\n\n The next step would be to teach read-tree, merge-recursive and\n clean (in C) to use setup_standard_excludes().\n\nHistory shows each of these were partially or fully fixed:\n\n * clean was taught the new trick in 1617adc7a0 (\"Teach git clean to\n use setup_standard_excludes()\", 2007-11-14).\n\n * read-tree was primarily used by checkout & merge scripts. checkout\n and merge later became builtins and were both fixed to use the new\n setup_standard_excludes() handling in fc001b526c (\"checkout,merge:\n loosen overwriting untracked file check based on info/exclude\",\n 2011-11-27). So the primary users were fixed, though read-tree\n itself was not.\n\n * merge-recursive has now been replaced as the default merge backend\n by merge-ort. merge-ort fixed this by using\n setup_standard_excludes() starting early in its implementation; see\n commit 6681ce5cf6 (\"merge-ort: add implementation of checkout()\",\n 2020-12-13), largely due to its design depending on checkout() and\n thus being influenced by the checkout code. However,\n merge-recursive itself was not fixed here, in part because its\n design meant it had difficulty differentiating between untracked\n files, ignored files, leftover tracked files that haven't been\n removed yet due to order of processing files, and files written by\n itself due to collisions).\n\nMake the conversion more complete by now handling read-tree and\nhandling at least the unpack_trees() portion of merge-recursive. While\nmerge-recursive is on its way out, fixing the unpack_trees() portion is\neasy and facilitates some of the later changes in this series. Note\nthat fixing read-tree makes the --exclude-per-directory option to\nread-tree useless, so we remove it from the documentation (though we\ncontinue to accept it if passed).\n\nThe read-tree changes happen to fix a bug in t1013.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-read-tree.txt | 18 +-----------------\n builtin/read-tree.c | 25 ++++++++++---------------\n merge-recursive.c | 11 ++++++++++-\n t/t1013-read-tree-submodule.sh | 1 -\n 4 files changed, 21 insertions(+), 34 deletions(-)\n\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 5fa8bab64c2..0222a27c5af 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -10,8 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git read-tree' [[-m [--trivial] [--aggressive] | --reset | --prefix=]\n-\t\t[-u [--exclude-per-directory=] | -i]]\n-\t\t[--index-output=] [--no-sparse-checkout]\n+\t\t[-u | -i]] [--index-output=] [--no-sparse-checkout]\n \t\t(--empty | [ []])\n \n \n@@ -88,21 +87,6 @@ OPTIONS\n \tThe command will refuse to overwrite entries that already\n \texisted in the original index file.\n \n---exclude-per-directory=::\n-\tWhen running the command with `-u` and `-m` options, the\n-\tmerge result may need to overwrite paths that are not\n-\ttracked in the current branch. The command usually\n-\trefuses to proceed with the merge to avoid losing such a\n-\tpath. However this safety valve sometimes gets in the\n-\tway. For example, it often happens that the other\n-\tbranch added a file that used to be a generated file in\n-\tyour branch, and the safety valve triggers when you try\n-\tto switch to that branch after you ran `make` but before\n-\trunning `make clean` to remove the generated file. This\n-\toption tells the command to read per-directory exclude\n-\tfile (usually '.gitignore') and allows such an untracked\n-\tbut explicitly ignored file to be overwritten.\n-\n --index-output=::\n \tInstead of writing the results out to `$GIT_INDEX_FILE`,\n \twrite the resulting index in the named file. While the\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 96102c222bf..73cb957a69b 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -38,7 +38,7 @@ static int list_tree(struct object_id *oid)\n }\n \n static const char * const read_tree_usage[] = {\n-\tN_(\"git read-tree [(-m [--trivial] [--aggressive] | --reset | --prefix=) [-u [--exclude-per-directory=] | -i]] [--no-sparse-checkout] [--index-output=] (--empty | [ []])\"),\n+\tN_(\"git read-tree [(-m [--trivial] [--aggressive] | --reset | --prefix=) [-u | -i]] [--no-sparse-checkout] [--index-output=] (--empty | [ []])\"),\n \tNULL\n };\n \n@@ -53,24 +53,16 @@ static int index_output_cb(const struct option *opt, const char *arg,\n static int exclude_per_directory_cb(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n {\n-\tstruct dir_struct *dir;\n \tstruct unpack_trees_options *opts;\n \n \tBUG_ON_OPT_NEG(unset);\n \n \topts = (struct unpack_trees_options *)opt->value;\n \n-\tif (opts->dir)\n-\t\tdie(\"more than one --exclude-per-directory given.\");\n-\n-\tdir = xcalloc(1, sizeof(*opts->dir));\n-\tdir->flags |= DIR_SHOW_IGNORED;\n-\tdir->exclude_per_dir = arg;\n-\topts->dir = dir;\n-\t/* We do not need to nor want to do read-directory\n-\t * here; we are merely interested in reusing the\n-\t * per directory ignore stack mechanism.\n-\t */\n+\tif (!opts->update)\n+\t\tdie(\"--exclude-per-directory is meaningless unless -u\");\n+\tif (strcmp(arg, \".gitignore\"))\n+\t\tdie(\"--exclude-per-directory argument must be .gitignore\");\n \treturn 0;\n }\n \n@@ -209,8 +201,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif ((opts.update || opts.index_only) && !opts.merge)\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n-\tif ((opts.dir && !opts.update))\n-\t\tdie(\"--exclude-per-directory is meaningless unless -u\");\n+\tif (opts.update && !opts.reset) {\n+\t\tCALLOC_ARRAY(opts.dir, 1);\n+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opts.dir);\n+\t}\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex e594d4c3fa1..233d9f686ad 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -408,8 +408,13 @@ static int unpack_trees_start(struct merge_options *opt,\n \tmemset(&opt->priv->unpack_opts, 0, sizeof(opt->priv->unpack_opts));\n \tif (opt->priv->call_depth)\n \t\topt->priv->unpack_opts.index_only = 1;\n-\telse\n+\telse {\n \t\topt->priv->unpack_opts.update = 1;\n+\t\t/* FIXME: should only do this if !overwrite_ignore */\n+\t\tCALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n+\t\topt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(opt->priv->unpack_opts.dir);\n+\t}\n \topt->priv->unpack_opts.merge = 1;\n \topt->priv->unpack_opts.head_idx = 2;\n \topt->priv->unpack_opts.fn = threeway_merge;\n@@ -423,6 +428,10 @@ static int unpack_trees_start(struct merge_options *opt,\n \tinit_tree_desc_from_tree(t+2, merge);\n \n \trc = unpack_trees(3, t, &opt->priv->unpack_opts);\n+\tif (opt->priv->unpack_opts.dir) {\n+\t\tdir_clear(opt->priv->unpack_opts.dir);\n+\t\tFREE_AND_NULL(opt->priv->unpack_opts.dir);\n+\t}\n \tcache_tree_free(&opt->repo->index->cache_tree);\n \n \t/*\ndiff --git a/t/t1013-read-tree-submodule.sh b/t/t1013-read-tree-submodule.sh\nindex b6df7444c05..bfc90d4cf27 100755\n--- a/t/t1013-read-tree-submodule.sh\n+++ b/t/t1013-read-tree-submodule.sh\n@@ -6,7 +6,6 @@ test_description='read-tree can handle submodules'\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \n KNOWN_FAILURE_DIRECTORY_SUBMODULE_CONFLICTS=1\n-KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1\n \n test_submodule_switch_recursing_with_args \"read-tree -u -m\"\n \n-- \ngitgitgadget\n\n"},{"id":"437154","messageId":"66270ffc74ecf4f2ce0ebc7379be84d5470c82ef.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 01/11] t2500: add various tests for nuking untracked files","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:38Z","receivedAt":"2021-09-27T16:34:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nNoting that unpack_trees treats reset=1 & update=1 as license to nuke\nuntracked files, I looked for code paths that use this combination and\ntried to generate testcases which demonstrated unintentional loss of\nuntracked files and directories. I found several.\n\nI also include testcases for `git reset --{hard,merge,keep}`. A hard\nreset is perhaps the most direct test of unpack_tree's reset=1 behavior,\nbut we cannot make `git reset --hard` preserve untracked files without\nsome migration work.\n\nAlso, the two commands `checkout --force` (because of the --force) and\n`read-tree --reset` (because it's plumbing and we need to keep it\nbackward compatible) were left out as we expect those to continue\nremoving untracked files and directories.\n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n 1 file changed, 244 insertions(+)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nnew file mode 100755\nindex 00000000000..2412d121ea8\n--- /dev/null\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -0,0 +1,244 @@\n+#!/bin/sh\n+\n+test_description='Test handling of overwriting untracked files'\n+\n+. ./test-lib.sh\n+\n+test_setup_reset () {\n+\tgit init reset_$1 &&\n+\t(\n+\t\tcd reset_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\tgit branch stable &&\n+\t\tgit branch work &&\n+\n+\t\tgit checkout work &&\n+\t\ttest_commit foo &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_success 'reset --hard will nuke untracked files/dirs' '\n+\ttest_setup_reset hard &&\n+\t(\n+\t\tcd reset_hard &&\n+\t\tgit ls-tree -r stable &&\n+\t\tgit log --all --name-status --oneline &&\n+\t\tgit ls-tree -r work &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\techo foo >expect &&\n+\n+\t\tgit reset --hard work &&\n+\n+\t\t# check that untracked directory foo.t/ was nuked\n+\t\ttest_path_is_file foo.t &&\n+\t\ttest_cmp expect foo.t\n+\t)\n+'\n+\n+test_expect_success 'reset --merge will preserve untracked files/dirs' '\n+\ttest_setup_reset merge &&\n+\t(\n+\t\tcd reset_merge &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating .foo.t. would lose untracked files\" error\n+\t)\n+'\n+\n+test_expect_success 'reset --keep will preserve untracked files/dirs' '\n+\ttest_setup_reset keep &&\n+\t(\n+\t\tcd reset_keep &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n+\t)\n+'\n+\n+test_setup_checkout_m () {\n+\tgit init checkout &&\n+\t(\n+\t\tcd checkout &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines file has some >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit branch stable &&\n+\n+\t\tgit switch -c work &&\n+\t\techo stuff >notes.txt &&\n+\t\ttest_write_lines file has some words >filler &&\n+\t\tgit add notes.txt filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_failure 'checkout -m does not nuke untracked file' '\n+\ttest_setup_checkout_m &&\n+\t(\n+\t\tcd checkout &&\n+\n+\t\t# Tweak filler\n+\t\ttest_write_lines this file has some >filler &&\n+\t\t# Make an untracked file, save its contents in \"expect\"\n+\t\techo precious >notes.txt &&\n+\t\tcp notes.txt expect &&\n+\n+\t\ttest_must_fail git checkout -m work &&\n+\t\ttest_cmp expect notes.txt\n+\t)\n+'\n+\n+test_setup_sequencing () {\n+\tgit init sequencing_$1 &&\n+\t(\n+\t\tcd sequencing_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines this file has some words >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tmkdir -p foo/bar &&\n+\t\ttest_commit foo/bar/baz &&\n+\n+\t\tgit branch simple &&\n+\t\tgit branch fooey &&\n+\n+\t\tgit checkout fooey &&\n+\t\tgit rm foo/bar/baz.t &&\n+\t\techo stuff >>filler &&\n+\t\tgit add -u &&\n+\t\tgit commit -m \"changes\" &&\n+\n+\t\tgit checkout simple &&\n+\t\techo items >>filler &&\n+\t\techo newstuff >>newfile &&\n+\t\tgit add filler newfile &&\n+\t\tgit commit -m another\n+\t)\n+}\n+\n+test_expect_failure 'git rebase --abort and untracked files' '\n+\ttest_setup_sequencing rebase_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_abort_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\ttest_must_fail git rebase simple &&\n+\n+\t\tcat init.t &&\n+\t\tgit rm init.t &&\n+\t\techo precious >init.t &&\n+\t\tcp init.t expect &&\n+\t\tgit status --porcelain &&\n+\t\ttest_must_fail git rebase --abort &&\n+\t\ttest_cmp expect init.t\n+\t)\n+'\n+\n+test_expect_failure 'git rebase fast forwarding and untracked files' '\n+\ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_fast_forward_and_untracked &&\n+\t\tgit checkout init &&\n+\t\techo precious >filler &&\n+\t\tcp filler expect &&\n+\t\ttest_must_fail git rebase init simple &&\n+\t\ttest_cmp expect filler\n+\t)\n+'\n+\n+test_expect_failure 'git rebase --autostash and untracked files' '\n+\ttest_setup_sequencing rebase_autostash_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_autostash_and_untracked &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit rebase --autostash init &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git stash and untracked files' '\n+\ttest_setup_sequencing stash_and_untracked_files &&\n+\t(\n+\t\tcd sequencing_stash_and_untracked_files &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit status --porcelain &&\n+\t\tgit stash push &&\n+\t\tgit status --porcelain &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+\ttest_setup_sequencing am_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_abort_and_untracked &&\n+\t\tgit format-patch -1 --stdout fooey >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete the conflicted file; we will stage and commit it later\n+\t\trm filler &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir filler &&\n+\t\techo foo >filler/file1 &&\n+\t\techo bar >filler/file2 &&\n+\n+\t\ttest_must_fail git am --abort 2>errors &&\n+\t\ttest_path_is_dir filler &&\n+\t\tgrep \"Updating .filler. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+\ttest_setup_sequencing am_skip_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_skip_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\tgit format-patch -1 --stdout simple >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete newfile\n+\t\trm newfile &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir newfile &&\n+\t\techo foo >newfile/file1 &&\n+\t\techo bar >newfile/file2 &&\n+\n+\t\t# Change our mind about resolutions, just skip this patch\n+\t\ttest_must_fail git am --skip 2>errors &&\n+\t\ttest_path_is_dir newfile &&\n+\t\tgrep \"Updating .newfile. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"437155","messageId":"b7fe354effff8da3de53bd9cc40a03b5fd455f67.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 06/11] Remove ignored files by default when they are in the way","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:43Z","receivedAt":"2021-09-27T16:34:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nChange several commands to remove ignored files by default when they are\nin the way. Since some commands (checkout, merge) take a\n--no-overwrite-ignore option to allow the user to configure this, and it\nmay make sense to add that option to more commands (and in the case of\nmerge, actually plumb that configuration option through to more of the\nbackends than just the fast-forwarding special case), add little\ncomments about where such flags would be used.\n\nIncidentally, this fixes a test failure in t7112.\n\nSigned-off-by: Elijah Newren \n---\n builtin/am.c | 3 +--\n builtin/clone.c | 3 +--\n builtin/merge.c | 3 +--\n builtin/reset.c | 3 +--\n builtin/stash.c | 3 +--\n merge-ort.c | 2 +-\n reset.c | 3 +--\n sequencer.c | 3 +--\n t/t7112-reset-submodule.sh | 1 -\n 9 files changed, 8 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 1ee70692bc3..57738eff0c5 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1919,8 +1919,7 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.merge = 1;\n \topts.reset = reset;\n \tif (!reset)\n-\t\t/* FIXME: Default should be to remove ignored files */\n-\t\topts.preserve_ignored = 1;\n+\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = twoway_merge;\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex be1c3840d62..11ec6c5f2c8 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -687,8 +687,7 @@ static int checkout(int submodule_progress)\n \topts.update = 1;\n \topts.merge = 1;\n \topts.clone = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\topts.preserve_ignored = 1;\n+\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = (option_verbosity >= 0);\n \topts.src_index = &the_index;\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 1e5fff095fc..0ccd5e1ac83 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -680,8 +680,7 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n \topts.verbose_update = 1;\n \topts.trivial_merges_only = 1;\n \topts.merge = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\topts.preserve_ignored = 1;\n+\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \ttrees[nr_trees] = parse_tree_indirect(common);\n \tif (!trees[nr_trees++])\n \t\treturn -1;\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 7f38656f018..5df01cc42e0 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -67,8 +67,7 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \tcase KEEP:\n \tcase MERGE:\n \t\topts.update = 1;\n-\t\t/* FIXME: Default should be to remove ignored files */\n-\t\topts.preserve_ignored = 1;\n+\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 88287b890d5..d60cdaf32f5 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -259,8 +259,7 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.reset = reset;\n \topts.update = update;\n \tif (update && !reset)\n-\t\t/* FIXME: Default should be to remove ignored files */\n-\t\topts.preserve_ignored = 1;\n+\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = oneway_merge;\n \n \tif (unpack_trees(nr_trees, t, &opts))\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 0d64ec716bd..04596b5e7b3 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -4045,7 +4045,7 @@ static int checkout(struct merge_options *opt,\n \tunpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n \tunpack_opts.verbose_update = (opt->verbosity > 2);\n \tunpack_opts.fn = twoway_merge;\n-\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n+\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore */\n \tparse_tree(prev);\n \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n \tparse_tree(next);\ndiff --git a/reset.c b/reset.c\nindex 41b3e2d88de..f40a8ecf663 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -56,8 +56,7 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n \tunpack_tree_opts.update = 1;\n \tunpack_tree_opts.merge = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\tunpack_tree_opts.preserve_ignored = 1;\n+\tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n \tif (!detach_head)\n \t\tunpack_tree_opts.reset = 1;\ndiff --git a/sequencer.c b/sequencer.c\nindex 098566c68d9..6872b7b00a4 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3699,8 +3699,7 @@ static int do_reset(struct repository *r,\n \tunpack_tree_opts.fn = oneway_merge;\n \tunpack_tree_opts.merge = 1;\n \tunpack_tree_opts.update = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\tunpack_tree_opts.preserve_ignored = 1;\n+\tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n \n \tif (repo_read_index_unmerged(r)) {\ndiff --git a/t/t7112-reset-submodule.sh b/t/t7112-reset-submodule.sh\nindex 19830d90365..a3e2413bc33 100755\n--- a/t/t7112-reset-submodule.sh\n+++ b/t/t7112-reset-submodule.sh\n@@ -6,7 +6,6 @@ test_description='reset can handle submodules'\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \n KNOWN_FAILURE_DIRECTORY_SUBMODULE_CONFLICTS=1\n-KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1\n \n test_submodule_switch_recursing_with_args \"reset --keep\"\n \n-- \ngitgitgadget\n\n"},{"id":"437156","messageId":"0d119142778dce8617dd9b2c102b5f5bfdc9dc0f.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 05/11] unpack-trees: make dir an internal-only struct","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:42Z","receivedAt":"2021-09-27T16:34:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAvoid accidental misuse or confusion over ownership by clearly making\nunpack_trees_options.dir an internal-only variable.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 7 +++++--\n unpack-trees.h | 2 +-\n 2 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1e4eae1dc7d..e067cce0fcd 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1694,9 +1694,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n+\tstruct dir_struct dir = DIR_INIT;\n \n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n+\tif (o->dir)\n+\t\tBUG(\"o->dir is for internal use only\");\n \n \ttrace_performance_enter();\n \ttrace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n@@ -1708,7 +1711,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t}\n \n \tif (!o->preserve_ignored) {\n-\t\tCALLOC_ARRAY(o->dir, 1);\n+\t\to->dir = &dir;\n \t\to->dir->flags |= DIR_SHOW_IGNORED;\n \t\tsetup_standard_excludes(o->dir);\n \t}\n@@ -1876,7 +1879,7 @@ done:\n \t\tclear_pattern_list(&pl);\n \tif (o->dir) {\n \t\tdir_clear(o->dir);\n-\t\tFREE_AND_NULL(o->dir);\n+\t\to->dir = NULL;\n \t}\n \ttrace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n \ttrace_performance_leave(\"unpack_trees\");\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex f98cfd49d7b..61da25dafee 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -67,7 +67,6 @@ struct unpack_trees_options {\n \t\t dry_run;\n \tconst char *prefix;\n \tint cache_bottom;\n-\tstruct dir_struct *dir;\n \tstruct pathspec *pathspec;\n \tmerge_fn_t fn;\n \tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n@@ -89,6 +88,7 @@ struct unpack_trees_options {\n \tstruct index_state result;\n \n \tstruct pattern_list *pl; /* for internal use */\n+\tstruct dir_struct *dir; /* for internal use only */\n \tstruct checkout_metadata meta;\n };\n \n-- \ngitgitgadget\n\n"},{"id":"437157","messageId":"f1a0700e598e52d6cdb507fe8a09c4fa9291c982.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:41Z","receivedAt":"2021-09-27T16:34:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCurrently, every caller of unpack_trees() that wants to ensure ignored\nfiles are overwritten by default needs to:\n * allocate unpack_trees_options.dir\n * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir->flags\n * call setup_standard_excludes\nAND then after the call to unpack_trees() needs to\n * call dir_clear()\n * deallocate unpack_trees_options.dir\nThat's a fair amount of boilerplate, and every caller uses identical\ncode. Make this easier by instead introducing a new boolean value where\nthe default value (0) does what we want so that new callers of\nunpack_trees() automatically get the appropriate behavior. And move all\nthe handling of unpack_trees_options.dir into unpack_trees() itself.\n\nWhile preserve_ignored = 0 is the behavior we feel is the appropriate\ndefault, we defer fixing commands to use the appropriate default until a\nlater commit. So, this commit introduces several locations where we\nmanually set preserve_ignored=1. This makes it clear where code paths\nwere previously preserving ignored files when they should not have been;\na future commit will flip these to instead use a value of 0 to get the\nbehavior we want.\n\nSigned-off-by: Elijah Newren \n---\n builtin/am.c | 3 +++\n builtin/checkout.c | 11 ++---------\n builtin/clone.c | 2 ++\n builtin/merge.c | 2 ++\n builtin/read-tree.c | 13 +++----------\n builtin/reset.c | 2 ++\n builtin/stash.c | 3 +++\n merge-ort.c | 8 +-------\n merge-recursive.c | 8 +-------\n merge.c | 8 +-------\n reset.c | 2 ++\n sequencer.c | 2 ++\n unpack-trees.c | 10 ++++++++++\n unpack-trees.h | 1 +\n 14 files changed, 35 insertions(+), 40 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex e4a0ff9cd7c..1ee70692bc3 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1918,6 +1918,9 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.update = 1;\n \topts.merge = 1;\n \topts.reset = reset;\n+\tif (!reset)\n+\t\t/* FIXME: Default should be to remove ignored files */\n+\t\topts.preserve_ignored = 1;\n \topts.fn = twoway_merge;\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 5335435d616..5e7957dd068 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -648,6 +648,7 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n \topts.skip_unmerged = !worktree;\n \topts.reset = 1;\n \topts.merge = 1;\n+\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\n@@ -746,11 +747,7 @@ static int merge_working_tree(const struct checkout_opts *opts,\n \t\t\t\t new_branch_info->commit ?\n \t\t\t\t &new_branch_info->commit->object.oid :\n \t\t\t\t &new_branch_info->oid, NULL);\n-\t\tif (opts->overwrite_ignore) {\n-\t\t\ttopts.dir = xcalloc(1, sizeof(*topts.dir));\n-\t\t\ttopts.dir->flags |= DIR_SHOW_IGNORED;\n-\t\t\tsetup_standard_excludes(topts.dir);\n-\t\t}\n+\t\ttopts.preserve_ignored = !opts->overwrite_ignore;\n \t\ttree = parse_tree_indirect(old_branch_info->commit ?\n \t\t\t\t\t &old_branch_info->commit->object.oid :\n \t\t\t\t\t the_hash_algo->empty_tree);\n@@ -760,10 +757,6 @@ static int merge_working_tree(const struct checkout_opts *opts,\n \t\tinit_tree_desc(&trees[1], tree->buffer, tree->size);\n \n \t\tret = unpack_trees(2, trees, &topts);\n-\t\tif (topts.dir) {\n-\t\t\tdir_clear(topts.dir);\n-\t\t\tFREE_AND_NULL(topts.dir);\n-\t\t}\n \t\tclear_unpack_trees_porcelain(&topts);\n \t\tif (ret == -1) {\n \t\t\t/*\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex ff1d3d447a3..be1c3840d62 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -687,6 +687,8 @@ static int checkout(int submodule_progress)\n \topts.update = 1;\n \topts.merge = 1;\n \topts.clone = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\topts.preserve_ignored = 1;\n \topts.fn = oneway_merge;\n \topts.verbose_update = (option_verbosity >= 0);\n \topts.src_index = &the_index;\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 3fbdacc7db4..1e5fff095fc 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -680,6 +680,8 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n \topts.verbose_update = 1;\n \topts.trivial_merges_only = 1;\n \topts.merge = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\topts.preserve_ignored = 1;\n \ttrees[nr_trees] = parse_tree_indirect(common);\n \tif (!trees[nr_trees++])\n \t\treturn -1;\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 73cb957a69b..443d206eca6 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -201,11 +201,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif ((opts.update || opts.index_only) && !opts.merge)\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n-\tif (opts.update && !opts.reset) {\n-\t\tCALLOC_ARRAY(opts.dir, 1);\n-\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(opts.dir);\n-\t}\n+\tif (opts.update && !opts.reset)\n+\t\topts.preserve_ignored = 0;\n+\t/* otherwise, opts.preserve_ignored is irrelevant */\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n@@ -245,11 +243,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (unpack_trees(nr_trees, t, &opts))\n \t\treturn 128;\n \n-\tif (opts.dir) {\n-\t\tdir_clear(opts.dir);\n-\t\tFREE_AND_NULL(opts.dir);\n-\t}\n-\n \tif (opts.debug_unpack || opts.dry_run)\n \t\treturn 0; /* do not write the index out */\n \ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..7f38656f018 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -67,6 +67,8 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \tcase KEEP:\n \tcase MERGE:\n \t\topts.update = 1;\n+\t\t/* FIXME: Default should be to remove ignored files */\n+\t\topts.preserve_ignored = 1;\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 8f42360ca91..88287b890d5 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -258,6 +258,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.merge = 1;\n \topts.reset = reset;\n \topts.update = update;\n+\tif (update && !reset)\n+\t\t/* FIXME: Default should be to remove ignored files */\n+\t\topts.preserve_ignored = 1;\n \topts.fn = oneway_merge;\n \n \tif (unpack_trees(nr_trees, t, &opts))\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 35aa979c3a4..0d64ec716bd 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -4045,11 +4045,7 @@ static int checkout(struct merge_options *opt,\n \tunpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n \tunpack_opts.verbose_update = (opt->verbosity > 2);\n \tunpack_opts.fn = twoway_merge;\n-\tif (1/* FIXME: opts->overwrite_ignore*/) {\n-\t\tCALLOC_ARRAY(unpack_opts.dir, 1);\n-\t\tunpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(unpack_opts.dir);\n-\t}\n+\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n \tparse_tree(prev);\n \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n \tparse_tree(next);\n@@ -4057,8 +4053,6 @@ static int checkout(struct merge_options *opt,\n \n \tret = unpack_trees(2, trees, &unpack_opts);\n \tclear_unpack_trees_porcelain(&unpack_opts);\n-\tdir_clear(unpack_opts.dir);\n-\tFREE_AND_NULL(unpack_opts.dir);\n \treturn ret;\n }\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 233d9f686ad..2be3f5d4044 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -411,9 +411,7 @@ static int unpack_trees_start(struct merge_options *opt,\n \telse {\n \t\topt->priv->unpack_opts.update = 1;\n \t\t/* FIXME: should only do this if !overwrite_ignore */\n-\t\tCALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n-\t\topt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(opt->priv->unpack_opts.dir);\n+\t\topt->priv->unpack_opts.preserve_ignored = 0;\n \t}\n \topt->priv->unpack_opts.merge = 1;\n \topt->priv->unpack_opts.head_idx = 2;\n@@ -428,10 +426,6 @@ static int unpack_trees_start(struct merge_options *opt,\n \tinit_tree_desc_from_tree(t+2, merge);\n \n \trc = unpack_trees(3, t, &opt->priv->unpack_opts);\n-\tif (opt->priv->unpack_opts.dir) {\n-\t\tdir_clear(opt->priv->unpack_opts.dir);\n-\t\tFREE_AND_NULL(opt->priv->unpack_opts.dir);\n-\t}\n \tcache_tree_free(&opt->repo->index->cache_tree);\n \n \t/*\ndiff --git a/merge.c b/merge.c\nindex 6e736881d90..2382ff66d35 100644\n--- a/merge.c\n+++ b/merge.c\n@@ -53,7 +53,6 @@ int checkout_fast_forward(struct repository *r,\n \tstruct unpack_trees_options opts;\n \tstruct tree_desc t[MAX_UNPACK_TREES];\n \tint i, nr_trees = 0;\n-\tstruct dir_struct dir = DIR_INIT;\n \tstruct lock_file lock_file = LOCK_INIT;\n \n \trefresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL);\n@@ -80,11 +79,7 @@ int checkout_fast_forward(struct repository *r,\n \t}\n \n \tmemset(&opts, 0, sizeof(opts));\n-\tif (overwrite_ignore) {\n-\t\tdir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&dir);\n-\t\topts.dir = &dir;\n-\t}\n+\topts.preserve_ignored = !overwrite_ignore;\n \n \topts.head_idx = 1;\n \topts.src_index = r->index;\n@@ -101,7 +96,6 @@ int checkout_fast_forward(struct repository *r,\n \t\tclear_unpack_trees_porcelain(&opts);\n \t\treturn -1;\n \t}\n-\tdir_clear(&dir);\n \tclear_unpack_trees_porcelain(&opts);\n \n \tif (write_locked_index(r->index, &lock_file, COMMIT_LOCK))\ndiff --git a/reset.c b/reset.c\nindex 79310ae071b..41b3e2d88de 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -56,6 +56,8 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n \tunpack_tree_opts.update = 1;\n \tunpack_tree_opts.merge = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\tunpack_tree_opts.preserve_ignored = 1;\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n \tif (!detach_head)\n \t\tunpack_tree_opts.reset = 1;\ndiff --git a/sequencer.c b/sequencer.c\nindex 614d56f5e21..098566c68d9 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3699,6 +3699,8 @@ static int do_reset(struct repository *r,\n \tunpack_tree_opts.fn = oneway_merge;\n \tunpack_tree_opts.merge = 1;\n \tunpack_tree_opts.update = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\tunpack_tree_opts.preserve_ignored = 1;\n \tinit_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n \n \tif (repo_read_index_unmerged(r)) {\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..1e4eae1dc7d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1707,6 +1707,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tensure_full_index(o->dst_index);\n \t}\n \n+\tif (!o->preserve_ignored) {\n+\t\tCALLOC_ARRAY(o->dir, 1);\n+\t\to->dir->flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(o->dir);\n+\t}\n+\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n@@ -1868,6 +1874,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n done:\n \tif (free_pattern_list)\n \t\tclear_pattern_list(&pl);\n+\tif (o->dir) {\n+\t\tdir_clear(o->dir);\n+\t\tFREE_AND_NULL(o->dir);\n+\t}\n \ttrace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n \ttrace_performance_leave(\"unpack_trees\");\n \treturn ret;\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 2d88b19dca7..f98cfd49d7b 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -49,6 +49,7 @@ struct unpack_trees_options {\n \tunsigned int reset,\n \t\t merge,\n \t\t update,\n+\t\t preserve_ignored,\n \t\t clone,\n \t\t index_only,\n \t\t nontrivial_merge,\n-- \ngitgitgadget\n\n"},{"id":"437158","messageId":"9eb20121fc3886436cef91670e32217466d03ba7.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 07/11] Change unpack_trees' 'reset' flag into an enum","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:44Z","receivedAt":"2021-09-27T16:34:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nTraditionally, unpack_trees_options->reset was used to signal that it\nwas okay to delete any untracked files in the way. This was used by\n`git read-tree --reset`, but then started appearing in other places as\nwell. However, many of the other uses should not be deleting untracked\nfiles in the way. Change this value to an enum so that a value of 1\n(i.e. \"true\") can be split into two:\n UNPACK_RESET_PROTECT_UNTRACKED,\n UNPACK_RESET_OVERWRITE_UNTRACKED\nIn order to catch accidental misuses (i.e. where folks call it the way\nthey traditionally used to), define the special enum value of\n UNPACK_RESET_INVALID = 1\nwhich will trigger a BUG().\n\nModify existing callers so that\n read-tree --reset\n reset --hard\n checkout --force\ncontinue using the UNPACK_RESET_OVERWRITE_UNTRACKED logic, while other\ncallers, including\n am\n checkout without --force\n stash (though currently dead code; reset always had a value of 0)\n numerous callers from rebase/sequencer to reset_head()\nwill use the new UNPACK_RESET_PROTECT_UNTRACKED value.\n\nAlso, note that it has been reported that 'git checkout \n' currently also allows overwriting untracked files[1]. That\ncase should also be fixed, but it does not use unpack_trees() and thus\nis outside the scope of the current changes.\n\n[1] https://lore.kernel.org/git/15dad590-087e-5a48-9238-5d2826950506@gmail.com/\n\nSigned-off-by: Elijah Newren \n---\n builtin/am.c | 5 ++---\n builtin/checkout.c | 5 +++--\n builtin/read-tree.c | 3 +++\n builtin/reset.c | 9 +++++++--\n builtin/stash.c | 4 ++--\n reset.c | 2 +-\n t/t2500-untracked-overwriting.sh | 6 +++---\n unpack-trees.c | 10 +++++++++-\n unpack-trees.h | 11 +++++++++--\n 9 files changed, 39 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 57738eff0c5..f296226e95f 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1917,9 +1917,8 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.dst_index = &the_index;\n \topts.update = 1;\n \topts.merge = 1;\n-\topts.reset = reset;\n-\tif (!reset)\n-\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n+\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n+\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = twoway_merge;\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 5e7957dd068..cbf73b8c9f6 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -646,9 +646,10 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n \topts.head_idx = -1;\n \topts.update = worktree;\n \topts.skip_unmerged = !worktree;\n-\topts.reset = 1;\n+\topts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n+\t\t\t\tUNPACK_RESET_PROTECT_UNTRACKED;\n+\topts.preserve_ignored = (!o->force && !o->overwrite_ignore);\n \topts.merge = 1;\n-\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 443d206eca6..2109c4c9e5c 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -166,6 +166,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n+\tif (opts.reset)\n+\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n+\n \t/*\n \t * NEEDSWORK\n \t *\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 5df01cc42e0..73935953494 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -71,9 +71,14 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\n-\t\t/* fallthrough */\n+\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n+\t\tbreak;\n+\tcase MIXED:\n+\t\topts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n+\t\t/* but opts.update=0, so working tree not updated */\n+\t\tbreak;\n \tdefault:\n-\t\topts.reset = 1;\n+\t\tBUG(\"invalid reset_type passed to reset_index\");\n \t}\n \n \tread_cache_unmerged();\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex d60cdaf32f5..0e3662a230c 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -256,9 +256,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n \topts.merge = 1;\n-\topts.reset = reset;\n+\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n \topts.update = update;\n-\tif (update && !reset)\n+\tif (update)\n \t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = oneway_merge;\n \ndiff --git a/reset.c b/reset.c\nindex f40a8ecf663..f214df3d96c 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -59,7 +59,7 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n \tif (!detach_head)\n-\t\tunpack_tree_opts.reset = 1;\n+\t\tunpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n \n \tif (repo_read_index_unmerged(r) < 0) {\n \t\tret = error(_(\"could not read index\"));\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 2412d121ea8..18604360df8 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -92,7 +92,7 @@ test_setup_checkout_m () {\n \t)\n }\n \n-test_expect_failure 'checkout -m does not nuke untracked file' '\n+test_expect_success 'checkout -m does not nuke untracked file' '\n \ttest_setup_checkout_m &&\n \t(\n \t\tcd checkout &&\n@@ -138,7 +138,7 @@ test_setup_sequencing () {\n \t)\n }\n \n-test_expect_failure 'git rebase --abort and untracked files' '\n+test_expect_success 'git rebase --abort and untracked files' '\n \ttest_setup_sequencing rebase_abort_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_abort_and_untracked &&\n@@ -155,7 +155,7 @@ test_expect_failure 'git rebase --abort and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git rebase fast forwarding and untracked files' '\n+test_expect_success 'git rebase fast forwarding and untracked files' '\n \ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_fast_forward_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e067cce0fcd..812e4c66713 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1696,6 +1696,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tint free_pattern_list = 0;\n \tstruct dir_struct dir = DIR_INIT;\n \n+\tif (o->reset == UNPACK_RESET_INVALID)\n+\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n+\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \tif (o->dir)\n@@ -1710,6 +1713,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tensure_full_index(o->dst_index);\n \t}\n \n+\tif (o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED &&\n+\t o->preserve_ignored)\n+\t\tBUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n+\n \tif (!o->preserve_ignored) {\n \t\to->dir = &dir;\n \t\to->dir->flags |= DIR_SHOW_IGNORED;\n@@ -2233,7 +2240,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n \n-\tif (o->index_only || o->reset || !o->update)\n+\tif (o->index_only || !o->update ||\n+\t o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED)\n \t\treturn 0;\n \n \tlen = check_leading_path(ce->name, ce_namelen(ce), 0);\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 61da25dafee..71ffb7eeb0c 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -45,9 +45,15 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n */\n void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n \n+enum unpack_trees_reset_type {\n+\tUNPACK_RESET_NONE = 0, /* traditional \"false\" value; still valid */\n+\tUNPACK_RESET_INVALID = 1, /* \"true\" no longer valid; use below values */\n+\tUNPACK_RESET_PROTECT_UNTRACKED,\n+\tUNPACK_RESET_OVERWRITE_UNTRACKED\n+};\n+\n struct unpack_trees_options {\n-\tunsigned int reset,\n-\t\t merge,\n+\tunsigned int merge,\n \t\t update,\n \t\t preserve_ignored,\n \t\t clone,\n@@ -65,6 +71,7 @@ struct unpack_trees_options {\n \t\t exiting_early,\n \t\t show_all_errors,\n \t\t dry_run;\n+\tenum unpack_trees_reset_type reset;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct pathspec *pathspec;\n-- \ngitgitgadget\n\n"},{"id":"437159","messageId":"e4c42d43b094070ae635431ed3d0903722f42039.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 08/11] unpack-trees: avoid nuking untracked dir in way of unmerged file","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:45Z","receivedAt":"2021-09-27T16:34:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 35 ++++++++++++++++++++++++++++----\n 2 files changed, 32 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 18604360df8..5ec66058cfc 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -197,7 +197,7 @@ test_expect_failure 'git stash and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \ttest_setup_sequencing am_abort_and_untracked &&\n \t(\n \t\tcd sequencing_am_abort_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 812e4c66713..080118f2325 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2178,9 +2178,15 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n+enum absent_checking_type {\n+\tCOMPLETELY_ABSENT,\n+\tABSENT_ANY_DIRECTORY\n+};\n+\n static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\t\t const struct cache_entry *ce, struct stat *st,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tconst struct cache_entry *result;\n@@ -2215,6 +2221,10 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \t}\n \n+\t/* If we only care about directories, then we can remove */\n+\tif (absent_type == ABSENT_ANY_DIRECTORY)\n+\t\treturn 0;\n+\n \t/*\n \t * The previous round may already have decided to\n \t * delete this path, which is in a subdirectory that\n@@ -2235,6 +2245,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n */\n static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tint len;\n@@ -2261,7 +2272,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t\t\t\t\t\tNULL, o);\n \t\t\telse\n \t\t\t\tret = check_ok_to_remove(path, len, DT_UNKNOWN, NULL,\n-\t\t\t\t\t\t\t &st, error_type, o);\n+\t\t\t\t\t\t\t &st, error_type,\n+\t\t\t\t\t\t\t absent_type, o);\n \t\t}\n \t\tfree(path);\n \t\treturn ret;\n@@ -2276,7 +2288,7 @@ static int verify_absent_1(const struct cache_entry *ce,\n \n \t\treturn check_ok_to_remove(ce->name, ce_namelen(ce),\n \t\t\t\t\t ce_to_dtype(ce), ce, &st,\n-\t\t\t\t\t error_type, o);\n+\t\t\t\t\t error_type, absent_type, o);\n \t}\n }\n \n@@ -2286,14 +2298,23 @@ static int verify_absent(const struct cache_entry *ce,\n {\n \tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n+}\n+\n+static int verify_absent_if_directory(const struct cache_entry *ce,\n+\t\t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t\t struct unpack_trees_options *o)\n+{\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\t\treturn 0;\n+\treturn verify_absent_1(ce, error_type, ABSENT_ANY_DIRECTORY, o);\n }\n \n static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types error_type,\n \t\t\t\tstruct unpack_trees_options *o)\n {\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n }\n \n static int merged_entry(const struct cache_entry *ce,\n@@ -2367,6 +2388,12 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t * Previously unmerged entry left as an existence\n \t\t * marker by read_index_unmerged();\n \t\t */\n+\t\tif (verify_absent_if_directory(merge,\n+\t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n+\t\t\tdiscard_cache_entry(merge);\n+\t\t\treturn -1;\n+\t\t}\n+\n \t\tinvalidate_ce_path(old, o);\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"437160","messageId":"1a770681704691e32722f1abbe292e8523c7f455.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 09/11] unpack-trees: avoid nuking untracked dir in way of locally deleted file","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:46Z","receivedAt":"2021-09-27T16:34:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 3 +++\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 5ec66058cfc..5c0bf4d21fc 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -218,7 +218,7 @@ test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \t)\n '\n \n-test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+test_expect_success 'git am --skip and untracked dir vs deleted file' '\n \ttest_setup_sequencing am_skip_and_untracked &&\n \t(\n \t\tcd sequencing_am_skip_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 080118f2325..a7e1712d236 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2411,7 +2411,10 @@ static int deleted_entry(const struct cache_entry *ce,\n \t\tif (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n \t\t\treturn -1;\n \t\treturn 0;\n+\t} else if (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o)) {\n+\t\treturn -1;\n \t}\n+\n \tif (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))\n \t\treturn -1;\n \tadd_entry(o, ce, CE_REMOVE, 0);\n-- \ngitgitgadget\n\n"},{"id":"437161","messageId":"6b42a80bf3d46e16980d0724e8b07101225239d0.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 10/11] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:47Z","receivedAt":"2021-09-27T16:34:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last few commits we focused on code in unpack-trees.c that\nmistakenly removed untracked files or directories. There may be more of\nthose, but in this commit we change our focus: callers of toplevel\ncommands that are expected to remove untracked files or directories.\n\nAs noted previously, we have toplevel commands that are expected to\ndelete untracked files such as 'read-tree --reset', 'reset --hard', and\n'checkout --force'. However, that does not mean that other highlevel\ncommands that happen to call these other commands thought about or\nconveyed to users the possibility that untracked files could be removed.\nAudit the code for such callsites, and add comments near existing\ncallsites to mention whether these are safe or not.\n\nMy auditing is somewhat incomplete, though; it skipped several cases:\n * git-rebase--preserve-merges.sh: is in the process of being\n deprecated/removed, so I won't leave a note that there are\n likely more bugs in that script.\n * contrib/git-new-workdir: why is the -f flag being used in a new\n empty directory?? It shouldn't hurt, but it seems useless.\n * git-p4.py: Don't see why -f is needed for a new dir (maybe it's\n not and is just superfluous), but I'm not at all familiar with\n the p4 stuff\n * git-archimport.perl: Don't care; arch is long since dead\n * git-cvs*.perl: Don't care; cvs is long since dead\n\nAlso, the reset --hard in builtin/worktree.c looks safe, due to only\nrunning in an empty directory.\n\nSigned-off-by: Elijah Newren \n---\n builtin/stash.c | 1 +\n builtin/submodule--helper.c | 4 ++++\n contrib/rerere-train.sh | 2 +-\n submodule.c | 1 +\n 4 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 0e3662a230c..aa31163a5a1 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1521,6 +1521,7 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\t} else {\n \t\t\tstruct child_process cp = CHILD_PROCESS_INIT;\n \t\t\tcp.git_cmd = 1;\n+\t\t\t/* BUG: this nukes untracked files in the way */\n \t\t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"-q\",\n \t\t\t\t \"--no-recurse-submodules\", NULL);\n \t\t\tif (run_command(&cp)) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 5336daf186d..e13f2a0bcd0 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -3090,6 +3090,10 @@ static int add_submodule(const struct add_data *add_data)\n \t\tprepare_submodule_repo_env(&cp.env_array);\n \t\tcp.git_cmd = 1;\n \t\tcp.dir = add_data->sm_path;\n+\t\t/*\n+\t\t * NOTE: we only get here if add_data->force is true, so\n+\t\t * passing --force to checkout is reasonable.\n+\t\t */\n \t\tstrvec_pushl(&cp.args, \"checkout\", \"-f\", \"-q\", NULL);\n \n \t\tif (add_data->branch) {\ndiff --git a/contrib/rerere-train.sh b/contrib/rerere-train.sh\nindex eeee45dd341..75125d6ae00 100755\n--- a/contrib/rerere-train.sh\n+++ b/contrib/rerere-train.sh\n@@ -91,7 +91,7 @@ do\n \t\tgit checkout -q $commit -- .\n \t\tgit rerere\n \tfi\n-\tgit reset -q --hard\n+\tgit reset -q --hard # Might nuke untracked files...\n done\n \n if test -z \"$branch\"\ndiff --git a/submodule.c b/submodule.c\nindex 78aed03d928..c8ba93cc708 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1908,6 +1908,7 @@ static void submodule_reset_index(const char *path)\n \n \tstrvec_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t get_super_prefix_or_empty(), path);\n+\t/* TODO: determine if this might overwright untracked files */\n \tstrvec_pushl(&cp.args, \"read-tree\", \"-u\", \"--reset\", NULL);\n \n \tstrvec_push(&cp.args, empty_tree_oid_hex());\n-- \ngitgitgadget\n\n"},{"id":"437162","messageId":"de416f887d7ce24f20ad3ad4cc838394d6523635.1632760428.git.gitgitgadget@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[PATCH v3 11/11] Documentation: call out commands that nuke untracked files/directories","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-27T16:33:48Z","receivedAt":"2021-09-27T16:34:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSome commands have traditionally also removed untracked files (or\ndirectories) that were in the way of a tracked file we needed. Document\nthese cases.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-checkout.txt | 5 +++--\n Documentation/git-read-tree.txt | 5 +++--\n Documentation/git-reset.txt | 3 ++-\n 3 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\nindex b1a6fe44997..d473c9bf387 100644\n--- a/Documentation/git-checkout.txt\n+++ b/Documentation/git-checkout.txt\n@@ -118,8 +118,9 @@ OPTIONS\n -f::\n --force::\n \tWhen switching branches, proceed even if the index or the\n-\tworking tree differs from `HEAD`. This is used to throw away\n-\tlocal changes.\n+\tworking tree differs from `HEAD`, and even if there are untracked\n+\tfiles in the way. This is used to throw away local changes and\n+\tany untracked files or directories that are in the way.\n +\n When checking out paths from the index, do not fail upon unmerged\n entries; instead, unmerged entries are ignored.\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 0222a27c5af..8c3aceb8324 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -38,8 +38,9 @@ OPTIONS\n \n --reset::\n \tSame as -m, except that unmerged entries are discarded instead\n-\tof failing. When used with `-u`, updates leading to loss of\n-\tworking tree changes will not abort the operation.\n+\tof failing. When used with `-u`, updates leading to loss of\n+\tworking tree changes or untracked files or directories will not\n+\tabort the operation.\n \n -u::\n \tAfter a successful merge, update the files in the work\ndiff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\nindex 252e2d4e47d..6f7685f53d5 100644\n--- a/Documentation/git-reset.txt\n+++ b/Documentation/git-reset.txt\n@@ -69,7 +69,8 @@ linkgit:git-add[1]).\n \n --hard::\n \tResets the index and working tree. Any changes to tracked files in the\n-\tworking tree since `` are discarded.\n+\tworking tree since `` are discarded. Any untracked files or\n+\tdirectories in the way of writing any tracked files are simply deleted.\n \n --merge::\n \tResets the index and updates the files in the working tree that are\n-- \ngitgitgadget\n"},{"id":"437187","messageId":"xmqq7df1pxgh.fsf@gitster.g","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Fix various issues around removal of untracked files/directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-27T20:36:46Z","receivedAt":"2021-09-27T20:36:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> Changes since v2 (all due to Junio's request to consolidate\n> unpack_trees_options.dir handling):\n\nHeh, don't blame me. I even explicitly said it was merely an\nobservation for longer term, not a suggestion to include the first\nstep for such a move in this series.\n"},{"id":"437188","messageId":"CABPp-BESmz+9vcd0r+5MT8vr7HodecUpLKNL=Hyx4qxLJv0ZCA@mail.gmail.com","threadId":"56534","inReplyTo":"xmqq7df1pxgh.fsf@gitster.g","subject":"Re: [PATCH v3 00/11] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-27T20:41:06Z","receivedAt":"2021-09-27T20:41:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Sep 27, 2021 at 1:36 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > Changes since v2 (all due to Junio's request to consolidate\n> > unpack_trees_options.dir handling):\n>\n> Heh, don't blame me. I even explicitly said it was merely an\n> observation for longer term, not a suggestion to include the first\n> step for such a move in this series.\n\nWell...the repetitive code for setting up and clearing out\nunpack_trees_options.dir that already existed (and which my series was\ncopying to more places) bugged me too, but I was worried that it was a\nbit messy to clean up (and the fact that it took five patches suggests\nit was). But then you also brought it up as an issue when reviewing,\nso I figured I might as well dive in...\n"},{"id":"437194","messageId":"CABPp-BEMeJ+-LPQigPQvS3Ez7nf2438rNK+dB5Q0xDHFwzY05w@mail.gmail.com","threadId":"56534","inReplyTo":"CABPp-BESmz+9vcd0r+5MT8vr7HodecUpLKNL=Hyx4qxLJv0ZCA@mail.gmail.com","subject":"Re: [PATCH v3 00/11] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-27T21:31:39Z","receivedAt":"2021-09-27T21:31:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Sep 27, 2021 at 1:41 PM Elijah Newren wrote:\n>\n> On Mon, Sep 27, 2021 at 1:36 PM Junio C Hamano wrote:\n> >\n> > \"Elijah Newren via GitGitGadget\" writes:\n> >\n> > > Changes since v2 (all due to Junio's request to consolidate\n> > > unpack_trees_options.dir handling):\n> >\n> > Heh, don't blame me. I even explicitly said it was merely an\n> > observation for longer term, not a suggestion to include the first\n> > step for such a move in this series.\n>\n> Well...the repetitive code for setting up and clearing out\n> unpack_trees_options.dir that already existed (and which my series was\n> copying to more places) bugged me too, but I was worried that it was a\n> bit messy to clean up (and the fact that it took five patches suggests\n> it was). But then you also brought it up as an issue when reviewing,\n> so I figured I might as well dive in...\n\nI guess I should add that some of your other review comments were\nrelated, e.g. your puzzlement/assumption that some of my changes\npreserved ignored files when untracked files were being overwritten\n(which was not what the patches actually did). Trying to make the\ncode clearer was in some ways easier by first consolidating all those\nother bits.\n"},{"id":"437448","messageId":"87ilyjviiy.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"f1a0700e598e52d6cdb507fe8a09c4fa9291c982.1632760428.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-29T09:22:24Z","receivedAt":"2021-09-29T09:27:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 27 2021, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren \n>\n> Currently, every caller of unpack_trees() that wants to ensure ignored\n> files are overwritten by default needs to:\n> * allocate unpack_trees_options.dir\n> * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir->flags\n> * call setup_standard_excludes\n> AND then after the call to unpack_trees() needs to\n> * call dir_clear()\n> * deallocate unpack_trees_options.dir\n> That's a fair amount of boilerplate, and every caller uses identical\n> code. Make this easier by instead introducing a new boolean value where\n> the default value (0) does what we want so that new callers of\n> unpack_trees() automatically get the appropriate behavior. And move all\n> the handling of unpack_trees_options.dir into unpack_trees() itself.\n>\n> While preserve_ignored = 0 is the behavior we feel is the appropriate\n> default, we defer fixing commands to use the appropriate default until a\n> later commit. So, this commit introduces several locations where we\n> manually set preserve_ignored=1. This makes it clear where code paths\n> were previously preserving ignored files when they should not have been;\n> a future commit will flip these to instead use a value of 0 to get the\n> behavior we want.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> builtin/am.c | 3 +++\n> builtin/checkout.c | 11 ++---------\n> builtin/clone.c | 2 ++\n> builtin/merge.c | 2 ++\n> builtin/read-tree.c | 13 +++----------\n> builtin/reset.c | 2 ++\n> builtin/stash.c | 3 +++\n> merge-ort.c | 8 +-------\n> merge-recursive.c | 8 +-------\n> merge.c | 8 +-------\n> reset.c | 2 ++\n> sequencer.c | 2 ++\n> unpack-trees.c | 10 ++++++++++\n> unpack-trees.h | 1 +\n> 14 files changed, 35 insertions(+), 40 deletions(-)\n>\n> diff --git a/builtin/am.c b/builtin/am.c\n> index e4a0ff9cd7c..1ee70692bc3 100644\n> --- a/builtin/am.c\n> +++ b/builtin/am.c\n> @@ -1918,6 +1918,9 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n> \topts.update = 1;\n> \topts.merge = 1;\n> \topts.reset = reset;\n> +\tif (!reset)\n> +\t\t/* FIXME: Default should be to remove ignored files */\n> +\t\topts.preserve_ignored = 1;\n> \topts.fn = twoway_merge;\n> \tinit_tree_desc(&t[0], head->buffer, head->size);\n> \tinit_tree_desc(&t[1], remote->buffer, remote->size);\n> diff --git a/builtin/checkout.c b/builtin/checkout.c\n> index 5335435d616..5e7957dd068 100644\n> --- a/builtin/checkout.c\n> +++ b/builtin/checkout.c\n> @@ -648,6 +648,7 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n> \topts.skip_unmerged = !worktree;\n> \topts.reset = 1;\n> \topts.merge = 1;\n> +\topts.preserve_ignored = 0;\n> \topts.fn = oneway_merge;\n> \topts.verbose_update = o->show_progress;\n> \topts.src_index = &the_index;\n> @@ -746,11 +747,7 @@ static int merge_working_tree(const struct checkout_opts *opts,\n> \t\t\t\t new_branch_info->commit ?\n> \t\t\t\t &new_branch_info->commit->object.oid :\n> \t\t\t\t &new_branch_info->oid, NULL);\n> -\t\tif (opts->overwrite_ignore) {\n> -\t\t\ttopts.dir = xcalloc(1, sizeof(*topts.dir));\n> -\t\t\ttopts.dir->flags |= DIR_SHOW_IGNORED;\n> -\t\t\tsetup_standard_excludes(topts.dir);\n> -\t\t}\n> +\t\ttopts.preserve_ignored = !opts->overwrite_ignore;\n> \t\ttree = parse_tree_indirect(old_branch_info->commit ?\n> \t\t\t\t\t &old_branch_info->commit->object.oid :\n> \t\t\t\t\t the_hash_algo->empty_tree);\n> @@ -760,10 +757,6 @@ static int merge_working_tree(const struct checkout_opts *opts,\n> \t\tinit_tree_desc(&trees[1], tree->buffer, tree->size);\n> \n> \t\tret = unpack_trees(2, trees, &topts);\n> -\t\tif (topts.dir) {\n> -\t\t\tdir_clear(topts.dir);\n> -\t\t\tFREE_AND_NULL(topts.dir);\n> -\t\t}\n> \t\tclear_unpack_trees_porcelain(&topts);\n> \t\tif (ret == -1) {\n> \t\t\t/*\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index ff1d3d447a3..be1c3840d62 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -687,6 +687,8 @@ static int checkout(int submodule_progress)\n> \topts.update = 1;\n> \topts.merge = 1;\n> \topts.clone = 1;\n> +\t/* FIXME: Default should be to remove ignored files */\n> +\topts.preserve_ignored = 1;\n> \topts.fn = oneway_merge;\n> \topts.verbose_update = (option_verbosity >= 0);\n> \topts.src_index = &the_index;\n> diff --git a/builtin/merge.c b/builtin/merge.c\n> index 3fbdacc7db4..1e5fff095fc 100644\n> --- a/builtin/merge.c\n> +++ b/builtin/merge.c\n> @@ -680,6 +680,8 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n> \topts.verbose_update = 1;\n> \topts.trivial_merges_only = 1;\n> \topts.merge = 1;\n> +\t/* FIXME: Default should be to remove ignored files */\n> +\topts.preserve_ignored = 1;\n> \ttrees[nr_trees] = parse_tree_indirect(common);\n> \tif (!trees[nr_trees++])\n> \t\treturn -1;\n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index 73cb957a69b..443d206eca6 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -201,11 +201,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> \tif ((opts.update || opts.index_only) && !opts.merge)\n> \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n> \t\t opts.update ? \"-u\" : \"-i\");\n> -\tif (opts.update && !opts.reset) {\n> -\t\tCALLOC_ARRAY(opts.dir, 1);\n> -\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n> -\t\tsetup_standard_excludes(opts.dir);\n> -\t}\n> +\tif (opts.update && !opts.reset)\n> +\t\topts.preserve_ignored = 0;\n> +\t/* otherwise, opts.preserve_ignored is irrelevant */\n> \tif (opts.merge && !opts.index_only)\n> \t\tsetup_work_tree();\n> \n> @@ -245,11 +243,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> \tif (unpack_trees(nr_trees, t, &opts))\n> \t\treturn 128;\n> \n> -\tif (opts.dir) {\n> -\t\tdir_clear(opts.dir);\n> -\t\tFREE_AND_NULL(opts.dir);\n> -\t}\n> -\n> \tif (opts.debug_unpack || opts.dry_run)\n> \t\treturn 0; /* do not write the index out */\n> \n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 51c9e2f43ff..7f38656f018 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -67,6 +67,8 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> \tcase KEEP:\n> \tcase MERGE:\n> \t\topts.update = 1;\n> +\t\t/* FIXME: Default should be to remove ignored files */\n> +\t\topts.preserve_ignored = 1;\n> \t\tbreak;\n> \tcase HARD:\n> \t\topts.update = 1;\n> diff --git a/builtin/stash.c b/builtin/stash.c\n> index 8f42360ca91..88287b890d5 100644\n> --- a/builtin/stash.c\n> +++ b/builtin/stash.c\n> @@ -258,6 +258,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n> \topts.merge = 1;\n> \topts.reset = reset;\n> \topts.update = update;\n> +\tif (update && !reset)\n> +\t\t/* FIXME: Default should be to remove ignored files */\n> +\t\topts.preserve_ignored = 1;\n> \topts.fn = oneway_merge;\n> \n> \tif (unpack_trees(nr_trees, t, &opts))\n> diff --git a/merge-ort.c b/merge-ort.c\n> index 35aa979c3a4..0d64ec716bd 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -4045,11 +4045,7 @@ static int checkout(struct merge_options *opt,\n> \tunpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n> \tunpack_opts.verbose_update = (opt->verbosity > 2);\n> \tunpack_opts.fn = twoway_merge;\n> -\tif (1/* FIXME: opts->overwrite_ignore*/) {\n> -\t\tCALLOC_ARRAY(unpack_opts.dir, 1);\n> -\t\tunpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n> -\t\tsetup_standard_excludes(unpack_opts.dir);\n> -\t}\n> +\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n> \tparse_tree(prev);\n> \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n> \tparse_tree(next);\n> @@ -4057,8 +4053,6 @@ static int checkout(struct merge_options *opt,\n> \n> \tret = unpack_trees(2, trees, &unpack_opts);\n> \tclear_unpack_trees_porcelain(&unpack_opts);\n> -\tdir_clear(unpack_opts.dir);\n> -\tFREE_AND_NULL(unpack_opts.dir);\n> \treturn ret;\n> }\n> \n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index 233d9f686ad..2be3f5d4044 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -411,9 +411,7 @@ static int unpack_trees_start(struct merge_options *opt,\n> \telse {\n> \t\topt->priv->unpack_opts.update = 1;\n> \t\t/* FIXME: should only do this if !overwrite_ignore */\n> -\t\tCALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n> -\t\topt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n> -\t\tsetup_standard_excludes(opt->priv->unpack_opts.dir);\n> +\t\topt->priv->unpack_opts.preserve_ignored = 0;\n> \t}\n> \topt->priv->unpack_opts.merge = 1;\n> \topt->priv->unpack_opts.head_idx = 2;\n> @@ -428,10 +426,6 @@ static int unpack_trees_start(struct merge_options *opt,\n> \tinit_tree_desc_from_tree(t+2, merge);\n> \n> \trc = unpack_trees(3, t, &opt->priv->unpack_opts);\n> -\tif (opt->priv->unpack_opts.dir) {\n> -\t\tdir_clear(opt->priv->unpack_opts.dir);\n> -\t\tFREE_AND_NULL(opt->priv->unpack_opts.dir);\n> -\t}\n> \tcache_tree_free(&opt->repo->index->cache_tree);\n> \n> \t/*\n> diff --git a/merge.c b/merge.c\n> index 6e736881d90..2382ff66d35 100644\n> --- a/merge.c\n> +++ b/merge.c\n> @@ -53,7 +53,6 @@ int checkout_fast_forward(struct repository *r,\n> \tstruct unpack_trees_options opts;\n> \tstruct tree_desc t[MAX_UNPACK_TREES];\n> \tint i, nr_trees = 0;\n> -\tstruct dir_struct dir = DIR_INIT;\n> \tstruct lock_file lock_file = LOCK_INIT;\n> \n> \trefresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL);\n> @@ -80,11 +79,7 @@ int checkout_fast_forward(struct repository *r,\n> \t}\n> \n> \tmemset(&opts, 0, sizeof(opts));\n> -\tif (overwrite_ignore) {\n> -\t\tdir.flags |= DIR_SHOW_IGNORED;\n> -\t\tsetup_standard_excludes(&dir);\n> -\t\topts.dir = &dir;\n> -\t}\n> +\topts.preserve_ignored = !overwrite_ignore;\n> \n> \topts.head_idx = 1;\n> \topts.src_index = r->index;\n> @@ -101,7 +96,6 @@ int checkout_fast_forward(struct repository *r,\n> \t\tclear_unpack_trees_porcelain(&opts);\n> \t\treturn -1;\n> \t}\n> -\tdir_clear(&dir);\n> \tclear_unpack_trees_porcelain(&opts);\n> \n> \tif (write_locked_index(r->index, &lock_file, COMMIT_LOCK))\n> diff --git a/reset.c b/reset.c\n> index 79310ae071b..41b3e2d88de 100644\n> --- a/reset.c\n> +++ b/reset.c\n> @@ -56,6 +56,8 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n> \tunpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n> \tunpack_tree_opts.update = 1;\n> \tunpack_tree_opts.merge = 1;\n> +\t/* FIXME: Default should be to remove ignored files */\n> +\tunpack_tree_opts.preserve_ignored = 1;\n> \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n> \tif (!detach_head)\n> \t\tunpack_tree_opts.reset = 1;\n> diff --git a/sequencer.c b/sequencer.c\n> index 614d56f5e21..098566c68d9 100644\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -3699,6 +3699,8 @@ static int do_reset(struct repository *r,\n> \tunpack_tree_opts.fn = oneway_merge;\n> \tunpack_tree_opts.merge = 1;\n> \tunpack_tree_opts.update = 1;\n> +\t/* FIXME: Default should be to remove ignored files */\n> +\tunpack_tree_opts.preserve_ignored = 1;\n> \tinit_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n> \n> \tif (repo_read_index_unmerged(r)) {\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 8ea0a542da8..1e4eae1dc7d 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1707,6 +1707,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \t\tensure_full_index(o->dst_index);\n> \t}\n> \n> +\tif (!o->preserve_ignored) {\n> +\t\tCALLOC_ARRAY(o->dir, 1);\n> +\t\to->dir->flags |= DIR_SHOW_IGNORED;\n> +\t\tsetup_standard_excludes(o->dir);\n> +\t}\n> +\n> \tif (!core_apply_sparse_checkout || !o->update)\n> \t\to->skip_sparse_checkout = 1;\n> \tif (!o->skip_sparse_checkout && !o->pl) {\n> @@ -1868,6 +1874,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> done:\n> \tif (free_pattern_list)\n> \t\tclear_pattern_list(&pl);\n> +\tif (o->dir) {\n> +\t\tdir_clear(o->dir);\n> +\t\tFREE_AND_NULL(o->dir);\n> +\t}\n> \ttrace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n> \ttrace_performance_leave(\"unpack_trees\");\n> \treturn ret;\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index 2d88b19dca7..f98cfd49d7b 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -49,6 +49,7 @@ struct unpack_trees_options {\n> \tunsigned int reset,\n> \t\t merge,\n> \t\t update,\n> +\t\t preserve_ignored,\n> \t\t clone,\n> \t\t index_only,\n> \t\t nontrivial_merge,\n\nI think getting rid of the boilerplate makes sense, but it doesn't sound\nfrom the commit message like you've considered just making that \"struct\ndir*\" member a \"struct dir\" instead.\n\nThat simplifies things a lot, i.e. we can just DIR_INIT it, and don't\nneed every caller to malloc/free it.\n\nSometimes a pointer makes sense, but in this case the \"struct\nunpack_trees_options\" can just own it.\n\nAs part of WIP leak fixes I have unsubmitted I'd implemented that, patch\nfollows below.\n\nI think the part of it that deals with managing the \"struct dir\" is much\nnicer, but you might still want to keep the \"preserve_ignored\" you've\nadded.\n\nOh, and I noticed I removed the dir_clear() here but didn't add it to\nclear_unpack_trees_porcelain(), that also needs to be done (and I did it\nin a later fix that I should squash in), but I can't be bothered to\nre-do the below diff just for that, and since the point is how we manage\nthe struct itself (the freeing is rather trivial...).\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 8c69dcdf72a..632da036717 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -747,9 +747,8 @@ static int merge_working_tree(const struct checkout_opts *opts,\n \t\t\t\t &new_branch_info->commit->object.oid :\n \t\t\t\t &new_branch_info->oid, NULL);\n \t\tif (opts->overwrite_ignore) {\n-\t\t\ttopts.dir = xcalloc(1, sizeof(*topts.dir));\n-\t\t\ttopts.dir->flags |= DIR_SHOW_IGNORED;\n-\t\t\tsetup_standard_excludes(topts.dir);\n+\t\t\ttopts.dir.flags |= DIR_SHOW_IGNORED;\n+\t\t\tsetup_standard_excludes(&topts.dir);\n \t\t}\n \t\ttree = parse_tree_indirect(old_branch_info->commit ?\n \t\t\t\t\t &old_branch_info->commit->object.oid :\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 485e7b04794..6d529c77c49 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -53,20 +53,17 @@ static int index_output_cb(const struct option *opt, const char *arg,\n static int exclude_per_directory_cb(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n {\n-\tstruct dir_struct *dir;\n \tstruct unpack_trees_options *opts;\n \n \tBUG_ON_OPT_NEG(unset);\n \n \topts = (struct unpack_trees_options *)opt->value;\n \n-\tif (opts->dir)\n+\tif (opts->dir.exclude_per_dir)\n \t\tdie(\"more than one --exclude-per-directory given.\");\n \n-\tdir = xcalloc(1, sizeof(*opts->dir));\n-\tdir->flags |= DIR_SHOW_IGNORED;\n-\tdir->exclude_per_dir = arg;\n-\topts->dir = dir;\n+\topts->dir.flags |= DIR_SHOW_IGNORED;\n+\topts->dir.exclude_per_dir = arg;\n \t/* We do not need to nor want to do read-directory\n \t * here; we are merely interested in reusing the\n \t * per directory ignore stack mechanism.\n@@ -209,7 +206,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif ((opts.update || opts.index_only) && !opts.merge)\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n-\tif ((opts.dir && !opts.update))\n+\tif ((opts.dir.exclude_per_dir && !opts.update))\n \t\tdie(\"--exclude-per-directory is meaningless unless -u\");\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 35aa979c3a4..e526b78b88d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -4021,9 +4021,8 @@ static int checkout(struct merge_options *opt,\n \t/* Switch the index/working copy from old to new */\n \tint ret;\n \tstruct tree_desc trees[2];\n-\tstruct unpack_trees_options unpack_opts;\n+\tstruct unpack_trees_options unpack_opts = UNPACK_TREES_OPTIONS_INIT;\n \n-\tmemset(&unpack_opts, 0, sizeof(unpack_opts));\n \tunpack_opts.head_idx = -1;\n \tunpack_opts.src_index = opt->repo->index;\n \tunpack_opts.dst_index = opt->repo->index;\n@@ -4046,9 +4045,8 @@ static int checkout(struct merge_options *opt,\n \tunpack_opts.verbose_update = (opt->verbosity > 2);\n \tunpack_opts.fn = twoway_merge;\n \tif (1/* FIXME: opts->overwrite_ignore*/) {\n-\t\tCALLOC_ARRAY(unpack_opts.dir, 1);\n-\t\tunpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(unpack_opts.dir);\n+\t\tunpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&unpack_opts.dir);\n \t}\n \tparse_tree(prev);\n \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n@@ -4057,8 +4055,6 @@ static int checkout(struct merge_options *opt,\n \n \tret = unpack_trees(2, trees, &unpack_opts);\n \tclear_unpack_trees_porcelain(&unpack_opts);\n-\tdir_clear(unpack_opts.dir);\n-\tFREE_AND_NULL(unpack_opts.dir);\n \treturn ret;\n }\n \ndiff --git a/merge.c b/merge.c\nindex 6e736881d90..9cb32990dd9 100644\n--- a/merge.c\n+++ b/merge.c\n@@ -50,10 +50,9 @@ int checkout_fast_forward(struct repository *r,\n \t\t\t int overwrite_ignore)\n {\n \tstruct tree *trees[MAX_UNPACK_TREES];\n-\tstruct unpack_trees_options opts;\n+\tstruct unpack_trees_options opts = UNPACK_TREES_OPTIONS_INIT;\n \tstruct tree_desc t[MAX_UNPACK_TREES];\n \tint i, nr_trees = 0;\n-\tstruct dir_struct dir = DIR_INIT;\n \tstruct lock_file lock_file = LOCK_INIT;\n \n \trefresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL);\n@@ -79,11 +78,9 @@ int checkout_fast_forward(struct repository *r,\n \t\tinit_tree_desc(t+i, trees[i]->buffer, trees[i]->size);\n \t}\n \n-\tmemset(&opts, 0, sizeof(opts));\n \tif (overwrite_ignore) {\n-\t\tdir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&dir);\n-\t\topts.dir = &dir;\n+\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&opts.dir);\n \t}\n \n \topts.head_idx = 1;\n@@ -101,7 +98,6 @@ int checkout_fast_forward(struct repository *r,\n \t\tclear_unpack_trees_porcelain(&opts);\n \t\treturn -1;\n \t}\n-\tdir_clear(&dir);\n \tclear_unpack_trees_porcelain(&opts);\n \n \tif (write_locked_index(r->index, &lock_file, COMMIT_LOCK))\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..33a2dc23ffc 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2081,7 +2081,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n \t */\n \tint namelen;\n \tint i;\n-\tstruct dir_struct d;\n+\tstruct dir_struct d = DIR_INIT;\n \tchar *pathbuf;\n \tint cnt = 0;\n \n@@ -2132,9 +2132,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n \t */\n \tpathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n \n-\tmemset(&d, 0, sizeof(d));\n-\tif (o->dir)\n-\t\td.exclude_per_dir = o->dir->exclude_per_dir;\n+\td.exclude_per_dir = o->dir.exclude_per_dir;\n \ti = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n \tif (i)\n \t\treturn add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\n@@ -2175,8 +2173,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \tif (ignore_case && icase_exists(o, name, len, st))\n \t\treturn 0;\n \n-\tif (o->dir &&\n-\t is_excluded(o->dir, o->src_index, name, &dtype))\n+\tif (is_excluded(&o->dir, o->src_index, name, &dtype))\n \t\t/*\n \t\t * ce->name is explicitly excluded, so it is Ok to\n \t\t * overwrite it.\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 2d88b19dca7..6fa6a4dfc3e 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -5,6 +5,7 @@\n #include \"strvec.h\"\n #include \"string-list.h\"\n #include \"tree-walk.h\"\n+#include \"dir.h\"\n \n #define MAX_UNPACK_TREES MAX_TRAVERSE_TREES\n \n@@ -66,7 +67,7 @@ struct unpack_trees_options {\n \t\t dry_run;\n \tconst char *prefix;\n \tint cache_bottom;\n-\tstruct dir_struct *dir;\n+\tstruct dir_struct dir;\n \tstruct pathspec *pathspec;\n \tmerge_fn_t fn;\n \tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n@@ -90,6 +91,9 @@ struct unpack_trees_options {\n \tstruct pattern_list *pl; /* for internal use */\n \tstruct checkout_metadata meta;\n };\n+#define UNPACK_TREES_OPTIONS_INIT { \\\n+\t.dir = DIR_INIT, \\\n+}\n \n int unpack_trees(unsigned n, struct tree_desc *t,\n \t\t struct unpack_trees_options *options);\n"},{"id":"437462","messageId":"CABPp-BEnGfxqKpqXGKopXFBBshY0tuimQLtwt58wKf9CJS8n5g@mail.gmail.com","threadId":"56534","inReplyTo":"87ilyjviiy.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-29T15:35:34Z","receivedAt":"2021-09-29T15:36:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Sep 29, 2021 at 2:27 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Mon, Sep 27 2021, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren \n> >\n> > Currently, every caller of unpack_trees() that wants to ensure ignored\n> > files are overwritten by default needs to:\n> > * allocate unpack_trees_options.dir\n> > * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir->flags\n> > * call setup_standard_excludes\n> > AND then after the call to unpack_trees() needs to\n> > * call dir_clear()\n> > * deallocate unpack_trees_options.dir\n> > That's a fair amount of boilerplate, and every caller uses identical\n> > code. Make this easier by instead introducing a new boolean value where\n> > the default value (0) does what we want so that new callers of\n> > unpack_trees() automatically get the appropriate behavior. And move all\n> > the handling of unpack_trees_options.dir into unpack_trees() itself.\n> >\n> > While preserve_ignored = 0 is the behavior we feel is the appropriate\n> > default, we defer fixing commands to use the appropriate default until a\n> > later commit. So, this commit introduces several locations where we\n> > manually set preserve_ignored=1. This makes it clear where code paths\n> > were previously preserving ignored files when they should not have been;\n> > a future commit will flip these to instead use a value of 0 to get the\n> > behavior we want.\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > builtin/am.c | 3 +++\n> > builtin/checkout.c | 11 ++---------\n> > builtin/clone.c | 2 ++\n> > builtin/merge.c | 2 ++\n> > builtin/read-tree.c | 13 +++----------\n> > builtin/reset.c | 2 ++\n> > builtin/stash.c | 3 +++\n> > merge-ort.c | 8 +-------\n> > merge-recursive.c | 8 +-------\n> > merge.c | 8 +-------\n> > reset.c | 2 ++\n> > sequencer.c | 2 ++\n> > unpack-trees.c | 10 ++++++++++\n> > unpack-trees.h | 1 +\n> > 14 files changed, 35 insertions(+), 40 deletions(-)\n> >\n> > diff --git a/builtin/am.c b/builtin/am.c\n> > index e4a0ff9cd7c..1ee70692bc3 100644\n> > --- a/builtin/am.c\n> > +++ b/builtin/am.c\n> > @@ -1918,6 +1918,9 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n> > opts.update = 1;\n> > opts.merge = 1;\n> > opts.reset = reset;\n> > + if (!reset)\n> > + /* FIXME: Default should be to remove ignored files */\n> > + opts.preserve_ignored = 1;\n> > opts.fn = twoway_merge;\n> > init_tree_desc(&t[0], head->buffer, head->size);\n> > init_tree_desc(&t[1], remote->buffer, remote->size);\n> > diff --git a/builtin/checkout.c b/builtin/checkout.c\n> > index 5335435d616..5e7957dd068 100644\n> > --- a/builtin/checkout.c\n> > +++ b/builtin/checkout.c\n> > @@ -648,6 +648,7 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n> > opts.skip_unmerged = !worktree;\n> > opts.reset = 1;\n> > opts.merge = 1;\n> > + opts.preserve_ignored = 0;\n> > opts.fn = oneway_merge;\n> > opts.verbose_update = o->show_progress;\n> > opts.src_index = &the_index;\n> > @@ -746,11 +747,7 @@ static int merge_working_tree(const struct checkout_opts *opts,\n> > new_branch_info->commit ?\n> > &new_branch_info->commit->object.oid :\n> > &new_branch_info->oid, NULL);\n> > - if (opts->overwrite_ignore) {\n> > - topts.dir = xcalloc(1, sizeof(*topts.dir));\n> > - topts.dir->flags |= DIR_SHOW_IGNORED;\n> > - setup_standard_excludes(topts.dir);\n> > - }\n> > + topts.preserve_ignored = !opts->overwrite_ignore;\n> > tree = parse_tree_indirect(old_branch_info->commit ?\n> > &old_branch_info->commit->object.oid :\n> > the_hash_algo->empty_tree);\n> > @@ -760,10 +757,6 @@ static int merge_working_tree(const struct checkout_opts *opts,\n> > init_tree_desc(&trees[1], tree->buffer, tree->size);\n> >\n> > ret = unpack_trees(2, trees, &topts);\n> > - if (topts.dir) {\n> > - dir_clear(topts.dir);\n> > - FREE_AND_NULL(topts.dir);\n> > - }\n> > clear_unpack_trees_porcelain(&topts);\n> > if (ret == -1) {\n> > /*\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index ff1d3d447a3..be1c3840d62 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -687,6 +687,8 @@ static int checkout(int submodule_progress)\n> > opts.update = 1;\n> > opts.merge = 1;\n> > opts.clone = 1;\n> > + /* FIXME: Default should be to remove ignored files */\n> > + opts.preserve_ignored = 1;\n> > opts.fn = oneway_merge;\n> > opts.verbose_update = (option_verbosity >= 0);\n> > opts.src_index = &the_index;\n> > diff --git a/builtin/merge.c b/builtin/merge.c\n> > index 3fbdacc7db4..1e5fff095fc 100644\n> > --- a/builtin/merge.c\n> > +++ b/builtin/merge.c\n> > @@ -680,6 +680,8 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n> > opts.verbose_update = 1;\n> > opts.trivial_merges_only = 1;\n> > opts.merge = 1;\n> > + /* FIXME: Default should be to remove ignored files */\n> > + opts.preserve_ignored = 1;\n> > trees[nr_trees] = parse_tree_indirect(common);\n> > if (!trees[nr_trees++])\n> > return -1;\n> > diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> > index 73cb957a69b..443d206eca6 100644\n> > --- a/builtin/read-tree.c\n> > +++ b/builtin/read-tree.c\n> > @@ -201,11 +201,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> > if ((opts.update || opts.index_only) && !opts.merge)\n> > die(\"%s is meaningless without -m, --reset, or --prefix\",\n> > opts.update ? \"-u\" : \"-i\");\n> > - if (opts.update && !opts.reset) {\n> > - CALLOC_ARRAY(opts.dir, 1);\n> > - opts.dir->flags |= DIR_SHOW_IGNORED;\n> > - setup_standard_excludes(opts.dir);\n> > - }\n> > + if (opts.update && !opts.reset)\n> > + opts.preserve_ignored = 0;\n> > + /* otherwise, opts.preserve_ignored is irrelevant */\n> > if (opts.merge && !opts.index_only)\n> > setup_work_tree();\n> >\n> > @@ -245,11 +243,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> > if (unpack_trees(nr_trees, t, &opts))\n> > return 128;\n> >\n> > - if (opts.dir) {\n> > - dir_clear(opts.dir);\n> > - FREE_AND_NULL(opts.dir);\n> > - }\n> > -\n> > if (opts.debug_unpack || opts.dry_run)\n> > return 0; /* do not write the index out */\n> >\n> > diff --git a/builtin/reset.c b/builtin/reset.c\n> > index 51c9e2f43ff..7f38656f018 100644\n> > --- a/builtin/reset.c\n> > +++ b/builtin/reset.c\n> > @@ -67,6 +67,8 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n> > case KEEP:\n> > case MERGE:\n> > opts.update = 1;\n> > + /* FIXME: Default should be to remove ignored files */\n> > + opts.preserve_ignored = 1;\n> > break;\n> > case HARD:\n> > opts.update = 1;\n> > diff --git a/builtin/stash.c b/builtin/stash.c\n> > index 8f42360ca91..88287b890d5 100644\n> > --- a/builtin/stash.c\n> > +++ b/builtin/stash.c\n> > @@ -258,6 +258,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n> > opts.merge = 1;\n> > opts.reset = reset;\n> > opts.update = update;\n> > + if (update && !reset)\n> > + /* FIXME: Default should be to remove ignored files */\n> > + opts.preserve_ignored = 1;\n> > opts.fn = oneway_merge;\n> >\n> > if (unpack_trees(nr_trees, t, &opts))\n> > diff --git a/merge-ort.c b/merge-ort.c\n> > index 35aa979c3a4..0d64ec716bd 100644\n> > --- a/merge-ort.c\n> > +++ b/merge-ort.c\n> > @@ -4045,11 +4045,7 @@ static int checkout(struct merge_options *opt,\n> > unpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n> > unpack_opts.verbose_update = (opt->verbosity > 2);\n> > unpack_opts.fn = twoway_merge;\n> > - if (1/* FIXME: opts->overwrite_ignore*/) {\n> > - CALLOC_ARRAY(unpack_opts.dir, 1);\n> > - unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n> > - setup_standard_excludes(unpack_opts.dir);\n> > - }\n> > + unpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n> > parse_tree(prev);\n> > init_tree_desc(&trees[0], prev->buffer, prev->size);\n> > parse_tree(next);\n> > @@ -4057,8 +4053,6 @@ static int checkout(struct merge_options *opt,\n> >\n> > ret = unpack_trees(2, trees, &unpack_opts);\n> > clear_unpack_trees_porcelain(&unpack_opts);\n> > - dir_clear(unpack_opts.dir);\n> > - FREE_AND_NULL(unpack_opts.dir);\n> > return ret;\n> > }\n> >\n> > diff --git a/merge-recursive.c b/merge-recursive.c\n> > index 233d9f686ad..2be3f5d4044 100644\n> > --- a/merge-recursive.c\n> > +++ b/merge-recursive.c\n> > @@ -411,9 +411,7 @@ static int unpack_trees_start(struct merge_options *opt,\n> > else {\n> > opt->priv->unpack_opts.update = 1;\n> > /* FIXME: should only do this if !overwrite_ignore */\n> > - CALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n> > - opt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n> > - setup_standard_excludes(opt->priv->unpack_opts.dir);\n> > + opt->priv->unpack_opts.preserve_ignored = 0;\n> > }\n> > opt->priv->unpack_opts.merge = 1;\n> > opt->priv->unpack_opts.head_idx = 2;\n> > @@ -428,10 +426,6 @@ static int unpack_trees_start(struct merge_options *opt,\n> > init_tree_desc_from_tree(t+2, merge);\n> >\n> > rc = unpack_trees(3, t, &opt->priv->unpack_opts);\n> > - if (opt->priv->unpack_opts.dir) {\n> > - dir_clear(opt->priv->unpack_opts.dir);\n> > - FREE_AND_NULL(opt->priv->unpack_opts.dir);\n> > - }\n> > cache_tree_free(&opt->repo->index->cache_tree);\n> >\n> > /*\n> > diff --git a/merge.c b/merge.c\n> > index 6e736881d90..2382ff66d35 100644\n> > --- a/merge.c\n> > +++ b/merge.c\n> > @@ -53,7 +53,6 @@ int checkout_fast_forward(struct repository *r,\n> > struct unpack_trees_options opts;\n> > struct tree_desc t[MAX_UNPACK_TREES];\n> > int i, nr_trees = 0;\n> > - struct dir_struct dir = DIR_INIT;\n> > struct lock_file lock_file = LOCK_INIT;\n> >\n> > refresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL);\n> > @@ -80,11 +79,7 @@ int checkout_fast_forward(struct repository *r,\n> > }\n> >\n> > memset(&opts, 0, sizeof(opts));\n> > - if (overwrite_ignore) {\n> > - dir.flags |= DIR_SHOW_IGNORED;\n> > - setup_standard_excludes(&dir);\n> > - opts.dir = &dir;\n> > - }\n> > + opts.preserve_ignored = !overwrite_ignore;\n> >\n> > opts.head_idx = 1;\n> > opts.src_index = r->index;\n> > @@ -101,7 +96,6 @@ int checkout_fast_forward(struct repository *r,\n> > clear_unpack_trees_porcelain(&opts);\n> > return -1;\n> > }\n> > - dir_clear(&dir);\n> > clear_unpack_trees_porcelain(&opts);\n> >\n> > if (write_locked_index(r->index, &lock_file, COMMIT_LOCK))\n> > diff --git a/reset.c b/reset.c\n> > index 79310ae071b..41b3e2d88de 100644\n> > --- a/reset.c\n> > +++ b/reset.c\n> > @@ -56,6 +56,8 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n> > unpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n> > unpack_tree_opts.update = 1;\n> > unpack_tree_opts.merge = 1;\n> > + /* FIXME: Default should be to remove ignored files */\n> > + unpack_tree_opts.preserve_ignored = 1;\n> > init_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n> > if (!detach_head)\n> > unpack_tree_opts.reset = 1;\n> > diff --git a/sequencer.c b/sequencer.c\n> > index 614d56f5e21..098566c68d9 100644\n> > --- a/sequencer.c\n> > +++ b/sequencer.c\n> > @@ -3699,6 +3699,8 @@ static int do_reset(struct repository *r,\n> > unpack_tree_opts.fn = oneway_merge;\n> > unpack_tree_opts.merge = 1;\n> > unpack_tree_opts.update = 1;\n> > + /* FIXME: Default should be to remove ignored files */\n> > + unpack_tree_opts.preserve_ignored = 1;\n> > init_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n> >\n> > if (repo_read_index_unmerged(r)) {\n> > diff --git a/unpack-trees.c b/unpack-trees.c\n> > index 8ea0a542da8..1e4eae1dc7d 100644\n> > --- a/unpack-trees.c\n> > +++ b/unpack-trees.c\n> > @@ -1707,6 +1707,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> > ensure_full_index(o->dst_index);\n> > }\n> >\n> > + if (!o->preserve_ignored) {\n> > + CALLOC_ARRAY(o->dir, 1);\n> > + o->dir->flags |= DIR_SHOW_IGNORED;\n> > + setup_standard_excludes(o->dir);\n> > + }\n> > +\n> > if (!core_apply_sparse_checkout || !o->update)\n> > o->skip_sparse_checkout = 1;\n> > if (!o->skip_sparse_checkout && !o->pl) {\n> > @@ -1868,6 +1874,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> > done:\n> > if (free_pattern_list)\n> > clear_pattern_list(&pl);\n> > + if (o->dir) {\n> > + dir_clear(o->dir);\n> > + FREE_AND_NULL(o->dir);\n> > + }\n> > trace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n> > trace_performance_leave(\"unpack_trees\");\n> > return ret;\n> > diff --git a/unpack-trees.h b/unpack-trees.h\n> > index 2d88b19dca7..f98cfd49d7b 100644\n> > --- a/unpack-trees.h\n> > +++ b/unpack-trees.h\n> > @@ -49,6 +49,7 @@ struct unpack_trees_options {\n> > unsigned int reset,\n> > merge,\n> > update,\n> > + preserve_ignored,\n> > clone,\n> > index_only,\n> > nontrivial_merge,\n>\n> I think getting rid of the boilerplate makes sense, but it doesn't sound\n> from the commit message like you've considered just making that \"struct\n> dir*\" member a \"struct dir\" instead.\n>\n> That simplifies things a lot, i.e. we can just DIR_INIT it, and don't\n> need every caller to malloc/free it.\n\nSee the next patch in the series. :-)\n\n> Sometimes a pointer makes sense, but in this case the \"struct\n> unpack_trees_options\" can just own it.\n\nI did make it internal to unpack_trees_options in the next patch, but\nkept it as a pointer just because that let me know whether it was used\nor not. I guess I could have added a boolean as well. But I don't\nactually allocate anything, because it's either a NULL pointer, or a\npointer to something on the stack. So, I do get to just use DIR_INIT.\n"},{"id":"437474","messageId":"87ee97utaq.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"CABPp-BEnGfxqKpqXGKopXFBBshY0tuimQLtwt58wKf9CJS8n5g@mail.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-29T18:30:08Z","receivedAt":"2021-09-29T18:32:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 29 2021, Elijah Newren wrote:\n\n> On Wed, Sep 29, 2021 at 2:27 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> On Mon, Sep 27 2021, Elijah Newren via GitGitGadget wrote:\n>>\n>> > From: Elijah Newren \n>> >\n>> > Currently, every caller of unpack_trees() that wants to ensure ignored\n>> > files are overwritten by default needs to:\n>> > * allocate unpack_trees_options.dir\n>> > * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir->flags\n>> > * call setup_standard_excludes\n>> > AND then after the call to unpack_trees() needs to\n>> > * call dir_clear()\n>> > * deallocate unpack_trees_options.dir\n>> > That's a fair amount of boilerplate, and every caller uses identical\n>> > code. Make this easier by instead introducing a new boolean value where\n>> > the default value (0) does what we want so that new callers of\n>> > unpack_trees() automatically get the appropriate behavior. And move all\n>> > the handling of unpack_trees_options.dir into unpack_trees() itself.\n>> >\n>> > While preserve_ignored = 0 is the behavior we feel is the appropriate\n>> > default, we defer fixing commands to use the appropriate default until a\n>> > later commit. So, this commit introduces several locations where we\n>> > manually set preserve_ignored=1. This makes it clear where code paths\n>> > were previously preserving ignored files when they should not have been;\n>> > a future commit will flip these to instead use a value of 0 to get the\n>> > behavior we want.\n>> >\n>> > Signed-off-by: Elijah Newren \n>> > ---\n>> > builtin/am.c | 3 +++\n>> > builtin/checkout.c | 11 ++---------\n>> > builtin/clone.c | 2 ++\n>> > builtin/merge.c | 2 ++\n>> > builtin/read-tree.c | 13 +++----------\n>> > builtin/reset.c | 2 ++\n>> > builtin/stash.c | 3 +++\n>> > merge-ort.c | 8 +-------\n>> > merge-recursive.c | 8 +-------\n>> > merge.c | 8 +-------\n>> > reset.c | 2 ++\n>> > sequencer.c | 2 ++\n>> > unpack-trees.c | 10 ++++++++++\n>> > unpack-trees.h | 1 +\n>> > 14 files changed, 35 insertions(+), 40 deletions(-)\n>> >\n>> > diff --git a/builtin/am.c b/builtin/am.c\n>> > index e4a0ff9cd7c..1ee70692bc3 100644\n>> > --- a/builtin/am.c\n>> > +++ b/builtin/am.c\n>> > @@ -1918,6 +1918,9 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n>> > opts.update = 1;\n>> > opts.merge = 1;\n>> > opts.reset = reset;\n>> > + if (!reset)\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + opts.preserve_ignored = 1;\n>> > opts.fn = twoway_merge;\n>> > init_tree_desc(&t[0], head->buffer, head->size);\n>> > init_tree_desc(&t[1], remote->buffer, remote->size);\n>> > diff --git a/builtin/checkout.c b/builtin/checkout.c\n>> > index 5335435d616..5e7957dd068 100644\n>> > --- a/builtin/checkout.c\n>> > +++ b/builtin/checkout.c\n>> > @@ -648,6 +648,7 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n>> > opts.skip_unmerged = !worktree;\n>> > opts.reset = 1;\n>> > opts.merge = 1;\n>> > + opts.preserve_ignored = 0;\n>> > opts.fn = oneway_merge;\n>> > opts.verbose_update = o->show_progress;\n>> > opts.src_index = &the_index;\n>> > @@ -746,11 +747,7 @@ static int merge_working_tree(const struct checkout_opts *opts,\n>> > new_branch_info->commit ?\n>> > &new_branch_info->commit->object.oid :\n>> > &new_branch_info->oid, NULL);\n>> > - if (opts->overwrite_ignore) {\n>> > - topts.dir = xcalloc(1, sizeof(*topts.dir));\n>> > - topts.dir->flags |= DIR_SHOW_IGNORED;\n>> > - setup_standard_excludes(topts.dir);\n>> > - }\n>> > + topts.preserve_ignored = !opts->overwrite_ignore;\n>> > tree = parse_tree_indirect(old_branch_info->commit ?\n>> > &old_branch_info->commit->object.oid :\n>> > the_hash_algo->empty_tree);\n>> > @@ -760,10 +757,6 @@ static int merge_working_tree(const struct checkout_opts *opts,\n>> > init_tree_desc(&trees[1], tree->buffer, tree->size);\n>> >\n>> > ret = unpack_trees(2, trees, &topts);\n>> > - if (topts.dir) {\n>> > - dir_clear(topts.dir);\n>> > - FREE_AND_NULL(topts.dir);\n>> > - }\n>> > clear_unpack_trees_porcelain(&topts);\n>> > if (ret == -1) {\n>> > /*\n>> > diff --git a/builtin/clone.c b/builtin/clone.c\n>> > index ff1d3d447a3..be1c3840d62 100644\n>> > --- a/builtin/clone.c\n>> > +++ b/builtin/clone.c\n>> > @@ -687,6 +687,8 @@ static int checkout(int submodule_progress)\n>> > opts.update = 1;\n>> > opts.merge = 1;\n>> > opts.clone = 1;\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + opts.preserve_ignored = 1;\n>> > opts.fn = oneway_merge;\n>> > opts.verbose_update = (option_verbosity >= 0);\n>> > opts.src_index = &the_index;\n>> > diff --git a/builtin/merge.c b/builtin/merge.c\n>> > index 3fbdacc7db4..1e5fff095fc 100644\n>> > --- a/builtin/merge.c\n>> > +++ b/builtin/merge.c\n>> > @@ -680,6 +680,8 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n>> > opts.verbose_update = 1;\n>> > opts.trivial_merges_only = 1;\n>> > opts.merge = 1;\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + opts.preserve_ignored = 1;\n>> > trees[nr_trees] = parse_tree_indirect(common);\n>> > if (!trees[nr_trees++])\n>> > return -1;\n>> > diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>> > index 73cb957a69b..443d206eca6 100644\n>> > --- a/builtin/read-tree.c\n>> > +++ b/builtin/read-tree.c\n>> > @@ -201,11 +201,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>> > if ((opts.update || opts.index_only) && !opts.merge)\n>> > die(\"%s is meaningless without -m, --reset, or --prefix\",\n>> > opts.update ? \"-u\" : \"-i\");\n>> > - if (opts.update && !opts.reset) {\n>> > - CALLOC_ARRAY(opts.dir, 1);\n>> > - opts.dir->flags |= DIR_SHOW_IGNORED;\n>> > - setup_standard_excludes(opts.dir);\n>> > - }\n>> > + if (opts.update && !opts.reset)\n>> > + opts.preserve_ignored = 0;\n>> > + /* otherwise, opts.preserve_ignored is irrelevant */\n>> > if (opts.merge && !opts.index_only)\n>> > setup_work_tree();\n>> >\n>> > @@ -245,11 +243,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>> > if (unpack_trees(nr_trees, t, &opts))\n>> > return 128;\n>> >\n>> > - if (opts.dir) {\n>> > - dir_clear(opts.dir);\n>> > - FREE_AND_NULL(opts.dir);\n>> > - }\n>> > -\n>> > if (opts.debug_unpack || opts.dry_run)\n>> > return 0; /* do not write the index out */\n>> >\n>> > diff --git a/builtin/reset.c b/builtin/reset.c\n>> > index 51c9e2f43ff..7f38656f018 100644\n>> > --- a/builtin/reset.c\n>> > +++ b/builtin/reset.c\n>> > @@ -67,6 +67,8 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n>> > case KEEP:\n>> > case MERGE:\n>> > opts.update = 1;\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + opts.preserve_ignored = 1;\n>> > break;\n>> > case HARD:\n>> > opts.update = 1;\n>> > diff --git a/builtin/stash.c b/builtin/stash.c\n>> > index 8f42360ca91..88287b890d5 100644\n>> > --- a/builtin/stash.c\n>> > +++ b/builtin/stash.c\n>> > @@ -258,6 +258,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n>> > opts.merge = 1;\n>> > opts.reset = reset;\n>> > opts.update = update;\n>> > + if (update && !reset)\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + opts.preserve_ignored = 1;\n>> > opts.fn = oneway_merge;\n>> >\n>> > if (unpack_trees(nr_trees, t, &opts))\n>> > diff --git a/merge-ort.c b/merge-ort.c\n>> > index 35aa979c3a4..0d64ec716bd 100644\n>> > --- a/merge-ort.c\n>> > +++ b/merge-ort.c\n>> > @@ -4045,11 +4045,7 @@ static int checkout(struct merge_options *opt,\n>> > unpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n>> > unpack_opts.verbose_update = (opt->verbosity > 2);\n>> > unpack_opts.fn = twoway_merge;\n>> > - if (1/* FIXME: opts->overwrite_ignore*/) {\n>> > - CALLOC_ARRAY(unpack_opts.dir, 1);\n>> > - unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n>> > - setup_standard_excludes(unpack_opts.dir);\n>> > - }\n>> > + unpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n>> > parse_tree(prev);\n>> > init_tree_desc(&trees[0], prev->buffer, prev->size);\n>> > parse_tree(next);\n>> > @@ -4057,8 +4053,6 @@ static int checkout(struct merge_options *opt,\n>> >\n>> > ret = unpack_trees(2, trees, &unpack_opts);\n>> > clear_unpack_trees_porcelain(&unpack_opts);\n>> > - dir_clear(unpack_opts.dir);\n>> > - FREE_AND_NULL(unpack_opts.dir);\n>> > return ret;\n>> > }\n>> >\n>> > diff --git a/merge-recursive.c b/merge-recursive.c\n>> > index 233d9f686ad..2be3f5d4044 100644\n>> > --- a/merge-recursive.c\n>> > +++ b/merge-recursive.c\n>> > @@ -411,9 +411,7 @@ static int unpack_trees_start(struct merge_options *opt,\n>> > else {\n>> > opt->priv->unpack_opts.update = 1;\n>> > /* FIXME: should only do this if !overwrite_ignore */\n>> > - CALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n>> > - opt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n>> > - setup_standard_excludes(opt->priv->unpack_opts.dir);\n>> > + opt->priv->unpack_opts.preserve_ignored = 0;\n>> > }\n>> > opt->priv->unpack_opts.merge = 1;\n>> > opt->priv->unpack_opts.head_idx = 2;\n>> > @@ -428,10 +426,6 @@ static int unpack_trees_start(struct merge_options *opt,\n>> > init_tree_desc_from_tree(t+2, merge);\n>> >\n>> > rc = unpack_trees(3, t, &opt->priv->unpack_opts);\n>> > - if (opt->priv->unpack_opts.dir) {\n>> > - dir_clear(opt->priv->unpack_opts.dir);\n>> > - FREE_AND_NULL(opt->priv->unpack_opts.dir);\n>> > - }\n>> > cache_tree_free(&opt->repo->index->cache_tree);\n>> >\n>> > /*\n>> > diff --git a/merge.c b/merge.c\n>> > index 6e736881d90..2382ff66d35 100644\n>> > --- a/merge.c\n>> > +++ b/merge.c\n>> > @@ -53,7 +53,6 @@ int checkout_fast_forward(struct repository *r,\n>> > struct unpack_trees_options opts;\n>> > struct tree_desc t[MAX_UNPACK_TREES];\n>> > int i, nr_trees = 0;\n>> > - struct dir_struct dir = DIR_INIT;\n>> > struct lock_file lock_file = LOCK_INIT;\n>> >\n>> > refresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL);\n>> > @@ -80,11 +79,7 @@ int checkout_fast_forward(struct repository *r,\n>> > }\n>> >\n>> > memset(&opts, 0, sizeof(opts));\n>> > - if (overwrite_ignore) {\n>> > - dir.flags |= DIR_SHOW_IGNORED;\n>> > - setup_standard_excludes(&dir);\n>> > - opts.dir = &dir;\n>> > - }\n>> > + opts.preserve_ignored = !overwrite_ignore;\n>> >\n>> > opts.head_idx = 1;\n>> > opts.src_index = r->index;\n>> > @@ -101,7 +96,6 @@ int checkout_fast_forward(struct repository *r,\n>> > clear_unpack_trees_porcelain(&opts);\n>> > return -1;\n>> > }\n>> > - dir_clear(&dir);\n>> > clear_unpack_trees_porcelain(&opts);\n>> >\n>> > if (write_locked_index(r->index, &lock_file, COMMIT_LOCK))\n>> > diff --git a/reset.c b/reset.c\n>> > index 79310ae071b..41b3e2d88de 100644\n>> > --- a/reset.c\n>> > +++ b/reset.c\n>> > @@ -56,6 +56,8 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n>> > unpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n>> > unpack_tree_opts.update = 1;\n>> > unpack_tree_opts.merge = 1;\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + unpack_tree_opts.preserve_ignored = 1;\n>> > init_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n>> > if (!detach_head)\n>> > unpack_tree_opts.reset = 1;\n>> > diff --git a/sequencer.c b/sequencer.c\n>> > index 614d56f5e21..098566c68d9 100644\n>> > --- a/sequencer.c\n>> > +++ b/sequencer.c\n>> > @@ -3699,6 +3699,8 @@ static int do_reset(struct repository *r,\n>> > unpack_tree_opts.fn = oneway_merge;\n>> > unpack_tree_opts.merge = 1;\n>> > unpack_tree_opts.update = 1;\n>> > + /* FIXME: Default should be to remove ignored files */\n>> > + unpack_tree_opts.preserve_ignored = 1;\n>> > init_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n>> >\n>> > if (repo_read_index_unmerged(r)) {\n>> > diff --git a/unpack-trees.c b/unpack-trees.c\n>> > index 8ea0a542da8..1e4eae1dc7d 100644\n>> > --- a/unpack-trees.c\n>> > +++ b/unpack-trees.c\n>> > @@ -1707,6 +1707,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> > ensure_full_index(o->dst_index);\n>> > }\n>> >\n>> > + if (!o->preserve_ignored) {\n>> > + CALLOC_ARRAY(o->dir, 1);\n>> > + o->dir->flags |= DIR_SHOW_IGNORED;\n>> > + setup_standard_excludes(o->dir);\n>> > + }\n>> > +\n>> > if (!core_apply_sparse_checkout || !o->update)\n>> > o->skip_sparse_checkout = 1;\n>> > if (!o->skip_sparse_checkout && !o->pl) {\n>> > @@ -1868,6 +1874,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> > done:\n>> > if (free_pattern_list)\n>> > clear_pattern_list(&pl);\n>> > + if (o->dir) {\n>> > + dir_clear(o->dir);\n>> > + FREE_AND_NULL(o->dir);\n>> > + }\n>> > trace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n>> > trace_performance_leave(\"unpack_trees\");\n>> > return ret;\n>> > diff --git a/unpack-trees.h b/unpack-trees.h\n>> > index 2d88b19dca7..f98cfd49d7b 100644\n>> > --- a/unpack-trees.h\n>> > +++ b/unpack-trees.h\n>> > @@ -49,6 +49,7 @@ struct unpack_trees_options {\n>> > unsigned int reset,\n>> > merge,\n>> > update,\n>> > + preserve_ignored,\n>> > clone,\n>> > index_only,\n>> > nontrivial_merge,\n>>\n>> I think getting rid of the boilerplate makes sense, but it doesn't sound\n>> from the commit message like you've considered just making that \"struct\n>> dir*\" member a \"struct dir\" instead.\n>>\n>> That simplifies things a lot, i.e. we can just DIR_INIT it, and don't\n>> need every caller to malloc/free it.\n>\n> See the next patch in the series. :-)\n\nAh!\n\n>> Sometimes a pointer makes sense, but in this case the \"struct\n>> unpack_trees_options\" can just own it.\n>\n> I did make it internal to unpack_trees_options in the next patch, but\n> kept it as a pointer just because that let me know whether it was used\n> or not. I guess I could have added a boolean as well. But I don't\n> actually allocate anything, because it's either a NULL pointer, or a\n> pointer to something on the stack. So, I do get to just use DIR_INIT.\n\nI think I'm probably missing something. I just made it allocated on the\nstack by the caller using \"struct unpack_trees_options\", but then you\nend up having a dir* in the struct, but that's only filled in as a\npointer to the stack variable? Maybe there's some subtlety I'm missing\nhere...\n"},{"id":"437494","messageId":"CABPp-BEEWPF6oBN69PH_GtmqqvAj1HMtDX5ipQxa2bLzyCCu0g@mail.gmail.com","threadId":"56534","inReplyTo":"87ee97utaq.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-30T04:25:20Z","receivedAt":"2021-09-30T04:25:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Sep 29, 2021 at 11:32 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Wed, Sep 29 2021, Elijah Newren wrote:\n>\n> > On Wed, Sep 29, 2021 at 2:27 AM Ævar Arnfjörð Bjarmason\n> > wrote:\n> >>\n...\n> >>\n> >> I think getting rid of the boilerplate makes sense, but it doesn't sound\n> >> from the commit message like you've considered just making that \"struct\n> >> dir*\" member a \"struct dir\" instead.\n> >>\n> >> That simplifies things a lot, i.e. we can just DIR_INIT it, and don't\n> >> need every caller to malloc/free it.\n> >\n> > See the next patch in the series. :-)\n>\n> Ah!\n>\n> >> Sometimes a pointer makes sense, but in this case the \"struct\n> >> unpack_trees_options\" can just own it.\n> >\n> > I did make it internal to unpack_trees_options in the next patch, but\n> > kept it as a pointer just because that let me know whether it was used\n> > or not. I guess I could have added a boolean as well. But I don't\n> > actually allocate anything, because it's either a NULL pointer, or a\n> > pointer to something on the stack. So, I do get to just use DIR_INIT.\n>\n> I think I'm probably missing something. I just made it allocated on the\n> stack by the caller using \"struct unpack_trees_options\", but then you\n> end up having a dir* in the struct, but that's only filled in as a\n> pointer to the stack variable? Maybe there's some subtlety I'm missing\n> here...\n\nAs per the next patch:\n\nint unpack_trees(..., struct unpack_trees_options *o)\n{\n struct dir_struct dir = DIR_INIT;\n ...\n if (!o->preserve_ignored) {\n /* Setup 'dir', make o->dir point to it */\n ....\n o->dir = &dir;\n }\n ...\n if (o->dir)\n /* cleanup */\n ....\n}\n\nThe caller doesn't touch o->dir (other than initializing it to zeros);\nunpack_trees() is wholly responsible for it. I'd kind of like to\nentirely remove dir from unpack_trees_options(), but I need a way of\npassing it down through all the other functions in unpack-trees.c, and\nleaving it in unpack_trees_options seems the easiest way to do so. So\nI just marked it as \"for internal use only\".\n"},{"id":"437541","messageId":"b23bb983-39e6-75ad-0cb5-d9a5ba2cd4d8@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Fix various issues around removal of untracked files/directories","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-09-30T14:00:48Z","receivedAt":"2021-09-30T14:00:53Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"[Resending as it did not seem to get through to the list last time]\n\nHi Elijah\n\nOn 27/09/2021 17:33, Elijah Newren via GitGitGadget wrote:\n> We have multiple codepaths that delete untracked files/directories but\n> shouldn't. There are also some codepaths where we delete untracked\n> files/directories intentionally (based on mailing list discussion), but\n> where that intent is not documented. We also have some codepaths that\n> preserve ignored files, which shouldn't. Fix the documentation, add several\n> new (mostly failing) testcases, fix some of the new testcases, and add\n> comments about some potential remaining problems. (I found these as a\n> side-effect of looking at [1], though [2] pointed out one explicitly while I\n> was working on it.)\n> \n> Note that I'm using Junio's declaration about checkout -f and reset --hard\n> (and also presuming that since read-tree --reset is porcelain that its\n> behavior should be left alone)[3] in this series.\n\nI've had a read through and I don't have any specific comments, I like \nthe way you have simplified adding the standard excludes for callers and \nmaking the existing value of reset invalid when converting to an enum. I \nthink there is a small risk someone will complain about read-tree \nchanging how it handles ignored files, but hopefully everyone was just \npassing \".gitignore\" to --exclude-per-directory and they wont mind \n'read-tree -m -u' removing ignored files now.\n\nBest Wishes\n\nPhillip\n"},{"id":"437543","messageId":"87lf3etaih.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"CABPp-BEEWPF6oBN69PH_GtmqqvAj1HMtDX5ipQxa2bLzyCCu0g@mail.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T14:04:58Z","receivedAt":"2021-09-30T14:16:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 29 2021, Elijah Newren wrote:\n\n> On Wed, Sep 29, 2021 at 11:32 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> On Wed, Sep 29 2021, Elijah Newren wrote:\n>>\n>> > On Wed, Sep 29, 2021 at 2:27 AM Ævar Arnfjörð Bjarmason\n>> > wrote:\n>> >>\n> ...\n>> >>\n>> >> I think getting rid of the boilerplate makes sense, but it doesn't sound\n>> >> from the commit message like you've considered just making that \"struct\n>> >> dir*\" member a \"struct dir\" instead.\n>> >>\n>> >> That simplifies things a lot, i.e. we can just DIR_INIT it, and don't\n>> >> need every caller to malloc/free it.\n>> >\n>> > See the next patch in the series. :-)\n>>\n>> Ah!\n>>\n>> >> Sometimes a pointer makes sense, but in this case the \"struct\n>> >> unpack_trees_options\" can just own it.\n>> >\n>> > I did make it internal to unpack_trees_options in the next patch, but\n>> > kept it as a pointer just because that let me know whether it was used\n>> > or not. I guess I could have added a boolean as well. But I don't\n>> > actually allocate anything, because it's either a NULL pointer, or a\n>> > pointer to something on the stack. So, I do get to just use DIR_INIT.\n>>\n>> I think I'm probably missing something. I just made it allocated on the\n>> stack by the caller using \"struct unpack_trees_options\", but then you\n>> end up having a dir* in the struct, but that's only filled in as a\n>> pointer to the stack variable? Maybe there's some subtlety I'm missing\n>> here...\n>\n> As per the next patch:\n>\n> int unpack_trees(..., struct unpack_trees_options *o)\n> {\n> struct dir_struct dir = DIR_INIT;\n> ...\n> if (!o->preserve_ignored) {\n> /* Setup 'dir', make o->dir point to it */\n> ....\n> o->dir = &dir;\n> }\n> ...\n> if (o->dir)\n> /* cleanup */\n> ....\n> }\n>\n> The caller doesn't touch o->dir (other than initializing it to zeros);\n> unpack_trees() is wholly responsible for it. I'd kind of like to\n> entirely remove dir from unpack_trees_options(), but I need a way of\n> passing it down through all the other functions in unpack-trees.c, and\n> leaving it in unpack_trees_options seems the easiest way to do so. So\n> I just marked it as \"for internal use only\".\n\nI think I understand *how* it works, I'm puzzled by why you went for\nthis whole level of indirection when you're using a struct on the stack\nin the end anyway, just ... put that in \"struct unpack_trees_options\"?\n\nAnyway, I see I have only myself to blame here, as you added these leak\nfixes in the v2 in response to some of my offhand comments.\n\nFWIW I then went on to do some deeper fixes not just on these leaks but\nthe surrounding leaks, which will be blocked by 2/11 & 05/11 of this\ntopic for a while. I suppose I only have myself to blame :)\n\nBelow is a patch-on-top that I think makes this whole thing much simpler\nby doing away with the pointer entirely.\n\nI suppose this is also a partial reply to\nhttps://lore.kernel.org/git/CABPp-BG_qigBoirMGR-Yk9Niyxt0UmYCEqojsYxbSEarLAmraA@mail.gmail.com/;\nbut I quite dislike this pattern of including a pointer like this where\nit's not needed just for the practicalities of memory management.\n\nI.e. here you use DIR_INIT. In my local patches to fix up the wider\nmemory leaks in this area I've got DIR_INIT also using a STRBUF_INIT,\nand DIR_INIT will in turn be referenced by a\nUNPACK_TREES_OPTIONS_INIT. It's quite nice if you're having to\ninitialize with \"UNPACK_TREES_OPTIONS_INIT\" have that initialization\nwork all the way down the chain, and not need e.g. a manual\nstrbuf_init(), dir_init() etc.\n\nI removed the dir_init() in ce93a4c6127 (dir.[ch]: replace dir_init()\nwith DIR_INIT, 2021-07-01), but would probably need to bring it back, of\ncourse you need some \"release()\" method for the\nUNPACK_TREES_OPTIONS_INIT, which in turn needs to call the dir_release()\n(well, \"dir_clear()\" in that case), and it needs to call\n\"strbuf_release()\". It's just nicer if that boilerplate is all on\ndestruction, but not also on struct/object setup.\n\nWe do need that setup in some cases (although a lot could just be\nreplaced by lazy initialization), but if we don't....\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a7e1712d236..de5cc6cd025 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1694,15 +1694,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n-\tstruct dir_struct dir = DIR_INIT;\n \n \tif (o->reset == UNPACK_RESET_INVALID)\n \t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n \n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n-\tif (o->dir)\n-\t\tBUG(\"o->dir is for internal use only\");\n \n \ttrace_performance_enter();\n \ttrace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n@@ -1718,9 +1715,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tBUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n \n \tif (!o->preserve_ignored) {\n-\t\to->dir = &dir;\n-\t\to->dir->flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(o->dir);\n+\t\to->dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&o->dir);\n \t}\n \n \tif (!core_apply_sparse_checkout || !o->update)\n@@ -1884,10 +1880,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n done:\n \tif (free_pattern_list)\n \t\tclear_pattern_list(&pl);\n-\tif (o->dir) {\n-\t\tdir_clear(o->dir);\n-\t\to->dir = NULL;\n-\t}\n+\tdir_clear(&o->dir);\n \ttrace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n \ttrace_performance_leave(\"unpack_trees\");\n \treturn ret;\n@@ -2153,8 +2146,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n \tpathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n \n \tmemset(&d, 0, sizeof(d));\n-\tif (o->dir)\n-\t\td.exclude_per_dir = o->dir->exclude_per_dir;\n+\td.exclude_per_dir = o->dir.exclude_per_dir;\n \ti = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n \tif (i)\n \t\treturn add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\n@@ -2201,8 +2193,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \tif (ignore_case && icase_exists(o, name, len, st))\n \t\treturn 0;\n \n-\tif (o->dir &&\n-\t is_excluded(o->dir, o->src_index, name, &dtype))\n+\tif (is_excluded(&o->dir, o->src_index, name, &dtype))\n \t\t/*\n \t\t * ce->name is explicitly excluded, so it is Ok to\n \t\t * overwrite it.\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 71ffb7eeb0c..a8afbb20170 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -5,6 +5,7 @@\n #include \"strvec.h\"\n #include \"string-list.h\"\n #include \"tree-walk.h\"\n+#include \"dir.h\"\n \n #define MAX_UNPACK_TREES MAX_TRAVERSE_TREES\n \n@@ -95,7 +96,7 @@ struct unpack_trees_options {\n \tstruct index_state result;\n \n \tstruct pattern_list *pl; /* for internal use */\n-\tstruct dir_struct *dir; /* for internal use only */\n+\tstruct dir_struct dir; /* for internal use only */\n \tstruct checkout_metadata meta;\n };\n \n"},{"id":"437597","messageId":"CABPp-BGi03JunRaMF_8SJKC00byOnq1kL3JyYhKWatz8-B4RsA@mail.gmail.com","threadId":"56534","inReplyTo":"87lf3etaih.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T01:53:49Z","receivedAt":"2021-10-01T01:54:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 30, 2021 at 7:15 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Wed, Sep 29 2021, Elijah Newren wrote:\n>\n> > On Wed, Sep 29, 2021 at 11:32 AM Ævar Arnfjörð Bjarmason\n> > wrote:\n> >>\n> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n> >>\n...\n> > As per the next patch:\n> >\n> > int unpack_trees(..., struct unpack_trees_options *o)\n> > {\n> > struct dir_struct dir = DIR_INIT;\n> > ...\n> > if (!o->preserve_ignored) {\n> > /* Setup 'dir', make o->dir point to it */\n> > ....\n> > o->dir = &dir;\n> > }\n> > ...\n> > if (o->dir)\n> > /* cleanup */\n> > ....\n> > }\n> >\n> > The caller doesn't touch o->dir (other than initializing it to zeros);\n> > unpack_trees() is wholly responsible for it. I'd kind of like to\n> > entirely remove dir from unpack_trees_options(), but I need a way of\n> > passing it down through all the other functions in unpack-trees.c, and\n> > leaving it in unpack_trees_options seems the easiest way to do so. So\n> > I just marked it as \"for internal use only\".\n>\n> I think I understand *how* it works, I'm puzzled by why you went for\n> this whole level of indirection when you're using a struct on the stack\n> in the end anyway, just ... put that in \"struct unpack_trees_options\"?\n>\n> Anyway, I see I have only myself to blame here, as you added these leak\n> fixes in the v2 in response to some of my offhand comments.\n>\n> FWIW I then went on to do some deeper fixes not just on these leaks but\n> the surrounding leaks, which will be blocked by 2/11 & 05/11 of this\n> topic for a while. I suppose I only have myself to blame :)\n>\n> Below is a patch-on-top that I think makes this whole thing much simpler\n> by doing away with the pointer entirely.\n>\n> I suppose this is also a partial reply to\n> https://lore.kernel.org/git/CABPp-BG_qigBoirMGR-Yk9Niyxt0UmYCEqojsYxbSEarLAmraA@mail.gmail.com/;\n> but I quite dislike this pattern of including a pointer like this where\n> it's not needed just for the practicalities of memory management.\n>\n> I.e. here you use DIR_INIT. In my local patches to fix up the wider\n> memory leaks in this area I've got DIR_INIT also using a STRBUF_INIT,\n> and DIR_INIT will in turn be referenced by a\n> UNPACK_TREES_OPTIONS_INIT. It's quite nice if you're having to\n> initialize with \"UNPACK_TREES_OPTIONS_INIT\" have that initialization\n> work all the way down the chain, and not need e.g. a manual\n> strbuf_init(), dir_init() etc.\n\nAnd you can keep using UNPACK_TREES_OPTIONS_INIT, because the\nunpack_trees_opts->dir should be initialized to NULL.\n\n> I removed the dir_init() in ce93a4c6127 (dir.[ch]: replace dir_init()\n> with DIR_INIT, 2021-07-01),\n\nI might be going on a tangent here, but looking at that patch, I'm\nworried that dir_init() was buggy and that you perpetuated that bug\nwith DIR_INIT. Note that dir_struct has a struct strbuf basebuf\nmember, which neither dir_init() or DIR_INIT initialize properly\n(using either strbuf_init() or STRBUF_INIT). As far as I can tell,\ndir.c relies on either strbuf_add() calls to just happen to work with\nthis incorrectly initialized strbuf, or else use the strbuf_init()\ncall in prep_exclude() to do so, using the following snippet:\n\n if (!dir->basebuf.buf)\n strbuf_init(&dir->basebuf, PATH_MAX);\n\nHowever, earlier in that same function we see\n\n if (stk->baselen <= baselen &&\n !strncmp(dir->basebuf.buf, base, stk->baselen))\n break;\n\nSo either that function can never have dir->basebuf.buf be NULL and\nthe strbuf_init() is dead code, or else it's possible for us to\ntrigger a segfault. If it's the former, it may just be a ticking time\nbomb that will transform into the latter with some other change,\nbecause it's not at all obvious to me how dir->basebuf gets\ninitialized appropriately to avoid that strncmp call. Perhaps there\nis some invariant where exclude_stack is only set up by previous calls\nto prep_exclude() and those won't set up exclude_stack until first\ninitializing basebuf. But that really at least deserves a comment\nabout how we're abusing basebuf, and would probably be cleaner if we\ninitialized basebuf to STRBUF_INIT.\n\n> but would probably need to bring it back, of\n\nIf you need to bring it back, it's unrelated to my changes here, and\nwould only be because of the lack of basebuf initialization above.\n\n> course you need some \"release()\" method for the\n> UNPACK_TREES_OPTIONS_INIT, which in turn needs to call the dir_release()\n> (well, \"dir_clear()\" in that case), and it needs to call\n> \"strbuf_release()\". It's just nicer if that boilerplate is all on\n> destruction, but not also on struct/object setup.\n\nThe caller should *not* be initializing or tearing down\nunpack_trees_options->dir beyond setting that field to NULL; it should\nthen leave it alone.\n\n> We do need that setup in some cases (although a lot could just be\n> replaced by lazy initialization), but if we don't....\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index a7e1712d236..de5cc6cd025 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1694,15 +1694,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> static struct cache_entry *dfc;\n> struct pattern_list pl;\n> int free_pattern_list = 0;\n> - struct dir_struct dir = DIR_INIT;\n>\n> if (o->reset == UNPACK_RESET_INVALID)\n> BUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n>\n> if (len > MAX_UNPACK_TREES)\n> die(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n> - if (o->dir)\n> - BUG(\"o->dir is for internal use only\");\n\nI think this was an important check that you've tossed without\nreplacement. Historically, callers set up and tweaked o->dir with\nvarious values. With my patch, we are no longer allowing that..which\nintroduces a transition problem -- people might have written or are\nnow writing patches that make new calls of unpack_trees() previous to\nthis change of mine, but submit them after this change of mine gets\nmerged. Without this check I added, they'd probably just do a\nmechanical `o->dir->` change to `o->dir.` and assume it's good...and\nthen possibly have ugly bugs to hunt down.\n\nSo, I think it's helpful to have a check that provides an early\nwarning that tweaking o->dir is not only no longer required, but also\nno longer allowed.\n\n> trace_performance_enter();\n> trace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n> @@ -1718,9 +1715,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> BUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n>\n> if (!o->preserve_ignored) {\n> - o->dir = &dir;\n> - o->dir->flags |= DIR_SHOW_IGNORED;\n> - setup_standard_excludes(o->dir);\n> + o->dir.flags |= DIR_SHOW_IGNORED;\n> + setup_standard_excludes(&o->dir);\n> }\n>\n> if (!core_apply_sparse_checkout || !o->update)\n> @@ -1884,10 +1880,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> done:\n> if (free_pattern_list)\n> clear_pattern_list(&pl);\n> - if (o->dir) {\n> - dir_clear(o->dir);\n> - o->dir = NULL;\n> - }\n> + dir_clear(&o->dir);\n\nUnconditionally calling dir_clear()...\n\n> trace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n> trace_performance_leave(\"unpack_trees\");\n> return ret;\n> @@ -2153,8 +2146,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n> pathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n>\n> memset(&d, 0, sizeof(d));\n> - if (o->dir)\n> - d.exclude_per_dir = o->dir->exclude_per_dir;\n> + d.exclude_per_dir = o->dir.exclude_per_dir;\n> i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n> if (i)\n> return add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\n> @@ -2201,8 +2193,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n> if (ignore_case && icase_exists(o, name, len, st))\n> return 0;\n>\n> - if (o->dir &&\n> - is_excluded(o->dir, o->src_index, name, &dtype))\n> + if (is_excluded(&o->dir, o->src_index, name, &dtype))\n\nUnconditionally calling is_excluded()...\n\n> /*\n> * ce->name is explicitly excluded, so it is Ok to\n> * overwrite it.\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index 71ffb7eeb0c..a8afbb20170 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -5,6 +5,7 @@\n> #include \"strvec.h\"\n> #include \"string-list.h\"\n> #include \"tree-walk.h\"\n> +#include \"dir.h\"\n>\n> #define MAX_UNPACK_TREES MAX_TRAVERSE_TREES\n>\n> @@ -95,7 +96,7 @@ struct unpack_trees_options {\n> struct index_state result;\n>\n> struct pattern_list *pl; /* for internal use */\n> - struct dir_struct *dir; /* for internal use only */\n> + struct dir_struct dir; /* for internal use only */\n> struct checkout_metadata meta;\n> };\n>\n\nNot only did you drop the important safety check that o->dir not be\nsetup by the caller (which needs to be reinstated in some form), your\nsolution also involves unconditionally calling dir_clear() and\nis_excluded(). It is not clear to me that those calls are safe...and\nthat they will continue to be safe in the future. Even if it is safe\nand will continue to be, I don't think this should be squashed into my\npatches. I think it should be a separate patch with its own commit\nmessage that explicitly calls out this assumption. Especially since\nthis is dir.c, which is an area where attempting to fix one very\nsimple little bug results in years of refactoring and fixing all kinds\nof historical messes, sometimes waiting a year and a half for\nresponses to RFCs/review requests, and where we have to sometimes just\ngive up on attempting to understand the purpose of various bits of\ncode and instead rely on the regression tests and hope they are good\nenough. I still think that dir.c deserves a little warning at the\ntop, like the one I suggested in [1].\n\n[1] https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/\n"},{"id":"437599","messageId":"CABPp-BHLyGP_V0P3jav2d6WR4CUpzZ4QwgazQJggG1diDJEaww@mail.gmail.com","threadId":"56534","inReplyTo":"aaa8ea3b-0902-f9e6-c1a4-0ca2b1b2f57b@gmail.com","subject":"Re: [PATCH v3 00/11] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T02:08:01Z","receivedAt":"2021-10-01T02:08:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Phillip,\n\nOn Thu, Sep 30, 2021 at 3:08 AM Phillip Wood wrote:\n>\n> Hi Elijah\n>\n> On 27/09/2021 17:33, Elijah Newren via GitGitGadget wrote:\n> > We have multiple codepaths that delete untracked files/directories but\n> > shouldn't. There are also some codepaths where we delete untracked\n> > files/directories intentionally (based on mailing list discussion), but\n> > where that intent is not documented. We also have some codepaths that\n> > preserve ignored files, which shouldn't. Fix the documentation, add several\n> > new (mostly failing) testcases, fix some of the new testcases, and add\n> > comments about some potential remaining problems. (I found these as a\n> > side-effect of looking at [1], though [2] pointed out one explicitly while I\n> > was working on it.)\n> >\n> > Note that I'm using Junio's declaration about checkout -f and reset --hard\n> > (and also presuming that since read-tree --reset is porcelain that its\n> > behavior should be left alone)[3] in this series.\n> >\n>\n> I've had a read through and I don't have any specific comments, I like\n> the way you have simplified adding the standard excludes for callers and\n> making the existing value of reset invalid when converting to an enum. I\n> think there is a small risk someone will complain about read-tree\n> changing how it handles ignored files, but hopefully everyone was just\n> passing \".gitignore\" to --exclude-per-directory and they wont mind\n> 'read-tree -m -u' removing ignored files now.\n\nThanks for taking a look!\n"},{"id":"437623","messageId":"87k0ixrv23.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"CABPp-BGi03JunRaMF_8SJKC00byOnq1kL3JyYhKWatz8-B4RsA@mail.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T08:15:22Z","receivedAt":"2021-10-01T08:47:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 30 2021, Elijah Newren wrote:\n\n> On Thu, Sep 30, 2021 at 7:15 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> On Wed, Sep 29 2021, Elijah Newren wrote:\n>>\n>> > On Wed, Sep 29, 2021 at 11:32 AM Ævar Arnfjörð Bjarmason\n>> > wrote:\n>> >>\n>> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n>> >>\n> ...\n>> > As per the next patch:\n>> >\n>> > int unpack_trees(..., struct unpack_trees_options *o)\n>> > {\n>> > struct dir_struct dir = DIR_INIT;\n>> > ...\n>> > if (!o->preserve_ignored) {\n>> > /* Setup 'dir', make o->dir point to it */\n>> > ....\n>> > o->dir = &dir;\n>> > }\n>> > ...\n>> > if (o->dir)\n>> > /* cleanup */\n>> > ....\n>> > }\n>> >\n>> > The caller doesn't touch o->dir (other than initializing it to zeros);\n>> > unpack_trees() is wholly responsible for it. I'd kind of like to\n>> > entirely remove dir from unpack_trees_options(), but I need a way of\n>> > passing it down through all the other functions in unpack-trees.c, and\n>> > leaving it in unpack_trees_options seems the easiest way to do so. So\n>> > I just marked it as \"for internal use only\".\n>>\n>> I think I understand *how* it works, I'm puzzled by why you went for\n>> this whole level of indirection when you're using a struct on the stack\n>> in the end anyway, just ... put that in \"struct unpack_trees_options\"?\n>>\n>> Anyway, I see I have only myself to blame here, as you added these leak\n>> fixes in the v2 in response to some of my offhand comments.\n>>\n>> FWIW I then went on to do some deeper fixes not just on these leaks but\n>> the surrounding leaks, which will be blocked by 2/11 & 05/11 of this\n>> topic for a while. I suppose I only have myself to blame :)\n>>\n>> Below is a patch-on-top that I think makes this whole thing much simpler\n>> by doing away with the pointer entirely.\n>>\n>> I suppose this is also a partial reply to\n>> https://lore.kernel.org/git/CABPp-BG_qigBoirMGR-Yk9Niyxt0UmYCEqojsYxbSEarLAmraA@mail.gmail.com/;\n>> but I quite dislike this pattern of including a pointer like this where\n>> it's not needed just for the practicalities of memory management.\n>>\n>> I.e. here you use DIR_INIT. In my local patches to fix up the wider\n>> memory leaks in this area I've got DIR_INIT also using a STRBUF_INIT,\n>> and DIR_INIT will in turn be referenced by a\n>> UNPACK_TREES_OPTIONS_INIT. It's quite nice if you're having to\n>> initialize with \"UNPACK_TREES_OPTIONS_INIT\" have that initialization\n>> work all the way down the chain, and not need e.g. a manual\n>> strbuf_init(), dir_init() etc.\n>\n> And you can keep using UNPACK_TREES_OPTIONS_INIT, because the\n> unpack_trees_opts->dir should be initialized to NULL.\n\nBut I don't want it initialized to NULL, I want DIR_INIT....\n\n>> I removed the dir_init() in ce93a4c6127 (dir.[ch]: replace dir_init()\n>> with DIR_INIT, 2021-07-01),\n>\n> I might be going on a tangent here, but looking at that patch, I'm\n> worried that dir_init() was buggy and that you perpetuated that bug\n> with DIR_INIT. Note that dir_struct has a struct strbuf basebuf\n> member, which neither dir_init() or DIR_INIT initialize properly\n> (using either strbuf_init() or STRBUF_INIT). As far as I can tell,\n> dir.c relies on either strbuf_add() calls to just happen to work with\n> this incorrectly initialized strbuf, or else use the strbuf_init()\n> call in prep_exclude() to do so, using the following snippet:\n>\n> if (!dir->basebuf.buf)\n> strbuf_init(&dir->basebuf, PATH_MAX);\n>\n> However, earlier in that same function we see\n>\n> if (stk->baselen <= baselen &&\n> !strncmp(dir->basebuf.buf, base, stk->baselen))\n> break;\n>\n> So either that function can never have dir->basebuf.buf be NULL and\n> the strbuf_init() is dead code, or else it's possible for us to\n> trigger a segfault. If it's the former, it may just be a ticking time\n> bomb that will transform into the latter with some other change,\n> because it's not at all obvious to me how dir->basebuf gets\n> initialized appropriately to avoid that strncmp call. Perhaps there\n> is some invariant where exclude_stack is only set up by previous calls\n> to prep_exclude() and those won't set up exclude_stack until first\n> initializing basebuf. But that really at least deserves a comment\n> about how we're abusing basebuf, and would probably be cleaner if we\n> initialized basebuf to STRBUF_INIT.\n\n...because yes, I forgot about that when sending you the diff-on-top,\nsorry. Yes that's buggy with the diff-on-top I sent you.\n\nI've got that fixed in the version I have. I.e. first I add a\nUNPACK_TREES_OPTIONS_INIT macro, then deal with that lazy initialization\ncase (at which point DIR_INIT starts initializing that strbuf), then\nchange the \"dir_struct\" from a pointer to embedding it, and finally fix\na memory leak with that new API.\n\nWIP patches here:\nhttps://github.com/avar/git/compare/avar/post-sanitize-leak-test-mode-add-and-use-revisions-release...avar/post-sanitize-leak-test-mode-unpack-trees-and-dir\n\n>> but would probably need to bring it back, of\n>\n> If you need to bring it back, it's unrelated to my changes here, and\n> would only be because of the lack of basebuf initialization above.\n\nYes, in this case. I mean that generally speaking I think it's a good\npattern to use to have structs be initialized by macros like this,\nbecause it means you can embed them N levels deep (as we sometimes do)\nwithout having to call functions to initialize them.\n\nSo yes, in this case as long as DIR_INIT is { 0 } it doesn't matter, but\nit does as soon as it has a member that needs initialization, and\ngenerally speaking for any FOO_INIT that needs a BAR_INIT ....\n\n>> course you need some \"release()\" method for the\n>> UNPACK_TREES_OPTIONS_INIT, which in turn needs to call the dir_release()\n>> (well, \"dir_clear()\" in that case), and it needs to call\n>> \"strbuf_release()\". It's just nicer if that boilerplate is all on\n>> destruction, but not also on struct/object setup.\n>\n> The caller should *not* be initializing or tearing down\n> unpack_trees_options->dir beyond setting that field to NULL; it should\n> then leave it alone.\n\ns/NULL/DIR_INIT/ in my version, but yes.\n\n>> We do need that setup in some cases (although a lot could just be\n>> replaced by lazy initialization), but if we don't....\n>>\n>> diff --git a/unpack-trees.c b/unpack-trees.c\n>> index a7e1712d236..de5cc6cd025 100644\n>> --- a/unpack-trees.c\n>> +++ b/unpack-trees.c\n>> @@ -1694,15 +1694,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> static struct cache_entry *dfc;\n>> struct pattern_list pl;\n>> int free_pattern_list = 0;\n>> - struct dir_struct dir = DIR_INIT;\n>>\n>> if (o->reset == UNPACK_RESET_INVALID)\n>> BUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n>>\n>> if (len > MAX_UNPACK_TREES)\n>> die(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n>> - if (o->dir)\n>> - BUG(\"o->dir is for internal use only\");\n>\n> I think this was an important check that you've tossed without\n> replacement. Historically, callers set up and tweaked o->dir with\n> various values. With my patch, we are no longer allowing that..which\n> introduces a transition problem -- people might have written or are\n> now writing patches that make new calls of unpack_trees() previous to\n> this change of mine, but submit them after this change of mine gets\n> merged. Without this check I added, they'd probably just do a\n> mechanical `o->dir->` change to `o->dir.` and assume it's good...and\n> then possibly have ugly bugs to hunt down.\n>\n> So, I think it's helpful to have a check that provides an early\n> warning that tweaking o->dir is not only no longer required, but also\n> no longer allowed.\n\nThe compiler will catch any such use of the pointer version on a\nmis-merge, or do you just mean that the person running into that might\nget the resolution wrong? I.e. before we could check o->dir being NULL\nfor \"do we have an exclude\", but &o->dir will always be true?\n\n>> trace_performance_enter();\n>> trace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n>> @@ -1718,9 +1715,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> BUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n>>\n>> if (!o->preserve_ignored) {\n>> - o->dir = &dir;\n>> - o->dir->flags |= DIR_SHOW_IGNORED;\n>> - setup_standard_excludes(o->dir);\n>> + o->dir.flags |= DIR_SHOW_IGNORED;\n>> + setup_standard_excludes(&o->dir);\n>> }\n>>\n>> if (!core_apply_sparse_checkout || !o->update)\n>> @@ -1884,10 +1880,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> done:\n>> if (free_pattern_list)\n>> clear_pattern_list(&pl);\n>> - if (o->dir) {\n>> - dir_clear(o->dir);\n>> - o->dir = NULL;\n>> - }\n>> + dir_clear(&o->dir);\n>\n> Unconditionally calling dir_clear()...\n\nAs before I'm not sure about bugs in the ad-hoc patch on top, but I\ndon't think this is a bug in my version linked above.\n\nI.e. it's zero'd out, and the dir_clear() either ends up calling\nfree(NULL) or tries to loop over 0..N where N will be 0, no?\n\n>> trace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n>> trace_performance_leave(\"unpack_trees\");\n>> return ret;\n>> @@ -2153,8 +2146,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n>> pathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n>>\n>> memset(&d, 0, sizeof(d));\n>> - if (o->dir)\n>> - d.exclude_per_dir = o->dir->exclude_per_dir;\n>> + d.exclude_per_dir = o->dir.exclude_per_dir;\n>> i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n>> if (i)\n>> return add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\n>> @@ -2201,8 +2193,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n>> if (ignore_case && icase_exists(o, name, len, st))\n>> return 0;\n>>\n>> - if (o->dir &&\n>> - is_excluded(o->dir, o->src_index, name, &dtype))\n>> + if (is_excluded(&o->dir, o->src_index, name, &dtype))\n>\n> Unconditionally calling is_excluded()...\n\nWhich will just return \"it's not\", won't it? Just lik dir_clear() deals\nwith an \"empty\" dir_struct. There's existing callers of both with that\npattern in e.g. builtin/{add,clean}.c.\n\nMaybe I've missed an edge case, but I think the only reason that \"o->dir\n&&\" was there was because it was dynamically malloc'd before, but in my\nversion where we'll always have it initialized...\n\n>> /*\n>> * ce->name is explicitly excluded, so it is Ok to\n>> * overwrite it.\n>> diff --git a/unpack-trees.h b/unpack-trees.h\n>> index 71ffb7eeb0c..a8afbb20170 100644\n>> --- a/unpack-trees.h\n>> +++ b/unpack-trees.h\n>> @@ -5,6 +5,7 @@\n>> #include \"strvec.h\"\n>> #include \"string-list.h\"\n>> #include \"tree-walk.h\"\n>> +#include \"dir.h\"\n>>\n>> #define MAX_UNPACK_TREES MAX_TRAVERSE_TREES\n>>\n>> @@ -95,7 +96,7 @@ struct unpack_trees_options {\n>> struct index_state result;\n>>\n>> struct pattern_list *pl; /* for internal use */\n>> - struct dir_struct *dir; /* for internal use only */\n>> + struct dir_struct dir; /* for internal use only */\n>> struct checkout_metadata meta;\n>> };\n>>\n>\n> Not only did you drop the important safety check that o->dir not be\n> setup by the caller (which needs to be reinstated in some form), your\n> solution also involves unconditionally calling dir_clear() and\n> is_excluded(). It is not clear to me that those calls are safe...and\n> that they will continue to be safe in the future.\n\nIt is a common pattern we rely on, e.g. strbuf_release() and various\nother custom free-like functions generally act as NOOP if they've got\nnothing to do, just like free()...\n\n> Even if it is safe\n> and will continue to be, I don't think this should be squashed into my\n> patches. I think it should be a separate patch with its own commit\n> message that explicitly calls out this assumption. Especially since\n> this is dir.c, which is an area where attempting to fix one very\n> simple little bug results in years of refactoring and fixing all kinds\n> of historical messes, sometimes waiting a year and a half for\n> responses to RFCs/review requests, and where we have to sometimes just\n> give up on attempting to understand the purpose of various bits of\n> code and instead rely on the regression tests and hope they are good\n> enough. I still think that dir.c deserves a little warning at the\n> top, like the one I suggested in [1].\n>\n> [1] https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/\n\n*nod* I can always submit something like this afterwards.\n\nJust on this series: Perhaps this discussion is a sign that this memory\nleak fixing should be its own cleanup series where we could hash out any\napproaches to doing that? I.e. as noted before I realize I'm to blame\nfor suggesting it in the first place, but those parts of these changes\ndon't seem like they're needed by other parts of the series (I tried\ncompiling with the two relevant patches ejected out).\n\nHaving a humongous set of memory leak fixes locally at this point, I\nthink it's generally not very worth the effort to fix a leak in b()\nwhere a() calls b() and b() calls c(), and all of [abc]() are\nleaking. I.e. often narrowly fixing leaks in b() will lead to different\nsolutions than if you're trying to resolve all of [abc](), as their\ninteraction comes into play.\n\nAside about safety: One thing I'll sometimes do when I'm unsure about\nthose sorts of fixes is to have my new INIT set a new \"sentinel\" field\nto \"12345\" or whatever, then just BUG() out in an entry point in the API\nthat you can't avoid calling if it's not set like that, e.g. dir_clear()\nor whatever the setup/work function is.\n\nWe don't have 100% test coverage, but we usually have at least *some*,\nand doing that is good about catching e.g. a memset() at a distance, as\nhappens in this code with the merge code embedding the relevant struct\nand memsetting it, which might be missed in some migration of just a\ngrep for \"struct dir_struct\" or whatever...\n"},{"id":"437667","messageId":"877dexrqvg.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"87k0ixrv23.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:53:43Z","receivedAt":"2021-10-01T10:17:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 01 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> Aside about safety: One thing I'll sometimes do when I'm unsure about\n> those sorts of fixes is to have my new INIT set a new \"sentinel\" field\n> to \"12345\" or whatever, then just BUG() out in an entry point in the API\n> that you can't avoid calling if it's not set like that, e.g. dir_clear()\n> or whatever the setup/work function is.\n\nFor reference: Something like the below, which passes with my WIP\npatches. Showing that no non-static entry point can reach the code in\nunpack-trees.c without \"foo\" being 12345, which can only be the case if\ncallers have used the macro (and the code internal to unpack-trees.c is\neasy enough to audit).\n\n unpack-trees.c | 25 +++++++++++++++++++++++++\n unpack-trees.h | 2 ++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d40af221e1c..f2365ecf215 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -199,6 +199,8 @@ void clear_unpack_trees_porcelain(struct unpack_trees_options *opts)\n {\n \tstrvec_clear(&opts->msgs_to_free);\n \tdir_clear(&opts->dir);\n+\tif (opts->foo != 12345)\n+\t\tBUG(\"noes\");\n \tmemset(opts->msgs, 0, sizeof(opts->msgs));\n }\n \n@@ -1702,6 +1704,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n@@ -1903,6 +1908,9 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n \tunsigned old_show_all_errors;\n \tint free_pattern_list = 0;\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \told_show_all_errors = o->show_all_errors;\n \to->show_all_errors = 1;\n \n@@ -2033,6 +2041,8 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n int verify_uptodate(const struct cache_entry *ce,\n \t\t struct unpack_trees_options *o)\n {\n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n \tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);\n@@ -2417,6 +2427,9 @@ int threeway_merge(const struct cache_entry * const *stages,\n \tint no_anc_exists = 1;\n \tint i;\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \tfor (i = 1; i < o->head_idx; i++) {\n \t\tif (!stages[i] || stages[i] == o->df_conflict_entry)\n \t\t\tany_anc_missing = 1;\n@@ -2580,6 +2593,9 @@ int twoway_merge(const struct cache_entry * const *src,\n \tconst struct cache_entry *oldtree = src[1];\n \tconst struct cache_entry *newtree = src[2];\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \tif (o->merge_size != 2)\n \t\treturn error(\"Cannot do a twoway merge of %d trees\",\n \t\t\t o->merge_size);\n@@ -2654,6 +2670,9 @@ int bind_merge(const struct cache_entry * const *src,\n \tconst struct cache_entry *old = src[0];\n \tconst struct cache_entry *a = src[1];\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \tif (o->merge_size != 1)\n \t\treturn error(\"Cannot do a bind merge of %d trees\",\n \t\t\t o->merge_size);\n@@ -2680,6 +2699,9 @@ int oneway_merge(const struct cache_entry * const *src,\n \tconst struct cache_entry *old = src[0];\n \tconst struct cache_entry *a = src[1];\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \tif (o->merge_size != 1)\n \t\treturn error(\"Cannot do a oneway merge of %d trees\",\n \t\t\t o->merge_size);\n@@ -2717,6 +2739,9 @@ int stash_worktree_untracked_merge(const struct cache_entry * const *src,\n \tconst struct cache_entry *worktree = src[1];\n \tconst struct cache_entry *untracked = src[2];\n \n+\tif (o->foo != 12345)\n+\t\tBUG(\"noes\");\n+\n \tif (o->merge_size != 2)\n \t\tBUG(\"invalid merge_size: %d\", o->merge_size);\n \ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 75b67f90ccd..8dae0938ad1 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -90,10 +90,12 @@ struct unpack_trees_options {\n \n \tstruct pattern_list *pl; /* for internal use */\n \tstruct checkout_metadata meta;\n+\tint foo;\n };\n #define UNPACK_TREES_OPTIONS_INIT { \\\n \t.msgs_to_free = STRVEC_INIT, \\\n \t.dir = DIR_INIT, \\\n+\t.foo = 12345, \\\n }\n \n void unpack_trees_init(struct unpack_trees_options *options);\n"},{"id":"437719","messageId":"CABPp-BE_aY4smj_b0+Zz=YrURKMniS=DmyMWVc=q2mVDL8zUOg@mail.gmail.com","threadId":"56534","inReplyTo":"87k0ixrv23.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-01T18:50:48Z","receivedAt":"2021-10-01T18:51:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 1, 2021 at 1:47 AM Ævar Arnfjörð Bjarmason wrote:\n>\n> On Thu, Sep 30 2021, Elijah Newren wrote:\n>\n> > On Thu, Sep 30, 2021 at 7:15 AM Ævar Arnfjörð Bjarmason\n> > wrote:\n> >>\n> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n> >>\n> >> > On Wed, Sep 29, 2021 at 11:32 AM Ævar Arnfjörð Bjarmason\n> >> > wrote:\n> >> >>\n> >> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n> >> >>\n> > ...\n> >> > As per the next patch:\n> >> >\n> >> > int unpack_trees(..., struct unpack_trees_options *o)\n> >> > {\n> >> > struct dir_struct dir = DIR_INIT;\n> >> > ...\n> >> > if (!o->preserve_ignored) {\n> >> > /* Setup 'dir', make o->dir point to it */\n> >> > ....\n> >> > o->dir = &dir;\n> >> > }\n> >> > ...\n> >> > if (o->dir)\n> >> > /* cleanup */\n> >> > ....\n> >> > }\n> >> >\n> >> > The caller doesn't touch o->dir (other than initializing it to zeros);\n> >> > unpack_trees() is wholly responsible for it. I'd kind of like to\n> >> > entirely remove dir from unpack_trees_options(), but I need a way of\n> >> > passing it down through all the other functions in unpack-trees.c, and\n> >> > leaving it in unpack_trees_options seems the easiest way to do so. So\n> >> > I just marked it as \"for internal use only\".\n> >>\n> >> I think I understand *how* it works, I'm puzzled by why you went for\n> >> this whole level of indirection when you're using a struct on the stack\n> >> in the end anyway, just ... put that in \"struct unpack_trees_options\"?\n> >>\n> >> Anyway, I see I have only myself to blame here, as you added these leak\n> >> fixes in the v2 in response to some of my offhand comments.\n> >>\n> >> FWIW I then went on to do some deeper fixes not just on these leaks but\n> >> the surrounding leaks, which will be blocked by 2/11 & 05/11 of this\n> >> topic for a while. I suppose I only have myself to blame :)\n> >>\n> >> Below is a patch-on-top that I think makes this whole thing much simpler\n> >> by doing away with the pointer entirely.\n> >>\n> >> I suppose this is also a partial reply to\n> >> https://lore.kernel.org/git/CABPp-BG_qigBoirMGR-Yk9Niyxt0UmYCEqojsYxbSEarLAmraA@mail.gmail.com/;\n> >> but I quite dislike this pattern of including a pointer like this where\n> >> it's not needed just for the practicalities of memory management.\n> >>\n> >> I.e. here you use DIR_INIT. In my local patches to fix up the wider\n> >> memory leaks in this area I've got DIR_INIT also using a STRBUF_INIT,\n> >> and DIR_INIT will in turn be referenced by a\n> >> UNPACK_TREES_OPTIONS_INIT. It's quite nice if you're having to\n> >> initialize with \"UNPACK_TREES_OPTIONS_INIT\" have that initialization\n> >> work all the way down the chain, and not need e.g. a manual\n> >> strbuf_init(), dir_init() etc.\n> >\n> > And you can keep using UNPACK_TREES_OPTIONS_INIT, because the\n> > unpack_trees_opts->dir should be initialized to NULL.\n>\n> But I don't want it initialized to NULL, I want DIR_INIT....\n\nWhy? For what purpose? How does that help anything? I've seen you\nsay that you want it, but I haven't yet seen you state how it helps\nyou do anything easier.\n\nWhat I really want, though, is not even to have it be a pointer, but\nto avoid exposing internal implementation details inside a struct that\nis meant to convey the public API. Instead unpack-trees should do\nsomething similar to merge-ort, where it hides all those internal-only\ndetails (by e.g. having a void* priv that happens to point to a struct\nunpack_trees_options_internal, the latter of which is only defined in\nunpack_trees.c). However, I didn't want to go through that work for\njust one member.\n\nBut you've inspired me to check if there are other fields that\nshouldn't be exposed. Turns out that there is a lot of cruft in\nunpack_trees_options that callers shouldn't be messing with (and which\nisn't at all clear to people trying to use the API): cache_bottom,\ndir, msgs, msgs_to_free, nontrivial_merge, skip_sparse_checkout,\nshow_all_errors (!), unpack_rejects, df_conflict_entry, merge_size,\nresult, and perhaps pl. A few of those have gotten slightly\nentangled. And there may have been others that people just started\nsetting because it was an existing field, and now I can't\ndifferentiate between an intentional API usage passing some kind of\ninteresting value and an accidental setting of something meant to be\ninternal.\n\nSo maybe I'll submit some patches on top that rip these direct members\nout of of unpack_trees_options and push them inside some opaque\nstruct.\n\n> >> I removed the dir_init() in ce93a4c6127 (dir.[ch]: replace dir_init()\n> >> with DIR_INIT, 2021-07-01),\n> >\n> > I might be going on a tangent here, but looking at that patch, I'm\n> > worried that dir_init() was buggy and that you perpetuated that bug\n> > with DIR_INIT. Note that dir_struct has a struct strbuf basebuf\n> > member, which neither dir_init() or DIR_INIT initialize properly\n> > (using either strbuf_init() or STRBUF_INIT). As far as I can tell,\n> > dir.c relies on either strbuf_add() calls to just happen to work with\n> > this incorrectly initialized strbuf, or else use the strbuf_init()\n> > call in prep_exclude() to do so, using the following snippet:\n> >\n> > if (!dir->basebuf.buf)\n> > strbuf_init(&dir->basebuf, PATH_MAX);\n> >\n> > However, earlier in that same function we see\n> >\n> > if (stk->baselen <= baselen &&\n> > !strncmp(dir->basebuf.buf, base, stk->baselen))\n> > break;\n> >\n> > So either that function can never have dir->basebuf.buf be NULL and\n> > the strbuf_init() is dead code, or else it's possible for us to\n> > trigger a segfault. If it's the former, it may just be a ticking time\n> > bomb that will transform into the latter with some other change,\n> > because it's not at all obvious to me how dir->basebuf gets\n> > initialized appropriately to avoid that strncmp call. Perhaps there\n> > is some invariant where exclude_stack is only set up by previous calls\n> > to prep_exclude() and those won't set up exclude_stack until first\n> > initializing basebuf. But that really at least deserves a comment\n> > about how we're abusing basebuf, and would probably be cleaner if we\n> > initialized basebuf to STRBUF_INIT.\n>\n> ...because yes, I forgot about that when sending you the diff-on-top,\n> sorry. Yes that's buggy with the diff-on-top I sent you.\n\nThat bug didn't come from the diff-on-top you sent me, it came from\nthe commit already merged to master -- ce93a4c6127 (dir.[ch]: replace\ndir_init() with DIR_INIT, 2021-07-01), merged as part of\nab/struct-init on Jul 16.\n\n> I've got that fixed in the version I have. I.e. first I add a\n> UNPACK_TREES_OPTIONS_INIT macro, then deal with that lazy initialization\n> case (at which point DIR_INIT starts initializing that strbuf), then\n> change the \"dir_struct\" from a pointer to embedding it, and finally fix\n> a memory leak with that new API.\n>\n> WIP patches here:\n> https://github.com/avar/git/compare/avar/post-sanitize-leak-test-mode-add-and-use-revisions-release...avar/post-sanitize-leak-test-mode-unpack-trees-and-dir\n\nYes, that fixes DIR_INIT nicely. Looks good!\n\n> >> but would probably need to bring it back, of\n> >\n> > If you need to bring it back, it's unrelated to my changes here, and\n> > would only be because of the lack of basebuf initialization above.\n>\n> Yes, in this case. I mean that generally speaking I think it's a good\n> pattern to use to have structs be initialized by macros like this,\n> because it means you can embed them N levels deep (as we sometimes do)\n> without having to call functions to initialize them.\n>\n> So yes, in this case as long as DIR_INIT is { 0 } it doesn't matter, but\n> it does as soon as it has a member that needs initialization, and\n> generally speaking for any FOO_INIT that needs a BAR_INIT ...\n\nCallers SHOULD NOT call a function to initialize\nunpacked_trees_opts->dir in my patches. It's a ****BUG**** if they do\nso. So if you're complaining that my changes force callers to also\ninvoke some additional function, then I think you're just not\nunderstanding my patch.\n\nSo, I still see no reason given for wanting opts->dir to be a struct.\nBut maybe we can fix this by just removing 'dir' (and several other\nmembers) from opts, so that callers can't initialize it in any way to\nanything.\n\n> >> course you need some \"release()\" method for the\n> >> UNPACK_TREES_OPTIONS_INIT, which in turn needs to call the dir_release()\n> >> (well, \"dir_clear()\" in that case), and it needs to call\n> >> \"strbuf_release()\". It's just nicer if that boilerplate is all on\n> >> destruction, but not also on struct/object setup.\n> >\n> > The caller should *not* be initializing or tearing down\n> > unpack_trees_options->dir beyond setting that field to NULL; it should\n> > then leave it alone.\n>\n> s/NULL/DIR_INIT/ in my version, but yes.\n>\n> >> We do need that setup in some cases (although a lot could just be\n> >> replaced by lazy initialization), but if we don't....\n> >>\n> >> diff --git a/unpack-trees.c b/unpack-trees.c\n> >> index a7e1712d236..de5cc6cd025 100644\n> >> --- a/unpack-trees.c\n> >> +++ b/unpack-trees.c\n> >> @@ -1694,15 +1694,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> >> static struct cache_entry *dfc;\n> >> struct pattern_list pl;\n> >> int free_pattern_list = 0;\n> >> - struct dir_struct dir = DIR_INIT;\n> >>\n> >> if (o->reset == UNPACK_RESET_INVALID)\n> >> BUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n> >>\n> >> if (len > MAX_UNPACK_TREES)\n> >> die(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n> >> - if (o->dir)\n> >> - BUG(\"o->dir is for internal use only\");\n> >\n> > I think this was an important check that you've tossed without\n> > replacement. Historically, callers set up and tweaked o->dir with\n> > various values. With my patch, we are no longer allowing that..which\n> > introduces a transition problem -- people might have written or are\n> > now writing patches that make new calls of unpack_trees() previous to\n> > this change of mine, but submit them after this change of mine gets\n> > merged. Without this check I added, they'd probably just do a\n> > mechanical `o->dir->` change to `o->dir.` and assume it's good...and\n> > then possibly have ugly bugs to hunt down.\n> >\n> > So, I think it's helpful to have a check that provides an early\n> > warning that tweaking o->dir is not only no longer required, but also\n> > no longer allowed.\n>\n> The compiler will catch any such use of the pointer version on a\n> mis-merge, or do you just mean that the person running into that might\n> get the resolution wrong? I.e. before we could check o->dir being NULL\n> for \"do we have an exclude\", but &o->dir will always be true?\n\nThe compiler catches and reports, and then the human sees the error\nand just transliterates \"o->dir->\" to \"o->dir.\". And then it\ncompiles, and the person assumes they fixed it correctly, but the\ntransliteration was WRONG and has subtle bugs because they had been\nsetting up o->dir with special values and something needs to warn them\nthat they shouldn't be touching o->dir anymore. You removed the\nsafety check that would have let them know that their straightforward\ntransliteration was wrong. I added that safety check intentionally,\nand don't like seeing it ripped out without a replacement.\n\n> >> trace_performance_enter();\n> >> trace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n> >> @@ -1718,9 +1715,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> >> BUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n> >>\n> >> if (!o->preserve_ignored) {\n> >> - o->dir = &dir;\n> >> - o->dir->flags |= DIR_SHOW_IGNORED;\n> >> - setup_standard_excludes(o->dir);\n> >> + o->dir.flags |= DIR_SHOW_IGNORED;\n> >> + setup_standard_excludes(&o->dir);\n> >> }\n> >>\n> >> if (!core_apply_sparse_checkout || !o->update)\n> >> @@ -1884,10 +1880,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> >> done:\n> >> if (free_pattern_list)\n> >> clear_pattern_list(&pl);\n> >> - if (o->dir) {\n> >> - dir_clear(o->dir);\n> >> - o->dir = NULL;\n> >> - }\n> >> + dir_clear(&o->dir);\n> >\n> > Unconditionally calling dir_clear()...\n>\n> As before I'm not sure about bugs in the ad-hoc patch on top, but I\n> don't think this is a bug in my version linked above.\n>\n> I.e. it's zero'd out, and the dir_clear() either ends up calling\n> free(NULL) or tries to loop over 0..N where N will be 0, no?\n>\n> >> trace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n> >> trace_performance_leave(\"unpack_trees\");\n> >> return ret;\n> >> @@ -2153,8 +2146,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n> >> pathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n> >>\n> >> memset(&d, 0, sizeof(d));\n> >> - if (o->dir)\n> >> - d.exclude_per_dir = o->dir->exclude_per_dir;\n> >> + d.exclude_per_dir = o->dir.exclude_per_dir;\n> >> i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n> >> if (i)\n> >> return add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\n> >> @@ -2201,8 +2193,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n> >> if (ignore_case && icase_exists(o, name, len, st))\n> >> return 0;\n> >>\n> >> - if (o->dir &&\n> >> - is_excluded(o->dir, o->src_index, name, &dtype))\n> >> + if (is_excluded(&o->dir, o->src_index, name, &dtype))\n> >\n> > Unconditionally calling is_excluded()...\n>\n> Which will just return \"it's not\", won't it? Just lik dir_clear() deals\n> with an \"empty\" dir_struct. There's existing callers of both with that\n> pattern in e.g. builtin/{add,clean}.c.\n\nThat's a question that someone writing this patch should investigate\nand document. I read through is_excluded() and the functions it\ncalls, and I _think_ it's possibly correct in the current code. But\nit's not trivial to verify. And I'd look a bit closer at it to be\nsure it's correct before making this change.\n\nYeah, I'm being pretty picky here, but this is dir.c we are dealing with.\n\n> Maybe I've missed an edge case, but I think the only reason that \"o->dir\n> &&\" was there was because it was dynamically malloc'd before, but in my\n> version where we'll always have it initialized...\n\nI think after checking the code to verify this, that it deserves being\nmentioned in the commit message (at least given that this is dir.c\nwe're talking about).\n\n> >> /*\n> >> * ce->name is explicitly excluded, so it is Ok to\n> >> * overwrite it.\n> >> diff --git a/unpack-trees.h b/unpack-trees.h\n> >> index 71ffb7eeb0c..a8afbb20170 100644\n> >> --- a/unpack-trees.h\n> >> +++ b/unpack-trees.h\n> >> @@ -5,6 +5,7 @@\n> >> #include \"strvec.h\"\n> >> #include \"string-list.h\"\n> >> #include \"tree-walk.h\"\n> >> +#include \"dir.h\"\n> >>\n> >> #define MAX_UNPACK_TREES MAX_TRAVERSE_TREES\n> >>\n> >> @@ -95,7 +96,7 @@ struct unpack_trees_options {\n> >> struct index_state result;\n> >>\n> >> struct pattern_list *pl; /* for internal use */\n> >> - struct dir_struct *dir; /* for internal use only */\n> >> + struct dir_struct dir; /* for internal use only */\n> >> struct checkout_metadata meta;\n> >> };\n> >>\n> >\n> > Not only did you drop the important safety check that o->dir not be\n> > setup by the caller (which needs to be reinstated in some form), your\n> > solution also involves unconditionally calling dir_clear() and\n> > is_excluded(). It is not clear to me that those calls are safe...and\n> > that they will continue to be safe in the future.\n>\n> It is a common pattern we rely on, e.g. strbuf_release() and various\n> other custom free-like functions generally act as NOOP if they've got\n> nothing to do, just like free()...\n\nYeah, \"common pattern\" is a useful way to want things to behave, and\npeople assuming that in other parts of git's codebase is probably\nfine, but I don't trust dir.c to already behave according to common\npatterns. My mistrust of it is deep enough that I think someone\nshould verify the relevant parts of dir.c do behave that way before\nmaking changes like this, and then explicitly call it out in the\ncommit message when the change is made. dir.c had some weird dragons\nin it. And after being repeatedly sucked back into dir.c problems for\nyears because of weird side effects, and seeing how many piles of\nnearly-cancelling bugs it had, I don't trust it anymore.\n\n> > Even if it is safe\n> > and will continue to be, I don't think this should be squashed into my\n> > patches. I think it should be a separate patch with its own commit\n> > message that explicitly calls out this assumption. Especially since\n> > this is dir.c, which is an area where attempting to fix one very\n> > simple little bug results in years of refactoring and fixing all kinds\n> > of historical messes, sometimes waiting a year and a half for\n> > responses to RFCs/review requests, and where we have to sometimes just\n> > give up on attempting to understand the purpose of various bits of\n> > code and instead rely on the regression tests and hope they are good\n> > enough. I still think that dir.c deserves a little warning at the\n> > top, like the one I suggested in [1].\n> >\n> > [1] https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/\n>\n> *nod* I can always submit something like this afterwards.\n\n:-)\n\n> Just on this series: Perhaps this discussion is a sign that this memory\n> leak fixing should be its own cleanup series where we could hash out any\n> approaches to doing that? I.e. as noted before I realize I'm to blame\n> for suggesting it in the first place, but those parts of these changes\n> don't seem like they're needed by other parts of the series (I tried\n> compiling with the two relevant patches ejected out).\n>\n> Having a humongous set of memory leak fixes locally at this point, I\n> think it's generally not very worth the effort to fix a leak in b()\n> where a() calls b() and b() calls c(), and all of [abc]() are\n> leaking. I.e. often narrowly fixing leaks in b() will lead to different\n> solutions than if you're trying to resolve all of [abc](), as their\n> interaction comes into play.\n\nThe patch we are commenting on isn't a leakfix, and is very much\nintegral to the series for reasons other than actual leakfix (which\noccurred two patches before this one). It's simplifying the API to\nnot require a bunch of boilerplate setup of opts->dir and instead\nrequiring callers to just set a simple boolean INSTEAD (and only if\nthey want non-default behavior, rather than requiring setting\nopts->dir by all those who did want default behavior). That way, when\nI need to make sure several additional callers who should have gotten\nthe default behavior actually get it, they don't have to call a bunch\nof functions to setup and cleanup opts->dir.\n\n> Aside about safety: One thing I'll sometimes do when I'm unsure about\n> those sorts of fixes is to have my new INIT set a new \"sentinel\" field\n> to \"12345\" or whatever, then just BUG() out in an entry point in the API\n> that you can't avoid calling if it's not set like that, e.g. dir_clear()\n> or whatever the setup/work function is.\n>\n> We don't have 100% test coverage, but we usually have at least *some*,\n> and doing that is good about catching e.g. a memset() at a distance, as\n> happens in this code with the merge code embedding the relevant struct\n> and memsetting it, which might be missed in some migration of just a\n> grep for \"struct dir_struct\" or whatever...\n"},{"id":"437752","messageId":"87k0ivpzfx.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"CABPp-BE_aY4smj_b0+Zz=YrURKMniS=DmyMWVc=q2mVDL8zUOg@mail.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-02T08:44:42Z","receivedAt":"2021-10-02T09:07:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 01 2021, Elijah Newren wrote:\n\n> On Fri, Oct 1, 2021 at 1:47 AM Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Thu, Sep 30 2021, Elijah Newren wrote:\n>>\n>> > On Thu, Sep 30, 2021 at 7:15 AM Ævar Arnfjörð Bjarmason\n>> > wrote:\n>> >>\n>> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n>> >>\n>> >> > On Wed, Sep 29, 2021 at 11:32 AM Ævar Arnfjörð Bjarmason\n>> >> > wrote:\n>> >> >>\n>> >> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n>> >> >>\n>> > ...\n>> >> > As per the next patch:\n>> >> >\n>> >> > int unpack_trees(..., struct unpack_trees_options *o)\n>> >> > {\n>> >> > struct dir_struct dir = DIR_INIT;\n>> >> > ...\n>> >> > if (!o->preserve_ignored) {\n>> >> > /* Setup 'dir', make o->dir point to it */\n>> >> > ....\n>> >> > o->dir = &dir;\n>> >> > }\n>> >> > ...\n>> >> > if (o->dir)\n>> >> > /* cleanup */\n>> >> > ....\n>> >> > }\n>> >> >\n>> >> > The caller doesn't touch o->dir (other than initializing it to zeros);\n>> >> > unpack_trees() is wholly responsible for it. I'd kind of like to\n>> >> > entirely remove dir from unpack_trees_options(), but I need a way of\n>> >> > passing it down through all the other functions in unpack-trees.c, and\n>> >> > leaving it in unpack_trees_options seems the easiest way to do so. So\n>> >> > I just marked it as \"for internal use only\".\n>> >>\n>> >> I think I understand *how* it works, I'm puzzled by why you went for\n>> >> this whole level of indirection when you're using a struct on the stack\n>> >> in the end anyway, just ... put that in \"struct unpack_trees_options\"?\n>> >>\n>> >> Anyway, I see I have only myself to blame here, as you added these leak\n>> >> fixes in the v2 in response to some of my offhand comments.\n>> >>\n>> >> FWIW I then went on to do some deeper fixes not just on these leaks but\n>> >> the surrounding leaks, which will be blocked by 2/11 & 05/11 of this\n>> >> topic for a while. I suppose I only have myself to blame :)\n>> >>\n>> >> Below is a patch-on-top that I think makes this whole thing much simpler\n>> >> by doing away with the pointer entirely.\n>> >>\n>> >> I suppose this is also a partial reply to\n>> >> https://lore.kernel.org/git/CABPp-BG_qigBoirMGR-Yk9Niyxt0UmYCEqojsYxbSEarLAmraA@mail.gmail.com/;\n>> >> but I quite dislike this pattern of including a pointer like this where\n>> >> it's not needed just for the practicalities of memory management.\n>> >>\n>> >> I.e. here you use DIR_INIT. In my local patches to fix up the wider\n>> >> memory leaks in this area I've got DIR_INIT also using a STRBUF_INIT,\n>> >> and DIR_INIT will in turn be referenced by a\n>> >> UNPACK_TREES_OPTIONS_INIT. It's quite nice if you're having to\n>> >> initialize with \"UNPACK_TREES_OPTIONS_INIT\" have that initialization\n>> >> work all the way down the chain, and not need e.g. a manual\n>> >> strbuf_init(), dir_init() etc.\n>> >\n>> > And you can keep using UNPACK_TREES_OPTIONS_INIT, because the\n>> > unpack_trees_opts->dir should be initialized to NULL.\n>>\n>> But I don't want it initialized to NULL, I want DIR_INIT....\n>\n> Why? For what purpose? How does that help anything? I've seen you\n> say that you want it, but I haven't yet seen you state how it helps\n> you do anything easier.\n\nUpthread of here in <87k0ixrv23.fsf@evledraar.gmail.com> I linked to WIP\npatches that get rid of lazy initialization in dir.c. I.e. this change:\nhttps://github.com/avar/git/commit/5a18133e927f82a9b6ffcb0c43c3f8657f8836af\n\n[I see from reading ahead a bit that you saw that later...]\n\nI.e. if you initialize it to NULL the embedded strbuf won't be in the\nright state to be used, which is why we initialize it on the fly in that\npart of prep_exclude(). If everything uses DIR_INIT we can just have it\nbe initialized already.\n\nMore generally I think that even if you don't have an immediate use-case\nlike that it makes sense to consistently use the macros, because once\nsomething *does* need a strbuf or whatever you're not needing to do a\nrefactoring of ensuring that everything that hardcoded a \"{ 0 }\" or its\nown memset() is converted to use it, or needing to do initialization\non-the-fly as in that code in prep_exclude()>\n\n> What I really want, though, is not even to have it be a pointer, but\n> to avoid exposing internal implementation details inside a struct that\n> is meant to convey the public API. Instead unpack-trees should do\n> something similar to merge-ort, where it hides all those internal-only\n> details (by e.g. having a void* priv that happens to point to a struct\n> unpack_trees_options_internal, the latter of which is only defined in\n> unpack_trees.c). However, I didn't want to go through that work for\n> just one member.\n\nYes, and I like the part of your change that's not having callers change\n\"struct dir_struct\" but instead just exposing a field in unpack_trees to\nflag the desired behavior, that that code wants to then use a \"struct\ndir_struct\" can just be an implementation detail.\n\n> But you've inspired me to check if there are other fields that\n> shouldn't be exposed. Turns out that there is a lot of cruft in\n> unpack_trees_options that callers shouldn't be messing with (and which\n> isn't at all clear to people trying to use the API): cache_bottom,\n> dir, msgs, msgs_to_free, nontrivial_merge, skip_sparse_checkout,\n> show_all_errors (!), unpack_rejects, df_conflict_entry, merge_size,\n> result, and perhaps pl. A few of those have gotten slightly\n> entangled. And there may have been others that people just started\n> setting because it was an existing field, and now I can't\n> differentiate between an intentional API usage passing some kind of\n> interesting value and an accidental setting of something meant to be\n> internal.\n>\n> So maybe I'll submit some patches on top that rip these direct members\n> out of of unpack_trees_options and push them inside some opaque\n> struct.\n\nSure, that sounds good. I only had a mild objection to doing it in a way\nwhere you'll need that sort of code I removed in the linked commit in\nprep_exclude() because you were trying not to expose that at any cost,\nincluding via some *_INIT macro. I.e. if it's private we can just name\nit \"priv_*\" or have a :\n\n struct dont_touch_this {\n struct dir_struct dir;\n };\n\nWhich are both ways of /messaging/ that it's private, and since the\ntarget audience is just the rest of the git.git codebase I think that\nultimately something that 1) sends the right message 2) makes accidents\npretty much impossible suffices. I.e. you don't accidentally introduce a\nnew API user accessing a field called \"->priv_*\" or\n\"->private_*\". Someone will review those patches...\n\n>> >> I removed the dir_init() in ce93a4c6127 (dir.[ch]: replace dir_init()\n>> >> with DIR_INIT, 2021-07-01),\n>> >\n>> > I might be going on a tangent here, but looking at that patch, I'm\n>> > worried that dir_init() was buggy and that you perpetuated that bug\n>> > with DIR_INIT. Note that dir_struct has a struct strbuf basebuf\n>> > member, which neither dir_init() or DIR_INIT initialize properly\n>> > (using either strbuf_init() or STRBUF_INIT). As far as I can tell,\n>> > dir.c relies on either strbuf_add() calls to just happen to work with\n>> > this incorrectly initialized strbuf, or else use the strbuf_init()\n>> > call in prep_exclude() to do so, using the following snippet:\n>> >\n>> > if (!dir->basebuf.buf)\n>> > strbuf_init(&dir->basebuf, PATH_MAX);\n>> >\n>> > However, earlier in that same function we see\n>> >\n>> > if (stk->baselen <= baselen &&\n>> > !strncmp(dir->basebuf.buf, base, stk->baselen))\n>> > break;\n>> >\n>> > So either that function can never have dir->basebuf.buf be NULL and\n>> > the strbuf_init() is dead code, or else it's possible for us to\n>> > trigger a segfault. If it's the former, it may just be a ticking time\n>> > bomb that will transform into the latter with some other change,\n>> > because it's not at all obvious to me how dir->basebuf gets\n>> > initialized appropriately to avoid that strncmp call. Perhaps there\n>> > is some invariant where exclude_stack is only set up by previous calls\n>> > to prep_exclude() and those won't set up exclude_stack until first\n>> > initializing basebuf. But that really at least deserves a comment\n>> > about how we're abusing basebuf, and would probably be cleaner if we\n>> > initialized basebuf to STRBUF_INIT.\n>>\n>> ...because yes, I forgot about that when sending you the diff-on-top,\n>> sorry. Yes that's buggy with the diff-on-top I sent you.\n>\n> That bug didn't come from the diff-on-top you sent me, it came from\n> the commit already merged to master -- ce93a4c6127 (dir.[ch]: replace\n> dir_init() with DIR_INIT, 2021-07-01), merged as part of\n> ab/struct-init on Jul 16.\n\nAh, I misunderstood you there. I'll look at that / fix it. Sorry.\n\n>> I've got that fixed in the version I have. I.e. first I add a\n>> UNPACK_TREES_OPTIONS_INIT macro, then deal with that lazy initialization\n>> case (at which point DIR_INIT starts initializing that strbuf), then\n>> change the \"dir_struct\" from a pointer to embedding it, and finally fix\n>> a memory leak with that new API.\n>>\n>> WIP patches here:\n>> https://github.com/avar/git/compare/avar/post-sanitize-leak-test-mode-add-and-use-revisions-release...avar/post-sanitize-leak-test-mode-unpack-trees-and-dir\n>\n> Yes, that fixes DIR_INIT nicely. Looks good!\n>\n>> >> but would probably need to bring it back, of\n>> >\n>> > If you need to bring it back, it's unrelated to my changes here, and\n>> > would only be because of the lack of basebuf initialization above.\n>>\n>> Yes, in this case. I mean that generally speaking I think it's a good\n>> pattern to use to have structs be initialized by macros like this,\n>> because it means you can embed them N levels deep (as we sometimes do)\n>> without having to call functions to initialize them.\n>>\n>> So yes, in this case as long as DIR_INIT is { 0 } it doesn't matter, but\n>> it does as soon as it has a member that needs initialization, and\n>> generally speaking for any FOO_INIT that needs a BAR_INIT ...\n>\n> Callers SHOULD NOT call a function to initialize\n> unpacked_trees_opts->dir in my patches. It's a ****BUG**** if they do\n> so. So if you're complaining that my changes force callers to also\n> invoke some additional function, then I think you're just not\n> understanding my patch.\n>\n> So, I still see no reason given for wanting opts->dir to be a struct.\n> But maybe we can fix this by just removing 'dir' (and several other\n> members) from opts, so that callers can't initialize it in any way to\n> anything.\n\nI think your patches are fine as-is, and yes that would be a bug without\nsome of the changes I have in that WIP series.\n\nAs noted in\nhttps://lore.kernel.org/git/87fstlrumj.fsf@evledraar.gmail.com/ I think\nwe're just having a side-discussion about init patterns, and what\ndirection to take these APIs when it comes to that...\n\n>> >> course you need some \"release()\" method for the\n>> >> UNPACK_TREES_OPTIONS_INIT, which in turn needs to call the dir_release()\n>> >> (well, \"dir_clear()\" in that case), and it needs to call\n>> >> \"strbuf_release()\". It's just nicer if that boilerplate is all on\n>> >> destruction, but not also on struct/object setup.\n>> >\n>> > The caller should *not* be initializing or tearing down\n>> > unpack_trees_options->dir beyond setting that field to NULL; it should\n>> > then leave it alone.\n>>\n>> s/NULL/DIR_INIT/ in my version, but yes.\n>>\n>> >> We do need that setup in some cases (although a lot could just be\n>> >> replaced by lazy initialization), but if we don't....\n>> >>\n>> >> diff --git a/unpack-trees.c b/unpack-trees.c\n>> >> index a7e1712d236..de5cc6cd025 100644\n>> >> --- a/unpack-trees.c\n>> >> +++ b/unpack-trees.c\n>> >> @@ -1694,15 +1694,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> >> static struct cache_entry *dfc;\n>> >> struct pattern_list pl;\n>> >> int free_pattern_list = 0;\n>> >> - struct dir_struct dir = DIR_INIT;\n>> >>\n>> >> if (o->reset == UNPACK_RESET_INVALID)\n>> >> BUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n>> >>\n>> >> if (len > MAX_UNPACK_TREES)\n>> >> die(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n>> >> - if (o->dir)\n>> >> - BUG(\"o->dir is for internal use only\");\n>> >\n>> > I think this was an important check that you've tossed without\n>> > replacement. Historically, callers set up and tweaked o->dir with\n>> > various values. With my patch, we are no longer allowing that..which\n>> > introduces a transition problem -- people might have written or are\n>> > now writing patches that make new calls of unpack_trees() previous to\n>> > this change of mine, but submit them after this change of mine gets\n>> > merged. Without this check I added, they'd probably just do a\n>> > mechanical `o->dir->` change to `o->dir.` and assume it's good...and\n>> > then possibly have ugly bugs to hunt down.\n>> >\n>> > So, I think it's helpful to have a check that provides an early\n>> > warning that tweaking o->dir is not only no longer required, but also\n>> > no longer allowed.\n>>\n>> The compiler will catch any such use of the pointer version on a\n>> mis-merge, or do you just mean that the person running into that might\n>> get the resolution wrong? I.e. before we could check o->dir being NULL\n>> for \"do we have an exclude\", but &o->dir will always be true?\n>\n> The compiler catches and reports, and then the human sees the error\n> and just transliterates \"o->dir->\" to \"o->dir.\". And then it\n> compiles, and the person assumes they fixed it correctly, but the\n> transliteration was WRONG and has subtle bugs because they had been\n> setting up o->dir with special values and something needs to warn them\n> that they shouldn't be touching o->dir anymore. You removed the\n> safety check that would have let them know that their straightforward\n> transliteration was wrong. I added that safety check intentionally,\n> and don't like seeing it ripped out without a replacement.\n\nI think that in general we've dealt with that in other places, e.g. if\nwe say have a \"struct strbuf *\" and memzero the containing struct we can\ncheck against NULL as an alias for \"do we have this at all\", but if it's\na \"struct strbuf\" initialized with STRBUF_INIT we'll either need a\n\"have_the_string\" side-member, or (if unambiguous) to check if .len == 0\n(or more nastily perhaps, check .alloc or if it's the slopbuf).\n\n>> >> trace_performance_enter();\n>> >> trace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n>> >> @@ -1718,9 +1715,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> >> BUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n>> >>\n>> >> if (!o->preserve_ignored) {\n>> >> - o->dir = &dir;\n>> >> - o->dir->flags |= DIR_SHOW_IGNORED;\n>> >> - setup_standard_excludes(o->dir);\n>> >> + o->dir.flags |= DIR_SHOW_IGNORED;\n>> >> + setup_standard_excludes(&o->dir);\n>> >> }\n>> >>\n>> >> if (!core_apply_sparse_checkout || !o->update)\n>> >> @@ -1884,10 +1880,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> >> done:\n>> >> if (free_pattern_list)\n>> >> clear_pattern_list(&pl);\n>> >> - if (o->dir) {\n>> >> - dir_clear(o->dir);\n>> >> - o->dir = NULL;\n>> >> - }\n>> >> + dir_clear(&o->dir);\n>> >\n>> > Unconditionally calling dir_clear()...\n>>\n>> As before I'm not sure about bugs in the ad-hoc patch on top, but I\n>> don't think this is a bug in my version linked above.\n>>\n>> I.e. it's zero'd out, and the dir_clear() either ends up calling\n>> free(NULL) or tries to loop over 0..N where N will be 0, no?\n>>\n>> >> trace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n>> >> trace_performance_leave(\"unpack_trees\");\n>> >> return ret;\n>> >> @@ -2153,8 +2146,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n>> >> pathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n>> >>\n>> >> memset(&d, 0, sizeof(d));\n>> >> - if (o->dir)\n>> >> - d.exclude_per_dir = o->dir->exclude_per_dir;\n>> >> + d.exclude_per_dir = o->dir.exclude_per_dir;\n>> >> i = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n>> >> if (i)\n>> >> return add_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\n>> >> @@ -2201,8 +2193,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n>> >> if (ignore_case && icase_exists(o, name, len, st))\n>> >> return 0;\n>> >>\n>> >> - if (o->dir &&\n>> >> - is_excluded(o->dir, o->src_index, name, &dtype))\n>> >> + if (is_excluded(&o->dir, o->src_index, name, &dtype))\n>> >\n>> > Unconditionally calling is_excluded()...\n>>\n>> Which will just return \"it's not\", won't it? Just lik dir_clear() deals\n>> with an \"empty\" dir_struct. There's existing callers of both with that\n>> pattern in e.g. builtin/{add,clean}.c.\n>\n> That's a question that someone writing this patch should investigate\n> and document. I read through is_excluded() and the functions it\n> calls, and I _think_ it's possibly correct in the current code. But\n> it's not trivial to verify. And I'd look a bit closer at it to be\n> sure it's correct before making this change.\n>\n> Yeah, I'm being pretty picky here, but this is dir.c we are dealing with.\n\nI'll make sure to make that clear / test it etc. if & when I do end up\nsubmitting that WIP stuff...\n\n>> Maybe I've missed an edge case, but I think the only reason that \"o->dir\n>> &&\" was there was because it was dynamically malloc'd before, but in my\n>> version where we'll always have it initialized...\n>\n> I think after checking the code to verify this, that it deserves being\n> mentioned in the commit message (at least given that this is dir.c\n> we're talking about).\n\n*nod*. See also my side-thread <877dexrqvg.fsf@evledraar.gmail.com> for\n some exhaustive verification. Will reference that...\n\n>> >> /*\n>> >> * ce->name is explicitly excluded, so it is Ok to\n>> >> * overwrite it.\n>> >> diff --git a/unpack-trees.h b/unpack-trees.h\n>> >> index 71ffb7eeb0c..a8afbb20170 100644\n>> >> --- a/unpack-trees.h\n>> >> +++ b/unpack-trees.h\n>> >> @@ -5,6 +5,7 @@\n>> >> #include \"strvec.h\"\n>> >> #include \"string-list.h\"\n>> >> #include \"tree-walk.h\"\n>> >> +#include \"dir.h\"\n>> >>\n>> >> #define MAX_UNPACK_TREES MAX_TRAVERSE_TREES\n>> >>\n>> >> @@ -95,7 +96,7 @@ struct unpack_trees_options {\n>> >> struct index_state result;\n>> >>\n>> >> struct pattern_list *pl; /* for internal use */\n>> >> - struct dir_struct *dir; /* for internal use only */\n>> >> + struct dir_struct dir; /* for internal use only */\n>> >> struct checkout_metadata meta;\n>> >> };\n>> >>\n>> >\n>> > Not only did you drop the important safety check that o->dir not be\n>> > setup by the caller (which needs to be reinstated in some form), your\n>> > solution also involves unconditionally calling dir_clear() and\n>> > is_excluded(). It is not clear to me that those calls are safe...and\n>> > that they will continue to be safe in the future.\n>>\n>> It is a common pattern we rely on, e.g. strbuf_release() and various\n>> other custom free-like functions generally act as NOOP if they've got\n>> nothing to do, just like free()...\n>\n> Yeah, \"common pattern\" is a useful way to want things to behave, and\n> people assuming that in other parts of git's codebase is probably\n> fine, but I don't trust dir.c to already behave according to common\n> patterns. My mistrust of it is deep enough that I think someone\n> should verify the relevant parts of dir.c do behave that way before\n> making changes like this, and then explicitly call it out in the\n> commit message when the change is made. dir.c had some weird dragons\n> in it. And after being repeatedly sucked back into dir.c problems for\n> years because of weird side effects, and seeing how many piles of\n> nearly-cancelling bugs it had, I don't trust it anymore.\n\n*nod*\n\n>> > Even if it is safe\n>> > and will continue to be, I don't think this should be squashed into my\n>> > patches. I think it should be a separate patch with its own commit\n>> > message that explicitly calls out this assumption. Especially since\n>> > this is dir.c, which is an area where attempting to fix one very\n>> > simple little bug results in years of refactoring and fixing all kinds\n>> > of historical messes, sometimes waiting a year and a half for\n>> > responses to RFCs/review requests, and where we have to sometimes just\n>> > give up on attempting to understand the purpose of various bits of\n>> > code and instead rely on the regression tests and hope they are good\n>> > enough. I still think that dir.c deserves a little warning at the\n>> > top, like the one I suggested in [1].\n>> >\n>> > [1] https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/\n>>\n>> *nod* I can always submit something like this afterwards.\n>\n> :-)\n>\n>> Just on this series: Perhaps this discussion is a sign that this memory\n>> leak fixing should be its own cleanup series where we could hash out any\n>> approaches to doing that? I.e. as noted before I realize I'm to blame\n>> for suggesting it in the first place, but those parts of these changes\n>> don't seem like they're needed by other parts of the series (I tried\n>> compiling with the two relevant patches ejected out).\n>>\n>> Having a humongous set of memory leak fixes locally at this point, I\n>> think it's generally not very worth the effort to fix a leak in b()\n>> where a() calls b() and b() calls c(), and all of [abc]() are\n>> leaking. I.e. often narrowly fixing leaks in b() will lead to different\n>> solutions than if you're trying to resolve all of [abc](), as their\n>> interaction comes into play.\n>\n> The patch we are commenting on isn't a leakfix, and is very much\n> integral to the series for reasons other than actual leakfix (which\n> occurred two patches before this one). It's simplifying the API to\n> not require a bunch of boilerplate setup of opts->dir and instead\n> requiring callers to just set a simple boolean INSTEAD (and only if\n> they want non-default behavior, rather than requiring setting\n> opts->dir by all those who did want default behavior). That way, when\n> I need to make sure several additional callers who should have gotten\n> the default behavior actually get it, they don't have to call a bunch\n> of functions to setup and cleanup opts->dir.\n\nI stand corrected. I have not given this series as a whole any careful\nreview, sorry. This whole side-thread came about because I noticed the\nmemory allocation part of this since it was something I was working in\non in that WIP...\n"},{"id":"437816","messageId":"87sfxhohsj.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"87k0ivpzfx.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-03T22:21:32Z","receivedAt":"2021-10-03T22:38:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Oct 02 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Oct 01 2021, Elijah Newren wrote:\n>\n>> On Fri, Oct 1, 2021 at 1:47 AM Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Thu, Sep 30 2021, Elijah Newren wrote:\n>>>\n>>> > On Thu, Sep 30, 2021 at 7:15 AM Ævar Arnfjörð Bjarmason\n>>> > wrote:\n>>> >>\n>>> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n[...]\n>>> > I might be going on a tangent here, but looking at that patch, I'm\n>>> > worried that dir_init() was buggy and that you perpetuated that bug\n>>> > with DIR_INIT. Note that dir_struct has a struct strbuf basebuf\n>>> > member, which neither dir_init() or DIR_INIT initialize properly\n>>> > (using either strbuf_init() or STRBUF_INIT). As far as I can tell,\n>>> > dir.c relies on either strbuf_add() calls to just happen to work with\n>>> > this incorrectly initialized strbuf, or else use the strbuf_init()\n>>> > call in prep_exclude() to do so, using the following snippet:\n>>> >\n>>> > if (!dir->basebuf.buf)\n>>> > strbuf_init(&dir->basebuf, PATH_MAX);\n>>> >\n>>> > However, earlier in that same function we see\n>>> >\n>>> > if (stk->baselen <= baselen &&\n>>> > !strncmp(dir->basebuf.buf, base, stk->baselen))\n>>> > break;\n>>> >\n>>> > So either that function can never have dir->basebuf.buf be NULL and\n>>> > the strbuf_init() is dead code, or else it's possible for us to\n>>> > trigger a segfault. If it's the former, it may just be a ticking time\n>>> > bomb that will transform into the latter with some other change,\n>>> > because it's not at all obvious to me how dir->basebuf gets\n>>> > initialized appropriately to avoid that strncmp call. Perhaps there\n>>> > is some invariant where exclude_stack is only set up by previous calls\n>>> > to prep_exclude() and those won't set up exclude_stack until first\n>>> > initializing basebuf. But that really at least deserves a comment\n>>> > about how we're abusing basebuf, and would probably be cleaner if we\n>>> > initialized basebuf to STRBUF_INIT.\n>>>\n>>> ...because yes, I forgot about that when sending you the diff-on-top,\n>>> sorry. Yes that's buggy with the diff-on-top I sent you.\n>>\n>> That bug didn't come from the diff-on-top you sent me, it came from\n>> the commit already merged to master -- ce93a4c6127 (dir.[ch]: replace\n>> dir_init() with DIR_INIT, 2021-07-01), merged as part of\n>> ab/struct-init on Jul 16.\n>\n> Ah, I misunderstood you there. I'll look at that / fix it. Sorry.\n\nJust to tie up this loose end: Yes this control flow suck, and I've got\nsome patches to unpack-trees.[ch] & dir.[ch] I'm about to submit to fix\nit. But just to comment on the existing behavior of the code, i.e. your\n(above):\n\n \"So either that function can never have dir->basebuf.buf be NULL and\n the strbuf_init() is dead code, or else it's possible for us to\n trigger a segfault.\".\n\nI hadn't had time to look into it when I said I'd fix it, but now that I\nhave I found thath there's nothing to fix, and this code wasn't buggy\neither before or after my ce93a4c6127 (dir.[ch]: replace dir_init() with\nDIR_INIT, 2021-07-01). I.e. we do have the invariant you mentioned.\n\nThe dir.[ch] API has always relied on the \"struct dir_struct\" being\nzero'd out. First with memset() before your eceba532141 (dir: fix\nproblematic API to avoid memory leaks, 2020-08-18), and after my\nce93a4c6127 with the DIR_INIT, which both amount to the same thing.\n\nWe both missed a caller that used neither dir_init() nor uses DIR_INIT\nnow, but it uses \"{ 0 }\", so it's always zero'd.\n\nNow, of course it being zero'd *would* segfault if you feed\n\"dir->basebuf.buf\" to strncmp() as you note above, but that code isn't\nreachable. The structure of that function is (pseudocode):\n\nvoid prep_exclude(...)\n{\n\tstruct exclude_stack *stk = NULL;\n\t[...]\n\n\twhile ((stk = dir->exclude_stack) != NULL)\n\t\t/* the strncmp() against \"dir->basebuf.buf\" is here */\n\n\t/* maybe we'll early return here */\n\n\tif (!dir->basebuf.buf)\n\t\tstrbuf_init(&dir->basebuf, PATH_MAX);\n\n\t/*\n * Code that sets dir->exclude_stack to non-NULL for the first\n\t * time follows...\n\t */\n}\n\nI.e. dir->exclude_stack is *only* referenced in this function and\ndir_clear() (where we also check it for NULL first).\n\nIt's state management between calls to prep_exclude(). So that that\ninitial while-loop can only be entered the the >1th time prep_exclude()\nis called.\n\nWe'll then either have reached that strbuf_init() already, or if we took\nan early return before the strbuf_init() we couldn't have set\ndir->exclude_stack either. So that \"dir->basebuf.buf\" dereference is\nsafe in either case.\n"},{"id":"437828","messageId":"RFC-patch-v4-01.10-3a3203beee6-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 01/10] t2500: add various tests for nuking untracked files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:50Z","receivedAt":"2021-10-04T01:12:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nNoting that unpack_trees treats reset=1 & update=1 as license to nuke\nuntracked files, I looked for code paths that use this combination and\ntried to generate testcases which demonstrated unintentional loss of\nuntracked files and directories. I found several.\n\nI also include testcases for `git reset --{hard,merge,keep}`. A hard\nreset is perhaps the most direct test of unpack_tree's reset=1 behavior,\nbut we cannot make `git reset --hard` preserve untracked files without\nsome migration work.\n\nAlso, the two commands `checkout --force` (because of the --force) and\n`read-tree --reset` (because it's plumbing and we need to keep it\nbackward compatible) were left out as we expect those to continue\nremoving untracked files and directories.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n 1 file changed, 244 insertions(+)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nnew file mode 100755\nindex 00000000000..2412d121ea8\n--- /dev/null\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -0,0 +1,244 @@\n+#!/bin/sh\n+\n+test_description='Test handling of overwriting untracked files'\n+\n+. ./test-lib.sh\n+\n+test_setup_reset () {\n+\tgit init reset_$1 &&\n+\t(\n+\t\tcd reset_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\tgit branch stable &&\n+\t\tgit branch work &&\n+\n+\t\tgit checkout work &&\n+\t\ttest_commit foo &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_success 'reset --hard will nuke untracked files/dirs' '\n+\ttest_setup_reset hard &&\n+\t(\n+\t\tcd reset_hard &&\n+\t\tgit ls-tree -r stable &&\n+\t\tgit log --all --name-status --oneline &&\n+\t\tgit ls-tree -r work &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\techo foo >expect &&\n+\n+\t\tgit reset --hard work &&\n+\n+\t\t# check that untracked directory foo.t/ was nuked\n+\t\ttest_path_is_file foo.t &&\n+\t\ttest_cmp expect foo.t\n+\t)\n+'\n+\n+test_expect_success 'reset --merge will preserve untracked files/dirs' '\n+\ttest_setup_reset merge &&\n+\t(\n+\t\tcd reset_merge &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating .foo.t. would lose untracked files\" error\n+\t)\n+'\n+\n+test_expect_success 'reset --keep will preserve untracked files/dirs' '\n+\ttest_setup_reset keep &&\n+\t(\n+\t\tcd reset_keep &&\n+\n+\t\tmkdir foo.t &&\n+\t\techo precious >foo.t/file &&\n+\t\tcp foo.t/file expect &&\n+\n+\t\ttest_must_fail git reset --merge work 2>error &&\n+\t\ttest_cmp expect foo.t/file &&\n+\t\tgrep \"Updating.*foo.t.*would lose untracked files\" error\n+\t)\n+'\n+\n+test_setup_checkout_m () {\n+\tgit init checkout &&\n+\t(\n+\t\tcd checkout &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines file has some >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit branch stable &&\n+\n+\t\tgit switch -c work &&\n+\t\techo stuff >notes.txt &&\n+\t\ttest_write_lines file has some words >filler &&\n+\t\tgit add notes.txt filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tgit checkout stable\n+\t)\n+}\n+\n+test_expect_failure 'checkout -m does not nuke untracked file' '\n+\ttest_setup_checkout_m &&\n+\t(\n+\t\tcd checkout &&\n+\n+\t\t# Tweak filler\n+\t\ttest_write_lines this file has some >filler &&\n+\t\t# Make an untracked file, save its contents in \"expect\"\n+\t\techo precious >notes.txt &&\n+\t\tcp notes.txt expect &&\n+\n+\t\ttest_must_fail git checkout -m work &&\n+\t\ttest_cmp expect notes.txt\n+\t)\n+'\n+\n+test_setup_sequencing () {\n+\tgit init sequencing_$1 &&\n+\t(\n+\t\tcd sequencing_$1 &&\n+\t\ttest_commit init &&\n+\n+\t\ttest_write_lines this file has some words >filler &&\n+\t\tgit add filler &&\n+\t\tgit commit -m filler &&\n+\n+\t\tmkdir -p foo/bar &&\n+\t\ttest_commit foo/bar/baz &&\n+\n+\t\tgit branch simple &&\n+\t\tgit branch fooey &&\n+\n+\t\tgit checkout fooey &&\n+\t\tgit rm foo/bar/baz.t &&\n+\t\techo stuff >>filler &&\n+\t\tgit add -u &&\n+\t\tgit commit -m \"changes\" &&\n+\n+\t\tgit checkout simple &&\n+\t\techo items >>filler &&\n+\t\techo newstuff >>newfile &&\n+\t\tgit add filler newfile &&\n+\t\tgit commit -m another\n+\t)\n+}\n+\n+test_expect_failure 'git rebase --abort and untracked files' '\n+\ttest_setup_sequencing rebase_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_abort_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\ttest_must_fail git rebase simple &&\n+\n+\t\tcat init.t &&\n+\t\tgit rm init.t &&\n+\t\techo precious >init.t &&\n+\t\tcp init.t expect &&\n+\t\tgit status --porcelain &&\n+\t\ttest_must_fail git rebase --abort &&\n+\t\ttest_cmp expect init.t\n+\t)\n+'\n+\n+test_expect_failure 'git rebase fast forwarding and untracked files' '\n+\ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_fast_forward_and_untracked &&\n+\t\tgit checkout init &&\n+\t\techo precious >filler &&\n+\t\tcp filler expect &&\n+\t\ttest_must_fail git rebase init simple &&\n+\t\ttest_cmp expect filler\n+\t)\n+'\n+\n+test_expect_failure 'git rebase --autostash and untracked files' '\n+\ttest_setup_sequencing rebase_autostash_and_untracked &&\n+\t(\n+\t\tcd sequencing_rebase_autostash_and_untracked &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit rebase --autostash init &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git stash and untracked files' '\n+\ttest_setup_sequencing stash_and_untracked_files &&\n+\t(\n+\t\tcd sequencing_stash_and_untracked_files &&\n+\t\tgit checkout simple &&\n+\t\tgit rm filler &&\n+\t\tmkdir filler &&\n+\t\techo precious >filler/file &&\n+\t\tcp filler/file expect &&\n+\t\tgit status --porcelain &&\n+\t\tgit stash push &&\n+\t\tgit status --porcelain &&\n+\t\ttest_path_is_file filler/file\n+\t)\n+'\n+\n+test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+\ttest_setup_sequencing am_abort_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_abort_and_untracked &&\n+\t\tgit format-patch -1 --stdout fooey >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete the conflicted file; we will stage and commit it later\n+\t\trm filler &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir filler &&\n+\t\techo foo >filler/file1 &&\n+\t\techo bar >filler/file2 &&\n+\n+\t\ttest_must_fail git am --abort 2>errors &&\n+\t\ttest_path_is_dir filler &&\n+\t\tgrep \"Updating .filler. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+\ttest_setup_sequencing am_skip_and_untracked &&\n+\t(\n+\t\tcd sequencing_am_skip_and_untracked &&\n+\t\tgit checkout fooey &&\n+\t\tgit format-patch -1 --stdout simple >changes.mbox &&\n+\t\ttest_must_fail git am --3way changes.mbox &&\n+\n+\t\t# Delete newfile\n+\t\trm newfile &&\n+\n+\t\t# Put an unrelated untracked directory there\n+\t\tmkdir newfile &&\n+\t\techo foo >newfile/file1 &&\n+\t\techo bar >newfile/file2 &&\n+\n+\t\t# Change our mind about resolutions, just skip this patch\n+\t\ttest_must_fail git am --skip 2>errors &&\n+\t\ttest_path_is_dir newfile &&\n+\t\tgrep \"Updating .newfile. would lose untracked files in it\" errors\n+\t)\n+'\n+\n+test_done\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437829","messageId":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"pull.1036.v3.git.1632760428.gitgitgadget@gmail.com","subject":"[RFC PATCH v4 00/10] Fix various issues around removal of untracked files/directories","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:49Z","receivedAt":"2021-10-04T01:12:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This is an RFC proposed v4 of Elijah's en/removing-untracked-fixes\nseries[1] based on top of my memory leak fixes in the \"unpack-trees\" &\n\"dir\" APIs[2].\n\nAs noted in [2] Elijah and I have been having a back & forth about the\napproach his series takes to fixing memory leaks in those APIs. I\nthink submitting working code is more productive than continuing that\npoint-by-point discussion, so here we are.\n\nI've avoided making any changes to this series except those narrowly\nrequired to rebase it on top of mine, and to those parts of Elijah's\ncommit messages that became outdated as a result. In particular\n3/10[3]'s is significantly changed, as much of its commit message\ndicusses complexities that have gone away due to my preceding\nseries[2].\n\nThe \"make dir an internal-only struct\" has been replaced by a commit\nthat renames that struct member from \"dir\" to \"private_dir\". I think\neven that is unnecessary as argued in [4], but I think the judgement\nthat something must be done to address that is Elijah's design\ndecision, so I did my best to retain it.\n\nI did drop the dynamic allocation & it being a pointer, since with my\npreceding [2] and subsequent unsubmitted memory leak fixes I've got on\ntop having it be embedded in \"struct unpack_trees_options\" makes\nthings easier to manage.\n\nHavingn read through all this code quite thoroughly at this point I do\nhave other comments on it, but I'll reserve those until we've found\nout what direction we're going forward with vis-a-vis what this will\nbe based on top of.\n\nI'm (obviously) hoping for an answer of either on top of my series[2],\nor alternatively that Elijah's series can stick to introducing the\n\"preserve_ignored\" flag, but not change how the memory\nmanagement/name/type of the embedded \"dir\" happens (and we could thus\nproceed in parallel).\n\nBut I'll hold off on any such general comments until we've got a way\nforward with this, since if I start commenting inline on patches in\nElijah's v3, or this RFC-v4 on something unrelated to this proposed\nre-arrangement that'll likely just confuse things, particularly as\nsome of those comments would be different depending on the base of his\nv3 v.s. my series[2] in this RFC v4.\n\n1. https://lore.kernel.org/git/pull.1036.v3.git.1632760428.gitgitgadget@gmail.com\n2. https://lore.kernel.org/git/cover-00.10-00000000000-20211004T002226Z-avarab@gmail.com/\n3. https://lore.kernel.org/git/RFC-patch-v4-03.10-739e9b871c4-20211004T004902Z-avarab@gmail.com\n4. https://lore.kernel.org/git/87k0ivpzfx.fsf@evledraar.gmail.com\n\nElijah Newren (10):\n t2500: add various tests for nuking untracked files\n read-tree, merge-recursive: overwrite ignored files by default\n unpack-trees: introduce preserve_ignored to unpack_trees_options\n unpack-trees: rename \"dir\" to \"private_dir\"\n Remove ignored files by default when they are in the way\n Change unpack_trees' 'reset' flag into an enum\n unpack-trees: avoid nuking untracked dir in way of unmerged file\n unpack-trees: avoid nuking untracked dir in way of locally deleted\n file\n Comment important codepaths regarding nuking untracked files/dirs\n Documentation: call out commands that nuke untracked files/directories\n\n Documentation/git-checkout.txt | 5 +-\n Documentation/git-read-tree.txt | 23 +--\n Documentation/git-reset.txt | 3 +-\n builtin/am.c | 3 +-\n builtin/checkout.c | 9 +-\n builtin/clone.c | 1 +\n builtin/merge.c | 1 +\n builtin/read-tree.c | 23 ++-\n builtin/reset.c | 10 +-\n builtin/stash.c | 5 +-\n builtin/submodule--helper.c | 4 +\n contrib/rerere-train.sh | 2 +-\n merge-ort.c | 5 +-\n merge-recursive.c | 5 +-\n merge.c | 6 +-\n reset.c | 3 +-\n sequencer.c | 1 +\n submodule.c | 1 +\n t/t1013-read-tree-submodule.sh | 1 -\n t/t2500-untracked-overwriting.sh | 244 +++++++++++++++++++++++++++++++\n t/t7112-reset-submodule.sh | 1 -\n unpack-trees.c | 59 +++++++-\n unpack-trees.h | 16 +-\n 23 files changed, 362 insertions(+), 69 deletions(-)\n create mode 100755 t/t2500-untracked-overwriting.sh\n\nRange-diff against v3:\n 1: 66270ffc74e ! 1: 3a3203beee6 t2500: add various tests for nuking untracked files\n @@ Commit message\n removing untracked files and directories.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## t/t2500-untracked-overwriting.sh (new) ##\n @@\n 2: 0c74285b253 < -: ----------- checkout, read-tree: fix leak of unpack_trees_options.dir\n 3: 2501a0c552a ! 2: 8e5f4006604 read-tree, merge-recursive: overwrite ignored files by default\n @@ Commit message\n The read-tree changes happen to fix a bug in t1013.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## Documentation/git-read-tree.txt ##\n @@ Documentation/git-read-tree.txt: SYNOPSIS\n @@ builtin/read-tree.c: static int list_tree(struct object_id *oid)\n \tNULL\n };\n \n -@@ builtin/read-tree.c: static int index_output_cb(const struct option *opt, const char *arg,\n - static int exclude_per_directory_cb(const struct option *opt, const char *arg,\n - \t\t\t\t int unset)\n - {\n --\tstruct dir_struct *dir;\n - \tstruct unpack_trees_options *opts;\n - \n - \tBUG_ON_OPT_NEG(unset);\n +@@ builtin/read-tree.c: static int exclude_per_directory_cb(const struct option *opt, const char *arg,\n \n \topts = (struct unpack_trees_options *)opt->value;\n \n --\tif (opts->dir)\n +-\tif (opts->dir.exclude_per_dir)\n -\t\tdie(\"more than one --exclude-per-directory given.\");\n -\n --\tdir = xcalloc(1, sizeof(*opts->dir));\n --\tdir->flags |= DIR_SHOW_IGNORED;\n --\tdir->exclude_per_dir = arg;\n --\topts->dir = dir;\n +-\topts->dir.flags |= DIR_SHOW_IGNORED;\n +-\topts->dir.exclude_per_dir = arg;\n -\t/* We do not need to nor want to do read-directory\n -\t * here; we are merely interested in reusing the\n -\t * per directory ignore stack mechanism.\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n \tif ((opts.update || opts.index_only) && !opts.merge)\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n --\tif ((opts.dir && !opts.update))\n +-\tif ((opts.dir.exclude_per_dir && !opts.update))\n -\t\tdie(\"--exclude-per-directory is meaningless unless -u\");\n +\tif (opts.update && !opts.reset) {\n -+\t\tCALLOC_ARRAY(opts.dir, 1);\n -+\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opts.dir);\n ++\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n ++\t\tsetup_standard_excludes(&opts.dir);\n +\t}\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n \n ## merge-recursive.c ##\n @@ merge-recursive.c: static int unpack_trees_start(struct merge_options *opt,\n - \tmemset(&opt->priv->unpack_opts, 0, sizeof(opt->priv->unpack_opts));\n + \tunpack_trees_options_init(&opt->priv->unpack_opts);\n \tif (opt->priv->call_depth)\n \t\topt->priv->unpack_opts.index_only = 1;\n -\telse\n +\telse {\n \t\topt->priv->unpack_opts.update = 1;\n -+\t\t/* FIXME: should only do this if !overwrite_ignore */\n -+\t\tCALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n -+\t\topt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(opt->priv->unpack_opts.dir);\n ++\t\topt->priv->unpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n ++\t\tsetup_standard_excludes(&opt->priv->unpack_opts.dir);\n +\t}\n \topt->priv->unpack_opts.merge = 1;\n \topt->priv->unpack_opts.head_idx = 2;\n \topt->priv->unpack_opts.fn = threeway_merge;\n -@@ merge-recursive.c: static int unpack_trees_start(struct merge_options *opt,\n - \tinit_tree_desc_from_tree(t+2, merge);\n - \n - \trc = unpack_trees(3, t, &opt->priv->unpack_opts);\n -+\tif (opt->priv->unpack_opts.dir) {\n -+\t\tdir_clear(opt->priv->unpack_opts.dir);\n -+\t\tFREE_AND_NULL(opt->priv->unpack_opts.dir);\n -+\t}\n - \tcache_tree_free(&opt->repo->index->cache_tree);\n - \n - \t/*\n \n ## t/t1013-read-tree-submodule.sh ##\n @@ t/t1013-read-tree-submodule.sh: test_description='read-tree can handle submodules'\n 4: f1a0700e598 ! 3: 739e9b871c4 unpack-trees: introduce preserve_ignored to unpack_trees_options\n @@ Commit message\n \n Currently, every caller of unpack_trees() that wants to ensure ignored\n files are overwritten by default needs to:\n - * allocate unpack_trees_options.dir\n - * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir->flags\n - * call setup_standard_excludes\n - AND then after the call to unpack_trees() needs to\n - * call dir_clear()\n - * deallocate unpack_trees_options.dir\n - That's a fair amount of boilerplate, and every caller uses identical\n - code. Make this easier by instead introducing a new boolean value where\n +\n + * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir.flags\n + * call setup_standard_excludes(&unpack_trees_options.dir)\n +\n + Avoid that boilerplate by introducing a new boolean value where\n the default value (0) does what we want so that new callers of\n unpack_trees() automatically get the appropriate behavior. And move all\n the handling of unpack_trees_options.dir into unpack_trees() itself.\n @@ Commit message\n behavior we want.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/am.c ##\n @@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n @@ builtin/checkout.c: static int merge_working_tree(const struct checkout_opts *op\n \t\t\t\t &new_branch_info->commit->object.oid :\n \t\t\t\t &new_branch_info->oid, NULL);\n -\t\tif (opts->overwrite_ignore) {\n --\t\t\ttopts.dir = xcalloc(1, sizeof(*topts.dir));\n --\t\t\ttopts.dir->flags |= DIR_SHOW_IGNORED;\n --\t\t\tsetup_standard_excludes(topts.dir);\n +-\t\t\ttopts.dir.flags |= DIR_SHOW_IGNORED;\n +-\t\t\tsetup_standard_excludes(&topts.dir);\n -\t\t}\n +\t\ttopts.preserve_ignored = !opts->overwrite_ignore;\n \t\ttree = parse_tree_indirect(old_branch_info->commit ?\n \t\t\t\t\t &old_branch_info->commit->object.oid :\n \t\t\t\t\t the_hash_algo->empty_tree);\n -@@ builtin/checkout.c: static int merge_working_tree(const struct checkout_opts *opts,\n - \t\tinit_tree_desc(&trees[1], tree->buffer, tree->size);\n - \n - \t\tret = unpack_trees(2, trees, &topts);\n --\t\tif (topts.dir) {\n --\t\t\tdir_clear(topts.dir);\n --\t\t\tFREE_AND_NULL(topts.dir);\n --\t\t}\n - \t\tclear_unpack_trees_porcelain(&topts);\n - \t\tif (ret == -1) {\n - \t\t\t/*\n \n ## builtin/clone.c ##\n @@ builtin/clone.c: static int checkout(int submodule_progress)\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n -\tif (opts.update && !opts.reset) {\n --\t\tCALLOC_ARRAY(opts.dir, 1);\n --\t\topts.dir->flags |= DIR_SHOW_IGNORED;\n --\t\tsetup_standard_excludes(opts.dir);\n +-\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n +-\t\tsetup_standard_excludes(&opts.dir);\n -\t}\n +\tif (opts.update && !opts.reset)\n +\t\topts.preserve_ignored = 0;\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n -@@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n - \tif (unpack_trees(nr_trees, t, &opts))\n - \t\treturn 128;\n - \n --\tif (opts.dir) {\n --\t\tdir_clear(opts.dir);\n --\t\tFREE_AND_NULL(opts.dir);\n --\t}\n --\n - \tif (opts.debug_unpack || opts.dry_run)\n - \t\treturn 0; /* do not write the index out */\n - \n \n ## builtin/reset.c ##\n @@ builtin/reset.c: static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n @@ builtin/stash.c: static int reset_tree(struct object_id *i_tree, int update, int\n +\t\topts.preserve_ignored = 1;\n \topts.fn = oneway_merge;\n \n - \tif (unpack_trees(nr_trees, t, &opts))\n + \tif (unpack_trees(nr_trees, t, &opts)) {\n \n ## merge-ort.c ##\n @@ merge-ort.c: static int checkout(struct merge_options *opt,\n @@ merge-ort.c: static int checkout(struct merge_options *opt,\n \tunpack_opts.verbose_update = (opt->verbosity > 2);\n \tunpack_opts.fn = twoway_merge;\n -\tif (1/* FIXME: opts->overwrite_ignore*/) {\n --\t\tCALLOC_ARRAY(unpack_opts.dir, 1);\n --\t\tunpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n --\t\tsetup_standard_excludes(unpack_opts.dir);\n +-\t\tunpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n +-\t\tsetup_standard_excludes(&unpack_opts.dir);\n -\t}\n +\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n \tparse_tree(prev);\n \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n \tparse_tree(next);\n -@@ merge-ort.c: static int checkout(struct merge_options *opt,\n - \n - \tret = unpack_trees(2, trees, &unpack_opts);\n - \tclear_unpack_trees_porcelain(&unpack_opts);\n --\tdir_clear(unpack_opts.dir);\n --\tFREE_AND_NULL(unpack_opts.dir);\n - \treturn ret;\n - }\n - \n \n ## merge-recursive.c ##\n @@ merge-recursive.c: static int unpack_trees_start(struct merge_options *opt,\n + \t\topt->priv->unpack_opts.index_only = 1;\n \telse {\n \t\topt->priv->unpack_opts.update = 1;\n - \t\t/* FIXME: should only do this if !overwrite_ignore */\n --\t\tCALLOC_ARRAY(opt->priv->unpack_opts.dir, 1);\n --\t\topt->priv->unpack_opts.dir->flags |= DIR_SHOW_IGNORED;\n --\t\tsetup_standard_excludes(opt->priv->unpack_opts.dir);\n +-\t\topt->priv->unpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n +-\t\tsetup_standard_excludes(&opt->priv->unpack_opts.dir);\n ++\t\t/* FIXME: should only do this if !overwrite_ignore */\n +\t\topt->priv->unpack_opts.preserve_ignored = 0;\n \t}\n \topt->priv->unpack_opts.merge = 1;\n \topt->priv->unpack_opts.head_idx = 2;\n -@@ merge-recursive.c: static int unpack_trees_start(struct merge_options *opt,\n - \tinit_tree_desc_from_tree(t+2, merge);\n - \n - \trc = unpack_trees(3, t, &opt->priv->unpack_opts);\n --\tif (opt->priv->unpack_opts.dir) {\n --\t\tdir_clear(opt->priv->unpack_opts.dir);\n --\t\tFREE_AND_NULL(opt->priv->unpack_opts.dir);\n --\t}\n - \tcache_tree_free(&opt->repo->index->cache_tree);\n - \n - \t/*\n \n ## merge.c ##\n @@ merge.c: int checkout_fast_forward(struct repository *r,\n - \tstruct unpack_trees_options opts;\n - \tstruct tree_desc t[MAX_UNPACK_TREES];\n - \tint i, nr_trees = 0;\n --\tstruct dir_struct dir = DIR_INIT;\n - \tstruct lock_file lock_file = LOCK_INIT;\n - \n - \trefresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL);\n -@@ merge.c: int checkout_fast_forward(struct repository *r,\n + \t\tinit_tree_desc(t+i, trees[i]->buffer, trees[i]->size);\n \t}\n \n - \tmemset(&opts, 0, sizeof(opts));\n -\tif (overwrite_ignore) {\n --\t\tdir.flags |= DIR_SHOW_IGNORED;\n --\t\tsetup_standard_excludes(&dir);\n --\t\topts.dir = &dir;\n +-\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n +-\t\tsetup_standard_excludes(&opts.dir);\n -\t}\n +-\n +\topts.preserve_ignored = !overwrite_ignore;\n - \n \topts.head_idx = 1;\n \topts.src_index = r->index;\n -@@ merge.c: int checkout_fast_forward(struct repository *r,\n - \t\tclear_unpack_trees_porcelain(&opts);\n - \t\treturn -1;\n - \t}\n --\tdir_clear(&dir);\n - \tclear_unpack_trees_porcelain(&opts);\n - \n - \tif (write_locked_index(r->index, &lock_file, COMMIT_LOCK))\n + \topts.dst_index = r->index;\n \n ## reset.c ##\n @@ reset.c: int reset_head(struct repository *r, struct object_id *oid, const char *action,\n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpac\n \t}\n \n +\tif (!o->preserve_ignored) {\n -+\t\tCALLOC_ARRAY(o->dir, 1);\n -+\t\to->dir->flags |= DIR_SHOW_IGNORED;\n -+\t\tsetup_standard_excludes(o->dir);\n ++\t\to->dir.flags |= DIR_SHOW_IGNORED;\n ++\t\tsetup_standard_excludes(&o->dir);\n +\t}\n +\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n -@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n - done:\n - \tif (free_pattern_list)\n - \t\tclear_pattern_list(&pl);\n -+\tif (o->dir) {\n -+\t\tdir_clear(o->dir);\n -+\t\tFREE_AND_NULL(o->dir);\n -+\t}\n - \ttrace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n - \ttrace_performance_leave(\"unpack_trees\");\n - \treturn ret;\n \n ## unpack-trees.h ##\n @@ unpack-trees.h: struct unpack_trees_options {\n 5: 0d119142778 ! 4: 296c1e03673 unpack-trees: make dir an internal-only struct\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - unpack-trees: make dir an internal-only struct\n + unpack-trees: rename \"dir\" to \"private_dir\"\n \n - Avoid accidental misuse or confusion over ownership by clearly making\n - unpack_trees_options.dir an internal-only variable.\n + Until the introduction of the \"preserve_ignored\" flag in the preceding\n + commit callers who wanted its behavior needed to adjust \"dir.flags\"\n + and call setup_standard_excludes() themselves.\n +\n + Now that we have no external users of \"dir\" anymore let's rename it to\n + \"private_dir\" and add a comment indicating that we'd like it not to be\n + messed with by external callers. This should avoid avoid accidental\n + misuse or confusion over its ownership.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## unpack-trees.c ##\n -@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n - \tstatic struct cache_entry *dfc;\n - \tstruct pattern_list pl;\n - \tint free_pattern_list = 0;\n -+\tstruct dir_struct dir = DIR_INIT;\n +@@ unpack-trees.c: void unpack_trees_options_init(struct unpack_trees_options *o)\n + void unpack_trees_options_release(struct unpack_trees_options *opts)\n + {\n + \tstrvec_clear(&opts->msgs_to_free);\n +-\tdir_clear(&opts->dir);\n ++\tdir_clear(&opts->private_dir);\n + }\n \n - \tif (len > MAX_UNPACK_TREES)\n - \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n -+\tif (o->dir)\n -+\t\tBUG(\"o->dir is for internal use only\");\n - \n - \ttrace_performance_enter();\n - \ttrace2_region_enter(\"unpack_trees\", \"unpack_trees\", the_repository);\n + static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t}\n \n \tif (!o->preserve_ignored) {\n --\t\tCALLOC_ARRAY(o->dir, 1);\n -+\t\to->dir = &dir;\n - \t\to->dir->flags |= DIR_SHOW_IGNORED;\n - \t\tsetup_standard_excludes(o->dir);\n - \t}\n -@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n - \t\tclear_pattern_list(&pl);\n - \tif (o->dir) {\n - \t\tdir_clear(o->dir);\n --\t\tFREE_AND_NULL(o->dir);\n -+\t\to->dir = NULL;\n +-\t\to->dir.flags |= DIR_SHOW_IGNORED;\n +-\t\tsetup_standard_excludes(&o->dir);\n ++\t\to->private_dir.flags |= DIR_SHOW_IGNORED;\n ++\t\tsetup_standard_excludes(&o->private_dir);\n \t}\n - \ttrace2_region_leave(\"unpack_trees\", \"unpack_trees\", the_repository);\n - \ttrace_performance_leave(\"unpack_trees\");\n + \n + \tif (!core_apply_sparse_checkout || !o->update)\n +@@ unpack-trees.c: static int verify_clean_subdirectory(const struct cache_entry *ce,\n + \t */\n + \tpathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n + \n +-\td.exclude_per_dir = o->dir.exclude_per_dir;\n ++\td.exclude_per_dir = o->private_dir.exclude_per_dir;\n + \ti = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n + \tdir_clear(&d);\n + \tfree(pathbuf);\n +@@ unpack-trees.c: static int check_ok_to_remove(const char *name, int len, int dtype,\n + \tif (ignore_case && icase_exists(o, name, len, st))\n + \t\treturn 0;\n + \n +-\tif (is_excluded(&o->dir, o->src_index, name, &dtype))\n ++\tif (is_excluded(&o->private_dir, o->src_index, name, &dtype))\n + \t\t/*\n + \t\t * ce->name is explicitly excluded, so it is Ok to\n + \t\t * overwrite it.\n \n ## unpack-trees.h ##\n @@ unpack-trees.h: struct unpack_trees_options {\n \t\t dry_run;\n \tconst char *prefix;\n \tint cache_bottom;\n --\tstruct dir_struct *dir;\n +-\tstruct dir_struct dir;\n ++\tstruct dir_struct private_dir; /* for internal use only */\n \tstruct pathspec *pathspec;\n \tmerge_fn_t fn;\n \tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n @@ unpack-trees.h: struct unpack_trees_options {\n - \tstruct index_state result;\n \n - \tstruct pattern_list *pl; /* for internal use */\n -+\tstruct dir_struct *dir; /* for internal use only */\n - \tstruct checkout_metadata meta;\n - };\n + #define UNPACK_TREES_OPTIONS_INIT { \\\n + \t.msgs_to_free = STRVEC_INIT, \\\n +-\t.dir = DIR_INIT, \\\n ++\t.private_dir = DIR_INIT, \\\n + }\n + void unpack_trees_options_init(struct unpack_trees_options *o);\n \n 6: b7fe354efff ! 5: 27496506430 Remove ignored files by default when they are in the way\n @@ Commit message\n Incidentally, this fixes a test failure in t7112.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/am.c ##\n @@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n @@ builtin/stash.c: static int reset_tree(struct object_id *i_tree, int update, int\n +\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = oneway_merge;\n \n - \tif (unpack_trees(nr_trees, t, &opts))\n + \tif (unpack_trees(nr_trees, t, &opts)) {\n \n ## merge-ort.c ##\n @@ merge-ort.c: static int checkout(struct merge_options *opt,\n 7: 9eb20121fc3 ! 6: 7b539a120b9 Change unpack_trees' 'reset' flag into an enum\n @@ Commit message\n [1] https://lore.kernel.org/git/15dad590-087e-5a48-9238-5d2826950506@gmail.com/\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/am.c ##\n @@ builtin/am.c: static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n @@ t/t2500-untracked-overwriting.sh: test_expect_failure 'git rebase --abort and un\n \n ## unpack-trees.c ##\n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n + \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n - \tstruct dir_struct dir = DIR_INIT;\n \n +\tif (o->reset == UNPACK_RESET_INVALID)\n +\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n +\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n - \tif (o->dir)\n + \n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tensure_full_index(o->dst_index);\n \t}\n @@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpac\n +\t\tBUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n +\n \tif (!o->preserve_ignored) {\n - \t\to->dir = &dir;\n - \t\to->dir->flags |= DIR_SHOW_IGNORED;\n + \t\to->private_dir.flags |= DIR_SHOW_IGNORED;\n + \t\tsetup_standard_excludes(&o->private_dir);\n @@ unpack-trees.c: static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n @@ unpack-trees.c: static int verify_absent_1(const struct cache_entry *ce,\n ## unpack-trees.h ##\n @@ unpack-trees.h: void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n */\n - void clear_unpack_trees_porcelain(struct unpack_trees_options *opts);\n + void unpack_trees_options_release(struct unpack_trees_options *opts);\n \n +enum unpack_trees_reset_type {\n +\tUNPACK_RESET_NONE = 0, /* traditional \"false\" value; still valid */\n @@ unpack-trees.h: struct unpack_trees_options {\n +\tenum unpack_trees_reset_type reset;\n \tconst char *prefix;\n \tint cache_bottom;\n - \tstruct pathspec *pathspec;\n + \tstruct dir_struct private_dir; /* for internal use only */\n 8: e4c42d43b09 ! 7: b6769f629ae unpack-trees: avoid nuking untracked dir in way of unmerged file\n @@ Commit message\n unpack-trees: avoid nuking untracked dir in way of unmerged file\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## t/t2500-untracked-overwriting.sh ##\n @@ t/t2500-untracked-overwriting.sh: test_expect_failure 'git stash and untracked files' '\n 9: 1a770681704 ! 8: 10a7cbf049e unpack-trees: avoid nuking untracked dir in way of locally deleted file\n @@ Commit message\n unpack-trees: avoid nuking untracked dir in way of locally deleted file\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## t/t2500-untracked-overwriting.sh ##\n @@ t/t2500-untracked-overwriting.sh: test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n10: 6b42a80bf3d ! 9: b2f27d961a9 Comment important codepaths regarding nuking untracked files/dirs\n @@ Commit message\n running in an empty directory.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/stash.c ##\n @@ builtin/stash.c: static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n11: de416f887d7 ! 10: e88f81baa50 Documentation: call out commands that nuke untracked files/directories\n @@ Commit message\n these cases.\n \n Signed-off-by: Elijah Newren \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## Documentation/git-checkout.txt ##\n @@ Documentation/git-checkout.txt: OPTIONS\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437830","messageId":"RFC-patch-v4-02.10-8e5f4006604-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 02/10] read-tree, merge-recursive: overwrite ignored files by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:51Z","receivedAt":"2021-10-04T01:12:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nThis fixes a long-standing patchwork of ignored files handling in\nread-tree and merge-recursive, called out and suggested by Junio long\nago. Quoting from commit dcf0c16ef1 (\"core.excludesfile clean-up\"\n2007-11-16):\n\n git-read-tree takes --exclude-per-directory=,\n not because the flexibility was needed. Again, this was\n because the option predates the standardization of the ignore\n files.\n\n ...\n\n On the other hand, I think it makes perfect sense to fix\n git-read-tree, git-merge-recursive and git-clean to follow the\n same rule as other commands. I do not think of a valid use case\n to give an exclude-per-directory that is nonstandard to\n read-tree command, outside a \"negative\" test in the t1004 test\n script.\n\n This patch is the first step to untangle this mess.\n\n The next step would be to teach read-tree, merge-recursive and\n clean (in C) to use setup_standard_excludes().\n\nHistory shows each of these were partially or fully fixed:\n\n * clean was taught the new trick in 1617adc7a0 (\"Teach git clean to\n use setup_standard_excludes()\", 2007-11-14).\n\n * read-tree was primarily used by checkout & merge scripts. checkout\n and merge later became builtins and were both fixed to use the new\n setup_standard_excludes() handling in fc001b526c (\"checkout,merge:\n loosen overwriting untracked file check based on info/exclude\",\n 2011-11-27). So the primary users were fixed, though read-tree\n itself was not.\n\n * merge-recursive has now been replaced as the default merge backend\n by merge-ort. merge-ort fixed this by using\n setup_standard_excludes() starting early in its implementation; see\n commit 6681ce5cf6 (\"merge-ort: add implementation of checkout()\",\n 2020-12-13), largely due to its design depending on checkout() and\n thus being influenced by the checkout code. However,\n merge-recursive itself was not fixed here, in part because its\n design meant it had difficulty differentiating between untracked\n files, ignored files, leftover tracked files that haven't been\n removed yet due to order of processing files, and files written by\n itself due to collisions).\n\nMake the conversion more complete by now handling read-tree and\nhandling at least the unpack_trees() portion of merge-recursive. While\nmerge-recursive is on its way out, fixing the unpack_trees() portion is\neasy and facilitates some of the later changes in this series. Note\nthat fixing read-tree makes the --exclude-per-directory option to\nread-tree useless, so we remove it from the documentation (though we\ncontinue to accept it if passed).\n\nThe read-tree changes happen to fix a bug in t1013.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/git-read-tree.txt | 18 +-----------------\n builtin/read-tree.c | 21 +++++++++------------\n merge-recursive.c | 5 ++++-\n t/t1013-read-tree-submodule.sh | 1 -\n 4 files changed, 14 insertions(+), 31 deletions(-)\n\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 5fa8bab64c2..0222a27c5af 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -10,8 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git read-tree' [[-m [--trivial] [--aggressive] | --reset | --prefix=]\n-\t\t[-u [--exclude-per-directory=] | -i]]\n-\t\t[--index-output=] [--no-sparse-checkout]\n+\t\t[-u | -i]] [--index-output=] [--no-sparse-checkout]\n \t\t(--empty | [ []])\n \n \n@@ -88,21 +87,6 @@ OPTIONS\n \tThe command will refuse to overwrite entries that already\n \texisted in the original index file.\n \n---exclude-per-directory=::\n-\tWhen running the command with `-u` and `-m` options, the\n-\tmerge result may need to overwrite paths that are not\n-\ttracked in the current branch. The command usually\n-\trefuses to proceed with the merge to avoid losing such a\n-\tpath. However this safety valve sometimes gets in the\n-\tway. For example, it often happens that the other\n-\tbranch added a file that used to be a generated file in\n-\tyour branch, and the safety valve triggers when you try\n-\tto switch to that branch after you ran `make` but before\n-\trunning `make clean` to remove the generated file. This\n-\toption tells the command to read per-directory exclude\n-\tfile (usually '.gitignore') and allows such an untracked\n-\tbut explicitly ignored file to be overwritten.\n-\n --index-output=::\n \tInstead of writing the results out to `$GIT_INDEX_FILE`,\n \twrite the resulting index in the named file. While the\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 8f1b8a7e74c..d0f88bbf3e3 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -38,7 +38,7 @@ static int list_tree(struct object_id *oid)\n }\n \n static const char * const read_tree_usage[] = {\n-\tN_(\"git read-tree [(-m [--trivial] [--aggressive] | --reset | --prefix=) [-u [--exclude-per-directory=] | -i]] [--no-sparse-checkout] [--index-output=] (--empty | [ []])\"),\n+\tN_(\"git read-tree [(-m [--trivial] [--aggressive] | --reset | --prefix=) [-u | -i]] [--no-sparse-checkout] [--index-output=] (--empty | [ []])\"),\n \tNULL\n };\n \n@@ -59,15 +59,10 @@ static int exclude_per_directory_cb(const struct option *opt, const char *arg,\n \n \topts = (struct unpack_trees_options *)opt->value;\n \n-\tif (opts->dir.exclude_per_dir)\n-\t\tdie(\"more than one --exclude-per-directory given.\");\n-\n-\topts->dir.flags |= DIR_SHOW_IGNORED;\n-\topts->dir.exclude_per_dir = arg;\n-\t/* We do not need to nor want to do read-directory\n-\t * here; we are merely interested in reusing the\n-\t * per directory ignore stack mechanism.\n-\t */\n+\tif (!opts->update)\n+\t\tdie(\"--exclude-per-directory is meaningless unless -u\");\n+\tif (strcmp(arg, \".gitignore\"))\n+\t\tdie(\"--exclude-per-directory argument must be .gitignore\");\n \treturn 0;\n }\n \n@@ -206,8 +201,10 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif ((opts.update || opts.index_only) && !opts.merge)\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n-\tif ((opts.dir.exclude_per_dir && !opts.update))\n-\t\tdie(\"--exclude-per-directory is meaningless unless -u\");\n+\tif (opts.update && !opts.reset) {\n+\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&opts.dir);\n+\t}\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 316cb2ca907..a4131b8837b 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -408,8 +408,11 @@ static int unpack_trees_start(struct merge_options *opt,\n \tunpack_trees_options_init(&opt->priv->unpack_opts);\n \tif (opt->priv->call_depth)\n \t\topt->priv->unpack_opts.index_only = 1;\n-\telse\n+\telse {\n \t\topt->priv->unpack_opts.update = 1;\n+\t\topt->priv->unpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&opt->priv->unpack_opts.dir);\n+\t}\n \topt->priv->unpack_opts.merge = 1;\n \topt->priv->unpack_opts.head_idx = 2;\n \topt->priv->unpack_opts.fn = threeway_merge;\ndiff --git a/t/t1013-read-tree-submodule.sh b/t/t1013-read-tree-submodule.sh\nindex b6df7444c05..bfc90d4cf27 100755\n--- a/t/t1013-read-tree-submodule.sh\n+++ b/t/t1013-read-tree-submodule.sh\n@@ -6,7 +6,6 @@ test_description='read-tree can handle submodules'\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \n KNOWN_FAILURE_DIRECTORY_SUBMODULE_CONFLICTS=1\n-KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1\n \n test_submodule_switch_recursing_with_args \"read-tree -u -m\"\n \n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437831","messageId":"RFC-patch-v4-03.10-739e9b871c4-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 03/10] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:52Z","receivedAt":"2021-10-04T01:12:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nCurrently, every caller of unpack_trees() that wants to ensure ignored\nfiles are overwritten by default needs to:\n\n * flip the DIR_SHOW_IGNORED flag in unpack_trees_options.dir.flags\n * call setup_standard_excludes(&unpack_trees_options.dir)\n\nAvoid that boilerplate by introducing a new boolean value where\nthe default value (0) does what we want so that new callers of\nunpack_trees() automatically get the appropriate behavior. And move all\nthe handling of unpack_trees_options.dir into unpack_trees() itself.\n\nWhile preserve_ignored = 0 is the behavior we feel is the appropriate\ndefault, we defer fixing commands to use the appropriate default until a\nlater commit. So, this commit introduces several locations where we\nmanually set preserve_ignored=1. This makes it clear where code paths\nwere previously preserving ignored files when they should not have been;\na future commit will flip these to instead use a value of 0 to get the\nbehavior we want.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/am.c | 3 +++\n builtin/checkout.c | 6 ++----\n builtin/clone.c | 2 ++\n builtin/merge.c | 2 ++\n builtin/read-tree.c | 7 +++----\n builtin/reset.c | 2 ++\n builtin/stash.c | 3 +++\n merge-ort.c | 5 +----\n merge-recursive.c | 4 ++--\n merge.c | 6 +-----\n reset.c | 2 ++\n sequencer.c | 2 ++\n unpack-trees.c | 5 +++++\n unpack-trees.h | 1 +\n 14 files changed, 31 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 4d4bb473c0f..3c6efe2a46b 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1918,6 +1918,9 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.update = 1;\n \topts.merge = 1;\n \topts.reset = reset;\n+\tif (!reset)\n+\t\t/* FIXME: Default should be to remove ignored files */\n+\t\topts.preserve_ignored = 1;\n \topts.fn = twoway_merge;\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex fd76b504861..0c5187025c5 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -648,6 +648,7 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n \topts.skip_unmerged = !worktree;\n \topts.reset = 1;\n \topts.merge = 1;\n+\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\n@@ -749,10 +750,7 @@ static int merge_working_tree(const struct checkout_opts *opts,\n \t\t\t\t new_branch_info->commit ?\n \t\t\t\t &new_branch_info->commit->object.oid :\n \t\t\t\t &new_branch_info->oid, NULL);\n-\t\tif (opts->overwrite_ignore) {\n-\t\t\ttopts.dir.flags |= DIR_SHOW_IGNORED;\n-\t\t\tsetup_standard_excludes(&topts.dir);\n-\t\t}\n+\t\ttopts.preserve_ignored = !opts->overwrite_ignore;\n \t\ttree = parse_tree_indirect(old_branch_info->commit ?\n \t\t\t\t\t &old_branch_info->commit->object.oid :\n \t\t\t\t\t the_hash_algo->empty_tree);\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex df3bb9a7884..e76c38e4e81 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -686,6 +686,8 @@ static int checkout(int submodule_progress)\n \topts.update = 1;\n \topts.merge = 1;\n \topts.clone = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\topts.preserve_ignored = 1;\n \topts.fn = oneway_merge;\n \topts.verbose_update = (option_verbosity >= 0);\n \topts.src_index = &the_index;\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 28089e2c5ed..55aac869e5a 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -680,6 +680,8 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n \topts.verbose_update = 1;\n \topts.trivial_merges_only = 1;\n \topts.merge = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\topts.preserve_ignored = 1;\n \ttrees[nr_trees] = parse_tree_indirect(common);\n \tif (!trees[nr_trees++])\n \t\treturn -1;\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex d0f88bbf3e3..7f3c987b126 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -201,10 +201,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif ((opts.update || opts.index_only) && !opts.merge)\n \t\tdie(\"%s is meaningless without -m, --reset, or --prefix\",\n \t\t opts.update ? \"-u\" : \"-i\");\n-\tif (opts.update && !opts.reset) {\n-\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&opts.dir);\n-\t}\n+\tif (opts.update && !opts.reset)\n+\t\topts.preserve_ignored = 0;\n+\t/* otherwise, opts.preserve_ignored is irrelevant */\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 713d084c3eb..73477239146 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -66,6 +66,8 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \tcase KEEP:\n \tcase MERGE:\n \t\topts.update = 1;\n+\t\t/* FIXME: Default should be to remove ignored files */\n+\t\topts.preserve_ignored = 1;\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex be6ecb1ae11..78492013529 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -257,6 +257,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.merge = 1;\n \topts.reset = reset;\n \topts.update = update;\n+\tif (update && !reset)\n+\t\t/* FIXME: Default should be to remove ignored files */\n+\t\topts.preserve_ignored = 1;\n \topts.fn = oneway_merge;\n \n \tif (unpack_trees(nr_trees, t, &opts)) {\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 0a5937364c9..e5620bda212 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -4044,10 +4044,7 @@ static int checkout(struct merge_options *opt,\n \tunpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n \tunpack_opts.verbose_update = (opt->verbosity > 2);\n \tunpack_opts.fn = twoway_merge;\n-\tif (1/* FIXME: opts->overwrite_ignore*/) {\n-\t\tunpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&unpack_opts.dir);\n-\t}\n+\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n \tparse_tree(prev);\n \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n \tparse_tree(next);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex a4131b8837b..5c6b95a79c0 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -410,8 +410,8 @@ static int unpack_trees_start(struct merge_options *opt,\n \t\topt->priv->unpack_opts.index_only = 1;\n \telse {\n \t\topt->priv->unpack_opts.update = 1;\n-\t\topt->priv->unpack_opts.dir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&opt->priv->unpack_opts.dir);\n+\t\t/* FIXME: should only do this if !overwrite_ignore */\n+\t\topt->priv->unpack_opts.preserve_ignored = 0;\n \t}\n \topt->priv->unpack_opts.merge = 1;\n \topt->priv->unpack_opts.head_idx = 2;\ndiff --git a/merge.c b/merge.c\nindex 2e3714ccaa0..e1f3165e407 100644\n--- a/merge.c\n+++ b/merge.c\n@@ -79,11 +79,7 @@ int checkout_fast_forward(struct repository *r,\n \t\tinit_tree_desc(t+i, trees[i]->buffer, trees[i]->size);\n \t}\n \n-\tif (overwrite_ignore) {\n-\t\topts.dir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&opts.dir);\n-\t}\n-\n+\topts.preserve_ignored = !overwrite_ignore;\n \topts.head_idx = 1;\n \topts.src_index = r->index;\n \topts.dst_index = r->index;\ndiff --git a/reset.c b/reset.c\nindex f4bf3fbfac0..cd344f47f13 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -56,6 +56,8 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n \tunpack_tree_opts.update = 1;\n \tunpack_tree_opts.merge = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\tunpack_tree_opts.preserve_ignored = 1;\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n \tif (!detach_head)\n \t\tunpack_tree_opts.reset = 1;\ndiff --git a/sequencer.c b/sequencer.c\nindex abd85b6c562..669ea15944c 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3698,6 +3698,8 @@ static int do_reset(struct repository *r,\n \tunpack_tree_opts.fn = oneway_merge;\n \tunpack_tree_opts.merge = 1;\n \tunpack_tree_opts.update = 1;\n+\t/* FIXME: Default should be to remove ignored files */\n+\tunpack_tree_opts.preserve_ignored = 1;\n \tinit_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n \n \tif (repo_read_index_unmerged(r)) {\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 260e7ec5bb4..02bc999c6c3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1711,6 +1711,11 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tensure_full_index(o->dst_index);\n \t}\n \n+\tif (!o->preserve_ignored) {\n+\t\to->dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&o->dir);\n+\t}\n+\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex a8d1f083b33..65a8d99d4ef 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -52,6 +52,7 @@ struct unpack_trees_options {\n \tunsigned int reset,\n \t\t merge,\n \t\t update,\n+\t\t preserve_ignored,\n \t\t clone,\n \t\t index_only,\n \t\t nontrivial_merge,\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437832","messageId":"RFC-patch-v4-04.10-296c1e03673-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 04/10] unpack-trees: rename \"dir\" to \"private_dir\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:53Z","receivedAt":"2021-10-04T01:12:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nUntil the introduction of the \"preserve_ignored\" flag in the preceding\ncommit callers who wanted its behavior needed to adjust \"dir.flags\"\nand call setup_standard_excludes() themselves.\n\nNow that we have no external users of \"dir\" anymore let's rename it to\n\"private_dir\" and add a comment indicating that we'd like it not to be\nmessed with by external callers. This should avoid avoid accidental\nmisuse or confusion over its ownership.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n unpack-trees.c | 10 +++++-----\n unpack-trees.h | 4 ++--\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 02bc999c6c3..512011cfa42 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -196,7 +196,7 @@ void unpack_trees_options_init(struct unpack_trees_options *o)\n void unpack_trees_options_release(struct unpack_trees_options *opts)\n {\n \tstrvec_clear(&opts->msgs_to_free);\n-\tdir_clear(&opts->dir);\n+\tdir_clear(&opts->private_dir);\n }\n \n static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n@@ -1712,8 +1712,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t}\n \n \tif (!o->preserve_ignored) {\n-\t\to->dir.flags |= DIR_SHOW_IGNORED;\n-\t\tsetup_standard_excludes(&o->dir);\n+\t\to->private_dir.flags |= DIR_SHOW_IGNORED;\n+\t\tsetup_standard_excludes(&o->private_dir);\n \t}\n \n \tif (!core_apply_sparse_checkout || !o->update)\n@@ -2141,7 +2141,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n \t */\n \tpathbuf = xstrfmt(\"%.*s/\", namelen, ce->name);\n \n-\td.exclude_per_dir = o->dir.exclude_per_dir;\n+\td.exclude_per_dir = o->private_dir.exclude_per_dir;\n \ti = read_directory(&d, o->src_index, pathbuf, namelen+1, NULL);\n \tdir_clear(&d);\n \tfree(pathbuf);\n@@ -2183,7 +2183,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \tif (ignore_case && icase_exists(o, name, len, st))\n \t\treturn 0;\n \n-\tif (is_excluded(&o->dir, o->src_index, name, &dtype))\n+\tif (is_excluded(&o->private_dir, o->src_index, name, &dtype))\n \t\t/*\n \t\t * ce->name is explicitly excluded, so it is Ok to\n \t\t * overwrite it.\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 65a8d99d4ef..2eb633bf771 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -70,7 +70,7 @@ struct unpack_trees_options {\n \t\t dry_run;\n \tconst char *prefix;\n \tint cache_bottom;\n-\tstruct dir_struct dir;\n+\tstruct dir_struct private_dir; /* for internal use only */\n \tstruct pathspec *pathspec;\n \tmerge_fn_t fn;\n \tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n@@ -97,7 +97,7 @@ struct unpack_trees_options {\n \n #define UNPACK_TREES_OPTIONS_INIT { \\\n \t.msgs_to_free = STRVEC_INIT, \\\n-\t.dir = DIR_INIT, \\\n+\t.private_dir = DIR_INIT, \\\n }\n void unpack_trees_options_init(struct unpack_trees_options *o);\n \n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437833","messageId":"RFC-patch-v4-05.10-27496506430-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 05/10] Remove ignored files by default when they are in the way","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:54Z","receivedAt":"2021-10-04T01:12:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nChange several commands to remove ignored files by default when they are\nin the way. Since some commands (checkout, merge) take a\n--no-overwrite-ignore option to allow the user to configure this, and it\nmay make sense to add that option to more commands (and in the case of\nmerge, actually plumb that configuration option through to more of the\nbackends than just the fast-forwarding special case), add little\ncomments about where such flags would be used.\n\nIncidentally, this fixes a test failure in t7112.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/am.c | 3 +--\n builtin/clone.c | 3 +--\n builtin/merge.c | 3 +--\n builtin/reset.c | 3 +--\n builtin/stash.c | 3 +--\n merge-ort.c | 2 +-\n reset.c | 3 +--\n sequencer.c | 3 +--\n t/t7112-reset-submodule.sh | 1 -\n 9 files changed, 8 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 3c6efe2a46b..8cb7e72e6c5 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1919,8 +1919,7 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.merge = 1;\n \topts.reset = reset;\n \tif (!reset)\n-\t\t/* FIXME: Default should be to remove ignored files */\n-\t\topts.preserve_ignored = 1;\n+\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = twoway_merge;\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex e76c38e4e81..49edb4a2aaa 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -686,8 +686,7 @@ static int checkout(int submodule_progress)\n \topts.update = 1;\n \topts.merge = 1;\n \topts.clone = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\topts.preserve_ignored = 1;\n+\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = (option_verbosity >= 0);\n \topts.src_index = &the_index;\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 55aac869e5a..7f990e36651 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -680,8 +680,7 @@ static int read_tree_trivial(struct object_id *common, struct object_id *head,\n \topts.verbose_update = 1;\n \topts.trivial_merges_only = 1;\n \topts.merge = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\topts.preserve_ignored = 1;\n+\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \ttrees[nr_trees] = parse_tree_indirect(common);\n \tif (!trees[nr_trees++])\n \t\treturn -1;\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 73477239146..9d1391335a1 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -66,8 +66,7 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \tcase KEEP:\n \tcase MERGE:\n \t\topts.update = 1;\n-\t\t/* FIXME: Default should be to remove ignored files */\n-\t\topts.preserve_ignored = 1;\n+\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 78492013529..92ad3241270 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -258,8 +258,7 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.reset = reset;\n \topts.update = update;\n \tif (update && !reset)\n-\t\t/* FIXME: Default should be to remove ignored files */\n-\t\topts.preserve_ignored = 1;\n+\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = oneway_merge;\n \n \tif (unpack_trees(nr_trees, t, &opts)) {\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e5620bda212..5c443b2b00d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -4044,7 +4044,7 @@ static int checkout(struct merge_options *opt,\n \tunpack_opts.quiet = 0; /* FIXME: sequencer might want quiet? */\n \tunpack_opts.verbose_update = (opt->verbosity > 2);\n \tunpack_opts.fn = twoway_merge;\n-\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore*/\n+\tunpack_opts.preserve_ignored = 0; /* FIXME: !opts->overwrite_ignore */\n \tparse_tree(prev);\n \tinit_tree_desc(&trees[0], prev->buffer, prev->size);\n \tparse_tree(next);\ndiff --git a/reset.c b/reset.c\nindex cd344f47f13..5f69311b9f4 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -56,8 +56,7 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;\n \tunpack_tree_opts.update = 1;\n \tunpack_tree_opts.merge = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\tunpack_tree_opts.preserve_ignored = 1;\n+\tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n \tif (!detach_head)\n \t\tunpack_tree_opts.reset = 1;\ndiff --git a/sequencer.c b/sequencer.c\nindex 669ea15944c..05395db9e01 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3698,8 +3698,7 @@ static int do_reset(struct repository *r,\n \tunpack_tree_opts.fn = oneway_merge;\n \tunpack_tree_opts.merge = 1;\n \tunpack_tree_opts.update = 1;\n-\t/* FIXME: Default should be to remove ignored files */\n-\tunpack_tree_opts.preserve_ignored = 1;\n+\tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, name, &oid, NULL);\n \n \tif (repo_read_index_unmerged(r)) {\ndiff --git a/t/t7112-reset-submodule.sh b/t/t7112-reset-submodule.sh\nindex 19830d90365..a3e2413bc33 100755\n--- a/t/t7112-reset-submodule.sh\n+++ b/t/t7112-reset-submodule.sh\n@@ -6,7 +6,6 @@ test_description='reset can handle submodules'\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \n KNOWN_FAILURE_DIRECTORY_SUBMODULE_CONFLICTS=1\n-KNOWN_FAILURE_SUBMODULE_OVERWRITE_IGNORED_UNTRACKED=1\n \n test_submodule_switch_recursing_with_args \"reset --keep\"\n \n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437834","messageId":"RFC-patch-v4-06.10-7b539a120b9-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 06/10] Change unpack_trees' 'reset' flag into an enum","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:55Z","receivedAt":"2021-10-04T01:12:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nTraditionally, unpack_trees_options->reset was used to signal that it\nwas okay to delete any untracked files in the way. This was used by\n`git read-tree --reset`, but then started appearing in other places as\nwell. However, many of the other uses should not be deleting untracked\nfiles in the way. Change this value to an enum so that a value of 1\n(i.e. \"true\") can be split into two:\n UNPACK_RESET_PROTECT_UNTRACKED,\n UNPACK_RESET_OVERWRITE_UNTRACKED\nIn order to catch accidental misuses (i.e. where folks call it the way\nthey traditionally used to), define the special enum value of\n UNPACK_RESET_INVALID = 1\nwhich will trigger a BUG().\n\nModify existing callers so that\n read-tree --reset\n reset --hard\n checkout --force\ncontinue using the UNPACK_RESET_OVERWRITE_UNTRACKED logic, while other\ncallers, including\n am\n checkout without --force\n stash (though currently dead code; reset always had a value of 0)\n numerous callers from rebase/sequencer to reset_head()\nwill use the new UNPACK_RESET_PROTECT_UNTRACKED value.\n\nAlso, note that it has been reported that 'git checkout \n' currently also allows overwriting untracked files[1]. That\ncase should also be fixed, but it does not use unpack_trees() and thus\nis outside the scope of the current changes.\n\n[1] https://lore.kernel.org/git/15dad590-087e-5a48-9238-5d2826950506@gmail.com/\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/am.c | 5 ++---\n builtin/checkout.c | 5 +++--\n builtin/read-tree.c | 3 +++\n builtin/reset.c | 9 +++++++--\n builtin/stash.c | 4 ++--\n reset.c | 2 +-\n t/t2500-untracked-overwriting.sh | 6 +++---\n unpack-trees.c | 10 +++++++++-\n unpack-trees.h | 11 +++++++++--\n 9 files changed, 39 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 8cb7e72e6c5..dfe2bf207e6 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1917,9 +1917,8 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)\n \topts.dst_index = &the_index;\n \topts.update = 1;\n \topts.merge = 1;\n-\topts.reset = reset;\n-\tif (!reset)\n-\t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n+\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n+\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = twoway_merge;\n \tinit_tree_desc(&t[0], head->buffer, head->size);\n \tinit_tree_desc(&t[1], remote->buffer, remote->size);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 0c5187025c5..ee3e450537f 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -646,9 +646,10 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,\n \topts.head_idx = -1;\n \topts.update = worktree;\n \topts.skip_unmerged = !worktree;\n-\topts.reset = 1;\n+\topts.reset = o->force ? UNPACK_RESET_OVERWRITE_UNTRACKED :\n+\t\t\t\tUNPACK_RESET_PROTECT_UNTRACKED;\n+\topts.preserve_ignored = (!o->force && !o->overwrite_ignore);\n \topts.merge = 1;\n-\topts.preserve_ignored = 0;\n \topts.fn = oneway_merge;\n \topts.verbose_update = o->show_progress;\n \topts.src_index = &the_index;\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 7f3c987b126..5e10caeff5a 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -166,6 +166,9 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n+\tif (opts.reset)\n+\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n+\n \t/*\n \t * NEEDSWORK\n \t *\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 9d1391335a1..00d2de392a8 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -70,9 +70,14 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t\n \t\tbreak;\n \tcase HARD:\n \t\topts.update = 1;\n-\t\t/* fallthrough */\n+\t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n+\t\tbreak;\n+\tcase MIXED:\n+\t\topts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n+\t\t/* but opts.update=0, so working tree not updated */\n+\t\tbreak;\n \tdefault:\n-\t\topts.reset = 1;\n+\t\tBUG(\"invalid reset_type passed to reset_index\");\n \t}\n \n \tread_cache_unmerged();\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 92ad3241270..061237cf9a4 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -255,9 +255,9 @@ static int reset_tree(struct object_id *i_tree, int update, int reset)\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n \topts.merge = 1;\n-\topts.reset = reset;\n+\topts.reset = reset ? UNPACK_RESET_PROTECT_UNTRACKED : 0;\n \topts.update = update;\n-\tif (update && !reset)\n+\tif (update)\n \t\topts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \topts.fn = oneway_merge;\n \ndiff --git a/reset.c b/reset.c\nindex 5f69311b9f4..5788b1926f3 100644\n--- a/reset.c\n+++ b/reset.c\n@@ -59,7 +59,7 @@ int reset_head(struct repository *r, struct object_id *oid, const char *action,\n \tunpack_tree_opts.preserve_ignored = 0; /* FIXME: !overwrite_ignore */\n \tinit_checkout_metadata(&unpack_tree_opts.meta, switch_to_branch, oid, NULL);\n \tif (!detach_head)\n-\t\tunpack_tree_opts.reset = 1;\n+\t\tunpack_tree_opts.reset = UNPACK_RESET_PROTECT_UNTRACKED;\n \n \tif (repo_read_index_unmerged(r) < 0) {\n \t\tret = error(_(\"could not read index\"));\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 2412d121ea8..18604360df8 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -92,7 +92,7 @@ test_setup_checkout_m () {\n \t)\n }\n \n-test_expect_failure 'checkout -m does not nuke untracked file' '\n+test_expect_success 'checkout -m does not nuke untracked file' '\n \ttest_setup_checkout_m &&\n \t(\n \t\tcd checkout &&\n@@ -138,7 +138,7 @@ test_setup_sequencing () {\n \t)\n }\n \n-test_expect_failure 'git rebase --abort and untracked files' '\n+test_expect_success 'git rebase --abort and untracked files' '\n \ttest_setup_sequencing rebase_abort_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_abort_and_untracked &&\n@@ -155,7 +155,7 @@ test_expect_failure 'git rebase --abort and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git rebase fast forwarding and untracked files' '\n+test_expect_success 'git rebase fast forwarding and untracked files' '\n \ttest_setup_sequencing rebase_fast_forward_and_untracked &&\n \t(\n \t\tcd sequencing_rebase_fast_forward_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 512011cfa42..37f769030ab 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1699,6 +1699,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n \n+\tif (o->reset == UNPACK_RESET_INVALID)\n+\t\tBUG(\"o->reset had a value of 1; should be UNPACK_TREES_*_UNTRACKED\");\n+\n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n@@ -1711,6 +1714,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tensure_full_index(o->dst_index);\n \t}\n \n+\tif (o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED &&\n+\t o->preserve_ignored)\n+\t\tBUG(\"UNPACK_RESET_OVERWRITE_UNTRACKED incompatible with preserved ignored files\");\n+\n \tif (!o->preserve_ignored) {\n \t\to->private_dir.flags |= DIR_SHOW_IGNORED;\n \t\tsetup_standard_excludes(&o->private_dir);\n@@ -2227,7 +2234,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \tint len;\n \tstruct stat st;\n \n-\tif (o->index_only || o->reset || !o->update)\n+\tif (o->index_only || !o->update ||\n+\t o->reset == UNPACK_RESET_OVERWRITE_UNTRACKED)\n \t\treturn 0;\n \n \tlen = check_leading_path(ce->name, ce_namelen(ce), 0);\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 2eb633bf771..1582599606a 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -48,9 +48,15 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n */\n void unpack_trees_options_release(struct unpack_trees_options *opts);\n \n+enum unpack_trees_reset_type {\n+\tUNPACK_RESET_NONE = 0, /* traditional \"false\" value; still valid */\n+\tUNPACK_RESET_INVALID = 1, /* \"true\" no longer valid; use below values */\n+\tUNPACK_RESET_PROTECT_UNTRACKED,\n+\tUNPACK_RESET_OVERWRITE_UNTRACKED\n+};\n+\n struct unpack_trees_options {\n-\tunsigned int reset,\n-\t\t merge,\n+\tunsigned int merge,\n \t\t update,\n \t\t preserve_ignored,\n \t\t clone,\n@@ -68,6 +74,7 @@ struct unpack_trees_options {\n \t\t exiting_early,\n \t\t show_all_errors,\n \t\t dry_run;\n+\tenum unpack_trees_reset_type reset;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct private_dir; /* for internal use only */\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437835","messageId":"RFC-patch-v4-07.10-b6769f629ae-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 07/10] unpack-trees: avoid nuking untracked dir in way of unmerged file","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:56Z","receivedAt":"2021-10-04T01:12:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 35 ++++++++++++++++++++++++++++----\n 2 files changed, 32 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 18604360df8..5ec66058cfc 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -197,7 +197,7 @@ test_expect_failure 'git stash and untracked files' '\n \t)\n '\n \n-test_expect_failure 'git am --abort and untracked dir vs. unmerged file' '\n+test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \ttest_setup_sequencing am_abort_and_untracked &&\n \t(\n \t\tcd sequencing_am_abort_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 37f769030ab..8408a8fcfff 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2173,9 +2173,15 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n+enum absent_checking_type {\n+\tCOMPLETELY_ABSENT,\n+\tABSENT_ANY_DIRECTORY\n+};\n+\n static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\t\t const struct cache_entry *ce, struct stat *st,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tconst struct cache_entry *result;\n@@ -2209,6 +2215,10 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \t}\n \n+\t/* If we only care about directories, then we can remove */\n+\tif (absent_type == ABSENT_ANY_DIRECTORY)\n+\t\treturn 0;\n+\n \t/*\n \t * The previous round may already have decided to\n \t * delete this path, which is in a subdirectory that\n@@ -2229,6 +2239,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n */\n static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t enum absent_checking_type absent_type,\n \t\t\t struct unpack_trees_options *o)\n {\n \tint len;\n@@ -2255,7 +2266,8 @@ static int verify_absent_1(const struct cache_entry *ce,\n \t\t\t\t\t\t\t\tNULL, o);\n \t\t\telse\n \t\t\t\tret = check_ok_to_remove(path, len, DT_UNKNOWN, NULL,\n-\t\t\t\t\t\t\t &st, error_type, o);\n+\t\t\t\t\t\t\t &st, error_type,\n+\t\t\t\t\t\t\t absent_type, o);\n \t\t}\n \t\tfree(path);\n \t\treturn ret;\n@@ -2270,7 +2282,7 @@ static int verify_absent_1(const struct cache_entry *ce,\n \n \t\treturn check_ok_to_remove(ce->name, ce_namelen(ce),\n \t\t\t\t\t ce_to_dtype(ce), ce, &st,\n-\t\t\t\t\t error_type, o);\n+\t\t\t\t\t error_type, absent_type, o);\n \t}\n }\n \n@@ -2280,14 +2292,23 @@ static int verify_absent(const struct cache_entry *ce,\n {\n \tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n+}\n+\n+static int verify_absent_if_directory(const struct cache_entry *ce,\n+\t\t\t\t enum unpack_trees_error_types error_type,\n+\t\t\t\t struct unpack_trees_options *o)\n+{\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\t\treturn 0;\n+\treturn verify_absent_1(ce, error_type, ABSENT_ANY_DIRECTORY, o);\n }\n \n static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types error_type,\n \t\t\t\tstruct unpack_trees_options *o)\n {\n-\treturn verify_absent_1(ce, error_type, o);\n+\treturn verify_absent_1(ce, error_type, COMPLETELY_ABSENT, o);\n }\n \n static int merged_entry(const struct cache_entry *ce,\n@@ -2361,6 +2382,12 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t * Previously unmerged entry left as an existence\n \t\t * marker by read_index_unmerged();\n \t\t */\n+\t\tif (verify_absent_if_directory(merge,\n+\t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n+\t\t\tdiscard_cache_entry(merge);\n+\t\t\treturn -1;\n+\t\t}\n+\n \t\tinvalidate_ce_path(old, o);\n \t}\n \n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437836","messageId":"RFC-patch-v4-08.10-10a7cbf049e-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 08/10] unpack-trees: avoid nuking untracked dir in way of locally deleted file","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:57Z","receivedAt":"2021-10-04T01:12:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t2500-untracked-overwriting.sh | 2 +-\n unpack-trees.c | 3 +++\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t2500-untracked-overwriting.sh b/t/t2500-untracked-overwriting.sh\nindex 5ec66058cfc..5c0bf4d21fc 100755\n--- a/t/t2500-untracked-overwriting.sh\n+++ b/t/t2500-untracked-overwriting.sh\n@@ -218,7 +218,7 @@ test_expect_success 'git am --abort and untracked dir vs. unmerged file' '\n \t)\n '\n \n-test_expect_failure 'git am --skip and untracked dir vs deleted file' '\n+test_expect_success 'git am --skip and untracked dir vs deleted file' '\n \ttest_setup_sequencing am_skip_and_untracked &&\n \t(\n \t\tcd sequencing_am_skip_and_untracked &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8408a8fcfff..703e7953d62 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2405,7 +2405,10 @@ static int deleted_entry(const struct cache_entry *ce,\n \t\tif (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))\n \t\t\treturn -1;\n \t\treturn 0;\n+\t} else if (verify_absent_if_directory(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o)) {\n+\t\treturn -1;\n \t}\n+\n \tif (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))\n \t\treturn -1;\n \tadd_entry(o, ce, CE_REMOVE, 0);\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437837","messageId":"RFC-patch-v4-09.10-b2f27d961a9-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 09/10] Comment important codepaths regarding nuking untracked files/dirs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:58Z","receivedAt":"2021-10-04T01:12:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nIn the last few commits we focused on code in unpack-trees.c that\nmistakenly removed untracked files or directories. There may be more of\nthose, but in this commit we change our focus: callers of toplevel\ncommands that are expected to remove untracked files or directories.\n\nAs noted previously, we have toplevel commands that are expected to\ndelete untracked files such as 'read-tree --reset', 'reset --hard', and\n'checkout --force'. However, that does not mean that other highlevel\ncommands that happen to call these other commands thought about or\nconveyed to users the possibility that untracked files could be removed.\nAudit the code for such callsites, and add comments near existing\ncallsites to mention whether these are safe or not.\n\nMy auditing is somewhat incomplete, though; it skipped several cases:\n * git-rebase--preserve-merges.sh: is in the process of being\n deprecated/removed, so I won't leave a note that there are\n likely more bugs in that script.\n * contrib/git-new-workdir: why is the -f flag being used in a new\n empty directory?? It shouldn't hurt, but it seems useless.\n * git-p4.py: Don't see why -f is needed for a new dir (maybe it's\n not and is just superfluous), but I'm not at all familiar with\n the p4 stuff\n * git-archimport.perl: Don't care; arch is long since dead\n * git-cvs*.perl: Don't care; cvs is long since dead\n\nAlso, the reset --hard in builtin/worktree.c looks safe, due to only\nrunning in an empty directory.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/stash.c | 1 +\n builtin/submodule--helper.c | 4 ++++\n contrib/rerere-train.sh | 2 +-\n submodule.c | 1 +\n 4 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 061237cf9a4..fabfb63632e 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1525,6 +1525,7 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\t} else {\n \t\t\tstruct child_process cp = CHILD_PROCESS_INIT;\n \t\t\tcp.git_cmd = 1;\n+\t\t\t/* BUG: this nukes untracked files in the way */\n \t\t\tstrvec_pushl(&cp.args, \"reset\", \"--hard\", \"-q\",\n \t\t\t\t \"--no-recurse-submodules\", NULL);\n \t\t\tif (run_command(&cp)) {\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 57f09925157..e40e4b6aacc 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -3091,6 +3091,10 @@ static int add_submodule(const struct add_data *add_data)\n \t\tprepare_submodule_repo_env(&cp.env_array);\n \t\tcp.git_cmd = 1;\n \t\tcp.dir = add_data->sm_path;\n+\t\t/*\n+\t\t * NOTE: we only get here if add_data->force is true, so\n+\t\t * passing --force to checkout is reasonable.\n+\t\t */\n \t\tstrvec_pushl(&cp.args, \"checkout\", \"-f\", \"-q\", NULL);\n \n \t\tif (add_data->branch) {\ndiff --git a/contrib/rerere-train.sh b/contrib/rerere-train.sh\nindex eeee45dd341..75125d6ae00 100755\n--- a/contrib/rerere-train.sh\n+++ b/contrib/rerere-train.sh\n@@ -91,7 +91,7 @@ do\n \t\tgit checkout -q $commit -- .\n \t\tgit rerere\n \tfi\n-\tgit reset -q --hard\n+\tgit reset -q --hard # Might nuke untracked files...\n done\n \n if test -z \"$branch\"\ndiff --git a/submodule.c b/submodule.c\nindex 03cf36707ae..9f3d9b7ee73 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1910,6 +1910,7 @@ static void submodule_reset_index(const char *path)\n \n \tstrvec_pushf(&cp.args, \"--super-prefix=%s%s/\",\n \t\t get_super_prefix_or_empty(), path);\n+\t/* TODO: determine if this might overwright untracked files */\n \tstrvec_pushl(&cp.args, \"read-tree\", \"-u\", \"--reset\", NULL);\n \n \tstrvec_push(&cp.args, empty_tree_oid_hex());\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437838","messageId":"RFC-patch-v4-10.10-e88f81baa50-20211004T004902Z-avarab@gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"[RFC PATCH v4 10/10] Documentation: call out commands that nuke untracked files/directories","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T01:11:59Z","receivedAt":"2021-10-04T01:12:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Elijah Newren \n\nSome commands have traditionally also removed untracked files (or\ndirectories) that were in the way of a tracked file we needed. Document\nthese cases.\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/git-checkout.txt | 5 +++--\n Documentation/git-read-tree.txt | 5 +++--\n Documentation/git-reset.txt | 3 ++-\n 3 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\nindex b1a6fe44997..d473c9bf387 100644\n--- a/Documentation/git-checkout.txt\n+++ b/Documentation/git-checkout.txt\n@@ -118,8 +118,9 @@ OPTIONS\n -f::\n --force::\n \tWhen switching branches, proceed even if the index or the\n-\tworking tree differs from `HEAD`. This is used to throw away\n-\tlocal changes.\n+\tworking tree differs from `HEAD`, and even if there are untracked\n+\tfiles in the way. This is used to throw away local changes and\n+\tany untracked files or directories that are in the way.\n +\n When checking out paths from the index, do not fail upon unmerged\n entries; instead, unmerged entries are ignored.\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 0222a27c5af..8c3aceb8324 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -38,8 +38,9 @@ OPTIONS\n \n --reset::\n \tSame as -m, except that unmerged entries are discarded instead\n-\tof failing. When used with `-u`, updates leading to loss of\n-\tworking tree changes will not abort the operation.\n+\tof failing. When used with `-u`, updates leading to loss of\n+\tworking tree changes or untracked files or directories will not\n+\tabort the operation.\n \n -u::\n \tAfter a successful merge, update the files in the work\ndiff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\nindex 252e2d4e47d..6f7685f53d5 100644\n--- a/Documentation/git-reset.txt\n+++ b/Documentation/git-reset.txt\n@@ -69,7 +69,8 @@ linkgit:git-add[1]).\n \n --hard::\n \tResets the index and working tree. Any changes to tracked files in the\n-\tworking tree since `` are discarded.\n+\tworking tree since `` are discarded. Any untracked files or\n+\tdirectories in the way of writing any tracked files are simply deleted.\n \n --merge::\n \tResets the index and updates the files in the working tree that are\n-- \n2.33.0.1404.g83021034c5d\n\n"},{"id":"437879","messageId":"CABPp-BFE_5zPCZY8adJSQchdQzxq3uH-oma-S=_Sw0OUXx03OQ@mail.gmail.com","threadId":"56534","inReplyTo":"87k0ivpzfx.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-04T13:45:00Z","receivedAt":"2021-10-04T13:54:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Oct 2, 2021 at 2:07 AM Ævar Arnfjörð Bjarmason wrote:\n>\n> On Fri, Oct 01 2021, Elijah Newren wrote:\n>\n...\n> > So maybe I'll submit some patches on top that rip these direct members\n> > out of of unpack_trees_options and push them inside some opaque\n> > struct.\n>\n> Sure, that sounds good. I only had a mild objection to doing it in a way\n> where you'll need that sort of code I removed in the linked commit in\n> prep_exclude() because you were trying not to expose that at any cost,\n> including via some *_INIT macro. I.e. if it's private we can just name\n> it \"priv_*\" or have a :\n>\n> struct dont_touch_this {\n> struct dir_struct dir;\n> };\n>\n> Which are both ways of /messaging/ that it's private, and since the\n> target audience is just the rest of the git.git codebase I think that\n> ultimately something that 1) sends the right message 2) makes accidents\n> pretty much impossible suffices. I.e. you don't accidentally introduce a\n> new API user accessing a field called \"->priv_*\" or\n> \"->private_*\". Someone will review those patches...\n\nAn internal struct with all the members meant to be internal-only\nprovides nearly all the advantages that I was going for with the\nopaque struct, while also being a smaller change than what I was\nthinking of doing. I like that idea; thanks for the suggestion.\n"},{"id":"437888","messageId":"CABPp-BFSFN3WM6q7KzkD5mhrwsz--St_-ej5LbaY8Yr2sZzj=w@mail.gmail.com","threadId":"56534","inReplyTo":"87sfxhohsj.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-04T13:45:00Z","receivedAt":"2021-10-04T13:55:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Oct 3, 2021 at 3:38 PM Ævar Arnfjörð Bjarmason wrote:\n>\n> On Sat, Oct 02 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n> > On Fri, Oct 01 2021, Elijah Newren wrote:\n> >\n> >> On Fri, Oct 1, 2021 at 1:47 AM Ævar Arnfjörð Bjarmason wrote:\n> >>>\n> >>> On Thu, Sep 30 2021, Elijah Newren wrote:\n> >>>\n> >>> > On Thu, Sep 30, 2021 at 7:15 AM Ævar Arnfjörð Bjarmason\n> >>> > wrote:\n> >>> >>\n> >>> >> On Wed, Sep 29 2021, Elijah Newren wrote:\n> [...]\n> >>> > I might be going on a tangent here, but looking at that patch, I'm\n> >>> > worried that dir_init() was buggy and that you perpetuated that bug\n> >>> > with DIR_INIT. Note that dir_struct has a struct strbuf basebuf\n> >>> > member, which neither dir_init() or DIR_INIT initialize properly\n> >>> > (using either strbuf_init() or STRBUF_INIT). As far as I can tell,\n> >>> > dir.c relies on either strbuf_add() calls to just happen to work with\n> >>> > this incorrectly initialized strbuf, or else use the strbuf_init()\n> >>> > call in prep_exclude() to do so, using the following snippet:\n> >>> >\n> >>> > if (!dir->basebuf.buf)\n> >>> > strbuf_init(&dir->basebuf, PATH_MAX);\n> >>> >\n> >>> > However, earlier in that same function we see\n> >>> >\n> >>> > if (stk->baselen <= baselen &&\n> >>> > !strncmp(dir->basebuf.buf, base, stk->baselen))\n> >>> > break;\n> >>> >\n> >>> > So either that function can never have dir->basebuf.buf be NULL and\n> >>> > the strbuf_init() is dead code, or else it's possible for us to\n> >>> > trigger a segfault. If it's the former, it may just be a ticking time\n> >>> > bomb that will transform into the latter with some other change,\n> >>> > because it's not at all obvious to me how dir->basebuf gets\n> >>> > initialized appropriately to avoid that strncmp call. Perhaps there\n> >>> > is some invariant where exclude_stack is only set up by previous calls\n> >>> > to prep_exclude() and those won't set up exclude_stack until first\n> >>> > initializing basebuf. But that really at least deserves a comment\n> >>> > about how we're abusing basebuf, and would probably be cleaner if we\n> >>> > initialized basebuf to STRBUF_INIT.\n> >>>\n> >>> ...because yes, I forgot about that when sending you the diff-on-top,\n> >>> sorry. Yes that's buggy with the diff-on-top I sent you.\n> >>\n> >> That bug didn't come from the diff-on-top you sent me, it came from\n> >> the commit already merged to master -- ce93a4c6127 (dir.[ch]: replace\n> >> dir_init() with DIR_INIT, 2021-07-01), merged as part of\n> >> ab/struct-init on Jul 16.\n> >\n> > Ah, I misunderstood you there. I'll look at that / fix it. Sorry.\n>\n> Just to tie up this loose end: Yes this control flow suck, and I've got\n> some patches to unpack-trees.[ch] & dir.[ch] I'm about to submit to fix\n> it. But just to comment on the existing behavior of the code, i.e. your\n> (above):\n>\n> \"So either that function can never have dir->basebuf.buf be NULL and\n> the strbuf_init() is dead code, or else it's possible for us to\n> trigger a segfault.\".\n>\n> I hadn't had time to look into it when I said I'd fix it, but now that I\n> have I found thath there's nothing to fix, and this code wasn't buggy\n> either before or after my ce93a4c6127 (dir.[ch]: replace dir_init() with\n> DIR_INIT, 2021-07-01). I.e. we do have the invariant you mentioned.\n>\n> The dir.[ch] API has always relied on the \"struct dir_struct\" being\n> zero'd out. First with memset() before your eceba532141 (dir: fix\n> problematic API to avoid memory leaks, 2020-08-18), and after my\n> ce93a4c6127 with the DIR_INIT, which both amount to the same thing.\n>\n> We both missed a caller that used neither dir_init() nor uses DIR_INIT\n> now, but it uses \"{ 0 }\", so it's always zero'd.\n>\n> Now, of course it being zero'd *would* segfault if you feed\n> \"dir->basebuf.buf\" to strncmp() as you note above, but that code isn't\n> reachable. The structure of that function is (pseudocode):\n>\n> void prep_exclude(...)\n> {\n> struct exclude_stack *stk = NULL;\n> [...]\n>\n> while ((stk = dir->exclude_stack) != NULL)\n> /* the strncmp() against \"dir->basebuf.buf\" is here */\n>\n> /* maybe we'll early return here */\n>\n> if (!dir->basebuf.buf)\n> strbuf_init(&dir->basebuf, PATH_MAX);\n>\n> /*\n> * Code that sets dir->exclude_stack to non-NULL for the first\n> * time follows...\n> */\n> }\n>\n> I.e. dir->exclude_stack is *only* referenced in this function and\n> dir_clear() (where we also check it for NULL first).\n>\n> It's state management between calls to prep_exclude(). So that that\n> initial while-loop can only be entered the the >1th time prep_exclude()\n> is called.\n>\n> We'll then either have reached that strbuf_init() already, or if we took\n> an early return before the strbuf_init() we couldn't have set\n> dir->exclude_stack either. So that \"dir->basebuf.buf\" dereference is\n> safe in either case.\n\nThanks for digging into this. I wonder if dir_struct could use some\nseparation of putting things inside an embedded internal struct as\nwell, similar to our discussions with unpack_trees_options.\n"},{"id":"437894","messageId":"8735pgop57.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"CABPp-BFE_5zPCZY8adJSQchdQzxq3uH-oma-S=_Sw0OUXx03OQ@mail.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T14:07:31Z","receivedAt":"2021-10-04T14:13:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 04 2021, Elijah Newren wrote:\n\n> On Sat, Oct 2, 2021 at 2:07 AM Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Fri, Oct 01 2021, Elijah Newren wrote:\n>>\n> ...\n>> > So maybe I'll submit some patches on top that rip these direct members\n>> > out of of unpack_trees_options and push them inside some opaque\n>> > struct.\n>>\n>> Sure, that sounds good. I only had a mild objection to doing it in a way\n>> where you'll need that sort of code I removed in the linked commit in\n>> prep_exclude() because you were trying not to expose that at any cost,\n>> including via some *_INIT macro. I.e. if it's private we can just name\n>> it \"priv_*\" or have a :\n>>\n>> struct dont_touch_this {\n>> struct dir_struct dir;\n>> };\n>>\n>> Which are both ways of /messaging/ that it's private, and since the\n>> target audience is just the rest of the git.git codebase I think that\n>> ultimately something that 1) sends the right message 2) makes accidents\n>> pretty much impossible suffices. I.e. you don't accidentally introduce a\n>> new API user accessing a field called \"->priv_*\" or\n>> \"->private_*\". Someone will review those patches...\n>\n> An internal struct with all the members meant to be internal-only\n> provides nearly all the advantages that I was going for with the\n> opaque struct, while also being a smaller change than what I was\n> thinking of doing. I like that idea; thanks for the suggestion.\n\nYeah, just to provide an explicit example something like the below. It\ncompiles to the same assembly (at least under -O3, didn't exhaustively\ntry other optimization levels).\n\nI'm rather \"meh\" on it v.s. just prefixing the relevant member names\nwith \"priv_\" or \"private_\", but it results in the same semantics &\nmachine code, so it's effectively just a way of doing the labeling for\nhuman consumption.\n\ndiff --git a/dir.c b/dir.c\nindex 39fce3bcba7..a714640e782 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1533,12 +1533,12 @@ static void prep_exclude(struct dir_struct *dir,\n \t * which originate from directories not in the prefix of the\n \t * path being checked.\n \t */\n-\twhile ((stk = dir->exclude_stack) != NULL) {\n+\twhile ((stk = dir->private.exclude_stack) != NULL) {\n \t\tif (stk->baselen <= baselen &&\n \t\t !strncmp(dir->basebuf.buf, base, stk->baselen))\n \t\t\tbreak;\n-\t\tpl = &group->pl[dir->exclude_stack->exclude_ix];\n-\t\tdir->exclude_stack = stk->prev;\n+\t\tpl = &group->pl[dir->private.exclude_stack->exclude_ix];\n+\t\tdir->private.exclude_stack = stk->prev;\n \t\tdir->pattern = NULL;\n \t\tfree((char *)pl->src); /* see strbuf_detach() below */\n \t\tclear_pattern_list(pl);\n@@ -1584,7 +1584,7 @@ static void prep_exclude(struct dir_struct *dir,\n \t\t\t\t\t\t base + current,\n \t\t\t\t\t\t cp - base - current);\n \t\t}\n-\t\tstk->prev = dir->exclude_stack;\n+\t\tstk->prev = dir->private.exclude_stack;\n \t\tstk->baselen = cp - base;\n \t\tstk->exclude_ix = group->nr;\n \t\tstk->ucd = untracked;\n@@ -1605,7 +1605,7 @@ static void prep_exclude(struct dir_struct *dir,\n \t\t\t dir->pattern->flags & PATTERN_FLAG_NEGATIVE)\n \t\t\t\tdir->pattern = NULL;\n \t\t\tif (dir->pattern) {\n-\t\t\t\tdir->exclude_stack = stk;\n+\t\t\t\tdir->private.exclude_stack = stk;\n \t\t\t\treturn;\n \t\t\t}\n \t\t}\n@@ -1662,7 +1662,7 @@ static void prep_exclude(struct dir_struct *dir,\n \t\t\tinvalidate_gitignore(dir->untracked, untracked);\n \t\t\toidcpy(&untracked->exclude_oid, &oid_stat.oid);\n \t\t}\n-\t\tdir->exclude_stack = stk;\n+\t\tdir->private.exclude_stack = stk;\n \t\tcurrent = stk->baselen;\n \t}\n \tstrbuf_setlen(&dir->basebuf, baselen);\n@@ -3302,7 +3302,7 @@ void dir_clear(struct dir_struct *dir)\n \tfree(dir->ignored);\n \tfree(dir->entries);\n \n-\tstk = dir->exclude_stack;\n+\tstk = dir->private.exclude_stack;\n \twhile (stk) {\n \t\tstruct exclude_stack *prev = stk->prev;\n \t\tfree(stk);\ndiff --git a/dir.h b/dir.h\nindex 83f46c0fb4c..d30d294308d 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -209,6 +209,11 @@ struct untracked_cache {\n * record the paths discovered. A single `struct dir_struct` is used regardless\n * of whether or not the traversal recursively descends into subdirectories.\n */\n+\n+struct dir_struct_private {\n+\tstruct exclude_stack *exclude_stack;\n+};\n+\n struct dir_struct {\n \n \t/* The number of members in `entries[]` array. */\n@@ -327,7 +332,7 @@ struct dir_struct {\n \t * (sub)directory in the traversal. Exclude points to the\n \t * matching exclude struct if the directory is excluded.\n \t */\n-\tstruct exclude_stack *exclude_stack;\n+\tstruct dir_struct_private private;\n \tstruct path_pattern *pattern;\n \tstruct strbuf basebuf;\n \n"},{"id":"437899","messageId":"CABPp-BEaJAG=pqyjCR93YUoqj06WwSqjiPENDHjgSTWwzd_C2A@mail.gmail.com","threadId":"56534","inReplyTo":"RFC-cover-v4-00.10-00000000000-20211004T004902Z-avarab@gmail.com","subject":"Re: [RFC PATCH v4 00/10] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-04T14:38:17Z","receivedAt":"2021-10-04T14:40:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Oct 3, 2021 at 6:12 PM Ævar Arnfjörð Bjarmason wrote:\n>\n> This is an RFC proposed v4 of Elijah's en/removing-untracked-fixes\n> series[1] based on top of my memory leak fixes in the \"unpack-trees\" &\n> \"dir\" APIs[2].\n>\n> As noted in [2] Elijah and I have been having a back & forth about the\n> approach his series takes to fixing memory leaks in those APIs. I\n> think submitting working code is more productive than continuing that\n> point-by-point discussion, so here we are.\n>\n> I've avoided making any changes to this series except those narrowly\n> required to rebase it on top of mine, and to those parts of Elijah's\n> commit messages that became outdated as a result. In particular\n> 3/10[3]'s is significantly changed, as much of its commit message\n> dicusses complexities that have gone away due to my preceding\n> series[2].\n>\n> The \"make dir an internal-only struct\" has been replaced by a commit\n> that renames that struct member from \"dir\" to \"private_dir\". I think\n> even that is unnecessary as argued in [4], but I think the judgement\n> that something must be done to address that is Elijah's design\n> decision, so I did my best to retain it.\n>\n> I did drop the dynamic allocation & it being a pointer, since with my\n> preceding [2] and subsequent unsubmitted memory leak fixes I've got on\n> top having it be embedded in \"struct unpack_trees_options\" makes\n> things easier to manage.\n>\n> Havingn read through all this code quite thoroughly at this point I do\n> have other comments on it, but I'll reserve those until we've found\n> out what direction we're going forward with vis-a-vis what this will\n> be based on top of.\n>\n> I'm (obviously) hoping for an answer of either on top of my series[2],\n> or alternatively that Elijah's series can stick to introducing the\n> \"preserve_ignored\" flag, but not change how the memory\n> management/name/type of the embedded \"dir\" happens (and we could thus\n> proceed in parallel).\n\n???\n\nThis really bothers me. I'm not quite sure how to put this into\nwords, so let me just try my best. Let me start out by saying that I\nthink you often provide good feedback and ideas. Sure, I sometimes\ndon't agree with some of the feedback or ideas, but overall your\nfeedback and contributions are definitely valuable. I also think your\nother series you rebased this on has some good ideas and some good\nbugfixes. There is something that seems off here, though.\n\nIn this particular case, to start with, Junio already said let's take\nv3 as-is[1]. So your series should be rebased on mine, not\nvice-versa.\n\nFurther while your other series that you are basing this on has some\nmemory leak fixes; to me, it mostly looks like refactorings for\nstylistic code changes. Even though some of those stylistic changes\nare good, making a series such as mine that includes bugfixes (to a\nuser reported bug no less), after multiple rounds and most reviewers\nare fine with it, suddenly depend on a new big and unrelated treewide\nstylistic refactoring series feels very off to me. But that doesn't\nquite fully explain my misgivings either; there's a bit more:\n\n * Junio has referred to several of your series as \"Meh\" and \"code\nchurn\". That makes me think we'd have a higher than normal chance of\na user-reported bug ending up blocked on unrelated stylistic changes.\n(Two of them actually, since I have another series depending on this\none that I've waited to submit until this merges to next.)\n * Your stylistic refactorings also manage to confuse the code in\nmerge-recursive.c, overall making the code potentially much harder to\nunderstand[2][3]. And you open a foot-gun in\nclear_unpack_trees_porcelain[3].\n * At least half the series of yours I've reviewed have had\nsignificant bugs[4][5][6] (in addition to [2] and [3]). This would be\nfine if it was complex code that had bugs we were fixing, or if we\nwere adding new features, but:\n * You submit a huge volume of patches, with a very\ndisproportionately high ratio of stylistic refactorings rather than\nbugfixes and new features. (This is by no means bad on its own, it's\nthe combination of this with other factors.)\n * You misrepresent my changes in multiple ways, including ways I had\npointed out corrections for in our previous discussions (including\nsome of which you acknowledged and agreed with), and you do so even\nafter you have rebased my patches and added your signed-off-by to them\nsuggesting you ought to be familiar with them[7].\n\nSo, I guess trying to distill what bugs me, I'd say: it seems to me\nthat you have ignored what Junio said about taking my series, and then\nyou rebased my series on top of unrelated stylistic churn, with that\nchurn containing three issues that trigger ongoing misgivings I have\nabout the care being put behind these refactorings, especially\nconsidering their value compared to the features and bugfixes we are\ngetting, and you seem to fail to try to understand my changes and\nmisrepresent them in the process. I hope I'm not overreacting but\nsomething feels wrong to me here.\n\n\nA big thumbs down on this reroll.\n\n[1] https://lore.kernel.org/git/xmqq35pk8ylz.fsf@gitster.g/\n[2] https://lore.kernel.org/git/CABPp-BFYxWXZQXvDSrM1Y1ZaQ1d2TENQDvx1cyawvrDO1OvExA@mail.gmail.com/\n[3] https://lore.kernel.org/git/CABPp-BH4ubjJ98Nvgp2iyKxmU9X+ypw4m1o=iL9Z4vSNZ-QTDw@mail.gmail.com/\n[4] https://lore.kernel.org/git/CABPp-BGE+e1er6qFuG90j9+eVG34O8TN=imX=jtcb+jbQjN1QQ@mail.gmail.com/\n[5] https://lore.kernel.org/git/CABPp-BEPkukGz32rrro1hzMvSQzX4v7U17CAcV-G2NS6v0u55g@mail.gmail.com/\n[6] https://lore.kernel.org/git/xmqqfstppxzm.fsf@gitster.g/ [Note:\nproblem was flagged by j6t; I was about to flag the same problem when\nI noticed he had already done so.]\n[7] https://lore.kernel.org/git/CABPp-BEr28xzbpEZc5dq-RVDupXy+h-+PH6CoANF4e0kmxqf0Q@mail.gmail.com/\n"},{"id":"437903","messageId":"CABPp-BHgot=CPNyK_xNfog_SqsNPNoCGfiSb-gZoS2sn_741dQ@mail.gmail.com","threadId":"56534","inReplyTo":"8735pgop57.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/11] unpack-trees: introduce preserve_ignored to unpack_trees_options","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-04T14:57:38Z","receivedAt":"2021-10-04T14:59:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 4, 2021 at 7:12 AM Ævar Arnfjörð Bjarmason wrote:\n>\n>\n> On Mon, Oct 04 2021, Elijah Newren wrote:\n>\n> > On Sat, Oct 2, 2021 at 2:07 AM Ævar Arnfjörð Bjarmason wrote:\n> >>\n> >> On Fri, Oct 01 2021, Elijah Newren wrote:\n> >>\n> > ...\n> >> > So maybe I'll submit some patches on top that rip these direct members\n> >> > out of of unpack_trees_options and push them inside some opaque\n> >> > struct.\n> >>\n> >> Sure, that sounds good. I only had a mild objection to doing it in a way\n> >> where you'll need that sort of code I removed in the linked commit in\n> >> prep_exclude() because you were trying not to expose that at any cost,\n> >> including via some *_INIT macro. I.e. if it's private we can just name\n> >> it \"priv_*\" or have a :\n> >>\n> >> struct dont_touch_this {\n> >> struct dir_struct dir;\n> >> };\n> >>\n> >> Which are both ways of /messaging/ that it's private, and since the\n> >> target audience is just the rest of the git.git codebase I think that\n> >> ultimately something that 1) sends the right message 2) makes accidents\n> >> pretty much impossible suffices. I.e. you don't accidentally introduce a\n> >> new API user accessing a field called \"->priv_*\" or\n> >> \"->private_*\". Someone will review those patches...\n> >\n> > An internal struct with all the members meant to be internal-only\n> > provides nearly all the advantages that I was going for with the\n> > opaque struct, while also being a smaller change than what I was\n> > thinking of doing. I like that idea; thanks for the suggestion.\n>\n> Yeah, just to provide an explicit example something like the below. It\n> compiles to the same assembly (at least under -O3, didn't exhaustively\n> try other optimization levels).\n>\n> I'm rather \"meh\" on it v.s. just prefixing the relevant member names\n> with \"priv_\" or \"private_\", but it results in the same semantics &\n> machine code, so it's effectively just a way of doing the labeling for\n> human consumption.\n>\n> diff --git a/dir.c b/dir.c\n> index 39fce3bcba7..a714640e782 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1533,12 +1533,12 @@ static void prep_exclude(struct dir_struct *dir,\n> * which originate from directories not in the prefix of the\n> * path being checked.\n> */\n> - while ((stk = dir->exclude_stack) != NULL) {\n> + while ((stk = dir->private.exclude_stack) != NULL) {\n> if (stk->baselen <= baselen &&\n> !strncmp(dir->basebuf.buf, base, stk->baselen))\n> break;\n> - pl = &group->pl[dir->exclude_stack->exclude_ix];\n> - dir->exclude_stack = stk->prev;\n> + pl = &group->pl[dir->private.exclude_stack->exclude_ix];\n> + dir->private.exclude_stack = stk->prev;\n> dir->pattern = NULL;\n> free((char *)pl->src); /* see strbuf_detach() below */\n> clear_pattern_list(pl);\n> @@ -1584,7 +1584,7 @@ static void prep_exclude(struct dir_struct *dir,\n> base + current,\n> cp - base - current);\n> }\n> - stk->prev = dir->exclude_stack;\n> + stk->prev = dir->private.exclude_stack;\n> stk->baselen = cp - base;\n> stk->exclude_ix = group->nr;\n> stk->ucd = untracked;\n> @@ -1605,7 +1605,7 @@ static void prep_exclude(struct dir_struct *dir,\n> dir->pattern->flags & PATTERN_FLAG_NEGATIVE)\n> dir->pattern = NULL;\n> if (dir->pattern) {\n> - dir->exclude_stack = stk;\n> + dir->private.exclude_stack = stk;\n> return;\n> }\n> }\n> @@ -1662,7 +1662,7 @@ static void prep_exclude(struct dir_struct *dir,\n> invalidate_gitignore(dir->untracked, untracked);\n> oidcpy(&untracked->exclude_oid, &oid_stat.oid);\n> }\n> - dir->exclude_stack = stk;\n> + dir->private.exclude_stack = stk;\n> current = stk->baselen;\n> }\n> strbuf_setlen(&dir->basebuf, baselen);\n> @@ -3302,7 +3302,7 @@ void dir_clear(struct dir_struct *dir)\n> free(dir->ignored);\n> free(dir->entries);\n>\n> - stk = dir->exclude_stack;\n> + stk = dir->private.exclude_stack;\n> while (stk) {\n> struct exclude_stack *prev = stk->prev;\n> free(stk);\n> diff --git a/dir.h b/dir.h\n> index 83f46c0fb4c..d30d294308d 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -209,6 +209,11 @@ struct untracked_cache {\n> * record the paths discovered. A single `struct dir_struct` is used regardless\n> * of whether or not the traversal recursively descends into subdirectories.\n> */\n> +\n> +struct dir_struct_private {\n> + struct exclude_stack *exclude_stack;\n> +};\n> +\n> struct dir_struct {\n>\n> /* The number of members in `entries[]` array. */\n> @@ -327,7 +332,7 @@ struct dir_struct {\n> * (sub)directory in the traversal. Exclude points to the\n> * matching exclude struct if the directory is excluded.\n> */\n> - struct exclude_stack *exclude_stack;\n> + struct dir_struct_private private;\n> struct path_pattern *pattern;\n> struct strbuf basebuf;\n\nYeah, that doesn't help much at all, and I'd argue even makes things\nworse, because you're just looking at a single member. This subtly\nimplies that all the other private variables are public API. The\ndir.h portion of the patch should look more like this:\n\n$ git diff -w dir.h\ndiff --git a/dir.h b/dir.h\nindex 83f46c0fb4..93a9f02688 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -214,14 +214,9 @@ struct dir_struct {\n /* The number of members in `entries[]` array. */\n int nr;\n\n- /* Internal use; keeps track of allocation of `entries[]` array.*/\n- int alloc;\n-\n /* The number of members in `ignored[]` array. */\n int ignored_nr;\n\n- int ignored_alloc;\n-\n /* bit-field of options */\n enum {\n\n@@ -301,11 +296,19 @@ struct dir_struct {\n */\n const char *exclude_per_dir;\n\n+ struct dir_struct_internal {\n+ /* Keeps track of allocation of `entries[]` array.*/\n+ int alloc;\n+\n+ /* Keeps track of allocation of `ignored[]` array. */\n+ int ignored_alloc;\n+\n /*\n * We maintain three groups of exclude pattern lists:\n *\n * EXC_CMDL lists patterns explicitly given on the command line.\n- * EXC_DIRS lists patterns obtained from per-directory ignore files.\n+ * EXC_DIRS lists patterns obtained from per-directory ignore\n+ * files.\n * EXC_FILE lists patterns from fallback ignore files, e.g.\n * - .git/info/exclude\n * - core.excludesfile\n@@ -340,6 +343,7 @@ struct dir_struct {\n /* Stats about the traversal */\n unsigned visited_paths;\n unsigned visited_directories;\n+ } internal;\n };\n\n #define DIR_INIT { 0 }\n\n\nThe above change would make it clear that there are 12 variables meant\nfor use only within dir.c that external callers should not be\ninitializing or reading for output after the fact -- and only 6 that\nare part of the public API that they need worry about. It also makes\nit easier for folks messing with dir.c to know which parts are just\ninternal state management, which I think would have made it easier to\nunderstand the weird basebuf/exclude_stack stuff in prep_exclude()\nthat you nicely tracked down. But overall, I'm really most happy\nabout the part of this patch that lets external callers realize they\nonly need to worry about 6 out of 18 fields and that they can ignore\nthe rest.\n\nunpack_trees_options should have something similar done with it, and\nmaybe some others.\n"},{"id":"437926","messageId":"87bl44n070.fsf@evledraar.gmail.com","threadId":"56534","inReplyTo":"CABPp-BEaJAG=pqyjCR93YUoqj06WwSqjiPENDHjgSTWwzd_C2A@mail.gmail.com","subject":"Re: [RFC PATCH v4 00/10] Fix various issues around removal of untracked files/directories","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-04T16:08:05Z","receivedAt":"2021-10-04T17:56:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 04 2021, Elijah Newren wrote:\n\n> On Sun, Oct 3, 2021 at 6:12 PM Ævar Arnfjörð Bjarmason wrote:\n>>\n>> This is an RFC proposed v4 of Elijah's en/removing-untracked-fixes\n>> series[1] based on top of my memory leak fixes in the \"unpack-trees\" &\n>> \"dir\" APIs[2].\n>>\n>> As noted in [2] Elijah and I have been having a back & forth about the\n>> approach his series takes to fixing memory leaks in those APIs. I\n>> think submitting working code is more productive than continuing that\n>> point-by-point discussion, so here we are.\n>>\n>> I've avoided making any changes to this series except those narrowly\n>> required to rebase it on top of mine, and to those parts of Elijah's\n>> commit messages that became outdated as a result. In particular\n>> 3/10[3]'s is significantly changed, as much of its commit message\n>> dicusses complexities that have gone away due to my preceding\n>> series[2].\n>>\n>> The \"make dir an internal-only struct\" has been replaced by a commit\n>> that renames that struct member from \"dir\" to \"private_dir\". I think\n>> even that is unnecessary as argued in [4], but I think the judgement\n>> that something must be done to address that is Elijah's design\n>> decision, so I did my best to retain it.\n>>\n>> I did drop the dynamic allocation & it being a pointer, since with my\n>> preceding [2] and subsequent unsubmitted memory leak fixes I've got on\n>> top having it be embedded in \"struct unpack_trees_options\" makes\n>> things easier to manage.\n>>\n>> Havingn read through all this code quite thoroughly at this point I do\n>> have other comments on it, but I'll reserve those until we've found\n>> out what direction we're going forward with vis-a-vis what this will\n>> be based on top of.\n>>\n>> I'm (obviously) hoping for an answer of either on top of my series[2],\n>> or alternatively that Elijah's series can stick to introducing the\n>> \"preserve_ignored\" flag, but not change how the memory\n>> management/name/type of the embedded \"dir\" happens (and we could thus\n>> proceed in parallel).\n>\n> ???\n>\n> This really bothers me. I'm not quite sure how to put this into\n> words, so let me just try my best. Let me start out by saying that I\n> think you often provide good feedback and ideas. Sure, I sometimes\n> don't agree with some of the feedback or ideas, but overall your\n> feedback and contributions are definitely valuable. I also think your\n> other series you rebased this on has some good ideas and some good\n> bugfixes. There is something that seems off here, though.\n\nJust for Junio / anyone else following along: let's drop this RFC & the\nrelateded/proposed \"unpack-trees & dir APIs: fix memory\nleaks\". Point-by-point commentary below (probably not needed/interesting\nfor those just interested in the state of those two serieses).\n\n> In this particular case, to start with, Junio already said let's take\n> v3 as-is[1]. So your series should be rebased on mine, not\n> vice-versa.\n\nI understand your annoyance at that, I wouldn't have submitted this if\nI'd seen that before, I somehow managed to miss that mail in my mail\nqueue. I believed the status was at \"Will merge to 'next'?\" upthread of\n[1].\n\nWe've then been having an extended back & forth about how to manage\n\"private\" data/structs/leak patterns starting at\nhttps://lore.kernel.org/git/87ilyjviiy.fsf@evledraar.gmail.com/.\n\nAt least some of which has been confused by my having quoted a working\nbut out of context diff from what I ended up submitting as\nhttps://lore.kernel.org/git/cover-00.10-00000000000-20211004T002226Z-avarab@gmail.com/\n\nSo, sorry about stepping on your toes. I figured having a discussion\nwith working patches would be more productive, and that it would help\nfocus on the important changes in your series.\n\nE.g. your 2nd and 3rd patch setup a \"dir_clear()\" that your 4th\nconsolidates, which as shown in this series are intermediate steps that\ncan be skipped. So perhaps there's some added churn, but also reduced\nchurn in your resulting series on top...\n\n> Further while your other series that you are basing this on has some\n> memory leak fixes; to me, it mostly looks like refactorings for\n> stylistic code changes. [...]\n\nAre you referring to the s/memset/UNPACK_TREES_OPTIONS_INIT/ bulk change\nat the start?\n\nI agree that it's not strictly necessary, but it's pretty much the same\nas your earlier eceba532141 (dir: fix problematic API to avoid memory\nleaks, 2020-08-18), and makes e.g. a later change you seemed to like\npossible:\nhttps://lore.kernel.org/git/CABPp-BFpyyJ-e8p5fbmCvyaEsfUow=RP45Nw0ckiwNEvVC4zrg@mail.gmail.com/\n\n> Even though some of those stylistic changes\n> are good, making a series such as mine that includes bugfixes (to a\n> user reported bug no less), after multiple rounds and most reviewers\n> are fine with it, suddenly depend on a new big and unrelated treewide\n> stylistic refactoring series feels very off to me. But that doesn't\n> quite fully explain my misgivings either; there's a bit more:\n\n[...]\n\n> * Junio has referred to several of your series as \"Meh\" and \"code\n> churn\". That makes me think we'd have a higher than normal chance of\n> a user-reported bug ending up blocked on unrelated stylistic changes.\n> (Two of them actually, since I have another series depending on this\n> one that I've waited to submit until this merges to next.)\n\nI'll stay out of this area for a while. Sorry about that.\n\n> [...]\n> * You misrepresent my changes in multiple ways, including ways I had\n> pointed out corrections for in our previous discussions (including\n> some of which you acknowledged and agreed with), and you do so even\n> after you have rebased my patches and added your signed-off-by to them\n> suggesting you ought to be familiar with them[7].\n\nYou're absolutely right about that, and the comment starting at \"*Sigh*\"\nin your linked [7] is entirely accurate. I'd like to apologize for that.\n\nIf I was in your shoes I'd be *very* annoyed at that chain from [7] to\nthe upthread cover-letter here making that false claim again.\n\nFor what it's worth I do know that your patches aren't allocating the\n\"struct dir_struct\" on the heap, having rebased them etc. But through\nsome combination of a brainfart and it being late when I wrote that CL\nlast night I falsely claimed that they did, PBCAK.\n\nWhat I *meant* to say was some summary that the your series's end state\nhas a pointer to a dir struct that's dynamically set up v.s. mine of\njust initializing it via the macro from the start.\n\n> So, I guess trying to distill what bugs me, I'd say: it seems to me\n> that you have ignored what Junio said about taking my series, and then\n> you rebased my series on top of unrelated stylistic churn, with that\n> churn containing three issues that trigger ongoing misgivings I have\n> about the care being put behind these refactorings, especially\n> considering their value compared to the features and bugfixes we are\n> getting, and you seem to fail to try to understand my changes and\n> misrepresent them in the process. I hope I'm not overreacting but\n> something feels wrong to me here.\n\nI don't think you're overreacting, and sorry again. Hopefully it helps\nsomewhat that I for the \"ignoring Junio [and charging ahead with this]\"\nand the 2nd false claim about about heap allocation I was (believe it or\nnot) just honestly mistaken instead of trying to get on your nerves.\n\nAs some of my early feedback on the whole topic of gitignore\nrelated-shredding/precious etc. should hopefully indicate I'm really\nhappy that you've picked up this topic.\n\nI thought this would help it along, but that's clearly not the\ncase. Sorry again.\n"},{"id":"437929","messageId":"xmqqh7dwwt6y.fsf@gitster.g","threadId":"56534","inReplyTo":"CABPp-BEaJAG=pqyjCR93YUoqj06WwSqjiPENDHjgSTWwzd_C2A@mail.gmail.com","subject":"Re: [RFC PATCH v4 00/10] Fix various issues around removal of untracked files/directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-04T18:17:25Z","receivedAt":"2021-10-04T18:17:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> So, I guess trying to distill what bugs me, I'd say: it seems to me\n> that you have ignored what Junio said about taking my series, and then\n> you rebased my series on top of unrelated stylistic churn, with that\n> churn containing three issues that trigger ongoing misgivings I have\n> about the care being put behind these refactorings, especially\n> considering their value compared to the features and bugfixes we are\n> getting, and you seem to fail to try to understand my changes and\n> misrepresent them in the process. I hope I'm not overreacting but\n> something feels wrong to me here.\n\nJust one thing. It is fairly easy to rebase and merge and generally\nmuck with somebody else's changes without fully understanding it. I\ndo that all the time ;-)\n\nI would prefer to see us assume misunderstanding by incompetence,\nrather than misrepresentation by malice to further one's own agenda.\n"},{"id":"437958","messageId":"CABPp-BEF8=yw__r3p=spt647+tLG0UshaXq45ePEsFvv89i9Ng@mail.gmail.com","threadId":"56534","inReplyTo":"87bl44n070.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH v4 00/10] Fix various issues around removal of untracked files/directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-05T07:40:38Z","receivedAt":"2021-10-05T07:40:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Ævar,\n\nOn Mon, Oct 4, 2021 at 10:56 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Mon, Oct 04 2021, Elijah Newren wrote:\n>\n> > So, I guess trying to distill what bugs me, I'd say: [...] I hope I'm not overreacting but\n> > something feels wrong to me here.\n>\n> I don't think you're overreacting, and sorry again. Hopefully it helps\n> somewhat that I for the \"ignoring Junio [and charging ahead with this]\"\n> and the 2nd false claim about about heap allocation I was (believe it or\n> not) just honestly mistaken instead of trying to get on your nerves.\n\nI really wish I could take back that email. And yes, I was totally\noverreacting (in part due to unrelated non-git stuff going on\nrecently). I should have waited a day, and then I'd probably realize\nit. I owe you an apology, Ævar. I'm very sorry.\n\nIn regards to the worst part of my email:\n\n\"\"\"\n * At least half the series of yours I've reviewed have had\nsignificant bugs[4][5][6] (in addition to [2] and [3]). This would be\nfine if it was complex code that had bugs we were fixing, or if we\nwere adding new features, but:\n\"\"\"\n\nI'm no stranger to introducing pretty bad bugs either, some caught in\nreview (one case of repository corruption caught in a review just last\nweek!), some making it into releases:\n\nhttps://lore.kernel.org/git/YVOn3hDsb5pnxR53@coredump.intra.peff.net/\nhttps://lore.kernel.org/git/DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com/\nhttps://lore.kernel.org/git/CABPp-BFWfwkYAPyySjWOMZ02_+YLf=TJ_aVMaHaizJWAsCL67g@mail.gmail.com/\nhttps://lore.kernel.org/git/CABPp-BENB=mqfFU4FGb2OS9VDV=9VdT71HhFLZwtyxD8MpdTMQ@mail.gmail.com/\nhttps://lore.kernel.org/git/CABPp-BEBKyE2NVfREov6k5qML5jryLjtzw=Y21EA=fHXA0PO5A@mail.gmail.com/\nhttps://lore.kernel.org/git/CABPp-BF8eokQTVwgo80ffq3tn=NA=mPf7oymce5P4itDyZBiMg@mail.gmail.com/\nhttps://lore.kernel.org/git/7v1uzu5a70.fsf@alter.siamese.dyndns.org/\nhttps://lore.kernel.org/git/CABPp-BHL4P0RxQ6OAuDSev9BXVM0uKTYD3M4JGTQvSwcBv4K0Q@mail.gmail.com/\n\nand I could easily find more, it was pretty easy to come up with that\nlist of bugs that still make me shudder. Touching difficult code, or\ncode one isn't quite as familiar with is more prone to bigger\nproblems. That doesn't mean we should avoid working on those areas,\nbut I'm afraid my email probably served only to scare people away from\ndoing so. Treewide refactoring is, by its nature, likely to take one\ninto many areas of the codebase, and thus it'd be natural to expect\nproblems that hopefully get caught in review. The fact that such\ntreewide refactorings did have problems, was by itself not my point\nand not what I'd consider unusual. I was attempting to make a more\nnuanced point about lots of treewide refactorings in a short time\nperiod coupled with lack of understanding of motivation for some of\nthose refactorings all combined with additional things on top, but\nutterly failed at anything more than coming across as a jerk. I'm\nsorry.\n\nThanks for responding to my email so diplomatically; I'm super\nimpressed with that. And just to be clear, I respect your\ncontributions and hold them in high esteem (prove, i18n, pcre2, faster\nsend-email to name just a few). I think that message was\nunfortunately completely lost in my email, which is plenty of reason\nthat I just shouldn't have sent it.\n"},{"id":"444012","messageId":"CA+6di1m91q70PfaFq0DKMsmd_Tb6XBB7H9AYPhwawX12cZgtGQ@mail.gmail.com","threadId":"56534","inReplyTo":"2501a0c552ad5147f61a96b9ebe45c5199e1dbfd.1632760428.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/11] read-tree, merge-recursive: overwrite ignored files by default","fromName":"Jack O'Connor","fromEmail":"oconnor663@gmail.com","sentAt":"2021-12-13T17:12:08Z","receivedAt":"2021-12-13T17:12:37Z","isPatch":true,"sender":{"key":"oconnor663@gmail.com","avatar":"https://gravatar.com/avatar/221557916c71e91c6adf326e4fcc9c8bd71dd4b855a7a2ad670107ff7155330d?d=mp&s=160"},"body":"> read-tree, merge-recursive: overwrite ignored files by default\n\nWhen this patch shipped in v1.34, a test broke in a project of mine\n(https://github.com/buildinspace/peru/blob/e9ba6e0024ea08105a8d027f958899cca39aeb9a/tests/test_cache.py#L111-L117)\nthat was relying on git read-tree *not* to respect .gitignore files.\n(Obligatory https://xkcd.com/1172.) That peru tool is using git\nplumbing commands to manage trees of files, but it tries to keep this\nimplementation detail internal, and behaving differently in the\npresence of a .gitignore file belonging to the user would leak this\ninternal implementation detail. I've been trying to figure out a way\nto reproduce the Git 1.33 behavior in Git 1.34, but so far I haven't\nfound any flags or configs to do that. (For example, putting !* in\n.git/info/exclude doesn't seem to help, I think because a .gitignore\nfile in the working tree takes precedence.) Can anyone suggest another\nworkaround?\n\nThis is my first mail to this list, so please let me know if I mess up\nthe etiquette.\n\n- Jack\n"},{"id":"444027","messageId":"CABPp-BEZj4opBu+wo2whOCHrck1qMjZXfDY9tAE5FbB=Wr9F_w@mail.gmail.com","threadId":"56534","inReplyTo":"CA+6di1m91q70PfaFq0DKMsmd_Tb6XBB7H9AYPhwawX12cZgtGQ@mail.gmail.com","subject":"Re: [PATCH v3 03/11] read-tree, merge-recursive: overwrite ignored files by default","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-13T20:10:31Z","receivedAt":"2021-12-13T20:10:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Dec 13, 2021 at 9:12 AM Jack O'Connor wrote:\n>\n> > read-tree, merge-recursive: overwrite ignored files by default\n>\n> When this patch shipped in v1.34, a test broke in a project of mine\n> (https://github.com/buildinspace/peru/blob/e9ba6e0024ea08105a8d027f958899cca39aeb9a/tests/test_cache.py#L111-L117)\n> that was relying on git read-tree *not* to respect .gitignore files.\n> (Obligatory https://xkcd.com/1172.) That peru tool is using git\n> plumbing commands to manage trees of files, but it tries to keep this\n> implementation detail internal, and behaving differently in the\n> presence of a .gitignore file belonging to the user would leak this\n> internal implementation detail. I've been trying to figure out a way\n> to reproduce the Git 1.33 behavior in Git 1.34, but so far I haven't\n> found any flags or configs to do that. (For example, putting !* in\n> .git/info/exclude doesn't seem to help, I think because a .gitignore\n> file in the working tree takes precedence.) Can anyone suggest another\n> workaround?\n>\n> This is my first mail to this list, so please let me know if I mess up\n> the etiquette.\n\nYour email is fine. :-) Interesting usage case; thanks for sending it along.\n\nDigging a bit into your repository, it appears this all started\nbecause you noticed that checkout would overwrite ignored files, and\nso you switched to reset --keep (in your 637d5c042262 (make cache\nexport refuse to pave .gitgnored files, 2014-07-22)) and then to\nread-tree (in your 057d1af600f9 (Rewrite `export_tree` to allow\ndeleted files., 2014-08-05)) to avoid having ignored files be\noverwritten. You could have stuck with `git checkout` all along, and\njust passed it the --no-overwrite-ignore flag. You probably just\nmissed the existence of that flag, because Duy forgot to document it\nfor 8 years (see git.git's commit 9d223d43e5 (\"doc: document\n--overwrite-ignore\", 2019-03-29)) Going back to checkout might\nprovide you a workaround. (Also, another random thing I noticed while\nlooking at your repo: `--diff-filter=d` is a much better way of\nchecking for not-deleted-changes than using `--diff-filter=ACMRTUXB`.\nNote the lowercase 'd' rather than uppercase.)\n\nYour report suggests more places should accept the\n--no-overwrite-ignore flag, which I alluded to as a possibility in the\nsixth patch in the series (\"Remove ignored files by default when they\nare in the way\"[1]) and the comments in the cover letter about\nprecious ignored files (under \"SIDENOTE about treating ignored files\nas precious\"[2]). And perhaps we could have a core.overwriteIgnore\nconfig option for setting a different global default (also as alluded\nto in my cover letter). Doing things would provide additional\nworkarounds, and finally provide the \"precious ignored\" concept that\nhas been discussed occasionally. I think it's not too hard to do that\non top of my previous patch series. I'll try to take a look after\nsome other in-flight series finally land.\n\n\n[1] https://lore.kernel.org/git/b7fe354effff8da3de53bd9cc40a03b5fd455f67.1632760428.git.gitgitgadget@gmail.com/\n[2] https://lore.kernel.org/git/pull.1036.v3.git.1632760428.gitgitgadget@gmail.com/\n"}]}