{"thread":{"id":"53444","subject":"[PATCH 1/7] subtree: handle multiple parents passed to cache_miss","startedAt":"2020-05-11T05:50:06Z","lastAt":"2020-10-07T19:46:45Z","messageCount":28,"participants":["Tom Clarkson via GitGitGadget","Ed Maste","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"397522","messageId":"pull.493.git.1589176201.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":null,"subject":"[PATCH 0/7] subtree: Fix handling of complex history","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:49:54Z","receivedAt":"2020-05-11T05:50:06Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"Fixes several issues that could occur when running subtree split on large\nrepos with more complex history.\n\n 1. A merge commit could bypass the known start point of the subtree, which\n    would cause the entire history to be processed recursively, leading to a\n    stack overflow / segfault after reading a few hundred commits. Older\n    commits are now explicitly recorded as irrelevant so that the recursive\n    process can terminate on any mainline commit rather than only on subtree\n    joins and initial commits.\n    \n    \n 2. It is possible for a repo to contain subtrees that lack the metadata\n    that is usually present in add/join commit messages (git-svn at least\n    can produce such a structure). The new use/ignore/map commands allow the\n    user to provide that information for any problematic commits.\n    \n    \n 3. A mainline commit that does not contain the subtree folder could be\n    erroneously identified as a subtree commit, which would add the entire\n    mainline history to the subtree. Commits will now only be used as is if\n    all their parents are already identified as subtree commits. While the\n    new code can still be tripped up by unusual folder structures, the\n    completely unambiguous solution turned out to involve a significant\n    performance penalty, and the new ignore / use commands provide a\n    workaround for that scenario.\n\nTom Clarkson (7):\n  subtree: handle multiple parents passed to cache_miss\n  subtree: exclude commits predating add from recursive processing\n  subtree: persist cache between split runs\n  subtree: add git subtree map command\n  subtree: add git subtree use and ignore commands\n  subtree: more robustly distinguish subtree and mainline commits\n  subtree: document new subtree commands\n\n contrib/subtree/git-subtree.sh  | 185 ++++++++++++++++++++++++++------\n contrib/subtree/git-subtree.txt |  24 +++++\n 2 files changed, 177 insertions(+), 32 deletions(-)\n\n\nbase-commit: af6b65d45ef179ed52087e80cb089f6b2349f4ec\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-493%2Ftqc%2Ftqc%2Fsubtree-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-493/tqc/tqc/subtree-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/493\n-- \ngitgitgadget\n"},{"id":"397521","messageId":"74fa6704905d5fa4b60adb2eefa0708223010e94.1589176201.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 1/7] subtree: handle multiple parents passed to cache_miss","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:49:55Z","receivedAt":"2020-05-11T05:50:07Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 868e18b9a1a..9867718503c 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -238,7 +238,7 @@ cache_miss () {\n }\n \n check_parents () {\n-\tmissed=$(cache_miss \"$1\")\n+\tmissed=$(cache_miss $1)\n \tlocal indent=$(($2 + 1))\n \tfor miss in $missed\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"397523","messageId":"87af5a316a2ca75e28fda466a04c27bdb02b96b2.1589176201.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 2/7] subtree: exclude commits predating add from recursive processing","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:49:56Z","receivedAt":"2020-05-11T05:50:07Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nInclude recursion depth in debug logs so we can see when the recursion is\ngetting out of hand.\n\nMaking the cache handle null mappings correctly and adding older commits\nto the cache allows the recursive algorithm to terminate at any point on\nmainline rather than needing to reach either the add point or the initial\ncommit.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 37 +++++++++++++++++++++++++++++++---\n 1 file changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 9867718503c..da0eede6979 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -244,7 +244,7 @@ check_parents () {\n \tdo\n \t\tif ! test -r \"$cachedir/notree/$miss\"\n \t\tthen\n-\t\t\tdebug \"  incorrect order: $miss\"\n+\t\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n \t\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n \t\tfi\n \tdone\n@@ -392,6 +392,26 @@ find_existing_splits () {\n \tdone\n }\n \n+find_mainline_ref () {\n+\tdebug \"Looking for first split...\"\n+\tdir=\"$1\"\n+\trevs=\"$2\"\n+\tmain=\n+\tsub=\n+\tlocal grep_format=\"^git-subtree-dir: $dir/*\\$\"\n+\tgit log --reverse --grep=\"$grep_format\" \\\n+\t\t--no-show-signature --pretty=format:'START %H%n%s%n%n%b%nEND%n' $revs |\n+\twhile read a b junk\n+\tdo\n+\t\tcase \"$a\" in\n+\t\tgit-subtree-mainline:)\n+\t\t\techo \"$b\"\n+\t\t\treturn\n+\t\t\t;;\n+\t\tesac\n+\tdone\n+}\n+\n copy_commit () {\n \t# We're going to set some environment vars here, so\n \t# do it in a subshell to get rid of them safely later\n@@ -646,9 +666,9 @@ process_split_commit () {\n \n \tprogress \"$revcount/$revmax ($createcount) [$extracount]\"\n \n-\tdebug \"Processing commit: $rev\"\n+\tdebug \"Processing commit: $rev ($indent)\"\n \texists=$(cache_get \"$rev\")\n-\tif test -n \"$exists\"\n+\tif test -z \"$(cache_miss \"$rev\")\"\n \tthen\n \t\tdebug \"  prior: $exists\"\n \t\treturn\n@@ -773,6 +793,17 @@ cmd_split () {\n \n \tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n \n+\tmainline=\"$(find_mainline_ref \"$dir\" \"$revs\")\"\n+\tif test -n \"$mainline\"\n+\tthen\n+\t\tdebug \"Mainline $mainline predates subtree add\"\n+\t\tgit rev-list --topo-order --skip=1 $mainline |\n+\t\twhile read rev\n+\t\tdo\n+\t\t\tcache_set \"$rev\" \"\"\n+\t\tdone || exit $?\n+\tfi\n+\n \t# We can't restrict rev-list to only $dir here, because some of our\n \t# parents have the $dir contents the root, and those won't match.\n \t# (and rev-list --follow doesn't seem to solve this)\n-- \ngitgitgadget\n\n"},{"id":"397524","messageId":"a67c256a598cfa8cb0bcc57fee89f16b0251a4a8.1589176202.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 4/7] subtree: add git subtree map command","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:49:58Z","receivedAt":"2020-05-11T05:50:09Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nAdds an entry to the subtree cache so that subsequent split runs can skip\nany commits that turn out to be problematic.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 21 +++++++++++++++++++--\n 1 file changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 90f92f4e949..f8ae1283cd3 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -15,6 +15,7 @@ git subtree merge --prefix=<prefix> <commit>\n git subtree pull  --prefix=<prefix> <repository> <ref>\n git subtree push  --prefix=<prefix> <repository> <ref>\n git subtree split --prefix=<prefix> <commit>\n+git subtree map   --prefix=<prefix> <mainline> <subtree>\n --\n h,help        show the help\n q             quiet\n@@ -161,7 +162,7 @@ command=\"$1\"\n shift\n \n case \"$command\" in\n-add|merge|pull)\n+add|merge|pull|map)\n \tdefault=\n \t;;\n split|push)\n@@ -192,7 +193,8 @@ dir=\"$(dirname \"$prefix/.\")\"\n \n if test \"$command\" != \"pull\" &&\n \t\ttest \"$command\" != \"add\" &&\n-\t\ttest \"$command\" != \"push\"\n+\t\ttest \"$command\" != \"push\" &&\n+\t\ttest \"$command\" != \"map\"\n then\n \trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n \tdirs=$(git rev-parse --no-revs --no-flags \"$@\") || exit $?\n@@ -795,6 +797,21 @@ cmd_add_commit () {\n \tsay \"Added dir '$dir'\"\n }\n \n+cmd_map () {\n+\toldrev=\"$1\"\n+\tnewrev=\"$2\"\n+\n+\tif test -z \"$oldrev\"\n+\tthen\n+\t\tdie \"You must provide a revision to map\"\n+\tfi\n+\n+\tcache_setup || exit $?\n+\tcache_set \"$oldrev\" \"$newrev\"\n+\n+\tsay \"Mapped $oldrev => $newrev\"\n+}\n+\n cmd_split () {\n \tdebug \"Splitting $dir...\"\n \tcache_setup || exit $?\n-- \ngitgitgadget\n\n"},{"id":"397525","messageId":"c892ee9828b019973152c6fa27b59a06c1d93e11.1589176201.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 3/7] subtree: persist cache between split runs","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:49:57Z","receivedAt":"2020-05-11T05:50:10Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nProvide a mechanism for handling problematic commits. If the algorithm\nin process_split_commit is getting something wrong, you can write a\ncorrected value to the cache before running split.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 37 ++++++++++++++++++++++++++--------\n 1 file changed, 29 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex da0eede6979..90f92f4e949 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -27,6 +27,7 @@ b,branch=     create a new branch from the split subtree\n ignore-joins  ignore prior --rejoin commits\n onto=         try connecting new tree to an existing one\n rejoin        merge the new branch back into HEAD\n+clear-cache   reset the subtree mapping cache\n  options for 'add', 'merge', and 'pull'\n squash        merge subtree changes as a single commit\n \"\n@@ -48,6 +49,7 @@ annotate=\n squash=\n message=\n prefix=\n+clearcache=\n \n debug () {\n \tif test -n \"$debug\"\n@@ -131,6 +133,9 @@ do\n \t--no-rejoin)\n \t\trejoin=\n \t\t;;\n+\t--clear-cache)\n+\t\tclearcache=1\n+\t\t;;\n \t--ignore-joins)\n \t\tignore_joins=1\n \t\t;;\n@@ -206,9 +211,13 @@ debug \"opts: {$*}\"\n debug\n \n cache_setup () {\n-\tcachedir=\"$GIT_DIR/subtree-cache/$$\"\n-\trm -rf \"$cachedir\" ||\n-\t\tdie \"Can't delete old cachedir: $cachedir\"\n+\tcachedir=\"$GIT_DIR/subtree-cache/$prefix\"\n+\tif test -n \"$clearcache\"\n+\tthen\n+\t\tdebug \"Clearing cache\"\n+\t\trm -rf \"$cachedir\" ||\n+\t\t\tdie \"Can't delete old cachedir: $cachedir\"\n+\tfi\n \tmkdir -p \"$cachedir\" ||\n \t\tdie \"Can't create new cachedir: $cachedir\"\n \tmkdir -p \"$cachedir/notree\" ||\n@@ -266,6 +275,16 @@ cache_set () {\n \techo \"$newrev\" >\"$cachedir/$oldrev\"\n }\n \n+cache_set_if_unset () {\n+\toldrev=\"$1\"\n+\tnewrev=\"$2\"\n+\tif test -e \"$cachedir/$oldrev\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$newrev\" >\"$cachedir/$oldrev\"\n+}\n+\n rev_exists () {\n \tif git rev-parse \"$1\" >/dev/null 2>&1\n \tthen\n@@ -375,13 +394,13 @@ find_existing_splits () {\n \t\t\tthen\n \t\t\t\t# squash commits refer to a subtree\n \t\t\t\tdebug \"  Squash: $sq from $sub\"\n-\t\t\t\tcache_set \"$sq\" \"$sub\"\n+\t\t\t\tcache_set_if_unset \"$sq\" \"$sub\"\n \t\t\tfi\n \t\t\tif test -n \"$main\" -a -n \"$sub\"\n \t\t\tthen\n \t\t\t\tdebug \"  Prior: $main -> $sub\"\n-\t\t\t\tcache_set $main $sub\n-\t\t\t\tcache_set $sub $sub\n+\t\t\t\tcache_set_if_unset $main $sub\n+\t\t\t\tcache_set_if_unset $sub $sub\n \t\t\t\ttry_remove_previous \"$main\"\n \t\t\t\ttry_remove_previous \"$sub\"\n \t\t\tfi\n@@ -690,6 +709,8 @@ process_split_commit () {\n \t\tif test -n \"$newparents\"\n \t\tthen\n \t\t\tcache_set \"$rev\" \"$rev\"\n+\t\telse\n+\t\t\tcache_set \"$rev\" \"\"\n \t\tfi\n \t\treturn\n \tfi\n@@ -787,7 +808,7 @@ cmd_split () {\n \t\t\t# the 'onto' history is already just the subdir, so\n \t\t\t# any parent we find there can be used verbatim\n \t\t\tdebug \"  cache: $rev\"\n-\t\t\tcache_set \"$rev\" \"$rev\"\n+\t\t\tcache_set_if_unset \"$rev\" \"$rev\"\n \t\tdone\n \tfi\n \n@@ -800,7 +821,7 @@ cmd_split () {\n \t\tgit rev-list --topo-order --skip=1 $mainline |\n \t\twhile read rev\n \t\tdo\n-\t\t\tcache_set \"$rev\" \"\"\n+\t\t\tcache_set_if_unset \"$rev\" \"\"\n \t\tdone || exit $?\n \tfi\n \n-- \ngitgitgadget\n\n"},{"id":"397526","messageId":"a76a49651bfb43f29892b0fec6887832f70b3e89.1589176202.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 5/7] subtree: add git subtree use and ignore commands","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:49:59Z","receivedAt":"2020-05-11T05:50:11Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nTell split to use or ignore larger sections of the history. In most cases\nsplit does this automatically based on metadata from subtree add.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 78 ++++++++++++++++++++++++++++------\n 1 file changed, 66 insertions(+), 12 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex f8ae1283cd3..1ca8b2f1101 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -9,13 +9,15 @@ then\n \tset -- -h\n fi\n OPTS_SPEC=\"\\\n-git subtree add   --prefix=<prefix> <commit>\n-git subtree add   --prefix=<prefix> <repository> <ref>\n-git subtree merge --prefix=<prefix> <commit>\n-git subtree pull  --prefix=<prefix> <repository> <ref>\n-git subtree push  --prefix=<prefix> <repository> <ref>\n-git subtree split --prefix=<prefix> <commit>\n-git subtree map   --prefix=<prefix> <mainline> <subtree>\n+git subtree add    --prefix=<prefix> <commit>\n+git subtree add    --prefix=<prefix> <repository> <ref>\n+git subtree merge  --prefix=<prefix> <commit>\n+git subtree pull   --prefix=<prefix> <repository> <ref>\n+git subtree push   --prefix=<prefix> <repository> <ref>\n+git subtree split  --prefix=<prefix> <commit>\n+git subtree map    --prefix=<prefix> <mainline> <subtree>\n+git subtree ignore --prefix=<prefix> <commit>\n+git subtree use    --prefix=<prefix> <commit>\n --\n h,help        show the help\n q             quiet\n@@ -162,7 +164,7 @@ command=\"$1\"\n shift\n \n case \"$command\" in\n-add|merge|pull|map)\n+add|merge|pull|map|ignore|use)\n \tdefault=\n \t;;\n split|push)\n@@ -433,6 +435,18 @@ find_mainline_ref () {\n \tdone\n }\n \n+exclude_processed_refs () {\n+\t\tif test -r \"$cachedir/processed\"\n+\t\tthen\n+\t\t\tcat \"$cachedir/processed\" |\n+\t\t\twhile read rev\n+\t\t\tdo\n+\t\t\t\tdebug \"read $rev\"\n+\t\t\t\techo \"^$rev\"\n+\t\t\tdone\n+\t\tfi\n+}\n+\n copy_commit () {\n \t# We're going to set some environment vars here, so\n \t# do it in a subshell to get rid of them safely later\n@@ -798,20 +812,60 @@ cmd_add_commit () {\n }\n \n cmd_map () {\n-\toldrev=\"$1\"\n-\tnewrev=\"$2\"\n \n-\tif test -z \"$oldrev\"\n+\tif test -z \"$1\"\n \tthen\n \t\tdie \"You must provide a revision to map\"\n \tfi\n \n+\toldrev=$(git rev-parse --revs-only \"$1\") || exit $?\n+\tnewrev=\n+\n+\tif test -n \"$2\"\n+\tthen\n+\t\tnewrev=$(git rev-parse --revs-only \"$2\") || exit $?\n+\tfi\n+\n \tcache_setup || exit $?\n \tcache_set \"$oldrev\" \"$newrev\"\n \n \tsay \"Mapped $oldrev => $newrev\"\n }\n \n+cmd_ignore () {\n+\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n+\tensure_single_rev $revs\n+\n+\tsay \"Ignoring $revs\"\n+\n+\tcache_setup || exit $?\n+\n+\tgit rev-list $revs |\n+\twhile read rev\n+\tdo\n+\t\tcache_set \"$rev\" \"\"\n+\tdone\n+\n+\techo \"$revs\" >>\"$cachedir/processed\"\n+}\n+\n+cmd_use () {\n+\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n+\tensure_single_rev $revs\n+\n+\tsay \"Using existing subtree $revs\"\n+\n+\tcache_setup || exit $?\n+\n+\tgit rev-list $revs |\n+\twhile read rev\n+\tdo\n+\t\tcache_set \"$rev\" \"$rev\"\n+\tdone\n+\n+\techo \"$revs\" >>\"$cachedir/processed\"\n+}\n+\n cmd_split () {\n \tdebug \"Splitting $dir...\"\n \tcache_setup || exit $?\n@@ -829,7 +883,7 @@ cmd_split () {\n \t\tdone\n \tfi\n \n-\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n+\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\") $(exclude_processed_refs)\"\n \n \tmainline=\"$(find_mainline_ref \"$dir\" \"$revs\")\"\n \tif test -n \"$mainline\"\n-- \ngitgitgadget\n\n"},{"id":"397527","messageId":"19db9cfb68c762a262f3f3b6a0781cdc6bf49bfe.1589176202.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 7/7] subtree: document new subtree commands","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:50:01Z","receivedAt":"2020-05-11T05:50:12Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.txt | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/contrib/subtree/git-subtree.txt b/contrib/subtree/git-subtree.txt\nindex 352deda69dc..a5a76e8ce69 100644\n--- a/contrib/subtree/git-subtree.txt\n+++ b/contrib/subtree/git-subtree.txt\n@@ -52,6 +52,12 @@ useful elsewhere, you can extract its entire history and publish\n that as its own git repository, without accidentally\n intermingling the history of your application project.\n \n+Although the relationship between subtree and mainline commits is stored\n+in regular git history, it is also cached between subtree runs. In most\n+cases this is merely a performance improvement, but for projects with\n+large and complex histories the cache can be manipulated directly\n+with the use, ignore and map commands.\n+\n [TIP]\n In order to keep your commit messages clean, we recommend that\n people split their commits between the subtrees and the main\n@@ -120,6 +126,21 @@ and friends will work as expected.\n Note that if you use '--squash' when you merge, you should usually not\n just '--rejoin' when you split.\n \n+ignore::\n+\tMark a commit and all of its history as irrelevant to subtree split.\n+\tIn most cases this would be handled automatically based on metadata\n+\tfrom subtree join commits. Intended for improving performance on\n+\textremely large repos and excluding complex history that turns out\n+\tto be otherwise problematic.\n+\n+use::\n+\tMark a commit and all of its history as part of an existing subtree.\n+\tIn normal circumstances this would be handled based on the metadata\n+\tfrom the subtree join commit. Similar to the --onto option of split.\n+\n+map::\n+\tManually override the normal output of split for a particular commit.\n+\tExtreme flexibility for advanced troubleshooting purposes only.\n \n OPTIONS\n -------\n@@ -142,6 +163,9 @@ OPTIONS\n \tThis option is only valid for add, merge and pull (unsure).\n \tSpecify <message> as the commit message for the merge commit.\n \n+--clear-cache::\n+\tReset the subtree cache and recalculate all subtree mappings from the\n+\tcommit history\n \n OPTIONS FOR add, merge, push, pull\n ----------------------------------\n-- \ngitgitgadget\n"},{"id":"397528","messageId":"27a43ea2c4d608192b22474800db645a52f2991b.1589176202.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH 6/7] subtree: more robustly distinguish subtree and mainline commits","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-11T05:50:00Z","receivedAt":"2020-05-11T05:50:14Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nPrevent a mainline commit without $dir being treated as a subtree\ncommit and pulling in the entire mainline history. Any valid subtree\ncommit will have only valid subtree commits as parents, which will be\nunchanged by check_parents.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 24 +++++++++++-------------\n 1 file changed, 11 insertions(+), 13 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 1ca8b2f1101..320e51b7e1a 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -224,8 +224,6 @@ cache_setup () {\n \tfi\n \tmkdir -p \"$cachedir\" ||\n \t\tdie \"Can't create new cachedir: $cachedir\"\n-\tmkdir -p \"$cachedir/notree\" ||\n-\t\tdie \"Can't create new cachedir: $cachedir/notree\"\n \tdebug \"Using cachedir: $cachedir\" >&2\n }\n \n@@ -255,18 +253,11 @@ check_parents () {\n \tlocal indent=$(($2 + 1))\n \tfor miss in $missed\n \tdo\n-\t\tif ! test -r \"$cachedir/notree/$miss\"\n-\t\tthen\n-\t\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n-\t\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n-\t\tfi\n+\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n+\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n \tdone\n }\n \n-set_notree () {\n-\techo \"1\" > \"$cachedir/notree/$1\"\n-}\n-\n cache_set () {\n \toldrev=\"$1\"\n \tnewrev=\"$2\"\n@@ -721,11 +712,18 @@ process_split_commit () {\n \t# vs. a mainline commit?  Does it matter?\n \tif test -z \"$tree\"\n \tthen\n-\t\tset_notree \"$rev\"\n \t\tif test -n \"$newparents\"\n \t\tthen\n-\t\t\tcache_set \"$rev\" \"$rev\"\n+\t\t\tif test \"$newparents\" = \"$parents\"\n+\t\t\tthen\n+\t\t\t\t# if all parents were subtrees, this can be a subtree commit\n+\t\t\t\tcache_set \"$rev\" \"$rev\"\n+\t\t\telse\n+\t\t\t\t# a mainline commit with tree missing is equivalent to the initial commit\n+\t\t\t\tcache_set \"$rev\" \"\"\n+\t\t\tfi\n \t\telse\n+\t\t\t# no parents with valid subtree mappings means a commit prior to subtree add\n \t\t\tcache_set \"$rev\" \"\"\n \t\tfi\n \t\treturn\n-- \ngitgitgadget\n\n"},{"id":"406886","messageId":"CAPyFy2BsFLZZeEY=54evEJwed9q+Dc3TX3s1O+cr-nqwjCRHvg@mail.gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] subtree: Fix handling of complex history","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2020-10-04T17:52:16Z","receivedAt":"2020-10-04T17:52:31Z","isPatch":true,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Mon, 11 May 2020 at 01:50, Tom Clarkson via GitGitGadget\n<gitgitgadget@gmail.com> wrote:\n>\n> Fixes several issues that could occur when running subtree split on large\n> repos with more complex history.\n\nI've been using this patch set locally and would very much like to see it land.\n"},{"id":"406962","messageId":"nycvar.QRO.7.76.6.2010042124320.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"CAPyFy2BsFLZZeEY=54evEJwed9q+Dc3TX3s1O+cr-nqwjCRHvg@mail.gmail.com","subject":"Re: [PATCH 0/7] subtree: Fix handling of complex history","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-04T19:27:58Z","receivedAt":"2020-10-05T13:18:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 4 Oct 2020, Ed Maste wrote:\n\n> On Mon, 11 May 2020 at 01:50, Tom Clarkson via GitGitGadget\n> <gitgitgadget@gmail.com> wrote:\n> >\n> > Fixes several issues that could occur when running subtree split on large\n> > repos with more complex history.\n>\n> I've been using this patch set locally and would very much like to see it land.\n\nFWIW there have been more comments in favor of this patch in\nhttps://github.com/gitgitgadget/git/pull/493.\n\nI guess what this patch series needs to proceed further is an ACK by\nAvery?\n\nCiao,\nDscho\n"},{"id":"406964","messageId":"xmqqft6s7h77.fsf@gitster.c.googlers.com","threadId":"53444","inReplyTo":"nycvar.QRO.7.76.6.2010042124320.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/7] subtree: Fix handling of complex history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-05T16:47:40Z","receivedAt":"2020-10-05T16:47:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> On Sun, 4 Oct 2020, Ed Maste wrote:\n>\n>> On Mon, 11 May 2020 at 01:50, Tom Clarkson via GitGitGadget\n>> <gitgitgadget@gmail.com> wrote:\n>> >\n>> > Fixes several issues that could occur when running subtree split on large\n>> > repos with more complex history.\n>>\n>> I've been using this patch set locally and would very much like to see it land.\n>\n> FWIW there have been more comments in favor of this patch in\n> https://github.com/gitgitgadget/git/pull/493.\n>\n> I guess what this patch series needs to proceed further is an ACK by\n> Avery?\n\nNot necessarily by Avery, but by somebody who knows the code better\nthan us ;-).\n\n"},{"id":"406969","messageId":"CAPyFy2BeHnacdqd_C8BdPntsN7FCoTciL+s8fDXGhBU0H01FaA@mail.gmail.com","threadId":"53444","inReplyTo":"nycvar.QRO.7.76.6.2010042124320.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/7] subtree: Fix handling of complex history","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2020-10-05T21:37:54Z","receivedAt":"2020-10-05T21:38:08Z","isPatch":true,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Mon, 5 Oct 2020 at 09:18, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n>\n> FWIW there have been more comments in favor of this patch in\n> https://github.com/gitgitgadget/git/pull/493.\n>\n> I guess what this patch series needs to proceed further is an ACK by\n> Avery?\n\nAvery says \"if it's good enough for Johannes it's good enough for me\"\nhttps://twitter.com/apenwarr/status/1313231132721401861\n"},{"id":"407010","messageId":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.git.1589176201.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] subtree: Fix handling of complex history","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:06Z","receivedAt":"2020-10-06T22:05:18Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"Fixes several issues that could occur when running subtree split on large\nrepos with more complex history.\n\n 1. A merge commit could bypass the known start point of the subtree, which\n    would cause the entire history to be processed recursively, leading to a\n    stack overflow / segfault after reading a few hundred commits. Older\n    commits are now explicitly recorded as irrelevant so that the recursive\n    process can terminate on any mainline commit rather than only on subtree\n    joins and initial commits.\n    \n    \n 2. It is possible for a repo to contain subtrees that lack the metadata\n    that is usually present in add/join commit messages (git-svn at least\n    can produce such a structure). The new use/ignore/map commands allow the\n    user to provide that information for any problematic commits.\n    \n    \n 3. A mainline commit that does not contain the subtree folder could be\n    erroneously identified as a subtree commit, which would add the entire\n    mainline history to the subtree. Commits will now only be used as is if\n    all their parents are already identified as subtree commits. While the\n    new code can still be tripped up by unusual folder structures, the\n    completely unambiguous solution turned out to involve a significant\n    performance penalty, and the new ignore / use commands provide a\n    workaround for that scenario.\n\nTom Clarkson (7):\n  subtree: handle multiple parents passed to cache_miss\n  subtree: exclude commits predating add from recursive processing\n  subtree: persist cache between split runs\n  subtree: add git subtree map command\n  subtree: add git subtree use and ignore commands\n  subtree: more robustly distinguish subtree and mainline commits\n  subtree: document new subtree commands\n\n contrib/subtree/git-subtree.sh  | 183 ++++++++++++++++++++++++++------\n contrib/subtree/git-subtree.txt |  24 +++++\n 2 files changed, 175 insertions(+), 32 deletions(-)\n\n\nbase-commit: 47ae905ffb98cc4d4fd90083da6bc8dab55d9ecc\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-493%2Ftqc%2Ftqc%2Fsubtree-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-493/tqc/tqc/subtree-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/493\n\nRange-diff vs v1:\n\n 1:  74fa670490 = 1:  9cff2a0cf6 subtree: handle multiple parents passed to cache_miss\n 2:  87af5a316a ! 2:  79b5f4a651 subtree: exclude commits predating add from recursive processing\n     @@ contrib/subtree/git-subtree.sh: find_existing_splits () {\n      +\tdebug \"Looking for first split...\"\n      +\tdir=\"$1\"\n      +\trevs=\"$2\"\n     -+\tmain=\n     -+\tsub=\n     -+\tlocal grep_format=\"^git-subtree-dir: $dir/*\\$\"\n     -+\tgit log --reverse --grep=\"$grep_format\" \\\n     ++\n     ++\tgit log --reverse --grep=\"^git-subtree-dir: $dir/*\\$\" \\\n      +\t\t--no-show-signature --pretty=format:'START %H%n%s%n%n%b%nEND%n' $revs |\n      +\twhile read a b junk\n      +\tdo\n 3:  c892ee9828 = 3:  8eec18388c subtree: persist cache between split runs\n 4:  a67c256a59 = 4:  1490ce1114 subtree: add git subtree map command\n 5:  a76a49651b = 5:  2d103292ce subtree: add git subtree use and ignore commands\n 6:  27a43ea2c4 = 6:  a7aaedfed3 subtree: more robustly distinguish subtree and mainline commits\n 7:  19db9cfb68 = 7:  fe2e4819b8 subtree: document new subtree commands\n\n-- \ngitgitgadget\n"},{"id":"407011","messageId":"1490ce111463b56eed35016c2e6232eea1faa3db.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] subtree: add git subtree map command","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:10Z","receivedAt":"2020-10-06T22:05:21Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nAdds an entry to the subtree cache so that subsequent split runs can skip\nany commits that turn out to be problematic.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 21 +++++++++++++++++++--\n 1 file changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex c21d620610..1559100c0e 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -15,6 +15,7 @@ git subtree merge --prefix=<prefix> <commit>\n git subtree pull  --prefix=<prefix> <repository> <ref>\n git subtree push  --prefix=<prefix> <repository> <ref>\n git subtree split --prefix=<prefix> <commit>\n+git subtree map   --prefix=<prefix> <mainline> <subtree>\n --\n h,help        show the help\n q             quiet\n@@ -161,7 +162,7 @@ command=\"$1\"\n shift\n \n case \"$command\" in\n-add|merge|pull)\n+add|merge|pull|map)\n \tdefault=\n \t;;\n split|push)\n@@ -192,7 +193,8 @@ dir=\"$(dirname \"$prefix/.\")\"\n \n if test \"$command\" != \"pull\" &&\n \t\ttest \"$command\" != \"add\" &&\n-\t\ttest \"$command\" != \"push\"\n+\t\ttest \"$command\" != \"push\" &&\n+\t\ttest \"$command\" != \"map\"\n then\n \trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n \tdirs=$(git rev-parse --no-revs --no-flags \"$@\") || exit $?\n@@ -793,6 +795,21 @@ cmd_add_commit () {\n \tsay \"Added dir '$dir'\"\n }\n \n+cmd_map () {\n+\toldrev=\"$1\"\n+\tnewrev=\"$2\"\n+\n+\tif test -z \"$oldrev\"\n+\tthen\n+\t\tdie \"You must provide a revision to map\"\n+\tfi\n+\n+\tcache_setup || exit $?\n+\tcache_set \"$oldrev\" \"$newrev\"\n+\n+\tsay \"Mapped $oldrev => $newrev\"\n+}\n+\n cmd_split () {\n \tdebug \"Splitting $dir...\"\n \tcache_setup || exit $?\n-- \ngitgitgadget\n\n"},{"id":"407012","messageId":"9cff2a0cf66f037bf10866b7787496c6cf02ad94.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] subtree: handle multiple parents passed to cache_miss","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:07Z","receivedAt":"2020-10-06T22:05:22Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 868e18b9a1..9867718503 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -238,7 +238,7 @@ cache_miss () {\n }\n \n check_parents () {\n-\tmissed=$(cache_miss \"$1\")\n+\tmissed=$(cache_miss $1)\n \tlocal indent=$(($2 + 1))\n \tfor miss in $missed\n \tdo\n-- \ngitgitgadget\n\n"},{"id":"407013","messageId":"2d103292cecfef3b29d221b3d8f05adb0add4475.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] subtree: add git subtree use and ignore commands","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:11Z","receivedAt":"2020-10-06T22:05:24Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nTell split to use or ignore larger sections of the history. In most cases\nsplit does this automatically based on metadata from subtree add.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 78 ++++++++++++++++++++++++++++------\n 1 file changed, 66 insertions(+), 12 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 1559100c0e..e56621a986 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -9,13 +9,15 @@ then\n \tset -- -h\n fi\n OPTS_SPEC=\"\\\n-git subtree add   --prefix=<prefix> <commit>\n-git subtree add   --prefix=<prefix> <repository> <ref>\n-git subtree merge --prefix=<prefix> <commit>\n-git subtree pull  --prefix=<prefix> <repository> <ref>\n-git subtree push  --prefix=<prefix> <repository> <ref>\n-git subtree split --prefix=<prefix> <commit>\n-git subtree map   --prefix=<prefix> <mainline> <subtree>\n+git subtree add    --prefix=<prefix> <commit>\n+git subtree add    --prefix=<prefix> <repository> <ref>\n+git subtree merge  --prefix=<prefix> <commit>\n+git subtree pull   --prefix=<prefix> <repository> <ref>\n+git subtree push   --prefix=<prefix> <repository> <ref>\n+git subtree split  --prefix=<prefix> <commit>\n+git subtree map    --prefix=<prefix> <mainline> <subtree>\n+git subtree ignore --prefix=<prefix> <commit>\n+git subtree use    --prefix=<prefix> <commit>\n --\n h,help        show the help\n q             quiet\n@@ -162,7 +164,7 @@ command=\"$1\"\n shift\n \n case \"$command\" in\n-add|merge|pull|map)\n+add|merge|pull|map|ignore|use)\n \tdefault=\n \t;;\n split|push)\n@@ -431,6 +433,18 @@ find_mainline_ref () {\n \tdone\n }\n \n+exclude_processed_refs () {\n+\t\tif test -r \"$cachedir/processed\"\n+\t\tthen\n+\t\t\tcat \"$cachedir/processed\" |\n+\t\t\twhile read rev\n+\t\t\tdo\n+\t\t\t\tdebug \"read $rev\"\n+\t\t\t\techo \"^$rev\"\n+\t\t\tdone\n+\t\tfi\n+}\n+\n copy_commit () {\n \t# We're going to set some environment vars here, so\n \t# do it in a subshell to get rid of them safely later\n@@ -796,20 +810,60 @@ cmd_add_commit () {\n }\n \n cmd_map () {\n-\toldrev=\"$1\"\n-\tnewrev=\"$2\"\n \n-\tif test -z \"$oldrev\"\n+\tif test -z \"$1\"\n \tthen\n \t\tdie \"You must provide a revision to map\"\n \tfi\n \n+\toldrev=$(git rev-parse --revs-only \"$1\") || exit $?\n+\tnewrev=\n+\n+\tif test -n \"$2\"\n+\tthen\n+\t\tnewrev=$(git rev-parse --revs-only \"$2\") || exit $?\n+\tfi\n+\n \tcache_setup || exit $?\n \tcache_set \"$oldrev\" \"$newrev\"\n \n \tsay \"Mapped $oldrev => $newrev\"\n }\n \n+cmd_ignore () {\n+\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n+\tensure_single_rev $revs\n+\n+\tsay \"Ignoring $revs\"\n+\n+\tcache_setup || exit $?\n+\n+\tgit rev-list $revs |\n+\twhile read rev\n+\tdo\n+\t\tcache_set \"$rev\" \"\"\n+\tdone\n+\n+\techo \"$revs\" >>\"$cachedir/processed\"\n+}\n+\n+cmd_use () {\n+\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n+\tensure_single_rev $revs\n+\n+\tsay \"Using existing subtree $revs\"\n+\n+\tcache_setup || exit $?\n+\n+\tgit rev-list $revs |\n+\twhile read rev\n+\tdo\n+\t\tcache_set \"$rev\" \"$rev\"\n+\tdone\n+\n+\techo \"$revs\" >>\"$cachedir/processed\"\n+}\n+\n cmd_split () {\n \tdebug \"Splitting $dir...\"\n \tcache_setup || exit $?\n@@ -827,7 +881,7 @@ cmd_split () {\n \t\tdone\n \tfi\n \n-\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n+\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\") $(exclude_processed_refs)\"\n \n \tmainline=\"$(find_mainline_ref \"$dir\" \"$revs\")\"\n \tif test -n \"$mainline\"\n-- \ngitgitgadget\n\n"},{"id":"407014","messageId":"a7aaedfed3785c6ca693f60f05e76156f68a5d39.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] subtree: more robustly distinguish subtree and mainline commits","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:12Z","receivedAt":"2020-10-06T22:05:24Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nPrevent a mainline commit without $dir being treated as a subtree\ncommit and pulling in the entire mainline history. Any valid subtree\ncommit will have only valid subtree commits as parents, which will be\nunchanged by check_parents.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 24 +++++++++++-------------\n 1 file changed, 11 insertions(+), 13 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex e56621a986..fa6293b372 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -224,8 +224,6 @@ cache_setup () {\n \tfi\n \tmkdir -p \"$cachedir\" ||\n \t\tdie \"Can't create new cachedir: $cachedir\"\n-\tmkdir -p \"$cachedir/notree\" ||\n-\t\tdie \"Can't create new cachedir: $cachedir/notree\"\n \tdebug \"Using cachedir: $cachedir\" >&2\n }\n \n@@ -255,18 +253,11 @@ check_parents () {\n \tlocal indent=$(($2 + 1))\n \tfor miss in $missed\n \tdo\n-\t\tif ! test -r \"$cachedir/notree/$miss\"\n-\t\tthen\n-\t\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n-\t\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n-\t\tfi\n+\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n+\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n \tdone\n }\n \n-set_notree () {\n-\techo \"1\" > \"$cachedir/notree/$1\"\n-}\n-\n cache_set () {\n \toldrev=\"$1\"\n \tnewrev=\"$2\"\n@@ -719,11 +710,18 @@ process_split_commit () {\n \t# vs. a mainline commit?  Does it matter?\n \tif test -z \"$tree\"\n \tthen\n-\t\tset_notree \"$rev\"\n \t\tif test -n \"$newparents\"\n \t\tthen\n-\t\t\tcache_set \"$rev\" \"$rev\"\n+\t\t\tif test \"$newparents\" = \"$parents\"\n+\t\t\tthen\n+\t\t\t\t# if all parents were subtrees, this can be a subtree commit\n+\t\t\t\tcache_set \"$rev\" \"$rev\"\n+\t\t\telse\n+\t\t\t\t# a mainline commit with tree missing is equivalent to the initial commit\n+\t\t\t\tcache_set \"$rev\" \"\"\n+\t\t\tfi\n \t\telse\n+\t\t\t# no parents with valid subtree mappings means a commit prior to subtree add\n \t\t\tcache_set \"$rev\" \"\"\n \t\tfi\n \t\treturn\n-- \ngitgitgadget\n\n"},{"id":"407015","messageId":"8eec18388c86071db47512b84118e3b9111bd34d.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] subtree: persist cache between split runs","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:09Z","receivedAt":"2020-10-06T22:05:25Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nProvide a mechanism for handling problematic commits. If the algorithm\nin process_split_commit is getting something wrong, you can write a\ncorrected value to the cache before running split.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 37 ++++++++++++++++++++++++++--------\n 1 file changed, 29 insertions(+), 8 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 160bad95c1..c21d620610 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -27,6 +27,7 @@ b,branch=     create a new branch from the split subtree\n ignore-joins  ignore prior --rejoin commits\n onto=         try connecting new tree to an existing one\n rejoin        merge the new branch back into HEAD\n+clear-cache   reset the subtree mapping cache\n  options for 'add', 'merge', and 'pull'\n squash        merge subtree changes as a single commit\n \"\n@@ -48,6 +49,7 @@ annotate=\n squash=\n message=\n prefix=\n+clearcache=\n \n debug () {\n \tif test -n \"$debug\"\n@@ -131,6 +133,9 @@ do\n \t--no-rejoin)\n \t\trejoin=\n \t\t;;\n+\t--clear-cache)\n+\t\tclearcache=1\n+\t\t;;\n \t--ignore-joins)\n \t\tignore_joins=1\n \t\t;;\n@@ -206,9 +211,13 @@ debug \"opts: {$*}\"\n debug\n \n cache_setup () {\n-\tcachedir=\"$GIT_DIR/subtree-cache/$$\"\n-\trm -rf \"$cachedir\" ||\n-\t\tdie \"Can't delete old cachedir: $cachedir\"\n+\tcachedir=\"$GIT_DIR/subtree-cache/$prefix\"\n+\tif test -n \"$clearcache\"\n+\tthen\n+\t\tdebug \"Clearing cache\"\n+\t\trm -rf \"$cachedir\" ||\n+\t\t\tdie \"Can't delete old cachedir: $cachedir\"\n+\tfi\n \tmkdir -p \"$cachedir\" ||\n \t\tdie \"Can't create new cachedir: $cachedir\"\n \tmkdir -p \"$cachedir/notree\" ||\n@@ -266,6 +275,16 @@ cache_set () {\n \techo \"$newrev\" >\"$cachedir/$oldrev\"\n }\n \n+cache_set_if_unset () {\n+\toldrev=\"$1\"\n+\tnewrev=\"$2\"\n+\tif test -e \"$cachedir/$oldrev\"\n+\tthen\n+\t\treturn\n+\tfi\n+\techo \"$newrev\" >\"$cachedir/$oldrev\"\n+}\n+\n rev_exists () {\n \tif git rev-parse \"$1\" >/dev/null 2>&1\n \tthen\n@@ -375,13 +394,13 @@ find_existing_splits () {\n \t\t\tthen\n \t\t\t\t# squash commits refer to a subtree\n \t\t\t\tdebug \"  Squash: $sq from $sub\"\n-\t\t\t\tcache_set \"$sq\" \"$sub\"\n+\t\t\t\tcache_set_if_unset \"$sq\" \"$sub\"\n \t\t\tfi\n \t\t\tif test -n \"$main\" -a -n \"$sub\"\n \t\t\tthen\n \t\t\t\tdebug \"  Prior: $main -> $sub\"\n-\t\t\t\tcache_set $main $sub\n-\t\t\t\tcache_set $sub $sub\n+\t\t\t\tcache_set_if_unset $main $sub\n+\t\t\t\tcache_set_if_unset $sub $sub\n \t\t\t\ttry_remove_previous \"$main\"\n \t\t\t\ttry_remove_previous \"$sub\"\n \t\t\tfi\n@@ -688,6 +707,8 @@ process_split_commit () {\n \t\tif test -n \"$newparents\"\n \t\tthen\n \t\t\tcache_set \"$rev\" \"$rev\"\n+\t\telse\n+\t\t\tcache_set \"$rev\" \"\"\n \t\tfi\n \t\treturn\n \tfi\n@@ -785,7 +806,7 @@ cmd_split () {\n \t\t\t# the 'onto' history is already just the subdir, so\n \t\t\t# any parent we find there can be used verbatim\n \t\t\tdebug \"  cache: $rev\"\n-\t\t\tcache_set \"$rev\" \"$rev\"\n+\t\t\tcache_set_if_unset \"$rev\" \"$rev\"\n \t\tdone\n \tfi\n \n@@ -798,7 +819,7 @@ cmd_split () {\n \t\tgit rev-list --topo-order --skip=1 $mainline |\n \t\twhile read rev\n \t\tdo\n-\t\t\tcache_set \"$rev\" \"\"\n+\t\t\tcache_set_if_unset \"$rev\" \"\"\n \t\tdone || exit $?\n \tfi\n \n-- \ngitgitgadget\n\n"},{"id":"407016","messageId":"79b5f4a65197cea26ddc080c19dd2c5c7d424fc1.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] subtree: exclude commits predating add from recursive processing","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:08Z","receivedAt":"2020-10-06T22:05:27Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nInclude recursion depth in debug logs so we can see when the recursion is\ngetting out of hand.\n\nMaking the cache handle null mappings correctly and adding older commits\nto the cache allows the recursive algorithm to terminate at any point on\nmainline rather than needing to reach either the add point or the initial\ncommit.\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.sh | 35 +++++++++++++++++++++++++++++++---\n 1 file changed, 32 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\nindex 9867718503..160bad95c1 100755\n--- a/contrib/subtree/git-subtree.sh\n+++ b/contrib/subtree/git-subtree.sh\n@@ -244,7 +244,7 @@ check_parents () {\n \tdo\n \t\tif ! test -r \"$cachedir/notree/$miss\"\n \t\tthen\n-\t\t\tdebug \"  incorrect order: $miss\"\n+\t\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n \t\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n \t\tfi\n \tdone\n@@ -392,6 +392,24 @@ find_existing_splits () {\n \tdone\n }\n \n+find_mainline_ref () {\n+\tdebug \"Looking for first split...\"\n+\tdir=\"$1\"\n+\trevs=\"$2\"\n+\n+\tgit log --reverse --grep=\"^git-subtree-dir: $dir/*\\$\" \\\n+\t\t--no-show-signature --pretty=format:'START %H%n%s%n%n%b%nEND%n' $revs |\n+\twhile read a b junk\n+\tdo\n+\t\tcase \"$a\" in\n+\t\tgit-subtree-mainline:)\n+\t\t\techo \"$b\"\n+\t\t\treturn\n+\t\t\t;;\n+\t\tesac\n+\tdone\n+}\n+\n copy_commit () {\n \t# We're going to set some environment vars here, so\n \t# do it in a subshell to get rid of them safely later\n@@ -646,9 +664,9 @@ process_split_commit () {\n \n \tprogress \"$revcount/$revmax ($createcount) [$extracount]\"\n \n-\tdebug \"Processing commit: $rev\"\n+\tdebug \"Processing commit: $rev ($indent)\"\n \texists=$(cache_get \"$rev\")\n-\tif test -n \"$exists\"\n+\tif test -z \"$(cache_miss \"$rev\")\"\n \tthen\n \t\tdebug \"  prior: $exists\"\n \t\treturn\n@@ -773,6 +791,17 @@ cmd_split () {\n \n \tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n \n+\tmainline=\"$(find_mainline_ref \"$dir\" \"$revs\")\"\n+\tif test -n \"$mainline\"\n+\tthen\n+\t\tdebug \"Mainline $mainline predates subtree add\"\n+\t\tgit rev-list --topo-order --skip=1 $mainline |\n+\t\twhile read rev\n+\t\tdo\n+\t\t\tcache_set \"$rev\" \"\"\n+\t\tdone || exit $?\n+\tfi\n+\n \t# We can't restrict rev-list to only $dir here, because some of our\n \t# parents have the $dir contents the root, and those won't match.\n \t# (and rev-list --follow doesn't seem to solve this)\n-- \ngitgitgadget\n\n"},{"id":"407017","messageId":"fe2e4819b869725f870cd3ce99f1f8150fe17dc1.1602021913.git.gitgitgadget@gmail.com","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] subtree: document new subtree commands","fromName":"Tom Clarkson via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-06T22:05:13Z","receivedAt":"2020-10-06T22:05:33Z","isPatch":true,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"From: Tom Clarkson <tom@tqclarkson.com>\n\nSigned-off-by: Tom Clarkson <tom@tqclarkson.com>\n---\n contrib/subtree/git-subtree.txt | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/contrib/subtree/git-subtree.txt b/contrib/subtree/git-subtree.txt\nindex 352deda69d..a5a76e8ce6 100644\n--- a/contrib/subtree/git-subtree.txt\n+++ b/contrib/subtree/git-subtree.txt\n@@ -52,6 +52,12 @@ useful elsewhere, you can extract its entire history and publish\n that as its own git repository, without accidentally\n intermingling the history of your application project.\n \n+Although the relationship between subtree and mainline commits is stored\n+in regular git history, it is also cached between subtree runs. In most\n+cases this is merely a performance improvement, but for projects with\n+large and complex histories the cache can be manipulated directly\n+with the use, ignore and map commands.\n+\n [TIP]\n In order to keep your commit messages clean, we recommend that\n people split their commits between the subtrees and the main\n@@ -120,6 +126,21 @@ and friends will work as expected.\n Note that if you use '--squash' when you merge, you should usually not\n just '--rejoin' when you split.\n \n+ignore::\n+\tMark a commit and all of its history as irrelevant to subtree split.\n+\tIn most cases this would be handled automatically based on metadata\n+\tfrom subtree join commits. Intended for improving performance on\n+\textremely large repos and excluding complex history that turns out\n+\tto be otherwise problematic.\n+\n+use::\n+\tMark a commit and all of its history as part of an existing subtree.\n+\tIn normal circumstances this would be handled based on the metadata\n+\tfrom the subtree join commit. Similar to the --onto option of split.\n+\n+map::\n+\tManually override the normal output of split for a particular commit.\n+\tExtreme flexibility for advanced troubleshooting purposes only.\n \n OPTIONS\n -------\n@@ -142,6 +163,9 @@ OPTIONS\n \tThis option is only valid for add, merge and pull (unsure).\n \tSpecify <message> as the commit message for the merge commit.\n \n+--clear-cache::\n+\tReset the subtree cache and recalculate all subtree mappings from the\n+\tcommit history\n \n OPTIONS FOR add, merge, push, pull\n ----------------------------------\n-- \ngitgitgadget\n"},{"id":"407051","messageId":"CAPyFy2DmZdWGe8VUAEtWrJEBMWZJfQ4VC0BzjFkQT3cmdrDf+Q@mail.gmail.com","threadId":"53444","inReplyTo":"9cff2a0cf66f037bf10866b7787496c6cf02ad94.1602021913.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/7] subtree: handle multiple parents passed to cache_miss","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2020-10-07T13:12:15Z","receivedAt":"2020-10-07T13:12:31Z","isPatch":true,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Tue, 6 Oct 2020 at 18:05, Tom Clarkson via GitGitGadget\n<gitgitgadget@gmail.com> wrote:\n>\n> From: Tom Clarkson <tom@tqclarkson.com>\n>\n> Signed-off-by: Tom Clarkson <tom@tqclarkson.com>\nReviewed-by: Ed Maste <emaste@FreeBSD.org>\n"},{"id":"407064","messageId":"nycvar.QRO.7.76.6.2010071211180.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"79b5f4a65197cea26ddc080c19dd2c5c7d424fc1.1602021913.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] subtree: exclude commits predating add from recursive processing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T15:36:28Z","receivedAt":"2020-10-07T15:36:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tom,\n\nOn Tue, 6 Oct 2020, Tom Clarkson via GitGitGadget wrote:\n\n> From: Tom Clarkson <tom@tqclarkson.com>\n>\n> Include recursion depth in debug logs so we can see when the recursion is\n> getting out of hand.\n>\n> Making the cache handle null mappings correctly and adding older commits\n> to the cache allows the recursive algorithm to terminate at any point on\n> mainline rather than needing to reach either the add point or the initial\n> commit.\n\nMakes sense.\n\n> diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\n> index 9867718503..160bad95c1 100755\n> --- a/contrib/subtree/git-subtree.sh\n> +++ b/contrib/subtree/git-subtree.sh\n> @@ -244,7 +244,7 @@ check_parents () {\n>  \tdo\n>  \t\tif ! test -r \"$cachedir/notree/$miss\"\n>  \t\tthen\n> -\t\t\tdebug \"  incorrect order: $miss\"\n> +\t\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n\nWithout any context, it is hard to understand what the `$indent` variable\nis supposed to mean, so it is unclear why we need to print it here.\n\nI _guess_ it is the degree removed from the first-parent lineage?\n\nIn any case, it does not hurt here, so I trust that it is good to include\nit in the debug output.\n\n>  \t\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n>  \t\tfi\n>  \tdone\n> @@ -392,6 +392,24 @@ find_existing_splits () {\n>  \tdone\n>  }\n>\n> +find_mainline_ref () {\n> +\tdebug \"Looking for first split...\"\n> +\tdir=\"$1\"\n> +\trevs=\"$2\"\n\nThe `git-subtree` script seems to rely on the `local` construct, using it\nin plenty of other circumstances. How about using it here, too?\n\n> +\n> +\tgit log --reverse --grep=\"^git-subtree-dir: $dir/*\\$\" \\\n> +\t\t--no-show-signature --pretty=format:'START %H%n%s%n%n%b%nEND%n' $revs |\n\nSince all you are interested in is the `git-subtree-mainline:` trailer,\nwouldn't a format like `%(trailers:key=git-subtree-mainline)` instead of\n`START %H%n%s%n%n%b%nEND%n`?\n\nSee\nhttps://git-scm.com/docs/git-log#Documentation/git-log.txt-emtrailersoptionsem\nfor more information about pretty formats.\n\nBTW I am super unfamiliar with `git subtree`'s inner workings, and\ntherefore it would help me incredibly if the commit message talked a bit\nabout the commit message layout (with a particular eye on\n`git-subtree-dir` and `git-subtree-mainline` which I guess are trailers\nadded by `git subtree`?)...\n\n> +\twhile read a b junk\n> +\tdo\n> +\t\tcase \"$a\" in\n> +\t\tgit-subtree-mainline:)\n> +\t\t\techo \"$b\"\n> +\t\t\treturn\n> +\t\t\t;;\n> +\t\tesac\n> +\tdone\n> +}\n> +\n>  copy_commit () {\n>  \t# We're going to set some environment vars here, so\n>  \t# do it in a subshell to get rid of them safely later\n> @@ -646,9 +664,9 @@ process_split_commit () {\n>\n>  \tprogress \"$revcount/$revmax ($createcount) [$extracount]\"\n>\n> -\tdebug \"Processing commit: $rev\"\n> +\tdebug \"Processing commit: $rev ($indent)\"\n>  \texists=$(cache_get \"$rev\")\n> -\tif test -n \"$exists\"\n> +\tif test -z \"$(cache_miss \"$rev\")\"\n>  \tthen\n>  \t\tdebug \"  prior: $exists\"\n\nI do not see the `exists` variable being used other than for the debug\nstatement. Maybe better something like this?\n\n\tdebug \"  prior found for $rev\"\n\n>  \t\treturn\n> @@ -773,6 +791,17 @@ cmd_split () {\n>\n>  \tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n>\n> +\tmainline=\"$(find_mainline_ref \"$dir\" \"$revs\")\"\n> +\tif test -n \"$mainline\"\n> +\tthen\n> +\t\tdebug \"Mainline $mainline predates subtree add\"\n> +\t\tgit rev-list --topo-order --skip=1 $mainline |\n> +\t\twhile read rev\n> +\t\tdo\n> +\t\t\tcache_set \"$rev\" \"\"\n\nAh, so they are not really \"null mappings\", but mapped to an empty string.\nMakes sense. Maybe adjust the commit message?\n\n> +\t\tdone || exit $?\n> +\tfi\n> +\n>  \t# We can't restrict rev-list to only $dir here, because some of our\n>  \t# parents have the $dir contents the root, and those won't match.\n>  \t# (and rev-list --follow doesn't seem to solve this)\n> --\n> gitgitgadget\n>\n>\n"},{"id":"407068","messageId":"nycvar.QRO.7.76.6.2010071750310.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"8eec18388c86071db47512b84118e3b9111bd34d.1602021913.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/7] subtree: persist cache between split runs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T16:06:26Z","receivedAt":"2020-10-07T16:06:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tom,\n\nOn Tue, 6 Oct 2020, Tom Clarkson via GitGitGadget wrote:\n\n> @@ -48,6 +49,7 @@ annotate=\n>  squash=\n>  message=\n>  prefix=\n> +clearcache=\n\nIt might be more consistent to call it `clear_cache` (i.e. with an\nunderscore), just like `ignore_joins`.\n\n>\n>  debug () {\n>  \tif test -n \"$debug\"\n> @@ -131,6 +133,9 @@ do\n>  \t--no-rejoin)\n>  \t\trejoin=\n>  \t\t;;\n> +\t--clear-cache)\n> +\t\tclearcache=1\n> +\t\t;;\n>  \t--ignore-joins)\n>  \t\tignore_joins=1\n>  \t\t;;\n> @@ -206,9 +211,13 @@ debug \"opts: {$*}\"\n>  debug\n>\n>  cache_setup () {\n> -\tcachedir=\"$GIT_DIR/subtree-cache/$$\"\n> -\trm -rf \"$cachedir\" ||\n> -\t\tdie \"Can't delete old cachedir: $cachedir\"\n> +\tcachedir=\"$GIT_DIR/subtree-cache/$prefix\"\n\nExcellent, the `prefix` should be \"unique enough\".\n\n> +\tif test -n \"$clearcache\"\n> +\tthen\n> +\t\tdebug \"Clearing cache\"\n> +\t\trm -rf \"$cachedir\" ||\n> +\t\t\tdie \"Can't delete old cachedir: $cachedir\"\n> +\tfi\n>  \tmkdir -p \"$cachedir\" ||\n>  \t\tdie \"Can't create new cachedir: $cachedir\"\n>  \tmkdir -p \"$cachedir/notree\" ||\n> @@ -266,6 +275,16 @@ cache_set () {\n>  \techo \"$newrev\" >\"$cachedir/$oldrev\"\n>  }\n>\n> +cache_set_if_unset () {\n> +\toldrev=\"$1\"\n> +\tnewrev=\"$2\"\n\n`local`? ;-)\n\n> +\tif test -e \"$cachedir/$oldrev\"\n> +\tthen\n> +\t\treturn\n> +\tfi\n> +\techo \"$newrev\" >\"$cachedir/$oldrev\"\n\nSo that directory contains commit mappings, a file for each mapped\nrevision.\n\nThinking back to patch 2/11, I am now no longer that sure that it makes\nsense to fill it up with every commit in that commit range: performance\nsuffers when directories contain too many files.\n\nFor example, I had a case in the past where it took a minute just to\nenumerate a directory, and even looking whether a file existed in that\ndirectory was not exactly fun.\n\nIn any case, I would write it slightly shorter:\n\n\ttest -e \"$cachedir/$oldrev\" ||\n\techo \"$newrev\" >\"$cachedir/$oldrev\"\n\n> +}\n> +\n>  rev_exists () {\n>  \tif git rev-parse \"$1\" >/dev/null 2>&1\n>  \tthen\n> @@ -375,13 +394,13 @@ find_existing_splits () {\n>  \t\t\tthen\n>  \t\t\t\t# squash commits refer to a subtree\n>  \t\t\t\tdebug \"  Squash: $sq from $sub\"\n> -\t\t\t\tcache_set \"$sq\" \"$sub\"\n> +\t\t\t\tcache_set_if_unset \"$sq\" \"$sub\"\n>  \t\t\tfi\n>  \t\t\tif test -n \"$main\" -a -n \"$sub\"\n>  \t\t\tthen\n>  \t\t\t\tdebug \"  Prior: $main -> $sub\"\n> -\t\t\t\tcache_set $main $sub\n> -\t\t\t\tcache_set $sub $sub\n> +\t\t\t\tcache_set_if_unset $main $sub\n> +\t\t\t\tcache_set_if_unset $sub $sub\n>  \t\t\t\ttry_remove_previous \"$main\"\n>  \t\t\t\ttry_remove_previous \"$sub\"\n>  \t\t\tfi\n> @@ -688,6 +707,8 @@ process_split_commit () {\n>  \t\tif test -n \"$newparents\"\n>  \t\tthen\n>  \t\t\tcache_set \"$rev\" \"$rev\"\n> +\t\telse\n> +\t\t\tcache_set \"$rev\" \"\"\n\nWas this hunk intended to be snuck in here? I can understand the\ns/cache_set/cache_set_if_unset/ changes, of course, but not this hunk.\n\n>  \t\tfi\n>  \t\treturn\n>  \tfi\n> @@ -785,7 +806,7 @@ cmd_split () {\n>  \t\t\t# the 'onto' history is already just the subdir, so\n>  \t\t\t# any parent we find there can be used verbatim\n>  \t\t\tdebug \"  cache: $rev\"\n> -\t\t\tcache_set \"$rev\" \"$rev\"\n> +\t\t\tcache_set_if_unset \"$rev\" \"$rev\"\n>  \t\tdone\n>  \tfi\n>\n> @@ -798,7 +819,7 @@ cmd_split () {\n>  \t\tgit rev-list --topo-order --skip=1 $mainline |\n>  \t\twhile read rev\n>  \t\tdo\n> -\t\t\tcache_set \"$rev\" \"\"\n> +\t\t\tcache_set_if_unset \"$rev\" \"\"\n\nOkay. A quite interesting question now would be: are there any callers of\n`cache_set` left? If so, why?\n\nThanks,\nDscho\n\n>  \t\tdone || exit $?\n>  \tfi\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"407070","messageId":"nycvar.QRO.7.76.6.2010071812400.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"2d103292cecfef3b29d221b3d8f05adb0add4475.1602021913.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/7] subtree: add git subtree use and ignore commands","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T16:29:44Z","receivedAt":"2020-10-07T16:29:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tom,\n\nOn Tue, 6 Oct 2020, Tom Clarkson via GitGitGadget wrote:\n\n> @@ -796,20 +810,60 @@ cmd_add_commit () {\n>  }\n>\n>  cmd_map () {\n> -\toldrev=\"$1\"\n> -\tnewrev=\"$2\"\n>\n> -\tif test -z \"$oldrev\"\n> +\tif test -z \"$1\"\n\nI'd like to keep the nice name. Maybe if it is `local`, there is no longer\na need to replace `$oldrev` by `$1`?\n\n>  \tthen\n>  \t\tdie \"You must provide a revision to map\"\n>  \tfi\n>\n> +\toldrev=$(git rev-parse --revs-only \"$1\") || exit $?\n> +\tnewrev=\n> +\n> +\tif test -n \"$2\"\n> +\tthen\n> +\t\tnewrev=$(git rev-parse --revs-only \"$2\") || exit $?\n> +\tfi\n> +\n\nWould it not make more sense to validate the parameters before calling\n`cmd_map`?\n\nIn any case, this strikes me like a subject for another commit.\n\nThanks,\nDscho\n\nP.S.: I'll have to stop reviewing here for the moment, not sure whether\nI'll come back to it later today or maybe tomorrow.\n\n\n>  \tcache_setup || exit $?\n>  \tcache_set \"$oldrev\" \"$newrev\"\n>\n>  \tsay \"Mapped $oldrev => $newrev\"\n>  }\n>\n> +cmd_ignore () {\n> +\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n> +\tensure_single_rev $revs\n> +\n> +\tsay \"Ignoring $revs\"\n> +\n> +\tcache_setup || exit $?\n> +\n> +\tgit rev-list $revs |\n> +\twhile read rev\n> +\tdo\n> +\t\tcache_set \"$rev\" \"\"\n> +\tdone\n> +\n> +\techo \"$revs\" >>\"$cachedir/processed\"\n> +}\n> +\n> +cmd_use () {\n> +\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n> +\tensure_single_rev $revs\n> +\n> +\tsay \"Using existing subtree $revs\"\n> +\n> +\tcache_setup || exit $?\n> +\n> +\tgit rev-list $revs |\n> +\twhile read rev\n> +\tdo\n> +\t\tcache_set \"$rev\" \"$rev\"\n> +\tdone\n> +\n> +\techo \"$revs\" >>\"$cachedir/processed\"\n> +}\n> +\n>  cmd_split () {\n>  \tdebug \"Splitting $dir...\"\n>  \tcache_setup || exit $?\n> @@ -827,7 +881,7 @@ cmd_split () {\n>  \t\tdone\n>  \tfi\n>\n> -\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n> +\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\") $(exclude_processed_refs)\"\n>\n>  \tmainline=\"$(find_mainline_ref \"$dir\" \"$revs\")\"\n>  \tif test -n \"$mainline\"\n> --\n> gitgitgadget\n>\n>\n"},{"id":"407071","messageId":"nycvar.QRO.7.76.6.2010071829590.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"CAPyFy2BeHnacdqd_C8BdPntsN7FCoTciL+s8fDXGhBU0H01FaA@mail.gmail.com","subject":"Re: [PATCH 0/7] subtree: Fix handling of complex history","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T16:31:34Z","receivedAt":"2020-10-07T16:31:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ed,\n\nOn Mon, 5 Oct 2020, Ed Maste wrote:\n\n> On Mon, 5 Oct 2020 at 09:18, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n> >\n> > FWIW there have been more comments in favor of this patch in\n> > https://github.com/gitgitgadget/git/pull/493.\n> >\n> > I guess what this patch series needs to proceed further is an ACK by\n> > Avery?\n>\n> Avery says \"if it's good enough for Johannes it's good enough for me\"\n> https://twitter.com/apenwarr/status/1313231132721401861\n\nWhat Avery does not quite understand is that Johannes has not a lot of\nclue about `git subtree`... ;-)\n\nSeriously, I am not a user, unfamiliar with the implementation details\n(although I am getting a bit more familiar through the review I started\nand plan on finishing later). My biggest connection with `git subtree` is\nthat there seem to be a couple of Git for Windows users who actively use\nthat command (which is the reason why we include it in Git for Windows,\nunlike many other things from `contrib/`).\n\nCiao,\nDscho\n"},{"id":"407091","messageId":"nycvar.QRO.7.76.6.2010072128130.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"a7aaedfed3785c6ca693f60f05e76156f68a5d39.1602021913.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/7] subtree: more robustly distinguish subtree and mainline commits","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T19:42:44Z","receivedAt":"2020-10-07T19:42:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 6 Oct 2020, Tom Clarkson via GitGitGadget wrote:\n\n> From: Tom Clarkson <tom@tqclarkson.com>\n>\n> Prevent a mainline commit without $dir being treated as a subtree\n> commit and pulling in the entire mainline history. Any valid subtree\n> commit will have only valid subtree commits as parents, which will be\n> unchanged by check_parents.\n\nI feel like this is only half the picture because I have a hard time\nstitching these two sentences together.\n\nAfter studying the code and your patch a bit, it appears to me that\n`process_split_commit()` calls `check_parents()` first, which will call\n`process_split_commit()` for all as yet unmapped parents. So basically, it\nrecurses until it found a commit all of whose parents are already mapped,\nthen permeates that information all the way back.\n\nDoesn't this cause serious issues with stack overflows and all for long\ncommit histories?\n\n> Signed-off-by: Tom Clarkson <tom@tqclarkson.com>\n> ---\n>  contrib/subtree/git-subtree.sh | 24 +++++++++++-------------\n>  1 file changed, 11 insertions(+), 13 deletions(-)\n>\n> diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh\n> index e56621a986..fa6293b372 100755\n> --- a/contrib/subtree/git-subtree.sh\n> +++ b/contrib/subtree/git-subtree.sh\n> @@ -224,8 +224,6 @@ cache_setup () {\n>  \tfi\n>  \tmkdir -p \"$cachedir\" ||\n>  \t\tdie \"Can't create new cachedir: $cachedir\"\n> -\tmkdir -p \"$cachedir/notree\" ||\n> -\t\tdie \"Can't create new cachedir: $cachedir/notree\"\n\nIt might make sense to talk about this a bit in the commit message.\nEssentially, you are replacing the `notree/<rev>` files by mapping `<rev>`\nto the empty string.\n\nThis makes me wonder, again, whether the file system layout of the cache\ncan hold up to the demands. If a main project were to merge a subtree\nwith, say, 10 million commits, wouldn't that mean that `git subtree` would\nnow fill one directory with 10 million files? I cannot imagine that this\nperforms well, still.\n\n>  \tdebug \"Using cachedir: $cachedir\" >&2\n>  }\n>\n> @@ -255,18 +253,11 @@ check_parents () {\n>  \tlocal indent=$(($2 + 1))\n>  \tfor miss in $missed\n>  \tdo\n> -\t\tif ! test -r \"$cachedir/notree/$miss\"\n> -\t\tthen\n> -\t\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n> -\t\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n> -\t\tfi\n> +\t\tdebug \"  unprocessed parent commit: $miss ($indent)\"\n> +\t\tprocess_split_commit \"$miss\" \"\" \"$indent\"\n\nThat makes sense to me, as the `missed` variable only contains as yet\nunmapped commits, therefore we do not have to have an equivalent `test -r`\ncheck.\n\nCiao,\nDscho\n\n>  \tdone\n>  }\n>\n> -set_notree () {\n> -\techo \"1\" > \"$cachedir/notree/$1\"\n> -}\n> -\n>  cache_set () {\n>  \toldrev=\"$1\"\n>  \tnewrev=\"$2\"\n> @@ -719,11 +710,18 @@ process_split_commit () {\n>  \t# vs. a mainline commit?  Does it matter?\n>  \tif test -z \"$tree\"\n>  \tthen\n> -\t\tset_notree \"$rev\"\n>  \t\tif test -n \"$newparents\"\n>  \t\tthen\n> -\t\t\tcache_set \"$rev\" \"$rev\"\n> +\t\t\tif test \"$newparents\" = \"$parents\"\n> +\t\t\tthen\n> +\t\t\t\t# if all parents were subtrees, this can be a subtree commit\n> +\t\t\t\tcache_set \"$rev\" \"$rev\"\n> +\t\t\telse\n> +\t\t\t\t# a mainline commit with tree missing is equivalent to the initial commit\n> +\t\t\t\tcache_set \"$rev\" \"\"\n> +\t\t\tfi\n>  \t\telse\n> +\t\t\t# no parents with valid subtree mappings means a commit prior to subtree add\n>  \t\t\tcache_set \"$rev\" \"\"\n>  \t\tfi\n>  \t\treturn\n> --\n> gitgitgadget\n>\n>\n"},{"id":"407092","messageId":"nycvar.QRO.7.76.6.2010072117340.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"fe2e4819b869725f870cd3ce99f1f8150fe17dc1.1602021913.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 7/7] subtree: document new subtree commands","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T19:43:28Z","receivedAt":"2020-10-07T19:43:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tom,\n\nOn Tue, 6 Oct 2020, Tom Clarkson via GitGitGadget wrote:\n\n> From: Tom Clarkson <tom@tqclarkson.com>\n>\n> Signed-off-by: Tom Clarkson <tom@tqclarkson.com>\n> ---\n>  contrib/subtree/git-subtree.txt | 24 ++++++++++++++++++++++++\n>  1 file changed, 24 insertions(+)\n>\n> diff --git a/contrib/subtree/git-subtree.txt b/contrib/subtree/git-subtree.txt\n> index 352deda69d..a5a76e8ce6 100644\n> --- a/contrib/subtree/git-subtree.txt\n> +++ b/contrib/subtree/git-subtree.txt\n> @@ -52,6 +52,12 @@ useful elsewhere, you can extract its entire history and publish\n>  that as its own git repository, without accidentally\n>  intermingling the history of your application project.\n>\n> +Although the relationship between subtree and mainline commits is stored\n\nAs far as I can see, this is the first time the term \"mainline commit\" is\nused in that file, and it has not really be defined what you mean by that.\nI *guess* you are referring to commits in the main project that did not\ncome from any subtree project.\n\nMaybe this can be described without needing a new term?\n\nCiao,\nDscho\n\n> +in regular git history, it is also cached between subtree runs. In most\n> +cases this is merely a performance improvement, but for projects with\n> +large and complex histories the cache can be manipulated directly\n> +with the use, ignore and map commands.\n> +\n>  [TIP]\n>  In order to keep your commit messages clean, we recommend that\n>  people split their commits between the subtrees and the main\n> @@ -120,6 +126,21 @@ and friends will work as expected.\n>  Note that if you use '--squash' when you merge, you should usually not\n>  just '--rejoin' when you split.\n>\n> +ignore::\n> +\tMark a commit and all of its history as irrelevant to subtree split.\n> +\tIn most cases this would be handled automatically based on metadata\n> +\tfrom subtree join commits. Intended for improving performance on\n> +\textremely large repos and excluding complex history that turns out\n> +\tto be otherwise problematic.\n> +\n> +use::\n> +\tMark a commit and all of its history as part of an existing subtree.\n> +\tIn normal circumstances this would be handled based on the metadata\n> +\tfrom the subtree join commit. Similar to the --onto option of split.\n> +\n> +map::\n> +\tManually override the normal output of split for a particular commit.\n> +\tExtreme flexibility for advanced troubleshooting purposes only.\n>\n>  OPTIONS\n>  -------\n> @@ -142,6 +163,9 @@ OPTIONS\n>  \tThis option is only valid for add, merge and pull (unsure).\n>  \tSpecify <message> as the commit message for the merge commit.\n>\n> +--clear-cache::\n> +\tReset the subtree cache and recalculate all subtree mappings from the\n> +\tcommit history\n>\n>  OPTIONS FOR add, merge, push, pull\n>  ----------------------------------\n> --\n> gitgitgadget\n>\n"},{"id":"407093","messageId":"nycvar.QRO.7.76.6.2010072143390.50@tvgsbejvaqbjf.bet","threadId":"53444","inReplyTo":"pull.493.v2.git.1602021913.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/7] subtree: Fix handling of complex history","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-07T19:46:36Z","receivedAt":"2020-10-07T19:46:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tom,\n\nOn Tue, 6 Oct 2020, Tom Clarkson via GitGitGadget wrote:\n\n> Fixes several issues that could occur when running subtree split on large\n> repos with more complex history.\n>\n>  1. A merge commit could bypass the known start point of the subtree, which\n>     would cause the entire history to be processed recursively, leading to a\n>     stack overflow / segfault after reading a few hundred commits. Older\n>     commits are now explicitly recorded as irrelevant so that the recursive\n>     process can terminate on any mainline commit rather than only on subtree\n>     joins and initial commits.\n>\n>\n>  2. It is possible for a repo to contain subtrees that lack the metadata\n>     that is usually present in add/join commit messages (git-svn at least\n>     can produce such a structure). The new use/ignore/map commands allow the\n>     user to provide that information for any problematic commits.\n>\n>\n>  3. A mainline commit that does not contain the subtree folder could be\n>     erroneously identified as a subtree commit, which would add the entire\n>     mainline history to the subtree. Commits will now only be used as is if\n>     all their parents are already identified as subtree commits. While the\n>     new code can still be tripped up by unusual folder structures, the\n>     completely unambiguous solution turned out to involve a significant\n>     performance penalty, and the new ignore / use commands provide a\n>     workaround for that scenario.\n\nI gave this as thorough a review as I can (which is not saying too much,\nas I am not exactly familiar with `git subtree`'s inner workings).\n\nHopefully some of my comments and suggestions are helpful.\n\nAt some stage, especially given the problems I pointed out with the\nimplementation detail that is a flat directory with a potentially insane\nnumber of files in it, I think it would make a lot of sense to go ahead\nand turn this into a built-in Git command, implemented in C, and with a\nmore robust file system layout of its cache.\n\nCiao,\nDscho\n"}]}