{"thread":{"id":"46797","subject":"[PATCH v3 0/4] filter-branch: support for incremental update + fix for ancient tag format","startedAt":"2017-09-21T07:49:15Z","lastAt":"2017-09-22T08:58:39Z","messageCount":8,"participants":["Ian Campbell","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":4},"messages":[{"id":"328555","messageId":"1505980146.4636.9.camel@hellion.org.uk","threadId":"46797","inReplyTo":null,"subject":"[PATCH v3 0/4] filter-branch: support for incremental update + fix for ancient tag format","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-21T07:49:06Z","receivedAt":"2017-09-21T07:49:15Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"This is the third version of my patches to add incremental support to\ngit-filter-branch. Since the last time I have replaced `git mktag --\nallow-missing-tagger` with `git hash-object -t tag -w --stdin`.\n\nI've force pushed to [1] (Travis is still running) and have set off the\nprocess of re-rewriting the devicetree tree from scratch (a multi-day\naffair) to validate (it's looking good).\n\nIan.\n\n[0] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/\n[1] https://github.com/ijc/git/tree/git-filter-branch\n"},{"id":"328556","messageId":"20170921074932.5490-2-ijc@hellion.org.uk","threadId":"46797","inReplyTo":"1505980146.4636.9.camel@hellion.org.uk","subject":"[PATCH v3 2/4] filter-branch: preserve and restore $GIT_AUTHOR_* and $GIT_COMMITTER_*","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-21T07:49:30Z","receivedAt":"2017-09-21T07:49:41Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"These are modified by set_ident() but a subsequent patch would like to operate\non their original values.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\n git-filter-branch.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 3da281f8a..9edb94206 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -219,6 +219,13 @@ trap 'cd \"$orig_dir\"; rm -rf \"$tempdir\"' 0\n ORIG_GIT_DIR=\"$GIT_DIR\"\n ORIG_GIT_WORK_TREE=\"$GIT_WORK_TREE\"\n ORIG_GIT_INDEX_FILE=\"$GIT_INDEX_FILE\"\n+ORIG_GIT_AUTHOR_NAME=\"$GIT_AUTHOR_NAME\"\n+ORIG_GIT_AUTHOR_EMAIL=\"$GIT_AUTHOR_EMAIL\"\n+ORIG_GIT_AUTHOR_DATE=\"$GIT_AUTHOR_DATE\"\n+ORIG_GIT_COMMITTER_NAME=\"$GIT_COMMITTER_NAME\"\n+ORIG_GIT_COMMITTER_EMAIL=\"$GIT_COMMITTER_EMAIL\"\n+ORIG_GIT_COMMITTER_DATE=\"$GIT_COMMITTER_DATE\"\n+\n GIT_WORK_TREE=.\n export GIT_DIR GIT_WORK_TREE\n \n@@ -545,6 +552,8 @@ if [ \"$filter_tag_name\" ]; then\n fi\n \n unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE\n+unset GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_AUTHOR_DATE\n+unset GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL GIT_COMMITTER_DATE\n test -z \"$ORIG_GIT_DIR\" || {\n \tGIT_DIR=\"$ORIG_GIT_DIR\" && export GIT_DIR\n }\n@@ -556,6 +565,30 @@ test -z \"$ORIG_GIT_INDEX_FILE\" || {\n \tGIT_INDEX_FILE=\"$ORIG_GIT_INDEX_FILE\" &&\n \texport GIT_INDEX_FILE\n }\n+test -z \"$ORIG_GIT_AUTHOR_NAME\" || {\n+\tGIT_AUTHOR_NAME=\"$ORIG_GIT_AUTHOR_NAME\" &&\n+\texport GIT_AUTHOR_NAME\n+}\n+test -z \"$ORIG_GIT_AUTHOR_EMAIL\" || {\n+\tGIT_AUTHOR_EMAIL=\"$ORIG_GIT_AUTHOR_EMAIL\" &&\n+\texport GIT_AUTHOR_EMAIL\n+}\n+test -z \"$ORIG_GIT_AUTHOR_DATE\" || {\n+\tGIT_AUTHOR_DATE=\"$ORIG_GIT_AUTHOR_DATE\" &&\n+\texport GIT_AUTHOR_DATE\n+}\n+test -z \"$ORIG_GIT_COMMITTER_NAME\" || {\n+\tGIT_COMMITTER_NAME=\"$ORIG_GIT_COMMITTER_NAME\" &&\n+\texport GIT_COMMITTER_NAME\n+}\n+test -z \"$ORIG_GIT_COMMITTER_EMAIL\" || {\n+\tGIT_COMMITTER_EMAIL=\"$ORIG_GIT_COMMITTER_EMAIL\" &&\n+\texport GIT_COMMITTER_EMAIL\n+}\n+test -z \"$ORIG_GIT_COMMITTER_DATE\" || {\n+\tGIT_COMMITTER_DATE=\"$ORIG_GIT_COMMITTER_DATE\" &&\n+\texport GIT_COMMITTER_DATE\n+}\n \n cd \"$orig_dir\"\n rm -rf \"$tempdir\"\n-- \n2.11.0\n\n"},{"id":"328557","messageId":"20170921074932.5490-4-ijc@hellion.org.uk","threadId":"46797","inReplyTo":"1505980146.4636.9.camel@hellion.org.uk","subject":"[PATCH v3 4/4] filter-branch: use hash-object instead of mktag","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-21T07:49:32Z","receivedAt":"2017-09-21T07:49:44Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"This allows us to recreate even historical tags which would now be consider\ninvalid, such as v2.6.12-rc2..v2.6.13-rc3 in the Linux kernel source tree which\nlack the `tagger` header.\n\n    $ git rev-parse v2.6.12-rc2\n    9e734775f7c22d2f89943ad6c745571f1930105f\n    $ git cat-file tag v2.6.12-rc2 | git mktag\n    error: char76: could not find \"tagger \"\n    fatal: invalid tag signature file\n    $ git cat-file tag v2.6.12-rc2 | git hash-object -t tag -w --stdin\n    9e734775f7c22d2f89943ad6c745571f1930105f\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\n git-filter-branch.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 956869b8e..3365a3b86 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -561,7 +561,7 @@ if [ \"$filter_tag_name\" ]; then\n \t\t\t\t\t}' \\\n \t\t\t\t    -e '/^-----BEGIN PGP SIGNATURE-----/q' \\\n \t\t\t\t    -e 'p' ) |\n-\t\t\t\tgit mktag) ||\n+\t\t\t\tgit hash-object -t tag -w --stdin) ||\n \t\t\t\tdie \"Could not create new tag object for $ref\"\n \t\t\tif git cat-file tag \"$ref\" | \\\n \t\t\t   sane_grep '^-----BEGIN PGP SIGNATURE-----' >/dev/null 2>&1\n-- \n2.11.0\n\n"},{"id":"328558","messageId":"20170921074932.5490-1-ijc@hellion.org.uk","threadId":"46797","inReplyTo":"1505980146.4636.9.camel@hellion.org.uk","subject":"[PATCH v3 1/4] filter-branch: reset $GIT_* before cleaning up","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-21T07:49:29Z","receivedAt":"2017-09-21T07:49:46Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"This is pure code motion to enable a subsequent patch to add code which needs\nto happen with the reset $GIT_* but before the temporary directory has been\ncleaned up.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\n git-filter-branch.sh | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 3a74602ef..3da281f8a 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -544,11 +544,6 @@ if [ \"$filter_tag_name\" ]; then\n \tdone\n fi\n \n-cd \"$orig_dir\"\n-rm -rf \"$tempdir\"\n-\n-trap - 0\n-\n unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE\n test -z \"$ORIG_GIT_DIR\" || {\n \tGIT_DIR=\"$ORIG_GIT_DIR\" && export GIT_DIR\n@@ -562,6 +557,11 @@ test -z \"$ORIG_GIT_INDEX_FILE\" || {\n \texport GIT_INDEX_FILE\n }\n \n+cd \"$orig_dir\"\n+rm -rf \"$tempdir\"\n+\n+trap - 0\n+\n if [ \"$(is_bare_repository)\" = false ]; then\n \tgit read-tree -u -m HEAD || exit\n fi\n-- \n2.11.0\n\n"},{"id":"328559","messageId":"20170921074932.5490-3-ijc@hellion.org.uk","threadId":"46797","inReplyTo":"1505980146.4636.9.camel@hellion.org.uk","subject":"[PATCH v3 3/4] filter-branch: stash away ref map in a branch","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-21T07:49:31Z","receivedAt":"2017-09-21T07:49:48Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"With \"--state-branch=<branchname>\" option, the mapping from old object names\nand filtered ones in ./map/ directory is stashed away in the object database,\nand the one from the previous run is read to populate the ./map/ directory,\nallowing for incremental updates of large trees.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\nI have been using this as part of the device tree extraction from the Linux\nkernel source since 2013, about time I sent the patch upstream!\n\nv2:\n- added several preceding cleanup patches, including:\n  - new: use of mktag --allow-missing tagger.\n  - split-out: preserving $GIT_*.\n- use git rev-parse rather than git show-ref.\n- improved error handling for Perl sub-processes.\n- collapsed some shell pipelines involving piping output of git and ls into\n  Perl into the Perl scripts.\n- style fixes for conditionals and sub-shells.\n- fixup indentation.\n- added documentation.\n- improved commit message.\n---\n Documentation/git-filter-branch.txt |  8 +++++-\n git-filter-branch.sh                | 49 ++++++++++++++++++++++++++++++++++++-\n 2 files changed, 55 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 9e5169aa6..bebdcdec5 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--commit-filter <command>] [--tag-name-filter <command>]\n \t[--subdirectory-filter <directory>] [--prune-empty]\n \t[--original <namespace>] [-d <directory>] [-f | --force]\n-\t[--] [<rev-list options>...]\n+\t[--state-branch <branch>] [--] [<rev-list options>...]\n \n DESCRIPTION\n -----------\n@@ -198,6 +198,12 @@ to other tags will be rewritten to point to the underlying commit.\n \tdirectory or when there are already refs starting with\n \t'refs/original/', unless forced.\n \n+--state-branch <branch>::\n+\tThis option will cause the mapping from old to new objects to\n+\tbe loaded from named branch upon startup and saved as a new\n+\tcommit to that branch upon exit, enabling incremental of large\n+\ttrees. If '<branch>' does not exist it will be created.\n+\n <rev-list options>...::\n \tArguments for 'git rev-list'.  All positive refs included by\n \tthese options are rewritten.  You may also specify options\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 9edb94206..956869b8e 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -86,7 +86,7 @@ USAGE=\"[--setup <command>] [--env-filter <command>]\n \t[--parent-filter <command>] [--msg-filter <command>]\n \t[--commit-filter <command>] [--tag-name-filter <command>]\n \t[--subdirectory-filter <directory>] [--original <namespace>]\n-\t[-d <directory>] [-f | --force]\n+\t[-d <directory>] [-f | --force] [--state-branch <branch>]\n \t[--] [<rev-list options>...]\"\n \n OPTIONS_SPEC=\n@@ -106,6 +106,7 @@ filter_msg=cat\n filter_commit=\n filter_tag_name=\n filter_subdir=\n+state_branch=\n orig_namespace=refs/original/\n force=\n prune_empty=\n@@ -181,6 +182,9 @@ do\n \t--original)\n \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n \t\t;;\n+\t--state-branch)\n+\t\tstate_branch=\"$OPTARG\"\n+\t\t;;\n \t*)\n \t\tusage\n \t\t;;\n@@ -259,6 +263,26 @@ export GIT_INDEX_FILE\n # map old->new commit ids for rewriting parents\n mkdir ../map || die \"Could not create map/ directory\"\n \n+if test -n \"$state_branch\"\n+then\n+\tstate_commit=$(git rev-parse --no-flags --revs-only \"$state_branch\")\n+\tif test -n \"$state_commit\"\n+\tthen\n+\t\techo \"Populating map from $state_branch ($state_commit)\" 1>&2\n+\t\tperl -e'open(MAP, \"-|\", \"git show $ARGV[0]:filter.map\") or die;\n+\t\t\twhile (<MAP>) {\n+\t\t\t\tm/(.*):(.*)/ or die;\n+\t\t\t\topen F, \">../map/$1\" or die;\n+\t\t\t\tprint F \"$2\" or die;\n+\t\t\t\tclose(F) or die;\n+\t\t\t}\n+\t\t\tclose(MAP) or die;' \"$state_commit\" \\\n+\t\t\t\t|| die \"Unable to load state from $state_branch:filter.map\"\n+\telse\n+\t\techo \"Branch $state_branch does not exist. Will create\" 1>&2\n+\tfi\n+fi\n+\n # we need \"--\" only if there are no path arguments in $@\n nonrevs=$(git rev-parse --no-revs \"$@\") || exit\n if test -z \"$nonrevs\"\n@@ -590,6 +614,29 @@ test -z \"$ORIG_GIT_COMMITTER_DATE\" || {\n \texport GIT_COMMITTER_DATE\n }\n \n+if test -n \"$state_branch\"\n+then\n+\techo \"Saving rewrite state to $state_branch\" 1>&2\n+\tstate_blob=$(\n+\t\tperl -e'opendir D, \"../map\" or die;\n+\t\t\topen H, \"|-\", \"git hash-object -w --stdin\" or die;\n+\t\t\tforeach (sort readdir(D)) {\n+\t\t\t\tnext if m/^\\.\\.?$/;\n+\t\t\t\topen F, \"<../map/$_\" or die;\n+\t\t\t\tchomp($f = <F>);\n+\t\t\t\tprint H \"$_:$f\\n\" or die;\n+\t\t\t}\n+\t\t\tclose(H) or die;' || die \"Unable to save state\")\n+\tstate_tree=$(/bin/echo -e \"100644 blob $state_blob\\tfilter.map\" | git mktree)\n+\tif test -n \"$state_commit\"\n+\tthen\n+\t\tstate_commit=$(/bin/echo \"Sync\" | git commit-tree \"$state_tree\" -p \"$state_commit\")\n+\telse\n+\t\tstate_commit=$(/bin/echo \"Sync\" | git commit-tree \"$state_tree\" )\n+\tfi\n+\tgit update-ref \"$state_branch\" \"$state_commit\"\n+fi\n+\n cd \"$orig_dir\"\n rm -rf \"$tempdir\"\n \n-- \n2.11.0\n\n"},{"id":"328619","messageId":"xmqqzi9njqec.fsf@gitster.mtv.corp.google.com","threadId":"46797","inReplyTo":"1505980146.4636.9.camel@hellion.org.uk","subject":"Re: [PATCH v3 0/4] filter-branch: support for incremental update + fix for ancient tag format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-22T04:42:03Z","receivedAt":"2017-09-22T04:42:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ian Campbell <ijc@hellion.org.uk> writes:\n\n> This is the third version of my patches to add incremental support to\n> git-filter-branch. Since the last time I have replaced `git mktag --\n> allow-missing-tagger` with `git hash-object -t tag -w --stdin`.\n>\n> I've force pushed to [1] (Travis is still running) and have set off the\n> process of re-rewriting the devicetree tree from scratch (a multi-day\n> affair) to validate (it's looking good).\n\nThanks.\n"},{"id":"328626","messageId":"1506069530.2682.41.camel@hellion.org.uk","threadId":"46797","inReplyTo":"xmqqzi9njqec.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 0/4] filter-branch: support for incremental update + fix for ancient tag format","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-22T08:38:50Z","receivedAt":"2017-09-22T08:39:10Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"On Fri, 2017-09-22 at 13:42 +0900, Junio C Hamano wrote:\n> Ian Campbell <ijc@hellion.org.uk> writes:\n> \n> > This is the third version of my patches to add incremental support to\n> > git-filter-branch. Since the last time I have replaced `git mktag --\n> > allow-missing-tagger` with `git hash-object -t tag -w --stdin`.\n> > \n> > I've force pushed to [1] (Travis is still running) and have set off the\n> > process of re-rewriting the devicetree tree from scratch (a multi-day\n> > affair) to validate (it's looking good).\n> \n> Thanks.\n\nTravis is happy and the dt reconvert looks sensible (only took 60 hours\n;-)).\n\nDon't know if this is useful to your workflow but:\n\nThe following changes since commit 4384e3cde2ce8ecd194202e171ae16333d241326:\n\n  Git 2.14 (2017-08-04 09:31:12 -0700)\n\nare available in the git repository at:\n\n  https://github.com/ijc/git git-filter-branch\n\nfor you to fetch changes up to e31c74f709fbf2827d57b4abf826bb836f120329:\n\n  filter-branch: use hash-object instead of mktag (2017-09-21 08:44:59 +0100)\n\n----------------------------------------------------------------\nIan Campbell (4):\n      filter-branch: reset $GIT_* before cleaning up\n      filter-branch: preserve and restore $GIT_AUTHOR_* and $GIT_COMMITTER_*\n      filter-branch: stash away ref map in a branch\n      filter-branch: use hash-object instead of mktag\n\n Documentation/git-filter-branch.txt |  8 +++++++-\n git-filter-branch.sh                | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 2 files changed, 94 insertions(+), 8 deletions(-)\n"},{"id":"328627","messageId":"xmqqk20rjeiv.fsf@gitster.mtv.corp.google.com","threadId":"46797","inReplyTo":"1506069530.2682.41.camel@hellion.org.uk","subject":"Re: [PATCH v3 0/4] filter-branch: support for incremental update + fix for ancient tag format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-22T08:58:32Z","receivedAt":"2017-09-22T08:58:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ian Campbell <ijc@hellion.org.uk> writes:\n\n> Travis is happy and the dt reconvert looks sensible (only took 60 hours\n> ;-)).\n\nGood.\n\n> Don't know if this is useful to your workflow but:\n>\n> The following changes since commit 4384e3cde2ce8ecd194202e171ae16333d241326:\n>\n>   Git 2.14 (2017-08-04 09:31:12 -0700)\n>\n> are available...\n\nThat should match (modulo that they lack my sign-off, for obvious\nreasons) what I have in the 'pu' branch, four commits on a single\nstrand of pearls ending at b2c1ca6b (\"filter-branch: use hash-object\ninstead of mktag\", 2017-09-21).\n\nThanks.\n"}]}