{"thread":{"id":"46768","subject":"[PATCH v2 1/4] mktag: add option which allows the tagger field to be omitted","startedAt":"2017-09-17T08:17:43Z","lastAt":"2017-09-19T06:42:46Z","messageCount":8,"participants":["Ian Campbell","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":4},"messages":[{"id":"328251","messageId":"20170917073657.31193-1-ijc@hellion.org.uk","threadId":"46768","inReplyTo":"1505633797.22447.4.camel@hellion.org.uk","subject":"[PATCH v2 1/4] mktag: add option which allows the tagger field to be omitted","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-17T07:36:54Z","receivedAt":"2017-09-17T08:17:43Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"This can be useful e.g. in `filter-branch` when rewritting tags produced by\nolder versions of Git, such as v2.6.12-rc2..v2.6.13-rc3 in the Linux kernel\nsource tree:\n\n        $ git cat-file tag v2.6.12-rc2\n        object 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2\n        type commit\n        tag v2.6.12-rc2\n\n        Linux v2.6.12-rc2 release\n        -----BEGIN PGP SIGNATURE-----\n        Version: GnuPG v1.2.4 (GNU/Linux)\n\n        iD8DBQBCbW8ZF3YsRnbiHLsRAgFRAKCq/TkuDaEombFABkPqYgGCgWN2lQCcC0qc\n        wznDbFU45A54dZC8RZ5JxyE=\n        =ESRP\n        -----END PGP SIGNATURE-----\n\n        $ git cat-file tag v2.6.12-rc2 | git mktag\n        error: char76: could not find \"tagger \"\n        fatal: invalid tag signature file\n        $ git cat-file tag v2.6.12-rc2 | git mktag --allow-missing-tagger\n        9e734775f7c22d2f89943ad6c745571f1930105f\n\nTo that end, pass the new option to `mktag` in `filter-branch`.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\n Documentation/git-mktag.txt |   9 +++-\n builtin/mktag.c             | 100 +++++++++++++++++++++++++-------------------\n git-filter-branch.sh        |   2 +-\n t/t3800-mktag.sh            |  33 ++++++++++++++-\n 4 files changed, 98 insertions(+), 46 deletions(-)\n\ndiff --git a/Documentation/git-mktag.txt b/Documentation/git-mktag.txt\nindex fa6a75612..c720c7419 100644\n--- a/Documentation/git-mktag.txt\n+++ b/Documentation/git-mktag.txt\n@@ -9,7 +9,7 @@ git-mktag - Creates a tag object\n SYNOPSIS\n --------\n [verse]\n-'git mktag'\n+'git mktag' [--allow-missing-tagger]\n \n DESCRIPTION\n -----------\n@@ -34,6 +34,13 @@ exists, is separated by a blank line from the header.  The\n message part may contain a signature that Git itself doesn't\n care about, but that can be verified with gpg.\n \n+OPTIONS\n+-------\n+--allow-missing-tagger::\n+\tAllow the `tagger` line in the header to be omitted. This is\n+\trarely desirable but may be useful in recreating tags created\n+\tby older Git.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 031b750f0..0f5dae8d5 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -1,4 +1,5 @@\n #include \"builtin.h\"\n+#include \"parse-options.h\"\n #include \"tag.h\"\n \n /*\n@@ -15,6 +16,8 @@\n  * the shortest possible tagger-line.\n  */\n \n+static int allow_missing_tagger;\n+\n /*\n  * We refuse to tag something we can't verify. Just because.\n  */\n@@ -41,8 +44,9 @@ static int verify_tag(char *buffer, unsigned long size)\n \tunsigned char sha1[20];\n \tconst char *object, *type_line, *tag_line, *tagger_line, *lb, *rb;\n \tsize_t len;\n+\tconst unsigned long min_size = allow_missing_tagger ? 71 : 84;\n \n-\tif (size < 84)\n+\tif (size < min_size)\n \t\treturn error(\"wanna fool me ? you obviously got the size wrong !\");\n \n \tbuffer[size] = 0;\n@@ -98,46 +102,46 @@ static int verify_tag(char *buffer, unsigned long size)\n \t/* Verify the tagger line */\n \ttagger_line = tag_line;\n \n-\tif (memcmp(tagger_line, \"tagger \", 7))\n+\tif (!memcmp(tagger_line, \"tagger \", 7)) {\n+\t\t/*\n+\t\t * Check for correct form for name and email\n+\t\t * i.e. \" <\" followed by \"> \" on _this_ line\n+\t\t * No angle brackets within the name or email address fields.\n+\t\t * No spaces within the email address field.\n+\t\t */\n+\t\ttagger_line += 7;\n+\t\tif (!(lb = strstr(tagger_line, \" <\")) || !(rb = strstr(lb+2, \"> \")) ||\n+\t\t\tstrpbrk(tagger_line, \"<>\\n\") != lb+1 ||\n+\t\t\tstrpbrk(lb+2, \"><\\n \") != rb)\n+\t\t\treturn error(\"char%\"PRIuMAX\": malformed tagger field\",\n+\t\t\t\t(uintmax_t) (tagger_line - buffer));\n+\n+\t\t/* Check for author name, at least one character, space is acceptable */\n+\t\tif (lb == tagger_line)\n+\t\t\treturn error(\"char%\"PRIuMAX\": missing tagger name\",\n+\t\t\t\t(uintmax_t) (tagger_line - buffer));\n+\n+\t\t/* timestamp, 1 or more digits followed by space */\n+\t\ttagger_line = rb + 2;\n+\t\tif (!(len = strspn(tagger_line, \"0123456789\")))\n+\t\t\treturn error(\"char%\"PRIuMAX\": missing tag timestamp\",\n+\t\t\t\t(uintmax_t) (tagger_line - buffer));\n+\t\ttagger_line += len;\n+\t\tif (*tagger_line != ' ')\n+\t\t\treturn error(\"char%\"PRIuMAX\": malformed tag timestamp\",\n+\t\t\t\t(uintmax_t) (tagger_line - buffer));\n+\t\ttagger_line++;\n+\n+\t\t/* timezone, 5 digits [+-]hhmm, max. 1400 */\n+\t\tif (!((tagger_line[0] == '+' || tagger_line[0] == '-') &&\n+\t\t      strspn(tagger_line+1, \"0123456789\") == 4 &&\n+\t\t      tagger_line[5] == '\\n' && atoi(tagger_line+1) <= 1400))\n+\t\t\treturn error(\"char%\"PRIuMAX\": malformed tag timezone\",\n+\t\t\t\t(uintmax_t) (tagger_line - buffer));\n+\t\ttagger_line += 6;\n+\t} else if (!allow_missing_tagger)\n \t\treturn error(\"char%\"PRIuMAX\": could not find \\\"tagger \\\"\",\n-\t\t\t(uintmax_t) (tagger_line - buffer));\n-\n-\t/*\n-\t * Check for correct form for name and email\n-\t * i.e. \" <\" followed by \"> \" on _this_ line\n-\t * No angle brackets within the name or email address fields.\n-\t * No spaces within the email address field.\n-\t */\n-\ttagger_line += 7;\n-\tif (!(lb = strstr(tagger_line, \" <\")) || !(rb = strstr(lb+2, \"> \")) ||\n-\t\tstrpbrk(tagger_line, \"<>\\n\") != lb+1 ||\n-\t\tstrpbrk(lb+2, \"><\\n \") != rb)\n-\t\treturn error(\"char%\"PRIuMAX\": malformed tagger field\",\n-\t\t\t(uintmax_t) (tagger_line - buffer));\n-\n-\t/* Check for author name, at least one character, space is acceptable */\n-\tif (lb == tagger_line)\n-\t\treturn error(\"char%\"PRIuMAX\": missing tagger name\",\n-\t\t\t(uintmax_t) (tagger_line - buffer));\n-\n-\t/* timestamp, 1 or more digits followed by space */\n-\ttagger_line = rb + 2;\n-\tif (!(len = strspn(tagger_line, \"0123456789\")))\n-\t\treturn error(\"char%\"PRIuMAX\": missing tag timestamp\",\n-\t\t\t(uintmax_t) (tagger_line - buffer));\n-\ttagger_line += len;\n-\tif (*tagger_line != ' ')\n-\t\treturn error(\"char%\"PRIuMAX\": malformed tag timestamp\",\n-\t\t\t(uintmax_t) (tagger_line - buffer));\n-\ttagger_line++;\n-\n-\t/* timezone, 5 digits [+-]hhmm, max. 1400 */\n-\tif (!((tagger_line[0] == '+' || tagger_line[0] == '-') &&\n-\t      strspn(tagger_line+1, \"0123456789\") == 4 &&\n-\t      tagger_line[5] == '\\n' && atoi(tagger_line+1) <= 1400))\n-\t\treturn error(\"char%\"PRIuMAX\": malformed tag timezone\",\n-\t\t\t(uintmax_t) (tagger_line - buffer));\n-\ttagger_line += 6;\n+\t\t\t     (uintmax_t) (tagger_line - buffer));\n \n \t/* Verify the blank line separating the header from the body */\n \tif (*tagger_line != '\\n')\n@@ -148,13 +152,25 @@ static int verify_tag(char *buffer, unsigned long size)\n \treturn 0;\n }\n \n+static char const * const mktag_usage[] = {\n+\tN_(\"git mktag [<options>]\"),\n+\tNULL\n+};\n+\n+static struct option mktag_opts[] = {\n+\tOPT_BOOL(0, \"allow-missing-tagger\", &allow_missing_tagger, N_(\"allow the tagger field to be omitted\")),\n+\tOPT_END(),\n+};\n+\n int cmd_mktag(int argc, const char **argv, const char *prefix)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tunsigned char result_sha1[20];\n \n-\tif (argc != 1)\n-\t\tusage(\"git mktag\");\n+\targc = parse_options(argc, argv, prefix, mktag_opts, mktag_usage, 0);\n+\n+\tif (argc != 0)\n+\t\tusage_with_options(mktag_usage, mktag_opts);\n \n \tif (strbuf_read(&buf, 0, 4096) < 0) {\n \t\tdie_errno(\"could not read from stdin\");\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 3a74602ef..05645064a 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -530,7 +530,7 @@ if [ \"$filter_tag_name\" ]; then\n \t\t\t\t\t}' \\\n \t\t\t\t    -e '/^-----BEGIN PGP SIGNATURE-----/q' \\\n \t\t\t\t    -e 'p' ) |\n-\t\t\t\tgit mktag) ||\n+\t\t\t\tgit mktag --allow-missing-tagger) ||\n \t\t\t\tdie \"Could not create new tag object for $ref\"\n \t\t\tif git cat-file tag \"$ref\" | \\\n \t\t\t   sane_grep '^-----BEGIN PGP SIGNATURE-----' >/dev/null 2>&1\ndiff --git a/t/t3800-mktag.sh b/t/t3800-mktag.sh\nindex 8eb47942e..3a77a26c8 100755\n--- a/t/t3800-mktag.sh\n+++ b/t/t3800-mktag.sh\n@@ -340,7 +340,36 @@ check_verify_failure 'detect invalid header entry' \\\n \t'^error: char124: trailing garbage in tag header$'\n \n ############################################################\n-# 24. create valid tag\n+# 24. missing tagger ok with --allow-missing-tagger\n+\n+cat >tag.sig <<EOF\n+object $head\n+type commit\n+tag mytag\n+\n+EOF\n+\n+test_expect_success \\\n+    'missing tagger with --allow-missing-tagger' \\\n+    'git mktag --allow-missing-tagger <tag.sig >.git/refs/tags/mytag 2>message'\n+\n+############################################################\n+# 25. detect invalid header entry with --allow-missing-tagger\n+\n+cat >tag.sig <<EOF\n+object $head\n+type commit\n+tag mytag\n+this line should not be here\n+EOF\n+\n+test_expect_success \\\n+    'detect invalid header entry with --allow-missing-tagger' \\\n+    '( test_must_fail git mktag --allow-missing-tagger <tag.sig 2>message ) &&\n+       grep \"^error: char70: trailing garbage in tag header$\" message'\n+\n+############################################################\n+# 26. create valid tag\n \n cat >tag.sig <<EOF\n object $head\n@@ -355,7 +384,7 @@ test_expect_success \\\n     'git mktag <tag.sig >.git/refs/tags/mytag 2>message'\n \n ############################################################\n-# 25. check mytag\n+# 27. check mytag\n \n test_expect_success \\\n     'check mytag' \\\n-- \n2.11.0\n\n"},{"id":"328252","messageId":"20170917073657.31193-4-ijc@hellion.org.uk","threadId":"46768","inReplyTo":"1505633797.22447.4.camel@hellion.org.uk","subject":"[PATCH v2 4/4] Subject: filter-branch: stash away ref map in a branch","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-17T07:36:57Z","receivedAt":"2017-09-17T08:17:45Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"With \"--state-branch=<branchname>\" option, the mapping from old object names\nand filtered ones in ./map/ directory is stashed away in the object database,\nand the one from the previous run is read to populate the ./map/ directory,\nallowing for incremental updates of large trees.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\nI have been using this as part of the device tree extraction from the Linux\nkernel source since 2013, about time I sent the patch upstream!\n\nv2:\n- added several preceding cleanup patches, including:\n  - new: use of mktag --allow-missing tagger.\n  - split-out: preserving $GIT_*.\n- use git rev-parse rather than git show-ref.\n- improved error handling for Perl sub-processes.\n- collapsed some shell pipelines involving piping output of git and ls into\n  Perl into the Perl scripts.\n- style fixes for conditionals and sub-shells.\n- fixup indentation.\n- added documentation.\n- improved commit message.\n---\n Documentation/git-filter-branch.txt |  8 +++++-\n git-filter-branch.sh                | 49 ++++++++++++++++++++++++++++++++++++-\n 2 files changed, 55 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 9e5169aa6..bebdcdec5 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--commit-filter <command>] [--tag-name-filter <command>]\n \t[--subdirectory-filter <directory>] [--prune-empty]\n \t[--original <namespace>] [-d <directory>] [-f | --force]\n-\t[--] [<rev-list options>...]\n+\t[--state-branch <branch>] [--] [<rev-list options>...]\n \n DESCRIPTION\n -----------\n@@ -198,6 +198,12 @@ to other tags will be rewritten to point to the underlying commit.\n \tdirectory or when there are already refs starting with\n \t'refs/original/', unless forced.\n \n+--state-branch <branch>::\n+\tThis option will cause the mapping from old to new objects to\n+\tbe loaded from named branch upon startup and saved as a new\n+\tcommit to that branch upon exit, enabling incremental of large\n+\ttrees. If '<branch>' does not exist it will be created.\n+\n <rev-list options>...::\n \tArguments for 'git rev-list'.  All positive refs included by\n \tthese options are rewritten.  You may also specify options\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex c88263965..ab927c62d 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -86,7 +86,7 @@ USAGE=\"[--setup <command>] [--env-filter <command>]\n \t[--parent-filter <command>] [--msg-filter <command>]\n \t[--commit-filter <command>] [--tag-name-filter <command>]\n \t[--subdirectory-filter <directory>] [--original <namespace>]\n-\t[-d <directory>] [-f | --force]\n+\t[-d <directory>] [-f | --force] [--state-branch <branch>]\n \t[--] [<rev-list options>...]\"\n \n OPTIONS_SPEC=\n@@ -106,6 +106,7 @@ filter_msg=cat\n filter_commit=\n filter_tag_name=\n filter_subdir=\n+state_branch=\n orig_namespace=refs/original/\n force=\n prune_empty=\n@@ -181,6 +182,9 @@ do\n \t--original)\n \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n \t\t;;\n+\t--state-branch)\n+\t\tstate_branch=\"$OPTARG\"\n+\t\t;;\n \t*)\n \t\tusage\n \t\t;;\n@@ -259,6 +263,26 @@ export GIT_INDEX_FILE\n # map old->new commit ids for rewriting parents\n mkdir ../map || die \"Could not create map/ directory\"\n \n+if test -n \"$state_branch\"\n+then\n+\tstate_commit=$(git rev-parse --no-flags --revs-only \"$state_branch\")\n+\tif test -n \"$state_commit\"\n+\tthen\n+\t\techo \"Populating map from $state_branch ($state_commit)\" 1>&2\n+\t\tperl -e'open(MAP, \"-|\", \"git show $ARGV[0]:filter.map\") or die;\n+\t\t\twhile (<MAP>) {\n+\t\t\t\tm/(.*):(.*)/ or die;\n+\t\t\t\topen F, \">../map/$1\" or die;\n+\t\t\t\tprint F \"$2\" or die;\n+\t\t\t\tclose(F) or die;\n+\t\t\t}\n+\t\t\tclose(MAP) or die;' \"$state_commit\" \\\n+\t\t\t\t|| die \"Unable to load state from $state_branch:filter.map\"\n+\telse\n+\t\techo \"Branch $state_branch does not exist. Will create\" 1>&2\n+\tfi\n+fi\n+\n # we need \"--\" only if there are no path arguments in $@\n nonrevs=$(git rev-parse --no-revs \"$@\") || exit\n if test -z \"$nonrevs\"\n@@ -590,6 +614,29 @@ test -z \"$ORIG_GIT_COMMITTER_DATE\" || {\n \texport GIT_COMMITTER_DATE\n }\n \n+if test -n \"$state_branch\"\n+then\n+\techo \"Saving rewrite state to $state_branch\" 1>&2\n+\tstate_blob=$(\n+\t\tperl -e'opendir D, \"../map\" or die;\n+\t\t\topen H, \"|-\", \"git hash-object -w --stdin\" or die;\n+\t\t\tforeach (sort readdir(D)) {\n+\t\t\t\tnext if m/^\\.\\.?$/;\n+\t\t\t\topen F, \"<../map/$_\" or die;\n+\t\t\t\tchomp($f = <F>);\n+\t\t\t\tprint H \"$_:$f\\n\" or die;\n+\t\t\t}\n+\t\t\tclose(H) or die;' || die \"Unable to save state\")\n+\tstate_tree=$(/bin/echo -e \"100644 blob $state_blob\\tfilter.map\" | git mktree)\n+\tif test -n \"$state_commit\"\n+\tthen\n+\t\tstate_commit=$(/bin/echo \"Sync\" | git commit-tree \"$state_tree\" -p \"$state_commit\")\n+\telse\n+\t\tstate_commit=$(/bin/echo \"Sync\" | git commit-tree \"$state_tree\" )\n+\tfi\n+\tgit update-ref \"$state_branch\" \"$state_commit\"\n+fi\n+\n cd \"$orig_dir\"\n rm -rf \"$tempdir\"\n \n-- \n2.11.0\n\n"},{"id":"328253","messageId":"20170917073657.31193-3-ijc@hellion.org.uk","threadId":"46768","inReplyTo":"1505633797.22447.4.camel@hellion.org.uk","subject":"[PATCH v2 3/4] filter-branch: preserve and restore $GIT_AUTHOR_* and $GIT_COMMITTER_*","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-17T07:36:56Z","receivedAt":"2017-09-17T08:17:46Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"These are modified by set_ident() but a subsequent patch would like to operate\non their original values.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\n git-filter-branch.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex e15c538f6..c88263965 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -219,6 +219,13 @@ trap 'cd \"$orig_dir\"; rm -rf \"$tempdir\"' 0\n ORIG_GIT_DIR=\"$GIT_DIR\"\n ORIG_GIT_WORK_TREE=\"$GIT_WORK_TREE\"\n ORIG_GIT_INDEX_FILE=\"$GIT_INDEX_FILE\"\n+ORIG_GIT_AUTHOR_NAME=\"$GIT_AUTHOR_NAME\"\n+ORIG_GIT_AUTHOR_EMAIL=\"$GIT_AUTHOR_EMAIL\"\n+ORIG_GIT_AUTHOR_DATE=\"$GIT_AUTHOR_DATE\"\n+ORIG_GIT_COMMITTER_NAME=\"$GIT_COMMITTER_NAME\"\n+ORIG_GIT_COMMITTER_EMAIL=\"$GIT_COMMITTER_EMAIL\"\n+ORIG_GIT_COMMITTER_DATE=\"$GIT_COMMITTER_DATE\"\n+\n GIT_WORK_TREE=.\n export GIT_DIR GIT_WORK_TREE\n \n@@ -545,6 +552,8 @@ if [ \"$filter_tag_name\" ]; then\n fi\n \n unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE\n+unset GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_AUTHOR_DATE\n+unset GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL GIT_COMMITTER_DATE\n test -z \"$ORIG_GIT_DIR\" || {\n \tGIT_DIR=\"$ORIG_GIT_DIR\" && export GIT_DIR\n }\n@@ -556,6 +565,30 @@ test -z \"$ORIG_GIT_INDEX_FILE\" || {\n \tGIT_INDEX_FILE=\"$ORIG_GIT_INDEX_FILE\" &&\n \texport GIT_INDEX_FILE\n }\n+test -z \"$ORIG_GIT_AUTHOR_NAME\" || {\n+\tGIT_AUTHOR_NAME=\"$ORIG_GIT_AUTHOR_NAME\" &&\n+\texport GIT_AUTHOR_NAME\n+}\n+test -z \"$ORIG_GIT_AUTHOR_EMAIL\" || {\n+\tGIT_AUTHOR_EMAIL=\"$ORIG_GIT_AUTHOR_EMAIL\" &&\n+\texport GIT_AUTHOR_EMAIL\n+}\n+test -z \"$ORIG_GIT_AUTHOR_DATE\" || {\n+\tGIT_AUTHOR_DATE=\"$ORIG_GIT_AUTHOR_DATE\" &&\n+\texport GIT_AUTHOR_DATE\n+}\n+test -z \"$ORIG_GIT_COMMITTER_NAME\" || {\n+\tGIT_COMMITTER_NAME=\"$ORIG_GIT_COMMITTER_NAME\" &&\n+\texport GIT_COMMITTER_NAME\n+}\n+test -z \"$ORIG_GIT_COMMITTER_EMAIL\" || {\n+\tGIT_COMMITTER_EMAIL=\"$ORIG_GIT_COMMITTER_EMAIL\" &&\n+\texport GIT_COMMITTER_EMAIL\n+}\n+test -z \"$ORIG_GIT_COMMITTER_DATE\" || {\n+\tGIT_COMMITTER_DATE=\"$ORIG_GIT_COMMITTER_DATE\" &&\n+\texport GIT_COMMITTER_DATE\n+}\n \n cd \"$orig_dir\"\n rm -rf \"$tempdir\"\n-- \n2.11.0\n\n"},{"id":"328254","messageId":"20170917073657.31193-2-ijc@hellion.org.uk","threadId":"46768","inReplyTo":"1505633797.22447.4.camel@hellion.org.uk","subject":"[PATCH v2 2/4] filter-branch: reset $GIT_* before cleaning up","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-17T07:36:55Z","receivedAt":"2017-09-17T08:17:48Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"This is pure code motion to enable a subsequent patch to add code which needs\nto happen with the reset $GIT_* but before the temporary directory has been\ncleaned up.\n\nSigned-off-by: Ian Campbell <ijc@hellion.org.uk>\n---\n git-filter-branch.sh | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 05645064a..e15c538f6 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -544,11 +544,6 @@ if [ \"$filter_tag_name\" ]; then\n \tdone\n fi\n \n-cd \"$orig_dir\"\n-rm -rf \"$tempdir\"\n-\n-trap - 0\n-\n unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE\n test -z \"$ORIG_GIT_DIR\" || {\n \tGIT_DIR=\"$ORIG_GIT_DIR\" && export GIT_DIR\n@@ -562,6 +557,11 @@ test -z \"$ORIG_GIT_INDEX_FILE\" || {\n \texport GIT_INDEX_FILE\n }\n \n+cd \"$orig_dir\"\n+rm -rf \"$tempdir\"\n+\n+trap - 0\n+\n if [ \"$(is_bare_repository)\" = false ]; then\n \tgit read-tree -u -m HEAD || exit\n fi\n-- \n2.11.0\n\n"},{"id":"328255","messageId":"1505633797.22447.4.camel@hellion.org.uk","threadId":"46768","inReplyTo":null,"subject":"[PATCH v2 0/4] filter-branch: support for incremental update + fix for ancient tag format","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-17T07:36:37Z","receivedAt":"2017-09-17T08:17:49Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"This is the second version of my patches to add incremental support to\ngit-filter-branch. Since the last time I have:\n * addressed the review feedback (see changelog embedded in final\n   patch)\n * switched to using the (newly introduced) `--allow-missing-tagger`\n   option to `git mktag` to allow early Linux kernel tags to be\n   rewritten\n * split out some of the preparatory changes to make the final patch\n   easier to read.\n\nI've force pushed to [1] where Travis seems happy and have set off the\nprocess of re-rewriting the devicetree tree from scratch (a multi-day\naffair) to validate (it's looking good).\n\nIan.\n\n[0] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/\n[1] https://github.com/ijc/git/tree/git-filter-branch\n"},{"id":"328256","messageId":"1505641403.22447.6.camel@hellion.org.uk","threadId":"46768","inReplyTo":"20170917073657.31193-4-ijc@hellion.org.uk","subject":"Re: [PATCH v2 4/4] Subject: filter-branch: stash away ref map in a branch","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-17T09:43:23Z","receivedAt":"2017-09-17T09:43:33Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"On Sun, 2017-09-17 at 08:36 +0100, Ian Campbell wrote:\n> +if test -n \"$state_branch\"\n> +then\n> > +\techo \"Saving rewrite state to $state_branch\" 1>&2\n> > +\tstate_blob=$(\n> > +\t\tperl -e'opendir D, \"../map\" or die;\n> > +\t\t\topen H, \"|-\", \"git hash-object -w --stdin\" or die;\n> > +\t\t\tforeach (sort readdir(D)) {\n> > +\t\t\t\tnext if m/^\\.\\.?$/;\n> > +\t\t\t\topen F, \"<../map/$_\" or die;\n> > +\t\t\t\tchomp($f = <F>);\n> > +\t\t\t\tprint H \"$_:$f\\n\" or die;\n> > +\t\t\t}\n> > +\t\t\tclose(H) or die;' || die \"Unable to save state\")\n\nOne things I've noticed is that for a full Linux tree history the\nfilter.map file is 50M+ which causes github to complain:\n\n    remote: warning: File filter.map is 54.40 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB\n\n(you can simulate this with `git log --pretty=format:\"%H:%H\"\nupstream/master`.) I suppose that's not a bad recommendation for any\ninfra, not just GH's.\n\nThe blob is compressed in the object store so there isn't _much_ point\nin compressing the map (also, it only goes down to ~30MB anyway so we\naren't buying all that much time), but I'm wondering if perhaps I\nshould look into a more intelligent representation, perhaps hashed by\nthe first two characters (as .git/objects is) to divide into several\nblobs and have two levels.\n\nI'm also wondering if the .git-rewrite/map directory, which will have\n70k+ (and growing) directory entries for a modern Linux tree, would\nbenefit from the same sort of thing. OTOH in this case the extra shell\nmachinations to turn abcdef123 into ab/cdef123 might overwhelm the\nsavings in directory lookup time (unless there is a helper already for\nthat. That assume that directory lookup is even a bottleneck, I've not\nmeasured but anecdotally/gut-feeling the commits-per-second does seem\nto be decreasing over the course of the filtering process.\n\nIan.\n"},{"id":"328337","messageId":"xmqqzi9rqtmn.fsf@gitster.mtv.corp.google.com","threadId":"46768","inReplyTo":"20170917073657.31193-1-ijc@hellion.org.uk","subject":"Re: [PATCH v2 1/4] mktag: add option which allows the tagger field to be omitted","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-19T03:01:20Z","receivedAt":"2017-09-19T03:01:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ian Campbell <ijc@hellion.org.uk> writes:\n\n> This can be useful e.g. in `filter-branch` when rewritting tags produced by\n> older versions of Git, such as v2.6.12-rc2..v2.6.13-rc3 in the Linux kernel\n> source tree:\n>\n>         $ git cat-file tag v2.6.12-rc2\n>         object 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2\n>         type commit\n>         tag v2.6.12-rc2\n>\n>         Linux v2.6.12-rc2 release\n>         -----BEGIN PGP SIGNATURE-----\n>         Version: GnuPG v1.2.4 (GNU/Linux)\n>\n>         iD8DBQBCbW8ZF3YsRnbiHLsRAgFRAKCq/TkuDaEombFABkPqYgGCgWN2lQCcC0qc\n>         wznDbFU45A54dZC8RZ5JxyE=\n>         =ESRP\n>         -----END PGP SIGNATURE-----\n>\n>         $ git cat-file tag v2.6.12-rc2 | git mktag\n>         error: char76: could not find \"tagger \"\n>         fatal: invalid tag signature file\n>         $ git cat-file tag v2.6.12-rc2 | git mktag --allow-missing-tagger\n>         9e734775f7c22d2f89943ad6c745571f1930105f\n>\n> To that end, pass the new option to `mktag` in `filter-branch`.\n\nHmph.  I cannot shake this nagging feeling that this is probably a\nsolution that is overly narrow to a single problem that won't scale\ninto the future.\n\nAs there is no guarantee that \"missing-tagger\" will stay to be the\nonly kind of broken tag objects we'd produce in one version, later\nwe notice our mistake and then forbid in another version, with the\napproach to add '--allow-missing-tagger' optoin would imply that\nwe'd end up adding more and more such options, and filter-branch\nwill need to use all these '--allow-this-breakage' options we would\never add.  Even though I fully agree with the problem you are trying\nto solve (i.e. we want to be able to replay an old history without\nour tool rejecting the data we have), it was my first reaction when\nI read this patch.  IOW, my first reaction was \"perhaps a single\noption '--allow-broken' to cover the currently known and any future\nshape of malformat over tag data is more appropriate\".\n\nBut then, if we look at the body of cmd_mktag(), it looks like this:\n\n\tint cmd_mktag(...)\n\t{\n\t\tread input into strbuf buf;\n\t\tcall verify_tag on buf to sanity check;\n\t\tcall write_sha1_file() the contents of buf as a tag;\n\t\treport the object name;\n\t}\n\nIf we drop the \"verification\" step from the above, that essentially\nbecomes an equivaent to \"hash-object -t tag -w --stdin\".\n\nSo I now have to wonder if it may be sufficient to use \"hash-object\"\nin filter-branch, without doing this \"allow malformed data that we\nwould not permit if the tag were being created today, only to help\nreplaying an old, already broken data\" change to \"git mktag\".\n\nIs there something that makes \"hash-object\" insufficient (like it\nstill does some extra checks we would want to disable and cannot\nwork as a replacement for your \"--allow-missing-tagger\")?\n\nThanks.\n\n> Signed-off-by: Ian Campbell <ijc@hellion.org.uk>\n> ---\n>  Documentation/git-mktag.txt |   9 +++-\n>  builtin/mktag.c             | 100 +++++++++++++++++++++++++-------------------\n>  git-filter-branch.sh        |   2 +-\n>  t/t3800-mktag.sh            |  33 ++++++++++++++-\n>  4 files changed, 98 insertions(+), 46 deletions(-)\n>\n> diff --git a/Documentation/git-mktag.txt b/Documentation/git-mktag.txt\n> index fa6a75612..c720c7419 100644\n> --- a/Documentation/git-mktag.txt\n> +++ b/Documentation/git-mktag.txt\n> @@ -9,7 +9,7 @@ git-mktag - Creates a tag object\n>  SYNOPSIS\n>  --------\n>  [verse]\n> -'git mktag'\n> +'git mktag' [--allow-missing-tagger]\n>  \n>  DESCRIPTION\n>  -----------\n> @@ -34,6 +34,13 @@ exists, is separated by a blank line from the header.  The\n>  message part may contain a signature that Git itself doesn't\n>  care about, but that can be verified with gpg.\n>  \n> +OPTIONS\n> +-------\n> +--allow-missing-tagger::\n> +\tAllow the `tagger` line in the header to be omitted. This is\n> +\trarely desirable but may be useful in recreating tags created\n> +\tby older Git.\n> +\n>  GIT\n>  ---\n>  Part of the linkgit:git[1] suite\n> diff --git a/builtin/mktag.c b/builtin/mktag.c\n> index 031b750f0..0f5dae8d5 100644\n> --- a/builtin/mktag.c\n> +++ b/builtin/mktag.c\n> @@ -1,4 +1,5 @@\n>  #include \"builtin.h\"\n> +#include \"parse-options.h\"\n>  #include \"tag.h\"\n>  \n>  /*\n> @@ -15,6 +16,8 @@\n>   * the shortest possible tagger-line.\n>   */\n>  \n> +static int allow_missing_tagger;\n> +\n>  /*\n>   * We refuse to tag something we can't verify. Just because.\n>   */\n> @@ -41,8 +44,9 @@ static int verify_tag(char *buffer, unsigned long size)\n>  \tunsigned char sha1[20];\n>  \tconst char *object, *type_line, *tag_line, *tagger_line, *lb, *rb;\n>  \tsize_t len;\n> +\tconst unsigned long min_size = allow_missing_tagger ? 71 : 84;\n>  \n> -\tif (size < 84)\n> +\tif (size < min_size)\n>  \t\treturn error(\"wanna fool me ? you obviously got the size wrong !\");\n>  \n>  \tbuffer[size] = 0;\n> @@ -98,46 +102,46 @@ static int verify_tag(char *buffer, unsigned long size)\n>  \t/* Verify the tagger line */\n>  \ttagger_line = tag_line;\n>  \n> -\tif (memcmp(tagger_line, \"tagger \", 7))\n> +\tif (!memcmp(tagger_line, \"tagger \", 7)) {\n> +\t\t/*\n> +\t\t * Check for correct form for name and email\n> +\t\t * i.e. \" <\" followed by \"> \" on _this_ line\n> +\t\t * No angle brackets within the name or email address fields.\n> +\t\t * No spaces within the email address field.\n> +\t\t */\n> +\t\ttagger_line += 7;\n> +\t\tif (!(lb = strstr(tagger_line, \" <\")) || !(rb = strstr(lb+2, \"> \")) ||\n> +\t\t\tstrpbrk(tagger_line, \"<>\\n\") != lb+1 ||\n> +\t\t\tstrpbrk(lb+2, \"><\\n \") != rb)\n> +\t\t\treturn error(\"char%\"PRIuMAX\": malformed tagger field\",\n> +\t\t\t\t(uintmax_t) (tagger_line - buffer));\n> +\n> +\t\t/* Check for author name, at least one character, space is acceptable */\n> +\t\tif (lb == tagger_line)\n> +\t\t\treturn error(\"char%\"PRIuMAX\": missing tagger name\",\n> +\t\t\t\t(uintmax_t) (tagger_line - buffer));\n> +\n> +\t\t/* timestamp, 1 or more digits followed by space */\n> +\t\ttagger_line = rb + 2;\n> +\t\tif (!(len = strspn(tagger_line, \"0123456789\")))\n> +\t\t\treturn error(\"char%\"PRIuMAX\": missing tag timestamp\",\n> +\t\t\t\t(uintmax_t) (tagger_line - buffer));\n> +\t\ttagger_line += len;\n> +\t\tif (*tagger_line != ' ')\n> +\t\t\treturn error(\"char%\"PRIuMAX\": malformed tag timestamp\",\n> +\t\t\t\t(uintmax_t) (tagger_line - buffer));\n> +\t\ttagger_line++;\n> +\n> +\t\t/* timezone, 5 digits [+-]hhmm, max. 1400 */\n> +\t\tif (!((tagger_line[0] == '+' || tagger_line[0] == '-') &&\n> +\t\t      strspn(tagger_line+1, \"0123456789\") == 4 &&\n> +\t\t      tagger_line[5] == '\\n' && atoi(tagger_line+1) <= 1400))\n> +\t\t\treturn error(\"char%\"PRIuMAX\": malformed tag timezone\",\n> +\t\t\t\t(uintmax_t) (tagger_line - buffer));\n> +\t\ttagger_line += 6;\n> +\t} else if (!allow_missing_tagger)\n>  \t\treturn error(\"char%\"PRIuMAX\": could not find \\\"tagger \\\"\",\n> -\t\t\t(uintmax_t) (tagger_line - buffer));\n> -\n> -\t/*\n> -\t * Check for correct form for name and email\n> -\t * i.e. \" <\" followed by \"> \" on _this_ line\n> -\t * No angle brackets within the name or email address fields.\n> -\t * No spaces within the email address field.\n> -\t */\n> -\ttagger_line += 7;\n> -\tif (!(lb = strstr(tagger_line, \" <\")) || !(rb = strstr(lb+2, \"> \")) ||\n> -\t\tstrpbrk(tagger_line, \"<>\\n\") != lb+1 ||\n> -\t\tstrpbrk(lb+2, \"><\\n \") != rb)\n> -\t\treturn error(\"char%\"PRIuMAX\": malformed tagger field\",\n> -\t\t\t(uintmax_t) (tagger_line - buffer));\n> -\n> -\t/* Check for author name, at least one character, space is acceptable */\n> -\tif (lb == tagger_line)\n> -\t\treturn error(\"char%\"PRIuMAX\": missing tagger name\",\n> -\t\t\t(uintmax_t) (tagger_line - buffer));\n> -\n> -\t/* timestamp, 1 or more digits followed by space */\n> -\ttagger_line = rb + 2;\n> -\tif (!(len = strspn(tagger_line, \"0123456789\")))\n> -\t\treturn error(\"char%\"PRIuMAX\": missing tag timestamp\",\n> -\t\t\t(uintmax_t) (tagger_line - buffer));\n> -\ttagger_line += len;\n> -\tif (*tagger_line != ' ')\n> -\t\treturn error(\"char%\"PRIuMAX\": malformed tag timestamp\",\n> -\t\t\t(uintmax_t) (tagger_line - buffer));\n> -\ttagger_line++;\n> -\n> -\t/* timezone, 5 digits [+-]hhmm, max. 1400 */\n> -\tif (!((tagger_line[0] == '+' || tagger_line[0] == '-') &&\n> -\t      strspn(tagger_line+1, \"0123456789\") == 4 &&\n> -\t      tagger_line[5] == '\\n' && atoi(tagger_line+1) <= 1400))\n> -\t\treturn error(\"char%\"PRIuMAX\": malformed tag timezone\",\n> -\t\t\t(uintmax_t) (tagger_line - buffer));\n> -\ttagger_line += 6;\n> +\t\t\t     (uintmax_t) (tagger_line - buffer));\n>  \n>  \t/* Verify the blank line separating the header from the body */\n>  \tif (*tagger_line != '\\n')\n> @@ -148,13 +152,25 @@ static int verify_tag(char *buffer, unsigned long size)\n>  \treturn 0;\n>  }\n>  \n> +static char const * const mktag_usage[] = {\n> +\tN_(\"git mktag [<options>]\"),\n> +\tNULL\n> +};\n> +\n> +static struct option mktag_opts[] = {\n> +\tOPT_BOOL(0, \"allow-missing-tagger\", &allow_missing_tagger, N_(\"allow the tagger field to be omitted\")),\n> +\tOPT_END(),\n> +};\n> +\n>  int cmd_mktag(int argc, const char **argv, const char *prefix)\n>  {\n>  \tstruct strbuf buf = STRBUF_INIT;\n>  \tunsigned char result_sha1[20];\n>  \n> -\tif (argc != 1)\n> -\t\tusage(\"git mktag\");\n> +\targc = parse_options(argc, argv, prefix, mktag_opts, mktag_usage, 0);\n> +\n> +\tif (argc != 0)\n> +\t\tusage_with_options(mktag_usage, mktag_opts);\n>  \n>  \tif (strbuf_read(&buf, 0, 4096) < 0) {\n>  \t\tdie_errno(\"could not read from stdin\");\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> index 3a74602ef..05645064a 100755\n> --- a/git-filter-branch.sh\n> +++ b/git-filter-branch.sh\n> @@ -530,7 +530,7 @@ if [ \"$filter_tag_name\" ]; then\n>  \t\t\t\t\t}' \\\n>  \t\t\t\t    -e '/^-----BEGIN PGP SIGNATURE-----/q' \\\n>  \t\t\t\t    -e 'p' ) |\n> -\t\t\t\tgit mktag) ||\n> +\t\t\t\tgit mktag --allow-missing-tagger) ||\n>  \t\t\t\tdie \"Could not create new tag object for $ref\"\n>  \t\t\tif git cat-file tag \"$ref\" | \\\n>  \t\t\t   sane_grep '^-----BEGIN PGP SIGNATURE-----' >/dev/null 2>&1\n> diff --git a/t/t3800-mktag.sh b/t/t3800-mktag.sh\n> index 8eb47942e..3a77a26c8 100755\n> --- a/t/t3800-mktag.sh\n> +++ b/t/t3800-mktag.sh\n> @@ -340,7 +340,36 @@ check_verify_failure 'detect invalid header entry' \\\n>  \t'^error: char124: trailing garbage in tag header$'\n>  \n>  ############################################################\n> -# 24. create valid tag\n> +# 24. missing tagger ok with --allow-missing-tagger\n> +\n> +cat >tag.sig <<EOF\n> +object $head\n> +type commit\n> +tag mytag\n> +\n> +EOF\n> +\n> +test_expect_success \\\n> +    'missing tagger with --allow-missing-tagger' \\\n> +    'git mktag --allow-missing-tagger <tag.sig >.git/refs/tags/mytag 2>message'\n> +\n> +############################################################\n> +# 25. detect invalid header entry with --allow-missing-tagger\n> +\n> +cat >tag.sig <<EOF\n> +object $head\n> +type commit\n> +tag mytag\n> +this line should not be here\n> +EOF\n> +\n> +test_expect_success \\\n> +    'detect invalid header entry with --allow-missing-tagger' \\\n> +    '( test_must_fail git mktag --allow-missing-tagger <tag.sig 2>message ) &&\n> +       grep \"^error: char70: trailing garbage in tag header$\" message'\n> +\n> +############################################################\n> +# 26. create valid tag\n>  \n>  cat >tag.sig <<EOF\n>  object $head\n> @@ -355,7 +384,7 @@ test_expect_success \\\n>      'git mktag <tag.sig >.git/refs/tags/mytag 2>message'\n>  \n>  ############################################################\n> -# 25. check mytag\n> +# 27. check mytag\n>  \n>  test_expect_success \\\n>      'check mytag' \\\n"},{"id":"328369","messageId":"1505803358.4636.4.camel@hellion.org.uk","threadId":"46768","inReplyTo":"xmqqzi9rqtmn.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 1/4] mktag: add option which allows the tagger field to be omitted","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2017-09-19T06:42:38Z","receivedAt":"2017-09-19T06:42:46Z","isPatch":true,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"On Tue, 2017-09-19 at 12:01 +0900, Junio C Hamano wrote:\n> \n> Hmph.  I cannot shake this nagging feeling that this is probably a\n> solution that is overly narrow to a single problem that won't scale\n> into the future.\n> \n> [...snip good point...]\n> \n> If we drop the \"verification\" step from the above, that essentially\n> becomes an equivaent to \"hash-object -t tag -w --stdin\".\n> \n> So I now have to wonder if it may be sufficient to use \"hash-object\"\n> in filter-branch, without doing this \"allow malformed data that we\n> would not permit if the tag were being created today, only to help\n> replaying an old, already broken data\" change to \"git mktag\".\n> \n> Is there something that makes \"hash-object\" insufficient (like it\n> still does some extra checks we would want to disable and cannot\n> work as a replacement for your \"--allow-missing-tagger\")?\n\nI've done a couple of quick tests and it looks like it will work. I'll\nrun a few more checks and repost.\n\nIan.\n"}]}