{"thread":{"id":"24323","subject":"[PATCH 0/2] non-incremental mode for fast-export","startedAt":"2010-07-07T20:46:00Z","lastAt":"2010-07-15T02:44:36Z","messageCount":5,"participants":["newren@gmail.com","Elijah Newren","Sverre Rabbelier"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"145052","messageId":"1278535562-14875-1-git-send-email-newren@gmail.com","threadId":"24323","inReplyTo":null,"subject":"[PATCH 0/2] non-incremental mode for fast-export","fromName":"","fromEmail":"newren@gmail.com","sentAt":"2010-07-07T20:46:00Z","receivedAt":"2010-07-07T20:46:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series fixes silently dropped files in uses of fast-export\ninvolving both --import-marks and limiting files by path. It also\nadds a new feature: a --full-tree option to switch from incremental to\ncomprehensive handling of files in commits. The two changes are\nrelated in that the bugfix consists of simply automatically activating\nthe new feature in a case where it is required for correct operation.\n\nThis obsoletes my previous (single) patch submission about the\n--full-tree option. Changes since that submission include splitting\nthis into two patches, automatically enabling the feature when needed\nfor correct behavior, and providing more explicit testcases that\nprobably explain the feature better.\n\nSince we're in feature freeze, it may make sense to split this patch\nseries so that the first patch can be applied now and keep the second\npatch in pu until feature freeze is over (the series is currently\nbased on maint). Since the second patch depends on the first, though,\nI thought it made sense to submit them together; let me know if you'd\nrather I submitted them in some other manner.\n\n(I'm not sure who to cc on this; other than Dscho who is out, I appear\nto be the biggest contributor to fast-export.)\n\nElijah Newren (2):\n fast-export: Fix dropping of files with --import-marks and path limiting\n fast-export: Add a --full-tree option\n\n Documentation/git-fast-export.txt | 6 +++\n builtin/fast-export.c | 11 +++++-\n t/t9350-fast-export.sh | 79 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 95 insertions(+), 1 deletions(-)\n"},{"id":"145053","messageId":"1278535562-14875-2-git-send-email-newren@gmail.com","threadId":"24323","inReplyTo":"1278535562-14875-1-git-send-email-newren@gmail.com","subject":"[PATCHv2 1/2] fast-export: Fix dropping of files with --import-marks and path limiting","fromName":"","fromEmail":"newren@gmail.com","sentAt":"2010-07-07T20:46:01Z","receivedAt":"2010-07-07T20:46:01Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSince fast-export operates by listing file changes since the (first) parent\ncommit, when using --import-marks and path limiting and using a wider list\nof paths than in previous runs, files from the new path(s) will silently be\nomitted from the result unless or until a commit which explicitly changes\nthose files. The resulting repository in such cases is broken and makes no\nsense.\n\nThis commit fixes this by having fast-export work with complete trees\ninstead of incremental changes (when both --import-marks and path limiting\nare used). It works by issuing a 'deleteall' directive with each commit and\nthen listing the full set of files that make up that commit, rather than\njust showing the list of files that have changed since the (first) parent\ncommit.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 9 ++++++++-\n t/t9350-fast-export.sh | 9 +++++++++\n 2 files changed, 17 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex c6dd71a..25d13a1 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -27,6 +27,7 @@ static enum { ABORT, VERBATIM, WARN, STRIP } signed_tag_mode = ABORT;\n static enum { ERROR, DROP, REWRITE } tag_of_filtered_mode = ABORT;\n static int fake_missing_tagger;\n static int no_data;\n+static int full_tree = 0;\n \n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -241,7 +242,8 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \t\tmessage += 2;\n \n \tif (commit->parents &&\n-\t get_object_mark(&commit->parents->item->object) != 0) {\n+\t get_object_mark(&commit->parents->item->object) != 0 &&\n+\t !full_tree) {\n \t\tparse_commit(commit->parents->item);\n \t\tdiff_tree_sha1(commit->parents->item->tree->object.sha1,\n \t\t\t commit->tree->object.sha1, \"\", &rev->diffopt);\n@@ -281,6 +283,8 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \t\ti++;\n \t}\n \n+\tif (full_tree)\n+\t\tprintf(\"deleteall\\n\");\n \tlog_tree_diff_flush(rev);\n \trev->diffopt.output_format = saved_output_format;\n \n@@ -608,6 +612,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \tif (import_filename)\n \t\timport_marks(import_filename);\n \n+\tif (import_filename && revs.prune_data)\n+\t\tfull_tree = 1;\n+\n \tget_tags_and_duplicates(&revs.pending, &extra_refs);\n \n \tif (prepare_revision_walk(&revs))\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex d43f37c..6069e1f 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -355,6 +355,15 @@ test_expect_failure 'no exact-ref revisions included' '\n \t)\n '\n \n+test_expect_success 'path limiting with import-marks does not lose unmodified files' '\n+\tgit checkout -b simple marks~2 &&\n+\tgit fast-export --export-marks=marks simple -- file > /dev/null &&\n+\techo more content >> file &&\n+\ttest_tick &&\n+\tgit commit -mnext file &&\n+\tgit fast-export --import-marks=marks simple -- file file0 | grep file0\n+'\n+\n test_expect_success 'set-up a few more tags for tag export tests' '\n \tgit checkout -f master &&\n \tHEAD_TREE=`git show -s --pretty=raw HEAD | grep tree | sed \"s/tree //\"` &&\n-- \n1.7.2.rc1.14.g19914\n"},{"id":"145054","messageId":"1278535562-14875-3-git-send-email-newren@gmail.com","threadId":"24323","inReplyTo":"1278535562-14875-1-git-send-email-newren@gmail.com","subject":"[PATCHv2 2/2] fast-export: Add a --full-tree option","fromName":"","fromEmail":"newren@gmail.com","sentAt":"2010-07-07T20:46:02Z","receivedAt":"2010-07-07T20:46:02Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis option adds symmetry with fast-import, enabling it to also work with\ncomplete trees instead of just incremental changes. It works by issuing a\n'deleteall' directive with each commit and then listing the full set of\nfiles that make up that commit, rather than just showing the list of files\nthat have changed since the (first) parent commit. Note that this\nfunctionality is automatically turned on when using --import-marks together\nwith path limiting in order to avoid dropping important but unchanged\nfiles.\n\nThis functionality is desired when using hand-written filters along with\n'fast-export | some-filter | fast-import' as it can be easier to write\n in terms of complete trees than incremental changes.\n\nWe could avoid the need to add this option by simply always turning it on.\nWhile the end result would be identical, it would slow things down slightly\nby printing many more filenames per commit which goes somewhat against the\n'fast' in 'fast-export'.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 +++\n builtin/fast-export.c | 4 ++-\n t/t9350-fast-export.sh | 70 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 79 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 98ec6b5..8a6a3cb 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -90,6 +90,12 @@ marks the same across runs.\n \tresulting stream can only be used by a repository which\n \talready contains the necessary objects.\n \n+--full-tree::\n+\tThis option will cause fast-export to issue a \"deleteall\"\n+\tdirective for each commit followed by a full list of all files\n+\tin the commit (as opposed to just listing the files which are\n+\tdifferent from the commit's first parent).\n+\n [git-rev-list-args...]::\n A list of arguments, acceptable to 'git rev-parse' and\n 'git rev-list', that specifies the specific objects and references\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 25d13a1..8c77602 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -27,7 +27,7 @@ static enum { ABORT, VERBATIM, WARN, STRIP } signed_tag_mode = ABORT;\n static enum { ERROR, DROP, REWRITE } tag_of_filtered_mode = ABORT;\n static int fake_missing_tagger;\n static int no_data;\n-static int full_tree = 0;\n+static int full_tree;\n \n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -588,6 +588,8 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\t\t \"Import marks from this file\"),\n \t\tOPT_BOOLEAN(0, \"fake-missing-tagger\", &fake_missing_tagger,\n \t\t\t \"Fake a tagger when tags lack one\"),\n+\t\tOPT_BOOLEAN(0, \"full-tree\", &full_tree,\n+\t\t\t \"Output full tree for each commit\"),\n \t\t{ OPTION_NEGBIT, 0, \"data\", &no_data, NULL,\n \t\t\t\"Skip output of blob data\",\n \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 6069e1f..b08954e 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -364,6 +364,76 @@ test_expect_success 'path limiting with import-marks does not lose unmodified fi\n \tgit fast-export --import-marks=marks simple -- file file0 | grep file0\n '\n \n+cat > expected << EOF\n+blob\n+mark :1\n+data 8\n+Wohlauf\n+\n+blob\n+mark :2\n+data 9\n+break it\n+\n+reset refs/heads/simple\n+commit refs/heads/simple\n+mark :3\n+author A U Thor 1112912053 -0700\n+committer C O Mitter 1112912053 -0700\n+data 8\n+initial\n+deleteall\n+M 100644 :1 file\n+M 100644 :2 file0\n+\n+blob\n+mark :4\n+data 9\n+die Luft\n+\n+blob\n+mark :5\n+data 12\n+geht frisch\n+\n+commit refs/heads/simple\n+mark :6\n+author A U Thor 1112912113 -0700\n+committer C O Mitter 1112912113 -0700\n+data 7\n+second\n+from :3\n+deleteall\n+M 100644 :4 file\n+M 100644 :2 file0\n+M 100644 :5 file2\n+\n+blob\n+mark :7\n+data 22\n+die Luft\n+more content\n+\n+commit refs/heads/simple\n+mark :8\n+author A U Thor 1112912773 -0700\n+committer C O Mitter 1112912773 -0700\n+data 5\n+next\n+from :6\n+deleteall\n+M 100644 :7 file\n+M 100644 :2 file0\n+M 100644 :5 file2\n+\n+EOF\n+\n+test_expect_success 'full-tree shows all files in commits' '\n+\tgit checkout -f simple &&\n+\tgit fast-export --full-tree simple > output &&\n+\ttest_cmp output expected\n+'\n+\n test_expect_success 'set-up a few more tags for tag export tests' '\n \tgit checkout -f master &&\n \tHEAD_TREE=`git show -s --pretty=raw HEAD | grep tree | sed \"s/tree //\"` &&\n-- \n1.7.2.rc1.14.g19914\n"},{"id":"145592","messageId":"AANLkTim9CU-2CNz8lUbKiKXHHICugMhts_1aIDP2_81g@mail.gmail.com","threadId":"24323","inReplyTo":"1278535562-14875-1-git-send-email-newren@gmail.com","subject":"Re: [PATCH 0/2] non-incremental mode for fast-export","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-07-15T01:03:56Z","receivedAt":"2010-07-15T01:03:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jul 7, 2010 at 2:46 PM, wrote:\n> This patch series fixes silently dropped files in uses of fast-export\n> involving both --import-marks and limiting files by path.  It also\n> adds a new feature: a --full-tree option to switch from incremental to\n> comprehensive handling of files in commits.  The two changes are\n> related in that the bugfix consists of simply automatically activating\n> the new feature in a case where it is required for correct operation.\n\nHmm..no comments. Am I (and others at my $dayjob) unique enough in\nour combination of options we use that no one else ever runs into this\nmangling of exported repositories?\n\nElijah\n"},{"id":"145593","messageId":"AANLkTilpIlOaupzzSdLWxjE3PAsZg9J9UB9_pM12H8wE@mail.gmail.com","threadId":"24323","inReplyTo":"AANLkTim9CU-2CNz8lUbKiKXHHICugMhts_1aIDP2_81g@mail.gmail.com","subject":"Re: [PATCH 0/2] non-incremental mode for fast-export","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-07-15T02:44:36Z","receivedAt":"2010-07-15T02:44:36Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Jul 14, 2010 at 20:03, Elijah Newren wrote:\n> Hmm..no comments.  Am I (and others at my $dayjob) unique enough in\n> our combination of options we use that no one else ever runs into this\n> mangling of exported repositories?\n\nThe first patch makes sense to me and should probably go into maint.\nThe second one is so trivial that even if you are the only user of it\nI can see no reason _not_ to include it. So even if I'm not exactly an\nexpert on fast-export, I think I know enough of fast-import to be able\nto say this should be merged :).\n\n-- \nCheers,\n\nSverre Rabbelier\n"}]}