git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 06/10] fast-export: when using paths, avoid corrupt stream with non-existent mark

From
Elijah Newren <newren@gmail.com>
Date
Nov 11, 2018, 06:23 UTC
Message-ID
<20181111062312.16342-7-newren@gmail.com>
In-Reply-To
<20181111062312.16342-1-newren@gmail.com>

If file paths are specified to fast-export and multiple refs point to a commit that does not touch any of the relevant file paths, then fast-export can hit problems. fast-export has a list of additional refs that it needs to explicitly set after exporting all blobs and commits, and when it tries to get_object_mark() on the relevant commit, it can get a mark of 0, i.e. "not found", because the commit in question did not touch the relevant paths and thus was not exported. Trying to import a stream with a mark corresponding to an unexported object will cause fast-import to crash.

Avoid this problem by taking the commit the ref points to and finding an ancestor of it that was exported, and make the ref point to that commit instead.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 builtin/fast-export.c  | 13 ++++++++++++-
 t/t9350-fast-export.sh | 24 ++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index a3c044b0af..5648a8ce9c 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -900,7 +900,18 @@ static void handle_tags_and_duplicates(void)
 			if (anonymize)
 				name = anonymize_refname(name);
 			/* create refs pointing to already seen commits */
-			commit = (struct commit *)object;
+			commit = rewrite_commit((struct commit *)object);
+			if (!commit) {
+				/*
+				 * Neither this object nor any of its
+				 * ancestors touch any relevant paths, so
+				 * it has been filtered to nothing.  Delete
+				 * it.
+				 */
+				printf("reset %s\nfrom %s\n\n",
+				       name, sha1_to_hex(null_sha1));
+				continue;
+			}
 			printf("reset %s\nfrom :%d\n\n", name,
 			       get_object_mark(&commit->object));
 			show_progress();
diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh
index 5bf21b4908..dbb560c110 100755
--- a/t/t9350-fast-export.sh
+++ b/t/t9350-fast-export.sh
@@ -386,6 +386,30 @@ test_expect_success 'path limiting with import-marks does not lose unmodified fi
 	grep file0 actual
 '
 
+test_expect_success 'avoid corrupt stream with non-existent mark' '
+	test_create_repo avoid_non_existent_mark &&
+	(
+		cd avoid_non_existent_mark &&
+
+		touch important-path &&
+		git add important-path &&
+		test_commit initial &&
+
+		touch ignored &&
+		git add ignored &&
+		test_commit whatever &&
+
+		git branch A &&
+		git branch B &&
+
+		echo foo >>important-path &&
+		git add important-path &&
+		test_commit more changes &&
+
+		git fast-export --all -- important-path | git fast-import --force
+	)
+'
+
 test_expect_success 'full-tree re-shows unmodified files'        '
 	git checkout -f simple &&
 	git fast-export --full-tree simple >actual &&
-- 
2.19.1.866.g82735bcbde
Previous: Jeff KingNext: Jeff King
Message 31 of 90 in “Import/Export as a fast way to purge files from Git?”
  1. Lars SchneiderSep 23, 2018
  2. Eric SunshineSep 23, 2018
  3. Lars SchneiderSep 23, 2018
  4. brian m. carlsonSep 23, 2018
  5. Jeff KingSep 23, 2018
  6. Elijah NewrenSep 24, 2018
  7. Lars SchneiderOct 31, 2018
  8. Elijah NewrenNov 1, 2018
  9. 00/10 fast export and import fixes and featuresElijah Newren, Nov 11, 2018
  10. 01/10 git-fast-import.txt: fix documentation for --quiet optionElijah Newren, Nov 11, 2018
  11. Jeff KingNov 11, 2018
  12. 02/10 git-fast-export.txt: clarify misleading documentation about rev-list argsElijah Newren, Nov 11, 2018
  13. Jeff KingNov 11, 2018
  14. Elijah NewrenNov 11, 2018
  15. Elijah NewrenNov 13, 2018
  16. Jonathan NiederNov 13, 2018
  17. Elijah NewrenNov 14, 2018
  18. 03/10 fast-export: use value from correct enumElijah Newren, Nov 11, 2018
  19. Jeff KingNov 11, 2018
  20. Ævar Arnfjörð BjarmasonNov 11, 2018
  21. Ævar Arnfjörð BjarmasonNov 12, 2018
  22. Jeff KingNov 12, 2018
  23. 04/10 fast-export: avoid dying when filtering by paths and old tags existElijah Newren, Nov 11, 2018
  24. Jeff KingNov 11, 2018
  25. Elijah NewrenNov 11, 2018
  26. Jeff KingNov 12, 2018
  27. brian m. carlsonNov 12, 2018
  28. Jeff KingNov 13, 2018
  29. 05/10 fast-export: move commit rewriting logic into a function for reuseElijah Newren, Nov 11, 2018
  30. Jeff KingNov 11, 2018
  31. 06/10 fast-export: when using paths, avoid corrupt stream with non-existent markElijah Newren, Nov 11, 2018
  32. Jeff KingNov 11, 2018
  33. Elijah NewrenNov 11, 2018
  34. Jeff KingNov 12, 2018
  35. Elijah NewrenNov 12, 2018
  36. 07/10 fast-export: ensure we export requested refsElijah Newren, Nov 11, 2018
  37. Jeff KingNov 11, 2018
  38. Elijah NewrenNov 11, 2018
  39. 08/10 fast-export: add --reference-excluded-parents optionElijah Newren, Nov 11, 2018
  40. Jeff KingNov 11, 2018
  41. 09/10 fast-export: add a --show-original-ids option to show original namesElijah Newren, Nov 11, 2018
  42. Jeff KingNov 11, 2018
  43. Elijah NewrenNov 11, 2018
  44. Jeff KingNov 12, 2018
  45. Elijah NewrenNov 12, 2018
  46. Jeff KingNov 12, 2018
  47. 10/10 fast-export: add --always-show-modify-after-renameElijah Newren, Nov 11, 2018
  48. Jeff KingNov 11, 2018
  49. Elijah NewrenNov 11, 2018
  50. Jeff KingNov 12, 2018
  51. Elijah NewrenNov 12, 2018
  52. Jeff KingNov 13, 2018
  53. Elijah NewrenNov 13, 2018
  54. Jeff KingNov 14, 2018
  55. Jeff KingNov 11, 2018
  56. Elijah NewrenNov 11, 2018
  57. Jeff KingNov 12, 2018
  58. 00/11 fast export and import fixes and featuresElijah Newren, Nov 14, 2018
  59. 07/11 fast-export: ensure we export requested refsElijah Newren, Nov 14, 2018
  60. 11/11 fast-export: add --always-show-modify-after-renameElijah Newren, Nov 14, 2018
  61. 01/11 git-fast-import.txt: fix documentation for --quiet optionElijah Newren, Nov 14, 2018
  62. 10/11 fast-export: add a --show-original-ids option to show original namesElijah Newren, Nov 14, 2018
  63. 05/11 fast-export: move commit rewriting logic into a function for reuseElijah Newren, Nov 14, 2018
  64. 09/11 fast-import: remove unmaintained duplicate documentationElijah Newren, Nov 14, 2018
  65. 04/11 fast-export: avoid dying when filtering by paths and old tags existElijah Newren, Nov 14, 2018
  66. SZEDER GáborNov 14, 2018
  67. Elijah NewrenNov 14, 2018
  68. 06/11 fast-export: when using paths, avoid corrupt stream with non-existent markElijah Newren, Nov 14, 2018
  69. 08/11 fast-export: add --reference-excluded-parents optionElijah Newren, Nov 14, 2018
  70. SZEDER GáborNov 14, 2018
  71. Elijah NewrenNov 14, 2018
  72. 03/11 fast-export: use value from correct enumElijah Newren, Nov 14, 2018
  73. 02/11 git-fast-export.txt: clarify misleading documentation about rev-list argsElijah Newren, Nov 14, 2018
  74. Jeff KingNov 14, 2018
  75. 00/11 fast export and import fixes and featuresElijah Newren, Nov 16, 2018
  76. 03/11 git-fast-export.txt: clarify misleading documentation about rev-list argsElijah Newren, Nov 16, 2018
  77. 07/11 fast-export: when using paths, avoid corrupt stream with non-existent markElijah Newren, Nov 16, 2018
  78. 01/11 fast-export: convert sha1 to oidElijah Newren, Nov 16, 2018
  79. 02/11 git-fast-import.txt: fix documentation for --quiet optionElijah Newren, Nov 16, 2018
  80. 08/11 fast-export: ensure we export requested refsElijah Newren, Nov 16, 2018
  81. 09/11 fast-export: add --reference-excluded-parents optionElijah Newren, Nov 16, 2018
  82. 10/11 fast-import: remove unmaintained duplicate documentationElijah Newren, Nov 16, 2018
  83. 06/11 fast-export: move commit rewriting logic into a function for reuseElijah Newren, Nov 16, 2018
  84. 11/11 fast-export: add a --show-original-ids option to show original namesElijah Newren, Nov 16, 2018
  85. SZEDER GáborNov 16, 2018
  86. 05/11 fast-export: avoid dying when filtering by paths and old tags existElijah Newren, Nov 16, 2018
  87. 04/11 fast-export: use value from correct enumElijah Newren, Nov 16, 2018
  88. Jeff KingNov 16, 2018
  89. Ævar Arnfjörð BjarmasonNov 12, 2018
  90. Elijah NewrenNov 12, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.