git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 07/10] fetch: implement fetch.fsck.*

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jul 27, 2018, 14:37 UTC
Message-ID
<20180727143720.14948-8-avarab@gmail.com>
In-Reply-To
<20180525192811.25680-1-avarab@gmail.com>

Implement support for fetch.fsck.* corresponding with the existing receive.fsck.*. This allows for pedantically cloning repositories with specific issues without turning off fetch.fsckObjects.

One such repository is https://github.com/robbyrussell/oh-my-zsh.git which before this change will emit this error when cloned with fetch.fsckObjects:

    error: object 2b7227859263b6aabcc28355b0b994995b7148b6: zeroPaddedFilemode: contains zero-padded file modes
    fatal: Error in object
    fatal: index-pack failed

Now with fetch.fsck.zeroPaddedFilemode=warn we'll warn about that issue, but the clone will succeed:

    warning: object 2b7227859263b6aabcc28355b0b994995b7148b6: zeroPaddedFilemode: contains zero-padded file modes
    warning: object a18c4d13c2a5fa2d4ecd5346c50e119b999b807d: zeroPaddedFilemode: contains zero-padded file modes
    warning: object 84df066176c8da3fd59b13731a86d90f4f1e5c9d: zeroPaddedFilemode: contains zero-padded file modes

The motivation for this is to be able to turn on fetch.fsckObjects globally across a fleet of computers but still be able to manually clone various legacy repositories by either white-listing specific issues, or better yet whitelist specific objects.

The use of --git-dir=* instead of -C in the tests could be considered somewhat archaic, but the tests I'm adding here are duplicating the corresponding receive.* tests with as few changes as possible.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/config.txt        | 20 +++++++++++---
 fetch-pack.c                    | 32 +++++++++++++++++++++--
 t/t5504-fetch-receive-strict.sh | 46 +++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 6 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 7ff453c53b..8dace49daa 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1467,6 +1467,16 @@ fetch.fsckObjects::
 	checked. Defaults to false. If not set, the value of
 	`transfer.fsckObjects` is used instead.
 
+fetch.fsck.<msg-id>::
+	Acts like `fsck.<msg-id>`, but is used by
+	linkgit:git-fetch-pack[1] instead of linkgit:git-fsck[1]. See
+	the `fsck.<msg-id>` documentation for details.
+
+fetch.fsck.skipList::
+	Acts like `fsck.skipList`, but is used by
+	linkgit:git-fetch-pack[1] instead of linkgit:git-fsck[1]. See
+	the `fsck.skipList` documentation for details.
+
 fetch.unpackLimit::
 	If the number of objects fetched over the Git native
 	transfer is below this
@@ -1602,10 +1612,12 @@ fsck.<msg-id>::
 	repositories containing such data.
 +
 Setting `fsck.<msg-id>` will be picked up by linkgit:git-fsck[1], but
-to accept pushes of such data set `receive.fsck.<msg-id>` instead.
+to accept pushes of such data set `receive.fsck.<msg-id>` instead, or
+to clone or fetch it set `fetch.fsck.<msg-id>`.
 +
 The rest of the documentation discusses `fsck.*` for brevity, but the
-same applies for the corresponding `receive.fsck.*` variables.
+same applies for the corresponding `receive.fsck.*` and
+`fetch.<msg-id>.*`. variables.
 +
 When `fsck.<msg-id>` is set, errors can be switched to warnings and
 vice versa by configuring the `fsck.<msg-id>` setting where the
@@ -1628,8 +1640,8 @@ fsck.skipList::
 	can be safely ignored such as invalid committer email addresses.
 	Note: corrupt objects cannot be skipped with this setting.
 +
-Like `fsck.<msg-id>` this variable has a corresponding
-`receive.fsck.skipList` variant.
+Like `fsck.<msg-id>` this variable has corresponding
+`receive.fsck.skipList` and `fetch.fsck.skipList` variants.
 
 gc.aggressiveDepth::
 	The depth parameter used in the delta compression
diff --git a/fetch-pack.c b/fetch-pack.c
index 7ccb9c0d45..aea2f6cf26 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -21,6 +21,7 @@
 #include "packfile.h"
 #include "object-store.h"
 #include "connected.h"
+#include "fsck.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;
@@ -35,6 +36,7 @@ static int agent_supported;
 static int server_supports_filtering;
 static struct lock_file shallow_lock;
 static const char *alternate_shallow_file;
+static struct strbuf fsck_msg_types = STRBUF_INIT;
 
 /* Remember to update object flag allocation in object.h */
 #define COMPLETE	(1U << 0)
@@ -937,7 +939,8 @@ static int get_pack(struct fetch_pack_args *args,
 			 */
 			argv_array_push(&cmd.args, "--fsck-objects");
 		else
-			argv_array_push(&cmd.args, "--strict");
+			argv_array_pushf(&cmd.args, "--strict%s",
+					 fsck_msg_types.buf);
 	}
 
 	cmd.in = demux.out;
@@ -1458,6 +1461,31 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	return ref;
 }
 
+static int fetch_pack_config_cb(const char *var, const char *value, void *cb)
+{
+	if (strcmp(var, "fetch.fsck.skiplist") == 0) {
+		const char *path;
+
+		if (git_config_pathname(&path, var, value))
+			return 1;
+		strbuf_addf(&fsck_msg_types, "%cskiplist=%s",
+			fsck_msg_types.len ? ',' : '=', path);
+		free((char *)path);
+		return 0;
+	}
+
+	if (skip_prefix(var, "fetch.fsck.", &var)) {
+		if (is_valid_msg_type(var, value))
+			strbuf_addf(&fsck_msg_types, "%c%s=%s",
+				fsck_msg_types.len ? ',' : '=', var, value);
+		else
+			warning("Skipping unknown msg id '%s'", var);
+		return 0;
+	}
+
+	return git_default_config(var, value, cb);
+}
+
 static void fetch_pack_config(void)
 {
 	git_config_get_int("fetch.unpacklimit", &fetch_unpack_limit);
@@ -1466,7 +1494,7 @@ static void fetch_pack_config(void)
 	git_config_get_bool("fetch.fsckobjects", &fetch_fsck_objects);
 	git_config_get_bool("transfer.fsckobjects", &transfer_fsck_objects);
 
-	git_config(git_default_config, NULL);
+	git_config(fetch_pack_config_cb, NULL);
 }
 
 static void fetch_pack_setup(void)
diff --git a/t/t5504-fetch-receive-strict.sh b/t/t5504-fetch-receive-strict.sh
index 57ff78c201..004bfebe98 100755
--- a/t/t5504-fetch-receive-strict.sh
+++ b/t/t5504-fetch-receive-strict.sh
@@ -145,6 +145,20 @@ test_expect_success 'push with receive.fsck.skipList' '
 	git push --porcelain dst bogus
 '
 
+test_expect_success 'fetch with fetch.fsck.skipList' '
+	commit="$(git hash-object -t commit -w --stdin <bogus-commit)" &&
+	refspec=refs/heads/bogus:refs/heads/bogus &&
+	git push . $commit:refs/heads/bogus &&
+	rm -rf dst &&
+	git init dst &&
+	git --git-dir=dst/.git config fetch.fsckObjects true &&
+	test_must_fail git --git-dir=dst/.git fetch "file://$(pwd)" $refspec &&
+	git --git-dir=dst/.git config fetch.fsck.skipList dst/.git/SKIP &&
+	echo $commit >dst/.git/SKIP &&
+	git --git-dir=dst/.git fetch "file://$(pwd)" $refspec
+'
+
+
 test_expect_success 'push with receive.fsck.missingEmail=warn' '
 	commit="$(git hash-object -t commit -w --stdin <bogus-commit)" &&
 	git push . $commit:refs/heads/bogus &&
@@ -163,6 +177,27 @@ test_expect_success 'push with receive.fsck.missingEmail=warn' '
 	! grep "missingEmail" act
 '
 
+test_expect_success 'fetch with fetch.fsck.missingEmail=warn' '
+	commit="$(git hash-object -t commit -w --stdin <bogus-commit)" &&
+	refspec=refs/heads/bogus:refs/heads/bogus &&
+	git push . $commit:refs/heads/bogus &&
+	rm -rf dst &&
+	git init dst &&
+	git --git-dir=dst/.git config fetch.fsckobjects true &&
+	test_must_fail git --git-dir=dst/.git fetch "file://$(pwd)" $refspec &&
+	git --git-dir=dst/.git config \
+		fetch.fsck.missingEmail warn &&
+	git --git-dir=dst/.git fetch "file://$(pwd)" $refspec >act 2>&1 &&
+	grep "missingEmail" act &&
+	rm -rf dst &&
+	git init dst &&
+	git --git-dir=dst/.git config fetch.fsckobjects true &&
+	git --git-dir=dst/.git config \
+		fetch.fsck.missingEmail ignore &&
+	git --git-dir=dst/.git fetch "file://$(pwd)" $refspec >act 2>&1 &&
+	! grep "missingEmail" act
+'
+
 test_expect_success \
 	'receive.fsck.unterminatedHeader=warn triggers error' '
 	rm -rf dst &&
@@ -174,4 +209,15 @@ test_expect_success \
 	grep "Cannot demote unterminatedheader" act
 '
 
+test_expect_success \
+	'fetch.fsck.unterminatedHeader=warn triggers error' '
+	rm -rf dst &&
+	git init dst &&
+	git --git-dir=dst/.git config fetch.fsckobjects true &&
+	git --git-dir=dst/.git config \
+		fetch.fsck.unterminatedheader warn &&
+	test_must_fail git --git-dir=dst/.git fetch "file://$(pwd)" HEAD &&
+	grep "Cannot demote unterminatedheader" act
+'
+
 test_done

-- 
2.18.0.345.g5c9ce644c3
Previous: Ævar Arnfjörð BjarmasonNext: Junio C Hamano
Message 53 of 69 in “BUG: No way to set fsck.<msg-id> when cloning”
  1. Ævar Arnfjörð BjarmasonMay 24, 2018
  2. Kevin DaudtMay 24, 2018
  3. Ævar Arnfjörð BjarmasonMay 24, 2018
  4. Jeff KingMay 24, 2018
  5. 0/4 fsck: doc fixes & fetch.fsck.* implementationÆvar Arnfjörð Bjarmason, May 24, 2018
  6. 1/4 config doc: don't describe *.fetchObjects twiceÆvar Arnfjörð Bjarmason, May 24, 2018
  7. Junio C HamanoMay 25, 2018
  8. 3/4 config doc: elaborate on what transfer.fsckObjects doesÆvar Arnfjörð Bjarmason, May 24, 2018
  9. Eric SunshineMay 24, 2018
  10. Junio C HamanoMay 25, 2018
  11. Ævar Arnfjörð BjarmasonMay 31, 2018
  12. 2/4 config doc: unify the description of fsck.* and receive.fsck.*Ævar Arnfjörð Bjarmason, May 24, 2018
  13. Eric SunshineMay 24, 2018
  14. Ævar Arnfjörð BjarmasonMay 24, 2018
  15. Eric SunshineMay 24, 2018
  16. Junio C HamanoMay 25, 2018
  17. 4/4 fetch: implement fetch.fsck.*Ævar Arnfjörð Bjarmason, May 24, 2018
  18. Junio C HamanoMay 25, 2018
  19. 0/5 fsck: doc fixes & fetch.fsck.* implementationÆvar Arnfjörð Bjarmason, May 25, 2018
  20. 1/5 config doc: don't describe *.fetchObjects twiceÆvar Arnfjörð Bjarmason, May 25, 2018
  21. Eric SunshineMay 25, 2018
  22. 2/5 config doc: unify the description of fsck.* and receive.fsck.*Ævar Arnfjörð Bjarmason, May 25, 2018
  23. Eric SunshineMay 25, 2018
  24. Junio C HamanoMay 28, 2018
  25. Ævar Arnfjörð BjarmasonMay 28, 2018
  26. Junio C HamanoMay 30, 2018
  27. Junio C HamanoMay 30, 2018
  28. Ævar Arnfjörð BjarmasonMay 31, 2018
  29. Junio C HamanoJun 1, 2018
  30. 4/5 config doc: mention future aspirations for transfer.fsckObjectsÆvar Arnfjörð Bjarmason, May 25, 2018
  31. Christian CouderMay 25, 2018
  32. 5/5 fetch: implement fetch.fsck.*Ævar Arnfjörð Bjarmason, May 25, 2018
  33. Junio C HamanoMay 30, 2018
  34. Ævar Arnfjörð BjarmasonMay 31, 2018
  35. 3/5 config doc: elaborate on what transfer.fsckObjects doesÆvar Arnfjörð Bjarmason, May 25, 2018
  36. Eric SunshineMay 25, 2018
  37. 00/10 fsck: doc fixes & fetch.fsck.* implementationÆvar Arnfjörð Bjarmason, Jul 27, 2018
  38. SZEDER GáborJul 30, 2018
  39. 01/10 receive.fsck.<msg-id> tests: remove dead codeÆvar Arnfjörð Bjarmason, Jul 27, 2018
  40. Junio C HamanoJul 27, 2018
  41. Ævar Arnfjörð BjarmasonJul 27, 2018
  42. Junio C HamanoJul 27, 2018
  43. 02/10 config doc: don't describe *.fetchObjects twiceÆvar Arnfjörð Bjarmason, Jul 27, 2018
  44. Junio C HamanoJul 27, 2018
  45. 03/10 config doc: unify the description of fsck.* and receive.fsck.*Ævar Arnfjörð Bjarmason, Jul 27, 2018
  46. Junio C HamanoJul 27, 2018
  47. 04/10 config doc: elaborate on what transfer.fsckObjects doesÆvar Arnfjörð Bjarmason, Jul 27, 2018
  48. Junio C HamanoJul 27, 2018
  49. 05/10 config doc: elaborate on fetch.fsckObjects securityÆvar Arnfjörð Bjarmason, Jul 27, 2018
  50. Junio C HamanoJul 27, 2018
  51. Ævar Arnfjörð BjarmasonJul 28, 2018
  52. 06/10 transfer.fsckObjects tests: untangle confusing setupÆvar Arnfjörð Bjarmason, Jul 27, 2018
  53. 07/10 fetch: implement fetch.fsck.*Ævar Arnfjörð Bjarmason, Jul 27, 2018
  54. Junio C HamanoJul 27, 2018
  55. Junio C HamanoJul 27, 2018
  56. Duy NguyenJul 30, 2018
  57. Ævar Arnfjörð BjarmasonJul 30, 2018
  58. 08/10 fsck: test & document {fetch,receive}.fsck.* config fallbackÆvar Arnfjörð Bjarmason, Jul 27, 2018
  59. Junio C HamanoJul 27, 2018
  60. 09/10 fsck: add stress tests for fsck.skipListÆvar Arnfjörð Bjarmason, Jul 27, 2018
  61. 10/10 fsck: test and document unknown fsck.<msg-id> valuesÆvar Arnfjörð Bjarmason, Jul 27, 2018
  62. Ævar Arnfjörð BjarmasonJul 27, 2018
  63. Junio C HamanoJul 27, 2018
  64. Ævar Arnfjörð BjarmasonJul 28, 2018
  65. Junio C HamanoJul 30, 2018
  66. Junio C HamanoMay 28, 2018
  67. Jeff KingMay 24, 2018
  68. Thomas BraunMay 24, 2018
  69. Ævar Arnfjörð BjarmasonMay 25, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.