{"thread":{"id":"55404","subject":"[PATCH 0/4] usage.c: add a non-fatal bug() + misc doc fixes","startedAt":"2021-03-28T02:27:32Z","lastAt":"2021-11-13T09:09:57Z","messageCount":245,"participants":["Ævar Arnfjörð Bjarmason","Eric Sunshine","Junio C Hamano","Jeff King","Johannes Sixt","Bagas Sanjaya","Jonathan Tan","Jonathan Nieder","Felipe Contreras","Andrei Rybak","Taylor Blau","Han Xin"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"420344","messageId":"cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com","threadId":"55404","inReplyTo":null,"subject":"[PATCH 0/4] usage.c: add a non-fatal bug() + misc doc fixes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:25:59Z","receivedAt":"2021-03-28T02:27:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix up some of the technical docs around error handling and add a\nbug() function, as explained in 4/4 this is for use in some\nfsck-related code.\n\nThis series does not make use of it, but I'll follow-up with one that\ndoes. I wanted to peel of this small cleanup series from that.\n\nI noticed some semi-related bugs in these error tracing functions to\ndo with the trace2 integration, noted in\nhttps://lore.kernel.org/git/87mtuoo4ym.fsf@evledraar.gmail.com/ this\ndoesn't attempt to fix those.\n\nÆvar Arnfjörð Bjarmason (4):\n usage.c: don't copy/paste the same comment three times\n api docs: document BUG() in api-error-handling.txt\n api docs: document that BUG() emits a trace2 error event\n usage.c: add a non-fatal bug() function to go with BUG()\n\n .../technical/api-error-handling.txt | 16 ++++++-\n Documentation/technical/api-trace2.txt | 4 +-\n git-compat-util.h | 3 ++\n run-command.c | 11 +++++\n t/helper/test-trace2.c | 14 +++++-\n t/t0210-trace2-normal.sh | 19 ++++++++\n usage.c | 46 ++++++++++++++-----\n 7 files changed, 95 insertions(+), 18 deletions(-)\n\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420345","messageId":"patch-3.5-f0e0d0daa6e-20210328T022343Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com","subject":"[PATCH 3/4] api docs: document that BUG() emits a trace2 error event","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:26:02Z","receivedAt":"2021-03-28T02:27:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Correct documentation added in e544221d97a (trace2:\nDocumentation/technical/api-trace2.txt, 2019-02-22) to state that\ncalling BUG() also emits an \"error\" event. See ee4512ed481 (trace2:\ncreate new combined trace facility, 2019-02-22) for the initial\nimplementation.\n\nThe BUG() function did not emit an event then however, that was only\nchanged later in 0a9dde4a04c (usage: trace2 BUG() invocations,\n2021-02-05), that commit changed the code, but didn't update any of\nthe docs.\n\nLet's also add a cross-reference from api-error-handling.txt.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/technical/api-error-handling.txt | 3 +++\n Documentation/technical/api-trace2.txt | 2 +-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt\nindex 71486abb2f0..8be4f4d0d6a 100644\n--- a/Documentation/technical/api-error-handling.txt\n+++ b/Documentation/technical/api-error-handling.txt\n@@ -23,6 +23,9 @@ various kinds.\n without running into too many problems. Like `error`, it\n returns -1 after reporting the situation to the caller.\n \n+These reports will be logged via the trace2 facility. See the \"error\"\n+event in link:api-trace2.txt[trace2 API].\n+\n Customizable error handlers\n ---------------------------\n \ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex c65ffafc485..3f52f981a2d 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -465,7 +465,7 @@ completed.)\n ------------\n \n `\"error\"`::\n-\tThis event is emitted when one of the `error()`, `die()`,\n+\tThis event is emitted when one of the `BUG()`, `error()`, `die()`,\n \t`warning()`, or `usage()` functions are called.\n +\n ------------\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420346","messageId":"patch-1.5-a7b329c21cf-20210328T022343Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com","subject":"[PATCH 1/4] usage.c: don't copy/paste the same comment three times","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:26:00Z","receivedAt":"2021-03-28T02:27:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In gee4512ed481 (trace2: create new combined trace facility,\n2019-02-22) we started with two copies of this comment,\n0ee10fd1296 (usage: add trace2 entry upon warning(), 2020-11-23) added\na third. Let's instead add an earlier comment that applies to all\nthese mostly-the-same functions.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n usage.c | 17 +++++------------\n 1 file changed, 5 insertions(+), 12 deletions(-)\n\ndiff --git a/usage.c b/usage.c\nindex 1b206de36d6..c7d233b0de9 100644\n--- a/usage.c\n+++ b/usage.c\n@@ -55,12 +55,13 @@ static NORETURN void usage_builtin(const char *err, va_list params)\n \texit(129);\n }\n \n+/*\n+ * We call trace2_cmd_error_va() in the below functions first and\n+ * expect it to va_copy 'params' before using it (because an 'ap' can\n+ * only be walked once).\n+ */\n static NORETURN void die_builtin(const char *err, va_list params)\n {\n-\t/*\n-\t * We call this trace2 function first and expect it to va_copy 'params'\n-\t * before using it (because an 'ap' can only be walked once).\n-\t */\n \ttrace2_cmd_error_va(err, params);\n \n \tvreportf(\"fatal: \", err, params);\n@@ -70,10 +71,6 @@ static NORETURN void die_builtin(const char *err, va_list params)\n \n static void error_builtin(const char *err, va_list params)\n {\n-\t/*\n-\t * We call this trace2 function first and expect it to va_copy 'params'\n-\t * before using it (because an 'ap' can only be walked once).\n-\t */\n \ttrace2_cmd_error_va(err, params);\n \n \tvreportf(\"error: \", err, params);\n@@ -81,10 +78,6 @@ static void error_builtin(const char *err, va_list params)\n \n static void warn_builtin(const char *warn, va_list params)\n {\n-\t/*\n-\t * We call this trace2 function first and expect it to va_copy 'params'\n-\t * before using it (because an 'ap' can only be walked once).\n-\t */\n \ttrace2_cmd_error_va(warn, params);\n \n \tvreportf(\"warning: \", warn, params);\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420347","messageId":"patch-2.5-8c8b1dfd184-20210328T022343Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com","subject":"[PATCH 2/4] api docs: document BUG() in api-error-handling.txt","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:26:01Z","receivedAt":"2021-03-28T02:27:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the BUG() function was added in d8193743e08 (usage.c: add BUG()\nfunction, 2017-05-12) these docs added in 1f23cfe0ef5 (doc: document\nerror handling functions and conventions, 2014-12-03) were not\nupdated. Let's do that.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/technical/api-error-handling.txt | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt\nindex ceeedd485c9..71486abb2f0 100644\n--- a/Documentation/technical/api-error-handling.txt\n+++ b/Documentation/technical/api-error-handling.txt\n@@ -1,8 +1,11 @@\n Error reporting in git\n ======================\n \n-`die`, `usage`, `error`, and `warning` report errors of various\n-kinds.\n+`BUG`, `die`, `usage`, `error`, and `warning` report errors of\n+various kinds.\n+\n+- `BUG` is for failed internal assertions that should never happen,\n+ i.e. a bug in git itself.\n \n - `die` is for fatal application errors. It prints a message to\n the user and exits with status 128.\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420348","messageId":"patch-4.5-515d146cac8-20210328T022343Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com","subject":"[PATCH 4/4] usage.c: add a non-fatal bug() function to go with BUG()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:26:03Z","receivedAt":"2021-03-28T02:27:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a bug() function that works like error() except the message is\nprefixed with \"bug:\" instead of \"error:\".\n\nThe reason this is needed is for e.g. the fsck code. If we encounter\nwhat we'd consider a BUG() in the middle of fsck traversal we'd still\nlike to try as hard as possible to go past that object and complete\nthe fsck, instead of hard dying. A follow-up commit will introduce\nsuch a use in object-file.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .../technical/api-error-handling.txt | 8 ++++-\n Documentation/technical/api-trace2.txt | 4 +--\n git-compat-util.h | 3 ++\n run-command.c | 11 +++++++\n t/helper/test-trace2.c | 14 +++++++--\n t/t0210-trace2-normal.sh | 19 ++++++++++++\n usage.c | 29 +++++++++++++++++++\n 7 files changed, 83 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt\nindex 8be4f4d0d6a..9d6ac6f6649 100644\n--- a/Documentation/technical/api-error-handling.txt\n+++ b/Documentation/technical/api-error-handling.txt\n@@ -1,7 +1,7 @@\n Error reporting in git\n ======================\n \n-`BUG`, `die`, `usage`, `error`, and `warning` report errors of\n+`BUG`, `bug`, `die`, `usage`, `error`, and `warning` report errors of\n various kinds.\n \n - `BUG` is for failed internal assertions that should never happen,\n@@ -18,6 +18,12 @@ various kinds.\n to the user and returns -1 for convenience in signaling the error\n to the caller.\n \n+- `bug` (lower-case, not `BUG`) is supposed to be used like `BUG` but\n+ has the same non-fatal semantics as `error`. It's meant to signal an\n+ internal bug in a library whose caller might still want to attempt\n+ some amount of graceful recovery, or to append other error output of\n+ their own.\n+\n - `warning` is for reporting situations that probably should not\n occur but which the user (and Git) can continue to work around\n without running into too many problems. Like `error`, it\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 3f52f981a2d..cafe373f405 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -465,8 +465,8 @@ completed.)\n ------------\n \n `\"error\"`::\n-\tThis event is emitted when one of the `BUG()`, `error()`, `die()`,\n-\t`warning()`, or `usage()` functions are called.\n+\tThis event is emitted when one of the `BUG()`, `bug()`, `error()`,\n+\t`die()`, `warning()`, or `usage()` functions are called.\n +\n ------------\n {\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 9ddf9d7044b..13c1dcf9dcc 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -463,6 +463,7 @@ NORETURN void usage(const char *err);\n NORETURN void usagef(const char *err, ...) __attribute__((format (printf, 1, 2)));\n NORETURN void die(const char *err, ...) __attribute__((format (printf, 1, 2)));\n NORETURN void die_errno(const char *err, ...) __attribute__((format (printf, 1, 2)));\n+int bug(const char *err, ...) __attribute__((format (printf, 1, 2)));\n int error(const char *err, ...) __attribute__((format (printf, 1, 2)));\n int error_errno(const char *err, ...) __attribute__((format (printf, 1, 2)));\n void warning(const char *err, ...) __attribute__((format (printf, 1, 2)));\n@@ -497,6 +498,8 @@ static inline int const_error(void)\n typedef void (*report_fn)(const char *, va_list params);\n \n void set_die_routine(NORETURN_PTR report_fn routine);\n+void set_bug_routine(report_fn routine);\n+report_fn get_bug_routine(void);\n void set_error_routine(report_fn routine);\n report_fn get_error_routine(void);\n void set_warn_routine(report_fn routine);\ndiff --git a/run-command.c b/run-command.c\nindex be6bc128cd9..8b818b063ff 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -348,6 +348,12 @@ static void fake_fatal(const char *err, va_list params)\n \tvreportf(\"fatal: \", err, params);\n }\n \n+static void child_bug_fn(const char *err, va_list params)\n+{\n+\tconst char msg[] = \"bug() should not be called in child\\n\";\n+\txwrite(2, msg, sizeof(msg) - 1);\n+}\n+\n static void child_error_fn(const char *err, va_list params)\n {\n \tconst char msg[] = \"error() should not be called in child\\n\";\n@@ -371,9 +377,12 @@ static void NORETURN child_die_fn(const char *err, va_list params)\n static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n {\n \tstatic void (*old_errfn)(const char *err, va_list params);\n+\tstatic void (*old_bugfn)(const char *err, va_list params);\n \n \told_errfn = get_error_routine();\n \tset_error_routine(fake_fatal);\n+\told_bugfn = get_bug_routine();\n+\tset_bug_routine(fake_fatal);\n \terrno = cerr->syserr;\n \n \tswitch (cerr->err) {\n@@ -399,6 +408,7 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)\n \t\terror_errno(\"cannot exec '%s'\", cmd->argv[0]);\n \t\tbreak;\n \t}\n+\tset_bug_routine(old_bugfn);\n \tset_error_routine(old_errfn);\n }\n \n@@ -789,6 +799,7 @@ int start_command(struct child_process *cmd)\n \t\t * called, they can take stdio locks and malloc.\n \t\t */\n \t\tset_die_routine(child_die_fn);\n+\t\tset_error_routine(child_bug_fn);\n \t\tset_error_routine(child_error_fn);\n \t\tset_warn_routine(child_warn_fn);\n \ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex f93633f895a..6248427e4bf 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -198,7 +198,7 @@ static int ut_006data(int argc, const char **argv)\n \treturn 0;\n }\n \n-static int ut_007bug(int argc, const char **argv)\n+static int ut_007BUG(int argc, const char **argv)\n {\n \t/*\n \t * Exercise BUG() to ensure that the message is printed to trace2.\n@@ -206,6 +206,15 @@ static int ut_007bug(int argc, const char **argv)\n \tBUG(\"the bug message\");\n }\n \n+static int ut_008bug(int argc, const char **argv)\n+{\n+\t/*\n+\t * Exercise BUG() to ensure that the message is printed to trace2.\n+\t */\n+\tbug(\"the bug message\");\n+\treturn 0;\n+}\n+\n /*\n * Usage:\n * test-tool trace2 \n@@ -222,7 +231,8 @@ static struct unit_test ut_table[] = {\n \t{ ut_004child, \"004child\", \"[]\" },\n \t{ ut_005exec, \"005exec\", \"\" },\n \t{ ut_006data, \"006data\", \"[ ]+\" },\n-\t{ ut_007bug, \"007bug\", \"\" },\n+\t{ ut_007BUG, \"007bug\", \"\" },\n+\t{ ut_008bug, \"008bug\", \"\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0210-trace2-normal.sh b/t/t0210-trace2-normal.sh\nindex 0cf3a63b75b..9c866af971f 100755\n--- a/t/t0210-trace2-normal.sh\n+++ b/t/t0210-trace2-normal.sh\n@@ -166,6 +166,25 @@ test_expect_success 'BUG messages are written to trace2' '\n \ttest_cmp expect actual\n '\n \n+# Verb 008bug\n+#\n+# Check that BUG writes to trace2\n+\n+test_expect_success 'bug messages are written to trace2' '\n+\ttest_when_finished \"rm trace.normal actual expect\" &&\n+\tGIT_TRACE2=\"$(pwd)/trace.normal\" test-tool trace2 008bug &&\n+\tperl \"$TEST_DIRECTORY/t0210/scrub_normal.perl\" actual &&\n+\tcat >expect <<-EOF &&\n+\t\tversion $V\n+\t\tstart _EXE_ trace2 008bug\n+\t\tcmd_name trace2 (trace2)\n+\t\terror the bug message\n+\t\texit elapsed:_TIME_ code:0\n+\t\tatexit elapsed:_TIME_ code:0\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n sane_unset GIT_TRACE2_BRIEF\n \n # Now test without environment variables and get all Trace2 settings\ndiff --git a/usage.c b/usage.c\nindex c7d233b0de9..34bd3abf048 100644\n--- a/usage.c\n+++ b/usage.c\n@@ -69,6 +69,13 @@ static NORETURN void die_builtin(const char *err, va_list params)\n \texit(128);\n }\n \n+static void bug_builtin(const char *err, va_list params)\n+{\n+\ttrace2_cmd_error_va(err, params);\n+\n+\tvreportf(\"bug: \", err, params);\n+}\n+\n static void error_builtin(const char *err, va_list params)\n {\n \ttrace2_cmd_error_va(err, params);\n@@ -109,6 +116,7 @@ static int die_is_recursing_builtin(void)\n * (ugh), so keep things static. */\n static NORETURN_PTR report_fn usage_routine = usage_builtin;\n static NORETURN_PTR report_fn die_routine = die_builtin;\n+static report_fn bug_routine = bug_builtin;\n static report_fn error_routine = error_builtin;\n static report_fn warn_routine = warn_builtin;\n static int (*die_is_recursing)(void) = die_is_recursing_builtin;\n@@ -118,11 +126,22 @@ void set_die_routine(NORETURN_PTR report_fn routine)\n \tdie_routine = routine;\n }\n \n+\n+void set_bug_routine(report_fn routine)\n+{\n+\tbug_routine = routine;\n+}\n+\n void set_error_routine(report_fn routine)\n {\n \terror_routine = routine;\n }\n \n+report_fn get_bug_routine(void)\n+{\n+\treturn bug_routine;\n+}\n+\n report_fn get_error_routine(void)\n {\n \treturn error_routine;\n@@ -223,6 +242,16 @@ int error_errno(const char *fmt, ...)\n \treturn -1;\n }\n \n+int bug(const char *err, ...)\n+{\n+\tva_list params;\n+\n+\tva_start(params, err);\n+\tbug_routine(err, params);\n+\tva_end(params);\n+\treturn -1;\n+}\n+\n #undef error\n int error(const char *err, ...)\n {\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420349","messageId":"CAPig+cQxhTVG_sG-LxSc4ZKEQDZTK5HZEXaHM-1G_12aGutJFQ@mail.gmail.com","threadId":"55404","inReplyTo":"patch-1.5-a7b329c21cf-20210328T022343Z-avarab@gmail.com","subject":"Re: [PATCH 1/4] usage.c: don't copy/paste the same comment three times","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-03-28T02:32:52Z","receivedAt":"2021-03-28T02:41:49Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Mar 27, 2021 at 10:28 PM Ævar Arnfjörð Bjarmason\n wrote:\n> In gee4512ed481 (trace2: create new combined trace facility,\n\ns/gee/ee/\n\n> 2019-02-22) we started with two copies of this comment,\n> 0ee10fd1296 (usage: add trace2 entry upon warning(), 2020-11-23) added\n> a third. Let's instead add an earlier comment that applies to all\n> these mostly-the-same functions.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n"},{"id":"420350","messageId":"patch-2.6-3e547289408-20210328T025618Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH 2/5] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:58:30Z","receivedAt":"2021-03-28T03:00:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe should instead simply use something like this test_create_repo\npattern. It's both less verbose, and makes things easier to debug as a\nfailing test can have their state left behind under -d without\ndamaging the state for other tests.\n\nBut let's punt on that general refactoring and just change this one\ntest, I'm going to change it further in subsequent commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..1563b35f88c 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,22 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n-\n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+test_expect_success 'object with hash mismatch' '\n+\ttest_create_repo hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420352","messageId":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","threadId":"55404","inReplyTo":"patch-4.5-515d146cac8-20210328T022343Z-avarab@gmail.com","subject":"[PATCH 0/5] fsck: improve error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:58:28Z","receivedAt":"2021-03-28T03:00:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting in a rather obscure edge case, and\nfixes up some object APIs along the way as needed.\n\nThis is based on my series to add a bug() function:\nhttps://lore.kernel.org/git/cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com/\n\nThe use of the new bug() function is in 5/5.\n\nÆvar Arnfjörð Bjarmason (5):\n cache.h: move object functions to object-store.h\n fsck tests: refactor one test to use a sub-repo\n fsck: don't hard die on invalid object types\n fsck: improve the error on invalid object types\n fsck: improve error on loose object hash mismatch\n\n builtin/cat-file.c | 7 +++--\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 ++++++++++++++---\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 10 ------\n object-file.c | 73 +++++++++++++++++++++++--------------------\n object-store.h | 19 +++++++++--\n object.c | 4 +--\n pack-check.c | 3 +-\n streaming.c | 5 ++-\n t/t1450-fsck.sh | 64 +++++++++++++++++++++++++++----------\n 12 files changed, 143 insertions(+), 77 deletions(-)\n\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420351","messageId":"patch-1.6-f8f00db8d31-20210328T025618Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH 1/5] cache.h: move object functions to object-store.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:58:29Z","receivedAt":"2021-03-28T03:00:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of some ancient object functions added in\ne.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n2005-06-01) from cache.h to object-store.h. This continues work\nstarted in cbd53a2193d (object-store: move object access functions to\nobject-store.h, 2018-05-15).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 10 ----------\n object-store.h | 9 +++++++++\n 2 files changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 6fda8091f11..0e387f84f67 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1279,16 +1279,6 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n-\n-int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n-\n-int finalize_object_file(const char *tmpfile, const char *filename);\n-\n-/* Helper to check and \"touch\" a file */\n-int check_and_freshen_file(const char *fn, int freshen);\n \n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb5..9117115a50c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,4 +477,13 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz);\n+int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int check_object_signature(struct repository *r, const struct object_id *oid,\n+\t\t\t void *buf, unsigned long size, const char *type);\n+int finalize_object_file(const char *tmpfile, const char *filename);\n+int check_and_freshen_file(const char *fn, int freshen);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420353","messageId":"patch-3.6-74654a01ba3-20210328T025618Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH 3/5] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:58:31Z","receivedAt":"2021-03-28T03:00:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change builtin/fsck.c to pass down a\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE. This changes this very ungraceful\nerror:\n\n $ git hash-object --stdin -w -t garbage --literally \n $ git fsck\n fatal: invalid object type\n $\n\nInto:\n\n $ git fsck\n error: hash mismatch for (expected )\n error: : object corrupt or missing: \n [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nBut why are we complaining about a \"hash mismatch\" for an object of a\ntype we don't know about? We shouldn't. This is the bare minimal\nchange needed to not make fsck hard die on a repository that's been\ncorrupted in this manner. In subsequent commits we'll teach fsck to\nrecognize this particular type of corruption and emit a better error\nmessage.\n\nThe parse_loose_header() function being changed here is only used in\nbuiltin/fsck.c, see f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) for its introduction.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 24 ++++++++++--------------\n object-store.h | 6 ++++--\n streaming.c | 5 ++++-\n t/t1450-fsck.sh | 9 +++++++++\n 5 files changed, 29 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 821e7798c70..d92c530863d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -602,7 +602,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tif (read_loose_object(path, oid, &type, &size, &contents,\n+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\ndiff --git a/object-file.c b/object-file.c\nindex 624af408cdc..26560a6281c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1294,8 +1294,9 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr,\n+\t\t struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1355,14 +1356,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1417,10 +1410,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2497,13 +2490,16 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2518,7 +2514,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, oi_flags);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/object-store.h b/object-store.h\nindex 9117115a50c..ab86c8bf32c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,7 +245,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n@@ -480,7 +481,8 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 800f07a52cc..e5d4dd2f654 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -341,6 +341,9 @@ static struct stream_vtbl loose_vtbl = {\n \n static open_method_decl(loose)\n {\n+\tstruct object_info oi2 = OBJECT_INFO_INIT;\n+\toi2.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -349,7 +352,7 @@ static open_method_decl(loose)\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi2, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 1563b35f88c..025dd1b491a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,4 +863,13 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck error and recovery on invalid object type' '\n+\ttest_create_repo garbage-type &&\n+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob out 2>err &&\n+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"dangling blob $empty_blob\" out\n+'\n+\n test_done\n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420354","messageId":"patch-4.6-d23fb5cd039-20210328T025618Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH 4/5] fsck: improve the error on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:58:32Z","receivedAt":"2021-03-28T03:00:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the work in the preceding commit and improve the error on:\n\n $ git hash-object --stdin -w -t garbage --literally (expected )\n error: : object corrupt or missing: \n [ other fsck output ]\n\nTo instead emit:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nThe complaint about a \"hash mismatch\" was simply an emergent property\nof how we'd fall though from read_loose_object() into fsck_loose()\nwhen we didn't get the data we expected. Now we'll correctly note that\nthe object type is invalid.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/cat-file.c | 7 +++++--\n builtin/fsck.c | 22 ++++++++++++++++++----\n object-file.c | 31 +++++++++++++++----------------\n object-store.h | 4 ++--\n t/t1450-fsck.sh | 23 ++++++++++++++++++++++-\n 5 files changed, 62 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5ebf13359e8..1063576a982 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -74,6 +74,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned flags = OBJECT_INFO_LOOKUP_REPLACE;\n \tconst char *path = force_path;\n+\tint ret;\n \n \tif (unknown_type)\n \t\tflags |= OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n@@ -92,7 +93,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tswitch (opt) {\n \tcase 't':\n \t\toi.type_name = &sb;\n-\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n+\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n+\t\tif (!unknown_type && ret < 0)\n \t\t\tdie(\"git cat-file: could not get object info\");\n \t\tif (sb.len) {\n \t\t\tprintf(\"%s\\n\", sb.buf);\n@@ -103,7 +105,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \n \tcase 's':\n \t\toi.sizep = &size;\n-\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n+\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n+\t\tif (!unknown_type && ret < 0)\n \t\t\tdie(\"git cat-file: could not get object info\");\n \t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)size);\n \t\treturn 0;\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d92c530863d..c8ab14d1545 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -601,12 +601,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n-\n-\tif (read_loose_object(path, oid, &type, &size, &contents,\n-\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n-\t\terrors_found |= ERROR_OBJECT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n+\tstruct object_info oi;\n+\tint found = 0;\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\n+\t}\n+\tif (type < 0) {\n+\t\tfound |= ERROR_OBJECT;\n+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t oid_to_hex(oid), sb.buf, path);\n+\t}\n+\tif (found) {\n+\t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 26560a6281c..e744a06637b 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1323,9 +1323,7 @@ int parse_loose_header(const char *hdr,\n \t * we're obtaining the type using '--allow-unknown-type'\n \t * option.\n \t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n+\tif (type < 0 && !(flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE))\n \t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n@@ -1407,14 +1405,17 @@ static int loose_object_info(struct repository *r,\n \t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n+\tif (status < 0) {\n \t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t} else {\n+\t\tstatus = parse_loose_header(hdr, oi, flags);\n+\t\tif (status < 0 && !(flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE))\n+\t\t\terror(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t}\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -2488,9 +2489,8 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n@@ -2498,8 +2498,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\toi.sizep = size;\n+\tenum object_type *type = oi->typep;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2514,9 +2514,9 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, oi_flags);\n-\tif (*type < 0) {\n-\t\terror(_(\"unable to parse header of %s\"), path);\n+\t*type = parse_loose_header(hdr, oi, oi_flags);\n+\tif (*type < 0 && !(oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n+\t\terror(_(\"unable to parse header %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n@@ -2532,8 +2532,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex ab86c8bf32c..786c5c34704 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -241,11 +241,11 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n *\n * Returns 0 on success, negative on error (details may be written to stderr).\n */\n+struct object_info;\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 025dd1b491a..214278e134a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\ttest_create_repo hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n@@ -868,7 +887,9 @@ test_expect_success 'fsck error and recovery on invalid object type' '\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob out 2>err &&\n-\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n+\tgrep error: err >err.errors &&\n+\ttest_line_count = 1 err.errors &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420355","messageId":"patch-5.6-bcec536b0f6-20210328T025618Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH 5/5] fsck: improve error on loose object hash mismatch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-28T02:58:33Z","receivedAt":"2021-03-28T03:00:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 13 +++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n object-file.c | 28 +++++++++++++++++++---------\n object-store.h | 4 +++-\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1450-fsck.sh | 8 +++++---\n 9 files changed, 44 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 85a76e0ef8b..bf0e266d83a 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex c8ab14d1545..365b9124bdc 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -604,20 +604,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n+\tstruct object_id real_oid = null_oid;\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n \t\tfound |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n+\t\tif (!oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n \t}\n \tif (type < 0) {\n \t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t oid_to_hex(oid), sb.buf, path);\n+\t\t oid_to_hex(&real_oid), sb.buf, path);\n \t}\n \tif (found) {\n \t\terrors_found |= ERROR_OBJECT;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 21899687e2c..93044e9e618 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1420,7 +1420,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex 41a399a69e4..cfecbcd664e 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -65,7 +65,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/object-file.c b/object-file.c\nindex e744a06637b..7aa80701aa7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -993,9 +993,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1003,8 +1005,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1029,9 +1031,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_fn(real_oid.hash, &c);\n+\tr->hash_algo->final_fn(real_oid->hash, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2489,6 +2491,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n@@ -2532,9 +2535,16 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n+\t\t\tif (oideq(real_oid, &null_oid))\n+\t\t\t\t/*\n+\t\t\t\t * Not a plain BUG() because if it\n+\t\t\t\t * does happen we're in the middle of\n+\t\t\t\t * an fsck we'd like to see to the\n+\t\t\t\t * end.\n+\t\t\t\t */\n+\t\t\t\tbug(\"BUG trying to compute hash for object at %s (expected %s)\",\n+\t\t\t\t path, oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex 786c5c34704..340b0f51f08 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -244,6 +244,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n struct object_info;\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n@@ -484,7 +485,8 @@ int unpack_loose_header(git_zstream *stream, unsigned char *map,\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n int finalize_object_file(const char *tmpfile, const char *filename);\n int check_and_freshen_file(const char *fn, int freshen);\n \ndiff --git a/object.c b/object.c\nindex 78343781ae7..1cb4b30acd7 100644\n--- a/object.c\n+++ b/object.c\n@@ -262,7 +262,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -273,7 +273,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex 4b089fe8ec0..e6aa4442c90 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 214278e134a..c7b084364b7 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -53,6 +53,7 @@ test_expect_success 'object with hash mismatch' '\n \t(\n \t\tcd hash-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -62,7 +63,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -71,6 +72,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t(\n \t\tcd hash-type-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -80,8 +82,8 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.31.1.445.g91d8e479b0a\n\n"},{"id":"420362","messageId":"xmqqh7kvolgn.fsf@gitster.g","threadId":"55404","inReplyTo":"patch-4.5-515d146cac8-20210328T022343Z-avarab@gmail.com","subject":"Re: [PATCH 4/4] usage.c: add a non-fatal bug() function to go with BUG()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-28T06:12:40Z","receivedAt":"2021-03-28T06:17:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Add a bug() function that works like error() except the message is\n> prefixed with \"bug:\" instead of \"error:\".\n>\n> The reason this is needed is for e.g. the fsck code. If we encounter\n> what we'd consider a BUG() in the middle of fsck traversal we'd still\n> like to try as hard as possible to go past that object and complete\n> the fsck, instead of hard dying. A follow-up commit will introduce\n> such a use in object-file.c.\n\nReading the description above, i.e. \"to go past that object\", the\nassumed use case seems to be to deal with a data error, not a\nprogram bug (which is where we use BUG()---e.g. one helper function\nin the fsck code detected that the caller wasn't careful enough to\nvet the data it has and called it with incoherent data). If we find\na tree entry whose mode bits implies that the object recorded in the\nentry ought to be a blob, and later find out that the object turns\nout to be a tree, that is a corrupt repository and the code that\ndetected is not buggy (and we shouldn't use BUG(), of course).\n\nSo, ... I am skeptical. If the code is prepared to handle breakage,\nwe would not want to die, but then I am not sure why it has to be\ndifferent from error().\n"},{"id":"420369","messageId":"YGAtkW2YWNbcAn8m@coredump.intra.peff.net","threadId":"55404","inReplyTo":"xmqqh7kvolgn.fsf@gitster.g","subject":"Re: [PATCH 4/4] usage.c: add a non-fatal bug() function to go with BUG()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-28T07:17:37Z","receivedAt":"2021-03-28T07:18:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 27, 2021 at 11:12:40PM -0700, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n> \n> > Add a bug() function that works like error() except the message is\n> > prefixed with \"bug:\" instead of \"error:\".\n> >\n> > The reason this is needed is for e.g. the fsck code. If we encounter\n> > what we'd consider a BUG() in the middle of fsck traversal we'd still\n> > like to try as hard as possible to go past that object and complete\n> > the fsck, instead of hard dying. A follow-up commit will introduce\n> > such a use in object-file.c.\n> \n> Reading the description above, i.e. \"to go past that object\", the\n> assumed use case seems to be to deal with a data error, not a\n> program bug (which is where we use BUG()---e.g. one helper function\n> in the fsck code detected that the caller wasn't careful enough to\n> vet the data it has and called it with incoherent data). If we find\n> a tree entry whose mode bits implies that the object recorded in the\n> entry ought to be a blob, and later find out that the object turns\n> out to be a tree, that is a corrupt repository and the code that\n> detected is not buggy (and we shouldn't use BUG(), of course).\n> \n> So, ... I am skeptical. If the code is prepared to handle breakage,\n> we would not want to die, but then I am not sure why it has to be\n> different from error().\n\nYeah, this seems like it is missing the point of BUG() completely. I\ntook a peek at patch 5/5 of the follow-on, which uses bug(). It looks\nlike it should really be an error() return or similar. The root cause\nwould be open_istream() on a loose object failing (which might be\ncorruption, or might even be a transient OS error!).\n\n-Peff\n"},{"id":"420374","messageId":"20ba35a4-6510-1a48-2ce6-5022e96926bd@kdbg.org","threadId":"55404","inReplyTo":"patch-4.6-d23fb5cd039-20210328T025618Z-avarab@gmail.com","subject":"Re: [PATCH 4/5] fsck: improve the error on invalid object types","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-03-28T08:56:24Z","receivedAt":"2021-03-28T08:57:30Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"A note on the subject line: we are all pretty sure that you want to\nimprove the code, not disimprove it. Therefore, the word \"improve\"\ncarries not meaning and only takes away space that could be used better.\nPerhaps\n\n fsck: report invalid types recorded in objects\n\nor something. Ditto for 5/5.\n\n-- Hannes\n"},{"id":"420432","messageId":"7bbcebcd-002b-0c40-2415-d29cb7a6e3ec@gmail.com","threadId":"55404","inReplyTo":"patch-2.5-8c8b1dfd184-20210328T022343Z-avarab@gmail.com","subject":"Re: [PATCH 2/4] api docs: document BUG() in api-error-handling.txt","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-03-29T05:37:12Z","receivedAt":"2021-03-29T05:46:24Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 28/03/21 09.26, Ævar Arnfjörð Bjarmason wrote:\n> -`die`, `usage`, `error`, and `warning` report errors of various\n> -kinds.\n> +`BUG`, `die`, `usage`, `error`, and `warning` report errors of\n> +various kinds.\n> +\n> +- `BUG` is for failed internal assertions that should never happen,\n> + i.e. a bug in git itself.\n> \n> - `die` is for fatal application errors. It prints a message to\n> the user and exits with status 128.\n> \nDocumentation looks OK.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"420466","messageId":"87tuoum6ru.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YGAtkW2YWNbcAn8m@coredump.intra.peff.net","subject":"Re: [PATCH 4/4] usage.c: add a non-fatal bug() function to go with BUG()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-29T13:25:09Z","receivedAt":"2021-03-29T13:26:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Mar 28 2021, Jeff King wrote:\n\n> On Sat, Mar 27, 2021 at 11:12:40PM -0700, Junio C Hamano wrote:\n>\n>> Ævar Arnfjörð Bjarmason writes:\n>> \n>> > Add a bug() function that works like error() except the message is\n>> > prefixed with \"bug:\" instead of \"error:\".\n>> >\n>> > The reason this is needed is for e.g. the fsck code. If we encounter\n>> > what we'd consider a BUG() in the middle of fsck traversal we'd still\n>> > like to try as hard as possible to go past that object and complete\n>> > the fsck, instead of hard dying. A follow-up commit will introduce\n>> > such a use in object-file.c.\n>> \n>> Reading the description above, i.e. \"to go past that object\", the\n>> assumed use case seems to be to deal with a data error, not a\n>> program bug (which is where we use BUG()---e.g. one helper function\n>> in the fsck code detected that the caller wasn't careful enough to\n>> vet the data it has and called it with incoherent data). If we find\n>> a tree entry whose mode bits implies that the object recorded in the\n>> entry ought to be a blob, and later find out that the object turns\n>> out to be a tree, that is a corrupt repository and the code that\n>> detected is not buggy (and we shouldn't use BUG(), of course).\n>> \n>> So, ... I am skeptical. If the code is prepared to handle breakage,\n>> we would not want to die, but then I am not sure why it has to be\n>> different from error().\n>\n> Yeah, this seems like it is missing the point of BUG() completely. I\n> took a peek at patch 5/5 of the follow-on, which uses bug(). It looks\n> like it should really be an error() return or similar. The root cause\n> would be open_istream() on a loose object failing (which might be\n> corruption, or might even be a transient OS error!).\n\nI don't feel strongly about this bug() thing, I'll drop it if you two\ndon't like it.\n\nBut that's not why I added it, yes you can now carefully read the code\nand reason that this code is unreachable now, as I think it is.\n\nBut it may not stay that way, refactoring how we handle I/O errors\netc. further down the stack is the sort of thing that if this bug()\nwasn't there would cause us to otherwise silently lose the\nerror. I.e. does check_object_signature() always promise to return\nnon-zero *only* if the signature isn't OK?\n\nSo maybe we are happy to just make that promise, in which case yes, this\nshould/could be an error() in this case.\n\nBut isn't this also useful for multi-threaded code? E.g. let's say fsck\nlearns to map-reduce its fsck-ing of objects across threads. One of them\nencounters a BUG(). Do we want to hard kill the whole thing or try to\nlimp ahead and report partial results from the other thread(s)?\n\nWe have than now with pack-objects/grep, but I'm struggling to find a\nuse-case for a partial grep result if e.g. PCRE fails with a BUG(...)\n...\n\n"},{"id":"420700","messageId":"YGRXomWwRYPdXZi3@coredump.intra.peff.net","threadId":"55404","inReplyTo":"87tuoum6ru.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/4] usage.c: add a non-fatal bug() function to go with BUG()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-31T11:06:10Z","receivedAt":"2021-03-31T11:07:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 29, 2021 at 03:25:09PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > Yeah, this seems like it is missing the point of BUG() completely. I\n> > took a peek at patch 5/5 of the follow-on, which uses bug(). It looks\n> > like it should really be an error() return or similar. The root cause\n> > would be open_istream() on a loose object failing (which might be\n> > corruption, or might even be a transient OS error!).\n> \n> I don't feel strongly about this bug() thing, I'll drop it if you two\n> don't like it.\n> \n> But that's not why I added it, yes you can now carefully read the code\n> and reason that this code is unreachable now, as I think it is.\n> \n> But it may not stay that way, refactoring how we handle I/O errors\n> etc. further down the stack is the sort of thing that if this bug()\n> wasn't there would cause us to otherwise silently lose the\n> error. I.e. does check_object_signature() always promise to return\n> non-zero *only* if the signature isn't OK?\n> \n> So maybe we are happy to just make that promise, in which case yes, this\n> should/could be an error() in this case.\n\nI didn't dig into what check_object_signature() promises, but I don't\nthink it matters for my argument. If the case you are looking at can be\ntriggered by bad on-disk data, transient OS errors, etc, then it should\nbe an error() or a die(), or whatever is appropriate for the code. If it\nis meant to be an invariant of the code that it should never trigger,\nthen it should be a BUG(), so that we loudly inform people that the\ncode's assumption has been violated.\n\nBut I do not see any point in a bug() that does not abort(). The point\nof BUG() is that nobody is supposed to see it, and we should be as loud\nas possible if we do.\n\nAnd if there is a call site that is in doubt about what it may be fed,\nthen it should just be an error() or die().\n\n> But isn't this also useful for multi-threaded code? E.g. let's say fsck\n> learns to map-reduce its fsck-ing of objects across threads. One of them\n> encounters a BUG(). Do we want to hard kill the whole thing or try to\n> limp ahead and report partial results from the other thread(s)?\n\nYes, we want to hard kill. The point of BUG() is that it is not supposed\nto happen, and there is no point in limping further.\n\n-Peff\n"},{"id":"421854","messageId":"patch-3.3-982f72345f1-20210413T090603Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.3-00000000000-20210413T090603Z-avarab@gmail.com","subject":"[PATCH v2 3/3] api docs: document that BUG() emits a trace2 error event","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:08:21Z","receivedAt":"2021-04-13T09:08:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Correct documentation added in e544221d97a (trace2:\nDocumentation/technical/api-trace2.txt, 2019-02-22) to state that\ncalling BUG() also emits an \"error\" event. See ee4512ed481 (trace2:\ncreate new combined trace facility, 2019-02-22) for the initial\nimplementation.\n\nThe BUG() function did not emit an event then however, that was only\nchanged later in 0a9dde4a04c (usage: trace2 BUG() invocations,\n2021-02-05), that commit changed the code, but didn't update any of\nthe docs.\n\nLet's also add a cross-reference from api-error-handling.txt.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/technical/api-error-handling.txt | 3 +++\n Documentation/technical/api-trace2.txt | 2 +-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt\nindex 71486abb2f0..8be4f4d0d6a 100644\n--- a/Documentation/technical/api-error-handling.txt\n+++ b/Documentation/technical/api-error-handling.txt\n@@ -23,6 +23,9 @@ various kinds.\n without running into too many problems. Like `error`, it\n returns -1 after reporting the situation to the caller.\n \n+These reports will be logged via the trace2 facility. See the \"error\"\n+event in link:api-trace2.txt[trace2 API].\n+\n Customizable error handlers\n ---------------------------\n \ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex c65ffafc485..3f52f981a2d 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -465,7 +465,7 @@ completed.)\n ------------\n \n `\"error\"`::\n-\tThis event is emitted when one of the `error()`, `die()`,\n+\tThis event is emitted when one of the `BUG()`, `error()`, `die()`,\n \t`warning()`, or `usage()` functions are called.\n +\n ------------\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421855","messageId":"cover-0.3-00000000000-20210413T090603Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.5-00000000000-20210328T022343Z-avarab@gmail.com","subject":"[PATCH v2 0/3] trace2 docs: note that BUG() sends an \"error\" event","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:08:18Z","receivedAt":"2021-04-13T09:08:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A trivial update to the trace2 docs to fix an omission\nwith \"BUG()\" not being listed alongside error(), die() etc.\n\nv1 of this[1] added a non-fatal-but-logging bug() function, per the\ndiscussion on v1 that's now gone.\n\n1. http://lore.kernel.org/git/cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com\n\nÆvar Arnfjörð Bjarmason (3):\n usage.c: don't copy/paste the same comment three times\n api docs: document BUG() in api-error-handling.txt\n api docs: document that BUG() emits a trace2 error event\n\n Documentation/technical/api-error-handling.txt | 10 ++++++++--\n Documentation/technical/api-trace2.txt | 2 +-\n usage.c | 17 +++++------------\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\nRange-diff against v1:\n1: a7b329c21cf ! 1: 2e4665b625b usage.c: don't copy/paste the same comment three times\n @@ Metadata\n ## Commit message ##\n usage.c: don't copy/paste the same comment three times\n \n - In gee4512ed481 (trace2: create new combined trace facility,\n + In ee4512ed481 (trace2: create new combined trace facility,\n 2019-02-22) we started with two copies of this comment,\n 0ee10fd1296 (usage: add trace2 entry upon warning(), 2020-11-23) added\n a third. Let's instead add an earlier comment that applies to all\n2: 8c8b1dfd184 = 2: ce78c79c9ac api docs: document BUG() in api-error-handling.txt\n3: f0e0d0daa6e = 3: 982f72345f1 api docs: document that BUG() emits a trace2 error event\n4: 515d146cac8 < -: ----------- usage.c: add a non-fatal bug() function to go with BUG()\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421856","messageId":"patch-1.3-2e4665b625b-20210413T090603Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.3-00000000000-20210413T090603Z-avarab@gmail.com","subject":"[PATCH v2 1/3] usage.c: don't copy/paste the same comment three times","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:08:19Z","receivedAt":"2021-04-13T09:08:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In ee4512ed481 (trace2: create new combined trace facility,\n2019-02-22) we started with two copies of this comment,\n0ee10fd1296 (usage: add trace2 entry upon warning(), 2020-11-23) added\na third. Let's instead add an earlier comment that applies to all\nthese mostly-the-same functions.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n usage.c | 17 +++++------------\n 1 file changed, 5 insertions(+), 12 deletions(-)\n\ndiff --git a/usage.c b/usage.c\nindex 1b206de36d6..c7d233b0de9 100644\n--- a/usage.c\n+++ b/usage.c\n@@ -55,12 +55,13 @@ static NORETURN void usage_builtin(const char *err, va_list params)\n \texit(129);\n }\n \n+/*\n+ * We call trace2_cmd_error_va() in the below functions first and\n+ * expect it to va_copy 'params' before using it (because an 'ap' can\n+ * only be walked once).\n+ */\n static NORETURN void die_builtin(const char *err, va_list params)\n {\n-\t/*\n-\t * We call this trace2 function first and expect it to va_copy 'params'\n-\t * before using it (because an 'ap' can only be walked once).\n-\t */\n \ttrace2_cmd_error_va(err, params);\n \n \tvreportf(\"fatal: \", err, params);\n@@ -70,10 +71,6 @@ static NORETURN void die_builtin(const char *err, va_list params)\n \n static void error_builtin(const char *err, va_list params)\n {\n-\t/*\n-\t * We call this trace2 function first and expect it to va_copy 'params'\n-\t * before using it (because an 'ap' can only be walked once).\n-\t */\n \ttrace2_cmd_error_va(err, params);\n \n \tvreportf(\"error: \", err, params);\n@@ -81,10 +78,6 @@ static void error_builtin(const char *err, va_list params)\n \n static void warn_builtin(const char *warn, va_list params)\n {\n-\t/*\n-\t * We call this trace2 function first and expect it to va_copy 'params'\n-\t * before using it (because an 'ap' can only be walked once).\n-\t */\n \ttrace2_cmd_error_va(warn, params);\n \n \tvreportf(\"warning: \", warn, params);\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421857","messageId":"patch-2.3-ce78c79c9ac-20210413T090603Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.3-00000000000-20210413T090603Z-avarab@gmail.com","subject":"[PATCH v2 2/3] api docs: document BUG() in api-error-handling.txt","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:08:20Z","receivedAt":"2021-04-13T09:08:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the BUG() function was added in d8193743e08 (usage.c: add BUG()\nfunction, 2017-05-12) these docs added in 1f23cfe0ef5 (doc: document\nerror handling functions and conventions, 2014-12-03) were not\nupdated. Let's do that.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/technical/api-error-handling.txt | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt\nindex ceeedd485c9..71486abb2f0 100644\n--- a/Documentation/technical/api-error-handling.txt\n+++ b/Documentation/technical/api-error-handling.txt\n@@ -1,8 +1,11 @@\n Error reporting in git\n ======================\n \n-`die`, `usage`, `error`, and `warning` report errors of various\n-kinds.\n+`BUG`, `die`, `usage`, `error`, and `warning` report errors of\n+various kinds.\n+\n+- `BUG` is for failed internal assertions that should never happen,\n+ i.e. a bug in git itself.\n \n - `die` is for fatal application errors. It prints a message to\n the user and exits with status 128.\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421858","messageId":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH v2 0/6] fsck: better \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:03Z","receivedAt":"2021-04-13T09:43:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of improved error reporting for fsck-ing bad loose\nobjects. See [1] for v1.\n\nThis is no longer based on the series to add a bug() function, since\nas noted in the re-roll of that[2] that function is gone. This version\nuses a plain BUG() for that condition.\n\nOther than that the only change is improved commit messages, and I\nadded a trivial patch to move read_loose_object() around in\nobject-store.h so I wouldn't need a forward declaration, and updated\nthe comment for that function.\n\n1. https://lore.kernel.org/git/cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/cover-0.3-00000000000-20210413T090603Z-avarab@gmail.com\n\n\nÆvar Arnfjörð Bjarmason (6):\n cache.h: move object functions to object-store.h\n fsck tests: refactor one test to use a sub-repo\n fsck: don't hard die on invalid object types\n object-store.h: move read_loose_object() below 'struct object_info'\n fsck: report invalid types recorded in objects\n fsck: report invalid object type-path combinations\n\n builtin/cat-file.c | 7 +++--\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 +++++++++++++++---\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 10 -------\n object-file.c | 66 +++++++++++++++++++++----------------------\n object-store.h | 39 ++++++++++++++++---------\n object.c | 4 +--\n pack-check.c | 3 +-\n streaming.c | 5 +++-\n t/t1450-fsck.sh | 64 ++++++++++++++++++++++++++++++-----------\n 12 files changed, 146 insertions(+), 87 deletions(-)\n\nRange-diff against v1:\n1: f8f00db8d31 = 1: 37c323a2410 cache.h: move object functions to object-store.h\n2: 3e547289408 = 2: 5a2cd6cca9c fsck tests: refactor one test to use a sub-repo\n3: 74654a01ba3 = 3: d0d9cb33315 fsck: don't hard die on invalid object types\n-: ----------- > 4: 81fffefcf99 object-store.h: move read_loose_object() below 'struct object_info'\n4: d23fb5cd039 ! 5: 5fb6ac4faee fsck: improve the error on invalid object types\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - fsck: improve the error on invalid object types\n + fsck: report invalid types recorded in objects\n \n Continue the work in the preceding commit and improve the error on:\n \n @@ object-file.c: int read_loose_object(const char *path,\n \t\t\tfree(*contents);\n \n ## object-store.h ##\n -@@ object-store.h: int force_object_loose(const struct object_id *oid, time_t mtime);\n +@@ object-store.h: int oid_object_info_extended(struct repository *r,\n + \n + /*\n + * Open the loose object at path, check its hash, and return the contents,\n ++ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n + * type, and size. If the object is a blob, then \"contents\" may return NULL,\n + * to allow streaming of large blobs.\n *\n - * Returns 0 on success, negative on error (details may be written to stderr).\n +@@ object-store.h: int oid_object_info_extended(struct repository *r,\n */\n -+struct object_info;\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n -\t\t enum object_type *type,\n @@ object-store.h: int force_object_loose(const struct object_id *oid, time_t mtime\n +\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n - /* Retry packed storage after checking packed and loose storage */\n + /*\n \n ## t/t1450-fsck.sh ##\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n5: bcec536b0f6 ! 6: 226d2031bcf fsck: improve error on loose object hash mismatch\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n + fsck: report invalid object type-path combinations\n +\n fsck: improve error on loose object hash mismatch\n \n Improve the error that's emitted in cases where we find a loose object\n @@ object-file.c: int read_loose_object(const char *path,\n -\t\t\t oid_to_hex(expected_oid));\n +\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n +\t\t\tif (oideq(real_oid, &null_oid))\n -+\t\t\t\t/*\n -+\t\t\t\t * Not a plain BUG() because if it\n -+\t\t\t\t * does happen we're in the middle of\n -+\t\t\t\t * an fsck we'd like to see to the\n -+\t\t\t\t * end.\n -+\t\t\t\t */\n -+\t\t\t\tbug(\"BUG trying to compute hash for object at %s (expected %s)\",\n -+\t\t\t\t path, oid_to_hex(expected_oid));\n ++\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\n \n ## object-store.h ##\n -@@ object-store.h: int force_object_loose(const struct object_id *oid, time_t mtime);\n - struct object_info;\n +@@ object-store.h: int oid_object_info_extended(struct repository *r,\n + */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n +\t\t struct object_id *real_oid,\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421859","messageId":"patch-1.6-37c323a2410-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","subject":"[PATCH v2 1/6] cache.h: move object functions to object-store.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:04Z","receivedAt":"2021-04-13T09:43:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of some ancient object functions added in\ne.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n2005-06-01) from cache.h to object-store.h. This continues work\nstarted in cbd53a2193d (object-store: move object access functions to\nobject-store.h, 2018-05-15).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 10 ----------\n object-store.h | 9 +++++++++\n 2 files changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 148d9ab5f18..f2e7fc615ba 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1279,16 +1279,6 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n-\n-int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n-\n-int finalize_object_file(const char *tmpfile, const char *filename);\n-\n-/* Helper to check and \"touch\" a file */\n-int check_and_freshen_file(const char *fn, int freshen);\n \n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb5..9117115a50c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,4 +477,13 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz);\n+int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int check_object_signature(struct repository *r, const struct object_id *oid,\n+\t\t\t void *buf, unsigned long size, const char *type);\n+int finalize_object_file(const char *tmpfile, const char *filename);\n+int check_and_freshen_file(const char *fn, int freshen);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421860","messageId":"patch-2.6-5a2cd6cca9c-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","subject":"[PATCH v2 2/6] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:05Z","receivedAt":"2021-04-13T09:43:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe should instead simply use something like this test_create_repo\npattern. It's both less verbose, and makes things easier to debug as a\nfailing test can have their state left behind under -d without\ndamaging the state for other tests.\n\nBut let's punt on that general refactoring and just change this one\ntest, I'm going to change it further in subsequent commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..1563b35f88c 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,22 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n-\n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+test_expect_success 'object with hash mismatch' '\n+\ttest_create_repo hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421861","messageId":"patch-4.6-81fffefcf99-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","subject":"[PATCH v2 4/6] object-store.h: move read_loose_object() below 'struct object_info'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:07Z","receivedAt":"2021-04-13T09:43:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the definition of read_loose_object() below \"struct\nobject_info\". In the next commit we'll add a \"struct object_info *\"\nparameter to it, moving it will avoid a forward declaration of the\nstruct.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-store.h | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex ab86c8bf32c..4680dc68ee4 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -234,20 +234,6 @@ int pretend_object_file(void *, unsigned long, enum object_type,\n \n int force_object_loose(const struct object_id *oid, time_t mtime);\n \n-/*\n- * Open the loose object at path, check its hash, and return the contents,\n- * type, and size. If the object is a blob, then \"contents\" may return NULL,\n- * to allow streaming of large blobs.\n- *\n- * Returns 0 on success, negative on error (details may be written to stderr).\n- */\n-int read_loose_object(const char *path,\n-\t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents,\n-\t\t unsigned int oi_flags);\n-\n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n \n@@ -388,6 +374,20 @@ int oid_object_info_extended(struct repository *r,\n \t\t\t const struct object_id *,\n \t\t\t struct object_info *, unsigned flags);\n \n+/*\n+ * Open the loose object at path, check its hash, and return the contents,\n+ * type, and size. If the object is a blob, then \"contents\" may return NULL,\n+ * to allow streaming of large blobs.\n+ *\n+ * Returns 0 on success, negative on error (details may be written to stderr).\n+ */\n+int read_loose_object(const char *path,\n+\t\t const struct object_id *expected_oid,\n+\t\t enum object_type *type,\n+\t\t unsigned long *size,\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n+\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421862","messageId":"patch-5.6-5fb6ac4faee-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","subject":"[PATCH v2 5/6] fsck: report invalid types recorded in objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:08Z","receivedAt":"2021-04-13T09:43:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the work in the preceding commit and improve the error on:\n\n $ git hash-object --stdin -w -t garbage --literally (expected )\n error: : object corrupt or missing: \n [ other fsck output ]\n\nTo instead emit:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nThe complaint about a \"hash mismatch\" was simply an emergent property\nof how we'd fall though from read_loose_object() into fsck_loose()\nwhen we didn't get the data we expected. Now we'll correctly note that\nthe object type is invalid.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/cat-file.c | 7 +++++--\n builtin/fsck.c | 22 ++++++++++++++++++----\n object-file.c | 31 +++++++++++++++----------------\n object-store.h | 4 ++--\n t/t1450-fsck.sh | 23 ++++++++++++++++++++++-\n 5 files changed, 62 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 5ebf13359e8..1063576a982 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -74,6 +74,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned flags = OBJECT_INFO_LOOKUP_REPLACE;\n \tconst char *path = force_path;\n+\tint ret;\n \n \tif (unknown_type)\n \t\tflags |= OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n@@ -92,7 +93,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tswitch (opt) {\n \tcase 't':\n \t\toi.type_name = &sb;\n-\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n+\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n+\t\tif (!unknown_type && ret < 0)\n \t\t\tdie(\"git cat-file: could not get object info\");\n \t\tif (sb.len) {\n \t\t\tprintf(\"%s\\n\", sb.buf);\n@@ -103,7 +105,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \n \tcase 's':\n \t\toi.sizep = &size;\n-\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n+\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n+\t\tif (!unknown_type && ret < 0)\n \t\t\tdie(\"git cat-file: could not get object info\");\n \t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)size);\n \t\treturn 0;\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 686f7cdfea0..878191e53ca 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -599,12 +599,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n-\n-\tif (read_loose_object(path, oid, &type, &size, &contents,\n-\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n-\t\terrors_found |= ERROR_OBJECT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n+\tstruct object_info oi;\n+\tint found = 0;\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\n+\t}\n+\tif (type < 0) {\n+\t\tfound |= ERROR_OBJECT;\n+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t oid_to_hex(oid), sb.buf, path);\n+\t}\n+\tif (found) {\n+\t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 26560a6281c..e744a06637b 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1323,9 +1323,7 @@ int parse_loose_header(const char *hdr,\n \t * we're obtaining the type using '--allow-unknown-type'\n \t * option.\n \t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n+\tif (type < 0 && !(flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE))\n \t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n@@ -1407,14 +1405,17 @@ static int loose_object_info(struct repository *r,\n \t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n+\tif (status < 0) {\n \t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t} else {\n+\t\tstatus = parse_loose_header(hdr, oi, flags);\n+\t\tif (status < 0 && !(flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE))\n+\t\t\terror(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t}\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -2488,9 +2489,8 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n@@ -2498,8 +2498,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\toi.sizep = size;\n+\tenum object_type *type = oi->typep;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2514,9 +2514,9 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, oi_flags);\n-\tif (*type < 0) {\n-\t\terror(_(\"unable to parse header of %s\"), path);\n+\t*type = parse_loose_header(hdr, oi, oi_flags);\n+\tif (*type < 0 && !(oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n+\t\terror(_(\"unable to parse header %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n@@ -2532,8 +2532,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex 4680dc68ee4..3d88b8a7cd3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -376,6 +376,7 @@ int oid_object_info_extended(struct repository *r,\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -383,9 +384,8 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n /*\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 025dd1b491a..214278e134a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\ttest_create_repo hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n@@ -868,7 +887,9 @@ test_expect_success 'fsck error and recovery on invalid object type' '\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob out 2>err &&\n-\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n+\tgrep error: err >err.errors &&\n+\ttest_line_count = 1 err.errors &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421863","messageId":"patch-3.6-d0d9cb33315-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","subject":"[PATCH v2 3/6] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:06Z","receivedAt":"2021-04-13T09:44:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change builtin/fsck.c to pass down a\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE. This changes this very ungraceful\nerror:\n\n $ git hash-object --stdin -w -t garbage --literally \n $ git fsck\n fatal: invalid object type\n $\n\nInto:\n\n $ git fsck\n error: hash mismatch for (expected )\n error: : object corrupt or missing: \n [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nBut why are we complaining about a \"hash mismatch\" for an object of a\ntype we don't know about? We shouldn't. This is the bare minimal\nchange needed to not make fsck hard die on a repository that's been\ncorrupted in this manner. In subsequent commits we'll teach fsck to\nrecognize this particular type of corruption and emit a better error\nmessage.\n\nThe parse_loose_header() function being changed here is only used in\nbuiltin/fsck.c, see f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) for its introduction.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 24 ++++++++++--------------\n object-store.h | 6 ++++--\n streaming.c | 5 ++++-\n t/t1450-fsck.sh | 9 +++++++++\n 5 files changed, 29 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 70ff95837ae..686f7cdfea0 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,7 +600,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tif (read_loose_object(path, oid, &type, &size, &contents,\n+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\ndiff --git a/object-file.c b/object-file.c\nindex 624af408cdc..26560a6281c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1294,8 +1294,9 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr,\n+\t\t struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1355,14 +1356,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1417,10 +1410,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2497,13 +2490,16 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2518,7 +2514,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, oi_flags);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/object-store.h b/object-store.h\nindex 9117115a50c..ab86c8bf32c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,7 +245,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n@@ -480,7 +481,8 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 800f07a52cc..e5d4dd2f654 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -341,6 +341,9 @@ static struct stream_vtbl loose_vtbl = {\n \n static open_method_decl(loose)\n {\n+\tstruct object_info oi2 = OBJECT_INFO_INIT;\n+\toi2.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -349,7 +352,7 @@ static open_method_decl(loose)\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi2, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 1563b35f88c..025dd1b491a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,4 +863,13 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck error and recovery on invalid object type' '\n+\ttest_create_repo garbage-type &&\n+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob out 2>err &&\n+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"dangling blob $empty_blob\" out\n+'\n+\n test_done\n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"421864","messageId":"patch-6.6-226d2031bcf-20210413T093734Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com","subject":"[PATCH v2 6/6] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:43:09Z","receivedAt":"2021-04-13T09:44:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"fsck: improve error on loose object hash mismatch\n\nImprove the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 13 +++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n object-file.c | 21 ++++++++++++---------\n object-store.h | 4 +++-\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1450-fsck.sh | 8 +++++---\n 9 files changed, 37 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 85a76e0ef8b..bf0e266d83a 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 878191e53ca..7713d992960 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -602,20 +602,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n+\tstruct object_id real_oid = null_oid;\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n \t\tfound |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n+\t\tif (!oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n \t}\n \tif (type < 0) {\n \t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t oid_to_hex(oid), sb.buf, path);\n+\t\t oid_to_hex(&real_oid), sb.buf, path);\n \t}\n \tif (found) {\n \t\terrors_found |= ERROR_OBJECT;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 15507b5cff0..d5fd81ebf39 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1421,7 +1421,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/object-file.c b/object-file.c\nindex e744a06637b..ca54e76fda2 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -993,9 +993,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1003,8 +1005,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1029,9 +1031,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_fn(real_oid.hash, &c);\n+\tr->hash_algo->final_fn(real_oid->hash, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2489,6 +2491,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n@@ -2532,9 +2535,9 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n+\t\t\tif (oideq(real_oid, &null_oid))\n+\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex 3d88b8a7cd3..c9c4d211de3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -384,6 +384,7 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n@@ -484,7 +485,8 @@ int unpack_loose_header(git_zstream *stream, unsigned char *map,\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n int finalize_object_file(const char *tmpfile, const char *filename);\n int check_and_freshen_file(const char *fn, int freshen);\n \ndiff --git a/object.c b/object.c\nindex 78343781ae7..1cb4b30acd7 100644\n--- a/object.c\n+++ b/object.c\n@@ -262,7 +262,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -273,7 +273,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex 4b089fe8ec0..e6aa4442c90 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 214278e134a..c7b084364b7 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -53,6 +53,7 @@ test_expect_success 'object with hash mismatch' '\n \t(\n \t\tcd hash-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -62,7 +63,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -71,6 +72,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t(\n \t\tcd hash-type-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -80,8 +82,8 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.31.1.645.g989d83ea6a6\n\n"},{"id":"422020","messageId":"YHgOwB1Pp2Z+LIPh@coredump.intra.peff.net","threadId":"55404","inReplyTo":"patch-2.3-ce78c79c9ac-20210413T090603Z-avarab@gmail.com","subject":"Re: [PATCH v2 2/3] api docs: document BUG() in api-error-handling.txt","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-15T10:00:32Z","receivedAt":"2021-04-15T10:00:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 11:08:20AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> When the BUG() function was added in d8193743e08 (usage.c: add BUG()\n> function, 2017-05-12) these docs added in 1f23cfe0ef5 (doc: document\n> error handling functions and conventions, 2014-12-03) were not\n> updated. Let's do that.\n\nWow, I had no idea this file even existed (most of the time I looked at\ntechnical/api-* the contents were along the lines of \"somebody should\nwrite this\").\n\nIMHO this is more evidence that this stuff should just go into header\nfiles, where people are more likely to see and update it.\n\n> diff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt\n> index ceeedd485c9..71486abb2f0 100644\n> --- a/Documentation/technical/api-error-handling.txt\n> +++ b/Documentation/technical/api-error-handling.txt\n> @@ -1,8 +1,11 @@\n> Error reporting in git\n> ======================\n> \n> -`die`, `usage`, `error`, and `warning` report errors of various\n> -kinds.\n> +`BUG`, `die`, `usage`, `error`, and `warning` report errors of\n> +various kinds.\n> +\n> +- `BUG` is for failed internal assertions that should never happen,\n> + i.e. a bug in git itself.\n\nYour change looks obviously correct, of course.\n\n-Peff\n"},{"id":"422021","messageId":"YHgQ5kjqQKb1zZpu@coredump.intra.peff.net","threadId":"55404","inReplyTo":"patch-1.3-2e4665b625b-20210413T090603Z-avarab@gmail.com","subject":"Re: [PATCH v2 1/3] usage.c: don't copy/paste the same comment three times","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-15T10:09:42Z","receivedAt":"2021-04-15T10:09:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 11:08:19AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> In ee4512ed481 (trace2: create new combined trace facility,\n> 2019-02-22) we started with two copies of this comment,\n> 0ee10fd1296 (usage: add trace2 entry upon warning(), 2020-11-23) added\n> a third. Let's instead add an earlier comment that applies to all\n> these mostly-the-same functions.\n\nI'm sometimes wary of this aggregating comments like this, because\nsomebody who is just reading the third function may not think to look\nfurther up to find the comment.\n\nBut this comment in particular does not seem dangerous if somebody\nmisses it (unlike comments that are warning people about bad things\nhappening).\n\n> + */\n> static NORETURN void die_builtin(const char *err, va_list params)\n> {\n> -\t/*\n> -\t * We call this trace2 function first and expect it to va_copy 'params'\n> -\t * before using it (because an 'ap' can only be walked once).\n> -\t */\n> \ttrace2_cmd_error_va(err, params);\n> \n> \tvreportf(\"fatal: \", err, params);\n\nTBH, I am not sure it adds all that much value in the first place. It is\nonly telling the reader that the code is not broken. But I kind of\nwonder if it should simply be doing the defensive thing anyway:\n\n va_list cp;\n va_copy(cp, params);\n trace2_cmd_error_va(err, params);\n va_end(cp);\n\nWe are relying on a subtle contract with the trace2 code, and there is\nno compile-time check that it will be upheld (and indeed, on many\nplatforms we might not even notice if it isn't, depending on how va_list\nis implemented). Looking at the trace2 code, we do not even enforce this\ncentrally. It is up to each target to remember to va_copy()!\n\nAnyway, that is somewhat outside the scope of your series (though I\nwould not be sad to see this comment go away entirely in favor of the\nmore defensive code above).\n\n-Peff\n"},{"id":"422022","messageId":"YHgQ+usXZseTclLT@coredump.intra.peff.net","threadId":"55404","inReplyTo":"cover-0.3-00000000000-20210413T090603Z-avarab@gmail.com","subject":"Re: [PATCH v2 0/3] trace2 docs: note that BUG() sends an \"error\" event","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-15T10:10:02Z","receivedAt":"2021-04-15T10:10:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 11:08:18AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> A trivial update to the trace2 docs to fix an omission\n> with \"BUG()\" not being listed alongside error(), die() etc.\n> \n> v1 of this[1] added a non-fatal-but-logging bug() function, per the\n> discussion on v1 that's now gone.\n\nAll three look reasonable to me. I left a few comments, but you may want\nto just ignore them. ;)\n\n-Peff\n"},{"id":"422754","messageId":"YILZHiuUyj0mt958@coredump.intra.peff.net","threadId":"55404","inReplyTo":"patch-3.6-d0d9cb33315-20210413T093734Z-avarab@gmail.com","subject":"Re: [PATCH v2 3/6] fsck: don't hard die on invalid object types","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-23T14:26:38Z","receivedAt":"2021-04-23T14:26:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 11:43:06AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Change builtin/fsck.c to pass down a\n> OBJECT_INFO_ALLOW_UNKNOWN_TYPE. This changes this very ungraceful\n> error:\n> \n> $ git hash-object --stdin -w -t garbage --literally \n> $ git fsck\n> fatal: invalid object type\n> $\n> \n> Into:\n> \n> $ git fsck\n> error: hash mismatch for (expected )\n> error: : object corrupt or missing: \n> [ the rest of the fsck output here, i.e. it didn't hard die ]\n> \n> We'll still exit with non-zero, but now we'll finish the rest of the\n> traversal. The tests that's being added here asserts that we'll still\n> complain about other fsck issues (e.g. an unrelated dangling blob).\n> \n> But why are we complaining about a \"hash mismatch\" for an object of a\n> type we don't know about? We shouldn't. This is the bare minimal\n> change needed to not make fsck hard die on a repository that's been\n> corrupted in this manner. In subsequent commits we'll teach fsck to\n> recognize this particular type of corruption and emit a better error\n> message.\n\nOK. The overall goal makes sense.\n\n> The parse_loose_header() function being changed here is only used in\n> builtin/fsck.c, see f6371f92104 (sha1_file: add read_loose_object()\n> function, 2017-01-13) for its introduction.\n\nThis left me scratching my head for a long time. Did you mean\nread_loose_object() in the beginning of the sentence?\n\n> -static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n> -\t\t\t\t unsigned int flags)\n> +int parse_loose_header(const char *hdr,\n> +\t\t struct object_info *oi,\n> +\t\t unsigned int flags)\n\nSo we are getting rid of the \"extended\" form and just making the\nnon-extended way take an OI. That seems kind of orthogonal...\n\n> --- a/streaming.c\n> +++ b/streaming.c\n> @@ -341,6 +341,9 @@ static struct stream_vtbl loose_vtbl = {\n> \n> static open_method_decl(loose)\n> {\n> +\tstruct object_info oi2 = OBJECT_INFO_INIT;\n> +\toi2.sizep = &st->size;\n> +\n\n...and is what leads us to having to touch this otherwise unrelated\nfunction.\n\nI don't mind _too_ much getting rid of a helper function that would have\nonly one caller remaining (though \"oi2\" is a bit mysterious here). But\nit seems like the patch would have been a lot easier to understand if\nthat were separately done (and explained). AFAICT the functional change\nhere is just passing the flag to read_loose_object(), which could be\ncalling parse_loose_header_extended(). I guess that would have to become\npublic, but that seems reasonable.\n\n-Peff\n"},{"id":"422755","messageId":"YILZTDzMoOVgy6El@coredump.intra.peff.net","threadId":"55404","inReplyTo":"patch-4.6-81fffefcf99-20210413T093734Z-avarab@gmail.com","subject":"Re: [PATCH v2 4/6] object-store.h: move read_loose_object() below 'struct object_info'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-23T14:27:24Z","receivedAt":"2021-04-23T14:27:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 11:43:07AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Move the definition of read_loose_object() below \"struct\n> object_info\". In the next commit we'll add a \"struct object_info *\"\n> parameter to it, moving it will avoid a forward declaration of the\n> struct.\n\nThis is a declaration, not a definition, no?\n\nNot a huge deal, I just expected to see the function body moving when I\nread the patch, but didn't.\n\n-Peff\n"},{"id":"422756","messageId":"YILbk34nwqCPmxGQ@coredump.intra.peff.net","threadId":"55404","inReplyTo":"patch-5.6-5fb6ac4faee-20210413T093734Z-avarab@gmail.com","subject":"Re: [PATCH v2 5/6] fsck: report invalid types recorded in objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-23T14:37:07Z","receivedAt":"2021-04-23T14:37:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 11:43:08AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Continue the work in the preceding commit and improve the error on:\n> \n> $ git hash-object --stdin -w -t garbage --literally $ git fsck\n> error: hash mismatch for (expected )\n> error: : object corrupt or missing: \n> [ other fsck output ]\n> \n> To instead emit:\n> \n> $ git fsck\n> error: : object is of unknown type 'garbage': \n> [ other fsck output ]\n\nSounds good.\n\n> @@ -92,7 +93,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n> \tswitch (opt) {\n> \tcase 't':\n> \t\toi.type_name = &sb;\n> -\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n> +\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n> +\t\tif (!unknown_type && ret < 0)\n> \t\t\tdie(\"git cat-file: could not get object info\");\n> \t\tif (sb.len) {\n> \t\t\tprintf(\"%s\\n\", sb.buf);\n\nSurprised to see changes to cat-file here, since the commit message is\nall about fsck. Did the semantics of oid_object_info_extended() change?\nI.e., this hunk implies to me that it is now returning -1 when we said\nunknown types were OK, and we got one. But in that case, how do we\ndistinguish that from a real error?\n\nOr more concretely, this patch causes this:\n\n $ git cat-file -t 1234567890123456789012345678901234567890\n fatal: git cat-file: could not get object info\n\n $ git.compile cat-file --allow-unknown-type -t 1234567890123456789012345678901234567890\n fatal: git cat-file 1234567890123456789012345678901234567890: bad file\n\nOr much worse, from the next hunk:\n\n $ git cat-file -s 1234567890123456789012345678901234567890\n fatal: git cat-file: could not get object info\n\n $ git cat-file --allow-unknown-type -s 1234567890123456789012345678901234567890\n 140732113568960\n\nThat seems wrong (so I think my \"this hunk implies\" is not true, but\nthen I am left with: what is the point of this hunk?).\n\n-Peff\n"},{"id":"422928","messageId":"87im493yos.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YILbk34nwqCPmxGQ@coredump.intra.peff.net","subject":"Re: [PATCH v2 5/6] fsck: report invalid types recorded in objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-26T14:28:30Z","receivedAt":"2021-04-26T14:31:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 23 2021, Jeff King wrote:\n\n> On Tue, Apr 13, 2021 at 11:43:08AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Continue the work in the preceding commit and improve the error on:\n>> \n>> $ git hash-object --stdin -w -t garbage --literally > $ git fsck\n>> error: hash mismatch for (expected )\n>> error: : object corrupt or missing: \n>> [ other fsck output ]\n>> \n>> To instead emit:\n>> \n>> $ git fsck\n>> error: : object is of unknown type 'garbage': \n>> [ other fsck output ]\n>\n> Sounds good.\n>\n>> @@ -92,7 +93,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n>> \tswitch (opt) {\n>> \tcase 't':\n>> \t\toi.type_name = &sb;\n>> -\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n>> +\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n>> +\t\tif (!unknown_type && ret < 0)\n>> \t\t\tdie(\"git cat-file: could not get object info\");\n>> \t\tif (sb.len) {\n>> \t\t\tprintf(\"%s\\n\", sb.buf);\n>\n> Surprised to see changes to cat-file here, since the commit message is\n> all about fsck. Did the semantics of oid_object_info_extended() change?\n> I.e., this hunk implies to me that it is now returning -1 when we said\n> unknown types were OK, and we got one. But in that case, how do we\n> distinguish that from a real error?\n>\n> Or more concretely, this patch causes this:\n>\n> $ git cat-file -t 1234567890123456789012345678901234567890\n> fatal: git cat-file: could not get object info\n>\n> $ git.compile cat-file --allow-unknown-type -t 1234567890123456789012345678901234567890\n> fatal: git cat-file 1234567890123456789012345678901234567890: bad file\n>\n> Or much worse, from the next hunk:\n>\n> $ git cat-file -s 1234567890123456789012345678901234567890\n> fatal: git cat-file: could not get object info\n>\n> $ git cat-file --allow-unknown-type -s 1234567890123456789012345678901234567890\n> 140732113568960\n>\n> That seems wrong (so I think my \"this hunk implies\" is not true, but\n> then I am left with: what is the point of this hunk?).\n\nThat's very well spotted.\n\nI started re-rolling this today but ran out of time. For what it's worth\nthe combination of this and 6/6 \"makes sense\" in the sense that all\ntests pass at the end of this series.\n\nBut the cases you're pointing out are ones we don't have tests for,\ni.e. the combination of \"allow unknown\" and a non-existing object, as\nopposed to a garbage one.\n\nHence the bug with passing up an invalid (uninitialized) size in that\ncase. It's fallout from other partial lib-ification changes of these\nAPIs, i.e. making them return bad values upstream instead of dying right\naway.\n\nI'll sort that out in some sensible way. Starting with adding meaningful\ntest coverage for the existing behavior.\n\n"},{"id":"422937","messageId":"YIbgEaY+cie/4h2X@coredump.intra.peff.net","threadId":"55404","inReplyTo":"87im493yos.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 5/6] fsck: report invalid types recorded in objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-26T15:45:21Z","receivedAt":"2021-04-26T15:45:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 26, 2021 at 04:28:30PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> >> @@ -92,7 +93,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n> >> \tswitch (opt) {\n> >> \tcase 't':\n> >> \t\toi.type_name = &sb;\n> >> -\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n> >> +\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n> >> +\t\tif (!unknown_type && ret < 0)\n> >> \t\t\tdie(\"git cat-file: could not get object info\");\n> >> \t\tif (sb.len) {\n> >> \t\t\tprintf(\"%s\\n\", sb.buf);\n> >\n> > Surprised to see changes to cat-file here, since the commit message is\n> > all about fsck. Did the semantics of oid_object_info_extended() change?\n> > I.e., this hunk implies to me that it is now returning -1 when we said\n> > unknown types were OK, and we got one. But in that case, how do we\n> > distinguish that from a real error?\n> >\n> > Or more concretely, this patch causes this:\n> >\n> > $ git cat-file -t 1234567890123456789012345678901234567890\n> > fatal: git cat-file: could not get object info\n> >\n> > $ git.compile cat-file --allow-unknown-type -t 1234567890123456789012345678901234567890\n> > fatal: git cat-file 1234567890123456789012345678901234567890: bad file\n> >\n> > Or much worse, from the next hunk:\n> >\n> > $ git cat-file -s 1234567890123456789012345678901234567890\n> > fatal: git cat-file: could not get object info\n> >\n> > $ git cat-file --allow-unknown-type -s 1234567890123456789012345678901234567890\n> > 140732113568960\n> >\n> > That seems wrong (so I think my \"this hunk implies\" is not true, but\n> > then I am left with: what is the point of this hunk?).\n> \n> That's very well spotted.\n> \n> I started re-rolling this today but ran out of time. For what it's worth\n> the combination of this and 6/6 \"makes sense\" in the sense that all\n> tests pass at the end of this series.\n> \n> But the cases you're pointing out are ones we don't have tests for,\n> i.e. the combination of \"allow unknown\" and a non-existing object, as\n> opposed to a garbage one.\n> \n> Hence the bug with passing up an invalid (uninitialized) size in that\n> case. It's fallout from other partial lib-ification changes of these\n> APIs, i.e. making them return bad values upstream instead of dying right\n> away.\n\nI'm not sure I understand. The problem seems solely in the hunk above.\nBefore, if we got an error from oid_object_info_extended(), we stopped\nimmediately. But after, we look at the results even though it told us\nthere was an error. In general, I would think that a \"-1\" return value\nfrom oid_object_info_extended() is \"all bets are off\" (remember that\nunlike oid_object_info(), this is a strict error return, and not trying\nto force the object type into the return value).\n\nAnd that's independent of what the other patches in the series are\ndoing, I think.\n\n> I'll sort that out in some sensible way. Starting with adding meaningful\n> test coverage for the existing behavior.\n\nYeah, that sounds fine. I think the current behavior there is perfectly\nreasonable (fail with \"could not get object info\").\n\n-Peff\n"},{"id":"425079","messageId":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.6-00000000000-20210328T025618Z-avarab@gmail.com","subject":"[PATCH v3 00/17] fsck: better \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:22:54Z","receivedAt":"2021-05-20T12:25:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of v2's 6 patch series[1], has turned into 17 now. Less\nscary than one might think though, it's mostly added test coverage +\nsplitting existing commits into more incremental chunks, but see the\nrange-diff. This should address all the feedback on v2 + more.\n\nA brief recap summary of what this is about: We now gracefully recover\ninstead of dying when fsck encounters types that aren't\nblob/commit/tree/tag. Those types don't exist in the wild, but you can\nmanually create them with \"git hash-object -t garbage --literally\".\n\nSo in some senes this matters to nobody, but I'm doing this as part of\ngeneral changes I've been pushing to make fsck/gc error reporting more\ngraceful, and errors more recoverable. We now have a few more places\nin object-file.c where we don't just die(), but properly return\nAPI-like return codes/data to the caller instead.\n\nThis does not contain any changes to how --allow-unknown-type\nhash-object's --literally etc. work, as I suggested we could do in\n[2]. Any such changes will need the API changes here, but these are\njust the narrow fsck fixes.\n\n1. https://lore.kernel.org/git/cover-0.6-00000000000-20210413T093734Z-avarab@gmail.com\n2. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nÆvar Arnfjörð Bjarmason (17):\n fsck tests: refactor one test to use a sub-repo\n fsck tests: add test for fsck-ing an unknown type\n cat-file tests: test for missing object with -t and -s\n cat-file tests: test that --allow-unknown-type isn't on by default\n rev-list tests: test for behavior with invalid object types\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n cache.h: move object functions to object-store.h\n object-file.c: make parse_loose_header_extended() public\n object-file.c: add missing braces to loose_object_info()\n object-file.c: stop dying in parse_loose_header()\n object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n fsck: don't hard die on invalid object types\n object-store.h: move read_loose_object() below 'struct object_info'\n fsck: report invalid types recorded in objects\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 ++++++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 10 ---\n object-file.c | 156 ++++++++++++++++++-------------------\n object-store.h | 60 +++++++++++----\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 10 ++-\n t/t1006-cat-file.sh | 169 +++++++++++++++++++++++++++++++++++++++++\n t/t1450-fsck.sh | 64 +++++++++++-----\n t/t6115-rev-list-du.sh | 11 +++\n 13 files changed, 387 insertions(+), 135 deletions(-)\n\nRange-diff against v2:\n 2: 5a2cd6cca9 = 1: aa38b2bf9e fsck tests: refactor one test to use a sub-repo\n -: ---------- > 2: 82b64abd25 fsck tests: add test for fsck-ing an unknown type\n -: ---------- > 3: 7c3c2fe25d cat-file tests: test for missing object with -t and -s\n -: ---------- > 4: 871b820003 cat-file tests: test that --allow-unknown-type isn't on by default\n -: ---------- > 5: b98da9cc89 rev-list tests: test for behavior with invalid object types\n -: ---------- > 6: 04cc1d20f6 cat-file tests: add corrupt loose object test\n -: ---------- > 7: 9217320888 cat-file tests: test for current --allow-unknown-type behavior\n 1: 37c323a241 = 8: 12dd453879 cache.h: move object functions to object-store.h\n 3: d0d9cb3331 ! 9: 6a5b78dcad fsck: don't hard die on invalid object types\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - fsck: don't hard die on invalid object types\n + object-file.c: make parse_loose_header_extended() public\n \n - Change builtin/fsck.c to pass down a\n - OBJECT_INFO_ALLOW_UNKNOWN_TYPE. This changes this very ungraceful\n - error:\n + Make the parse_loose_header_extended() function public and remove the\n + parse_loose_header() wrapper. The only direct user of it outside of\n + object-file.c itself was in streaming.c, that caller can simply pass\n + the required \"struct object-info *\" instead.\n \n - $ git hash-object --stdin -w -t garbage --literally \n - $ git fsck\n - fatal: invalid object type\n - $\n -\n - Into:\n -\n - $ git fsck\n - error: hash mismatch for (expected )\n - error: : object corrupt or missing: \n - [ the rest of the fsck output here, i.e. it didn't hard die ]\n -\n - We'll still exit with non-zero, but now we'll finish the rest of the\n - traversal. The tests that's being added here asserts that we'll still\n - complain about other fsck issues (e.g. an unrelated dangling blob).\n -\n - But why are we complaining about a \"hash mismatch\" for an object of a\n - type we don't know about? We shouldn't. This is the bare minimal\n - change needed to not make fsck hard die on a repository that's been\n - corrupted in this manner. In subsequent commits we'll teach fsck to\n - recognize this particular type of corruption and emit a better error\n - message.\n -\n - The parse_loose_header() function being changed here is only used in\n - builtin/fsck.c, see f6371f92104 (sha1_file: add read_loose_object()\n - function, 2017-01-13) for its introduction.\n + This change is being done in preparation for teaching\n + read_loose_object() to accept a flag to pass to\n + parse_loose_header(). It isn't strictly necessary for that change, we\n + could simply use parse_loose_header_extended() there, but will leave\n + the API in a better end state.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n - ## builtin/fsck.c ##\n -@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n - \tvoid *contents;\n - \tint eaten;\n - \n --\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n -+\tif (read_loose_object(path, oid, &type, &size, &contents,\n -+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n - \t\terrors_found |= ERROR_OBJECT;\n - \t\terror(_(\"%s: object corrupt or missing: %s\"),\n - \t\t oid_to_hex(oid), path);\n -\n ## object-file.c ##\n @@ object-file.c: static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \n \tif (status >= 0 && oi->contentp) {\n @@ object-file.c: int read_loose_object(const char *path,\n - \t\t const struct object_id *expected_oid,\n - \t\t enum object_type *type,\n - \t\t unsigned long *size,\n --\t\t void **contents)\n -+\t\t void **contents,\n -+\t\t unsigned int oi_flags)\n - {\n - \tint ret = -1;\n - \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n @@ object-file.c: int read_loose_object(const char *path,\n \t}\n \n -\t*type = parse_loose_header(hdr, size);\n -+\t*type = parse_loose_header(hdr, &oi, oi_flags);\n ++\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \n ## object-store.h ##\n -@@ object-store.h: int read_loose_object(const char *path,\n - \t\t const struct object_id *expected_oid,\n - \t\t enum object_type *type,\n - \t\t unsigned long *size,\n --\t\t void **contents);\n -+\t\t void **contents,\n -+\t\t unsigned int oi_flags);\n - \n - /* Retry packed storage after checking packed and loose storage */\n - #define HAS_OBJECT_RECHECK_PACKED 1\n @@ object-store.h: int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n @@ object-store.h: int for_each_packed_object(each_packed_object_fn, void *,\n int finalize_object_file(const char *tmpfile, const char *filename);\n \n ## streaming.c ##\n -@@ streaming.c: static struct stream_vtbl loose_vtbl = {\n - \n - static open_method_decl(loose)\n +@@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n + \t\t\t const struct object_id *oid,\n + \t\t\t enum object_type *type)\n {\n -+\tstruct object_info oi2 = OBJECT_INFO_INIT;\n -+\toi2.sizep = &st->size;\n ++\tstruct object_info oi = OBJECT_INFO_INIT;\n ++\toi.sizep = &st->size;\n +\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n -@@ streaming.c: static open_method_decl(loose)\n +@@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n -\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n -+\t (parse_loose_header(st->u.loose.hdr, &oi2, 0) < 0)) {\n ++\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n -\n - ## t/t1450-fsck.sh ##\n -@@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n - \ttest_i18ngrep \"bad index file\" errors\n - '\n - \n -+test_expect_success 'fsck error and recovery on invalid object type' '\n -+\ttest_create_repo garbage-type &&\n -+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob out 2>err &&\n -+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n -+\tgrep \"dangling blob $empty_blob\" out\n -+'\n -+\n - test_done\n -: ---------- > 10: 5d31d7e1a5 object-file.c: add missing braces to loose_object_info()\n -: ---------- > 11: ee28089219 object-file.c: stop dying in parse_loose_header()\n -: ---------- > 12: 77f2cd439c object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n -: ---------- > 13: d22d5b8b85 object-file.c: return -1, not \"status\" from unpack_loose_header()\n -: ---------- > 14: 260e9888a3 fsck: don't hard die on invalid object types\n 4: 81fffefcf9 ! 15: e2afb813b2 object-store.h: move read_loose_object() below 'struct object_info'\n @@ Metadata\n ## Commit message ##\n object-store.h: move read_loose_object() below 'struct object_info'\n \n - Move the definition of read_loose_object() below \"struct\n + Move the declaration of read_loose_object() below \"struct\n object_info\". In the next commit we'll add a \"struct object_info *\"\n parameter to it, moving it will avoid a forward declaration of the\n struct.\n 5: 5fb6ac4fae ! 16: 328f05c51b fsck: report invalid types recorded in objects\n @@ Commit message\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n - ## builtin/cat-file.c ##\n -@@ builtin/cat-file.c: static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n - \tstruct strbuf sb = STRBUF_INIT;\n - \tunsigned flags = OBJECT_INFO_LOOKUP_REPLACE;\n - \tconst char *path = force_path;\n -+\tint ret;\n - \n - \tif (unknown_type)\n - \t\tflags |= OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n -@@ builtin/cat-file.c: static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n - \tswitch (opt) {\n - \tcase 't':\n - \t\toi.type_name = &sb;\n --\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n -+\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n -+\t\tif (!unknown_type && ret < 0)\n - \t\t\tdie(\"git cat-file: could not get object info\");\n - \t\tif (sb.len) {\n - \t\t\tprintf(\"%s\\n\", sb.buf);\n -@@ builtin/cat-file.c: static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n - \n - \tcase 's':\n - \t\toi.sizep = &size;\n --\t\tif (oid_object_info_extended(the_repository, &oid, &oi, flags) < 0)\n -+\t\tret = oid_object_info_extended(the_repository, &oid, &oi, flags);\n -+\t\tif (!unknown_type && ret < 0)\n - \t\t\tdie(\"git cat-file: could not get object info\");\n - \t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)size);\n - \t\treturn 0;\n -\n ## builtin/fsck.c ##\n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *p\n \n \n ## object-file.c ##\n -@@ object-file.c: int parse_loose_header(const char *hdr,\n - \t * we're obtaining the type using '--allow-unknown-type'\n - \t * option.\n - \t */\n --\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n --\t\ttype = 0;\n --\telse if (type < 0)\n -+\tif (type < 0 && !(flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE))\n - \t\tdie(_(\"invalid object type\"));\n - \tif (oi->typep)\n - \t\t*oi->typep = type;\n -@@ object-file.c: static int loose_object_info(struct repository *r,\n - \t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n - \t\tstatus = error(_(\"unable to unpack %s header\"),\n - \t\t\t oid_to_hex(oid));\n --\tif (status < 0)\n -+\tif (status < 0) {\n - \t\t; /* Do nothing */\n --\telse if (hdrbuf.len) {\n -+\t} else if (hdrbuf.len) {\n - \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n - \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n - \t\t\t\t oid_to_hex(oid));\n --\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n --\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n -+\t} else {\n -+\t\tstatus = parse_loose_header(hdr, oi, flags);\n -+\t\tif (status < 0 && !(flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE))\n -+\t\t\terror(_(\"unable to parse %s header\"), oid_to_hex(oid));\n -+\t}\n - \n - \tif (status >= 0 && oi->contentp) {\n - \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n @@ object-file.c: static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n @@ object-file.c: int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n -\tstruct object_info oi = OBJECT_INFO_INIT;\n + \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n +-\toi.typep = type;\n -\toi.sizep = size;\n +\tenum object_type *type = oi->typep;\n +\tunsigned long *size = oi->sizep;\n @@ object-file.c: int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n --\t*type = parse_loose_header(hdr, &oi, oi_flags);\n --\tif (*type < 0) {\n --\t\terror(_(\"unable to parse header of %s\"), path);\n -+\t*type = parse_loose_header(hdr, oi, oi_flags);\n -+\tif (*type < 0 && !(oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n -+\t\terror(_(\"unable to parse header %s\"), path);\n +-\tif (parse_loose_header(hdr, &oi) < 0) {\n ++\tif (parse_loose_header(hdr, oi) < 0) {\n + \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n - \t}\n @@ object-file.c: int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n @@ t/t1450-fsck.sh: test_expect_success 'fsck error and recovery on invalid object type' '\n - \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob out 2>err &&\n + \tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n +-\ttest_line_count = 2 errors &&\n +-\tgrep \"error: hash mismatch for\" err &&\n -\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n ++\ttest_line_count = 1 errors &&\n +\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n -+\tgrep error: err >err.errors &&\n -+\ttest_line_count = 1 err.errors &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n 6: 226d2031bc ! 17: c5e6686765 fsck: report invalid object type-path combinations\n @@ Metadata\n ## Commit message ##\n fsck: report invalid object type-path combinations\n \n - fsck: improve error on loose object hash mismatch\n -\n Improve the error that's emitted in cases where we find a loose object\n we parse, but which isn't at the location we expect it to be.\n \n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *p\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n -+\tstruct object_id real_oid = null_oid;\n ++\tstruct object_id real_oid = *null_oid();\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n @@ object-file.c: int check_object_signature(struct repository *r, const struct obj\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n --\tr->hash_algo->final_fn(real_oid.hash, &c);\n -+\tr->hash_algo->final_fn(real_oid->hash, &c);\n +-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n ++\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n -\treturn !oideq(oid, &real_oid) ? -1 : 0;\n +\treturn !oideq(oid, real_oid) ? -1 : 0;\n @@ object-file.c: int read_loose_object(const char *path,\n -\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n -\t\t\t oid_to_hex(expected_oid));\n +\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n -+\t\t\tif (oideq(real_oid, &null_oid))\n ++\t\t\tif (oideq(real_oid, null_oid()))\n +\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n @@ object-store.h: int oid_object_info_extended(struct repository *r,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n @@ object-store.h: int unpack_loose_header(git_zstream *stream, unsigned char *map,\n - int parse_loose_header(const char *hdr, struct object_info *oi,\n - \t\t unsigned int flags);\n + int parse_loose_header(const char *hdr, struct object_info *oi);\n + \n int check_object_signature(struct repository *r, const struct object_id *oid,\n -\t\t\t void *buf, unsigned long size, const char *type);\n +\t\t\t void *buf, unsigned long size, const char *type,\n @@ pack-check.c: static int verify_packfile(struct repository *r,\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\n \n + ## t/t1006-cat-file.sh ##\n +@@ t/t1006-cat-file.sh: test_expect_success 'cat-file -t and -s on corrupt loose object' '\n + \t\t# Swap the two to corrupt the repository\n + \t\tmv -v \"$other_path\" \"$empty_path\" &&\n + \t\ttest_must_fail git fsck 2>err.fsck &&\n +-\t\tgrep \"hash mismatch\" err.fsck &&\n ++\t\tgrep \"hash-path mismatch\" err.fsck &&\n + \n + \t\t# confirm that cat-file is reading the new swapped-in\n + \t\t# blob...\n +\n ## t/t1450-fsck.sh ##\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n \t(\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425080","messageId":"patch-01.17-aa38b2bf9e-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 01/17] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:22:55Z","receivedAt":"2021-05-20T12:25:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe should instead simply use something like this test_create_repo\npattern. It's both less verbose, and makes things easier to debug as a\nfailing test can have their state left behind under -d without\ndamaging the state for other tests.\n\nBut let's punt on that general refactoring and just change this one\ntest, I'm going to change it further in subsequent commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5..1563b35f88 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,22 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n-\n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+test_expect_success 'object with hash mismatch' '\n+\ttest_create_repo hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425081","messageId":"patch-02.17-82b64abd25-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 02/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:22:56Z","receivedAt":"2021-05-20T12:25:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 1563b35f88..f36ec1e2f4 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,4 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\ttest_create_repo garbage-type &&\n+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_done\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425082","messageId":"patch-04.17-871b820003-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 04/17] cat-file tests: test that --allow-unknown-type isn't on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:22:58Z","receivedAt":"2021-05-20T12:25:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests added in the tests for the\n--allow-unknown-type feature, added in 39e4ae38804 (cat-file: teach\ncat-file a '--allow-unknown-type' option, 2015-05-03).\n\nBefore this change all the tests would succeed if --allow-unknown-type\nwas on by default, let's fix that by asserting that -t and -s die on a\n\"garbage\" type without --allow-unknown-type.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex b71ef94329..dc01d7c4a9 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -347,6 +347,20 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -363,6 +377,21 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-EOF &&\n+\terror: unable to unpack $bogus_sha1 header\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct when type is large\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425083","messageId":"patch-03.17-7c3c2fe25d-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 03/17] cat-file tests: test for missing object with -t and -s","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:22:57Z","receivedAt":"2021-05-20T12:25:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test for what happens when the -t and -s flags are asked to operate on\na missing object, this extends tests added in 3e370f9faf0 (t1006: add\ntests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n-s flags are the only ones that can be combined with\n--allow-unknown-type, so let's test with and without that flag.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5d2dc99b74..b71ef94329 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,6 +315,33 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n+missing_oid=$(test_oid deadbeef)\n+test_expect_success 'error on type of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -t $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -t --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n+test_expect_success 'error on size of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -s $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -s --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n bogus_type=\"bogus\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425084","messageId":"patch-05.17-b98da9cc89-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 05/17] rev-list tests: test for behavior with invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:22:59Z","receivedAt":"2021-05-20T12:26:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\nadded in 16950f8384a (rev-list: add --disk-usage option for\ncalculating disk usage, 2021-02-09) to test for what happens when it's\nasked to calculate the disk usage of invalid object types.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t6115-rev-list-du.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t6115-rev-list-du.sh b/t/t6115-rev-list-du.sh\nindex b4aef32b71..edb2ed5584 100755\n--- a/t/t6115-rev-list-du.sh\n+++ b/t/t6115-rev-list-du.sh\n@@ -48,4 +48,15 @@ check_du HEAD\n check_du --objects HEAD\n check_du --objects HEAD^..HEAD\n \n+test_expect_success 'setup garbage repository' '\n+\tgit clone --bare . garbage.git &&\n+\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally garbage.git/refs/tags/garbage-tag &&\n+\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n+'\n+\n test_done\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425085","messageId":"patch-06.17-04cc1d20f6-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 06/17] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:00Z","receivedAt":"2021-05-20T12:26:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex dc01d7c4a9..4a76ff024d 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -404,6 +404,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -v \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -v other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425086","messageId":"patch-07.17-9217320888-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 07/17] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:01Z","receivedAt":"2021-05-20T12:26:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 4a76ff024d..d3d3fd733a 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -361,6 +361,46 @@ test_expect_success 'die on broken object under -t and -s without --allow-unknow\n \ttest_must_be_empty out.actual\n '\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_type $bogus_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -372,6 +412,27 @@ test_expect_success \"Size of broken object is correct\" '\n \tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n \ttest_cmp expect actual\n '\n+\n+test_expect_success 'the --allow-unknown-type option does not consider replacement refs' '\n+\tcat >expect <<-EOF &&\n+\t$bogus_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425087","messageId":"patch-09.17-6a5b78dcad-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 09/17] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:03Z","receivedAt":"2021-05-20T12:26:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 21 ++++++++-------------\n object-store.h | 3 ++-\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..527f435381 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1340,8 +1340,9 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr,\n+\t\t struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1401,14 +1402,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1463,10 +1456,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2549,6 +2542,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2563,7 +2558,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/object-store.h b/object-store.h\nindex 9117115a50..d443964447 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -480,7 +480,8 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c..8beac62cbb 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425088","messageId":"patch-11.17-ee28089219-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 11/17] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:05Z","receivedAt":"2021-05-20T12:26:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Start the libification of parse_loose_header() by making it return\nerror codes and data instead of invoking die() by itself. For now\nwe'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller, but in subsequent\ncommits we'll also libify those.\n\nThe reason this makes sense is that with the refactoring of\nparse_loose_header_extended() in an earlier commit the public\ninterface for parse_loose_header() no longer just accepts a \"unsigned\nlong *sizep\". Rather it accepts a \"struct object_info *\", that\nstructure will be populated with information about the object.\n\nIt thus makes sense to further libify the interface so that it stops\ncalling die() when it encounters OBJ_BAD, and instead rely on its\ncallers to check the populated \"oi->typep\".\n\nThis also allows us to simplify away the\nunpack_loose_header_to_strbuf() function added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03). Its code was mostly copy/pasted between it and both\nof unpack_loose_header() and unpack_loose_short_header(). We now have\na single unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead, but let's\nleave this in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 105 ++++++++++++++++++++-----------------------------\n object-store.h | 25 ++++++++++--\n streaming.c | 7 +++-\n 3 files changed, 70 insertions(+), 67 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 115054389c..d4bdf86657 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1210,11 +1210,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1225,44 +1226,25 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n \tif (status < Z_OK)\n \t\treturn status;\n \n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. We abort early\n+\t * unless under we're running as e.g. \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1340,9 +1322,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr,\n-\t\t struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1364,15 +1344,6 @@ int parse_loose_header(const char *hdr,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1399,7 +1370,14 @@ int parse_loose_header(const char *hdr,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1410,9 +1388,12 @@ static int loose_object_info(struct repository *r,\n \tunsigned long mapsize;\n \tvoid *map;\n \tgit_zstream stream;\n+\tint hdr_ret;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1443,27 +1424,23 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\n+\thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL);\n+\tif (hdr_ret < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n-\n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n+\tif (!status && parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0) {\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n+\tif (!allow_unknown && *oi->typep < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -1481,7 +1458,8 @@ static int loose_object_info(struct repository *r,\n \t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n-\tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n \treturn (status < 0) ? status : 0;\n }\n@@ -2547,6 +2525,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2557,17 +2536,19 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex d443964447..740edcac30 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,11 +477,30 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n+\n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown.\n+ */\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n+\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb..c3dc241d6a 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -233,8 +234,10 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n+\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n+\t *type < 0) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425089","messageId":"patch-10.17-5d31d7e1a5-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 10/17] object-file.c: add missing braces to loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:04Z","receivedAt":"2021-05-20T12:26:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the formatting in loose_object_info() to conform with our usual\ncoding style:\n\n When there are multiple arms to a conditional and some of them\n require braces, enclose even a single line block in braces for\n consistency -- Documentation/CodingGuidelines\n\nThis formatting-only change makes a subsequent commit easier to read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 527f435381..115054389c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1450,17 +1450,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t}\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -1469,8 +1472,9 @@ static int loose_object_info(struct repository *r,\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n \t\t}\n-\t} else\n+\t} else {\n \t\tgit_inflate_end(&stream);\n+\t}\n \n \tmunmap(map, mapsize);\n \tif (status && oi->typep)\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425090","messageId":"patch-08.17-12dd453879-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 08/17] cache.h: move object functions to object-store.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:02Z","receivedAt":"2021-05-20T12:26:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of some ancient object functions added in\ne.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n2005-06-01) from cache.h to object-store.h. This continues work\nstarted in cbd53a2193d (object-store: move object access functions to\nobject-store.h, 2018-05-15).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 10 ----------\n object-store.h | 9 +++++++++\n 2 files changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd3..32ea1ea047 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1302,16 +1302,6 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n-\n-int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n-\n-int finalize_object_file(const char *tmpfile, const char *filename);\n-\n-/* Helper to check and \"touch\" a file */\n-int check_and_freshen_file(const char *fn, int freshen);\n \n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb..9117115a50 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,4 +477,13 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz);\n+int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int check_object_signature(struct repository *r, const struct object_id *oid,\n+\t\t\t void *buf, unsigned long size, const char *type);\n+int finalize_object_file(const char *tmpfile, const char *filename);\n+int check_and_freshen_file(const char *fn, int freshen);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425091","messageId":"patch-14.17-260e9888a3-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 14/17] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:08Z","receivedAt":"2021-05-20T12:26:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: hash mismatch for (expected )\n error: : object corrupt or missing: \n [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it. See f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) for the introduction of read_loose_object().\n\nWhy are we complaining about a \"hash mismatch\" for an object of a type\nwe don't know about? We shouldn't. This is the bare minimal change\nneeded to not make fsck hard die on a repository that's been corrupted\nin this manner. In subsequent commits we'll teach fsck to recognize\nthis particular type of corruption and emit a better error message.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 11 ++++++++---\n object-store.h | 3 ++-\n t/t1450-fsck.sh | 14 +++++++-------\n 4 files changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 87a99b0108..38b515deb6 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,7 +600,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tif (read_loose_object(path, oid, &type, &size, &contents,\n+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\ndiff --git a/object-file.c b/object-file.c\nindex 0de699de98..0e8a024eb3 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2522,7 +2522,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n@@ -2530,6 +2531,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \toi.typep = type;\n \toi.sizep = size;\n \n@@ -2552,8 +2554,11 @@ int read_loose_object(const char *path,\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n+\tif (!allow_unknown && *type < 0) {\n+\t\terror(_(\"header for %s declares an unknown type\"), path);\n+\t\tgit_inflate_end(&stream);\n+\t\tgoto out;\n+\t}\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex 9accb614fc..790e8b1798 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,7 +245,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f36ec1e2f4..e7e8decebb 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,16 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \ttest_create_repo garbage-type &&\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n-\tfatal: invalid object type\n-\tEOF\n-\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n-\ttest_cmp err.expect err.actual &&\n-\ttest_must_be_empty out.actual\n+\ttest_must_fail git -C garbage-type fsck >out 2>err &&\n+\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\ttest_line_count = 2 errors &&\n+\tgrep \"error: hash mismatch for\" err &&\n+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"dangling blob $empty_blob\" out\n '\n \n test_done\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425092","messageId":"patch-17.17-c5e6686765-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 17/17] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:11Z","receivedAt":"2021-05-20T12:26:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 13 +++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n object-file.c | 21 ++++++++++++---------\n object-store.h | 4 +++-\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 10 files changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 3c20f164f0..48a3b6a7f8 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 32f11dc1fe..96df1aadbf 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -602,20 +602,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n+\tstruct object_id real_oid = *null_oid();\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n \t\tfound |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n+\t\tif (!oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n \t}\n \tif (type < 0) {\n \t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t oid_to_hex(oid), sb.buf, path);\n+\t\t oid_to_hex(&real_oid), sb.buf, path);\n \t}\n \tif (found) {\n \t\terrors_found |= ERROR_OBJECT;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 3fbc5d7077..bf860b6555 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1421,7 +1421,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd36..3b2dbbb37e 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/object-file.c b/object-file.c\nindex f24fa2fe4a..bf9ac38d73 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1039,9 +1039,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1049,8 +1051,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1075,9 +1077,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2520,6 +2522,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n@@ -2569,9 +2572,9 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n+\t\t\tif (oideq(real_oid, null_oid()))\n+\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex ba6e5d76c0..60b566a63b 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -384,6 +384,7 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n@@ -505,7 +506,8 @@ int unpack_loose_header(git_zstream *stream, unsigned char *map,\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n int finalize_object_file(const char *tmpfile, const char *filename);\n int check_and_freshen_file(const char *fn, int freshen);\n \ndiff --git a/object.c b/object.c\nindex 14188453c5..5467ead328 100644\n--- a/object.c\n+++ b/object.c\n@@ -261,7 +261,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -272,7 +272,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex 4b089fe8ec..e6aa4442c9 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex f12b06150e..0bb5ee97de 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -490,7 +490,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -v \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex bc541af2cf..d76293c495 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -53,6 +53,7 @@ test_expect_success 'object with hash mismatch' '\n \t(\n \t\tcd hash-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -62,7 +63,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -71,6 +72,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t(\n \t\tcd hash-type-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -80,8 +82,8 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425093","messageId":"patch-15.17-e2afb813b2-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 15/17] object-store.h: move read_loose_object() below 'struct object_info'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:09Z","receivedAt":"2021-05-20T12:26:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of read_loose_object() below \"struct\nobject_info\". In the next commit we'll add a \"struct object_info *\"\nparameter to it, moving it will avoid a forward declaration of the\nstruct.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-store.h | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex 790e8b1798..698a701d70 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -234,20 +234,6 @@ int pretend_object_file(void *, unsigned long, enum object_type,\n \n int force_object_loose(const struct object_id *oid, time_t mtime);\n \n-/*\n- * Open the loose object at path, check its hash, and return the contents,\n- * type, and size. If the object is a blob, then \"contents\" may return NULL,\n- * to allow streaming of large blobs.\n- *\n- * Returns 0 on success, negative on error (details may be written to stderr).\n- */\n-int read_loose_object(const char *path,\n-\t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents,\n-\t\t unsigned int oi_flags);\n-\n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n \n@@ -388,6 +374,20 @@ int oid_object_info_extended(struct repository *r,\n \t\t\t const struct object_id *,\n \t\t\t struct object_info *, unsigned flags);\n \n+/*\n+ * Open the loose object at path, check its hash, and return the contents,\n+ * type, and size. If the object is a blob, then \"contents\" may return NULL,\n+ * to allow streaming of large blobs.\n+ *\n+ * Returns 0 on success, negative on error (details may be written to stderr).\n+ */\n+int read_loose_object(const char *path,\n+\t\t const struct object_id *expected_oid,\n+\t\t enum object_type *type,\n+\t\t unsigned long *size,\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n+\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425094","messageId":"patch-13.17-d22d5b8b85-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 13/17] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:07Z","receivedAt":"2021-05-20T12:26:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 7623ada1aa..0de699de98 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1229,7 +1229,7 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425095","messageId":"patch-12.17-77f2cd439c-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 12/17] object-file.c: return -2 on \"header too long\" in unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:06Z","receivedAt":"2021-05-20T12:26:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return -2 saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 15 ++++++++++-----\n object-store.h | 6 ++++--\n t/t1006-cat-file.sh | 2 +-\n 3 files changed, 15 insertions(+), 8 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex d4bdf86657..7623ada1aa 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1240,10 +1240,10 @@ int unpack_loose_header(git_zstream *stream,\n \t/*\n \t * We have a header longer than MAX_HEADER_LEN. We abort early\n \t * unless under we're running as e.g. \"cat-file\n-\t * --allow-unknown-type\".\n+\t * --allow-unknown-type\". A -2 is \"header too long\"\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn -2;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1264,7 +1264,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn -2;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1433,9 +1433,14 @@ static int loose_object_info(struct repository *r,\n \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n \tif (hdr_ret < 0) {\n-\t\tstatus = error(_(\"unable to unpack %s header\"),\n-\t\t\t oid_to_hex(oid));\n+\t\tif (hdr_ret == -2)\n+\t\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\telse\n+\t\t\tstatus = error(_(\"unable to unpack %s header\"),\n+\t\t\t\t oid_to_hex(oid));\n \t}\n+\n \tif (!status && parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0) {\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\ndiff --git a/object-store.h b/object-store.h\nindex 740edcac30..9accb614fc 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -481,13 +481,15 @@ int for_each_packed_object(each_packed_object_fn, void *,\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns 0 on success. Returns negative values on error. If the\n+ * header exceeds MAX_HEADER_LEN -2 will be returned.\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), -2 will still be returned from this\n+ * function to indicate that the header was too long.\n */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex d3d3fd733a..f12b06150e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -440,7 +440,7 @@ bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_t\n \n test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n \tcat >err.expect <<-EOF &&\n-\terror: unable to unpack $bogus_sha1 header\n+\terror: header for $bogus_sha1 too long, exceeds 32 bytes\n \tfatal: git cat-file: could not get object info\n \tEOF\n \n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425096","messageId":"patch-16.17-328f05c51b-20210520T111610Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v3 16/17] fsck: report invalid types recorded in objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-20T11:23:10Z","receivedAt":"2021-05-20T12:26:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the work in the preceding commit and improve the error on:\n\n $ git hash-object --stdin -w -t garbage --literally (expected )\n error: : object corrupt or missing: \n [ other fsck output ]\n\nTo instead emit:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nThe complaint about a \"hash mismatch\" was simply an emergent property\nof how we'd fall though from read_loose_object() into fsck_loose()\nwhen we didn't get the data we expected. Now we'll correctly note that\nthe object type is invalid.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 22 ++++++++++++++++++----\n object-file.c | 13 +++++--------\n object-store.h | 4 ++--\n t/t1450-fsck.sh | 24 +++++++++++++++++++++---\n 4 files changed, 46 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 38b515deb6..32f11dc1fe 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -599,12 +599,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n-\n-\tif (read_loose_object(path, oid, &type, &size, &contents,\n-\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n-\t\terrors_found |= ERROR_OBJECT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n+\tstruct object_info oi;\n+\tint found = 0;\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\n+\t}\n+\tif (type < 0) {\n+\t\tfound |= ERROR_OBJECT;\n+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t oid_to_hex(oid), sb.buf, path);\n+\t}\n+\tif (found) {\n+\t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 0e8a024eb3..f24fa2fe4a 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2520,9 +2520,8 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n@@ -2530,10 +2529,9 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tenum object_type *type = oi->typep;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2549,7 +2547,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n@@ -2571,8 +2569,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex 698a701d70..ba6e5d76c0 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -376,6 +376,7 @@ int oid_object_info_extended(struct repository *r,\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -383,9 +384,8 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n /*\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex e7e8decebb..bc541af2cf 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\ttest_create_repo hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n@@ -869,9 +888,8 @@ test_expect_success 'fsck error and recovery on invalid object type' '\n \tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally out 2>err &&\n \tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n-\ttest_line_count = 2 errors &&\n-\tgrep \"error: hash mismatch for\" err &&\n-\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\ttest_line_count = 1 errors &&\n+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n-- \n2.32.0.rc0.406.g73369325f8d\n\n"},{"id":"425694","messageId":"20210527170832.2669483-1-jonathantanmy@google.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 00/17] fsck: better \"invalid object\" error reporting","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-27T17:08:32Z","receivedAt":"2021-05-27T17:08:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> So in some senes this matters to nobody, but I'm doing this as part of\n> general changes I've been pushing to make fsck/gc error reporting more\n> graceful, and errors more recoverable. We now have a few more places\n> in object-file.c where we don't just die(), but properly return\n> API-like return codes/data to the caller instead.\n\nWell, I guess it's useful if somehow your repository got corrupted, and\nyou want to pinpoint where it occurred.\n\n> Ævar Arnfjörð Bjarmason (17):\n> fsck tests: refactor one test to use a sub-repo\n> fsck tests: add test for fsck-ing an unknown type\n> cat-file tests: test for missing object with -t and -s\n> cat-file tests: test that --allow-unknown-type isn't on by default\n> rev-list tests: test for behavior with invalid object types\n> cat-file tests: add corrupt loose object test\n> cat-file tests: test for current --allow-unknown-type behavior\n> cache.h: move object functions to object-store.h\n> object-file.c: make parse_loose_header_extended() public\n> object-file.c: add missing braces to loose_object_info()\n> object-file.c: stop dying in parse_loose_header()\n> object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n> object-file.c: return -1, not \"status\" from unpack_loose_header()\n> fsck: don't hard die on invalid object types\n> object-store.h: move read_loose_object() below 'struct object_info'\n> fsck: report invalid types recorded in objects\n> fsck: report invalid object type-path combinations\n\nMy main comment as a reviewer is I think that there are a lot of\nunrelated changes in this patch set - in particular, the first 7 tests\n(1 fsck test that refactors something unrelated, 1 fsck test that I\npresume will be overridden later, and 5 tests for other commands\nunrelated to fsck). I'll also give comments on individual patches.\n"},{"id":"425700","messageId":"20210527175039.2672716-1-jonathantanmy@google.com","threadId":"55404","inReplyTo":"patch-11.17-ee28089219-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 11/17] object-file.c: stop dying in parse_loose_header()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-27T17:50:39Z","receivedAt":"2021-05-27T17:50:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Start the libification of parse_loose_header() by making it return\n> error codes and data instead of invoking die() by itself. For now\n> we'll move the relevant die() call to loose_object_info() and\n> read_loose_object() to keep this change smaller, but in subsequent\n> commits we'll also libify those.\n> \n> The reason this makes sense is that with the refactoring of\n> parse_loose_header_extended() in an earlier commit the public\n> interface for parse_loose_header() no longer just accepts a \"unsigned\n> long *sizep\". Rather it accepts a \"struct object_info *\", that\n> structure will be populated with information about the object.\n> \n> It thus makes sense to further libify the interface so that it stops\n> calling die() when it encounters OBJ_BAD, and instead rely on its\n> callers to check the populated \"oi->typep\".\n> \n> This also allows us to simplify away the\n> unpack_loose_header_to_strbuf() function added in\n> 46f034483eb (sha1_file: support reading from a loose object of unknown\n> type, 2015-05-03). Its code was mostly copy/pasted between it and both\n> of unpack_loose_header() and unpack_loose_short_header(). We now have\n> a single unpack_loose_header() function which accepts an optional\n> \"struct strbuf *\" instead.\n> \n> I think the remaining unpack_loose_header() function could be further\n> simplified, we're carrying some complexity just to be able to emit a\n> garbage type longer than MAX_HEADER_LEN, we could alternatively just\n> say \"we found a garbage type ...\" instead, but let's\n> leave this in place for now.\n\nLooking at the patch itself, this patch:\n 1. Combines unpack_loose_header(), unpack_loose_short_header(), and\n unpack_loose_header_to_strbuf() into one function. It does different\n things depending on whether the struct strbuf * is provided.\n 2. Updates parse_loose_header() to:\n a. never die upon invalid object type\n b. not accept a flags argument (which was used solely to control\n whether it died or not upon invalid object type)\n 3. Updates the callers of these functions.\n\nI think 2b should have been mentioned in the commit message. (And\noverall I think that the commit message could be shorter, but what\ninformation to include and exclude is a subjective matter, so I won't\nconcern myself so much about that.)\n\nAlso, I think that 1 should be split into its own commit. As it is, I'm\nnot even sure if it's a good idea - for me, it is confusing for a\nfunction to consume or not consume more of the stream depending on\nwhether an argument is NULL or not.\n\n> \t/*\n> \t * Check if entire header is unpacked in the first iteration.\n> \t */\n> \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * We have a header longer than MAX_HEADER_LEN. We abort early\n> +\t * unless under we're running as e.g. \"cat-file\n> +\t * --allow-unknown-type\".\n> +\t */\n> +\tif (!header)\n> +\t\treturn -1;\n\nWhat do you mean by \"unless under we're running as\"? And how would we\nknow at this point that we're running as \"cat-file --allow-unknown-type\"\njust by checking \"header\"?\n\n> -\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n> -\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n> -\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n> -\t\t\t\t oid_to_hex(oid));\n> -\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n> +\n> +\thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n> +\t\t\t\t allow_unknown ? &hdrbuf : NULL);\n> +\tif (hdr_ret < 0) {\n> \t\tstatus = error(_(\"unable to unpack %s header\"),\n> \t\t\t oid_to_hex(oid));\n> \t}\n\nThis hunk would go into the split commit (for the unpack_loose_header()\nrefactoring) I suggested above.\n\n> -\n> -\tif (status < 0) {\n> -\t\t/* Do nothing */\n> -\t} else if (hdrbuf.len) {\n> -\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n> -\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n> -\t\t\t\t oid_to_hex(oid));\n> -\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n> +\tif (!status && parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0) {\n> \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n> \t}\n> +\tif (!allow_unknown && *oi->typep < 0)\n> +\t\tdie(_(\"invalid object type\"));\n\nI think this change doesn't need to be so big? The oi->typep check could\njust go in the \"else if\" branch that happens if --allow-unknown-type is\nnot set.\n\n> diff --git a/object-store.h b/object-store.h\n> index d443964447..740edcac30 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -477,11 +477,30 @@ int for_each_object_in_pack(struct packed_git *p,\n> int for_each_packed_object(each_packed_object_fn, void *,\n> \t\t\t enum for_each_object_flags flags);\n> \n> +/**\n> + * unpack_loose_header() initializes the data stream needed to unpack\n> + * a loose object header.\n> + *\n> + * Returns 0 on success. Returns negative values on error.\n> + *\n> + * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n> + * \"hdrbuf\" argument is non-NULL. This is intended for use with\n> + * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n> + * reporting. The full header will be extracted to \"hdrbuf\" for use\n> + * with parse_loose_header().\n> + */\n> int unpack_loose_header(git_zstream *stream, unsigned char *map,\n> \t\t\tunsigned long mapsize, void *buffer,\n> +\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n\nParsing up to MAX_HEADER_LEN only occasionally is confusing. Could we\nalways make it parse up to MAX_HEADER_LEN, and then put a truncated\nheader in hdrbuf if it is too big (and therefore invalid)?\n\n> +/**\n> + * parse_loose_header() parses the starting \" \\0\" of an\n> + * object. If it doesn't follow that format -1 is returned. To check\n> + * the validity of the populate the \"typep\" in the \"struct\n> + * object_info\". It will be OBJ_BAD if the object type is unknown.\n> + */\n> +int parse_loose_header(const char *hdr, struct object_info *oi);\n\nAlso mention what happens if the size is invalid.\n"},{"id":"425701","messageId":"20210527175433.2673306-1-jonathantanmy@google.com","threadId":"55404","inReplyTo":"patch-12.17-77f2cd439c-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 12/17] object-file.c: return -2 on \"header too long\" in unpack_loose_header()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-27T17:54:33Z","receivedAt":"2021-05-27T17:54:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/object-store.h b/object-store.h\n> index 740edcac30..9accb614fc 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -481,13 +481,15 @@ int for_each_packed_object(each_packed_object_fn, void *,\n> * unpack_loose_header() initializes the data stream needed to unpack\n> * a loose object header.\n> *\n> - * Returns 0 on success. Returns negative values on error.\n> + * Returns 0 on success. Returns negative values on error. If the\n> + * header exceeds MAX_HEADER_LEN -2 will be returned.\n> *\n> * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n> * \"hdrbuf\" argument is non-NULL. This is intended for use with\n> * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n> * reporting. The full header will be extracted to \"hdrbuf\" for use\n> - * with parse_loose_header().\n> + * with parse_loose_header(), -2 will still be returned from this\n> + * function to indicate that the header was too long.\n> */\n> int unpack_loose_header(git_zstream *stream, unsigned char *map,\n> \t\t\tunsigned long mapsize, void *buffer,\n\nCan the return type be an enum in this case?\n\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index d3d3fd733a..f12b06150e 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -440,7 +440,7 @@ bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_t\n> \n> test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n> \tcat >err.expect <<-EOF &&\n> -\terror: unable to unpack $bogus_sha1 header\n> +\terror: header for $bogus_sha1 too long, exceeds 32 bytes\n> \tfatal: git cat-file: could not get object info\n> \tEOF\n\nAh, the error message is much more informative - thanks.\n"},{"id":"425702","messageId":"20210527181848.2679147-1-jonathantanmy@google.com","threadId":"55404","inReplyTo":"patch-14.17-260e9888a3-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 14/17] fsck: don't hard die on invalid object types","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-27T18:18:48Z","receivedAt":"2021-05-27T18:18:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/object-file.c b/object-file.c\n> index 0de699de98..0e8a024eb3 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2522,7 +2522,8 @@ int read_loose_object(const char *path,\n> \t\t const struct object_id *expected_oid,\n> \t\t enum object_type *type,\n> \t\t unsigned long *size,\n> -\t\t void **contents)\n> +\t\t void **contents,\n> +\t\t unsigned int oi_flags)\n> {\n> \tint ret = -1;\n> \tvoid *map = NULL;\n> @@ -2530,6 +2531,7 @@ int read_loose_object(const char *path,\n> \tgit_zstream stream;\n> \tchar hdr[MAX_HEADER_LEN];\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> +\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n> \toi.typep = type;\n> \toi.sizep = size;\n> \n> @@ -2552,8 +2554,11 @@ int read_loose_object(const char *path,\n> \t\tgit_inflate_end(&stream);\n> \t\tgoto out;\n> \t}\n> -\tif (*type < 0)\n> -\t\tdie(_(\"invalid object type\"));\n> +\tif (!allow_unknown && *type < 0) {\n> +\t\terror(_(\"header for %s declares an unknown type\"), path);\n> +\t\tgit_inflate_end(&stream);\n> +\t\tgoto out;\n> +\t}\n> \n> \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n> \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n\nSo instead of dying, we print an error and behave as if the object was\ninvalid for some other reason. Makes sense.\n"},{"id":"425703","messageId":"20210527182412.2681093-1-jonathantanmy@google.com","threadId":"55404","inReplyTo":"patch-16.17-328f05c51b-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 16/17] fsck: report invalid types recorded in objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-27T18:24:12Z","receivedAt":"2021-05-27T18:24:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/object-store.h b/object-store.h\n> index 698a701d70..ba6e5d76c0 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -376,6 +376,7 @@ int oid_object_info_extended(struct repository *r,\n> \n> /*\n> * Open the loose object at path, check its hash, and return the contents,\n> + * use the \"oi\" argument to assert things about the object, or e.g. populate its\n> * type, and size. If the object is a blob, then \"contents\" may return NULL,\n> * to allow streaming of large blobs.\n> *\n> @@ -383,9 +384,8 @@ int oid_object_info_extended(struct repository *r,\n> */\n> int read_loose_object(const char *path,\n> \t\t const struct object_id *expected_oid,\n> -\t\t enum object_type *type,\n> -\t\t unsigned long *size,\n> \t\t void **contents,\n> +\t\t struct object_info *oi,\n> \t\t unsigned int oi_flags);\n> \n> /*\n\nWhat do you mean by using \"oi\" to assert things? As far as I know, it's\njust a way to specify which information you want (by assigning pointers\nto the appropriate fields).\n\n> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n> index e7e8decebb..bc541af2cf 100755\n> --- a/t/t1450-fsck.sh\n> +++ b/t/t1450-fsck.sh\n> @@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n> \t)\n> '\n> \n> +test_expect_success 'object with hash and type mismatch' '\n> +\ttest_create_repo hash-type-mismatch &&\n> +\t(\n> +\t\tcd hash-type-mismatch &&\n> +\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n> +\t\told=$(test_oid_to_path \"$oid\") &&\n> +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n> +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n> +\t\tmv .git/objects/$old .git/objects/$new &&\n> +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n> +\t\ttree=$(git write-tree) &&\n> +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n> +\t\tgit update-ref refs/heads/bogus $cmt &&\n> +\t\ttest_must_fail git fsck 2>out &&\n> +\t\tgrep \"^error: hash mismatch for \" out &&\n> +\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n\nI don't think we need to know the precise quote used - it's simpler to\njust put \".\" where we need the single quote. Same for the other case\nbelow.\n"},{"id":"425704","messageId":"20210527182811.2682205-1-jonathantanmy@google.com","threadId":"55404","inReplyTo":"patch-17.17-c5e6686765-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 17/17] fsck: report invalid object type-path combinations","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-05-27T18:28:11Z","receivedAt":"2021-05-27T18:28:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Improve the error that's emitted in cases where we find a loose object\n> we parse, but which isn't at the location we expect it to be.\n\nI'll hold off reviewing this for 2 reasons - firstly, I don't think it's\nrelated to the other patches that are about reporting a wrong object\ntype, and secondly, in the case of repository corruption, I don't see\nhow useful computing (and reporting) the hash of a corrupt object is.\n\nIf someone else wants to take a look at this, that would be great, but\notherwise I would suggest splitting this into its own patch set.\n"},{"id":"425710","messageId":"YLAMdYtd77l2g6SP@google.com","threadId":"55404","inReplyTo":"patch-04.17-871b820003-20210520T111610Z-avarab@gmail.com","subject":"Re: [PATCH v3 04/17] cat-file tests: test that --allow-unknown-type isn't on by default","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2021-05-27T21:17:41Z","receivedAt":"2021-05-27T21:17:48Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nÆvar Arnfjörð Bjarmason wrote:\n\n> Fix a blindspot in the tests added in the tests for the\n> --allow-unknown-type feature, added in 39e4ae38804 (cat-file: teach\n> cat-file a '--allow-unknown-type' option, 2015-05-03).\n>\n> Before this change all the tests would succeed if --allow-unknown-type\n> was on by default, let's fix that by asserting that -t and -s die on a\n> \"garbage\" type without --allow-unknown-type.\n\nnit: \"tests added in the tests\" seems oddly repetitive.\n\nMore importantly, I'm curious about the desired behavior here. The\nidea behind cat-file --allow-unknown-type is that I can use it to\ninspect an invalid object, for example after it has been reported by\ngit fsck. The commit that introduced it (39e4ae3880, \"cat-file: teach\ncat-file a '--allow-unknown-type' option\", 2015-05-03) gives the hint\n\"query broken/corrupt objects\" in the documentation, so I figure\nthat's what it's for, and I'm sympathetic.\n\nBut: why is that an option instead of something that we always do?\n\nIn other words, is there some situation where I would not want the\nmore permissive behavior from cat-file against a bad object?\n\nThanks,\nJonathan\n"},{"id":"425721","messageId":"xmqqim3390d9.fsf@gitster.g","threadId":"55404","inReplyTo":"20210527170832.2669483-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 00/17] fsck: better \"invalid object\" error reporting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-28T00:18:42Z","receivedAt":"2021-05-28T00:18:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> My main comment as a reviewer is I think that there are a lot of\n> unrelated changes in this patch set ...\n\nThanks for reviewing. I share the same feeling, not specifically\nabout this series, but I find that \"doing too many while-at-it\nchanges\" is shared among the topics by the same author, and I often\nwish each topic were more focused.\n\n"},{"id":"425729","messageId":"87cztblfh8.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YLAMdYtd77l2g6SP@google.com","subject":"Re: [PATCH v3 04/17] cat-file tests: test that --allow-unknown-type isn't on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-28T03:10:26Z","receivedAt":"2021-05-28T03:11:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 27 2021, Jonathan Nieder wrote:\n\n> Hi,\n>\n> Ævar Arnfjörð Bjarmason wrote:\n>\n>> Fix a blindspot in the tests added in the tests for the\n>> --allow-unknown-type feature, added in 39e4ae38804 (cat-file: teach\n>> cat-file a '--allow-unknown-type' option, 2015-05-03).\n>>\n>> Before this change all the tests would succeed if --allow-unknown-type\n>> was on by default, let's fix that by asserting that -t and -s die on a\n>> \"garbage\" type without --allow-unknown-type.\n>\n> nit: \"tests added in the tests\" seems oddly repetitive.\n>\n> More importantly, I'm curious about the desired behavior here. The\n> idea behind cat-file --allow-unknown-type is that I can use it to\n> inspect an invalid object, for example after it has been reported by\n> git fsck. The commit that introduced it (39e4ae3880, \"cat-file: teach\n> cat-file a '--allow-unknown-type' option\", 2015-05-03) gives the hint\n> \"query broken/corrupt objects\" in the documentation, so I figure\n> that's what it's for, and I'm sympathetic.\n>\n> But: why is that an option instead of something that we always do?\n>\n> In other words, is there some situation where I would not want the\n> more permissive behavior from cat-file against a bad object?\n\nYes. I suggested as much in\nhttps://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nFor this series though I'm sticking to testing for the existing behavior\n+ fixing the immediate fsck issues. I've got some local patches queued\nup for after this topic lands (after I re-roll it, re-submit etc.) that\ndo that.\n"},{"id":"425740","messageId":"60b0829b6eed1_4a64b2084c@natae.notmuch","threadId":"55404","inReplyTo":"xmqqim3390d9.fsf@gitster.g","subject":"Re: [PATCH v3 00/17] fsck: better \"invalid object\" error reporting","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-05-28T05:41:47Z","receivedAt":"2021-05-28T05:41:51Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Junio C Hamano wrote:\n> Jonathan Tan writes:\n> \n> > My main comment as a reviewer is I think that there are a lot of\n> > unrelated changes in this patch set ...\n> \n> Thanks for reviewing. I share the same feeling, not specifically\n> about this series, but I find that \"doing too many while-at-it\n> changes\" is shared among the topics by the same author, and I often\n> wish each topic were more focused.\n\nThe author has a name.\n\nI understand why as a reviewer you want a small patch series, but as a\npatch-writer you want your code to land on master.\n\nPerhaps if there was an actual incentive to split a patch series more\npeople would do so, but in my personal experience that has not been the\ncase.\n\nIf I had to name the reason why some of my patches have landed on master I\nwould say it's *arbitrary*. Maybe you catch reviewers on a good mood, or\nthe maintainer in-between release candidates. But regardless of the\nactual reason, patch-series' size doesn't seem to be a huge factor.\n\nAs exhibit I can five two patch-series of mine:\n\n 1. https://lore.kernel.org/git/20201223144845.143039-1-felipe.contreras@gmail.com/\n 2. https://lore.kernel.org/git/20210426161458.49860-1-felipe.contreras@gmail.com/\n\nThe first one is 4 patches. The second one is 43.\n\nThe second one receved feedback from the maintainer. The first one was\ncomplerely ignored. Neither were acceped.\n\nThis is not intended to point fingers at anyone, merely to state the\na mathematical fact.\n\n\nSplitting a patch series is usually more work. If there's no real\nincentive for a submitter to do so, why would she/he?\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"428399","messageId":"patch-01.21-2e37971c016-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 01/21] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:39Z","receivedAt":"2021-06-24T19:24:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe should instead simply use something like this test_create_repo\npattern. It's both less verbose, and makes things easier to debug as a\nfailing test can have their state left behind under -d without\ndamaging the state for other tests.\n\nBut let's punt on that general refactoring and just change this one\ntest, I'm going to change it further in subsequent commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..1563b35f88c 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,22 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n-\n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+test_expect_success 'object with hash mismatch' '\n+\ttest_create_repo hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428400","messageId":"patch-02.21-79630a99433-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 02/21] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:40Z","receivedAt":"2021-06-24T19:24:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 1563b35f88c..f36ec1e2f4a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,4 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\ttest_create_repo garbage-type &&\n+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_done\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428401","messageId":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com","subject":"[PATCH v4 00/21] fsck: better \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:38Z","receivedAt":"2021-06-24T19:24:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A late re-roll of the v3[1] that's in \"seen\" and causing a couple of\nCI errors, both are solved with this version. For a recap of what this\nis about see v3's summary[1].\n\nOne was a \"mv -v\" issue on OSX, it asks interactively, and defaults to\nno, now moved to \"mv -f\" (as in other tests that move .git/objects/*\nfiles directly).\n\nThe other happened on one of the docker32 Linux boxes, and turned out\nto be an uninitialized variable bug, which happened to be initialized\nto a negative value there, and zero (or positive) everywhere else.\n\nI also went through all the commentary on the v3 (particularly good\nfeedback from Jonathan Tan) and hopefully addressed everything in one\nway or another. In particular the 12/21 is new here, split off from\nwhat's now 14/21. That change was the most complex one in the previous\nseries.\n\n1. https://lore.kernel.org/git/cover-00.17-0000000000-20210520T111610Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (21):\n fsck tests: refactor one test to use a sub-repo\n fsck tests: add test for fsck-ing an unknown type\n cat-file tests: test for missing object with -t and -s\n cat-file tests: test that --allow-unknown-type isn't on by default\n rev-list tests: test for behavior with invalid object types\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n cache.h: move object functions to object-store.h\n object-file.c: don't set \"typep\" when returning non-zero\n object-file.c: make parse_loose_header_extended() public\n object-file.c: add missing braces to loose_object_info()\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: split up ternary in parse_loose_header()\n object-file.c: stop dying in parse_loose_header()\n object-file.c: guard against future bugs in loose_object_info()\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n fsck: don't hard die on invalid object types\n object-store.h: move read_loose_object() below 'struct object_info'\n fsck: report invalid types recorded in objects\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 ++++++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 10 ---\n object-file.c | 178 +++++++++++++++++++++--------------------\n object-store.h | 62 +++++++++++---\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 10 ++-\n t/t1006-cat-file.sh | 169 ++++++++++++++++++++++++++++++++++++++\n t/t1450-fsck.sh | 64 +++++++++++----\n t/t6115-rev-list-du.sh | 11 +++\n 13 files changed, 407 insertions(+), 139 deletions(-)\n\nRange-diff against v3:\n 1: aa38b2bf9e7 = 1: 2e37971c016 fsck tests: refactor one test to use a sub-repo\n 2: 82b64abd250 = 2: 79630a99433 fsck tests: add test for fsck-ing an unknown type\n 3: 7c3c2fe25d9 = 3: 2b5366bfb9d cat-file tests: test for missing object with -t and -s\n 4: 871b8200035 ! 4: ea9a5ef0920 cat-file tests: test that --allow-unknown-type isn't on by default\n @@ Metadata\n ## Commit message ##\n cat-file tests: test that --allow-unknown-type isn't on by default\n \n - Fix a blindspot in the tests added in the tests for the\n - --allow-unknown-type feature, added in 39e4ae38804 (cat-file: teach\n - cat-file a '--allow-unknown-type' option, 2015-05-03).\n + Fix a blindspot in the tests for the --allow-unknown-type feature\n + added in 39e4ae38804 (cat-file: teach cat-file a\n + '--allow-unknown-type' option, 2015-05-03). We should check that\n + --allow-unknown-type isn't on by default.\n \n Before this change all the tests would succeed if --allow-unknown-type\n was on by default, let's fix that by asserting that -t and -s die on a\n 5: b98da9cc89e = 5: 8eaf0e6ddda rev-list tests: test for behavior with invalid object types\n 6: 04cc1d20f62 ! 6: f0e9d92414e cat-file tests: add corrupt loose object test\n @@ t/t1006-cat-file.sh: test_expect_success \"Size of large broken object is correct\n +\t\ttest_cmp out.expect out.actual &&\n +\n +\t\t# Swap the two to corrupt the repository\n -+\t\tmv -v \"$other_path\" \"$empty_path\" &&\n ++\t\tmv -f \"$other_path\" \"$empty_path\" &&\n +\t\ttest_must_fail git fsck 2>err.fsck &&\n +\t\tgrep \"hash mismatch\" err.fsck &&\n +\n @@ t/t1006-cat-file.sh: test_expect_success \"Size of large broken object is correct\n +\n +\t\t# So far \"cat-file\" has been happy to spew the found\n +\t\t# content out as-is. Try to make it zlib-invalid.\n -+\t\tmv -v other.blob \"$empty_path\" &&\n ++\t\tmv -f other.blob \"$empty_path\" &&\n +\t\ttest_must_fail git fsck 2>err.fsck &&\n +\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n +\t)\n 7: 9217320888f = 7: d797d2e8e9d cat-file tests: test for current --allow-unknown-type behavior\n 8: 12dd4538794 = 8: 96310a0bb59 cache.h: move object functions to object-store.h\n -: ----------- > 9: 54fb9189408 object-file.c: don't set \"typep\" when returning non-zero\n 9: 6a5b78dcad8 = 10: 9d36fcbc44a object-file.c: make parse_loose_header_extended() public\n10: 5d31d7e1a54 ! 11: 74c308adc19 object-file.c: add missing braces to loose_object_info()\n @@ object-file.c: static int loose_object_info(struct repository *r,\n +\t}\n \n \tmunmap(map, mapsize);\n - \tif (status && oi->typep)\n + \tif (oi->sizep == &size_scratch)\n11: ee28089219f ! 12: 3f52149bfde object-file.c: stop dying in parse_loose_header()\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - object-file.c: stop dying in parse_loose_header()\n + object-file.c: simplify unpack_loose_short_header()\n \n - Start the libification of parse_loose_header() by making it return\n - error codes and data instead of invoking die() by itself. For now\n - we'll move the relevant die() call to loose_object_info() and\n - read_loose_object() to keep this change smaller, but in subsequent\n - commits we'll also libify those.\n + Combine the unpack_loose_short_header(),\n + unpack_loose_header_to_strbuf() and unpack_loose_header() functions\n + into one.\n \n - The reason this makes sense is that with the refactoring of\n - parse_loose_header_extended() in an earlier commit the public\n - interface for parse_loose_header() no longer just accepts a \"unsigned\n - long *sizep\". Rather it accepts a \"struct object_info *\", that\n - structure will be populated with information about the object.\n -\n - It thus makes sense to further libify the interface so that it stops\n - calling die() when it encounters OBJ_BAD, and instead rely on its\n - callers to check the populated \"oi->typep\".\n -\n - This also allows us to simplify away the\n - unpack_loose_header_to_strbuf() function added in\n + The unpack_loose_header_to_strbuf() function was added in\n 46f034483eb (sha1_file: support reading from a loose object of unknown\n - type, 2015-05-03). Its code was mostly copy/pasted between it and both\n - of unpack_loose_header() and unpack_loose_short_header(). We now have\n - a single unpack_loose_header() function which accepts an optional\n + type, 2015-05-03).\n +\n + Its code was mostly copy/pasted between it and both of\n + unpack_loose_header() and unpack_loose_short_header(). We now have a\n + single unpack_loose_header() function which accepts an optional\n \"struct strbuf *\" instead.\n \n I think the remaining unpack_loose_header() function could be further\n simplified, we're carrying some complexity just to be able to emit a\n garbage type longer than MAX_HEADER_LEN, we could alternatively just\n - say \"we found a garbage type ...\" instead, but let's\n - leave this in place for now.\n + say \"we found a garbage type ...\" instead. But let's\n + leave the current behavior in place for now.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ object-file.c: static int unpack_loose_short_header(git_zstream *stream,\n \t\treturn 0;\n \n +\t/*\n -+\t * We have a header longer than MAX_HEADER_LEN. We abort early\n -+\t * unless under we're running as e.g. \"cat-file\n ++\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n ++\t * here is only non-NULL when we run \"cat-file\n +\t * --allow-unknown-type\".\n +\t */\n +\tif (!header)\n @@ object-file.c: static int unpack_loose_short_header(git_zstream *stream,\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n -@@ object-file.c: static void *unpack_loose_rest(git_zstream *stream,\n - * too permissive for what we want to check. So do an anal\n - * object header parse by hand.\n - */\n --int parse_loose_header(const char *hdr,\n --\t\t struct object_info *oi,\n --\t\t unsigned int flags)\n -+int parse_loose_header(const char *hdr, struct object_info *oi)\n - {\n - \tconst char *type_buf = hdr;\n - \tunsigned long size;\n -@@ object-file.c: int parse_loose_header(const char *hdr,\n - \ttype = type_from_string_gently(type_buf, type_len, 1);\n - \tif (oi->type_name)\n - \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n --\t/*\n --\t * Set type to 0 if its an unknown object and\n --\t * we're obtaining the type using '--allow-unknown-type'\n --\t * option.\n --\t */\n --\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n --\t\ttype = 0;\n --\telse if (type < 0)\n --\t\tdie(_(\"invalid object type\"));\n - \tif (oi->typep)\n - \t\t*oi->typep = type;\n - \n -@@ object-file.c: int parse_loose_header(const char *hdr,\n - \t/*\n - \t * The length must be followed by a zero byte\n - \t */\n --\treturn *hdr ? -1 : type;\n -+\tif (*hdr)\n -+\t\treturn -1;\n -+\n -+\t/*\n -+\t * The format is valid, but the type may still be bogus. The\n -+\t * Caller needs to check its oi->typep.\n -+\t */\n -+\treturn 0;\n - }\n - \n - static int loose_object_info(struct repository *r,\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \tunsigned long mapsize;\n \tvoid *map;\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n -+\tenum object_type type_scratch;\n +\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \n - \tif (!oi->sizep)\n - \t\toi->sizep = &size_scratch;\n -+\tif (!oi->typep)\n -+\t\toi->typep = &type_scratch;\n - \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n -\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n --\n --\tif (status < 0) {\n --\t\t/* Do nothing */\n --\t} else if (hdrbuf.len) {\n --\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n --\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n --\t\t\t\t oid_to_hex(oid));\n --\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n -+\tif (!status && parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0) {\n - \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n - \t}\n -+\tif (!allow_unknown && *oi->typep < 0)\n -+\t\tdie(_(\"invalid object type\"));\n - \n - \tif (status >= 0 && oi->contentp) {\n - \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n -@@ object-file.c: static int loose_object_info(struct repository *r,\n - \t\t*oi->typep = status;\n - \tif (oi->sizep == &size_scratch)\n - \t\toi->sizep = NULL;\n --\tstrbuf_release(&hdrbuf);\n -+\tif (oi->typep == &type_scratch)\n -+\t\toi->typep = NULL;\n - \toi->whence = OI_LOOSE;\n - \treturn (status < 0) ? status : 0;\n - }\n -@@ object-file.c: int read_loose_object(const char *path,\n - \tgit_zstream stream;\n - \tchar hdr[MAX_HEADER_LEN];\n - \tstruct object_info oi = OBJECT_INFO_INIT;\n -+\toi.typep = type;\n - \toi.sizep = size;\n - \n - \t*contents = NULL;\n @@ object-file.c: int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n @@ object-file.c: int read_loose_object(const char *path,\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\n - \n --\t*type = parse_loose_header(hdr, &oi, 0);\n --\tif (*type < 0) {\n -+\tif (parse_loose_header(hdr, &oi) < 0) {\n - \t\terror(_(\"unable to parse header of %s\"), path);\n - \t\tgit_inflate_end(&stream);\n - \t\tgoto out;\n - \t}\n -+\tif (*type < 0)\n -+\t\tdie(_(\"invalid object type\"));\n - \n - \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n - \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \n ## object-store.h ##\n @@ object-store.h: int for_each_object_in_pack(struct packed_git *p,\n @@ object-store.h: int for_each_object_in_pack(struct packed_git *p,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n -\t\t\tunsigned long bufsiz);\n --int parse_loose_header(const char *hdr, struct object_info *oi,\n --\t\t unsigned int flags);\n +\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n -+\n -+/**\n -+ * parse_loose_header() parses the starting \" \\0\" of an\n -+ * object. If it doesn't follow that format -1 is returned. To check\n -+ * the validity of the populate the \"typep\" in the \"struct\n -+ * object_info\". It will be OBJ_BAD if the object type is unknown.\n -+ */\n -+int parse_loose_header(const char *hdr, struct object_info *oi);\n -+\n + int parse_loose_header(const char *hdr, struct object_info *oi,\n + \t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n - \t\t\t void *buf, unsigned long size, const char *type);\n - int finalize_object_file(const char *tmpfile, const char *filename);\n \n ## streaming.c ##\n -@@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n - {\n - \tstruct object_info oi = OBJECT_INFO_INIT;\n - \toi.sizep = &st->size;\n -+\toi.typep = type;\n - \n - \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n - \tif (!st->u.loose.mapped)\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n -\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n --\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n +\t\t\t\t sizeof(st->u.loose.hdr),\n +\t\t\t\t NULL) < 0) ||\n -+\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n -+\t *type < 0) {\n + \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n - \t\treturn -1;\n -: ----------- > 13: ba632be1520 object-file.c: split up ternary in parse_loose_header()\n -: ----------- > 14: ea4f446f5b1 object-file.c: stop dying in parse_loose_header()\n -: ----------- > 15: aacef784eab object-file.c: guard against future bugs in loose_object_info()\n13: d22d5b8b85e = 16: 050cfc7808c object-file.c: return -1, not \"status\" from unpack_loose_header()\n12: 77f2cd439c6 ! 17: 78e3152fd94 object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n @@ Commit message\n MAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n \n + I tried setting up an enum just for these three return values, but I\n + think the result was less readable. Let's consider doing that if we\n + gain even more return values. For now let's do the next best thing and\n + enumerate our known return values, and BUG() if we encounter one we\n + don't know about.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## object-file.c ##\n @@ object-file.c: int unpack_loose_header(git_zstream *stream,\n - \t/*\n - \t * We have a header longer than MAX_HEADER_LEN. We abort early\n - \t * unless under we're running as e.g. \"cat-file\n --\t * --allow-unknown-type\".\n -+\t * --allow-unknown-type\". A -2 is \"header too long\"\n + \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n -\t\treturn -1;\n @@ object-file.c: int unpack_loose_header(git_zstream *stream,\n \n static void *unpack_loose_rest(git_zstream *stream,\n @@ object-file.c: static int loose_object_info(struct repository *r,\n + \n \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n - \tif (hdr_ret < 0) {\n --\t\tstatus = error(_(\"unable to unpack %s header\"),\n --\t\t\t oid_to_hex(oid));\n -+\t\tif (hdr_ret == -2)\n -+\t\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n -+\t\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n -+\t\telse\n -+\t\t\tstatus = error(_(\"unable to unpack %s header\"),\n -+\t\t\t\t oid_to_hex(oid));\n - \t}\n -+\n - \tif (!status && parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0) {\n - \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n +-\tif (hdr_ret < 0) {\n ++\tswitch (hdr_ret) {\n ++\tcase 0:\n ++\t\tbreak;\n ++\tcase -1:\n + \t\tstatus = error(_(\"unable to unpack %s header\"),\n + \t\t\t oid_to_hex(oid));\n ++\t\tbreak;\n ++\tcase -2:\n ++\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n ++\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n ++\t\tbreak;\n ++\tdefault:\n ++\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n \t}\n + \tif (!status) {\n + \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n \n ## object-store.h ##\n @@ object-store.h: int for_each_packed_object(each_packed_object_fn, void *,\n14: 260e9888a3e = 18: f9bb1b799ac fsck: don't hard die on invalid object types\n15: e2afb813b28 = 19: acbea7e2a2a object-store.h: move read_loose_object() below 'struct object_info'\n16: 328f05c51b3 = 20: edc28de229d fsck: report invalid types recorded in objects\n17: c5e6686765d ! 21: e588c05f461 fsck: report invalid object type-path combinations\n @@ pack-check.c: static int verify_packfile(struct repository *r,\n ## t/t1006-cat-file.sh ##\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n - \t\tmv -v \"$other_path\" \"$empty_path\" &&\n + \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n -\t\tgrep \"hash mismatch\" err.fsck &&\n +\t\tgrep \"hash-path mismatch\" err.fsck &&\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428402","messageId":"patch-03.21-2b5366bfb9d-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 03/21] cat-file tests: test for missing object with -t and -s","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:41Z","receivedAt":"2021-06-24T19:24:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test for what happens when the -t and -s flags are asked to operate on\na missing object, this extends tests added in 3e370f9faf0 (t1006: add\ntests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n-s flags are the only ones that can be combined with\n--allow-unknown-type, so let's test with and without that flag.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5d2dc99b74a..b71ef94329e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,6 +315,33 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n+missing_oid=$(test_oid deadbeef)\n+test_expect_success 'error on type of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -t $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -t --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n+test_expect_success 'error on size of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -s $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -s --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n bogus_type=\"bogus\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428403","messageId":"patch-04.21-ea9a5ef0920-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 04/21] cat-file tests: test that --allow-unknown-type isn't on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:42Z","receivedAt":"2021-06-24T19:24:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the --allow-unknown-type feature\nadded in 39e4ae38804 (cat-file: teach cat-file a\n'--allow-unknown-type' option, 2015-05-03). We should check that\n--allow-unknown-type isn't on by default.\n\nBefore this change all the tests would succeed if --allow-unknown-type\nwas on by default, let's fix that by asserting that -t and -s die on a\n\"garbage\" type without --allow-unknown-type.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex b71ef94329e..dc01d7c4a9a 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -347,6 +347,20 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -363,6 +377,21 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-EOF &&\n+\terror: unable to unpack $bogus_sha1 header\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct when type is large\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428404","messageId":"patch-05.21-8eaf0e6ddda-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 05/21] rev-list tests: test for behavior with invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:43Z","receivedAt":"2021-06-24T19:24:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\nadded in 16950f8384a (rev-list: add --disk-usage option for\ncalculating disk usage, 2021-02-09) to test for what happens when it's\nasked to calculate the disk usage of invalid object types.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t6115-rev-list-du.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t6115-rev-list-du.sh b/t/t6115-rev-list-du.sh\nindex b4aef32b713..edb2ed55846 100755\n--- a/t/t6115-rev-list-du.sh\n+++ b/t/t6115-rev-list-du.sh\n@@ -48,4 +48,15 @@ check_du HEAD\n check_du --objects HEAD\n check_du --objects HEAD^..HEAD\n \n+test_expect_success 'setup garbage repository' '\n+\tgit clone --bare . garbage.git &&\n+\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally garbage.git/refs/tags/garbage-tag &&\n+\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n+'\n+\n test_done\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428405","messageId":"patch-06.21-f0e9d92414e-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 06/21] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:44Z","receivedAt":"2021-06-24T19:24:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex dc01d7c4a9a..7f10a92f0e4 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -404,6 +404,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428406","messageId":"patch-07.21-d797d2e8e9d-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 07/21] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:45Z","receivedAt":"2021-06-24T19:24:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 7f10a92f0e4..86fd2a90ca7 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -361,6 +361,46 @@ test_expect_success 'die on broken object under -t and -s without --allow-unknow\n \ttest_must_be_empty out.actual\n '\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_type $bogus_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -372,6 +412,27 @@ test_expect_success \"Size of broken object is correct\" '\n \tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n \ttest_cmp expect actual\n '\n+\n+test_expect_success 'the --allow-unknown-type option does not consider replacement refs' '\n+\tcat >expect <<-EOF &&\n+\t$bogus_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428407","messageId":"patch-08.21-96310a0bb59-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 08/21] cache.h: move object functions to object-store.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:46Z","receivedAt":"2021-06-24T19:24:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of some ancient object functions added in\ne.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n2005-06-01) from cache.h to object-store.h. This continues work\nstarted in cbd53a2193d (object-store: move object access functions to\nobject-store.h, 2018-05-15).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 10 ----------\n object-store.h | 9 +++++++++\n 2 files changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd36..32ea1ea0474 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1302,16 +1302,6 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n-\n-int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n-\n-int finalize_object_file(const char *tmpfile, const char *filename);\n-\n-/* Helper to check and \"touch\" a file */\n-int check_and_freshen_file(const char *fn, int freshen);\n \n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb5..9117115a50c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,4 +477,13 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz);\n+int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int check_object_signature(struct repository *r, const struct object_id *oid,\n+\t\t\t void *buf, unsigned long size, const char *type);\n+int finalize_object_file(const char *tmpfile, const char *filename);\n+int check_and_freshen_file(const char *fn, int freshen);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428408","messageId":"patch-10.21-9d36fcbc44a-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 10/21] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:48Z","receivedAt":"2021-06-24T19:24:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 21 ++++++++-------------\n object-store.h | 3 ++-\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 9210e2e6fe4..e0ba1842272 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1340,8 +1340,9 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr,\n+\t\t struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1401,14 +1402,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1463,10 +1456,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2547,6 +2540,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2561,7 +2556,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/object-store.h b/object-store.h\nindex 9117115a50c..d443964447c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -480,7 +480,8 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428409","messageId":"patch-09.21-54fb9189408-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 09/21] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:47Z","receivedAt":"2021-06-24T19:24:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b22..9210e2e6fe4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1480,8 +1480,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428410","messageId":"patch-11.21-74c308adc19-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 11/21] object-file.c: add missing braces to loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:49Z","receivedAt":"2021-06-24T19:24:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the formatting in loose_object_info() to conform with our usual\ncoding style:\n\n When there are multiple arms to a conditional and some of them\n require braces, enclose even a single line block in braces for\n consistency -- Documentation/CodingGuidelines\n\nThis formatting-only change makes a subsequent commit easier to read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex e0ba1842272..646ca7f85d6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1450,17 +1450,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t}\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -1469,8 +1472,9 @@ static int loose_object_info(struct repository *r,\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n \t\t}\n-\t} else\n+\t} else {\n \t\tgit_inflate_end(&stream);\n+\t}\n \n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428411","messageId":"patch-12.21-3f52149bfde-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 12/21] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:50Z","receivedAt":"2021-06-24T19:24:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 60 ++++++++++++++++++--------------------------------\n object-store.h | 14 +++++++++++-\n streaming.c | 3 ++-\n 3 files changed, 37 insertions(+), 40 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 646ca7f85d6..ef3a1517fed 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1210,11 +1210,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1225,44 +1226,25 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n \tif (status < Z_OK)\n \t\treturn status;\n \n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1410,9 +1392,11 @@ static int loose_object_info(struct repository *r,\n \tunsigned long mapsize;\n \tvoid *map;\n \tgit_zstream stream;\n+\tint hdr_ret;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1446,11 +1430,10 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\n+\thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL);\n+\tif (hdr_ret < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n@@ -2555,7 +2538,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/object-store.h b/object-store.h\nindex d443964447c..31327a7f6c3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,9 +477,21 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz);\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428412","messageId":"patch-13.21-ba632be1520-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 13/21] object-file.c: split up ternary in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:51Z","receivedAt":"2021-06-24T19:24:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This minor formatting change serves to make a subsequent patch easier\nto read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex ef3a1517fed..e51cf2ca33e 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1381,7 +1381,10 @@ int parse_loose_header(const char *hdr,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\treturn type;\n }\n \n static int loose_object_info(struct repository *r,\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428413","messageId":"patch-14.21-ea4f446f5b1-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 14/21] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:52Z","receivedAt":"2021-06-24T19:24:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Start the libification of parse_loose_header() by making it return\nerror codes and data instead of invoking die() by itself. For now\nwe'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller, but in subsequent\ncommits we'll also libify those.\n\nSince the refactoring of parse_loose_header_extended() into\nparse_loose_header() in an earlier commit, its interface accepts a\n\"unsigned long *sizep\". Rather it accepts a \"struct object_info *\",\nthat structure will be populated with information about the object.\n\nIt thus makes sense to further libify the interface so that it stops\ncalling die() when it encounters OBJ_BAD, and instead rely on its\ncallers to check the populated \"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nIn another case added in c84a1f3ed4d (sha1_file: refactor read_object,\n2017-06-21) (but the behavior pre-dated that) we did checks of \"status\n>= 0\", because at that point \"status\" had become the return value of\nparse_loose_header(). I.e. a non-negative \"enum object_type\" (unless\nwe -1, aka. OBJ_BAD).\n\nNow that parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 53 ++++++++++++++++++++++++++------------------------\n object-store.h | 13 +++++++++++--\n streaming.c | 4 +++-\n 3 files changed, 42 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex e51cf2ca33e..31263335af9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1322,9 +1322,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr,\n-\t\t struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1346,15 +1344,6 @@ int parse_loose_header(const char *hdr,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1384,7 +1373,11 @@ int parse_loose_header(const char *hdr,\n \tif (*hdr)\n \t\treturn -1;\n \n-\treturn type;\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1399,6 +1392,8 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n+\tint parsed_header = 0;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1430,6 +1425,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1440,18 +1437,20 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n-\n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\tif (!status) {\n+\t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n+\t\t\t/*\n+\t\t\t * oi->{sizep,typep} are meaningless unless\n+\t\t\t * parse_loose_header() returns >= 0.\n+\t\t\t */\n+\t\t\tparsed_header = 1;\n+\t\telse\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n+\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n-\tif (status >= 0 && oi->contentp) {\n+\tif (parsed_header && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n \t\t\t\t\t\t *oi->sizep, oid);\n \t\tif (!*oi->contentp) {\n@@ -1466,6 +1465,8 @@ static int loose_object_info(struct repository *r,\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n \treturn (status < 0) ? status : 0;\n }\n@@ -2531,6 +2532,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2547,12 +2549,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex 31327a7f6c3..65a8e4dc6a8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -492,8 +492,17 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+\n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n+\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..c3dc241d6a5 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -235,7 +236,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr),\n \t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n+\t *type < 0) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428414","messageId":"patch-15.21-aacef784eab-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 15/21] object-file.c: guard against future bugs in loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:53Z","receivedAt":"2021-06-24T19:24:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"An earlier version of the preceding commit had a subtle bug where our\n\"type_scratch\" (later assigned to \"oi->typep\") would be uninitialized\nand used in the \"!allow_unknown\" case, at which point it would contain\na nonsensical value if we'd failed to call parse_loose_header().\n\nThe preceding commit introduced \"parsed_header\" variable to check for\nthis case, but I think we can do better, let's carry a \"oi_header\"\nvariable initially set to NULL, and only set it to \"oi\" once we're\npast parse_loose_header().\n\nThis is functionally the same thing, but hopefully makes it even more\nobvious in the future that we must not access the \"typep\" and\n\"sizep\" (or \"type_name\") unless parse_loose_header() succeeds, but\nthat accessing other fields set earlier (such as the \"disk_sizep\" set\nearlier) is OK.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 31263335af9..d41f444e6cc 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1393,7 +1393,7 @@ static int loose_object_info(struct repository *r,\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n \tenum object_type type_scratch;\n-\tint parsed_header = 0;\n+\tstruct object_info *oi_header = NULL;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1441,18 +1441,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n \t\t\t/*\n \t\t\t * oi->{sizep,typep} are meaningless unless\n-\t\t\t * parse_loose_header() returns >= 0.\n+\t\t\t * parse_loose_header() returns >= 0. Let's\n+\t\t\t * access them as \"oi_header\" (just an alias\n+\t\t\t * for \"oi\") below to make that intent clear.\n \t\t\t */\n-\t\t\tparsed_header = 1;\n+\t\t\toi_header = oi;\n \t\telse\n \t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n-\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n+\tif (!allow_unknown && oi_header && *oi_header->typep < 0)\n \t\tdie(_(\"invalid object type\"));\n \n-\tif (parsed_header && oi->contentp) {\n+\tif (oi_header && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n+\t\t\t\t\t\t *oi_header->sizep, oid);\n \t\tif (!*oi->contentp) {\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428415","messageId":"patch-16.21-050cfc7808c-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 16/21] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:54Z","receivedAt":"2021-06-24T19:24:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex d41f444e6cc..956ca260518 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1229,7 +1229,7 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428416","messageId":"patch-17.21-78e3152fd94-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 17/21] object-file.c: return -2 on \"header too long\" in unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:55Z","receivedAt":"2021-06-24T19:24:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return -2 saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nI tried setting up an enum just for these three return values, but I\nthink the result was less readable. Let's consider doing that if we\ngain even more return values. For now let's do the next best thing and\nenumerate our known return values, and BUG() if we encounter one we\ndon't know about.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 +++++++++++++---\n object-store.h | 6 ++++--\n t/t1006-cat-file.sh | 2 +-\n 3 files changed, 18 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 956ca260518..1866115a1c5 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1243,7 +1243,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn -2;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1264,7 +1264,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn -2;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1433,9 +1433,19 @@ static int loose_object_info(struct repository *r,\n \n \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n-\tif (hdr_ret < 0) {\n+\tswitch (hdr_ret) {\n+\tcase 0:\n+\t\tbreak;\n+\tcase -1:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n+\t\tbreak;\n+\tcase -2:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n \t}\n \tif (!status) {\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\ndiff --git a/object-store.h b/object-store.h\nindex 65a8e4dc6a8..1151ce8e820 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -481,13 +481,15 @@ int for_each_packed_object(each_packed_object_fn, void *,\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns 0 on success. Returns negative values on error. If the\n+ * header exceeds MAX_HEADER_LEN -2 will be returned.\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), -2 will still be returned from this\n+ * function to indicate that the header was too long.\n */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 86fd2a90ca7..06d38e1fae6 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -440,7 +440,7 @@ bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_t\n \n test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n \tcat >err.expect <<-EOF &&\n-\terror: unable to unpack $bogus_sha1 header\n+\terror: header for $bogus_sha1 too long, exceeds 32 bytes\n \tfatal: git cat-file: could not get object info\n \tEOF\n \n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428417","messageId":"patch-18.21-f9bb1b799ac-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 18/21] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:56Z","receivedAt":"2021-06-24T19:24:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: hash mismatch for (expected )\n error: : object corrupt or missing: \n [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it. See f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) for the introduction of read_loose_object().\n\nWhy are we complaining about a \"hash mismatch\" for an object of a type\nwe don't know about? We shouldn't. This is the bare minimal change\nneeded to not make fsck hard die on a repository that's been corrupted\nin this manner. In subsequent commits we'll teach fsck to recognize\nthis particular type of corruption and emit a better error message.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 11 ++++++++---\n object-store.h | 3 ++-\n t/t1450-fsck.sh | 14 +++++++-------\n 4 files changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..082dadd5629 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -601,7 +601,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tif (read_loose_object(path, oid, &type, &size, &contents,\n+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\ndiff --git a/object-file.c b/object-file.c\nindex 1866115a1c5..8fb55fc6f58 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2536,7 +2536,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n@@ -2544,6 +2545,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \toi.typep = type;\n \toi.sizep = size;\n \n@@ -2566,8 +2568,11 @@ int read_loose_object(const char *path,\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n+\tif (!allow_unknown && *type < 0) {\n+\t\terror(_(\"header for %s declares an unknown type\"), path);\n+\t\tgit_inflate_end(&stream);\n+\t\tgoto out;\n+\t}\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex 1151ce8e820..94ff03072c1 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,7 +245,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f36ec1e2f4a..e7e8decebbd 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,16 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \ttest_create_repo garbage-type &&\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n-\tfatal: invalid object type\n-\tEOF\n-\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n-\ttest_cmp err.expect err.actual &&\n-\ttest_must_be_empty out.actual\n+\ttest_must_fail git -C garbage-type fsck >out 2>err &&\n+\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\ttest_line_count = 2 errors &&\n+\tgrep \"error: hash mismatch for\" err &&\n+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"dangling blob $empty_blob\" out\n '\n \n test_done\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428418","messageId":"patch-19.21-acbea7e2a2a-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 19/21] object-store.h: move read_loose_object() below 'struct object_info'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:57Z","receivedAt":"2021-06-24T19:24:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of read_loose_object() below \"struct\nobject_info\". In the next commit we'll add a \"struct object_info *\"\nparameter to it, moving it will avoid a forward declaration of the\nstruct.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-store.h | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex 94ff03072c1..72d668b1674 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -234,20 +234,6 @@ int pretend_object_file(void *, unsigned long, enum object_type,\n \n int force_object_loose(const struct object_id *oid, time_t mtime);\n \n-/*\n- * Open the loose object at path, check its hash, and return the contents,\n- * type, and size. If the object is a blob, then \"contents\" may return NULL,\n- * to allow streaming of large blobs.\n- *\n- * Returns 0 on success, negative on error (details may be written to stderr).\n- */\n-int read_loose_object(const char *path,\n-\t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents,\n-\t\t unsigned int oi_flags);\n-\n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n \n@@ -388,6 +374,20 @@ int oid_object_info_extended(struct repository *r,\n \t\t\t const struct object_id *,\n \t\t\t struct object_info *, unsigned flags);\n \n+/*\n+ * Open the loose object at path, check its hash, and return the contents,\n+ * type, and size. If the object is a blob, then \"contents\" may return NULL,\n+ * to allow streaming of large blobs.\n+ *\n+ * Returns 0 on success, negative on error (details may be written to stderr).\n+ */\n+int read_loose_object(const char *path,\n+\t\t const struct object_id *expected_oid,\n+\t\t enum object_type *type,\n+\t\t unsigned long *size,\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n+\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428419","messageId":"patch-20.21-edc28de229d-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 20/21] fsck: report invalid types recorded in objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:58Z","receivedAt":"2021-06-24T19:24:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the work in the preceding commit and improve the error on:\n\n $ git hash-object --stdin -w -t garbage --literally (expected )\n error: : object corrupt or missing: \n [ other fsck output ]\n\nTo instead emit:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nThe complaint about a \"hash mismatch\" was simply an emergent property\nof how we'd fall though from read_loose_object() into fsck_loose()\nwhen we didn't get the data we expected. Now we'll correctly note that\nthe object type is invalid.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 22 ++++++++++++++++++----\n object-file.c | 13 +++++--------\n object-store.h | 4 ++--\n t/t1450-fsck.sh | 24 +++++++++++++++++++++---\n 4 files changed, 46 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 082dadd5629..07af0434db6 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,12 +600,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n-\n-\tif (read_loose_object(path, oid, &type, &size, &contents,\n-\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n-\t\terrors_found |= ERROR_OBJECT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n+\tstruct object_info oi;\n+\tint found = 0;\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\n+\t}\n+\tif (type < 0) {\n+\t\tfound |= ERROR_OBJECT;\n+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t oid_to_hex(oid), sb.buf, path);\n+\t}\n+\tif (found) {\n+\t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 8fb55fc6f58..e550ea0c7cf 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2534,9 +2534,8 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n@@ -2544,10 +2543,9 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tenum object_type *type = oi->typep;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2563,7 +2561,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n@@ -2585,8 +2583,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex 72d668b1674..96a5970f314 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -376,6 +376,7 @@ int oid_object_info_extended(struct repository *r,\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -383,9 +384,8 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n /*\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex e7e8decebbd..bc541af2cfc 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\ttest_create_repo hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv .git/objects/$old .git/objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n@@ -869,9 +888,8 @@ test_expect_success 'fsck error and recovery on invalid object type' '\n \tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally out 2>err &&\n \tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n-\ttest_line_count = 2 errors &&\n-\tgrep \"error: hash mismatch for\" err &&\n-\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\ttest_line_count = 1 errors &&\n+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428420","messageId":"patch-21.21-e588c05f461-20210624T191755Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v4 21/21] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T19:23:59Z","receivedAt":"2021-06-24T19:24:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 13 +++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n object-file.c | 21 ++++++++++++---------\n object-store.h | 4 +++-\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 10 files changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 3c20f164f0f..48a3b6a7f8f 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 07af0434db6..158b9dac9b3 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -603,20 +603,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n+\tstruct object_id real_oid = *null_oid();\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n \t\tfound |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n+\t\tif (!oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n \t}\n \tif (type < 0) {\n \t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t oid_to_hex(oid), sb.buf, path);\n+\t\t oid_to_hex(&real_oid), sb.buf, path);\n \t}\n \tif (found) {\n \t\terrors_found |= ERROR_OBJECT;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 3fbc5d70777..bf860b6555e 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1421,7 +1421,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/object-file.c b/object-file.c\nindex e550ea0c7cf..923ff759e19 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1039,9 +1039,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1049,8 +1051,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1075,9 +1077,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2534,6 +2536,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n@@ -2583,9 +2586,9 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n+\t\t\tif (oideq(real_oid, null_oid()))\n+\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex 96a5970f314..9fc69016361 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -384,6 +384,7 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n@@ -507,7 +508,8 @@ int unpack_loose_header(git_zstream *stream, unsigned char *map,\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n int finalize_object_file(const char *tmpfile, const char *filename);\n int check_and_freshen_file(const char *fn, int freshen);\n \ndiff --git a/object.c b/object.c\nindex 14188453c56..5467ead3285 100644\n--- a/object.c\n+++ b/object.c\n@@ -261,7 +261,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -272,7 +272,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex 4b089fe8ec0..e6aa4442c90 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 06d38e1fae6..72386cfec0e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -490,7 +490,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex bc541af2cfc..d76293c495a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -53,6 +53,7 @@ test_expect_success 'object with hash mismatch' '\n \t(\n \t\tcd hash-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -62,7 +63,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -71,6 +72,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t(\n \t\tcd hash-type-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -80,8 +82,8 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.32.0.606.g2e440ee2c94\n\n"},{"id":"428433","messageId":"d77583dc-5dd6-7992-bf48-456f14e36207@gmail.com","threadId":"55404","inReplyTo":"patch-01.21-2e37971c016-20210624T191755Z-avarab@gmail.com","subject":"Re: [PATCH v4 01/21] fsck tests: refactor one test to use a sub-repo","fromName":"Andrei Rybak","fromEmail":"rybak.a.v@gmail.com","sentAt":"2021-06-24T22:00:04Z","receivedAt":"2021-06-24T22:00:15Z","isPatch":true,"sender":{"key":"rybak.a.v@gmail.com","avatar":"https://avatars.githubusercontent.com/u/624072?v=4"},"body":"On 24/06/2021 21:23, Ævar Arnfjörð Bjarmason wrote:\n> Refactor one of the fsck tests to use a throwaway repository. It's a\n> pervasive pattern in t1450-fsck.sh to spend a lot of effort on the\n> teardown of a tests so we're not leaving corrupt content for the next\n> test.\n> \n> We should instead simply use something like this test_create_repo\n> pattern. It's both less verbose, and makes things easier to debug as a\n> failing test can have their state left behind under -d without\n> damaging the state for other tests.\n> \n> But let's punt on that general refactoring and just change this one\n> test, I'm going to change it further in subsequent commits.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n> 1 file changed, 16 insertions(+), 18 deletions(-)\n> \n> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n> index 5071ac63a5b..1563b35f88c 100755\n> --- a/t/t1450-fsck.sh\n> +++ b/t/t1450-fsck.sh\n> @@ -48,24 +48,22 @@ remove_object () {\n> \trm \"$(sha1_file \"$1\")\"\n> }\n> \n> -test_expect_success 'object with bad sha1' '\n> -\tsha=$(echo blob | git hash-object -w --stdin) &&\n> -\told=$(test_oid_to_path \"$sha\") &&\n> -\tnew=$(dirname $old)/$(test_oid ff_2) &&\n> -\tsha=\"$(dirname $new)$(basename $new)\" &&\n> -\tmv .git/objects/$old .git/objects/$new &&\n> -\ttest_when_finished \"remove_object $sha\" &&\n> -\tgit update-index --add --cacheinfo 100644 $sha foo &&\n> -\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n> -\ttree=$(git write-tree) &&\n> -\ttest_when_finished \"remove_object $tree\" &&\n> -\tcmt=$(echo bogus | git commit-tree $tree) &&\n> -\ttest_when_finished \"remove_object $cmt\" &&\n> -\tgit update-ref refs/heads/bogus $cmt &&\n> -\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n> -\n> -\ttest_must_fail git fsck 2>out &&\n> -\ttest_i18ngrep \"$sha.*corrupt\" out\n> +test_expect_success 'object with hash mismatch' '\n> +\ttest_create_repo hash-mismatch &&\n\nThis patch was originally sent to ML on 2021-03-28:\n\thttps://lore.kernel.org/git/patch-2.6-3e547289408-20210328T025618Z-avarab@gmail.com/\n\nsince then, however, commit f0d4d398e2 (test-lib: split up and deprecate\ntest_create_repo(), 2021-05-10) has been merged ;-) so this line should\nbe:\n\n\tgit init hash-mismatch &&\n\n\n> +\t(\n> +\t\tcd hash-mismatch &&\n> +\t\toid=$(echo blob | git hash-object -w --stdin) &&\n> +\t\told=$(test_oid_to_path \"$oid\") &&\n> +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n> +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n> +\t\tmv .git/objects/$old .git/objects/$new &&\n> +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n> +\t\ttree=$(git write-tree) &&\n> +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n> +\t\tgit update-ref refs/heads/bogus $cmt &&\n> +\t\ttest_must_fail git fsck 2>out &&\n> +\t\ttest_i18ngrep \"$oid.*corrupt\" out\n> +\t)\n> '\n> \n> test_expect_success 'branch pointing to non-commit' '\n> \n\n"},{"id":"428436","messageId":"87mtrezya9.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"d77583dc-5dd6-7992-bf48-456f14e36207@gmail.com","subject":"Re: [PATCH v4 01/21] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T22:34:31Z","receivedAt":"2021-06-24T22:42:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jun 25 2021, Andrei Rybak wrote:\n\n> On 24/06/2021 21:23, Ævar Arnfjörð Bjarmason wrote:\n>> Refactor one of the fsck tests to use a throwaway repository. It's a\n>> pervasive pattern in t1450-fsck.sh to spend a lot of effort on the\n>> teardown of a tests so we're not leaving corrupt content for the next\n>> test.\n>> We should instead simply use something like this test_create_repo\n>> pattern. It's both less verbose, and makes things easier to debug as a\n>> failing test can have their state left behind under -d without\n>> damaging the state for other tests.\n>> But let's punt on that general refactoring and just change this one\n>> test, I'm going to change it further in subsequent commits.\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n>> 1 file changed, 16 insertions(+), 18 deletions(-)\n>> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n>> index 5071ac63a5b..1563b35f88c 100755\n>> --- a/t/t1450-fsck.sh\n>> +++ b/t/t1450-fsck.sh\n>> @@ -48,24 +48,22 @@ remove_object () {\n>> \trm \"$(sha1_file \"$1\")\"\n>> }\n>> -test_expect_success 'object with bad sha1' '\n>> -\tsha=$(echo blob | git hash-object -w --stdin) &&\n>> -\told=$(test_oid_to_path \"$sha\") &&\n>> -\tnew=$(dirname $old)/$(test_oid ff_2) &&\n>> -\tsha=\"$(dirname $new)$(basename $new)\" &&\n>> -\tmv .git/objects/$old .git/objects/$new &&\n>> -\ttest_when_finished \"remove_object $sha\" &&\n>> -\tgit update-index --add --cacheinfo 100644 $sha foo &&\n>> -\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n>> -\ttree=$(git write-tree) &&\n>> -\ttest_when_finished \"remove_object $tree\" &&\n>> -\tcmt=$(echo bogus | git commit-tree $tree) &&\n>> -\ttest_when_finished \"remove_object $cmt\" &&\n>> -\tgit update-ref refs/heads/bogus $cmt &&\n>> -\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n>> -\n>> -\ttest_must_fail git fsck 2>out &&\n>> -\ttest_i18ngrep \"$sha.*corrupt\" out\n>> +test_expect_success 'object with hash mismatch' '\n>> +\ttest_create_repo hash-mismatch &&\n>\n> This patch was originally sent to ML on 2021-03-28:\n> \thttps://lore.kernel.org/git/patch-2.6-3e547289408-20210328T025618Z-avarab@gmail.com/\n>\n> since then, however, commit f0d4d398e2 (test-lib: split up and deprecate\n> test_create_repo(), 2021-05-10) has been merged ;-) so this line should\n> be:\n>\n> \tgit init hash-mismatch &&\n\nThanks, you'd think that the author of that code would be in a better\nposition than most to get this right, but apparently not :)\n\nThis series originally pre-dates that work, I see I have a couple of\nmore test_create_repo() in other patches here, will fix for a v5 pending\nmore discussion on v4, thanks!\n"},{"id":"429612","messageId":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com","subject":"[PATCH v5 00/21] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:03Z","receivedAt":"2021-07-10T13:37:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting, see the examples in the commit\nmessages of 18/21, 20/21 and 21/21, to get there I've lib-ified more\nthigs in object-file.c and the general object APIs, i.e. now we'll\nreturn error codes instead of calling die() in these cases.\n\nThe fsck improvements are rather obscure & trivial in the grand scheme\nof things, but the object API improvements make it easier to work with\nin general.\n\nA trivial re-roll of v4 to s/test_create_repo/git init/g, pointed out\nby Andrei Rybak, I changed them to \"git init --bare\" while I was at\nit. For v4 see:\n\nhttps://lore.kernel.org/git/cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (21):\n fsck tests: refactor one test to use a sub-repo\n fsck tests: add test for fsck-ing an unknown type\n cat-file tests: test for missing object with -t and -s\n cat-file tests: test that --allow-unknown-type isn't on by default\n rev-list tests: test for behavior with invalid object types\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n cache.h: move object functions to object-store.h\n object-file.c: don't set \"typep\" when returning non-zero\n object-file.c: make parse_loose_header_extended() public\n object-file.c: add missing braces to loose_object_info()\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: split up ternary in parse_loose_header()\n object-file.c: stop dying in parse_loose_header()\n object-file.c: guard against future bugs in loose_object_info()\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n fsck: don't hard die on invalid object types\n object-store.h: move read_loose_object() below 'struct object_info'\n fsck: report invalid types recorded in objects\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 ++++++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 10 ---\n object-file.c | 178 +++++++++++++++++++++--------------------\n object-store.h | 62 +++++++++++---\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 10 ++-\n t/t1006-cat-file.sh | 169 ++++++++++++++++++++++++++++++++++++++\n t/t1450-fsck.sh | 64 +++++++++++----\n t/t6115-rev-list-du.sh | 11 +++\n 13 files changed, 407 insertions(+), 139 deletions(-)\n\nRange-diff against v4:\n 1: 2e37971c016 ! 1: a1259cdedcb fsck tests: refactor one test to use a sub-repo\n @@ t/t1450-fsck.sh: remove_object () {\n -\ttest_must_fail git fsck 2>out &&\n -\ttest_i18ngrep \"$sha.*corrupt\" out\n +test_expect_success 'object with hash mismatch' '\n -+\ttest_create_repo hash-mismatch &&\n ++\tgit init --bare hash-mismatch &&\n +\t(\n +\t\tcd hash-mismatch &&\n +\t\toid=$(echo blob | git hash-object -w --stdin) &&\n +\t\told=$(test_oid_to_path \"$oid\") &&\n +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n -+\t\tmv .git/objects/$old .git/objects/$new &&\n ++\t\tmv objects/$old objects/$new &&\n +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n +\t\ttree=$(git write-tree) &&\n +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n 2: 79630a99433 ! 2: 634f991d7c6 fsck tests: add test for fsck-ing an unknown type\n @@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n '\n \n +test_expect_success 'fsck hard errors on an invalid object type' '\n -+\ttest_create_repo garbage-type &&\n ++\tgit init --bare garbage-type &&\n +\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n 3: 2b5366bfb9d = 3: ce9dcc423e9 cat-file tests: test for missing object with -t and -s\n 4: ea9a5ef0920 = 4: 50a20741e86 cat-file tests: test that --allow-unknown-type isn't on by default\n 5: 8eaf0e6ddda = 5: f8d0b630d0a rev-list tests: test for behavior with invalid object types\n 6: f0e9d92414e = 6: 43335e653b8 cat-file tests: add corrupt loose object test\n 7: d797d2e8e9d = 7: a00dfea3fb8 cat-file tests: test for current --allow-unknown-type behavior\n 8: 96310a0bb59 = 8: 387d7f08e61 cache.h: move object functions to object-store.h\n 9: 54fb9189408 = 9: e9520953956 object-file.c: don't set \"typep\" when returning non-zero\n10: 9d36fcbc44a = 10: a8b408eefe6 object-file.c: make parse_loose_header_extended() public\n11: 74c308adc19 = 11: 31eee4da0e1 object-file.c: add missing braces to loose_object_info()\n12: 3f52149bfde = 12: dae5cfabd57 object-file.c: simplify unpack_loose_short_header()\n13: ba632be1520 = 13: 0d8385d8d12 object-file.c: split up ternary in parse_loose_header()\n14: ea4f446f5b1 = 14: d1522291aee object-file.c: stop dying in parse_loose_header()\n15: aacef784eab = 15: 13d4141a21b object-file.c: guard against future bugs in loose_object_info()\n16: 050cfc7808c = 16: 912c9edf362 object-file.c: return -1, not \"status\" from unpack_loose_header()\n17: 78e3152fd94 = 17: 7e101f97646 object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n18: f9bb1b799ac ! 18: 3c04065b0b0 fsck: don't hard die on invalid object types\n @@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n \n -test_expect_success 'fsck hard errors on an invalid object type' '\n +test_expect_success 'fsck error and recovery on invalid object type' '\n - \ttest_create_repo garbage-type &&\n + \tgit init --bare garbage-type &&\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n19: acbea7e2a2a = 19: ad920362594 object-store.h: move read_loose_object() below 'struct object_info'\n20: edc28de229d ! 20: 02a148af5cf fsck: report invalid types recorded in objects\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n '\n \n +test_expect_success 'object with hash and type mismatch' '\n -+\ttest_create_repo hash-type-mismatch &&\n ++\tgit init --bare hash-type-mismatch &&\n +\t(\n +\t\tcd hash-type-mismatch &&\n +\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n +\t\told=$(test_oid_to_path \"$oid\") &&\n +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n -+\t\tmv .git/objects/$old .git/objects/$new &&\n ++\t\tmv objects/$old objects/$new &&\n +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n +\t\ttree=$(git write-tree) &&\n +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n21: e588c05f461 = 21: 730e0a6f805 fsck: report invalid object type-path combinations\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429613","messageId":"patch-01.21-a1259cdedcb-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 01/21] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:04Z","receivedAt":"2021-07-10T13:37:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe should instead simply use something like this test_create_repo\npattern. It's both less verbose, and makes things easier to debug as a\nfailing test can have their state left behind under -d without\ndamaging the state for other tests.\n\nBut let's punt on that general refactoring and just change this one\ntest, I'm going to change it further in subsequent commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..7becab5ba1e 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,22 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n-\n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+test_expect_success 'object with hash mismatch' '\n+\tgit init --bare hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429614","messageId":"patch-05.21-f8d0b630d0a-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 05/21] rev-list tests: test for behavior with invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:08Z","receivedAt":"2021-07-10T13:37:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\nadded in 16950f8384a (rev-list: add --disk-usage option for\ncalculating disk usage, 2021-02-09) to test for what happens when it's\nasked to calculate the disk usage of invalid object types.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t6115-rev-list-du.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t6115-rev-list-du.sh b/t/t6115-rev-list-du.sh\nindex b4aef32b713..edb2ed55846 100755\n--- a/t/t6115-rev-list-du.sh\n+++ b/t/t6115-rev-list-du.sh\n@@ -48,4 +48,15 @@ check_du HEAD\n check_du --objects HEAD\n check_du --objects HEAD^..HEAD\n \n+test_expect_success 'setup garbage repository' '\n+\tgit clone --bare . garbage.git &&\n+\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally garbage.git/refs/tags/garbage-tag &&\n+\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n+'\n+\n test_done\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429615","messageId":"patch-02.21-634f991d7c6-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 02/21] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:05Z","receivedAt":"2021-07-10T13:37:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 7becab5ba1e..f10d6f7b7e8 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,4 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\tgit init --bare garbage-type &&\n+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_done\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429616","messageId":"patch-06.21-43335e653b8-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 06/21] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:09Z","receivedAt":"2021-07-10T13:37:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex dc01d7c4a9a..7f10a92f0e4 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -404,6 +404,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429617","messageId":"patch-04.21-50a20741e86-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 04/21] cat-file tests: test that --allow-unknown-type isn't on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:07Z","receivedAt":"2021-07-10T13:37:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the --allow-unknown-type feature\nadded in 39e4ae38804 (cat-file: teach cat-file a\n'--allow-unknown-type' option, 2015-05-03). We should check that\n--allow-unknown-type isn't on by default.\n\nBefore this change all the tests would succeed if --allow-unknown-type\nwas on by default, let's fix that by asserting that -t and -s die on a\n\"garbage\" type without --allow-unknown-type.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex b71ef94329e..dc01d7c4a9a 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -347,6 +347,20 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -363,6 +377,21 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-EOF &&\n+\terror: unable to unpack $bogus_sha1 header\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct when type is large\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429620","messageId":"patch-03.21-ce9dcc423e9-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 03/21] cat-file tests: test for missing object with -t and -s","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:06Z","receivedAt":"2021-07-10T13:37:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test for what happens when the -t and -s flags are asked to operate on\na missing object, this extends tests added in 3e370f9faf0 (t1006: add\ntests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n-s flags are the only ones that can be combined with\n--allow-unknown-type, so let's test with and without that flag.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5d2dc99b74a..b71ef94329e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,6 +315,33 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n+missing_oid=$(test_oid deadbeef)\n+test_expect_success 'error on type of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -t $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -t --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n+test_expect_success 'error on size of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -s $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -s --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n bogus_type=\"bogus\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429618","messageId":"patch-07.21-a00dfea3fb8-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 07/21] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:10Z","receivedAt":"2021-07-10T13:37:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 7f10a92f0e4..86fd2a90ca7 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -361,6 +361,46 @@ test_expect_success 'die on broken object under -t and -s without --allow-unknow\n \ttest_must_be_empty out.actual\n '\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_type $bogus_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -372,6 +412,27 @@ test_expect_success \"Size of broken object is correct\" '\n \tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n \ttest_cmp expect actual\n '\n+\n+test_expect_success 'the --allow-unknown-type option does not consider replacement refs' '\n+\tcat >expect <<-EOF &&\n+\t$bogus_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429621","messageId":"patch-08.21-387d7f08e61-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 08/21] cache.h: move object functions to object-store.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:11Z","receivedAt":"2021-07-10T13:37:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of some ancient object functions added in\ne.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n2005-06-01) from cache.h to object-store.h. This continues work\nstarted in cbd53a2193d (object-store: move object access functions to\nobject-store.h, 2018-05-15).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 10 ----------\n object-store.h | 9 +++++++++\n 2 files changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd36..32ea1ea0474 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1302,16 +1302,6 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n-\n-int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n-\n-int finalize_object_file(const char *tmpfile, const char *filename);\n-\n-/* Helper to check and \"touch\" a file */\n-int check_and_freshen_file(const char *fn, int freshen);\n \n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb5..9117115a50c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,4 +477,13 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz);\n+int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int check_object_signature(struct repository *r, const struct object_id *oid,\n+\t\t\t void *buf, unsigned long size, const char *type);\n+int finalize_object_file(const char *tmpfile, const char *filename);\n+int check_and_freshen_file(const char *fn, int freshen);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429622","messageId":"patch-10.21-a8b408eefe6-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 10/21] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:13Z","receivedAt":"2021-07-10T13:37:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 21 ++++++++-------------\n object-store.h | 3 ++-\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 9210e2e6fe4..e0ba1842272 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1340,8 +1340,9 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr,\n+\t\t struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1401,14 +1402,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1463,10 +1456,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2547,6 +2540,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2561,7 +2556,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/object-store.h b/object-store.h\nindex 9117115a50c..d443964447c 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -480,7 +480,8 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429619","messageId":"patch-09.21-e9520953956-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 09/21] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:12Z","receivedAt":"2021-07-10T13:37:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b22..9210e2e6fe4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1480,8 +1480,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429623","messageId":"patch-12.21-dae5cfabd57-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 12/21] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:15Z","receivedAt":"2021-07-10T13:37:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 60 ++++++++++++++++++--------------------------------\n object-store.h | 14 +++++++++++-\n streaming.c | 3 ++-\n 3 files changed, 37 insertions(+), 40 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 646ca7f85d6..ef3a1517fed 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1210,11 +1210,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1225,44 +1226,25 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n \tif (status < Z_OK)\n \t\treturn status;\n \n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1410,9 +1392,11 @@ static int loose_object_info(struct repository *r,\n \tunsigned long mapsize;\n \tvoid *map;\n \tgit_zstream stream;\n+\tint hdr_ret;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1446,11 +1430,10 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\n+\thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL);\n+\tif (hdr_ret < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n@@ -2555,7 +2538,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/object-store.h b/object-store.h\nindex d443964447c..31327a7f6c3 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -477,9 +477,21 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz);\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429625","messageId":"patch-11.21-31eee4da0e1-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 11/21] object-file.c: add missing braces to loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:14Z","receivedAt":"2021-07-10T13:37:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the formatting in loose_object_info() to conform with our usual\ncoding style:\n\n When there are multiple arms to a conditional and some of them\n require braces, enclose even a single line block in braces for\n consistency -- Documentation/CodingGuidelines\n\nThis formatting-only change makes a subsequent commit easier to read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex e0ba1842272..646ca7f85d6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1450,17 +1450,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t}\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -1469,8 +1472,9 @@ static int loose_object_info(struct repository *r,\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n \t\t}\n-\t} else\n+\t} else {\n \t\tgit_inflate_end(&stream);\n+\t}\n \n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429624","messageId":"patch-13.21-0d8385d8d12-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 13/21] object-file.c: split up ternary in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:16Z","receivedAt":"2021-07-10T13:37:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This minor formatting change serves to make a subsequent patch easier\nto read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex ef3a1517fed..e51cf2ca33e 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1381,7 +1381,10 @@ int parse_loose_header(const char *hdr,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\treturn type;\n }\n \n static int loose_object_info(struct repository *r,\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429631","messageId":"patch-14.21-d1522291aee-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 14/21] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:17Z","receivedAt":"2021-07-10T13:37:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Start the libification of parse_loose_header() by making it return\nerror codes and data instead of invoking die() by itself. For now\nwe'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller, but in subsequent\ncommits we'll also libify those.\n\nSince the refactoring of parse_loose_header_extended() into\nparse_loose_header() in an earlier commit, its interface accepts a\n\"unsigned long *sizep\". Rather it accepts a \"struct object_info *\",\nthat structure will be populated with information about the object.\n\nIt thus makes sense to further libify the interface so that it stops\ncalling die() when it encounters OBJ_BAD, and instead rely on its\ncallers to check the populated \"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nIn another case added in c84a1f3ed4d (sha1_file: refactor read_object,\n2017-06-21) (but the behavior pre-dated that) we did checks of \"status\n>= 0\", because at that point \"status\" had become the return value of\nparse_loose_header(). I.e. a non-negative \"enum object_type\" (unless\nwe -1, aka. OBJ_BAD).\n\nNow that parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 53 ++++++++++++++++++++++++++------------------------\n object-store.h | 13 +++++++++++--\n streaming.c | 4 +++-\n 3 files changed, 42 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex e51cf2ca33e..31263335af9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1322,9 +1322,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr,\n-\t\t struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1346,15 +1344,6 @@ int parse_loose_header(const char *hdr,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1384,7 +1373,11 @@ int parse_loose_header(const char *hdr,\n \tif (*hdr)\n \t\treturn -1;\n \n-\treturn type;\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1399,6 +1392,8 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n+\tint parsed_header = 0;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1430,6 +1425,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1440,18 +1437,20 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n-\n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\tif (!status) {\n+\t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n+\t\t\t/*\n+\t\t\t * oi->{sizep,typep} are meaningless unless\n+\t\t\t * parse_loose_header() returns >= 0.\n+\t\t\t */\n+\t\t\tparsed_header = 1;\n+\t\telse\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n+\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n-\tif (status >= 0 && oi->contentp) {\n+\tif (parsed_header && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n \t\t\t\t\t\t *oi->sizep, oid);\n \t\tif (!*oi->contentp) {\n@@ -1466,6 +1465,8 @@ static int loose_object_info(struct repository *r,\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n \treturn (status < 0) ? status : 0;\n }\n@@ -2531,6 +2532,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2547,12 +2549,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex 31327a7f6c3..65a8e4dc6a8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -492,8 +492,17 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+\n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n+\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..c3dc241d6a5 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -235,7 +236,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr),\n \t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n+\t *type < 0) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429628","messageId":"patch-15.21-13d4141a21b-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 15/21] object-file.c: guard against future bugs in loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:18Z","receivedAt":"2021-07-10T13:37:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"An earlier version of the preceding commit had a subtle bug where our\n\"type_scratch\" (later assigned to \"oi->typep\") would be uninitialized\nand used in the \"!allow_unknown\" case, at which point it would contain\na nonsensical value if we'd failed to call parse_loose_header().\n\nThe preceding commit introduced \"parsed_header\" variable to check for\nthis case, but I think we can do better, let's carry a \"oi_header\"\nvariable initially set to NULL, and only set it to \"oi\" once we're\npast parse_loose_header().\n\nThis is functionally the same thing, but hopefully makes it even more\nobvious in the future that we must not access the \"typep\" and\n\"sizep\" (or \"type_name\") unless parse_loose_header() succeeds, but\nthat accessing other fields set earlier (such as the \"disk_sizep\" set\nearlier) is OK.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 31263335af9..d41f444e6cc 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1393,7 +1393,7 @@ static int loose_object_info(struct repository *r,\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n \tenum object_type type_scratch;\n-\tint parsed_header = 0;\n+\tstruct object_info *oi_header = NULL;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1441,18 +1441,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n \t\t\t/*\n \t\t\t * oi->{sizep,typep} are meaningless unless\n-\t\t\t * parse_loose_header() returns >= 0.\n+\t\t\t * parse_loose_header() returns >= 0. Let's\n+\t\t\t * access them as \"oi_header\" (just an alias\n+\t\t\t * for \"oi\") below to make that intent clear.\n \t\t\t */\n-\t\t\tparsed_header = 1;\n+\t\t\toi_header = oi;\n \t\telse\n \t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n-\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n+\tif (!allow_unknown && oi_header && *oi_header->typep < 0)\n \t\tdie(_(\"invalid object type\"));\n \n-\tif (parsed_header && oi->contentp) {\n+\tif (oi_header && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n+\t\t\t\t\t\t *oi_header->sizep, oid);\n \t\tif (!*oi->contentp) {\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429632","messageId":"patch-16.21-912c9edf362-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 16/21] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:19Z","receivedAt":"2021-07-10T13:37:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex d41f444e6cc..956ca260518 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1229,7 +1229,7 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429626","messageId":"patch-17.21-7e101f97646-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 17/21] object-file.c: return -2 on \"header too long\" in unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:20Z","receivedAt":"2021-07-10T13:37:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return -2 saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nI tried setting up an enum just for these three return values, but I\nthink the result was less readable. Let's consider doing that if we\ngain even more return values. For now let's do the next best thing and\nenumerate our known return values, and BUG() if we encounter one we\ndon't know about.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 +++++++++++++---\n object-store.h | 6 ++++--\n t/t1006-cat-file.sh | 2 +-\n 3 files changed, 18 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 956ca260518..1866115a1c5 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1243,7 +1243,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn -2;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1264,7 +1264,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn -2;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1433,9 +1433,19 @@ static int loose_object_info(struct repository *r,\n \n \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n-\tif (hdr_ret < 0) {\n+\tswitch (hdr_ret) {\n+\tcase 0:\n+\t\tbreak;\n+\tcase -1:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n+\t\tbreak;\n+\tcase -2:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n \t}\n \tif (!status) {\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\ndiff --git a/object-store.h b/object-store.h\nindex 65a8e4dc6a8..1151ce8e820 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -481,13 +481,15 @@ int for_each_packed_object(each_packed_object_fn, void *,\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns 0 on success. Returns negative values on error. If the\n+ * header exceeds MAX_HEADER_LEN -2 will be returned.\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), -2 will still be returned from this\n+ * function to indicate that the header was too long.\n */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 86fd2a90ca7..06d38e1fae6 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -440,7 +440,7 @@ bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_t\n \n test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n \tcat >err.expect <<-EOF &&\n-\terror: unable to unpack $bogus_sha1 header\n+\terror: header for $bogus_sha1 too long, exceeds 32 bytes\n \tfatal: git cat-file: could not get object info\n \tEOF\n \n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429627","messageId":"patch-19.21-ad920362594-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 19/21] object-store.h: move read_loose_object() below 'struct object_info'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:22Z","receivedAt":"2021-07-10T13:37:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of read_loose_object() below \"struct\nobject_info\". In the next commit we'll add a \"struct object_info *\"\nparameter to it, moving it will avoid a forward declaration of the\nstruct.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-store.h | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex 94ff03072c1..72d668b1674 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -234,20 +234,6 @@ int pretend_object_file(void *, unsigned long, enum object_type,\n \n int force_object_loose(const struct object_id *oid, time_t mtime);\n \n-/*\n- * Open the loose object at path, check its hash, and return the contents,\n- * type, and size. If the object is a blob, then \"contents\" may return NULL,\n- * to allow streaming of large blobs.\n- *\n- * Returns 0 on success, negative on error (details may be written to stderr).\n- */\n-int read_loose_object(const char *path,\n-\t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents,\n-\t\t unsigned int oi_flags);\n-\n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n \n@@ -388,6 +374,20 @@ int oid_object_info_extended(struct repository *r,\n \t\t\t const struct object_id *,\n \t\t\t struct object_info *, unsigned flags);\n \n+/*\n+ * Open the loose object at path, check its hash, and return the contents,\n+ * type, and size. If the object is a blob, then \"contents\" may return NULL,\n+ * to allow streaming of large blobs.\n+ *\n+ * Returns 0 on success, negative on error (details may be written to stderr).\n+ */\n+int read_loose_object(const char *path,\n+\t\t const struct object_id *expected_oid,\n+\t\t enum object_type *type,\n+\t\t unsigned long *size,\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n+\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429630","messageId":"patch-18.21-3c04065b0b0-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 18/21] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:21Z","receivedAt":"2021-07-10T13:37:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: hash mismatch for (expected )\n error: : object corrupt or missing: \n [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it. See f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) for the introduction of read_loose_object().\n\nWhy are we complaining about a \"hash mismatch\" for an object of a type\nwe don't know about? We shouldn't. This is the bare minimal change\nneeded to not make fsck hard die on a repository that's been corrupted\nin this manner. In subsequent commits we'll teach fsck to recognize\nthis particular type of corruption and emit a better error message.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 11 ++++++++---\n object-store.h | 3 ++-\n t/t1450-fsck.sh | 14 +++++++-------\n 4 files changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..082dadd5629 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -601,7 +601,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tif (read_loose_object(path, oid, &type, &size, &contents,\n+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\ndiff --git a/object-file.c b/object-file.c\nindex 1866115a1c5..8fb55fc6f58 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2536,7 +2536,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n@@ -2544,6 +2545,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \toi.typep = type;\n \toi.sizep = size;\n \n@@ -2566,8 +2568,11 @@ int read_loose_object(const char *path,\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n+\tif (!allow_unknown && *type < 0) {\n+\t\terror(_(\"header for %s declares an unknown type\"), path);\n+\t\tgit_inflate_end(&stream);\n+\t\tgoto out;\n+\t}\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex 1151ce8e820..94ff03072c1 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,7 +245,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f10d6f7b7e8..d8303db9709 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,16 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \tgit init --bare garbage-type &&\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n-\tfatal: invalid object type\n-\tEOF\n-\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n-\ttest_cmp err.expect err.actual &&\n-\ttest_must_be_empty out.actual\n+\ttest_must_fail git -C garbage-type fsck >out 2>err &&\n+\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\ttest_line_count = 2 errors &&\n+\tgrep \"error: hash mismatch for\" err &&\n+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"dangling blob $empty_blob\" out\n '\n \n test_done\n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429629","messageId":"patch-20.21-02a148af5cf-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 20/21] fsck: report invalid types recorded in objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:23Z","receivedAt":"2021-07-10T13:37:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the work in the preceding commit and improve the error on:\n\n $ git hash-object --stdin -w -t garbage --literally (expected )\n error: : object corrupt or missing: \n [ other fsck output ]\n\nTo instead emit:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nThe complaint about a \"hash mismatch\" was simply an emergent property\nof how we'd fall though from read_loose_object() into fsck_loose()\nwhen we didn't get the data we expected. Now we'll correctly note that\nthe object type is invalid.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 22 ++++++++++++++++++----\n object-file.c | 13 +++++--------\n object-store.h | 4 ++--\n t/t1450-fsck.sh | 24 +++++++++++++++++++++---\n 4 files changed, 46 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 082dadd5629..07af0434db6 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,12 +600,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n-\n-\tif (read_loose_object(path, oid, &type, &size, &contents,\n-\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n-\t\terrors_found |= ERROR_OBJECT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n+\tstruct object_info oi;\n+\tint found = 0;\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\n+\t}\n+\tif (type < 0) {\n+\t\tfound |= ERROR_OBJECT;\n+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t oid_to_hex(oid), sb.buf, path);\n+\t}\n+\tif (found) {\n+\t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 8fb55fc6f58..e550ea0c7cf 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2534,9 +2534,8 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n@@ -2544,10 +2543,9 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tenum object_type *type = oi->typep;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2563,7 +2561,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n@@ -2585,8 +2583,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex 72d668b1674..96a5970f314 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -376,6 +376,7 @@ int oid_object_info_extended(struct repository *r,\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -383,9 +384,8 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n /*\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex d8303db9709..da2658155c7 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\tgit init --bare hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n@@ -869,9 +888,8 @@ test_expect_success 'fsck error and recovery on invalid object type' '\n \tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally out 2>err &&\n \tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n-\ttest_line_count = 2 errors &&\n-\tgrep \"error: hash mismatch for\" err &&\n-\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\ttest_line_count = 1 errors &&\n+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"429633","messageId":"patch-21.21-730e0a6f805-20210710T133203Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v5 21/21] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T13:37:24Z","receivedAt":"2021-07-10T13:37:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 13 +++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n object-file.c | 21 ++++++++++++---------\n object-store.h | 4 +++-\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 10 files changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 3c20f164f0f..48a3b6a7f8f 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 07af0434db6..158b9dac9b3 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -603,20 +603,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n+\tstruct object_id real_oid = *null_oid();\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n \t\tfound |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n+\t\tif (!oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n \t}\n \tif (type < 0) {\n \t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t oid_to_hex(oid), sb.buf, path);\n+\t\t oid_to_hex(&real_oid), sb.buf, path);\n \t}\n \tif (found) {\n \t\terrors_found |= ERROR_OBJECT;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 3fbc5d70777..bf860b6555e 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1421,7 +1421,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/object-file.c b/object-file.c\nindex e550ea0c7cf..923ff759e19 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1039,9 +1039,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1049,8 +1051,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1075,9 +1077,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2534,6 +2536,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n@@ -2583,9 +2586,9 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n+\t\t\tif (oideq(real_oid, null_oid()))\n+\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex 96a5970f314..9fc69016361 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -384,6 +384,7 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n@@ -507,7 +508,8 @@ int unpack_loose_header(git_zstream *stream, unsigned char *map,\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n int finalize_object_file(const char *tmpfile, const char *filename);\n int check_and_freshen_file(const char *fn, int freshen);\n \ndiff --git a/object.c b/object.c\nindex 14188453c56..5467ead3285 100644\n--- a/object.c\n+++ b/object.c\n@@ -261,7 +261,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -272,7 +272,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex 4b089fe8ec0..e6aa4442c90 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 06d38e1fae6..72386cfec0e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -490,7 +490,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex da2658155c7..7d0d57564b5 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -53,6 +53,7 @@ test_expect_success 'object with hash mismatch' '\n \t(\n \t\tcd hash-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -62,7 +63,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -71,6 +72,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t(\n \t\tcd hash-type-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -80,8 +82,8 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.32.0.636.g43e71d69cff\n\n"},{"id":"434826","messageId":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com","subject":"[PATCH v6 00/22] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:57:55Z","receivedAt":"2021-09-07T10:58:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting, see the examples in the commit\nmessages of 19/22, 21/22 and 22/22. To get there I've lib-ified more\nthigs in object-file.c and the general object APIs, i.e. now we'll\nreturn error codes instead of calling die() in these cases.\n\nThis series has been in \"needs review\" state for a while. This re-roll\nis mainly to bump it for the list's attention, but while I was at it I\naddressed point from Jonathan Tan raised in a previous round: use an\nenum instead of int for the unpack_loose_header() return value.\n\nI think the v3 of this got a detailed review, and the v3->v4 delta\nwasn't that big (although one commit in particular is a bit tricky):\n\n https://lore.kernel.org/git/cover-00.21-00000000000-20210624T191754Z-avarab@gmail.com/\n\nI.e. it's just this part being substantially different from v3:\n\n https://lore.kernel.org/git/patch-12.21-3f52149bfde-20210624T191755Z-avarab@gmail.com/\n https://lore.kernel.org/git/patch-13.21-ba632be1520-20210624T191755Z-avarab@gmail.com/\n https://lore.kernel.org/git/patch-14.21-ea4f446f5b1-20210624T191755Z-avarab@gmail.com/\n\nThe v5 was then a trivial change of moving away from the\ntest_create_repo() test helper:\nhttps://lore.kernel.org/git/cover-00.21-00000000000-20210710T133203Z-avarab@gmail.com/\n\nPerhaps this whole thing just needs to be split up, e.g. the first 7\ncommits could be some \"improve fsck tests\" series, the middle part\nsome general small refactoring, and the real meaty work as of 14/22\nits own third series...\n\nBut I've resisted that because I think while this is rather long the\nfirst parts adding tests for missing behavior should assure reviewers\nthat the later changes are all properly tested.\n\nI know the end-goal here isn't all that exciting in itself, but I've\nreally wanted to improve various things around fsck-ing, bad object\nreporting etc., and that's been stalled on this first step for a\nwhile.\n\nÆvar Arnfjörð Bjarmason (22):\n fsck tests: refactor one test to use a sub-repo\n fsck tests: add test for fsck-ing an unknown type\n cat-file tests: test for missing object with -t and -s\n cat-file tests: test that --allow-unknown-type isn't on by default\n rev-list tests: test for behavior with invalid object types\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n object-file.c: don't set \"typep\" when returning non-zero\n cache.h: move object functions to object-store.h\n object-file.c: make parse_loose_header_extended() public\n object-file.c: add missing braces to loose_object_info()\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: split up ternary in parse_loose_header()\n object-file.c: stop dying in parse_loose_header()\n object-file.c: guard against future bugs in loose_object_info()\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n object-file.c: use \"enum\" return type for unpack_loose_header()\n fsck: don't hard die on invalid object types\n object-store.h: move read_loose_object() below 'struct object_info'\n fsck: report invalid types recorded in objects\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 ++++++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 10 ---\n object-file.c | 178 +++++++++++++++++++++--------------------\n object-store.h | 75 ++++++++++++++---\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 29 ++++---\n t/t1006-cat-file.sh | 169 ++++++++++++++++++++++++++++++++++++++\n t/t1450-fsck.sh | 64 +++++++++++----\n t/t6115-rev-list-du.sh | 11 +++\n 13 files changed, 432 insertions(+), 146 deletions(-)\n\nRange-diff against v5:\n 1: a1259cdedcb = 1: ebe89f65354 fsck tests: refactor one test to use a sub-repo\n 2: 634f991d7c6 = 2: 9072eef3be3 fsck tests: add test for fsck-ing an unknown type\n 3: ce9dcc423e9 = 3: d442a309178 cat-file tests: test for missing object with -t and -s\n 4: 50a20741e86 = 4: 0358273022f cat-file tests: test that --allow-unknown-type isn't on by default\n 5: f8d0b630d0a = 5: 82db40ebf8a rev-list tests: test for behavior with invalid object types\n 6: 43335e653b8 = 6: d1ffd21acc5 cat-file tests: add corrupt loose object test\n 7: a00dfea3fb8 = 7: 22ab12c2282 cat-file tests: test for current --allow-unknown-type behavior\n 9: e9520953956 = 8: 38e4266772d object-file.c: don't set \"typep\" when returning non-zero\n 8: 387d7f08e61 = 9: 5b9278e7bb4 cache.h: move object functions to object-store.h\n10: a8b408eefe6 = 10: b15ad53414b object-file.c: make parse_loose_header_extended() public\n11: 31eee4da0e1 = 11: 326eb74545d object-file.c: add missing braces to loose_object_info()\n12: dae5cfabd57 = 12: 4f829e9b727 object-file.c: simplify unpack_loose_short_header()\n13: 0d8385d8d12 = 13: 90489d9e6ec object-file.c: split up ternary in parse_loose_header()\n14: d1522291aee = 14: 7c9819d37c5 object-file.c: stop dying in parse_loose_header()\n15: 13d4141a21b = 15: 3fb660ff944 object-file.c: guard against future bugs in loose_object_info()\n16: 912c9edf362 = 16: 9e7dbfb4aa3 object-file.c: return -1, not \"status\" from unpack_loose_header()\n17: 7e101f97646 ! 17: f28c4f0dfb4 object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n @@ Commit message\n MAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n \n - I tried setting up an enum just for these three return values, but I\n - think the result was less readable. Let's consider doing that if we\n - gain even more return values. For now let's do the next best thing and\n - enumerate our known return values, and BUG() if we encounter one we\n - don't know about.\n -\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## object-file.c ##\n -: ----------- > 18: 1b7173a5b5b object-file.c: use \"enum\" return type for unpack_loose_header()\n18: 3c04065b0b0 = 19: ad1614dbb8d fsck: don't hard die on invalid object types\n19: ad920362594 = 20: 3bf3cf2299d object-store.h: move read_loose_object() below 'struct object_info'\n20: 02a148af5cf = 21: 974f650cddf fsck: report invalid types recorded in objects\n21: 730e0a6f805 ! 22: 804673a17b0 fsck: report invalid object type-path combinations\n @@ object-store.h: int oid_object_info_extended(struct repository *r,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n -@@ object-store.h: int unpack_loose_header(git_zstream *stream, unsigned char *map,\n +@@ object-store.h: enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434827","messageId":"patch-v6-01.22-ebe89f65354-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 01/22] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:57:56Z","receivedAt":"2021-09-07T10:58:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe should instead simply use something like this test_create_repo\npattern. It's both less verbose, and makes things easier to debug as a\nfailing test can have their state left behind under -d without\ndamaging the state for other tests.\n\nBut let's punt on that general refactoring and just change this one\ntest, I'm going to change it further in subsequent commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 34 ++++++++++++++++------------------\n 1 file changed, 16 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..7becab5ba1e 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,22 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n-\n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+test_expect_success 'object with hash mismatch' '\n+\tgit init --bare hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434828","messageId":"patch-v6-02.22-9072eef3be3-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 02/22] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:57:57Z","receivedAt":"2021-09-07T10:58:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 7becab5ba1e..f10d6f7b7e8 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,4 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\tgit init --bare garbage-type &&\n+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_done\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434829","messageId":"patch-v6-03.22-d442a309178-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 03/22] cat-file tests: test for missing object with -t and -s","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:57:58Z","receivedAt":"2021-09-07T10:58:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test for what happens when the -t and -s flags are asked to operate on\na missing object, this extends tests added in 3e370f9faf0 (t1006: add\ntests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n-s flags are the only ones that can be combined with\n--allow-unknown-type, so let's test with and without that flag.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..3a7b138fe4e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,6 +315,33 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n+missing_oid=$(test_oid deadbeef)\n+test_expect_success 'error on type of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -t $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -t --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n+test_expect_success 'error on size of missing object' '\n+\tcat >expect.err <<-\\EOF &&\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\ttest_must_fail git cat-file -s $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err &&\n+\n+\ttest_must_fail git cat-file -s --allow-unknown-type $missing_oid >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect.err err\n+'\n+\n bogus_type=\"bogus\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434830","messageId":"patch-v6-04.22-0358273022f-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 04/22] cat-file tests: test that --allow-unknown-type isn't on by default","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:57:59Z","receivedAt":"2021-09-07T10:58:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the --allow-unknown-type feature\nadded in 39e4ae38804 (cat-file: teach cat-file a\n'--allow-unknown-type' option, 2015-05-03). We should check that\n--allow-unknown-type isn't on by default.\n\nBefore this change all the tests would succeed if --allow-unknown-type\nwas on by default, let's fix that by asserting that -t and -s die on a\n\"garbage\" type without --allow-unknown-type.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 3a7b138fe4e..5e05ea0861e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -347,6 +347,20 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -363,6 +377,21 @@ bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n+test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n+\tcat >err.expect <<-EOF &&\n+\terror: unable to unpack $bogus_sha1 header\n+\tfatal: git cat-file: could not get object info\n+\tEOF\n+\n+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual &&\n+\n+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\ttest_must_be_empty out.actual\n+'\n+\n test_expect_success \"Type of broken object is correct when type is large\" '\n \techo $bogus_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434831","messageId":"patch-v6-05.22-82db40ebf8a-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 05/22] rev-list tests: test for behavior with invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:00Z","receivedAt":"2021-09-07T10:58:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\nadded in 16950f8384a (rev-list: add --disk-usage option for\ncalculating disk usage, 2021-02-09) to test for what happens when it's\nasked to calculate the disk usage of invalid object types.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t6115-rev-list-du.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t6115-rev-list-du.sh b/t/t6115-rev-list-du.sh\nindex b4aef32b713..edb2ed55846 100755\n--- a/t/t6115-rev-list-du.sh\n+++ b/t/t6115-rev-list-du.sh\n@@ -48,4 +48,15 @@ check_du HEAD\n check_du --objects HEAD\n check_du --objects HEAD^..HEAD\n \n+test_expect_success 'setup garbage repository' '\n+\tgit clone --bare . garbage.git &&\n+\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally garbage.git/refs/tags/garbage-tag &&\n+\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n+'\n+\n test_done\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434832","messageId":"patch-v6-06.22-d1ffd21acc5-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 06/22] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:01Z","receivedAt":"2021-09-07T10:58:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5e05ea0861e..8f3516db188 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -404,6 +404,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434833","messageId":"patch-v6-07.22-22ab12c2282-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 07/22] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:02Z","receivedAt":"2021-09-07T10:58:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 8f3516db188..98729f1edfc 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -361,6 +361,46 @@ test_expect_success 'die on broken object under -t and -s without --allow-unknow\n \ttest_must_be_empty out.actual\n '\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_type $bogus_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n@@ -372,6 +412,27 @@ test_expect_success \"Size of broken object is correct\" '\n \tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n \ttest_cmp expect actual\n '\n+\n+test_expect_success 'the --allow-unknown-type option does not consider replacement refs' '\n+\tcat >expect <<-EOF &&\n+\t$bogus_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n bogus_content=\"bogus\"\n bogus_size=$(strlen \"$bogus_content\")\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434834","messageId":"patch-v6-08.22-38e4266772d-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 08/22] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:03Z","receivedAt":"2021-09-07T10:58:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex a8be8994814..bda3497d5ca 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1503,8 +1503,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434835","messageId":"patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 09/22] cache.h: move object functions to object-store.h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:04Z","receivedAt":"2021-09-07T10:58:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of some ancient object functions added in\ne.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n2005-06-01) from cache.h to object-store.h. This continues work\nstarted in cbd53a2193d (object-store: move object access functions to\nobject-store.h, 2018-05-15).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 10 ----------\n object-store.h | 9 +++++++++\n 2 files changed, 9 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d23de693680..11a04a93436 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1313,16 +1313,6 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n-\n-int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n-\n-int finalize_object_file(const char *tmpfile, const char *filename);\n-\n-/* Helper to check and \"touch\" a file */\n-int check_and_freshen_file(const char *fn, int freshen);\n \n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\ndiff --git a/object-store.h b/object-store.h\nindex d24915ced1b..eb4876ec983 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -485,4 +485,13 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz);\n+int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int check_object_signature(struct repository *r, const struct object_id *oid,\n+\t\t\t void *buf, unsigned long size, const char *type);\n+int finalize_object_file(const char *tmpfile, const char *filename);\n+int check_and_freshen_file(const char *fn, int freshen);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434836","messageId":"patch-v6-10.22-b15ad53414b-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 10/22] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:05Z","receivedAt":"2021-09-07T10:58:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 21 ++++++++-------------\n object-store.h | 3 ++-\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex bda3497d5ca..7a47af68bd8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1363,8 +1363,9 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr,\n+\t\t struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1424,14 +1425,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1486,10 +1479,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2573,6 +2566,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2587,7 +2582,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/object-store.h b/object-store.h\nindex eb4876ec983..25e641a606f 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -488,7 +488,8 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434837","messageId":"patch-v6-11.22-326eb74545d-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 11/22] object-file.c: add missing braces to loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:06Z","receivedAt":"2021-09-07T10:58:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the formatting in loose_object_info() to conform with our usual\ncoding style:\n\n When there are multiple arms to a conditional and some of them\n require braces, enclose even a single line block in braces for\n consistency -- Documentation/CodingGuidelines\n\nThis formatting-only change makes a subsequent commit easier to read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 7a47af68bd8..878a4298c9b 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1473,17 +1473,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t}\n \n \tif (status >= 0 && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n@@ -1492,8 +1495,9 @@ static int loose_object_info(struct repository *r,\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n \t\t}\n-\t} else\n+\t} else {\n \t\tgit_inflate_end(&stream);\n+\t}\n \n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434838","messageId":"patch-v6-12.22-4f829e9b727-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 12/22] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:07Z","receivedAt":"2021-09-07T10:58:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 60 ++++++++++++++++++--------------------------------\n object-store.h | 14 +++++++++++-\n streaming.c | 3 ++-\n 3 files changed, 37 insertions(+), 40 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 878a4298c9b..2dd4cdd1ae0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1233,11 +1233,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1248,44 +1249,25 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n \tif (status < Z_OK)\n \t\treturn status;\n \n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1433,9 +1415,11 @@ static int loose_object_info(struct repository *r,\n \tunsigned long mapsize;\n \tvoid *map;\n \tgit_zstream stream;\n+\tint hdr_ret;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1469,11 +1453,10 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\n+\thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL);\n+\tif (hdr_ret < 0) {\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n@@ -2581,7 +2564,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/object-store.h b/object-store.h\nindex 25e641a606f..4064710ae29 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -485,9 +485,21 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz);\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\n int check_object_signature(struct repository *r, const struct object_id *oid,\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434839","messageId":"patch-v6-14.22-7c9819d37c5-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 14/22] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:09Z","receivedAt":"2021-09-07T10:58:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Start the libification of parse_loose_header() by making it return\nerror codes and data instead of invoking die() by itself. For now\nwe'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller, but in subsequent\ncommits we'll also libify those.\n\nSince the refactoring of parse_loose_header_extended() into\nparse_loose_header() in an earlier commit, its interface accepts a\n\"unsigned long *sizep\". Rather it accepts a \"struct object_info *\",\nthat structure will be populated with information about the object.\n\nIt thus makes sense to further libify the interface so that it stops\ncalling die() when it encounters OBJ_BAD, and instead rely on its\ncallers to check the populated \"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nIn another case added in c84a1f3ed4d (sha1_file: refactor read_object,\n2017-06-21) (but the behavior pre-dated that) we did checks of \"status\n>= 0\", because at that point \"status\" had become the return value of\nparse_loose_header(). I.e. a non-negative \"enum object_type\" (unless\nwe -1, aka. OBJ_BAD).\n\nNow that parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 53 ++++++++++++++++++++++++++------------------------\n object-store.h | 13 +++++++++++--\n streaming.c | 4 +++-\n 3 files changed, 42 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 7c6a865a6c0..d656960422d 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1345,9 +1345,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr,\n-\t\t struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1369,15 +1367,6 @@ int parse_loose_header(const char *hdr,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1407,7 +1396,11 @@ int parse_loose_header(const char *hdr,\n \tif (*hdr)\n \t\treturn -1;\n \n-\treturn type;\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1422,6 +1415,8 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n+\tint parsed_header = 0;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1453,6 +1448,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1463,18 +1460,20 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t}\n-\n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\tif (!status) {\n+\t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n+\t\t\t/*\n+\t\t\t * oi->{sizep,typep} are meaningless unless\n+\t\t\t * parse_loose_header() returns >= 0.\n+\t\t\t */\n+\t\t\tparsed_header = 1;\n+\t\telse\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n+\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n-\tif (status >= 0 && oi->contentp) {\n+\tif (parsed_header && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n \t\t\t\t\t\t *oi->sizep, oid);\n \t\tif (!*oi->contentp) {\n@@ -1489,6 +1488,8 @@ static int loose_object_info(struct repository *r,\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n \treturn (status < 0) ? status : 0;\n }\n@@ -2557,6 +2558,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2573,12 +2575,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex 4064710ae29..584bf5556af 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -500,8 +500,17 @@ int for_each_packed_object(each_packed_object_fn, void *,\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\n \t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+\n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n+\n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\n int finalize_object_file(const char *tmpfile, const char *filename);\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..c3dc241d6a5 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -235,7 +236,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr),\n \t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n+\t *type < 0) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434840","messageId":"patch-v6-13.22-90489d9e6ec-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 13/22] object-file.c: split up ternary in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:08Z","receivedAt":"2021-09-07T10:58:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This minor formatting change serves to make a subsequent patch easier\nto read.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 2dd4cdd1ae0..7c6a865a6c0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1404,7 +1404,10 @@ int parse_loose_header(const char *hdr,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\treturn type;\n }\n \n static int loose_object_info(struct repository *r,\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434841","messageId":"patch-v6-16.22-9e7dbfb4aa3-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 16/22] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:11Z","receivedAt":"2021-09-07T10:58:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex ae6a37ab5fb..11df4485147 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1252,7 +1252,7 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434842","messageId":"patch-v6-17.22-f28c4f0dfb4-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 17/22] object-file.c: return -2 on \"header too long\" in unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:12Z","receivedAt":"2021-09-07T10:58:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return -2 saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 16 +++++++++++++---\n object-store.h | 6 ++++--\n t/t1006-cat-file.sh | 2 +-\n 3 files changed, 18 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 11df4485147..0cb5287d3ef 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1266,7 +1266,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn -2;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1287,7 +1287,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn -2;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1456,9 +1456,19 @@ static int loose_object_info(struct repository *r,\n \n \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n-\tif (hdr_ret < 0) {\n+\tswitch (hdr_ret) {\n+\tcase 0:\n+\t\tbreak;\n+\tcase -1:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n+\t\tbreak;\n+\tcase -2:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n \t}\n \tif (!status) {\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\ndiff --git a/object-store.h b/object-store.h\nindex 584bf5556af..e896b813f24 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -489,13 +489,15 @@ int for_each_packed_object(each_packed_object_fn, void *,\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns 0 on success. Returns negative values on error. If the\n+ * header exceeds MAX_HEADER_LEN -2 will be returned.\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), -2 will still be returned from this\n+ * function to indicate that the header was too long.\n */\n int unpack_loose_header(git_zstream *stream, unsigned char *map,\n \t\t\tunsigned long mapsize, void *buffer,\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 98729f1edfc..43a9f4e7f0c 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -440,7 +440,7 @@ bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_t\n \n test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n \tcat >err.expect <<-EOF &&\n-\terror: unable to unpack $bogus_sha1 header\n+\terror: header for $bogus_sha1 too long, exceeds 32 bytes\n \tfatal: git cat-file: could not get object info\n \tEOF\n \n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434843","messageId":"patch-v6-15.22-3fb660ff944-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 15/22] object-file.c: guard against future bugs in loose_object_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:10Z","receivedAt":"2021-09-07T10:59:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"An earlier version of the preceding commit had a subtle bug where our\n\"type_scratch\" (later assigned to \"oi->typep\") would be uninitialized\nand used in the \"!allow_unknown\" case, at which point it would contain\na nonsensical value if we'd failed to call parse_loose_header().\n\nThe preceding commit introduced \"parsed_header\" variable to check for\nthis case, but I think we can do better, let's carry a \"oi_header\"\nvariable initially set to NULL, and only set it to \"oi\" once we're\npast parse_loose_header().\n\nThis is functionally the same thing, but hopefully makes it even more\nobvious in the future that we must not access the \"typep\" and\n\"sizep\" (or \"type_name\") unless parse_loose_header() succeeds, but\nthat accessing other fields set earlier (such as the \"disk_sizep\" set\nearlier) is OK.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex d656960422d..ae6a37ab5fb 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1416,7 +1416,7 @@ static int loose_object_info(struct repository *r,\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n \tenum object_type type_scratch;\n-\tint parsed_header = 0;\n+\tstruct object_info *oi_header = NULL;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1464,18 +1464,20 @@ static int loose_object_info(struct repository *r,\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n \t\t\t/*\n \t\t\t * oi->{sizep,typep} are meaningless unless\n-\t\t\t * parse_loose_header() returns >= 0.\n+\t\t\t * parse_loose_header() returns >= 0. Let's\n+\t\t\t * access them as \"oi_header\" (just an alias\n+\t\t\t * for \"oi\") below to make that intent clear.\n \t\t\t */\n-\t\t\tparsed_header = 1;\n+\t\t\toi_header = oi;\n \t\telse\n \t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \t}\n-\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n+\tif (!allow_unknown && oi_header && *oi_header->typep < 0)\n \t\tdie(_(\"invalid object type\"));\n \n-\tif (parsed_header && oi->contentp) {\n+\tif (oi_header && oi->contentp) {\n \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n+\t\t\t\t\t\t *oi_header->sizep, oid);\n \t\tif (!*oi->contentp) {\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434844","messageId":"patch-v6-18.22-1b7173a5b5b-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 18/22] object-file.c: use \"enum\" return type for unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:13Z","receivedAt":"2021-09-07T10:59:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In the preceding commits we changed and documented\nunpack_loose_header() from return any negative value or zero, to only\n-2, -1 or 0. Let's instead add an \"enum unpack_loose_header_result\"\ntype and use it, and have the compiler assert that we're exhaustively\ncovering all return values. This gets rid of the need for having a\n\"default\" BUG() case in loose_object_info().\n\nI'm on the fence about whether this is more readable or worth it, but\nsince it was suggested in [1] to do this let's go for it.\n\n1. https://lore.kernel.org/git/20210527175433.2673306-1-jonathantanmy@google.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 20 ++++++++++----------\n object-store.h | 27 ++++++++++++++++++++-------\n streaming.c | 27 ++++++++++++++++-----------\n 3 files changed, 46 insertions(+), 28 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 0cb5287d3ef..9484c7ce2be 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1233,10 +1233,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz,\n-\t\t\tstruct strbuf *header)\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *header)\n {\n \tint status;\n \n@@ -1411,7 +1413,7 @@ static int loose_object_info(struct repository *r,\n \tunsigned long mapsize;\n \tvoid *map;\n \tgit_zstream stream;\n-\tint hdr_ret;\n+\tenum unpack_loose_header_result hdr_ret;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n@@ -1457,18 +1459,16 @@ static int loose_object_info(struct repository *r,\n \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n \tswitch (hdr_ret) {\n-\tcase 0:\n+\tcase UNPACK_LOOSE_HEADER_RESULT_OK:\n \t\tbreak;\n-\tcase -1:\n+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t\tbreak;\n-\tcase -2:\n+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG:\n \t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n \t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n \t\tbreak;\n-\tdefault:\n-\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n \t}\n \tif (!status) {\n \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\ndiff --git a/object-store.h b/object-store.h\nindex e896b813f24..ac55b02f15a 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -485,23 +485,36 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+enum unpack_loose_header_result {\n+\tUNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG = -2,\n+\tUNPACK_LOOSE_HEADER_RESULT_BAD = -1,\n+\tUNPACK_LOOSE_HEADER_RESULT_OK,\n+\n+};\n+\n /**\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error. If the\n- * header exceeds MAX_HEADER_LEN -2 will be returned.\n+ * Returns UNPACK_LOOSE_HEADER_RESULT_OK on success. Returns\n+ * UNPACK_LOOSE_HEADER_RESULT_BAD values on error, or if the header\n+ * exceeds MAX_HEADER_LEN UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG will\n+ * be returned.\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header(), -2 will still be returned from this\n- * function to indicate that the header was too long.\n+ * with parse_loose_header(), UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG\n+ * will still be returned from this function to indicate that the\n+ * header was too long.\n */\n-int unpack_loose_header(git_zstream *stream, unsigned char *map,\n-\t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *hdrbuf);\n \n /**\n * parse_loose_header() parses the starting \" \\0\" of an\ndiff --git a/streaming.c b/streaming.c\nindex c3dc241d6a5..3e5045c004d 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -224,24 +224,25 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t enum object_type *type)\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tenum unpack_loose_header_result hdr_ret;\n \toi.sizep = &st->size;\n \toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n-\tif ((unpack_loose_header(&st->z,\n-\t\t\t\t st->u.loose.mapped,\n-\t\t\t\t st->u.loose.mapsize,\n-\t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr),\n-\t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n-\t *type < 0) {\n-\t\tgit_inflate_end(&st->z);\n-\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-\t\treturn -1;\n+\thdr_ret = unpack_loose_header(&st->z, st->u.loose.mapped,\n+\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n+\t\t\t\t sizeof(st->u.loose.hdr), NULL);\n+\tswitch (hdr_ret) {\n+\tcase UNPACK_LOOSE_HEADER_RESULT_OK:\n+\t\tbreak;\n+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD:\n+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG:\n+\t\tgoto error;\n \t}\n+\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n+\t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n \tst->u.loose.hdr_avail = st->z.total_out;\n@@ -250,6 +251,10 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->read = read_istream_loose;\n \n \treturn 0;\n+error:\n+\tgit_inflate_end(&st->z);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn -1;\n }\n \n \n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434845","messageId":"patch-v6-19.22-ad1614dbb8d-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 19/22] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:14Z","receivedAt":"2021-09-07T10:59:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: hash mismatch for (expected )\n error: : object corrupt or missing: \n [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it. See f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) for the introduction of read_loose_object().\n\nWhy are we complaining about a \"hash mismatch\" for an object of a type\nwe don't know about? We shouldn't. This is the bare minimal change\nneeded to not make fsck hard die on a repository that's been corrupted\nin this manner. In subsequent commits we'll teach fsck to recognize\nthis particular type of corruption and emit a better error message.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 11 ++++++++---\n object-store.h | 3 ++-\n t/t1450-fsck.sh | 14 +++++++-------\n 4 files changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..082dadd5629 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -601,7 +601,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tif (read_loose_object(path, oid, &type, &size, &contents,\n+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\ndiff --git a/object-file.c b/object-file.c\nindex 9484c7ce2be..0e6937fad73 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2562,7 +2562,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n@@ -2570,6 +2571,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \toi.typep = type;\n \toi.sizep = size;\n \n@@ -2592,8 +2594,11 @@ int read_loose_object(const char *path,\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n+\tif (!allow_unknown && *type < 0) {\n+\t\terror(_(\"header for %s declares an unknown type\"), path);\n+\t\tgit_inflate_end(&stream);\n+\t\tgoto out;\n+\t}\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/object-store.h b/object-store.h\nindex ac55b02f15a..c268662f5ba 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -253,7 +253,8 @@ int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n \t\t enum object_type *type,\n \t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f10d6f7b7e8..d8303db9709 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -863,16 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \tgit init --bare garbage-type &&\n \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n-\tfatal: invalid object type\n-\tEOF\n-\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n-\ttest_cmp err.expect err.actual &&\n-\ttest_must_be_empty out.actual\n+\ttest_must_fail git -C garbage-type fsck >out 2>err &&\n+\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\ttest_line_count = 2 errors &&\n+\tgrep \"error: hash mismatch for\" err &&\n+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\tgrep \"dangling blob $empty_blob\" out\n '\n \n test_done\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434846","messageId":"patch-v6-20.22-3bf3cf2299d-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 20/22] object-store.h: move read_loose_object() below 'struct object_info'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:15Z","receivedAt":"2021-09-07T10:59:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of read_loose_object() below \"struct\nobject_info\". In the next commit we'll add a \"struct object_info *\"\nparameter to it, moving it will avoid a forward declaration of the\nstruct.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-store.h | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex c268662f5ba..dc638335e7d 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -242,20 +242,6 @@ int pretend_object_file(void *, unsigned long, enum object_type,\n \n int force_object_loose(const struct object_id *oid, time_t mtime);\n \n-/*\n- * Open the loose object at path, check its hash, and return the contents,\n- * type, and size. If the object is a blob, then \"contents\" may return NULL,\n- * to allow streaming of large blobs.\n- *\n- * Returns 0 on success, negative on error (details may be written to stderr).\n- */\n-int read_loose_object(const char *path,\n-\t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents,\n-\t\t unsigned int oi_flags);\n-\n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\n \n@@ -396,6 +382,20 @@ int oid_object_info_extended(struct repository *r,\n \t\t\t const struct object_id *,\n \t\t\t struct object_info *, unsigned flags);\n \n+/*\n+ * Open the loose object at path, check its hash, and return the contents,\n+ * type, and size. If the object is a blob, then \"contents\" may return NULL,\n+ * to allow streaming of large blobs.\n+ *\n+ * Returns 0 on success, negative on error (details may be written to stderr).\n+ */\n+int read_loose_object(const char *path,\n+\t\t const struct object_id *expected_oid,\n+\t\t enum object_type *type,\n+\t\t unsigned long *size,\n+\t\t void **contents,\n+\t\t unsigned int oi_flags);\n+\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434847","messageId":"patch-v6-21.22-974f650cddf-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 21/22] fsck: report invalid types recorded in objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:16Z","receivedAt":"2021-09-07T10:59:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Continue the work in the preceding commit and improve the error on:\n\n $ git hash-object --stdin -w -t garbage --literally (expected )\n error: : object corrupt or missing: \n [ other fsck output ]\n\nTo instead emit:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nThe complaint about a \"hash mismatch\" was simply an emergent property\nof how we'd fall though from read_loose_object() into fsck_loose()\nwhen we didn't get the data we expected. Now we'll correctly note that\nthe object type is invalid.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 22 ++++++++++++++++++----\n object-file.c | 13 +++++--------\n object-store.h | 4 ++--\n t/t1450-fsck.sh | 24 +++++++++++++++++++++---\n 4 files changed, 46 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 082dadd5629..07af0434db6 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,12 +600,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n-\n-\tif (read_loose_object(path, oid, &type, &size, &contents,\n-\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n-\t\terrors_found |= ERROR_OBJECT;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n+\tstruct object_info oi;\n+\tint found = 0;\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object corrupt or missing: %s\"),\n \t\t oid_to_hex(oid), path);\n+\t}\n+\tif (type < 0) {\n+\t\tfound |= ERROR_OBJECT;\n+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t oid_to_hex(oid), sb.buf, path);\n+\t}\n+\tif (found) {\n+\t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 0e6937fad73..f4850ba62b4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2560,9 +2560,8 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n {\n \tint ret = -1;\n@@ -2570,10 +2569,9 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tenum object_type *type = oi->typep;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2589,7 +2587,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n@@ -2611,8 +2609,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex dc638335e7d..f3045148b89 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -384,6 +384,7 @@ int oid_object_info_extended(struct repository *r,\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -391,9 +392,8 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n \t\t void **contents,\n+\t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n \n /*\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex d8303db9709..da2658155c7 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -66,6 +66,25 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\tgit init --bare hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n@@ -869,9 +888,8 @@ test_expect_success 'fsck error and recovery on invalid object type' '\n \tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally out 2>err &&\n \tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n-\ttest_line_count = 2 errors &&\n-\tgrep \"error: hash mismatch for\" err &&\n-\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n+\ttest_line_count = 1 errors &&\n+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n \tgrep \"dangling blob $empty_blob\" out\n '\n \n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"434848","messageId":"patch-v6-22.22-804673a17b0-20210907T104559Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v6 22/22] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T10:58:17Z","receivedAt":"2021-09-07T10:59:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 13 +++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n object-file.c | 21 ++++++++++++---------\n object-store.h | 4 +++-\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 10 files changed, 38 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 3c20f164f0f..48a3b6a7f8f 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 07af0434db6..158b9dac9b3 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -603,20 +603,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct strbuf sb = STRBUF_INIT;\n \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \tstruct object_info oi;\n+\tstruct object_id real_oid = *null_oid();\n \tint found = 0;\n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n \t\tfound |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n+\t\tif (!oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n \t}\n \tif (type < 0) {\n \t\tfound |= ERROR_OBJECT;\n \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t oid_to_hex(oid), sb.buf, path);\n+\t\t oid_to_hex(&real_oid), sb.buf, path);\n \t}\n \tif (found) {\n \t\terrors_found |= ERROR_OBJECT;\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 8336466865c..9f540e0236a 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1419,7 +1419,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/object-file.c b/object-file.c\nindex f4850ba62b4..07b3e4d9b4b 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1062,9 +1062,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1072,8 +1074,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1098,9 +1100,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2560,6 +2562,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags)\n@@ -2609,9 +2612,9 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n+\t\t\tif (oideq(real_oid, null_oid()))\n+\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex f3045148b89..3c4ada23f5d 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -392,6 +392,7 @@ int oid_object_info_extended(struct repository *r,\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi,\n \t\t unsigned int oi_flags);\n@@ -528,7 +529,8 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n int finalize_object_file(const char *tmpfile, const char *filename);\n int check_and_freshen_file(const char *fn, int freshen);\n \ndiff --git a/object.c b/object.c\nindex 4e85955a941..23a24e678a8 100644\n--- a/object.c\n+++ b/object.c\n@@ -279,7 +279,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -290,7 +290,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex c8e560d71ab..3f418e3a6af 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 43a9f4e7f0c..39fe11bc92c 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -490,7 +490,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex da2658155c7..7d0d57564b5 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -53,6 +53,7 @@ test_expect_success 'object with hash mismatch' '\n \t(\n \t\tcd hash-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -62,7 +63,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\ttest_i18ngrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -71,6 +72,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t(\n \t\tcd hash-type-mismatch &&\n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -80,8 +82,8 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.33.0.815.g21c7aaf6073\n\n"},{"id":"436148","messageId":"YUOds4kcHRgMk5nC@nand.local","threadId":"55404","inReplyTo":"patch-v6-01.22-ebe89f65354-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 01/22] fsck tests: refactor one test to use a sub-repo","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T19:40:35Z","receivedAt":"2021-09-16T19:40:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Refactor one of the fsck tests to use a throwaway repository. It's a\n> pervasive pattern in t1450-fsck.sh to spend a lot of effort on the\n> teardown of a tests so we're not leaving corrupt content for the next\n> test.\n\nOK. I seem to recall you advocating against this pattern elsewhere[1], but\nthis is a good example of why it can sometimes make writing tests much\neasier when not having to reason about what leaks out of running a test.\n\n[1]: https://lore.kernel.org/git/87zgsnj0q0.fsf@evledraar.gmail.com/,\nalthough after re-reading it it looks like you were more focused on the\nunnecessary \"rm -fr repo\" there and not the \"git init +\ntest_when_finished rm -fr\" pattern.\n\n> -test_expect_success 'object with bad sha1' '\n> -\tsha=$(echo blob | git hash-object -w --stdin) &&\n> -\told=$(test_oid_to_path \"$sha\") &&\n> -\tnew=$(dirname $old)/$(test_oid ff_2) &&\n> -\tsha=\"$(dirname $new)$(basename $new)\" &&\n> -\tmv .git/objects/$old .git/objects/$new &&\n> -\ttest_when_finished \"remove_object $sha\" &&\n> -\tgit update-index --add --cacheinfo 100644 $sha foo &&\n> -\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n> -\ttree=$(git write-tree) &&\n> -\ttest_when_finished \"remove_object $tree\" &&\n> -\tcmt=$(echo bogus | git commit-tree $tree) &&\n> -\ttest_when_finished \"remove_object $cmt\" &&\n> -\tgit update-ref refs/heads/bogus $cmt &&\n> -\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n> -\n> -\ttest_must_fail git fsck 2>out &&\n> -\ttest_i18ngrep \"$sha.*corrupt\" out\n> +test_expect_success 'object with hash mismatch' '\n> +\tgit init --bare hash-mismatch &&\n> +\t(\n> +\t\tcd hash-mismatch &&\n> +\t\toid=$(echo blob | git hash-object -w --stdin) &&\n> +\t\told=$(test_oid_to_path \"$oid\") &&\n> +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n> +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n> +\t\tmv objects/$old objects/$new &&\n> +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n> +\t\ttree=$(git write-tree) &&\n> +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n> +\t\tgit update-ref refs/heads/bogus $cmt &&\n> +\t\ttest_must_fail git fsck 2>out &&\n> +\t\ttest_i18ngrep \"$oid.*corrupt\" out\n> +\t)\n> '\n\nThis all looks fine to me. The translation is s/sha/oid and removing all\nof the now-unnecessary test_when_finished calls.\n\nBut the test_i18ngrep (which isn't new) could probably also stand to get\ncleaned up and converted to a normal grep.\n\nThanks,\nTaylor\n"},{"id":"436150","messageId":"YUOgKKRf9ZpMzNaS@nand.local","threadId":"55404","inReplyTo":"patch-v6-02.22-9072eef3be3-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 02/22] fsck tests: add test for fsck-ing an unknown type","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T19:51:04Z","receivedAt":"2021-09-16T19:51:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:57:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix a blindspot in the fsck tests by checking what we do when we\n> encounter an unknown \"garbage\" type produced with hash-object's\n> --literally option.\n>\n> This behavior needs to be improved, which'll be done in subsequent\n> patches, but for now let's test for the current behavior.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t1450-fsck.sh | 12 ++++++++++++\n> 1 file changed, 12 insertions(+)\n>\n> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n> index 7becab5ba1e..f10d6f7b7e8 100755\n> --- a/t/t1450-fsck.sh\n> +++ b/t/t1450-fsck.sh\n> @@ -863,4 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n> \ttest_i18ngrep \"bad index file\" errors\n> '\n>\n> +test_expect_success 'fsck hard errors on an invalid object type' '\n> +\tgit init --bare garbage-type &&\n\nI wondered whether it was really possible to not cover this, since I\nfigured such a test may have just been hiding elsewhere. But we really\ndo seem to be lacking coverage. So, adding this test is good.\n\n> +\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob +\tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally Test for what happens when the -t and -s flags are asked to operate on\n> a missing object, this extends tests added in 3e370f9faf0 (t1006: add\n> tests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n> -s flags are the only ones that can be combined with\n> --allow-unknown-type, so let's test with and without that flag.\n\nI'm a little skeptical to have tests for all four pairs of `-t` or `-s`\nand \"with `--allow-unknown-type` and without `--allow-unknown-type`\".\n\nTesting both the presence and absence of `--allow-unknown-type` seems\nuseful to me, but I'm not sure what testing `-t` and `-s` separately\nbuys us.\n\n(If you really feel the need test both, I'd encourage looping like:\n\n for arg in -t -s\n do\n test_must_fail git cat-file $arg $missing_oid >out 2>err &&\n test_must_be_empty out &&\n test_cmp expect.err err &&\n\n test_must_fail git cat-file $arg --allow-unknown-type $missing_oid >out 2>err &&\n test_must_be_empty out &&\n test_cmp expect.err err\n done &&\n\nbut I would be equally or perhaps even happier to just have one of the\ntwo tests).\n\nThanks,\nTaylor\n"},{"id":"436158","messageId":"YUOipZRE8tu+n8H6@nand.local","threadId":"55404","inReplyTo":"YUOhqnj7vwr00Qap@nand.local","subject":"Re: [PATCH v6 03/22] cat-file tests: test for missing object with -t and -s","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T20:01:41Z","receivedAt":"2021-09-16T20:01:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 16, 2021 at 03:57:30PM -0400, Taylor Blau wrote:\n> On Tue, Sep 07, 2021 at 12:57:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > Test for what happens when the -t and -s flags are asked to operate on\n> > a missing object, this extends tests added in 3e370f9faf0 (t1006: add\n> > tests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n> > -s flags are the only ones that can be combined with\n> > --allow-unknown-type, so let's test with and without that flag.\n>\n> I'm a little skeptical to have tests for all four pairs of `-t` or `-s`\n> and \"with `--allow-unknown-type` and without `--allow-unknown-type`\".\n\nAh. Reading the next patch makes me feel even more certain of this\nadvice. Consider squashing this and the next patch with my suggestion\nto use a loop below?\n\nThanks,\nTaylor\n"},{"id":"436160","messageId":"YUOryeVj1WNk1Sqd@nand.local","threadId":"55404","inReplyTo":"patch-v6-05.22-82db40ebf8a-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 05/22] rev-list tests: test for behavior with invalid object types","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T20:40:41Z","receivedAt":"2021-09-16T20:40:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:00PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\n> added in 16950f8384a (rev-list: add --disk-usage option for\n> calculating disk usage, 2021-02-09) to test for what happens when it's\n> asked to calculate the disk usage of invalid object types.\n\nI'm not sure that I agree this is a blindspot, or at least one worth\ntesting. Is the goal to add tests to every Git command that might have\nto do something with a corrupt object and make sure that it is handled\ncorrectly?\n\nI'm not sure that doing so would be useful, or at the very least that it\nwould be worth the effort. That's not to say I'm not interested in\nhaving tests fail when we don't handle corrupt objects correctly, but\nmore to say that I think there are so many parts of Git that might touch\na corrupt object that trying to test all of them seems like a losing\nbattle.\n\nAssuming that this is a useful direction, though...\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t6115-rev-list-du.sh | 11 +++++++++++\n> 1 file changed, 11 insertions(+)\n>\n> diff --git a/t/t6115-rev-list-du.sh b/t/t6115-rev-list-du.sh\n> index b4aef32b713..edb2ed55846 100755\n> --- a/t/t6115-rev-list-du.sh\n> +++ b/t/t6115-rev-list-du.sh\n> @@ -48,4 +48,15 @@ check_du HEAD\n> check_du --objects HEAD\n> check_du --objects HEAD^..HEAD\n>\n> +test_expect_success 'setup garbage repository' '\n> +\tgit clone --bare . garbage.git &&\n\nSince this is cloned within the working directory, should we bother to\nclean this up to avoid munging with future tests?\n\n> +\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally +\tgit -C garbage.git rev-list --objects --all --disk-usage &&\n> +\n> +\t# Manually create a ref because \"update-ref\", \"tag\" etc. have\n> +\t# no corresponding --literally option.\n> +\techo $garbage_oid >garbage.git/refs/tags/garbage-tag &&\n> +\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n\nSee also my earlier comment about this being much more readable in a\nsub-shell.\n\nThanks,\nTaylor\n"},{"id":"436165","messageId":"YUO3OjbnpqwEWeDv@nand.local","threadId":"55404","inReplyTo":"patch-v6-08.22-38e4266772d-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 08/22] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T21:29:30Z","receivedAt":"2021-09-16T21:29:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:03PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> When the loose_object_info() function returns an error stop faking up\n> the \"oi->typep\" to OBJ_BAD. Let the return value of the function\n> itself suffice. This code cleanup simplifies subsequent changes.\n\nThe obvious danger (which you mention) is that somebody is relying on\nwhat typep points to, and is reading it even if we returned non-zero\nfrom whatever called this function.\n\nHopefully nobody is, but this change makes me a little uncomfortable\nnonetheless, since there are so many potential callers (even though this\nfunction has only one caller, it doesn't take long before the number of\nindirect callers explodes).\n\nSo it would be nice if we could do without it, but you claim that it\nsimplifies changes that happen later on. So let's continue to see if we\nreally do need it...\n\nThanks,\nTaylor\n"},{"id":"436167","messageId":"YUO4HTcONhu15CSB@nand.local","threadId":"55404","inReplyTo":"patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 09/22] cache.h: move object functions to object-store.h","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T21:33:17Z","receivedAt":"2021-09-16T21:33:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:04PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Move the declaration of some ancient object functions added in\n> e.g. c4483576b8d (Add \"unpack_sha1_header()\" helper function,\n> 2005-06-01) from cache.h to object-store.h. This continues work\n> started in cbd53a2193d (object-store: move object access functions to\n> object-store.h, 2018-05-15).\n\nThis builds with DEVELOPER=1, likely as a result of all of the includes\non object-store.h added in cbd53a2193d.\n\n> diff --git a/cache.h b/cache.h\n> index d23de693680..11a04a93436 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1313,16 +1313,6 @@ char *xdg_cache_home(const char *filename);\n>\n> int git_open_cloexec(const char *name, int flags);\n> #define git_open(name) git_open_cloexec(name, O_RDONLY)\n> -int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n> -int parse_loose_header(const char *hdr, unsigned long *sizep);\n> -\n> -int check_object_signature(struct repository *r, const struct object_id *oid,\n> -\t\t\t void *buf, unsigned long size, const char *type);\n> -\n> -int finalize_object_file(const char *tmpfile, const char *filename);\n> -\n> -/* Helper to check and \"touch\" a file */\n\nI'm fine to drop this comment, by the way, since it does not add any\nexplanation to what a function called check_and_freshen_file() might do\n;).\n\n> -int check_and_freshen_file(const char *fn, int freshen);\n\nEverything else looks fine, although it's unclear how this is related to\nthe rest of your series.\n\nThanks,\nTaylor\n"},{"id":"436168","messageId":"YUO5m9gukagBGiF3@nand.local","threadId":"55404","inReplyTo":"patch-v6-10.22-b15ad53414b-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 10/22] object-file.c: make parse_loose_header_extended() public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T21:39:39Z","receivedAt":"2021-09-16T21:39:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:05PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Make the parse_loose_header_extended() function public and remove the\n> parse_loose_header() wrapper. The only direct user of it outside of\n> object-file.c itself was in streaming.c, that caller can simply pass\n> the required \"struct object-info *\" instead.\n>\n> This change is being done in preparation for teaching\n> read_loose_object() to accept a flag to pass to\n> parse_loose_header(). It isn't strictly necessary for that change, we\n> could simply use parse_loose_header_extended() there, but will leave\n> the API in a better end state.\n\nAll seems reasonable. I agree that this is not a necessary step, but at\nleast the clean-up is self contained and an easy enough read.\n\nThe flag that read_loose_object() is going to start passing to\nparse_loose_header() is left a bit vague, but I'll continue reading to\nfigure out what it is.\n\nThanks,\nTaylor\n"},{"id":"436171","messageId":"YUO9lrYloKGXILSA@coredump.intra.peff.net","threadId":"55404","inReplyTo":"YUO3OjbnpqwEWeDv@nand.local","subject":"Re: [PATCH v6 08/22] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-16T21:56:38Z","receivedAt":"2021-09-16T21:56:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 16, 2021 at 05:29:30PM -0400, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 12:58:03PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > When the loose_object_info() function returns an error stop faking up\n> > the \"oi->typep\" to OBJ_BAD. Let the return value of the function\n> > itself suffice. This code cleanup simplifies subsequent changes.\n> \n> The obvious danger (which you mention) is that somebody is relying on\n> what typep points to, and is reading it even if we returned non-zero\n> from whatever called this function.\n> \n> Hopefully nobody is, but this change makes me a little uncomfortable\n> nonetheless, since there are so many potential callers (even though this\n> function has only one caller, it doesn't take long before the number of\n> indirect callers explodes).\n> \n> So it would be nice if we could do without it, but you claim that it\n> simplifies changes that happen later on. So let's continue to see if we\n> really do need it...\n\nI'm actually reasonable comfortable with this patch. If we return an\nerror from the *_object_info() functions, then I think all bets are off\non what is in the resulting object_info struct. E.g., we'd already leave\nsizep uninitialized in such a case.\n\nIt feels like oi->typep may be a little bit special because we conflate\n\"error\" and \"type\" in the return from oid_object_info(). But\noid_object_info_extended() does not do that, and the innards of\noid_object_info() do the right thing.\n\nOf course we _have_ been setting typep in this way for a while, so it's\nworth making sure nobody is depending on. Notably packed_object_info()\ndoes not behave in this way; if it hits an error, typep may be left\nunset. So any oid_object_info_extended() callers depending on this were\nalready potentially buggy. I'd be OK with a quick sweep of the hits of\n\"git grep typep\" here.\n\nI just did that, and all the sites look pretty reasonable (they call\noid_object_info_extended() and bail as soon as they see that it fails).\n\n-Peff\n"},{"id":"436173","messageId":"YUO97RoOzY639hYW@nand.local","threadId":"55404","inReplyTo":"patch-v6-13.22-90489d9e6ec-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 13/22] object-file.c: split up ternary in parse_loose_header()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-16T21:58:05Z","receivedAt":"2021-09-16T21:58:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:08PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> This minor formatting change serves to make a subsequent patch easier\n> to read.\n\nHmm. I'm not sure if I agree.\n\nAs far as I can tell from reading the subsequent patch, this is designed\nto make it easier to add a comment above \"return type\" that pertains\njust to the case when !*hdr.\n\nI think it would have been fine to go from the ternary to this style\nwith the comment in a single patch. But I also think it would have been\nfine to add the comment above the ternary and instead start it off by\nsaying \"when !*hdr, ...\".\n\nThanks,\nTaylor\n"},{"id":"436193","messageId":"87fsu4f729.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YUOhqnj7vwr00Qap@nand.local","subject":"Re: [PATCH v6 03/22] cat-file tests: test for missing object with -t and -s","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-16T22:52:11Z","receivedAt":"2021-09-16T23:13:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 16 2021, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 12:57:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Test for what happens when the -t and -s flags are asked to operate on\n>> a missing object, this extends tests added in 3e370f9faf0 (t1006: add\n>> tests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n>> -s flags are the only ones that can be combined with\n>> --allow-unknown-type, so let's test with and without that flag.\n>\n> I'm a little skeptical to have tests for all four pairs of `-t` or `-s`\n> and \"with `--allow-unknown-type` and without `--allow-unknown-type`\".\n>\n> Testing both the presence and absence of `--allow-unknown-type` seems\n> useful to me, but I'm not sure what testing `-t` and `-s` separately\n> buys us.\n>\n> (If you really feel the need test both, I'd encourage looping like:\n\nThanks, I'll try to simplify it.\n\n> for arg in -t -s\n> do\n> test_must_fail git cat-file $arg $missing_oid >out 2>err &&\n> test_must_be_empty out &&\n> test_cmp expect.err err &&\n>\n> test_must_fail git cat-file $arg --allow-unknown-type $missing_oid >out 2>err &&\n> test_must_be_empty out &&\n> test_cmp expect.err err\n> done &&\n>\n> but I would be equally or perhaps even happier to just have one of the\n> two tests).\n\nA loop like that can be further simplified as just (just inlining\narg=-s):\n\n\ttest_must_fail git cat-file -s $missing_oid >out 2>err &&\n\ttest_must_be_empty out &&\n\ttest_cmp expect.err err &&\n\n\ttest_must_fail git cat-file -s --allow-unknown-type $missing_oid >out 2>err &&\n\ttest_must_be_empty out &&\n\ttest_cmp expect.err err\n\n:)\n\nI.e. unless you end &&-chains in loops in the test framework with an ||\nreturn 1 you're only testing your last iteration. Aside from whatever\nI'm doing here I generally prefer to either just spell it out twice (if\nsmall enough), or:\n\n for arg in -t -s\n do\n test_expect_success '...' \"[... use $arg ...]\"\n done\n\nWhich both nicely get around the issue of that easy-to-make mistake.\n\nWe've got some in-tree tests that are broken this way, well, at least\n4cf67869b2a (list-objects.c: don't segfault for missing cmdline objects,\n2018-12-05). But I think I'll leave that for a #leftoverbits submission\ngiven my outstanding patch queue..., oh there's another one in\nt1010-mktree.sh ... :)\n"},{"id":"436194","messageId":"YUP+Ni1tFKolLZd4@nand.local","threadId":"55404","inReplyTo":"patch-v6-14.22-7c9819d37c5-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 14/22] object-file.c: stop dying in parse_loose_header()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T02:32:22Z","receivedAt":"2021-09-17T02:32:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:09PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> It thus makes sense to further libify the interface so that it stops\n> calling die() when it encounters OBJ_BAD, and instead rely on its\n> callers to check the populated \"oi->typep\".\n\nHmm. I thought we got rid of this behavior in a previous commit? Perhaps\nI'm thinking of something else, but I would certainly appreciate a\nclarification :).\n\n> @@ -1369,15 +1367,6 @@ int parse_loose_header(const char *hdr,\n> \ttype = type_from_string_gently(type_buf, type_len, 1);\n> \tif (oi->type_name)\n> \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n> -\t/*\n> -\t * Set type to 0 if its an unknown object and\n> -\t * we're obtaining the type using '--allow-unknown-type'\n> -\t * option.\n> -\t */\n> -\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n> -\t\ttype = 0;\n> -\telse if (type < 0)\n> -\t\tdie(_(\"invalid object type\"));\n\nGood, this part moved to loose_object_info() as you said it would.\n\n> @@ -1463,18 +1460,20 @@ static int loose_object_info(struct repository *r,\n> \t\tstatus = error(_(\"unable to unpack %s header\"),\n> \t\t\t oid_to_hex(oid));\n> \t}\n> -\n> -\tif (status < 0) {\n> -\t\t/* Do nothing */\n> -\t} else if (hdrbuf.len) {\n> -\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n> -\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n> -\t\t\t\t oid_to_hex(oid));\n> -\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n> -\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n> +\tif (!status) {\n> +\t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n> +\t\t\t/*\n> +\t\t\t * oi->{sizep,typep} are meaningless unless\n> +\t\t\t * parse_loose_header() returns >= 0.\n> +\t\t\t */\n\nThis double negative is a little confusing. Clearer to say:\n\"oi->{size,type}p is meaningless if parse_loose_header() returns < 0\"?\n\nBut I was also a little confused to see that the expression we are\nchecking here is just that parse_loose_header() returned zero. What\nabout other positive values?\n\nI think we should either update the comment to say \"unless it returns\nzero\" or the conditional expression to check for >= 0.\n\n> diff --git a/object-store.h b/object-store.h\n> index 4064710ae29..584bf5556af 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -500,8 +500,17 @@ int for_each_packed_object(each_packed_object_fn, void *,\n> int unpack_loose_header(git_zstream *stream, unsigned char *map,\n> \t\t\tunsigned long mapsize, void *buffer,\n> \t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n> -int parse_loose_header(const char *hdr, struct object_info *oi,\n> -\t\t unsigned int flags);\n> +\n> +/**\n> + * parse_loose_header() parses the starting \" \\0\" of an\n> + * object. If it doesn't follow that format -1 is returned. To check\n> + * the validity of the populate the \"typep\" in the \"struct\n> + * object_info\". It will be OBJ_BAD if the object type is unknown. The\n> + * parsed can be retrieved via \"oi->sizep\", and from there\n> + * passed to unpack_loose_rest().\n> + */\n> +int parse_loose_header(const char *hdr, struct object_info *oi);\n\nOK, I guess this must be what I was confused about earlier (that I\nthought we didn't support reading typep if returning OBJ_BAD). But it\nseems odd to me that we would get rid of it elsewhere, yet continue\nusing this pattern here.\n\nOr am I mistaken that the two are different?\n\nThanks,\nTaylor\n"},{"id":"436195","messageId":"YUP+5W9rAWwNQ23t@nand.local","threadId":"55404","inReplyTo":"patch-v6-15.22-3fb660ff944-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 15/22] object-file.c: guard against future bugs in loose_object_info()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T02:35:17Z","receivedAt":"2021-09-17T02:37:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:10PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> An earlier version of the preceding commit had a subtle bug where our\n> \"type_scratch\" (later assigned to \"oi->typep\") would be uninitialized\n> and used in the \"!allow_unknown\" case, at which point it would contain\n> a nonsensical value if we'd failed to call parse_loose_header().\n>\n> The preceding commit introduced \"parsed_header\" variable to check for\n> this case, but I think we can do better, let's carry a \"oi_header\"\n> variable initially set to NULL, and only set it to \"oi\" once we're\n> past parse_loose_header().\n\nEverything in this patch seems OK to me.\n\nFor what it's worth, I think that this could likely have been folded\ninto the previous commit. I was just a little surprised to see\nparsed_header go away after I had just a minute or two again spent time\nthinking about what it was for.\n\nThanks,\nTaylor\n"},{"id":"436196","messageId":"YUQBPXPdCxumPWhL@nand.local","threadId":"55404","inReplyTo":"patch-v6-18.22-1b7173a5b5b-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 18/22] object-file.c: use \"enum\" return type for unpack_loose_header()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T02:45:17Z","receivedAt":"2021-09-17T02:45:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:13PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> In the preceding commits we changed and documented\n> unpack_loose_header() from return any negative value or zero, to only\n> -2, -1 or 0. Let's instead add an \"enum unpack_loose_header_result\"\n> type and use it, and have the compiler assert that we're exhaustively\n> covering all return values. This gets rid of the need for having a\n> \"default\" BUG() case in loose_object_info().\n>\n> I'm on the fence about whether this is more readable or worth it, but\n> since it was suggested in [1] to do this let's go for it.\n\n:-). The first hunk is quite a long line, but I think that only suggests\nthe enum has a long name. I also can't think of anything shorter, so I\nthink what you have is just fine.\n\nI do think that this is an improvement in readability, and for what it's\nworth I am a fan of the previous two changes as well.\n\nAs a workflow comment, I would have perhaps done these conversions a\nlittle earlier, maybe in these steps:\n\n - First a patch to introduce unpack_loose_header_result with just OK\n and BAD, and then converted all callers that return negative numbers\n to return BAD (and all others to return OK).\n\n - Then a second patch to convert some of the BAD returns into\n BAD_TOO_LONG.\n\nThat gets things done in two patches, instead of three, at the cost of a\nslightly more complicated first patch. But I think you also get some\nmore insight into why we're making the change in the first place instead\nof having to read through a couple of commits to get there.\n\nIn any case, what you have is certainly fine, and I don't think that one\napproach is any better or worse than the other. Just mentioning it in\ncase it's something may try in the future.\n\nThis patch looks good.\n\nThanks,\nTaylor\n"},{"id":"436197","messageId":"YUQNcPAs35WhJHKo@nand.local","threadId":"55404","inReplyTo":"patch-v6-19.22-ad1614dbb8d-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 19/22] fsck: don't hard die on invalid object types","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T03:37:20Z","receivedAt":"2021-09-17T03:37:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:14PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Change the error fsck emits on invalid object types, such as:\n>\n> $ git hash-object --stdin -w -t garbage --literally \n>\n> >From the very ungraceful error of:\n>\n> $ git fsck\n> fatal: invalid object type\n> $\n>\n> To:\n>\n> $ git fsck\n> error: hash mismatch for (expected )\n> error: : object corrupt or missing: \n> [ the rest of the fsck output here, i.e. it didn't hard die ]\n\nGreat. I don't love the second error (since it doesn't really give the\nuser any new information when read after the first) but that's fsck's\nfault, and not your patch's.\n\n> To do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\n> flag from read_loose_object() through to parse_loose_header(). Since\n> the read_loose_object() function is only used in builtin/fsck.c we can\n> simply change it. See f6371f92104 (sha1_file: add read_loose_object()\n> function, 2017-01-13) for the introduction of read_loose_object().\n>\n> Why are we complaining about a \"hash mismatch\" for an object of a type\n> we don't know about? We shouldn't. This is the bare minimal change\n> needed to not make fsck hard die on a repository that's been corrupted\n> in this manner. In subsequent commits we'll teach fsck to recognize\n> this particular type of corruption and emit a better error message.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/fsck.c | 3 ++-\n> object-file.c | 11 ++++++++---\n> object-store.h | 3 ++-\n> t/t1450-fsck.sh | 14 +++++++-------\n> 4 files changed, 19 insertions(+), 12 deletions(-)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index b42b6fe21f7..082dadd5629 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -601,7 +601,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \tvoid *contents;\n> \tint eaten;\n>\n> -\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n> +\tif (read_loose_object(path, oid, &type, &size, &contents,\n> +\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n> \t\terrors_found |= ERROR_OBJECT;\n> \t\terror(_(\"%s: object corrupt or missing: %s\"),\n> \t\t oid_to_hex(oid), path);\n> diff --git a/object-file.c b/object-file.c\n> index 9484c7ce2be..0e6937fad73 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2562,7 +2562,8 @@ int read_loose_object(const char *path,\n> \t\t const struct object_id *expected_oid,\n> \t\t enum object_type *type,\n> \t\t unsigned long *size,\n> -\t\t void **contents)\n> +\t\t void **contents,\n> +\t\t unsigned int oi_flags)\n> {\n> \tint ret = -1;\n> \tvoid *map = NULL;\n> @@ -2570,6 +2571,7 @@ int read_loose_object(const char *path,\n> \tgit_zstream stream;\n> \tchar hdr[MAX_HEADER_LEN];\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> +\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n> \toi.typep = type;\n> \toi.sizep = size;\n>\n> @@ -2592,8 +2594,11 @@ int read_loose_object(const char *path,\n> \t\tgit_inflate_end(&stream);\n> \t\tgoto out;\n> \t}\n> -\tif (*type < 0)\n> -\t\tdie(_(\"invalid object type\"));\n> +\tif (!allow_unknown && *type < 0) {\n> +\t\terror(_(\"header for %s declares an unknown type\"), path);\n> +\t\tgit_inflate_end(&stream);\n> +\t\tgoto out;\n> +\t}\n\nHmm. I'm not sure that I new test for this error (which may be\nuninteresting, in which case it is fine to skip).\n>\n> -test_expect_success 'fsck hard errors on an invalid object type' '\n> +test_expect_success 'fsck error and recovery on invalid object type' '\n> \tgit init --bare garbage-type &&\n> \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob \tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally -\tcat >err.expect <<-\\EOF &&\n> -\tfatal: invalid object type\n> -\tEOF\n> -\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n> -\ttest_cmp err.expect err.actual &&\n> -\ttest_must_be_empty out.actual\n> +\ttest_must_fail git -C garbage-type fsck >out 2>err &&\n> +\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n> +\ttest_line_count = 2 errors &&\n> +\tgrep \"error: hash mismatch for\" err &&\n> +\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n> +\tgrep \"dangling blob $empty_blob\" out\n> '\n\nGreat.\n\nThanks,\nTaylor\n"},{"id":"436200","messageId":"YUQSNgC8yCw6ICY9@nand.local","threadId":"55404","inReplyTo":"patch-v6-21.22-974f650cddf-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 21/22] fsck: report invalid types recorded in objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T03:57:42Z","receivedAt":"2021-09-17T03:57:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Continue the work in the preceding commit and improve the error on:\n>\n> $ git hash-object --stdin -w -t garbage --literally $ git fsck\n> error: hash mismatch for (expected )\n> error: : object corrupt or missing: \n> [ other fsck output ]\n>\n> To instead emit:\n>\n> $ git fsck\n> error: : object is of unknown type 'garbage': \n> [ other fsck output ]\n>\n> The complaint about a \"hash mismatch\" was simply an emergent property\n> of how we'd fall though from read_loose_object() into fsck_loose()\n> when we didn't get the data we expected. Now we'll correctly note that\n> the object type is invalid.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/fsck.c | 22 ++++++++++++++++++----\n> object-file.c | 13 +++++--------\n> object-store.h | 4 ++--\n> t/t1450-fsck.sh | 24 +++++++++++++++++++++---\n> 4 files changed, 46 insertions(+), 17 deletions(-)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 082dadd5629..07af0434db6 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -600,12 +600,26 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \tunsigned long size;\n> \tvoid *contents;\n> \tint eaten;\n> -\n> -\tif (read_loose_object(path, oid, &type, &size, &contents,\n> -\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n> -\t\terrors_found |= ERROR_OBJECT;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n> +\tstruct object_info oi;\n> +\tint found = 0;\n> +\toi.type_name = &sb;\n> +\toi.sizep = &size;\n> +\toi.typep = &type;\n> +\n> +\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n\nOK, now we pass a struct object_info instead of pointers to type and\nsize separately. Makes sense.\n\n> +\t\tfound |= ERROR_OBJECT;\n\nAnd found tracks the error we found when trying to read this loose\nobject, if any. Having a separate variable makes sense, since we only\nwant to avoid calling fsck_obj() if we found any errors for this object\nwhile trying to call read_loose_object().\n\n> \t\terror(_(\"%s: object corrupt or missing: %s\"),\n> \t\t oid_to_hex(oid), path);\n> +\t}\n> +\tif (type < 0) {\n> +\t\tfound |= ERROR_OBJECT;\n> +\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n> +\t\t oid_to_hex(oid), sb.buf, path);\n> +\t}\n> +\tif (found) {\n> +\t\terrors_found |= ERROR_OBJECT;\n\nPerhaps errors_found |= found ?\n\n> \t\treturn 0; /* keep checking other objects */\n> \t}\n>\n> diff --git a/object-file.c b/object-file.c\n> index 0e6937fad73..f4850ba62b4 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2560,9 +2560,8 @@ static int check_stream_oid(git_zstream *stream,\n>\n> int read_loose_object(const char *path,\n> \t\t const struct object_id *expected_oid,\n> -\t\t enum object_type *type,\n> -\t\t unsigned long *size,\n> \t\t void **contents,\n> +\t\t struct object_info *oi,\n> \t\t unsigned int oi_flags)\n\nAll of the changes in this function make perfect sense, except...\n> {\n> \tint ret = -1;\n> @@ -2570,10 +2569,9 @@ int read_loose_object(const char *path,\n> \tunsigned long mapsize;\n> \tgit_zstream stream;\n> \tchar hdr[MAX_HEADER_LEN];\n> -\tstruct object_info oi = OBJECT_INFO_INIT;\n> \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n> -\toi.typep = type;\n> -\toi.sizep = size;\n> +\tenum object_type *type = oi->typep;\n> +\tunsigned long *size = oi->sizep;\n\n...I see that size is used in check_object_signature(), but I don't see\nany uses for type. Am I missing it?\n\nThe tests look good to me.\n\nThanks,\nTaylor\n"},{"id":"436201","messageId":"YUQUKvM3yjx9yDQI@nand.local","threadId":"55404","inReplyTo":"patch-v6-22.22-804673a17b0-20210907T104559Z-avarab@gmail.com","subject":"Re: [PATCH v6 22/22] fsck: report invalid object type-path combinations","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T04:06:02Z","receivedAt":"2021-09-17T04:08:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:58:17PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Improve the error that's emitted in cases where we find a loose object\n> we parse, but which isn't at the location we expect it to be.\n>\n> Before this change we'd prefix the error with a not-a-OID derived from\n> the path at which the object was found, due to an emergent behavior in\n> how we'd end up with an \"OID\" in these codepaths.\n>\n> Now we'll instead say what object we hashed, and what path it was\n> found at. Before this patch series e.g.:\n>\n> $ git hash-object --stdin -w -t blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n> $ mv objects/e6/ objects/e7\n>\n> Would emit (\"[...]\" used to abbreviate the OIDs):\n>\n> git fsck\n> error: hash mismatch for ./objects/e7/9d[...] (expected e79d[...])\n> error: e79d[...]: object corrupt or missing: ./objects/e7/9d[...]\n>\n> Now we'll instead emit:\n>\n> error: e69d[...]: hash-path mismatch, found at: ./objects/e7/9d[...]\n\nLovely!\n\n> @@ -603,20 +603,25 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n> \tstruct object_info oi;\n> +\tstruct object_id real_oid = *null_oid();\n> \tint found = 0;\n> \toi.type_name = &sb;\n> \toi.sizep = &size;\n> \toi.typep = &type;\n>\n> -\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n> +\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n> \t\tfound |= ERROR_OBJECT;\n> -\t\terror(_(\"%s: object corrupt or missing: %s\"),\n> -\t\t oid_to_hex(oid), path);\n> +\t\tif (!oideq(&real_oid, oid))\n> +\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n> +\t\t\t oid_to_hex(&real_oid), path);\n> +\t\telse\n> +\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n> +\t\t\t oid_to_hex(oid), path);\n\nNice; this is the important part that this patch is changing, and the\nlogic is very nice. Before it read \"anytime read_loose_object fails,\nit's an error\" to \"it's still an error, but we can handle the case where\nthe real OID and the one we expected were different separately from\ngeneric corruption\".\n\n> \t}\n> \tif (type < 0) {\n> \t\tfound |= ERROR_OBJECT;\n> \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n> -\t\t oid_to_hex(oid), sb.buf, path);\n> +\t\t oid_to_hex(&real_oid), sb.buf, path);\n\nCould go either way on this hunk, but I think that I err slightly on\nyour side now that we have access to the \"real_oid\".\n\nThe rest of the code and test changes in this patch look good to me.\n\nThanks,\nTaylor\n"},{"id":"436202","messageId":"YUQUpCeFmQ1eSn7R@nand.local","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"Re: [PATCH v6 00/22] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-17T04:08:04Z","receivedAt":"2021-09-17T04:08:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 12:57:55PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> This improves fsck error reporting, see the examples in the commit\n> messages of 19/22, 21/22 and 22/22. To get there I've lib-ified more\n> thigs in object-file.c and the general object APIs, i.e. now we'll\n> return error codes instead of calling die() in these cases.\n>\n> This series has been in \"needs review\" state for a while. This re-roll\n> is mainly to bump it for the list's attention, but while I was at it I\n> addressed point from Jonathan Tan raised in a previous round: use an\n> enum instead of int for the unpack_loose_header() return value.\n\nI took a thorough look through this series, and left a handful of minor\ncomments. I didn't spot any glaring issues, and think that this series\nis in pretty good shape.\n\nI do admit there were quite a large number of patches to get to the\ncouple of changes at the end. I left some thoughts throughout for places\nthat I would have combined things / presented them in a different order\nor similar.\n\nI don't think you should spend much time changing the structure now that\nit's been looked at with close eyes, but just some idle thoughts for\nother large series you might send in the future.\n\nThanks,\nTaylor\n"},{"id":"436219","messageId":"87tuijeels.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YUOds4kcHRgMk5nC@nand.local","subject":"Re: [PATCH v6 01/22] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-17T09:27:19Z","receivedAt":"2021-09-17T09:28:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 16 2021, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 12:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Refactor one of the fsck tests to use a throwaway repository. It's a\n>> pervasive pattern in t1450-fsck.sh to spend a lot of effort on the\n>> teardown of a tests so we're not leaving corrupt content for the next\n>> test.\n>\n> OK. I seem to recall you advocating against this pattern elsewhere[1], but\n> this is a good example of why it can sometimes make writing tests much\n> easier when not having to reason about what leaks out of running a test.\n>\n> [1]: https://lore.kernel.org/git/87zgsnj0q0.fsf@evledraar.gmail.com/,\n> although after re-reading it it looks like you were more focused on the\n> unnecessary \"rm -fr repo\" there and not the \"git init +\n> test_when_finished rm -fr\" pattern.\n\nI was referring to a different pattern there, replied in some detail at\nhttps://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n\n>> -test_expect_success 'object with bad sha1' '\n>> -\tsha=$(echo blob | git hash-object -w --stdin) &&\n>> -\told=$(test_oid_to_path \"$sha\") &&\n>> -\tnew=$(dirname $old)/$(test_oid ff_2) &&\n>> -\tsha=\"$(dirname $new)$(basename $new)\" &&\n>> -\tmv .git/objects/$old .git/objects/$new &&\n>> -\ttest_when_finished \"remove_object $sha\" &&\n>> -\tgit update-index --add --cacheinfo 100644 $sha foo &&\n>> -\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n>> -\ttree=$(git write-tree) &&\n>> -\ttest_when_finished \"remove_object $tree\" &&\n>> -\tcmt=$(echo bogus | git commit-tree $tree) &&\n>> -\ttest_when_finished \"remove_object $cmt\" &&\n>> -\tgit update-ref refs/heads/bogus $cmt &&\n>> -\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n>> -\n>> -\ttest_must_fail git fsck 2>out &&\n>> -\ttest_i18ngrep \"$sha.*corrupt\" out\n>> +test_expect_success 'object with hash mismatch' '\n>> +\tgit init --bare hash-mismatch &&\n>> +\t(\n>> +\t\tcd hash-mismatch &&\n>> +\t\toid=$(echo blob | git hash-object -w --stdin) &&\n>> +\t\told=$(test_oid_to_path \"$oid\") &&\n>> +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n>> +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n>> +\t\tmv objects/$old objects/$new &&\n>> +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n>> +\t\ttree=$(git write-tree) &&\n>> +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n>> +\t\tgit update-ref refs/heads/bogus $cmt &&\n>> +\t\ttest_must_fail git fsck 2>out &&\n>> +\t\ttest_i18ngrep \"$oid.*corrupt\" out\n>> +\t)\n>> '\n>\n> This all looks fine to me. The translation is s/sha/oid and removing all\n> of the now-unnecessary test_when_finished calls.\n>\n> But the test_i18ngrep (which isn't new) could probably also stand to get\n> cleaned up and converted to a normal grep.\n\nThanks, I missed that one!\n"},{"id":"436221","messageId":"87pmt7edx4.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YUOgKKRf9ZpMzNaS@nand.local","subject":"Re: [PATCH v6 02/22] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-17T09:39:02Z","receivedAt":"2021-09-17T09:42:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 16 2021, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 12:57:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Fix a blindspot in the fsck tests by checking what we do when we\n>> encounter an unknown \"garbage\" type produced with hash-object's\n>> --literally option.\n>>\n>> This behavior needs to be improved, which'll be done in subsequent\n>> patches, but for now let's test for the current behavior.\n>>\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> t/t1450-fsck.sh | 12 ++++++++++++\n>> 1 file changed, 12 insertions(+)\n>>\n>> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n>> index 7becab5ba1e..f10d6f7b7e8 100755\n>> --- a/t/t1450-fsck.sh\n>> +++ b/t/t1450-fsck.sh\n>> @@ -863,4 +863,16 @@ test_expect_success 'detect corrupt index file in fsck' '\n>> \ttest_i18ngrep \"bad index file\" errors\n>> '\n>>\n>> +test_expect_success 'fsck hard errors on an invalid object type' '\n>> +\tgit init --bare garbage-type &&\n>\n> I wondered whether it was really possible to not cover this, since I\n> figured such a test may have just been hiding elsewhere. But we really\n> do seem to be lacking coverage. So, adding this test is good.\n>\n>> +\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob > +\tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally \n> I'm nitpicking, but I find the -C garbage-type pattern less than ideal\n> for two reasons:\n>\n> - It makes every line longer (since \"-C garbage type\" is wider than an\n> 8-wide tab, even indenting this in a subshell would take up fewer\n> characters visually)\n>\n> - It pollutes the current directory with things like \"err.expect\" and\n> \"err.actual\" that have nothing to do with the current directory (and\n> much more to do with the garbage-type repository within it).\n>\n> So I don't care, really, but it may be better to just put all of this in\n> a subshell.\n\nYes, it does look much nicer like that. Thanks!\n\nSome aspects of style I use I have some informed/strong opinion about,\nlike the teardown/setup pattern noted in [1], but for some other stuff\nlike this ... I think I was just following the pattern of some recent\ntest I'd read or something.\n\nWell, one advantage of using \"git -C\" is that if it fails you can cd to\nthe trash directory and run the command you saw fail as-is without\ncd-ing further, and in that case the \"polluting\" is a feature, you can\ncat the top-level expect/actual consistently.\n\nBut I think on balance having the test itself be easier to read is more\nimportant, so I'm going with the subshell.\n\n1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n"},{"id":"436225","messageId":"87lf3ve789.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"YUOryeVj1WNk1Sqd@nand.local","subject":"Re: [PATCH v6 05/22] rev-list tests: test for behavior with invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-17T11:59:10Z","receivedAt":"2021-09-17T12:07:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 16 2021, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 12:58:00PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\n>> added in 16950f8384a (rev-list: add --disk-usage option for\n>> calculating disk usage, 2021-02-09) to test for what happens when it's\n>> asked to calculate the disk usage of invalid object types.\n>\n> I'm not sure that I agree this is a blindspot, or at least one worth\n> testing. Is the goal to add tests to every Git command that might have\n> to do something with a corrupt object and make sure that it is handled\n> correctly?\n>\n> I'm not sure that doing so would be useful, or at the very least that\n> it would be worth the effort. [...] I think there are so many parts of\n> Git that might touch a corrupt object that trying to test all of them\n> seems like a losing battle.\n\nI'll drop it since it doesn't have anything directly to do with this\nseries. This slipped in from the work I meant to follow-up after this\nwith.\n\nThis isn't just any random command that might come across an invalid\nobject though, it's specifically reporting object sizes. Once we change\nthat to not die we'll we'll want to see how invalid objects are handled\nby it. Will the disk size be reported as -1? 0? ~0?\n\n> [...]\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> t/t6115-rev-list-du.sh | 11 +++++++++++\n>> 1 file changed, 11 insertions(+)\n>>\n>> diff --git a/t/t6115-rev-list-du.sh b/t/t6115-rev-list-du.sh\n>> index b4aef32b713..edb2ed55846 100755\n>> --- a/t/t6115-rev-list-du.sh\n>> +++ b/t/t6115-rev-list-du.sh\n>> @@ -48,4 +48,15 @@ check_du HEAD\n>> check_du --objects HEAD\n>> check_du --objects HEAD^..HEAD\n>>\n>> +test_expect_success 'setup garbage repository' '\n>> +\tgit clone --bare . garbage.git &&\n>\n> Since this is cloned within the working directory, should we bother to\n> clean this up to avoid munging with future tests?\n\nIn general (and I had some other replies with this) I think no, if a an\nindividual test is picking a unique name for its data it doesn't need to\nbother with test_when_finished, it can just leave the cleanup to the\neventual trash directory cleanup.\n\n>> +\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally > +\tgit -C garbage.git rev-list --objects --all --disk-usage &&\n>> +\n>> +\t# Manually create a ref because \"update-ref\", \"tag\" etc. have\n>> +\t# no corresponding --literally option.\n>> +\techo $garbage_oid >garbage.git/refs/tags/garbage-tag &&\n>> +\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n>\n> See also my earlier comment about this being much more readable in a\n> sub-shell.\n\n*nod*\n"},{"id":"436532","messageId":"patch-v7-01.17-752cef556c2-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 01/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:05Z","receivedAt":"2021-09-21T02:58:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..969bfbbdd8f 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -865,4 +865,21 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\tgit init --bare garbage-type &&\n+\t(\n+\t\tcd garbage-type &&\n+\n+\t\tempty=$(git hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck >out 2>err &&\n+\t\ttest_cmp err.expect err &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436533","messageId":"patch-v7-02.17-612003bdd2c-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 02/17] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:06Z","receivedAt":"2021-09-21T02:58:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe can instead use the pattern of creating a named sub-repository,\nthen we don't have to worry about cleaning up after ourselves, nobody\nwill care what state the broken \"hash-mismatch\" repository is after\nthis test runs.\n\nSee [1] for related discussion on various \"modern\" test patterns that\ncan be used to avoid verbosity and increase reliability.\n\n1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 35 ++++++++++++++++++-----------------\n 1 file changed, 18 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 969bfbbdd8f..f8edd15abf8 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,25 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n+test_expect_success 'object with hash mismatch' '\n+\tgit init --bare hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n \n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436534","messageId":"patch-v7-03.17-1e40a4235e9-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 03/17] fsck tests: test current hash/type mismatch behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:07Z","receivedAt":"2021-09-21T02:58:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If fsck we move an object around between .git/objects/?? directories\nto simulate a hash mismatch \"git fsck\" will currently hard die() in\nobject-file.c. This behavior will be fixed in subsequent commits, but\nlet's test for it as-is for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f8edd15abf8..175ed304637 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -69,6 +69,30 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\tgit init --bare hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436536","messageId":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com","subject":"[PATCH v7 00/17] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:04Z","receivedAt":"2021-09-21T02:58:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting, see the examples in the commit\nmessages of 16/17 and 17/17. To get there I've lib-ified more things\nin object-file.c and the general object APIs, i.e. now we'll return\nerror codes instead of calling die() in these cases.\n\nv6 of this got a very detailed review from Taylor Blau (thanks a\nlot!), for the v6 see:\nhttps://lore.kernel.org/git/cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com/\n\nThis should address all of the things brought up, and more. After\nleaving this series for a while I came up with ways to simplify it\neven more, so now it's 17 instead of 22 patches!\n\nSo things like:\n\n * The move of functions from cache.h to object-store.h is gone, that\n still makes sense to do, but can be left for later.\n\n * A large part of the mid-series is squashed together and\n re-arranged, e.g. I moved the migration of unpack_loose_header() to\n to an enum earlier, which simplified later steps. The 15/17 rewrite\n of much of parse_loose_header() is now much simpler.\n\n * I attempted to address the comments about the tests with some\n for-loops and boilerplate-generated testing, maybe some of it's a\n bit too ugly, but it's both less copy/pasting now, and more cases\n (e.g. the \"cat-file -p\" case) are tested.\n\n * We now test for what happens/error reporting when we append garbage\n to a loose object.\n\n * Many more small changes / improvements / simplifications, see the\n range-diff below, but given its size perhaps a re-read is easier...\n\nÆvar Arnfjörð Bjarmason (17):\n fsck tests: add test for fsck-ing an unknown type\n fsck tests: refactor one test to use a sub-repo\n fsck tests: test current hash/type mismatch behavior\n fsck tests: test for garbage appended to a loose object\n cat-file tests: move bogus_* variable declarations earlier\n cat-file tests: test for missing/bogus object with -t, -s and -p\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n object-file.c: don't set \"typep\" when returning non-zero\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: make parse_loose_header_extended() public\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: use \"enum\" return type for unpack_loose_header()\n object-file.c: return ULHR_TOO_LONG on \"header too long\"\n object-file.c: stop dying in parse_loose_header()\n fsck: don't hard die on invalid object types\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 +++++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 45 ++++++++-\n object-file.c | 176 +++++++++++++++------------------\n object-store.h | 7 +-\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 27 +++--\n t/oid-info/oid | 2 +\n t/t1006-cat-file.sh | 223 +++++++++++++++++++++++++++++++++++++++---\n t/t1450-fsck.sh | 99 +++++++++++++++----\n 13 files changed, 463 insertions(+), 158 deletions(-)\n\nRange-diff against v6:\n 2: 9072eef3be3 ! 1: 752cef556c2 fsck tests: add test for fsck-ing an unknown type\n @@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n \n +test_expect_success 'fsck hard errors on an invalid object type' '\n +\tgit init --bare garbage-type &&\n -+\tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n -+\tfatal: invalid object type\n -+\tEOF\n -+\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n -+\ttest_cmp err.expect err.actual &&\n -+\ttest_must_be_empty out.actual\n ++\t(\n ++\t\tcd garbage-type &&\n ++\n ++\t\tempty=$(git hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n ++\t\tfatal: invalid object type\n ++\t\tEOF\n ++\t\ttest_must_fail git fsck >out 2>err &&\n ++\t\ttest_cmp err.expect err &&\n ++\t\ttest_must_be_empty out\n ++\t)\n +'\n +\n test_done\n 1: ebe89f65354 ! 2: 612003bdd2c fsck tests: refactor one test to use a sub-repo\n @@ Commit message\n teardown of a tests so we're not leaving corrupt content for the next\n test.\n \n - We should instead simply use something like this test_create_repo\n - pattern. It's both less verbose, and makes things easier to debug as a\n - failing test can have their state left behind under -d without\n - damaging the state for other tests.\n + We can instead use the pattern of creating a named sub-repository,\n + then we don't have to worry about cleaning up after ourselves, nobody\n + will care what state the broken \"hash-mismatch\" repository is after\n + this test runs.\n \n - But let's punt on that general refactoring and just change this one\n - test, I'm going to change it further in subsequent commits.\n + See [1] for related discussion on various \"modern\" test patterns that\n + can be used to avoid verbosity and increase reliability.\n +\n + 1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ t/t1450-fsck.sh: remove_object () {\n -\ttest_when_finished \"remove_object $cmt\" &&\n -\tgit update-ref refs/heads/bogus $cmt &&\n -\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n --\n --\ttest_must_fail git fsck 2>out &&\n --\ttest_i18ngrep \"$sha.*corrupt\" out\n +test_expect_success 'object with hash mismatch' '\n +\tgit init --bare hash-mismatch &&\n +\t(\n +\t\tcd hash-mismatch &&\n + \n +-\ttest_must_fail git fsck 2>out &&\n +-\ttest_i18ngrep \"$sha.*corrupt\" out\n +\t\toid=$(echo blob | git hash-object -w --stdin) &&\n +\t\told=$(test_oid_to_path \"$oid\") &&\n +\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n +\t\toid=\"$(dirname $new)$(basename $new)\" &&\n ++\n +\t\tmv objects/$old objects/$new &&\n +\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n +\t\ttree=$(git write-tree) &&\n +\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n +\t\tgit update-ref refs/heads/bogus $cmt &&\n ++\n +\t\ttest_must_fail git fsck 2>out &&\n -+\t\ttest_i18ngrep \"$oid.*corrupt\" out\n ++\t\tgrep \"$oid.*corrupt\" out\n +\t)\n '\n \n 3: d442a309178 ! 3: 1e40a4235e9 cat-file tests: test for missing object with -t and -s\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - cat-file tests: test for missing object with -t and -s\n + fsck tests: test current hash/type mismatch behavior\n \n - Test for what happens when the -t and -s flags are asked to operate on\n - a missing object, this extends tests added in 3e370f9faf0 (t1006: add\n - tests for git cat-file --allow-unknown-type, 2015-05-03). The -t and\n - -s flags are the only ones that can be combined with\n - --allow-unknown-type, so let's test with and without that flag.\n + If fsck we move an object around between .git/objects/?? directories\n + to simulate a hash mismatch \"git fsck\" will currently hard die() in\n + object-file.c. This behavior will be fixed in subsequent commits, but\n + let's test for it as-is for now.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n - ## t/t1006-cat-file.sh ##\n -@@ t/t1006-cat-file.sh: test_expect_success '%(deltabase) reports packed delta bases' '\n - \t}\n + ## t/t1450-fsck.sh ##\n +@@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n + \t)\n '\n \n -+missing_oid=$(test_oid deadbeef)\n -+test_expect_success 'error on type of missing object' '\n -+\tcat >expect.err <<-\\EOF &&\n -+\tfatal: git cat-file: could not get object info\n -+\tEOF\n -+\ttest_must_fail git cat-file -t $missing_oid >out 2>err &&\n -+\ttest_must_be_empty out &&\n -+\ttest_cmp expect.err err &&\n ++test_expect_success 'object with hash and type mismatch' '\n ++\tgit init --bare hash-type-mismatch &&\n ++\t(\n ++\t\tcd hash-type-mismatch &&\n +\n -+\ttest_must_fail git cat-file -t --allow-unknown-type $missing_oid >out 2>err &&\n -+\ttest_must_be_empty out &&\n -+\ttest_cmp expect.err err\n -+'\n ++\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n ++\t\told=$(test_oid_to_path \"$oid\") &&\n ++\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n ++\t\toid=\"$(dirname $new)$(basename $new)\" &&\n +\n -+test_expect_success 'error on size of missing object' '\n -+\tcat >expect.err <<-\\EOF &&\n -+\tfatal: git cat-file: could not get object info\n -+\tEOF\n -+\ttest_must_fail git cat-file -s $missing_oid >out 2>err &&\n -+\ttest_must_be_empty out &&\n -+\ttest_cmp expect.err err &&\n ++\t\tmv objects/$old objects/$new &&\n ++\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n ++\t\ttree=$(git write-tree) &&\n ++\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n ++\t\tgit update-ref refs/heads/bogus $cmt &&\n +\n -+\ttest_must_fail git cat-file -s --allow-unknown-type $missing_oid >out 2>err &&\n -+\ttest_must_be_empty out &&\n -+\ttest_cmp expect.err err\n ++\t\tcat >expect <<-\\EOF &&\n ++\t\tfatal: invalid object type\n ++\t\tEOF\n ++\t\ttest_must_fail git fsck 2>actual &&\n ++\t\ttest_cmp expect actual\n ++\t)\n +'\n +\n - bogus_type=\"bogus\"\n - bogus_content=\"bogus\"\n - bogus_size=$(strlen \"$bogus_content\")\n + test_expect_success 'branch pointing to non-commit' '\n + \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n + \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n 5: 82db40ebf8a ! 4: 854991c1543 rev-list tests: test for behavior with invalid object types\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - rev-list tests: test for behavior with invalid object types\n + fsck tests: test for garbage appended to a loose object\n \n - Fix a blindspot in the tests for the \"rev-list --disk-usage\" feature\n - added in 16950f8384a (rev-list: add --disk-usage option for\n - calculating disk usage, 2021-02-09) to test for what happens when it's\n - asked to calculate the disk usage of invalid object types.\n + There wasn't any output tests for this scenario, let's ensure that we\n + don't regress on it in the changes that come after this.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n - ## t/t6115-rev-list-du.sh ##\n -@@ t/t6115-rev-list-du.sh: check_du HEAD\n - check_du --objects HEAD\n - check_du --objects HEAD^..HEAD\n + ## t/t1450-fsck.sh ##\n +@@ t/t1450-fsck.sh: test_expect_success 'object with hash and type mismatch' '\n + \t)\n + '\n \n -+test_expect_success 'setup garbage repository' '\n -+\tgit clone --bare . garbage.git &&\n -+\tgarbage_oid=$(git -C garbage.git hash-object -t garbage -w --stdin --literally >$oidf &&\n +\n -+\t# Manually create a ref because \"update-ref\", \"tag\" etc. have\n -+\t# no corresponding --literally option.\n -+\techo $garbage_oid >garbage.git/refs/tags/garbage-tag &&\n -+\ttest_must_fail git -C garbage.git rev-list --objects --all --disk-usage\n ++\t\tcat >expect.error <<-EOF &&\n ++\t\terror: garbage at end of loose object '\\''$oid'\\''\n ++\t\terror: unable to unpack contents of ./$oidf\n ++\t\terror: $oid: object corrupt or missing: ./$oidf\n ++\t\tEOF\n ++\t\ttest_must_fail git fsck 2>actual &&\n ++\t\tgrep ^error: actual >error &&\n ++\t\ttest_cmp expect.error error\n ++\t)\n +'\n +\n - test_done\n + test_expect_success 'branch pointing to non-commit' '\n + \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n + \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n19: ad1614dbb8d ! 5: fc93c2c2530 fsck: don't hard die on invalid object types\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - fsck: don't hard die on invalid object types\n + cat-file tests: move bogus_* variable declarations earlier\n \n - Change the error fsck emits on invalid object types, such as:\n -\n - $ git hash-object --stdin -w -t garbage --literally \n -\n - From the very ungraceful error of:\n -\n - $ git fsck\n - fatal: invalid object type\n - $\n -\n - To:\n -\n - $ git fsck\n - error: hash mismatch for (expected )\n - error: : object corrupt or missing: \n - [ the rest of the fsck output here, i.e. it didn't hard die ]\n -\n - We'll still exit with non-zero, but now we'll finish the rest of the\n - traversal. The tests that's being added here asserts that we'll still\n - complain about other fsck issues (e.g. an unrelated dangling blob).\n -\n - To do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\n - flag from read_loose_object() through to parse_loose_header(). Since\n - the read_loose_object() function is only used in builtin/fsck.c we can\n - simply change it. See f6371f92104 (sha1_file: add read_loose_object()\n - function, 2017-01-13) for the introduction of read_loose_object().\n -\n - Why are we complaining about a \"hash mismatch\" for an object of a type\n - we don't know about? We shouldn't. This is the bare minimal change\n - needed to not make fsck hard die on a repository that's been corrupted\n - in this manner. In subsequent commits we'll teach fsck to recognize\n - this particular type of corruption and emit a better error message.\n + Change the short/long bogus bogus object type variables into a form\n + where the two sets can be used concurrently. This'll be used by\n + subsequently added tests.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n - ## builtin/fsck.c ##\n -@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n - \tvoid *contents;\n - \tint eaten;\n + ## t/t1006-cat-file.sh ##\n +@@ t/t1006-cat-file.sh: test_expect_success '%(deltabase) reports packed delta bases' '\n + \t}\n + '\n \n --\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n -+\tif (read_loose_object(path, oid, &type, &size, &contents,\n -+\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n - \t\terrors_found |= ERROR_OBJECT;\n - \t\terror(_(\"%s: object corrupt or missing: %s\"),\n - \t\t oid_to_hex(oid), path);\n -\n - ## object-file.c ##\n -@@ object-file.c: int read_loose_object(const char *path,\n - \t\t const struct object_id *expected_oid,\n - \t\t enum object_type *type,\n - \t\t unsigned long *size,\n --\t\t void **contents)\n -+\t\t void **contents,\n -+\t\t unsigned int oi_flags)\n - {\n - \tint ret = -1;\n - \tvoid *map = NULL;\n -@@ object-file.c: int read_loose_object(const char *path,\n - \tgit_zstream stream;\n - \tchar hdr[MAX_HEADER_LEN];\n - \tstruct object_info oi = OBJECT_INFO_INIT;\n -+\tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n - \toi.typep = type;\n - \toi.sizep = size;\n +-bogus_type=\"bogus\"\n +-bogus_content=\"bogus\"\n +-bogus_size=$(strlen \"$bogus_content\")\n +-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n ++test_expect_success 'setup bogus data' '\n ++\tbogus_short_type=\"bogus\" &&\n ++\tbogus_short_content=\"bogus\" &&\n ++\tbogus_short_size=$(strlen \"$bogus_short_content\") &&\n ++\tbogus_short_sha1=$(echo_without_newline \"$bogus_short_content\" | git hash-object -t $bogus_short_type --literally -w --stdin) &&\n ++\n ++\tbogus_long_type=\"abcdefghijklmnopqrstuvwxyz1234679\" &&\n ++\tbogus_long_content=\"bogus\" &&\n ++\tbogus_long_size=$(strlen \"$bogus_long_content\") &&\n ++\tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n ++'\n \n -@@ object-file.c: int read_loose_object(const char *path,\n - \t\tgit_inflate_end(&stream);\n - \t\tgoto out;\n - \t}\n --\tif (*type < 0)\n --\t\tdie(_(\"invalid object type\"));\n -+\tif (!allow_unknown && *type < 0) {\n -+\t\terror(_(\"header for %s declares an unknown type\"), path);\n -+\t\tgit_inflate_end(&stream);\n -+\t\tgoto out;\n -+\t}\n + test_expect_success \"Type of broken object is correct\" '\n +-\techo $bogus_type >expect &&\n +-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n ++\techo $bogus_short_type >expect &&\n ++\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n + \ttest_cmp expect actual\n + '\n \n - \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n - \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n -\n - ## object-store.h ##\n -@@ object-store.h: int read_loose_object(const char *path,\n - \t\t const struct object_id *expected_oid,\n - \t\t enum object_type *type,\n - \t\t unsigned long *size,\n --\t\t void **contents);\n -+\t\t void **contents,\n -+\t\t unsigned int oi_flags);\n + test_expect_success \"Size of broken object is correct\" '\n +-\techo $bogus_size >expect &&\n +-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n ++\techo $bogus_short_size >expect &&\n ++\tgit cat-file -s --allow-unknown-type $bogus_short_sha1 >actual &&\n + \ttest_cmp expect actual\n + '\n +-bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n +-bogus_content=\"bogus\"\n +-bogus_size=$(strlen \"$bogus_content\")\n +-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n - /* Retry packed storage after checking packed and loose storage */\n - #define HAS_OBJECT_RECHECK_PACKED 1\n -\n - ## t/t1450-fsck.sh ##\n -@@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n - \ttest_i18ngrep \"bad index file\" errors\n + test_expect_success \"Type of broken object is correct when type is large\" '\n +-\techo $bogus_type >expect &&\n +-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n ++\techo $bogus_long_type >expect &&\n ++\tgit cat-file -t --allow-unknown-type $bogus_long_sha1 >actual &&\n + \ttest_cmp expect actual\n '\n \n --test_expect_success 'fsck hard errors on an invalid object type' '\n -+test_expect_success 'fsck error and recovery on invalid object type' '\n - \tgit init --bare garbage-type &&\n - \tempty_blob=$(git -C garbage-type hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n --\tfatal: invalid object type\n --\tEOF\n --\ttest_must_fail git -C garbage-type fsck >out.actual 2>err.actual &&\n --\ttest_cmp err.expect err.actual &&\n --\ttest_must_be_empty out.actual\n -+\ttest_must_fail git -C garbage-type fsck >out 2>err &&\n -+\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n -+\ttest_line_count = 2 errors &&\n -+\tgrep \"error: hash mismatch for\" err &&\n -+\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n -+\tgrep \"dangling blob $empty_blob\" out\n + test_expect_success \"Size of large broken object is correct when type is large\" '\n +-\techo $bogus_size >expect &&\n +-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n ++\techo $bogus_long_size >expect &&\n ++\tgit cat-file -s --allow-unknown-type $bogus_long_sha1 >actual &&\n + \ttest_cmp expect actual\n '\n \n - test_done\n 4: 0358273022f ! 6: 051088aa114 cat-file tests: test that --allow-unknown-type isn't on by default\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - cat-file tests: test that --allow-unknown-type isn't on by default\n + cat-file tests: test for missing/bogus object with -t, -s and -p\n \n - Fix a blindspot in the tests for the --allow-unknown-type feature\n - added in 39e4ae38804 (cat-file: teach cat-file a\n - '--allow-unknown-type' option, 2015-05-03). We should check that\n - --allow-unknown-type isn't on by default.\n + When we look up a missing object with cat_one_file() what error we\n + print out currently depends on whether we'll error out early in\n + get_oid_with_context(), or if we'll get an error later from\n + oid_object_info_extended().\n \n - Before this change all the tests would succeed if --allow-unknown-type\n - was on by default, let's fix that by asserting that -t and -s die on a\n - \"garbage\" type without --allow-unknown-type.\n + The --allow-unknown-type flag then changes whether we pass the\n + \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\" flag to get_oid_with_context() or\n + not.\n +\n + The \"-p\" flag is yet another special-case in printing the same output\n + on the deadbeef OID as we'd emit on the deadbeef_short OID for the\n + \"-s\" and \"-t\" options, it also doesn't support the\n + \"--allow-unknown-type\" flag at all.\n +\n + Let's test the combination of the two sets of [-t, -s, -p] and\n + [--{no-}allow-unknown-type] (the --no-allow-unknown-type is implicit\n + in not supplying it), as well as a [missing,bogus] object pair.\n +\n + This extends tests added in 3e370f9faf0 (t1006: add tests for git\n + cat-file --allow-unknown-type, 2015-05-03).\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## t/oid-info/oid ##\n +@@ t/oid-info/oid: numeric\t\tsha1:0123456789012345678901234567890123456789\n + numeric\t\tsha256:0123456789012345678901234567890123456789012345678901234567890123\n + deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n + deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n ++deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n ++deadbee_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\n +\n ## t/t1006-cat-file.sh ##\n -@@ t/t1006-cat-file.sh: bogus_content=\"bogus\"\n - bogus_size=$(strlen \"$bogus_content\")\n - bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n +@@ t/t1006-cat-file.sh: test_expect_success 'setup bogus data' '\n + \tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n + '\n \n -+test_expect_success 'die on broken object under -t and -s without --allow-unknown-type' '\n -+\tcat >err.expect <<-\\EOF &&\n -+\tfatal: invalid object type\n -+\tEOF\n ++for arg1 in '' --allow-unknown-type\n ++do\n ++\tfor arg2 in -s -t -p\n ++\tdo\n ++\t\tif test $arg1 = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n ++\t\tthen\n ++\t\t\tcontinue\n ++\t\tfi\n ++\n +\n -+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n -+\ttest_cmp err.expect err.actual &&\n -+\ttest_must_be_empty out.actual &&\n ++\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus short OID\" '\n ++\t\t\tcat >expect <<-\\EOF &&\n ++\t\t\tfatal: invalid object type\n ++\t\t\tEOF\n +\n -+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n -+\ttest_cmp err.expect err.actual &&\n -+\ttest_must_be_empty out.actual\n -+'\n ++\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n ++\t\t\tthen\n ++\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n ++\t\t\telse\n ++\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_short_sha1 >out 2>actual &&\n ++\t\t\t\ttest_must_be_empty out &&\n ++\t\t\t\ttest_cmp expect actual\n ++\t\t\tfi\n ++\t\t'\n ++\n ++\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus full OID\" '\n ++\t\t\tif test \"$arg2\" = \"-p\"\n ++\t\t\tthen\n ++\t\t\t\tcat >expect <<-EOF\n ++\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n ++\t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n ++\t\t\t\tEOF\n ++\t\t\telse\n ++\t\t\t\tcat >expect <<-EOF\n ++\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n ++\t\t\t\tfatal: git cat-file: could not get object info\n ++\t\t\t\tEOF\n ++\t\t\tfi &&\n ++\n ++\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n ++\t\t\tthen\n ++\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n ++\t\t\telse\n ++\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_long_sha1 >out 2>actual &&\n ++\t\t\t\ttest_must_be_empty out &&\n ++\t\t\t\ttest_cmp expect actual\n ++\t\t\tfi\n ++\t\t'\n ++\n ++\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing short OID\" '\n ++\t\t\tcat >expect.err <<-EOF &&\n ++\t\t\tfatal: Not a valid object name $(test_oid deadbeef_short)\n ++\t\t\tEOF\n ++\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef_short) >out 2>err.actual &&\n ++\t\t\ttest_must_be_empty out\n ++\t\t'\n ++\n ++\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing full OID\" '\n ++\t\t\tif test \"$arg2\" = \"-p\"\n ++\t\t\tthen\n ++\t\t\t\tcat >expect.err <<-EOF\n ++\t\t\t\tfatal: Not a valid object name $(test_oid deadbeef)\n ++\t\t\t\tEOF\n ++\t\t\telse\n ++\t\t\t\tcat >expect.err <<-\\EOF\n ++\t\t\t\tfatal: git cat-file: could not get object info\n ++\t\t\t\tEOF\n ++\t\t\tfi &&\n ++\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef) >out 2>err.actual &&\n ++\t\t\ttest_must_be_empty out &&\n ++\t\t\ttest_cmp expect.err err.actual\n ++\t\t'\n ++\tdone\n ++done\n +\n test_expect_success \"Type of broken object is correct\" '\n - \techo $bogus_type >expect &&\n - \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n -@@ t/t1006-cat-file.sh: bogus_content=\"bogus\"\n - bogus_size=$(strlen \"$bogus_content\")\n - bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n - \n -+test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n -+\tcat >err.expect <<-EOF &&\n -+\terror: unable to unpack $bogus_sha1 header\n -+\tfatal: git cat-file: could not get object info\n -+\tEOF\n -+\n -+\ttest_must_fail git cat-file -t $bogus_sha1 >out.actual 2>err.actual &&\n -+\ttest_cmp err.expect err.actual &&\n -+\ttest_must_be_empty out.actual &&\n -+\n -+\ttest_must_fail git cat-file -s $bogus_sha1 >out.actual 2>err.actual &&\n -+\ttest_cmp err.expect err.actual &&\n -+\ttest_must_be_empty out.actual\n -+'\n -+\n - test_expect_success \"Type of broken object is correct when type is large\" '\n - \techo $bogus_type >expect &&\n - \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n + \techo $bogus_short_type >expect &&\n + \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n 6: d1ffd21acc5 = 7: 20bd81c1af0 cat-file tests: add corrupt loose object test\n 7: 22ab12c2282 ! 8: cd1d52b8a07 cat-file tests: test for current --allow-unknown-type behavior\n @@ Commit message\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## t/t1006-cat-file.sh ##\n -@@ t/t1006-cat-file.sh: test_expect_success 'die on broken object under -t and -s without --allow-unknow\n - \ttest_must_be_empty out.actual\n - '\n +@@ t/t1006-cat-file.sh: do\n + \tdone\n + done\n \n +test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n -+\tgit cat-file -e $bogus_sha1\n ++\tgit cat-file -e $bogus_short_sha1\n +'\n +\n +test_expect_success '-e can not be combined with --allow-unknown-type' '\n -+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_sha1\n ++\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_short_sha1\n +'\n +\n +test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n -+\ttest_must_fail git cat-file -p $bogus_sha1 &&\n -+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_sha1\n ++\ttest_must_fail git cat-file -p $bogus_short_sha1 &&\n ++\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_short_sha1\n +'\n +\n +test_expect_success ' does not work with objects of broken types' '\n +\tcat >err.expect <<-\\EOF &&\n +\tfatal: invalid object type \"bogus\"\n +\tEOF\n -+\ttest_must_fail git cat-file $bogus_type $bogus_sha1 2>err.actual &&\n ++\ttest_must_fail git cat-file $bogus_short_type $bogus_short_sha1 2>err.actual &&\n +\ttest_cmp err.expect err.actual\n +'\n +\n +test_expect_success 'broken types combined with --batch and --batch-check' '\n -+\techo $bogus_sha1 >bogus-oid &&\n ++\techo $bogus_short_sha1 >bogus-oid &&\n +\n +\tcat >err.expect <<-\\EOF &&\n +\tfatal: invalid object type\n @@ t/t1006-cat-file.sh: test_expect_success 'die on broken object under -t and -s w\n +\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect &&\n - \tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n -@@ t/t1006-cat-file.sh: test_expect_success \"Size of broken object is correct\" '\n - \tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n - \ttest_cmp expect actual\n - '\n +\n +test_expect_success 'the --allow-unknown-type option does not consider replacement refs' '\n +\tcat >expect <<-EOF &&\n -+\t$bogus_type\n ++\t$bogus_short_type\n +\tEOF\n -+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n ++\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n +\ttest_cmp expect actual &&\n +\n +\t# Create it manually, as \"git replace\" will die on bogus\n +\t# types.\n +\thead=$(git rev-parse --verify HEAD) &&\n ++\ttest_when_finished \"rm -rf .git/refs/replace\" &&\n +\tmkdir -p .git/refs/replace &&\n -+\techo $head >.git/refs/replace/$bogus_sha1 &&\n ++\techo $head >.git/refs/replace/$bogus_short_sha1 &&\n +\n +\tcat >expect <<-EOF &&\n +\tcommit\n +\tEOF\n -+\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n ++\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n +\ttest_cmp expect actual\n +'\n +\n - bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n - bogus_content=\"bogus\"\n - bogus_size=$(strlen \"$bogus_content\")\n + test_expect_success \"Type of broken object is correct\" '\n + \techo $bogus_short_type >expect &&\n + \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n 8: 38e4266772d = 9: d9f5adfc74b object-file.c: don't set \"typep\" when returning non-zero\n 9: 5b9278e7bb4 < -: ----------- cache.h: move object functions to object-store.h\n16: 9e7dbfb4aa3 ! 10: 51d14bc9274 object-file.c: return -1, not \"status\" from unpack_loose_header()\n @@ Commit message\n \n ## object-file.c ##\n @@ object-file.c: int unpack_loose_header(git_zstream *stream,\n - \tstatus = git_inflate(stream, 0);\n - \tobj_read_lock();\n + \t\t\t\t\t buffer, bufsiz);\n + \n \tif (status < Z_OK)\n -\t\treturn status;\n +\t\treturn -1;\n \n - \t/*\n - \t * Check if entire header is unpacked in the first iteration.\n + \t/* Make sure we have the terminating NUL */\n + \tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n10: b15ad53414b ! 11: f43cfd8a5ed object-file.c: make parse_loose_header_extended() public\n @@ Commit message\n could simply use parse_loose_header_extended() there, but will leave\n the API in a better end state.\n \n + It would be a better end-state to have already moved the declaration\n + of these functions to object-store.h to avoid the forward declaration\n + of \"struct object_info\" in cache.h, but let's leave that cleanup for\n + some other time.\n +\n + 1. https://lore.kernel.org/git/patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com/\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## cache.h ##\n +@@ cache.h: char *xdg_cache_home(const char *filename);\n + int git_open_cloexec(const char *name, int flags);\n + #define git_open(name) git_open_cloexec(name, O_RDONLY)\n + int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n +-int parse_loose_header(const char *hdr, unsigned long *sizep);\n ++struct object_info;\n ++int parse_loose_header(const char *hdr, struct object_info *oi,\n ++\t\t unsigned int flags);\n + \n + int check_object_signature(struct repository *r, const struct object_id *oid,\n + \t\t\t void *buf, unsigned long size, const char *type);\n +\n ## object-file.c ##\n @@ object-file.c: static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n @@ object-file.c: static void *unpack_loose_rest(git_zstream *stream,\n */\n -static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n -\t\t\t\t unsigned int flags)\n -+int parse_loose_header(const char *hdr,\n -+\t\t struct object_info *oi,\n ++int parse_loose_header(const char *hdr, struct object_info *oi,\n +\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n @@ object-file.c: int read_loose_object(const char *path,\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \n - ## object-store.h ##\n -@@ object-store.h: int for_each_packed_object(each_packed_object_fn, void *,\n - int unpack_loose_header(git_zstream *stream, unsigned char *map,\n - \t\t\tunsigned long mapsize, void *buffer,\n - \t\t\tunsigned long bufsiz);\n --int parse_loose_header(const char *hdr, unsigned long *sizep);\n -+int parse_loose_header(const char *hdr, struct object_info *oi,\n -+\t\t unsigned int flags);\n - int check_object_signature(struct repository *r, const struct object_id *oid,\n - \t\t\t void *buf, unsigned long size, const char *type);\n - int finalize_object_file(const char *tmpfile, const char *filename);\n -\n ## streaming.c ##\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n11: 326eb74545d < -: ----------- object-file.c: add missing braces to loose_object_info()\n12: 4f829e9b727 ! 12: 50d938f7f3c object-file.c: simplify unpack_loose_short_header()\n @@ Commit message\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## cache.h ##\n +@@ cache.h: char *xdg_cache_home(const char *filename);\n + \n + int git_open_cloexec(const char *name, int flags);\n + #define git_open(name) git_open_cloexec(name, O_RDONLY)\n +-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n ++\n ++/**\n ++ * unpack_loose_header() initializes the data stream needed to unpack\n ++ * a loose object header.\n ++ *\n ++ * Returns 0 on success. Returns negative values on error.\n ++ *\n ++ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n ++ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n ++ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n ++ * reporting. The full header will be extracted to \"hdrbuf\" for use\n ++ * with parse_loose_header().\n ++ */\n ++int unpack_loose_header(git_zstream *stream, unsigned char *map,\n ++\t\t\tunsigned long mapsize, void *buffer,\n ++\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n + struct object_info;\n + int parse_loose_header(const char *hdr, struct object_info *oi,\n + \t\t unsigned int flags);\n +\n ## object-file.c ##\n @@ object-file.c: void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n @@ object-file.c: static int unpack_loose_short_header(git_zstream *stream,\n -\tint status = unpack_loose_short_header(stream, map, mapsize,\n -\t\t\t\t\t buffer, bufsiz);\n -\n - \tif (status < Z_OK)\n - \t\treturn status;\n - \n +-\tif (status < Z_OK)\n +-\t\treturn -1;\n +-\n -\t/* Make sure we have the terminating NUL */\n -\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n -\t\treturn -1;\n @@ object-file.c: static int unpack_loose_short_header(git_zstream *stream,\n -\tint status;\n -\n -\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n --\tif (status < Z_OK)\n --\t\treturn -1;\n --\n - \t/*\n - \t * Check if entire header is unpacked in the first iteration.\n - \t */\n + \tif (status < Z_OK)\n + \t\treturn -1;\n + \n +@@ object-file.c: static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n @@ object-file.c: static int unpack_loose_short_header(git_zstream *stream,\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n @@ object-file.c: static int loose_object_info(struct repository *r,\n - \tunsigned long mapsize;\n - \tvoid *map;\n - \tgit_zstream stream;\n -+\tint hdr_ret;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n @@ object-file.c: static int loose_object_info(struct repository *r,\n -\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n -\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n -\t\t\t\t oid_to_hex(oid));\n --\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n +-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n +\n -+\thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n -+\t\t\t\t allow_unknown ? &hdrbuf : NULL);\n -+\tif (hdr_ret < 0) {\n ++\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n ++\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n - \t}\n + \tif (status < 0)\n @@ object-file.c: int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n @@ object-file.c: int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n - ## object-store.h ##\n -@@ object-store.h: int for_each_object_in_pack(struct packed_git *p,\n - int for_each_packed_object(each_packed_object_fn, void *,\n - \t\t\t enum for_each_object_flags flags);\n - \n -+/**\n -+ * unpack_loose_header() initializes the data stream needed to unpack\n -+ * a loose object header.\n -+ *\n -+ * Returns 0 on success. Returns negative values on error.\n -+ *\n -+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n -+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n -+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n -+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n -+ * with parse_loose_header().\n -+ */\n - int unpack_loose_header(git_zstream *stream, unsigned char *map,\n - \t\t\tunsigned long mapsize, void *buffer,\n --\t\t\tunsigned long bufsiz);\n -+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n - int parse_loose_header(const char *hdr, struct object_info *oi,\n - \t\t unsigned int flags);\n - int check_object_signature(struct repository *r, const struct object_id *oid,\n -\n ## streaming.c ##\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n13: 90489d9e6ec < -: ----------- object-file.c: split up ternary in parse_loose_header()\n18: 1b7173a5b5b ! 13: 755fde00b46 object-file.c: use \"enum\" return type for unpack_loose_header()\n @@ Metadata\n ## Commit message ##\n object-file.c: use \"enum\" return type for unpack_loose_header()\n \n - In the preceding commits we changed and documented\n - unpack_loose_header() from return any negative value or zero, to only\n - -2, -1 or 0. Let's instead add an \"enum unpack_loose_header_result\"\n - type and use it, and have the compiler assert that we're exhaustively\n - covering all return values. This gets rid of the need for having a\n - \"default\" BUG() case in loose_object_info().\n + In a preceding commit we changed and documented unpack_loose_header()\n + from its previous behavior of returning any negative value or zero, to\n + only -1 or 0.\n \n - I'm on the fence about whether this is more readable or worth it, but\n - since it was suggested in [1] to do this let's go for it.\n -\n - 1. https://lore.kernel.org/git/20210527175433.2673306-1-jonathantanmy@google.com/\n + Let's add an \"enum unpack_loose_header_result\" type and use it for\n + these return values, and have the compiler assert that we're\n + exhaustively covering all of them.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## cache.h ##\n +@@ cache.h: int git_open_cloexec(const char *name, int flags);\n + * unpack_loose_header() initializes the data stream needed to unpack\n + * a loose object header.\n + *\n +- * Returns 0 on success. Returns negative values on error.\n ++ * Returns:\n ++ *\n ++ * - ULHR_OK on success\n ++ * - ULHR_BAD on error\n + *\n + * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n + * \"hdrbuf\" argument is non-NULL. This is intended for use with\n +@@ cache.h: int git_open_cloexec(const char *name, int flags);\n + * reporting. The full header will be extracted to \"hdrbuf\" for use\n + * with parse_loose_header().\n + */\n +-int unpack_loose_header(git_zstream *stream, unsigned char *map,\n +-\t\t\tunsigned long mapsize, void *buffer,\n +-\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n ++enum unpack_loose_header_result {\n ++\tULHR_OK,\n ++\tULHR_BAD,\n ++};\n ++enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n ++\t\t\t\t\t\t unsigned char *map,\n ++\t\t\t\t\t\t unsigned long mapsize,\n ++\t\t\t\t\t\t void *buffer,\n ++\t\t\t\t\t\t unsigned long bufsiz,\n ++\t\t\t\t\t\t struct strbuf *hdrbuf);\n ++\n + struct object_info;\n + int parse_loose_header(const char *hdr, struct object_info *oi,\n + \t\t unsigned int flags);\n +\n ## object-file.c ##\n @@ object-file.c: void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n @@ object-file.c: void *map_loose_object(struct repository *r,\n {\n \tint status;\n \n +@@ object-file.c: int unpack_loose_header(git_zstream *stream,\n + \tstatus = git_inflate(stream, 0);\n + \tobj_read_lock();\n + \tif (status < Z_OK)\n +-\t\treturn -1;\n ++\t\treturn ULHR_BAD;\n + \n + \t/*\n + \t * Check if entire header is unpacked in the first iteration.\n + \t */\n + \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n +-\t\treturn 0;\n ++\t\treturn ULHR_OK;\n + \n + \t/*\n + \t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n +@@ object-file.c: int unpack_loose_header(git_zstream *stream,\n + \t * --allow-unknown-type\".\n + \t */\n + \tif (!header)\n +-\t\treturn -1;\n ++\t\treturn ULHR_BAD;\n + \n + \t/*\n + \t * buffer[0..bufsiz] was not large enough. Copy the partial\n +@@ object-file.c: int unpack_loose_header(git_zstream *stream,\n + \t\tstream->next_out = buffer;\n + \t\tstream->avail_out = bufsiz;\n + \t} while (status != Z_STREAM_END);\n +-\treturn -1;\n ++\treturn ULHR_BAD;\n + }\n + \n + static void *unpack_loose_rest(git_zstream *stream,\n @@ object-file.c: static int loose_object_info(struct repository *r,\n - \tunsigned long mapsize;\n - \tvoid *map;\n - \tgit_zstream stream;\n --\tint hdr_ret;\n -+\tenum unpack_loose_header_result hdr_ret;\n - \tchar hdr[MAX_HEADER_LEN];\n - \tstruct strbuf hdrbuf = STRBUF_INIT;\n - \tunsigned long size_scratch;\n -@@ object-file.c: static int loose_object_info(struct repository *r,\n - \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n - \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n - \tswitch (hdr_ret) {\n --\tcase 0:\n -+\tcase UNPACK_LOOSE_HEADER_RESULT_OK:\n - \t\tbreak;\n --\tcase -1:\n -+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD:\n + \tif (oi->disk_sizep)\n + \t\t*oi->disk_sizep = mapsize;\n + \n +-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n +-\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n ++\tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n ++\t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n ++\tcase ULHR_OK:\n ++\t\tbreak;\n ++\tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n - \t\tbreak;\n --\tcase -2:\n -+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG:\n - \t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n - \t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n - \t\tbreak;\n --\tdefault:\n --\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n - \t}\n - \tif (!status) {\n - \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n -\n - ## object-store.h ##\n -@@ object-store.h: int for_each_object_in_pack(struct packed_git *p,\n - int for_each_packed_object(each_packed_object_fn, void *,\n - \t\t\t enum for_each_object_flags flags);\n - \n -+enum unpack_loose_header_result {\n -+\tUNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG = -2,\n -+\tUNPACK_LOOSE_HEADER_RESULT_BAD = -1,\n -+\tUNPACK_LOOSE_HEADER_RESULT_OK,\n -+\n -+};\n +-\tif (status < 0)\n +-\t\t; /* Do nothing */\n +-\telse if (hdrbuf.len) {\n ++\t\tbreak;\n ++\t}\n +\n - /**\n - * unpack_loose_header() initializes the data stream needed to unpack\n - * a loose object header.\n - *\n -- * Returns 0 on success. Returns negative values on error. If the\n -- * header exceeds MAX_HEADER_LEN -2 will be returned.\n -+ * Returns UNPACK_LOOSE_HEADER_RESULT_OK on success. Returns\n -+ * UNPACK_LOOSE_HEADER_RESULT_BAD values on error, or if the header\n -+ * exceeds MAX_HEADER_LEN UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG will\n -+ * be returned.\n - *\n - * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n - * \"hdrbuf\" argument is non-NULL. This is intended for use with\n - * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n - * reporting. The full header will be extracted to \"hdrbuf\" for use\n -- * with parse_loose_header(), -2 will still be returned from this\n -- * function to indicate that the header was too long.\n -+ * with parse_loose_header(), UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG\n -+ * will still be returned from this function to indicate that the\n -+ * header was too long.\n - */\n --int unpack_loose_header(git_zstream *stream, unsigned char *map,\n --\t\t\tunsigned long mapsize, void *buffer,\n --\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n -+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n -+\t\t\t\t\t\t unsigned char *map,\n -+\t\t\t\t\t\t unsigned long mapsize,\n -+\t\t\t\t\t\t void *buffer,\n -+\t\t\t\t\t\t unsigned long bufsiz,\n -+\t\t\t\t\t\t struct strbuf *hdrbuf);\n - \n - /**\n - * parse_loose_header() parses the starting \" \\0\" of an\n ++\tif (status < 0) {\n ++\t\t/* Do nothing */\n ++\t} else if (hdrbuf.len) {\n + \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n + \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n + \t\t\t\t oid_to_hex(oid));\n \n ## streaming.c ##\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n - \t\t\t enum object_type *type)\n - {\n - \tstruct object_info oi = OBJECT_INFO_INIT;\n -+\tenum unpack_loose_header_result hdr_ret;\n - \toi.sizep = &st->size;\n - \toi.typep = type;\n - \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct reposi\n -\t\t\t\t st->u.loose.hdr,\n -\t\t\t\t sizeof(st->u.loose.hdr),\n -\t\t\t\t NULL) < 0) ||\n --\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n --\t *type < 0) {\n +-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n -\t\tgit_inflate_end(&st->z);\n -\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n -\t\treturn -1;\n -+\thdr_ret = unpack_loose_header(&st->z, st->u.loose.mapped,\n -+\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n -+\t\t\t\t sizeof(st->u.loose.hdr), NULL);\n -+\tswitch (hdr_ret) {\n -+\tcase UNPACK_LOOSE_HEADER_RESULT_OK:\n ++\tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n ++\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n ++\t\t\t\t sizeof(st->u.loose.hdr), NULL)) {\n ++\tcase ULHR_OK:\n +\t\tbreak;\n -+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD:\n -+\tcase UNPACK_LOOSE_HEADER_RESULT_BAD_TOO_LONG:\n ++\tcase ULHR_BAD:\n +\t\tgoto error;\n \t}\n -+\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n ++\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n +\t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n17: f28c4f0dfb4 ! 14: 522d71eb19d object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - object-file.c: return -2 on \"header too long\" in unpack_loose_header()\n + object-file.c: return ULHR_TOO_LONG on \"header too long\"\n \n Split up the return code for \"header too long\" from the generic\n negative return value unpack_loose_header() returns, and report via\n @@ Commit message\n As a test added earlier in this series in t1006-cat-file.sh shows\n we'll correctly emit zlib errors from zlib.c already in this case, so\n we have no need to carry those return codes further down the\n - stack. Let's instead just return -2 saying we ran into the\n + stack. Let's instead just return ULHR_TOO_LONG saying we ran into the\n MAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## cache.h ##\n +@@ cache.h: int git_open_cloexec(const char *name, int flags);\n + *\n + * - ULHR_OK on success\n + * - ULHR_BAD on error\n ++ * - ULHR_TOO_LONG if the header was too long\n + *\n + * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n + * \"hdrbuf\" argument is non-NULL. This is intended for use with\n + * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n + * reporting. The full header will be extracted to \"hdrbuf\" for use\n +- * with parse_loose_header().\n ++ * with parse_loose_header(), ULHR_TOO_LONG will still be returned\n ++ * from this function to indicate that the header was too long.\n + */\n + enum unpack_loose_header_result {\n + \tULHR_OK,\n + \tULHR_BAD,\n ++\tULHR_TOO_LONG,\n + };\n + enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n + \t\t\t\t\t\t unsigned char *map,\n +\n ## object-file.c ##\n -@@ object-file.c: int unpack_loose_header(git_zstream *stream,\n +@@ object-file.c: enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n --\t\treturn -1;\n -+\t\treturn -2;\n +-\t\treturn ULHR_BAD;\n ++\t\treturn ULHR_TOO_LONG;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n -@@ object-file.c: int unpack_loose_header(git_zstream *stream,\n +@@ object-file.c: enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n --\treturn -1;\n -+\treturn -2;\n +-\treturn ULHR_BAD;\n ++\treturn ULHR_TOO_LONG;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n @@ object-file.c: static int loose_object_info(struct repository *r,\n - \n - \thdr_ret = unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n - \t\t\t\t allow_unknown ? &hdrbuf : NULL);\n --\tif (hdr_ret < 0) {\n -+\tswitch (hdr_ret) {\n -+\tcase 0:\n -+\t\tbreak;\n -+\tcase -1:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n -+\t\tbreak;\n -+\tcase -2:\n + \t\tbreak;\n ++\tcase ULHR_TOO_LONG:\n +\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n +\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n +\t\tbreak;\n -+\tdefault:\n -+\t\tBUG(\"unknown hdr_ret value %d\", hdr_ret);\n \t}\n - \tif (!status) {\n - \t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n + \n + \tif (status < 0) {\n \n - ## object-store.h ##\n -@@ object-store.h: int for_each_packed_object(each_packed_object_fn, void *,\n - * unpack_loose_header() initializes the data stream needed to unpack\n - * a loose object header.\n - *\n -- * Returns 0 on success. Returns negative values on error.\n -+ * Returns 0 on success. Returns negative values on error. If the\n -+ * header exceeds MAX_HEADER_LEN -2 will be returned.\n - *\n - * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n - * \"hdrbuf\" argument is non-NULL. This is intended for use with\n - * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n - * reporting. The full header will be extracted to \"hdrbuf\" for use\n -- * with parse_loose_header().\n -+ * with parse_loose_header(), -2 will still be returned from this\n -+ * function to indicate that the header was too long.\n - */\n - int unpack_loose_header(git_zstream *stream, unsigned char *map,\n - \t\t\tunsigned long mapsize, void *buffer,\n + ## streaming.c ##\n +@@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n + \tcase ULHR_OK:\n + \t\tbreak;\n + \tcase ULHR_BAD:\n ++\tcase ULHR_TOO_LONG:\n + \t\tgoto error;\n + \t}\n + \tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n \n ## t/t1006-cat-file.sh ##\n -@@ t/t1006-cat-file.sh: bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_t\n - \n - test_expect_success 'die on broken object with large type under -t and -s without --allow-unknown-type' '\n - \tcat >err.expect <<-EOF &&\n --\terror: unable to unpack $bogus_sha1 header\n -+\terror: header for $bogus_sha1 too long, exceeds 32 bytes\n - \tfatal: git cat-file: could not get object info\n - \tEOF\n - \n +@@ t/t1006-cat-file.sh: do\n + \t\t\tif test \"$arg2\" = \"-p\"\n + \t\t\tthen\n + \t\t\t\tcat >expect <<-EOF\n +-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n ++\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n + \t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n + \t\t\t\tEOF\n + \t\t\telse\n + \t\t\t\tcat >expect <<-EOF\n +-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n ++\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n + \t\t\t\tfatal: git cat-file: could not get object info\n + \t\t\t\tEOF\n + \t\t\tfi &&\n14: 7c9819d37c5 ! 15: 1ca875395c1 object-file.c: stop dying in parse_loose_header()\n @@ Metadata\n ## Commit message ##\n object-file.c: stop dying in parse_loose_header()\n \n - Start the libification of parse_loose_header() by making it return\n - error codes and data instead of invoking die() by itself. For now\n - we'll move the relevant die() call to loose_object_info() and\n - read_loose_object() to keep this change smaller, but in subsequent\n - commits we'll also libify those.\n + Make parse_loose_header() return error codes and data instead of\n + invoking die() by itself.\n \n - Since the refactoring of parse_loose_header_extended() into\n - parse_loose_header() in an earlier commit, its interface accepts a\n - \"unsigned long *sizep\". Rather it accepts a \"struct object_info *\",\n - that structure will be populated with information about the object.\n + For now we'll move the relevant die() call to loose_object_info() and\n + read_loose_object() to keep this change smaller. In a subsequent\n + commit we'll make read_loose_object() return an error code instead of\n + dying. We should also address the \"allow_unknown\" case (should be\n + moved to builtin/cat-file.c), but for now I'll be leaving it.\n \n - It thus makes sense to further libify the interface so that it stops\n - calling die() when it encounters OBJ_BAD, and instead rely on its\n - callers to check the populated \"oi->typep\".\n + For making parse_loose_header() not die() change its prototype to\n + accept a \"struct object_info *\" instead of the \"unsigned long *sizep\"\n + it accepted before. Its callers can now check the populated populated\n + \"oi->typep\".\n \n Because of this we don't need to pass in the \"unsigned int flags\"\n which we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\n @@ Commit message\n variable. In some cases we set it to the return value of \"error()\",\n i.e. -1, and later checked if \"status < 0\" was true.\n \n - In another case added in c84a1f3ed4d (sha1_file: refactor read_object,\n - 2017-06-21) (but the behavior pre-dated that) we did checks of \"status\n - >= 0\", because at that point \"status\" had become the return value of\n - parse_loose_header(). I.e. a non-negative \"enum object_type\" (unless\n - we -1, aka. OBJ_BAD).\n + Since 93cff9a978e (sha1_loose_object_info: return error for corrupted\n + objects, 2017-04-01) the return value of loose_object_info() (then\n + named sha1_loose_object_info()) had been a \"status\" variable that be\n + any negative value, as we were expecting to return the \"enum\n + object_type\".\n \n - Now that parse_loose_header() will return 0 on success instead of the\n + The only negative type happens to be OBJ_BAD, but the code still\n + assumed that more might be added. This was then used later in\n + e.g. c84a1f3ed4d (sha1_file: refactor read_object, 2017-06-21). Now\n + that parse_loose_header() will return 0 on success instead of the\n type (which it'll stick into the \"struct object_info\") we don't need\n to conflate these two cases in its callers.\n \n + Since parse_loose_header() doesn't need to return an arbitrary\n + \"status\" we only need to treat its \"ret < 0\" specially, but can\n + idiomatically overwrite it with our own error() return. This along\n + with having made unpack_loose_header() return an \"enum\n + unpack_loose_header_result\" in an earlier commit means that we can\n + move the previously nested if/else cases mostly into the \"ULHR_OK\"\n + branch of the \"switch\" statement.\n +\n + We should be less silent if we reach that \"status = -1\" branch, which\n + happens if we've got trailing garbage in loose objects, see\n + f6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\n + for a better way to handle it. For now let's punt on it, a subsequent\n + commit will address that edge case.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + ## cache.h ##\n +@@ cache.h: enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n + \t\t\t\t\t\t unsigned long bufsiz,\n + \t\t\t\t\t\t struct strbuf *hdrbuf);\n + \n ++/**\n ++ * parse_loose_header() parses the starting \" \\0\" of an\n ++ * object. If it doesn't follow that format -1 is returned. To check\n ++ * the validity of the populate the \"typep\" in the \"struct\n ++ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n ++ * parsed can be retrieved via \"oi->sizep\", and from there\n ++ * passed to unpack_loose_rest().\n ++ */\n + struct object_info;\n +-int parse_loose_header(const char *hdr, struct object_info *oi,\n +-\t\t unsigned int flags);\n ++int parse_loose_header(const char *hdr, struct object_info *oi);\n + \n + int check_object_signature(struct repository *r, const struct object_id *oid,\n + \t\t\t void *buf, unsigned long size, const char *type);\n +\n ## object-file.c ##\n @@ object-file.c: static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n --int parse_loose_header(const char *hdr,\n --\t\t struct object_info *oi,\n +-int parse_loose_header(const char *hdr, struct object_info *oi,\n -\t\t unsigned int flags)\n +int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n -@@ object-file.c: int parse_loose_header(const char *hdr,\n +@@ object-file.c: int parse_loose_header(const char *hdr, struct object_info *oi,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n @@ object-file.c: int parse_loose_header(const char *hdr,\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n -@@ object-file.c: int parse_loose_header(const char *hdr,\n - \tif (*hdr)\n - \t\treturn -1;\n - \n --\treturn type;\n +@@ object-file.c: int parse_loose_header(const char *hdr, struct object_info *oi,\n + \t/*\n + \t * The length must be followed by a zero byte\n + \t */\n +-\treturn *hdr ? -1 : type;\n ++\tif (*hdr)\n ++\t\treturn -1;\n ++\n +\t/*\n +\t * The format is valid, but the type may still be bogus. The\n +\t * Caller needs to check its oi->typep.\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n +\tenum object_type type_scratch;\n -+\tint parsed_header = 0;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n @@ object-file.c: static int loose_object_info(struct repository *r,\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n @@ object-file.c: static int loose_object_info(struct repository *r,\n + \tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n + \t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n + \tcase ULHR_OK:\n ++\t\tif (parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0)\n ++\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n ++\t\telse if (!allow_unknown && *oi->typep < 0)\n ++\t\t\tdie(_(\"invalid object type\"));\n ++\n ++\t\tif (!oi->contentp)\n ++\t\t\tbreak;\n ++\t\t*oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);\n ++\t\tif (*oi->contentp)\n ++\t\t\tgoto cleanup;\n ++\n ++\t\tstatus = -1;\n + \t\tbreak;\n + \tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n - \t\t\t oid_to_hex(oid));\n +@@ object-file.c: static int loose_object_info(struct repository *r,\n + \t\tbreak;\n \t}\n --\n + \n -\tif (status < 0) {\n -\t\t/* Do nothing */\n -\t} else if (hdrbuf.len) {\n -\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n -\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n -\t\t\t\t oid_to_hex(oid));\n --\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0) {\n +-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n -\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n -+\tif (!status) {\n -+\t\tif (!parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi))\n -+\t\t\t/*\n -+\t\t\t * oi->{sizep,typep} are meaningless unless\n -+\t\t\t * parse_loose_header() returns >= 0.\n -+\t\t\t */\n -+\t\t\tparsed_header = 1;\n -+\t\telse\n -+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n - \t}\n -+\tif (!allow_unknown && parsed_header && *oi->typep < 0)\n -+\t\tdie(_(\"invalid object type\"));\n - \n +-\n -\tif (status >= 0 && oi->contentp) {\n -+\tif (parsed_header && oi->contentp) {\n - \t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n - \t\t\t\t\t\t *oi->sizep, oid);\n - \t\tif (!*oi->contentp) {\n -@@ object-file.c: static int loose_object_info(struct repository *r,\n +-\t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n +-\t\t\t\t\t\t *oi->sizep, oid);\n +-\t\tif (!*oi->contentp) {\n +-\t\t\tgit_inflate_end(&stream);\n +-\t\t\tstatus = -1;\n +-\t\t}\n +-\t} else\n +-\t\tgit_inflate_end(&stream);\n +-\n ++\tgit_inflate_end(&stream);\n ++cleanup:\n + \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n +\tif (oi->typep == &type_scratch)\n +\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n - \treturn (status < 0) ? status : 0;\n +-\treturn (status < 0) ? status : 0;\n ++\treturn status;\n }\n + \n + int obj_read_use_lock = 0;\n @@ object-file.c: int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n @@ object-file.c: int read_loose_object(const char *path,\n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \n - ## object-store.h ##\n -@@ object-store.h: int for_each_packed_object(each_packed_object_fn, void *,\n - int unpack_loose_header(git_zstream *stream, unsigned char *map,\n - \t\t\tunsigned long mapsize, void *buffer,\n - \t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n --int parse_loose_header(const char *hdr, struct object_info *oi,\n --\t\t unsigned int flags);\n -+\n -+/**\n -+ * parse_loose_header() parses the starting \" \\0\" of an\n -+ * object. If it doesn't follow that format -1 is returned. To check\n -+ * the validity of the populate the \"typep\" in the \"struct\n -+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n -+ * parsed can be retrieved via \"oi->sizep\", and from there\n -+ * passed to unpack_loose_rest().\n -+ */\n -+int parse_loose_header(const char *hdr, struct object_info *oi);\n -+\n - int check_object_signature(struct repository *r, const struct object_id *oid,\n - \t\t\t void *buf, unsigned long size, const char *type);\n - int finalize_object_file(const char *tmpfile, const char *filename);\n -\n ## streaming.c ##\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct reposi\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n @@ streaming.c: static int open_istream_loose(struct git_istream *st, struct repository *r,\n - \t\t\t\t st->u.loose.hdr,\n - \t\t\t\t sizeof(st->u.loose.hdr),\n - \t\t\t\t NULL) < 0) ||\n --\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n -+\t (parse_loose_header(st->u.loose.hdr, &oi) < 0) ||\n -+\t *type < 0) {\n - \t\tgit_inflate_end(&st->z);\n - \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n - \t\treturn -1;\n + \tcase ULHR_TOO_LONG:\n + \t\tgoto error;\n + \t}\n +-\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n ++\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n + \t\tgoto error;\n + \n + \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n15: 3fb660ff944 < -: ----------- object-file.c: guard against future bugs in loose_object_info()\n20: 3bf3cf2299d < -: ----------- object-store.h: move read_loose_object() below 'struct object_info'\n21: 974f650cddf ! 16: d38067feab3 fsck: report invalid types recorded in objects\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - fsck: report invalid types recorded in objects\n + fsck: don't hard die on invalid object types\n \n - Continue the work in the preceding commit and improve the error on:\n + Change the error fsck emits on invalid object types, such as:\n \n $ git hash-object --stdin -w -t garbage --literally \n +\n + From the very ungraceful error of:\n +\n $ git fsck\n - error: hash mismatch for (expected )\n - error: : object corrupt or missing: \n - [ other fsck output ]\n + fatal: invalid object type\n + $\n \n - To instead emit:\n + To:\n \n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n \n - The complaint about a \"hash mismatch\" was simply an emergent property\n - of how we'd fall though from read_loose_object() into fsck_loose()\n - when we didn't get the data we expected. Now we'll correctly note that\n - the object type is invalid.\n + We'll still exit with non-zero, but now we'll finish the rest of the\n + traversal. The tests that's being added here asserts that we'll still\n + complain about other fsck issues (e.g. an unrelated dangling blob).\n +\n + To do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\n + flag from read_loose_object() through to parse_loose_header(). Since\n + the read_loose_object() function is only used in builtin/fsck.c we can\n + simply change it to accept a \"struct object_info\" (which contains the\n + OBJECT_INFO_ALLOW_UNKNOWN_TYPE in its flags). See\n + f6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\n + for the introduction of read_loose_object().\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *p\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n --\n --\tif (read_loose_object(path, oid, &type, &size, &contents,\n --\t\t\t OBJECT_INFO_ALLOW_UNKNOWN_TYPE) < 0) {\n --\t\terrors_found |= ERROR_OBJECT;\n +\tstruct strbuf sb = STRBUF_INIT;\n -+\tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n -+\tstruct object_info oi;\n -+\tint found = 0;\n ++\tstruct object_info oi = OBJECT_INFO_INIT;\n ++\tint err = 0;\n + \n +-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n +\toi.type_name = &sb;\n +\toi.sizep = &size;\n +\toi.typep = &type;\n +\n -+\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n -+\t\tfound |= ERROR_OBJECT;\n - \t\terror(_(\"%s: object corrupt or missing: %s\"),\n - \t\t oid_to_hex(oid), path);\n -+\t}\n -+\tif (type < 0) {\n -+\t\tfound |= ERROR_OBJECT;\n -+\t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n -+\t\t oid_to_hex(oid), sb.buf, path);\n -+\t}\n -+\tif (found) {\n -+\t\terrors_found |= ERROR_OBJECT;\n ++\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n ++\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n ++\t\t\t oid_to_hex(oid), path);\n ++\tif (type < 0)\n ++\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n ++\t\t\t oid_to_hex(oid), sb.buf, path);\n ++\tif (err) {\n + \t\terrors_found |= ERROR_OBJECT;\n +-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n +-\t\t oid_to_hex(oid), path);\n \t\treturn 0; /* keep checking other objects */\n \t}\n \n @@ object-file.c: static int check_stream_oid(git_zstream *stream,\n \t\t const struct object_id *expected_oid,\n -\t\t enum object_type *type,\n -\t\t unsigned long *size,\n - \t\t void **contents,\n -+\t\t struct object_info *oi,\n - \t\t unsigned int oi_flags)\n +-\t\t void **contents)\n ++\t\t void **contents,\n ++\t\t struct object_info *oi)\n {\n \tint ret = -1;\n -@@ object-file.c: int read_loose_object(const char *path,\n + \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n -\tstruct object_info oi = OBJECT_INFO_INIT;\n - \tint allow_unknown = oi_flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n -\toi.typep = type;\n -\toi.sizep = size;\n -+\tenum object_type *type = oi->typep;\n +\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n @@ object-file.c: int read_loose_object(const char *path,\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n + \t}\n +-\tif (*type < 0)\n +-\t\tdie(_(\"invalid object type\"));\n + \n +-\tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n ++\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n + \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n + \t\t\tgoto out;\n + \t} else {\n @@ object-file.c: int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n @@ object-file.c: int read_loose_object(const char *path,\n \t\t\tfree(*contents);\n \n ## object-store.h ##\n -@@ object-store.h: int oid_object_info_extended(struct repository *r,\n +@@ object-store.h: int force_object_loose(const struct object_id *oid, time_t mtime);\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n @@ object-store.h: int oid_object_info_extended(struct repository *r,\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n -@@ object-store.h: int oid_object_info_extended(struct repository *r,\n +@@ object-store.h: int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n -\t\t enum object_type *type,\n -\t\t unsigned long *size,\n - \t\t void **contents,\n -+\t\t struct object_info *oi,\n - \t\t unsigned int oi_flags);\n +-\t\t void **contents);\n ++\t\t void **contents,\n ++\t\t struct object_info *oi);\n \n - /*\n + /* Retry packed storage after checking packed and loose storage */\n + #define HAS_OBJECT_RECHECK_PACKED 1\n \n ## t/t1450-fsck.sh ##\n -@@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n - \t)\n - '\n +@@ t/t1450-fsck.sh: test_expect_success 'object with hash and type mismatch' '\n + \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n + \t\tgit update-ref refs/heads/bogus $cmt &&\n \n -+test_expect_success 'object with hash and type mismatch' '\n -+\tgit init --bare hash-type-mismatch &&\n -+\t(\n -+\t\tcd hash-type-mismatch &&\n -+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n -+\t\told=$(test_oid_to_path \"$oid\") &&\n -+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n -+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n -+\t\tmv objects/$old objects/$new &&\n -+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n -+\t\ttree=$(git write-tree) &&\n -+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n -+\t\tgit update-ref refs/heads/bogus $cmt &&\n +-\t\tcat >expect <<-\\EOF &&\n +-\t\tfatal: invalid object type\n +-\t\tEOF\n +-\t\ttest_must_fail git fsck 2>actual &&\n +-\t\ttest_cmp expect actual\n ++\n +\t\ttest_must_fail git fsck 2>out &&\n +\t\tgrep \"^error: hash mismatch for \" out &&\n +\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n -+\t)\n -+'\n -+\n - test_expect_success 'branch pointing to non-commit' '\n - \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n - \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n -@@ t/t1450-fsck.sh: test_expect_success 'fsck error and recovery on invalid object type' '\n - \tgarbage_blob=$(git -C garbage-type hash-object --stdin -w -t garbage --literally out 2>err &&\n - \tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n --\ttest_line_count = 2 errors &&\n --\tgrep \"error: hash mismatch for\" err &&\n --\tgrep \"$garbage_blob: object corrupt or missing:\" err &&\n -+\ttest_line_count = 1 errors &&\n -+\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n - \tgrep \"dangling blob $empty_blob\" out\n + \t)\n + '\n + \n +@@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n + \ttest_i18ngrep \"bad index file\" errors\n + '\n + \n +-test_expect_success 'fsck hard errors on an invalid object type' '\n ++test_expect_success 'fsck error and recovery on invalid object type' '\n + \tgit init --bare garbage-type &&\n + \t(\n + \t\tcd garbage-type &&\n +@@ t/t1450-fsck.sh: test_expect_success 'fsck hard errors on an invalid object type' '\n + \t\tfatal: invalid object type\n + \t\tEOF\n + \t\ttest_must_fail git fsck >out 2>err &&\n +-\t\ttest_cmp err.expect err &&\n +-\t\ttest_must_be_empty out\n ++\t\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n ++\t\ttest_line_count = 1 errors &&\n ++\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n ++\t\tgrep \"dangling blob $empty_blob\" out\n + \t)\n '\n \n22: 804673a17b0 ! 17: b07e892fc19 fsck: report invalid object type-path combinations\n @@ Commit message\n value for checking whether we got far enough to be certain that the\n issue was indeed this OID mismatch.\n \n - In the case of check_object_signature() I don't really trust all the\n - moving parts there to behave consistently, in the face of future\n - refactorings. Getting it wrong would mean that we'd potentially emit\n - no error at all on a failing check_object_signature(), or worse\n - misreport whatever issue we encountered. So let's use the new bug()\n - function to ferry and return code up to fsck_loose() in that case.\n + We need to add the \"object corrupt or missing\" special-case to deal\n + with cases where read_loose_object() will return an error before\n + completing check_object_signature(), e.g. if we have an error in\n + unpack_loose_rest() because we find garbage after the valid gzip\n + content:\n +\n + $ git hash-object --stdin -w -t blob >objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391\n + $ git fsck\n + error: garbage at end of loose object 'e69d[...]'\n + error: unable to unpack contents of ./objects/e6/9d[...]\n + error: e69d[...]: object corrupt or missing: ./objects/e6/9d[...]\n +\n + There is currently some weird messaging in the edge case when the two\n + are combined, i.e. because we're not explicitly passing along an error\n + state about this specific scenario from check_stream_oid() via\n + read_loose_object() we'll end up printing the null OID if an object is\n + of an unknown type *and* it can't be unpacked by zlib, e.g.:\n +\n + $ git hash-object --stdin -w -t garbage --literally >objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n + $ /usr/bin/git fsck\n + fatal: invalid object type\n + $ ~/g/git/git fsck\n + error: garbage at end of loose object '8315a83d2acc4c174aed59430f9a9c4ed926440f'\n + error: unable to unpack contents of ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n + error: 8315a83d2acc4c174aed59430f9a9c4ed926440f: object corrupt or missing: ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n + error: 0000000000000000000000000000000000000000: object is of unknown type 'garbage': ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n + [...]\n +\n + I think it's OK to leave that for future improvements, which would\n + involve enum-ifying more error state as we've done with \"enum\n + unpack_loose_header_result\" in preceding commits. In these\n + increasingly more obscure cases the worst that can happen is that\n + we'll get slightly nonsensical or inapplicable error messages.\n +\n + There's other such potential edge cases, all of which might produce\n + some confusing messaging, but still be handled correctly as far as\n + passing along errors goes. E.g. if check_object_signature() returns\n + and oideq(real_oid, null_oid()) is true, which could happen if it\n + returns -1 due to the read_istream() call having failed.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ builtin/fast-export.c: static void export_blob(const struct object_id *oid)\n \n ## builtin/fsck.c ##\n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n + \tstruct object *obj;\n + \tenum object_type type;\n + \tunsigned long size;\n +-\tvoid *contents;\n ++\tunsigned char *contents = NULL;\n + \tint eaten;\n \tstruct strbuf sb = STRBUF_INIT;\n - \tunsigned int oi_flags = OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n - \tstruct object_info oi;\n + \tstruct object_info oi = OBJECT_INFO_INIT;\n +-\tint err = 0;\n +\tstruct object_id real_oid = *null_oid();\n - \tint found = 0;\n ++\tint ret;\n + \n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n --\tif (read_loose_object(path, oid, &contents, &oi, oi_flags) < 0) {\n -+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi, oi_flags) < 0) {\n - \t\tfound |= ERROR_OBJECT;\n --\t\terror(_(\"%s: object corrupt or missing: %s\"),\n --\t\t oid_to_hex(oid), path);\n -+\t\tif (!oideq(&real_oid, oid))\n +-\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n +-\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n +-\t\t\t oid_to_hex(oid), path);\n ++\tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n ++\tif (ret < 0) {\n ++\t\tif (contents && !oideq(&real_oid, oid))\n +\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n +\t\t\t oid_to_hex(&real_oid), path);\n +\t\telse\n +\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n +\t\t\t oid_to_hex(oid), path);\n - \t}\n - \tif (type < 0) {\n - \t\tfound |= ERROR_OBJECT;\n - \t\terror(_(\"%s: object is of unknown type '%s': %s\"),\n --\t\t oid_to_hex(oid), sb.buf, path);\n -+\t\t oid_to_hex(&real_oid), sb.buf, path);\n - \t}\n - \tif (found) {\n ++\t}\n + \tif (type < 0)\n +-\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n +-\t\t\t oid_to_hex(oid), sb.buf, path);\n +-\tif (err) {\n ++\t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n ++\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n ++\tif (ret < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n + \t\treturn 0; /* keep checking other objects */\n + \t}\n \n ## builtin/index-pack.c ##\n @@ builtin/index-pack.c: static void fix_unresolved_deltas(struct hashfile *f)\n @@ builtin/mktag.c: static int verify_object_in_tag(struct object_id *tagged_oid, i\n \n \treturn ret;\n \n + ## cache.h ##\n +@@ cache.h: struct object_info;\n + int parse_loose_header(const char *hdr, struct object_info *oi);\n + \n + int check_object_signature(struct repository *r, const struct object_id *oid,\n +-\t\t\t void *buf, unsigned long size, const char *type);\n ++\t\t\t void *buf, unsigned long size, const char *type,\n ++\t\t\t struct object_id *real_oidp);\n + \n + int finalize_object_file(const char *tmpfile, const char *filename);\n + \n +\n ## object-file.c ##\n @@ object-file.c: void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n @@ object-file.c: static int check_stream_oid(git_zstream *stream,\n \t\t const struct object_id *expected_oid,\n +\t\t struct object_id *real_oid,\n \t\t void **contents,\n - \t\t struct object_info *oi,\n - \t\t unsigned int oi_flags)\n + \t\t struct object_info *oi)\n + {\n +@@ object-file.c: int read_loose_object(const char *path,\n + \tchar hdr[MAX_HEADER_LEN];\n + \tunsigned long *size = oi->sizep;\n + \n +-\t*contents = NULL;\n +-\n + \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n + \tif (!map) {\n + \t\terror_errno(_(\"unable to mmap %s\"), path);\n @@ object-file.c: int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n @@ object-file.c: int read_loose_object(const char *path,\n -\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n -\t\t\t oid_to_hex(expected_oid));\n +\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n -+\t\t\tif (oideq(real_oid, null_oid()))\n -+\t\t\t\tBUG(\"should only get OID mismatch errors with mapped contents\");\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\n \n ## object-store.h ##\n -@@ object-store.h: int oid_object_info_extended(struct repository *r,\n +@@ object-store.h: int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n +\t\t struct object_id *real_oid,\n \t\t void **contents,\n - \t\t struct object_info *oi,\n - \t\t unsigned int oi_flags);\n -@@ object-store.h: enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n - int parse_loose_header(const char *hdr, struct object_info *oi);\n - \n - int check_object_signature(struct repository *r, const struct object_id *oid,\n --\t\t\t void *buf, unsigned long size, const char *type);\n -+\t\t\t void *buf, unsigned long size, const char *type,\n -+\t\t\t struct object_id *real_oidp);\n - int finalize_object_file(const char *tmpfile, const char *filename);\n - int check_and_freshen_file(const char *fn, int freshen);\n + \t\t struct object_info *oi);\n \n \n ## object.c ##\n @@ t/t1006-cat-file.sh: test_expect_success 'cat-file -t and -s on corrupt loose ob\n \n ## t/t1450-fsck.sh ##\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n - \t(\n \t\tcd hash-mismatch &&\n + \n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n +\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash mismatch' '\n - \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n + \n \t\ttest_must_fail git fsck 2>out &&\n --\t\ttest_i18ngrep \"$oid.*corrupt\" out\n +-\t\tgrep \"$oid.*corrupt\" out\n +\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n @@ t/t1450-fsck.sh: test_expect_success 'object with hash and type mismatch' '\n - \t(\n \t\tcd hash-type-mismatch &&\n + \n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n +\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n @@ t/t1450-fsck.sh: test_expect_success 'object with hash and type mismatch' '\n - \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n - \t\tgit update-ref refs/heads/bogus $cmt &&\n + \n + \n \t\ttest_must_fail git fsck 2>out &&\n -\t\tgrep \"^error: hash mismatch for \" out &&\n -\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436535","messageId":"patch-v7-07.17-20bd81c1af0-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 07/17] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:11Z","receivedAt":"2021-09-21T02:58:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex af59613250b..8bbc34efb0c 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -426,6 +426,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436537","messageId":"patch-v7-06.17-051088aa114-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 06/17] cat-file tests: test for missing/bogus object with -t, -s and -p","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:10Z","receivedAt":"2021-09-21T02:58:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When we look up a missing object with cat_one_file() what error we\nprint out currently depends on whether we'll error out early in\nget_oid_with_context(), or if we'll get an error later from\noid_object_info_extended().\n\nThe --allow-unknown-type flag then changes whether we pass the\n\"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\" flag to get_oid_with_context() or\nnot.\n\nThe \"-p\" flag is yet another special-case in printing the same output\non the deadbeef OID as we'd emit on the deadbeef_short OID for the\n\"-s\" and \"-t\" options, it also doesn't support the\n\"--allow-unknown-type\" flag at all.\n\nLet's test the combination of the two sets of [-t, -s, -p] and\n[--{no-}allow-unknown-type] (the --no-allow-unknown-type is implicit\nin not supplying it), as well as a [missing,bogus] object pair.\n\nThis extends tests added in 3e370f9faf0 (t1006: add tests for git\ncat-file --allow-unknown-type, 2015-05-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/oid-info/oid | 2 ++\n t/t1006-cat-file.sh | 75 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 77 insertions(+)\n\ndiff --git a/t/oid-info/oid b/t/oid-info/oid\nindex a754970523c..ecffa9045f9 100644\n--- a/t/oid-info/oid\n+++ b/t/oid-info/oid\n@@ -27,3 +27,5 @@ numeric\t\tsha1:0123456789012345678901234567890123456789\n numeric\t\tsha256:0123456789012345678901234567890123456789012345678901234567890123\n deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n+deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n+deadbee_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex ea6a53d425b..af59613250b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -327,6 +327,81 @@ test_expect_success 'setup bogus data' '\n \tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n '\n \n+for arg1 in '' --allow-unknown-type\n+do\n+\tfor arg2 in -s -t -p\n+\tdo\n+\t\tif test $arg1 = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus short OID\" '\n+\t\t\tcat >expect <<-\\EOF &&\n+\t\t\tfatal: invalid object type\n+\t\t\tEOF\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_short_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_long_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing short OID\" '\n+\t\t\tcat >expect.err <<-EOF &&\n+\t\t\tfatal: Not a valid object name $(test_oid deadbeef_short)\n+\t\t\tEOF\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef_short) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect.err <<-EOF\n+\t\t\t\tfatal: Not a valid object name $(test_oid deadbeef)\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect.err <<-\\EOF\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out &&\n+\t\t\ttest_cmp expect.err err.actual\n+\t\t'\n+\tdone\n+done\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436538","messageId":"patch-v7-05.17-fc93c2c2530-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 05/17] cat-file tests: move bogus_* variable declarations earlier","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:09Z","receivedAt":"2021-09-21T02:58:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the short/long bogus bogus object type variables into a form\nwhere the two sets can be used concurrently. This'll be used by\nsubsequently added tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 35 +++++++++++++++++++----------------\n 1 file changed, 19 insertions(+), 16 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..ea6a53d425b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,36 +315,39 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n-bogus_type=\"bogus\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n+test_expect_success 'setup bogus data' '\n+\tbogus_short_type=\"bogus\" &&\n+\tbogus_short_content=\"bogus\" &&\n+\tbogus_short_size=$(strlen \"$bogus_short_content\") &&\n+\tbogus_short_sha1=$(echo_without_newline \"$bogus_short_content\" | git hash-object -t $bogus_short_type --literally -w --stdin) &&\n+\n+\tbogus_long_type=\"abcdefghijklmnopqrstuvwxyz1234679\" &&\n+\tbogus_long_content=\"bogus\" &&\n+\tbogus_long_size=$(strlen \"$bogus_long_content\") &&\n+\tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n+'\n \n test_expect_success \"Type of broken object is correct\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of broken object is correct\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n-bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n test_expect_success \"Type of broken object is correct when type is large\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of large broken object is correct when type is large\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436539","messageId":"patch-v7-04.17-854991c1543-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 04/17] fsck tests: test for garbage appended to a loose object","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:08Z","receivedAt":"2021-09-21T02:58:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There wasn't any output tests for this scenario, let's ensure that we\ndon't regress on it in the changes that come after this.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 175ed304637..bd696d21dba 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -93,6 +93,26 @@ test_expect_success 'object with hash and type mismatch' '\n \t)\n '\n \n+test_expect_success POSIXPERM 'zlib corrupt loose object output ' '\n+\tgit init --bare corrupt-loose-output &&\n+\t(\n+\t\tcd corrupt-loose-output &&\n+\t\toid=$(git hash-object -w --stdin --literally >$oidf &&\n+\n+\t\tcat >expect.error <<-EOF &&\n+\t\terror: garbage at end of loose object '\\''$oid'\\''\n+\t\terror: unable to unpack contents of ./$oidf\n+\t\terror: $oid: object corrupt or missing: ./$oidf\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\tgrep ^error: actual >error &&\n+\t\ttest_cmp expect.error error\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436540","messageId":"patch-v7-09.17-d9f5adfc74b-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 09/17] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:13Z","receivedAt":"2021-09-21T02:58:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex a8be8994814..bda3497d5ca 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1503,8 +1503,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436541","messageId":"patch-v7-08.17-cd1d52b8a07-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 08/17] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:12Z","receivedAt":"2021-09-21T02:58:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 8bbc34efb0c..269ab7e4729 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -402,6 +402,67 @@ do\n \tdone\n done\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_short_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_short_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_short_type $bogus_short_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_short_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect <<-EOF &&\n+\t$bogus_short_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\ttest_when_finished \"rm -rf .git/refs/replace\" &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_short_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436542","messageId":"patch-v7-10.17-51d14bc9274-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 10/17] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:14Z","receivedAt":"2021-09-21T02:58:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex bda3497d5ca..774ec8c866f 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1262,7 +1262,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t buffer, bufsiz);\n \n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/* Make sure we have the terminating NUL */\n \tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436543","messageId":"patch-v7-11.17-f43cfd8a5ed-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 11/17] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:15Z","receivedAt":"2021-09-21T02:58:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nIt would be a better end-state to have already moved the declaration\nof these functions to object-store.h to avoid the forward declaration\nof \"struct object_info\" in cache.h, but let's leave that cleanup for\nsome other time.\n\n1. https://lore.kernel.org/git/patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 4 +++-\n object-file.c | 20 +++++++-------------\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d23de693680..33cacbd22ac 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1314,7 +1314,9 @@ char *xdg_cache_home(const char *filename);\n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+struct object_info;\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 774ec8c866f..33a01ac203f 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1363,8 +1363,8 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1424,14 +1424,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1486,10 +1478,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2573,6 +2565,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2587,7 +2581,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436544","messageId":"patch-v7-14.17-522d71eb19d-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 14/17] object-file.c: return ULHR_TOO_LONG on \"header too long\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:18Z","receivedAt":"2021-09-21T02:58:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return ULHR_TOO_LONG saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 5 ++++-\n object-file.c | 8 ++++++--\n streaming.c | 1 +\n t/t1006-cat-file.sh | 4 ++--\n 4 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 90dde86828e..49b18f2755c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1322,16 +1322,19 @@ int git_open_cloexec(const char *name, int flags);\n *\n * - ULHR_OK on success\n * - ULHR_BAD on error\n+ * - ULHR_TOO_LONG if the header was too long\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), ULHR_TOO_LONG will still be returned\n+ * from this function to indicate that the header was too long.\n */\n enum unpack_loose_header_result {\n \tULHR_OK,\n \tULHR_BAD,\n+\tULHR_TOO_LONG,\n };\n enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned char *map,\ndiff --git a/object-file.c b/object-file.c\nindex b214a152ca8..ca4abe172ce 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1268,7 +1268,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn ULHR_BAD;\n+\t\treturn ULHR_TOO_LONG;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1289,7 +1289,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn ULHR_BAD;\n+\treturn ULHR_TOO_LONG;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1462,6 +1462,10 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t\tbreak;\n+\tcase ULHR_TOO_LONG:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n \t}\n \n \tif (status < 0) {\ndiff --git a/streaming.c b/streaming.c\nindex 6df0247a4cb..bd89c50e7b3 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -235,6 +235,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_OK:\n \t\tbreak;\n \tcase ULHR_BAD:\n+\tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n \tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 269ab7e4729..711dcc6d795 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -356,12 +356,12 @@ do\n \t\t\tif test \"$arg2\" = \"-p\"\n \t\t\tthen\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n \t\t\t\tEOF\n \t\t\telse\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: git cat-file: could not get object info\n \t\t\t\tEOF\n \t\t\tfi &&\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436545","messageId":"patch-v7-12.17-50d938f7f3c-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 12/17] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:16Z","receivedAt":"2021-09-21T02:58:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 17 ++++++++++++++-\n object-file.c | 58 ++++++++++++++++++---------------------------------\n streaming.c | 3 ++-\n 3 files changed, 38 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 33cacbd22ac..9ad81d452ad 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1313,7 +1313,22 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n+\n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 33a01ac203f..8dd35f768bb 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1233,11 +1233,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1248,35 +1249,8 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n \tif (status < Z_OK)\n \t\treturn -1;\n \n@@ -1286,6 +1260,14 @@ static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1435,6 +1417,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1468,11 +1451,9 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \tif (status < 0)\n@@ -2576,7 +2557,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436546","messageId":"patch-v7-15.17-1ca875395c1-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 15/17] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:19Z","receivedAt":"2021-09-21T02:58:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make parse_loose_header() return error codes and data instead of\ninvoking die() by itself.\n\nFor now we'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller. In a subsequent\ncommit we'll make read_loose_object() return an error code instead of\ndying. We should also address the \"allow_unknown\" case (should be\nmoved to builtin/cat-file.c), but for now I'll be leaving it.\n\nFor making parse_loose_header() not die() change its prototype to\naccept a \"struct object_info *\" instead of the \"unsigned long *sizep\"\nit accepted before. Its callers can now check the populated populated\n\"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nSince 93cff9a978e (sha1_loose_object_info: return error for corrupted\nobjects, 2017-04-01) the return value of loose_object_info() (then\nnamed sha1_loose_object_info()) had been a \"status\" variable that be\nany negative value, as we were expecting to return the \"enum\nobject_type\".\n\nThe only negative type happens to be OBJ_BAD, but the code still\nassumed that more might be added. This was then used later in\ne.g. c84a1f3ed4d (sha1_file: refactor read_object, 2017-06-21). Now\nthat parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSince parse_loose_header() doesn't need to return an arbitrary\n\"status\" we only need to treat its \"ret < 0\" specially, but can\nidiomatically overwrite it with our own error() return. This along\nwith having made unpack_loose_header() return an \"enum\nunpack_loose_header_result\" in an earlier commit means that we can\nmove the previously nested if/else cases mostly into the \"ULHR_OK\"\nbranch of the \"switch\" statement.\n\nWe should be less silent if we reach that \"status = -1\" branch, which\nhappens if we've got trailing garbage in loose objects, see\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor a better way to handle it. For now let's punt on it, a subsequent\ncommit will address that edge case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 11 +++++++--\n object-file.c | 67 +++++++++++++++++++++++++--------------------------\n streaming.c | 3 ++-\n 3 files changed, 44 insertions(+), 37 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 49b18f2755c..23f0534b70e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1343,9 +1343,16 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned long bufsiz,\n \t\t\t\t\t\t struct strbuf *hdrbuf);\n \n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n struct object_info;\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex ca4abe172ce..1af914c19c6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1347,8 +1347,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1370,15 +1369,6 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1405,7 +1395,14 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1419,6 +1416,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1450,6 +1448,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1457,6 +1457,18 @@ static int loose_object_info(struct repository *r,\n \tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n \tcase ULHR_OK:\n+\t\tif (parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0)\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t\telse if (!allow_unknown && *oi->typep < 0)\n+\t\t\tdie(_(\"invalid object type\"));\n+\n+\t\tif (!oi->contentp)\n+\t\t\tbreak;\n+\t\t*oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);\n+\t\tif (*oi->contentp)\n+\t\t\tgoto cleanup;\n+\n+\t\tstatus = -1;\n \t\tbreak;\n \tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n@@ -1468,31 +1480,16 @@ static int loose_object_info(struct repository *r,\n \t\tbreak;\n \t}\n \n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n-\n-\tif (status >= 0 && oi->contentp) {\n-\t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n-\t\tif (!*oi->contentp) {\n-\t\t\tgit_inflate_end(&stream);\n-\t\t\tstatus = -1;\n-\t\t}\n-\t} else\n-\t\tgit_inflate_end(&stream);\n-\n+\tgit_inflate_end(&stream);\n+cleanup:\n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n-\treturn (status < 0) ? status : 0;\n+\treturn status;\n }\n \n int obj_read_use_lock = 0;\n@@ -2559,6 +2556,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2575,12 +2573,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/streaming.c b/streaming.c\nindex bd89c50e7b3..fe54665d86e 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -238,7 +239,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n-\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n \t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436547","messageId":"patch-v7-13.17-755fde00b46-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 13/17] object-file.c: use \"enum\" return type for unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:17Z","receivedAt":"2021-09-21T02:58:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In a preceding commit we changed and documented unpack_loose_header()\nfrom its previous behavior of returning any negative value or zero, to\nonly -1 or 0.\n\nLet's add an \"enum unpack_loose_header_result\" type and use it for\nthese return values, and have the compiler assert that we're\nexhaustively covering all of them.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 19 +++++++++++++++----\n object-file.c | 34 +++++++++++++++++++++-------------\n streaming.c | 23 +++++++++++++----------\n 3 files changed, 49 insertions(+), 27 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9ad81d452ad..90dde86828e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1318,7 +1318,10 @@ int git_open_cloexec(const char *name, int flags);\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns:\n+ *\n+ * - ULHR_OK on success\n+ * - ULHR_BAD on error\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n@@ -1326,9 +1329,17 @@ int git_open_cloexec(const char *name, int flags);\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n * with parse_loose_header().\n */\n-int unpack_loose_header(git_zstream *stream, unsigned char *map,\n-\t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n+enum unpack_loose_header_result {\n+\tULHR_OK,\n+\tULHR_BAD,\n+};\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *hdrbuf);\n+\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 8dd35f768bb..b214a152ca8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1233,10 +1233,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz,\n-\t\t\tstruct strbuf *header)\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *header)\n {\n \tint status;\n \n@@ -1252,13 +1254,13 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn 0;\n+\t\treturn ULHR_OK;\n \n \t/*\n \t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n@@ -1266,7 +1268,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1287,7 +1289,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn ULHR_BAD;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1452,13 +1454,19 @@ static int loose_object_info(struct repository *r,\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n-\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n+\tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t\tbreak;\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..6df0247a4cb 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -229,17 +229,16 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n-\tif ((unpack_loose_header(&st->z,\n-\t\t\t\t st->u.loose.mapped,\n-\t\t\t\t st->u.loose.mapsize,\n-\t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr),\n-\t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n-\t\tgit_inflate_end(&st->z);\n-\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-\t\treturn -1;\n+\tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n+\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n+\t\t\t\t sizeof(st->u.loose.hdr), NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n+\t\tgoto error;\n \t}\n+\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n \tst->u.loose.hdr_avail = st->z.total_out;\n@@ -248,6 +247,10 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->read = read_istream_loose;\n \n \treturn 0;\n+error:\n+\tgit_inflate_end(&st->z);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn -1;\n }\n \n \n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436548","messageId":"patch-v7-16.17-d38067feab3-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 16/17] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:20Z","receivedAt":"2021-09-21T02:58:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it to accept a \"struct object_info\" (which contains the\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE in its flags). See\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor the introduction of read_loose_object().\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 17 ++++++++++++++---\n object-file.c | 18 ++++++------------\n object-store.h | 6 +++---\n t/t1450-fsck.sh | 17 +++++++++--------\n 4 files changed, 32 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..3b046820750 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,11 +600,22 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint err = 0;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n+\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\tif (type < 0)\n+\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(oid), sb.buf, path);\n+\tif (err) {\n \t\terrors_found |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex 1af914c19c6..be568ade95b 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2546,18 +2546,15 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t struct object_info *oi)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2573,15 +2570,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \n-\tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n+\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \t\t\tgoto out;\n \t} else {\n@@ -2592,8 +2587,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex b4dc6668aa2..e8b4d87b898 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -244,6 +244,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -251,9 +252,8 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t struct object_info *oi);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex bd696d21dba..167c319823a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -85,11 +85,10 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n-\t\tcat >expect <<-\\EOF &&\n-\t\tfatal: invalid object type\n-\t\tEOF\n-\t\ttest_must_fail git fsck 2>actual &&\n-\t\ttest_cmp expect actual\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n@@ -910,7 +909,7 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \tgit init --bare garbage-type &&\n \t(\n \t\tcd garbage-type &&\n@@ -922,8 +921,10 @@ test_expect_success 'fsck hard errors on an invalid object type' '\n \t\tfatal: invalid object type\n \t\tEOF\n \t\ttest_must_fail git fsck >out 2>err &&\n-\t\ttest_cmp err.expect err &&\n-\t\ttest_must_be_empty out\n+\t\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\t\ttest_line_count = 1 errors &&\n+\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n+\t\tgrep \"dangling blob $empty_blob\" out\n \t)\n '\n \n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436549","messageId":"patch-v7-17.17-b07e892fc19-20210920T190305Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v7 17/17] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T19:04:21Z","receivedAt":"2021-09-21T02:59:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob >objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391\n $ git fsck\n error: garbage at end of loose object 'e69d[...]'\n error: unable to unpack contents of ./objects/e6/9d[...]\n error: e69d[...]: object corrupt or missing: ./objects/e6/9d[...]\n\nThere is currently some weird messaging in the edge case when the two\nare combined, i.e. because we're not explicitly passing along an error\nstate about this specific scenario from check_stream_oid() via\nread_loose_object() we'll end up printing the null OID if an object is\nof an unknown type *and* it can't be unpacked by zlib, e.g.:\n\n $ git hash-object --stdin -w -t garbage --literally >objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n $ /usr/bin/git fsck\n fatal: invalid object type\n $ ~/g/git/git fsck\n error: garbage at end of loose object '8315a83d2acc4c174aed59430f9a9c4ed926440f'\n error: unable to unpack contents of ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 8315a83d2acc4c174aed59430f9a9c4ed926440f: object corrupt or missing: ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 0000000000000000000000000000000000000000: object is of unknown type 'garbage': ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n [...]\n\nI think it's OK to leave that for future improvements, which would\ninvolve enum-ifying more error state as we've done with \"enum\nunpack_loose_header_result\" in preceding commits. In these\nincreasingly more obscure cases the worst that can happen is that\nwe'll get slightly nonsensical or inapplicable error messages.\n\nThere's other such potential edge cases, all of which might produce\nsome confusing messaging, but still be handled correctly as far as\npassing along errors goes. E.g. if check_object_signature() returns\nand oideq(real_oid, null_oid()) is true, which could happen if it\nreturns -1 due to the read_istream() call having failed.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 23 +++++++++++++++--------\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n cache.h | 3 ++-\n object-file.c | 21 ++++++++++-----------\n object-store.h | 1 +\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 11 files changed, 42 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 95e8e89e81f..8e2caf72819 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 3b046820750..d925cdbae5c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -598,23 +598,30 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct object *obj;\n \tenum object_type type;\n \tunsigned long size;\n-\tvoid *contents;\n+\tunsigned char *contents = NULL;\n \tint eaten;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n-\tint err = 0;\n+\tstruct object_id real_oid = *null_oid();\n+\tint ret;\n \n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n-\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n-\t\t\t oid_to_hex(oid), path);\n+\tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n+\tif (ret < 0) {\n+\t\tif (contents && !oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\t}\n \tif (type < 0)\n-\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t\t oid_to_hex(oid), sb.buf, path);\n-\tif (err) {\n+\t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n+\tif (ret < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 6cc48902170..17c4b1d3ead 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1415,7 +1415,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/cache.h b/cache.h\nindex 23f0534b70e..44b11f52362 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1355,7 +1355,8 @@ struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n \n int finalize_object_file(const char *tmpfile, const char *filename);\n \ndiff --git a/object-file.c b/object-file.c\nindex be568ade95b..ff0e465d556 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1062,9 +1062,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1072,8 +1074,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1098,9 +1100,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2546,6 +2548,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi)\n {\n@@ -2556,8 +2559,6 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n-\t*contents = NULL;\n-\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n@@ -2587,9 +2588,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex e8b4d87b898..77aa3d857cc 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -252,6 +252,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \ndiff --git a/object.c b/object.c\nindex 4e85955a941..23a24e678a8 100644\n--- a/object.c\n+++ b/object.c\n@@ -279,7 +279,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -290,7 +290,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex c8e560d71ab..3f418e3a6af 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 711dcc6d795..1f7cc0717b7 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -512,7 +512,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 167c319823a..eb0e772f098 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -54,6 +54,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcd hash-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -65,7 +66,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -75,6 +76,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcd hash-type-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -87,8 +89,8 @@ test_expect_success 'object with hash and type mismatch' '\n \n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.33.0.1098.g29a6526ae47\n\n"},{"id":"436555","messageId":"YUlRv7LMDyB2Nw4L@nand.local","threadId":"55404","inReplyTo":"patch-v7-06.17-051088aa114-20210920T190305Z-avarab@gmail.com","subject":"Re: [PATCH v7 06/17] cat-file tests: test for missing/bogus object with -t, -s and -p","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-21T03:30:07Z","receivedAt":"2021-09-21T03:31:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 20, 2021 at 09:04:10PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> diff --git a/t/oid-info/oid b/t/oid-info/oid\n> index a754970523c..ecffa9045f9 100644\n> --- a/t/oid-info/oid\n> +++ b/t/oid-info/oid\n> @@ -27,3 +27,5 @@ numeric\t\tsha1:0123456789012345678901234567890123456789\n> numeric\t\tsha256:0123456789012345678901234567890123456789012345678901234567890123\n> deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n> deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n> +deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n> +deadbee_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\n\nThis jumped out at me while I was reading it. In the second line,\ns/deadbee_short/deadbeef_short/ ?\n\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index ea6a53d425b..af59613250b 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -327,6 +327,81 @@ test_expect_success 'setup bogus data' '\n> \tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n> '\n>\n> +for arg1 in '' --allow-unknown-type\n> +do\n> +\tfor arg2 in -s -t -p\n> +\tdo\n\nThis is quite the loop! I appreciate the extra thoroughness, although it\nmay come at some extra cost of intertwining all of these combinations of\ntests together.\n\nBut that may be warranted, since they are all related. But it's not a\nfull matrix of all possible combinations; e.g., \"--allow-unknown-type\"\ndoes not go with \"-p\".\n\nSo this may be the best that we can do. It's definitely a mouthful, but\nI think it's overall an easier read than what we had in the previous\nversion. And it's definitely more thorough, which is good. Thanks for\nspending the time improving this test.\n\nThanks,\nTaylor\n"},{"id":"437224","messageId":"patch-v8-01.17-b999ab695d9-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 01/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:42Z","receivedAt":"2021-09-28T02:19:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..969bfbbdd8f 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -865,4 +865,21 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\tgit init --bare garbage-type &&\n+\t(\n+\t\tcd garbage-type &&\n+\n+\t\tempty=$(git hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck >out 2>err &&\n+\t\ttest_cmp err.expect err &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437225","messageId":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com","subject":"[PATCH v8 00/17] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:41Z","receivedAt":"2021-09-28T02:19:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting, see the examples in the commit\nmessages of 16/17 and 17/17. To get there I've lib-ified more things\nin object-file.c and the general object APIs, i.e. now we'll return\nerror codes instead of calling die() in these cases.\n\nv6 of this got a very detailed review from Taylor Blau (thanks a\nlot!), for the v6 see:\nhttps://lore.kernel.org/git/cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com/\n\nThe v7 had a couple of trivial shellscripting issues, a typo'd\ntest_oid variable, and a warning on a \"test\" comparison. For v7 see\nhttps://lore.kernel.org/git/cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (17):\n fsck tests: add test for fsck-ing an unknown type\n fsck tests: refactor one test to use a sub-repo\n fsck tests: test current hash/type mismatch behavior\n fsck tests: test for garbage appended to a loose object\n cat-file tests: move bogus_* variable declarations earlier\n cat-file tests: test for missing/bogus object with -t, -s and -p\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n object-file.c: don't set \"typep\" when returning non-zero\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: make parse_loose_header_extended() public\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: use \"enum\" return type for unpack_loose_header()\n object-file.c: return ULHR_TOO_LONG on \"header too long\"\n object-file.c: stop dying in parse_loose_header()\n fsck: don't hard die on invalid object types\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 28 +++++-\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 45 ++++++++-\n object-file.c | 176 +++++++++++++++------------------\n object-store.h | 7 +-\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 27 +++--\n t/oid-info/oid | 2 +\n t/t1006-cat-file.sh | 223 +++++++++++++++++++++++++++++++++++++++---\n t/t1450-fsck.sh | 99 +++++++++++++++----\n 13 files changed, 463 insertions(+), 158 deletions(-)\n\nRange-diff against v7:\n 1: 752cef556c2 = 1: b999ab695d9 fsck tests: add test for fsck-ing an unknown type\n 2: 612003bdd2c = 2: e01c21378a4 fsck tests: refactor one test to use a sub-repo\n 3: 1e40a4235e9 = 3: 93197a7bcee fsck tests: test current hash/type mismatch behavior\n 4: 854991c1543 = 4: 277188dd58d fsck tests: test for garbage appended to a loose object\n 5: fc93c2c2530 = 5: ab2ea1beaaf cat-file tests: move bogus_* variable declarations earlier\n 6: 051088aa114 ! 6: 91229b94fac cat-file tests: test for missing/bogus object with -t, -s and -p\n @@ t/oid-info/oid: numeric\t\tsha1:0123456789012345678901234567890123456789\n deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n +deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n -+deadbee_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\n ++deadbeef_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\n \n ## t/t1006-cat-file.sh ##\n @@ t/t1006-cat-file.sh: test_expect_success 'setup bogus data' '\n @@ t/t1006-cat-file.sh: test_expect_success 'setup bogus data' '\n +do\n +\tfor arg2 in -s -t -p\n +\tdo\n -+\t\tif test $arg1 = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n ++\t\tif test \"$arg1\" = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n +\t\tthen\n +\t\t\tcontinue\n +\t\tfi\n 7: 20bd81c1af0 = 7: 9e95e134d30 cat-file tests: add corrupt loose object test\n 8: cd1d52b8a07 = 8: 215f98ad369 cat-file tests: test for current --allow-unknown-type behavior\n 9: d9f5adfc74b = 9: 3e1df3594df object-file.c: don't set \"typep\" when returning non-zero\n10: 51d14bc9274 = 10: b96828f3d5b object-file.c: return -1, not \"status\" from unpack_loose_header()\n11: f43cfd8a5ed = 11: 273acb45517 object-file.c: make parse_loose_header_extended() public\n12: 50d938f7f3c = 12: 314d34357dd object-file.c: simplify unpack_loose_short_header()\n13: 755fde00b46 = 13: 07481bcb55c object-file.c: use \"enum\" return type for unpack_loose_header()\n14: 522d71eb19d = 14: 42b8d135c8c object-file.c: return ULHR_TOO_LONG on \"header too long\"\n15: 1ca875395c1 = 15: 106b7461ce9 object-file.c: stop dying in parse_loose_header()\n16: d38067feab3 = 16: d01223ae322 fsck: don't hard die on invalid object types\n17: b07e892fc19 = 17: 7f394a991a6 fsck: report invalid object type-path combinations\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437226","messageId":"patch-v8-02.17-e01c21378a4-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 02/17] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:43Z","receivedAt":"2021-09-28T02:19:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe can instead use the pattern of creating a named sub-repository,\nthen we don't have to worry about cleaning up after ourselves, nobody\nwill care what state the broken \"hash-mismatch\" repository is after\nthis test runs.\n\nSee [1] for related discussion on various \"modern\" test patterns that\ncan be used to avoid verbosity and increase reliability.\n\n1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 35 ++++++++++++++++++-----------------\n 1 file changed, 18 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 969bfbbdd8f..f8edd15abf8 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,25 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n+test_expect_success 'object with hash mismatch' '\n+\tgit init --bare hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n \n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437227","messageId":"patch-v8-03.17-93197a7bcee-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 03/17] fsck tests: test current hash/type mismatch behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:44Z","receivedAt":"2021-09-28T02:19:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If fsck we move an object around between .git/objects/?? directories\nto simulate a hash mismatch \"git fsck\" will currently hard die() in\nobject-file.c. This behavior will be fixed in subsequent commits, but\nlet's test for it as-is for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f8edd15abf8..175ed304637 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -69,6 +69,30 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\tgit init --bare hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437228","messageId":"patch-v8-04.17-277188dd58d-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 04/17] fsck tests: test for garbage appended to a loose object","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:45Z","receivedAt":"2021-09-28T02:19:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There wasn't any output tests for this scenario, let's ensure that we\ndon't regress on it in the changes that come after this.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 175ed304637..bd696d21dba 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -93,6 +93,26 @@ test_expect_success 'object with hash and type mismatch' '\n \t)\n '\n \n+test_expect_success POSIXPERM 'zlib corrupt loose object output ' '\n+\tgit init --bare corrupt-loose-output &&\n+\t(\n+\t\tcd corrupt-loose-output &&\n+\t\toid=$(git hash-object -w --stdin --literally >$oidf &&\n+\n+\t\tcat >expect.error <<-EOF &&\n+\t\terror: garbage at end of loose object '\\''$oid'\\''\n+\t\terror: unable to unpack contents of ./$oidf\n+\t\terror: $oid: object corrupt or missing: ./$oidf\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\tgrep ^error: actual >error &&\n+\t\ttest_cmp expect.error error\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437229","messageId":"patch-v8-05.17-ab2ea1beaaf-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 05/17] cat-file tests: move bogus_* variable declarations earlier","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:46Z","receivedAt":"2021-09-28T02:19:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the short/long bogus bogus object type variables into a form\nwhere the two sets can be used concurrently. This'll be used by\nsubsequently added tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 35 +++++++++++++++++++----------------\n 1 file changed, 19 insertions(+), 16 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..ea6a53d425b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,36 +315,39 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n-bogus_type=\"bogus\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n+test_expect_success 'setup bogus data' '\n+\tbogus_short_type=\"bogus\" &&\n+\tbogus_short_content=\"bogus\" &&\n+\tbogus_short_size=$(strlen \"$bogus_short_content\") &&\n+\tbogus_short_sha1=$(echo_without_newline \"$bogus_short_content\" | git hash-object -t $bogus_short_type --literally -w --stdin) &&\n+\n+\tbogus_long_type=\"abcdefghijklmnopqrstuvwxyz1234679\" &&\n+\tbogus_long_content=\"bogus\" &&\n+\tbogus_long_size=$(strlen \"$bogus_long_content\") &&\n+\tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n+'\n \n test_expect_success \"Type of broken object is correct\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of broken object is correct\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n-bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n test_expect_success \"Type of broken object is correct when type is large\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of large broken object is correct when type is large\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437230","messageId":"patch-v8-06.17-91229b94fac-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 06/17] cat-file tests: test for missing/bogus object with -t, -s and -p","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:47Z","receivedAt":"2021-09-28T02:19:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When we look up a missing object with cat_one_file() what error we\nprint out currently depends on whether we'll error out early in\nget_oid_with_context(), or if we'll get an error later from\noid_object_info_extended().\n\nThe --allow-unknown-type flag then changes whether we pass the\n\"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\" flag to get_oid_with_context() or\nnot.\n\nThe \"-p\" flag is yet another special-case in printing the same output\non the deadbeef OID as we'd emit on the deadbeef_short OID for the\n\"-s\" and \"-t\" options, it also doesn't support the\n\"--allow-unknown-type\" flag at all.\n\nLet's test the combination of the two sets of [-t, -s, -p] and\n[--{no-}allow-unknown-type] (the --no-allow-unknown-type is implicit\nin not supplying it), as well as a [missing,bogus] object pair.\n\nThis extends tests added in 3e370f9faf0 (t1006: add tests for git\ncat-file --allow-unknown-type, 2015-05-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/oid-info/oid | 2 ++\n t/t1006-cat-file.sh | 75 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 77 insertions(+)\n\ndiff --git a/t/oid-info/oid b/t/oid-info/oid\nindex a754970523c..7547d2c7903 100644\n--- a/t/oid-info/oid\n+++ b/t/oid-info/oid\n@@ -27,3 +27,5 @@ numeric\t\tsha1:0123456789012345678901234567890123456789\n numeric\t\tsha256:0123456789012345678901234567890123456789012345678901234567890123\n deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n+deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n+deadbeef_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex ea6a53d425b..abf57339a29 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -327,6 +327,81 @@ test_expect_success 'setup bogus data' '\n \tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n '\n \n+for arg1 in '' --allow-unknown-type\n+do\n+\tfor arg2 in -s -t -p\n+\tdo\n+\t\tif test \"$arg1\" = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus short OID\" '\n+\t\t\tcat >expect <<-\\EOF &&\n+\t\t\tfatal: invalid object type\n+\t\t\tEOF\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_short_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_long_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing short OID\" '\n+\t\t\tcat >expect.err <<-EOF &&\n+\t\t\tfatal: Not a valid object name $(test_oid deadbeef_short)\n+\t\t\tEOF\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef_short) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect.err <<-EOF\n+\t\t\t\tfatal: Not a valid object name $(test_oid deadbeef)\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect.err <<-\\EOF\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out &&\n+\t\t\ttest_cmp expect.err err.actual\n+\t\t'\n+\tdone\n+done\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437231","messageId":"patch-v8-07.17-9e95e134d30-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 07/17] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:48Z","receivedAt":"2021-09-28T02:19:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex abf57339a29..15774979ad3 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -426,6 +426,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437232","messageId":"patch-v8-08.17-215f98ad369-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 08/17] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:49Z","receivedAt":"2021-09-28T02:19:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 15774979ad3..5b16c69c286 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -402,6 +402,67 @@ do\n \tdone\n done\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_short_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_short_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_short_type $bogus_short_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_short_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect <<-EOF &&\n+\t$bogus_short_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\ttest_when_finished \"rm -rf .git/refs/replace\" &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_short_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437233","messageId":"patch-v8-09.17-3e1df3594df-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 09/17] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:50Z","receivedAt":"2021-09-28T02:19:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex be4f94ecf3b..766ba88b851 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1525,8 +1525,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437234","messageId":"patch-v8-11.17-273acb45517-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 11/17] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:52Z","receivedAt":"2021-09-28T02:19:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nIt would be a better end-state to have already moved the declaration\nof these functions to object-store.h to avoid the forward declaration\nof \"struct object_info\" in cache.h, but let's leave that cleanup for\nsome other time.\n\n1. https://lore.kernel.org/git/patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 4 +++-\n object-file.c | 20 +++++++-------------\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..35f254dae4a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1320,7 +1320,9 @@ char *xdg_cache_home(const char *filename);\n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+struct object_info;\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 8475b128944..6b91c4edcf6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1385,8 +1385,8 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1446,14 +1446,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1508,10 +1500,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2599,6 +2591,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2613,7 +2607,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437235","messageId":"patch-v8-10.17-b96828f3d5b-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 10/17] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:51Z","receivedAt":"2021-09-28T02:19:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 766ba88b851..8475b128944 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1284,7 +1284,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t buffer, bufsiz);\n \n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/* Make sure we have the terminating NUL */\n \tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437236","messageId":"patch-v8-12.17-314d34357dd-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 12/17] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:53Z","receivedAt":"2021-09-28T02:19:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 17 ++++++++++++++-\n object-file.c | 58 ++++++++++++++++++---------------------------------\n streaming.c | 3 ++-\n 3 files changed, 38 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 35f254dae4a..d7189aed8fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1319,7 +1319,22 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n+\n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 6b91c4edcf6..1327872cbf4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1255,11 +1255,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1270,35 +1271,8 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n \tif (status < Z_OK)\n \t\treturn -1;\n \n@@ -1308,6 +1282,14 @@ static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1457,6 +1439,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1490,11 +1473,9 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \tif (status < 0)\n@@ -2602,7 +2583,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437237","messageId":"patch-v8-14.17-42b8d135c8c-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 14/17] object-file.c: return ULHR_TOO_LONG on \"header too long\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:55Z","receivedAt":"2021-09-28T02:19:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return ULHR_TOO_LONG saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 5 ++++-\n object-file.c | 8 ++++++--\n streaming.c | 1 +\n t/t1006-cat-file.sh | 4 ++--\n 4 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7239e20a625..8e05392fda8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1328,16 +1328,19 @@ int git_open_cloexec(const char *name, int flags);\n *\n * - ULHR_OK on success\n * - ULHR_BAD on error\n+ * - ULHR_TOO_LONG if the header was too long\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), ULHR_TOO_LONG will still be returned\n+ * from this function to indicate that the header was too long.\n */\n enum unpack_loose_header_result {\n \tULHR_OK,\n \tULHR_BAD,\n+\tULHR_TOO_LONG,\n };\n enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned char *map,\ndiff --git a/object-file.c b/object-file.c\nindex e0f508415dd..3589c5a2e33 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1290,7 +1290,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn ULHR_BAD;\n+\t\treturn ULHR_TOO_LONG;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1311,7 +1311,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn ULHR_BAD;\n+\treturn ULHR_TOO_LONG;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1484,6 +1484,10 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t\tbreak;\n+\tcase ULHR_TOO_LONG:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n \t}\n \n \tif (status < 0) {\ndiff --git a/streaming.c b/streaming.c\nindex 6df0247a4cb..bd89c50e7b3 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -235,6 +235,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_OK:\n \t\tbreak;\n \tcase ULHR_BAD:\n+\tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n \tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5b16c69c286..a5e7401af8b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -356,12 +356,12 @@ do\n \t\t\tif test \"$arg2\" = \"-p\"\n \t\t\tthen\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n \t\t\t\tEOF\n \t\t\telse\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: git cat-file: could not get object info\n \t\t\t\tEOF\n \t\t\tfi &&\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437238","messageId":"patch-v8-13.17-07481bcb55c-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 13/17] object-file.c: use \"enum\" return type for unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:54Z","receivedAt":"2021-09-28T02:19:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In a preceding commit we changed and documented unpack_loose_header()\nfrom its previous behavior of returning any negative value or zero, to\nonly -1 or 0.\n\nLet's add an \"enum unpack_loose_header_result\" type and use it for\nthese return values, and have the compiler assert that we're\nexhaustively covering all of them.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 19 +++++++++++++++----\n object-file.c | 34 +++++++++++++++++++++-------------\n streaming.c | 23 +++++++++++++----------\n 3 files changed, 49 insertions(+), 27 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d7189aed8fc..7239e20a625 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1324,7 +1324,10 @@ int git_open_cloexec(const char *name, int flags);\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns:\n+ *\n+ * - ULHR_OK on success\n+ * - ULHR_BAD on error\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n@@ -1332,9 +1335,17 @@ int git_open_cloexec(const char *name, int flags);\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n * with parse_loose_header().\n */\n-int unpack_loose_header(git_zstream *stream, unsigned char *map,\n-\t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n+enum unpack_loose_header_result {\n+\tULHR_OK,\n+\tULHR_BAD,\n+};\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *hdrbuf);\n+\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 1327872cbf4..e0f508415dd 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1255,10 +1255,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz,\n-\t\t\tstruct strbuf *header)\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *header)\n {\n \tint status;\n \n@@ -1274,13 +1276,13 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn 0;\n+\t\treturn ULHR_OK;\n \n \t/*\n \t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n@@ -1288,7 +1290,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1309,7 +1311,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn ULHR_BAD;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1474,13 +1476,19 @@ static int loose_object_info(struct repository *r,\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n-\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n+\tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t\tbreak;\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..6df0247a4cb 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -229,17 +229,16 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n-\tif ((unpack_loose_header(&st->z,\n-\t\t\t\t st->u.loose.mapped,\n-\t\t\t\t st->u.loose.mapsize,\n-\t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr),\n-\t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n-\t\tgit_inflate_end(&st->z);\n-\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-\t\treturn -1;\n+\tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n+\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n+\t\t\t\t sizeof(st->u.loose.hdr), NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n+\t\tgoto error;\n \t}\n+\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n \tst->u.loose.hdr_avail = st->z.total_out;\n@@ -248,6 +247,10 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->read = read_istream_loose;\n \n \treturn 0;\n+error:\n+\tgit_inflate_end(&st->z);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn -1;\n }\n \n \n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437239","messageId":"patch-v8-15.17-106b7461ce9-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 15/17] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:56Z","receivedAt":"2021-09-28T02:19:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make parse_loose_header() return error codes and data instead of\ninvoking die() by itself.\n\nFor now we'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller. In a subsequent\ncommit we'll make read_loose_object() return an error code instead of\ndying. We should also address the \"allow_unknown\" case (should be\nmoved to builtin/cat-file.c), but for now I'll be leaving it.\n\nFor making parse_loose_header() not die() change its prototype to\naccept a \"struct object_info *\" instead of the \"unsigned long *sizep\"\nit accepted before. Its callers can now check the populated populated\n\"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nSince 93cff9a978e (sha1_loose_object_info: return error for corrupted\nobjects, 2017-04-01) the return value of loose_object_info() (then\nnamed sha1_loose_object_info()) had been a \"status\" variable that be\nany negative value, as we were expecting to return the \"enum\nobject_type\".\n\nThe only negative type happens to be OBJ_BAD, but the code still\nassumed that more might be added. This was then used later in\ne.g. c84a1f3ed4d (sha1_file: refactor read_object, 2017-06-21). Now\nthat parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSince parse_loose_header() doesn't need to return an arbitrary\n\"status\" we only need to treat its \"ret < 0\" specially, but can\nidiomatically overwrite it with our own error() return. This along\nwith having made unpack_loose_header() return an \"enum\nunpack_loose_header_result\" in an earlier commit means that we can\nmove the previously nested if/else cases mostly into the \"ULHR_OK\"\nbranch of the \"switch\" statement.\n\nWe should be less silent if we reach that \"status = -1\" branch, which\nhappens if we've got trailing garbage in loose objects, see\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor a better way to handle it. For now let's punt on it, a subsequent\ncommit will address that edge case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 11 +++++++--\n object-file.c | 67 +++++++++++++++++++++++++--------------------------\n streaming.c | 3 ++-\n 3 files changed, 44 insertions(+), 37 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8e05392fda8..6c5f00c82d5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1349,9 +1349,16 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned long bufsiz,\n \t\t\t\t\t\t struct strbuf *hdrbuf);\n \n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n struct object_info;\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 3589c5a2e33..a70669700d0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1369,8 +1369,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1392,15 +1391,6 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1427,7 +1417,14 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1441,6 +1438,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1472,6 +1470,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1479,6 +1479,18 @@ static int loose_object_info(struct repository *r,\n \tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n \tcase ULHR_OK:\n+\t\tif (parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0)\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t\telse if (!allow_unknown && *oi->typep < 0)\n+\t\t\tdie(_(\"invalid object type\"));\n+\n+\t\tif (!oi->contentp)\n+\t\t\tbreak;\n+\t\t*oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);\n+\t\tif (*oi->contentp)\n+\t\t\tgoto cleanup;\n+\n+\t\tstatus = -1;\n \t\tbreak;\n \tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n@@ -1490,31 +1502,16 @@ static int loose_object_info(struct repository *r,\n \t\tbreak;\n \t}\n \n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n-\n-\tif (status >= 0 && oi->contentp) {\n-\t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n-\t\tif (!*oi->contentp) {\n-\t\t\tgit_inflate_end(&stream);\n-\t\t\tstatus = -1;\n-\t\t}\n-\t} else\n-\t\tgit_inflate_end(&stream);\n-\n+\tgit_inflate_end(&stream);\n+cleanup:\n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n-\treturn (status < 0) ? status : 0;\n+\treturn status;\n }\n \n int obj_read_use_lock = 0;\n@@ -2585,6 +2582,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2601,12 +2599,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/streaming.c b/streaming.c\nindex bd89c50e7b3..fe54665d86e 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -238,7 +239,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n-\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n \t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437240","messageId":"patch-v8-16.17-d01223ae322-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 16/17] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:57Z","receivedAt":"2021-09-28T02:19:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it to accept a \"struct object_info\" (which contains the\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE in its flags). See\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor the introduction of read_loose_object().\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 17 ++++++++++++++---\n object-file.c | 18 ++++++------------\n object-store.h | 6 +++---\n t/t1450-fsck.sh | 17 +++++++++--------\n 4 files changed, 32 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..3b046820750 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,11 +600,22 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint err = 0;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n+\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\tif (type < 0)\n+\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(oid), sb.buf, path);\n+\tif (err) {\n \t\terrors_found |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex a70669700d0..fe95285f405 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2572,18 +2572,15 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t struct object_info *oi)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2599,15 +2596,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \n-\tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n+\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \t\t\tgoto out;\n \t} else {\n@@ -2618,8 +2613,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex c5130d8baea..c90c41a07f7 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,6 +245,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -252,9 +253,8 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t struct object_info *oi);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex bd696d21dba..167c319823a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -85,11 +85,10 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n-\t\tcat >expect <<-\\EOF &&\n-\t\tfatal: invalid object type\n-\t\tEOF\n-\t\ttest_must_fail git fsck 2>actual &&\n-\t\ttest_cmp expect actual\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n@@ -910,7 +909,7 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \tgit init --bare garbage-type &&\n \t(\n \t\tcd garbage-type &&\n@@ -922,8 +921,10 @@ test_expect_success 'fsck hard errors on an invalid object type' '\n \t\tfatal: invalid object type\n \t\tEOF\n \t\ttest_must_fail git fsck >out 2>err &&\n-\t\ttest_cmp err.expect err &&\n-\t\ttest_must_be_empty out\n+\t\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\t\ttest_line_count = 1 errors &&\n+\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n+\t\tgrep \"dangling blob $empty_blob\" out\n \t)\n '\n \n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437241","messageId":"patch-v8-17.17-7f394a991a6-20210928T021616Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v8 17/17] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-28T02:18:58Z","receivedAt":"2021-09-28T02:19:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob >objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391\n $ git fsck\n error: garbage at end of loose object 'e69d[...]'\n error: unable to unpack contents of ./objects/e6/9d[...]\n error: e69d[...]: object corrupt or missing: ./objects/e6/9d[...]\n\nThere is currently some weird messaging in the edge case when the two\nare combined, i.e. because we're not explicitly passing along an error\nstate about this specific scenario from check_stream_oid() via\nread_loose_object() we'll end up printing the null OID if an object is\nof an unknown type *and* it can't be unpacked by zlib, e.g.:\n\n $ git hash-object --stdin -w -t garbage --literally >objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n $ /usr/bin/git fsck\n fatal: invalid object type\n $ ~/g/git/git fsck\n error: garbage at end of loose object '8315a83d2acc4c174aed59430f9a9c4ed926440f'\n error: unable to unpack contents of ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 8315a83d2acc4c174aed59430f9a9c4ed926440f: object corrupt or missing: ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 0000000000000000000000000000000000000000: object is of unknown type 'garbage': ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n [...]\n\nI think it's OK to leave that for future improvements, which would\ninvolve enum-ifying more error state as we've done with \"enum\nunpack_loose_header_result\" in preceding commits. In these\nincreasingly more obscure cases the worst that can happen is that\nwe'll get slightly nonsensical or inapplicable error messages.\n\nThere's other such potential edge cases, all of which might produce\nsome confusing messaging, but still be handled correctly as far as\npassing along errors goes. E.g. if check_object_signature() returns\nand oideq(real_oid, null_oid()) is true, which could happen if it\nreturns -1 due to the read_istream() call having failed.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 23 +++++++++++++++--------\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n cache.h | 3 ++-\n object-file.c | 21 ++++++++++-----------\n object-store.h | 1 +\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 11 files changed, 42 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 95e8e89e81f..8e2caf72819 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 3b046820750..d925cdbae5c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -598,23 +598,30 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct object *obj;\n \tenum object_type type;\n \tunsigned long size;\n-\tvoid *contents;\n+\tunsigned char *contents = NULL;\n \tint eaten;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n-\tint err = 0;\n+\tstruct object_id real_oid = *null_oid();\n+\tint ret;\n \n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n-\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n-\t\t\t oid_to_hex(oid), path);\n+\tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n+\tif (ret < 0) {\n+\t\tif (contents && !oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\t}\n \tif (type < 0)\n-\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t\t oid_to_hex(oid), sb.buf, path);\n-\tif (err) {\n+\t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n+\tif (ret < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7ce69c087ec..15ae406e6b7 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1415,7 +1415,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/cache.h b/cache.h\nindex 6c5f00c82d5..e2a203073ea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1361,7 +1361,8 @@ struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n \n int finalize_object_file(const char *tmpfile, const char *filename);\n \ndiff --git a/object-file.c b/object-file.c\nindex fe95285f405..49561e31551 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1084,9 +1084,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1094,8 +1096,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1120,9 +1122,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2572,6 +2574,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi)\n {\n@@ -2582,8 +2585,6 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n-\t*contents = NULL;\n-\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n@@ -2613,9 +2614,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex c90c41a07f7..17b072e5a19 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -253,6 +253,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \ndiff --git a/object.c b/object.c\nindex 4e85955a941..23a24e678a8 100644\n--- a/object.c\n+++ b/object.c\n@@ -279,7 +279,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -290,7 +290,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex c8e560d71ab..3f418e3a6af 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex a5e7401af8b..0f52ca9cc82 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -512,7 +512,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 167c319823a..eb0e772f098 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -54,6 +54,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcd hash-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -65,7 +66,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -75,6 +76,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcd hash-type-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -87,8 +89,8 @@ test_expect_success 'object with hash and type mismatch' '\n \n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.33.0.1327.g9926af6cb02\n\n"},{"id":"437478","messageId":"YVTDgJ7wFl9DCjS+@nand.local","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"Re: [PATCH v8 00/17] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T19:50:24Z","receivedAt":"2021-09-29T19:50:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 28, 2021 at 04:18:41AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> This improves fsck error reporting, see the examples in the commit\n> messages of 16/17 and 17/17. To get there I've lib-ified more things\n> in object-file.c and the general object APIs, i.e. now we'll return\n> error codes instead of calling die() in these cases.\n>\n> v6 of this got a very detailed review from Taylor Blau (thanks a\n> lot!), for the v6 see:\n> https://lore.kernel.org/git/cover-v6-00.22-00000000000-20210907T104558Z-avarab@gmail.com/\n>\n> The v7 had a couple of trivial shellscripting issues, a typo'd\n> test_oid variable, and a warning on a \"test\" comparison. For v7 see\n> https://lore.kernel.org/git/cover-v7-00.17-00000000000-20210920T190304Z-avarab@gmail.com/\n\nThanks; I looked at the range-diff and it addresses both of my comments.\n\nThis series looks good to me.\n\nThanks,\nTaylor\n"},{"id":"437522","messageId":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com","subject":"[PATCH v9 00/17] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:05Z","receivedAt":"2021-09-30T13:37:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting, see the examples in the commit\nmessages of 16/17 and 17/17. To get there I've lib-ified more things\nin object-file.c and the general object APIs, i.e. now we'll return\nerror codes instead of calling die() in these cases.\n\nStatus of this: Since v6 this series has been getting a thorough\nreview from Taylor Blau, thanks again Taylor! See [1] for the v8, [2]\nfor Taylor's ack on the [2], and [3] for my own status update on the\nlast What's Cooking regarding the v8.\n\nThe only change since v8 is the plugging of a memory leak introduced\nin the previous 16/17. I've been doing integration of my local pending\npatches using some follow-up work for the in-flight\nab/sanitize-leak-ci topic, which is already proving quite useful.\n\n1. https://lore.kernel.org/git/cover-v8-00.17-00000000000-20210928T021616Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/YVTDgJ7wFl9DCjS+@nand.local/\n3. https://lore.kernel.org/git/87czotzaru.fsf@evledraar.gmail.com/\n\nÆvar Arnfjörð Bjarmason (17):\n fsck tests: add test for fsck-ing an unknown type\n fsck tests: refactor one test to use a sub-repo\n fsck tests: test current hash/type mismatch behavior\n fsck tests: test for garbage appended to a loose object\n cat-file tests: move bogus_* variable declarations earlier\n cat-file tests: test for missing/bogus object with -t, -s and -p\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n object-file.c: don't set \"typep\" when returning non-zero\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: make parse_loose_header_extended() public\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: use \"enum\" return type for unpack_loose_header()\n object-file.c: return ULHR_TOO_LONG on \"header too long\"\n object-file.c: stop dying in parse_loose_header()\n fsck: don't hard die on invalid object types\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 37 +++++--\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 45 ++++++++-\n object-file.c | 176 +++++++++++++++------------------\n object-store.h | 7 +-\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 27 +++--\n t/oid-info/oid | 2 +\n t/t1006-cat-file.sh | 223 +++++++++++++++++++++++++++++++++++++++---\n t/t1450-fsck.sh | 99 +++++++++++++++----\n 13 files changed, 468 insertions(+), 162 deletions(-)\n\nRange-diff against v8:\n 1: b999ab695d9 = 1: 520732612f7 fsck tests: add test for fsck-ing an unknown type\n 2: e01c21378a4 = 2: af7086623fe fsck tests: refactor one test to use a sub-repo\n 3: 93197a7bcee = 3: 102bc4f0176 fsck tests: test current hash/type mismatch behavior\n 4: 277188dd58d = 4: ff7fc09d5a1 fsck tests: test for garbage appended to a loose object\n 5: ab2ea1beaaf = 5: 278df093239 cat-file tests: move bogus_* variable declarations earlier\n 6: 91229b94fac = 6: 290bf983590 cat-file tests: test for missing/bogus object with -t, -s and -p\n 7: 9e95e134d30 = 7: a41b2c571e5 cat-file tests: add corrupt loose object test\n 8: 215f98ad369 = 8: cedeb117330 cat-file tests: test for current --allow-unknown-type behavior\n 9: 3e1df3594df = 9: 6f0673d38c8 object-file.c: don't set \"typep\" when returning non-zero\n10: b96828f3d5b = 10: 6637e8fd2ca object-file.c: return -1, not \"status\" from unpack_loose_header()\n11: 273acb45517 = 11: 51db08ebbae object-file.c: make parse_loose_header_extended() public\n12: 314d34357dd = 12: dffe5581f6f object-file.c: simplify unpack_loose_short_header()\n13: 07481bcb55c = 13: eb7c949c8b7 object-file.c: use \"enum\" return type for unpack_loose_header()\n14: 42b8d135c8c = 14: f4cc7271df7 object-file.c: return ULHR_TOO_LONG on \"header too long\"\n15: 106b7461ce9 = 15: 25d6ec668d4 object-file.c: stop dying in parse_loose_header()\n16: d01223ae322 ! 16: 6ce0414b2b7 fsck: don't hard die on invalid object types\n @@ Commit message\n f6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\n for the introduction of read_loose_object().\n \n + Since we're now passing in a \"oi.type_name\" we'll have to clean up the\n + allocated \"strbuf sb\". That we're doing it right is asserted by\n + e.g. the \"fsck notices broken commit\" test added in 03818a4a94c\n + (split_ident: parse timestamp from end of line, 2013-10-14). To do\n + that switch to a \"goto cleanup\" pattern, and while we're at it factor\n + out the already duplicated free(content) to use that pattern.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/fsck.c ##\n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *p\n \t\terrors_found |= ERROR_OBJECT;\n -\t\terror(_(\"%s: object corrupt or missing: %s\"),\n -\t\t oid_to_hex(oid), path);\n - \t\treturn 0; /* keep checking other objects */\n +-\t\treturn 0; /* keep checking other objects */\n ++\t\tgoto cleanup;\n + \t}\n + \n + \tif (!contents && type != OBJ_BLOB)\n +@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n + \t\terrors_found |= ERROR_OBJECT;\n + \t\terror(_(\"%s: object could not be parsed: %s\"),\n + \t\t oid_to_hex(oid), path);\n +-\t\tif (!eaten)\n +-\t\t\tfree(contents);\n +-\t\treturn 0; /* keep checking other objects */\n ++\t\tgoto cleanup_eaten;\n \t}\n \n + \tobj->flags &= ~(REACHABLE | SEEN);\n +@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n + \tif (fsck_obj(obj, contents, size))\n + \t\terrors_found |= ERROR_OBJECT;\n + \n ++cleanup_eaten:\n + \tif (!eaten)\n + \t\tfree(contents);\n ++cleanup:\n ++\tstrbuf_release(&sb);\n + \treturn 0; /* keep checking other objects, even if we saw an error */\n + }\n + \n \n ## object-file.c ##\n @@ object-file.c: static int check_stream_oid(git_zstream *stream,\n17: 7f394a991a6 ! 17: 8d926e41fc3 fsck: report invalid object type-path combinations\n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *p\n +\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n +\tif (ret < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n - \t\treturn 0; /* keep checking other objects */\n + \t\tgoto cleanup;\n \t}\n \n ## builtin/index-pack.c ##\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437523","messageId":"patch-v9-01.17-520732612f7-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 01/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:06Z","receivedAt":"2021-09-30T13:37:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..969bfbbdd8f 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -865,4 +865,21 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\tgit init --bare garbage-type &&\n+\t(\n+\t\tcd garbage-type &&\n+\n+\t\tempty=$(git hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck >out 2>err &&\n+\t\ttest_cmp err.expect err &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437524","messageId":"patch-v9-03.17-102bc4f0176-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 03/17] fsck tests: test current hash/type mismatch behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:08Z","receivedAt":"2021-09-30T13:37:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If fsck we move an object around between .git/objects/?? directories\nto simulate a hash mismatch \"git fsck\" will currently hard die() in\nobject-file.c. This behavior will be fixed in subsequent commits, but\nlet's test for it as-is for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f8edd15abf8..175ed304637 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -69,6 +69,30 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\tgit init --bare hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437525","messageId":"patch-v9-02.17-af7086623fe-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 02/17] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:07Z","receivedAt":"2021-09-30T13:37:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe can instead use the pattern of creating a named sub-repository,\nthen we don't have to worry about cleaning up after ourselves, nobody\nwill care what state the broken \"hash-mismatch\" repository is after\nthis test runs.\n\nSee [1] for related discussion on various \"modern\" test patterns that\ncan be used to avoid verbosity and increase reliability.\n\n1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 35 ++++++++++++++++++-----------------\n 1 file changed, 18 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 969bfbbdd8f..f8edd15abf8 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,25 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n+test_expect_success 'object with hash mismatch' '\n+\tgit init --bare hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n \n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437526","messageId":"patch-v9-04.17-ff7fc09d5a1-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 04/17] fsck tests: test for garbage appended to a loose object","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:09Z","receivedAt":"2021-09-30T13:37:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There wasn't any output tests for this scenario, let's ensure that we\ndon't regress on it in the changes that come after this.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 175ed304637..bd696d21dba 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -93,6 +93,26 @@ test_expect_success 'object with hash and type mismatch' '\n \t)\n '\n \n+test_expect_success POSIXPERM 'zlib corrupt loose object output ' '\n+\tgit init --bare corrupt-loose-output &&\n+\t(\n+\t\tcd corrupt-loose-output &&\n+\t\toid=$(git hash-object -w --stdin --literally >$oidf &&\n+\n+\t\tcat >expect.error <<-EOF &&\n+\t\terror: garbage at end of loose object '\\''$oid'\\''\n+\t\terror: unable to unpack contents of ./$oidf\n+\t\terror: $oid: object corrupt or missing: ./$oidf\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\tgrep ^error: actual >error &&\n+\t\ttest_cmp expect.error error\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437527","messageId":"patch-v9-05.17-278df093239-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 05/17] cat-file tests: move bogus_* variable declarations earlier","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:10Z","receivedAt":"2021-09-30T13:37:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the short/long bogus bogus object type variables into a form\nwhere the two sets can be used concurrently. This'll be used by\nsubsequently added tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 35 +++++++++++++++++++----------------\n 1 file changed, 19 insertions(+), 16 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..ea6a53d425b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,36 +315,39 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n-bogus_type=\"bogus\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n+test_expect_success 'setup bogus data' '\n+\tbogus_short_type=\"bogus\" &&\n+\tbogus_short_content=\"bogus\" &&\n+\tbogus_short_size=$(strlen \"$bogus_short_content\") &&\n+\tbogus_short_sha1=$(echo_without_newline \"$bogus_short_content\" | git hash-object -t $bogus_short_type --literally -w --stdin) &&\n+\n+\tbogus_long_type=\"abcdefghijklmnopqrstuvwxyz1234679\" &&\n+\tbogus_long_content=\"bogus\" &&\n+\tbogus_long_size=$(strlen \"$bogus_long_content\") &&\n+\tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n+'\n \n test_expect_success \"Type of broken object is correct\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of broken object is correct\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n-bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n test_expect_success \"Type of broken object is correct when type is large\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of large broken object is correct when type is large\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437528","messageId":"patch-v9-06.17-290bf983590-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 06/17] cat-file tests: test for missing/bogus object with -t, -s and -p","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:11Z","receivedAt":"2021-09-30T13:37:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When we look up a missing object with cat_one_file() what error we\nprint out currently depends on whether we'll error out early in\nget_oid_with_context(), or if we'll get an error later from\noid_object_info_extended().\n\nThe --allow-unknown-type flag then changes whether we pass the\n\"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\" flag to get_oid_with_context() or\nnot.\n\nThe \"-p\" flag is yet another special-case in printing the same output\non the deadbeef OID as we'd emit on the deadbeef_short OID for the\n\"-s\" and \"-t\" options, it also doesn't support the\n\"--allow-unknown-type\" flag at all.\n\nLet's test the combination of the two sets of [-t, -s, -p] and\n[--{no-}allow-unknown-type] (the --no-allow-unknown-type is implicit\nin not supplying it), as well as a [missing,bogus] object pair.\n\nThis extends tests added in 3e370f9faf0 (t1006: add tests for git\ncat-file --allow-unknown-type, 2015-05-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/oid-info/oid | 2 ++\n t/t1006-cat-file.sh | 75 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 77 insertions(+)\n\ndiff --git a/t/oid-info/oid b/t/oid-info/oid\nindex a754970523c..7547d2c7903 100644\n--- a/t/oid-info/oid\n+++ b/t/oid-info/oid\n@@ -27,3 +27,5 @@ numeric\t\tsha1:0123456789012345678901234567890123456789\n numeric\t\tsha256:0123456789012345678901234567890123456789012345678901234567890123\n deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n+deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n+deadbeef_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex ea6a53d425b..abf57339a29 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -327,6 +327,81 @@ test_expect_success 'setup bogus data' '\n \tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n '\n \n+for arg1 in '' --allow-unknown-type\n+do\n+\tfor arg2 in -s -t -p\n+\tdo\n+\t\tif test \"$arg1\" = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus short OID\" '\n+\t\t\tcat >expect <<-\\EOF &&\n+\t\t\tfatal: invalid object type\n+\t\t\tEOF\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_short_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_long_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing short OID\" '\n+\t\t\tcat >expect.err <<-EOF &&\n+\t\t\tfatal: Not a valid object name $(test_oid deadbeef_short)\n+\t\t\tEOF\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef_short) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect.err <<-EOF\n+\t\t\t\tfatal: Not a valid object name $(test_oid deadbeef)\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect.err <<-\\EOF\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out &&\n+\t\t\ttest_cmp expect.err err.actual\n+\t\t'\n+\tdone\n+done\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437529","messageId":"patch-v9-07.17-a41b2c571e5-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 07/17] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:12Z","receivedAt":"2021-09-30T13:37:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex abf57339a29..15774979ad3 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -426,6 +426,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437530","messageId":"patch-v9-08.17-cedeb117330-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 08/17] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:13Z","receivedAt":"2021-09-30T13:37:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 15774979ad3..5b16c69c286 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -402,6 +402,67 @@ do\n \tdone\n done\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_short_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_short_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_short_type $bogus_short_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_short_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect <<-EOF &&\n+\t$bogus_short_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\ttest_when_finished \"rm -rf .git/refs/replace\" &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_short_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437531","messageId":"patch-v9-09.17-6f0673d38c8-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 09/17] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:14Z","receivedAt":"2021-09-30T13:37:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex be4f94ecf3b..766ba88b851 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1525,8 +1525,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437532","messageId":"patch-v9-10.17-6637e8fd2ca-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 10/17] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:15Z","receivedAt":"2021-09-30T13:37:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 766ba88b851..8475b128944 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1284,7 +1284,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t buffer, bufsiz);\n \n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/* Make sure we have the terminating NUL */\n \tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437533","messageId":"patch-v9-12.17-dffe5581f6f-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 12/17] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:17Z","receivedAt":"2021-09-30T13:37:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 17 ++++++++++++++-\n object-file.c | 58 ++++++++++++++++++---------------------------------\n streaming.c | 3 ++-\n 3 files changed, 38 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 35f254dae4a..d7189aed8fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1319,7 +1319,22 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n+\n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 6b91c4edcf6..1327872cbf4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1255,11 +1255,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1270,35 +1271,8 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n \tif (status < Z_OK)\n \t\treturn -1;\n \n@@ -1308,6 +1282,14 @@ static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1457,6 +1439,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1490,11 +1473,9 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \tif (status < 0)\n@@ -2602,7 +2583,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437534","messageId":"patch-v9-11.17-51db08ebbae-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 11/17] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:16Z","receivedAt":"2021-09-30T13:37:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nIt would be a better end-state to have already moved the declaration\nof these functions to object-store.h to avoid the forward declaration\nof \"struct object_info\" in cache.h, but let's leave that cleanup for\nsome other time.\n\n1. https://lore.kernel.org/git/patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 4 +++-\n object-file.c | 20 +++++++-------------\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..35f254dae4a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1320,7 +1320,9 @@ char *xdg_cache_home(const char *filename);\n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+struct object_info;\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 8475b128944..6b91c4edcf6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1385,8 +1385,8 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1446,14 +1446,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1508,10 +1500,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2599,6 +2591,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2613,7 +2607,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437535","messageId":"patch-v9-13.17-eb7c949c8b7-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 13/17] object-file.c: use \"enum\" return type for unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:18Z","receivedAt":"2021-09-30T13:38:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In a preceding commit we changed and documented unpack_loose_header()\nfrom its previous behavior of returning any negative value or zero, to\nonly -1 or 0.\n\nLet's add an \"enum unpack_loose_header_result\" type and use it for\nthese return values, and have the compiler assert that we're\nexhaustively covering all of them.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 19 +++++++++++++++----\n object-file.c | 34 +++++++++++++++++++++-------------\n streaming.c | 23 +++++++++++++----------\n 3 files changed, 49 insertions(+), 27 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d7189aed8fc..7239e20a625 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1324,7 +1324,10 @@ int git_open_cloexec(const char *name, int flags);\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns:\n+ *\n+ * - ULHR_OK on success\n+ * - ULHR_BAD on error\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n@@ -1332,9 +1335,17 @@ int git_open_cloexec(const char *name, int flags);\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n * with parse_loose_header().\n */\n-int unpack_loose_header(git_zstream *stream, unsigned char *map,\n-\t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n+enum unpack_loose_header_result {\n+\tULHR_OK,\n+\tULHR_BAD,\n+};\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *hdrbuf);\n+\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 1327872cbf4..e0f508415dd 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1255,10 +1255,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz,\n-\t\t\tstruct strbuf *header)\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *header)\n {\n \tint status;\n \n@@ -1274,13 +1276,13 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn 0;\n+\t\treturn ULHR_OK;\n \n \t/*\n \t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n@@ -1288,7 +1290,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1309,7 +1311,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn ULHR_BAD;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1474,13 +1476,19 @@ static int loose_object_info(struct repository *r,\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n-\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n+\tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t\tbreak;\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..6df0247a4cb 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -229,17 +229,16 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n-\tif ((unpack_loose_header(&st->z,\n-\t\t\t\t st->u.loose.mapped,\n-\t\t\t\t st->u.loose.mapsize,\n-\t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr),\n-\t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n-\t\tgit_inflate_end(&st->z);\n-\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-\t\treturn -1;\n+\tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n+\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n+\t\t\t\t sizeof(st->u.loose.hdr), NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n+\t\tgoto error;\n \t}\n+\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n \tst->u.loose.hdr_avail = st->z.total_out;\n@@ -248,6 +247,10 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->read = read_istream_loose;\n \n \treturn 0;\n+error:\n+\tgit_inflate_end(&st->z);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn -1;\n }\n \n \n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437536","messageId":"patch-v9-14.17-f4cc7271df7-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 14/17] object-file.c: return ULHR_TOO_LONG on \"header too long\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:19Z","receivedAt":"2021-09-30T13:38:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return ULHR_TOO_LONG saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 5 ++++-\n object-file.c | 8 ++++++--\n streaming.c | 1 +\n t/t1006-cat-file.sh | 4 ++--\n 4 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7239e20a625..8e05392fda8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1328,16 +1328,19 @@ int git_open_cloexec(const char *name, int flags);\n *\n * - ULHR_OK on success\n * - ULHR_BAD on error\n+ * - ULHR_TOO_LONG if the header was too long\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), ULHR_TOO_LONG will still be returned\n+ * from this function to indicate that the header was too long.\n */\n enum unpack_loose_header_result {\n \tULHR_OK,\n \tULHR_BAD,\n+\tULHR_TOO_LONG,\n };\n enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned char *map,\ndiff --git a/object-file.c b/object-file.c\nindex e0f508415dd..3589c5a2e33 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1290,7 +1290,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn ULHR_BAD;\n+\t\treturn ULHR_TOO_LONG;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1311,7 +1311,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn ULHR_BAD;\n+\treturn ULHR_TOO_LONG;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1484,6 +1484,10 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t\tbreak;\n+\tcase ULHR_TOO_LONG:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n \t}\n \n \tif (status < 0) {\ndiff --git a/streaming.c b/streaming.c\nindex 6df0247a4cb..bd89c50e7b3 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -235,6 +235,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_OK:\n \t\tbreak;\n \tcase ULHR_BAD:\n+\tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n \tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5b16c69c286..a5e7401af8b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -356,12 +356,12 @@ do\n \t\t\tif test \"$arg2\" = \"-p\"\n \t\t\tthen\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n \t\t\t\tEOF\n \t\t\telse\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: git cat-file: could not get object info\n \t\t\t\tEOF\n \t\t\tfi &&\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437537","messageId":"patch-v9-15.17-25d6ec668d4-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 15/17] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:20Z","receivedAt":"2021-09-30T13:38:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make parse_loose_header() return error codes and data instead of\ninvoking die() by itself.\n\nFor now we'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller. In a subsequent\ncommit we'll make read_loose_object() return an error code instead of\ndying. We should also address the \"allow_unknown\" case (should be\nmoved to builtin/cat-file.c), but for now I'll be leaving it.\n\nFor making parse_loose_header() not die() change its prototype to\naccept a \"struct object_info *\" instead of the \"unsigned long *sizep\"\nit accepted before. Its callers can now check the populated populated\n\"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nSince 93cff9a978e (sha1_loose_object_info: return error for corrupted\nobjects, 2017-04-01) the return value of loose_object_info() (then\nnamed sha1_loose_object_info()) had been a \"status\" variable that be\nany negative value, as we were expecting to return the \"enum\nobject_type\".\n\nThe only negative type happens to be OBJ_BAD, but the code still\nassumed that more might be added. This was then used later in\ne.g. c84a1f3ed4d (sha1_file: refactor read_object, 2017-06-21). Now\nthat parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSince parse_loose_header() doesn't need to return an arbitrary\n\"status\" we only need to treat its \"ret < 0\" specially, but can\nidiomatically overwrite it with our own error() return. This along\nwith having made unpack_loose_header() return an \"enum\nunpack_loose_header_result\" in an earlier commit means that we can\nmove the previously nested if/else cases mostly into the \"ULHR_OK\"\nbranch of the \"switch\" statement.\n\nWe should be less silent if we reach that \"status = -1\" branch, which\nhappens if we've got trailing garbage in loose objects, see\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor a better way to handle it. For now let's punt on it, a subsequent\ncommit will address that edge case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 11 +++++++--\n object-file.c | 67 +++++++++++++++++++++++++--------------------------\n streaming.c | 3 ++-\n 3 files changed, 44 insertions(+), 37 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8e05392fda8..6c5f00c82d5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1349,9 +1349,16 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned long bufsiz,\n \t\t\t\t\t\t struct strbuf *hdrbuf);\n \n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n struct object_info;\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 3589c5a2e33..a70669700d0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1369,8 +1369,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1392,15 +1391,6 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1427,7 +1417,14 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1441,6 +1438,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1472,6 +1470,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1479,6 +1479,18 @@ static int loose_object_info(struct repository *r,\n \tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n \tcase ULHR_OK:\n+\t\tif (parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0)\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t\telse if (!allow_unknown && *oi->typep < 0)\n+\t\t\tdie(_(\"invalid object type\"));\n+\n+\t\tif (!oi->contentp)\n+\t\t\tbreak;\n+\t\t*oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);\n+\t\tif (*oi->contentp)\n+\t\t\tgoto cleanup;\n+\n+\t\tstatus = -1;\n \t\tbreak;\n \tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n@@ -1490,31 +1502,16 @@ static int loose_object_info(struct repository *r,\n \t\tbreak;\n \t}\n \n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n-\n-\tif (status >= 0 && oi->contentp) {\n-\t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n-\t\tif (!*oi->contentp) {\n-\t\t\tgit_inflate_end(&stream);\n-\t\t\tstatus = -1;\n-\t\t}\n-\t} else\n-\t\tgit_inflate_end(&stream);\n-\n+\tgit_inflate_end(&stream);\n+cleanup:\n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n-\treturn (status < 0) ? status : 0;\n+\treturn status;\n }\n \n int obj_read_use_lock = 0;\n@@ -2585,6 +2582,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2601,12 +2599,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/streaming.c b/streaming.c\nindex bd89c50e7b3..fe54665d86e 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -238,7 +239,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n-\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n \t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437538","messageId":"patch-v9-16.17-6ce0414b2b7-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 16/17] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:21Z","receivedAt":"2021-09-30T13:38:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it to accept a \"struct object_info\" (which contains the\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE in its flags). See\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor the introduction of read_loose_object().\n\nSince we're now passing in a \"oi.type_name\" we'll have to clean up the\nallocated \"strbuf sb\". That we're doing it right is asserted by\ne.g. the \"fsck notices broken commit\" test added in 03818a4a94c\n(split_ident: parse timestamp from end of line, 2013-10-14). To do\nthat switch to a \"goto cleanup\" pattern, and while we're at it factor\nout the already duplicated free(content) to use that pattern.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 26 +++++++++++++++++++-------\n object-file.c | 18 ++++++------------\n object-store.h | 6 +++---\n t/t1450-fsck.sh | 17 +++++++++--------\n 4 files changed, 37 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..623f8fc3194 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -600,12 +600,23 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint err = 0;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\toi.type_name = &sb;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n+\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\tif (type < 0)\n+\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(oid), sb.buf, path);\n+\tif (err) {\n \t\terrors_found |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n-\t\treturn 0; /* keep checking other objects */\n+\t\tgoto cleanup;\n \t}\n \n \tif (!contents && type != OBJ_BLOB)\n@@ -618,9 +629,7 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \t\terrors_found |= ERROR_OBJECT;\n \t\terror(_(\"%s: object could not be parsed: %s\"),\n \t\t oid_to_hex(oid), path);\n-\t\tif (!eaten)\n-\t\t\tfree(contents);\n-\t\treturn 0; /* keep checking other objects */\n+\t\tgoto cleanup_eaten;\n \t}\n \n \tobj->flags &= ~(REACHABLE | SEEN);\n@@ -628,8 +637,11 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tif (fsck_obj(obj, contents, size))\n \t\terrors_found |= ERROR_OBJECT;\n \n+cleanup_eaten:\n \tif (!eaten)\n \t\tfree(contents);\n+cleanup:\n+\tstrbuf_release(&sb);\n \treturn 0; /* keep checking other objects, even if we saw an error */\n }\n \ndiff --git a/object-file.c b/object-file.c\nindex a70669700d0..fe95285f405 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2572,18 +2572,15 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t struct object_info *oi)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2599,15 +2596,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \n-\tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n+\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \t\t\tgoto out;\n \t} else {\n@@ -2618,8 +2613,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex c5130d8baea..c90c41a07f7 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,6 +245,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -252,9 +253,8 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t struct object_info *oi);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex bd696d21dba..167c319823a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -85,11 +85,10 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n-\t\tcat >expect <<-\\EOF &&\n-\t\tfatal: invalid object type\n-\t\tEOF\n-\t\ttest_must_fail git fsck 2>actual &&\n-\t\ttest_cmp expect actual\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n@@ -910,7 +909,7 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \tgit init --bare garbage-type &&\n \t(\n \t\tcd garbage-type &&\n@@ -922,8 +921,10 @@ test_expect_success 'fsck hard errors on an invalid object type' '\n \t\tfatal: invalid object type\n \t\tEOF\n \t\ttest_must_fail git fsck >out 2>err &&\n-\t\ttest_cmp err.expect err &&\n-\t\ttest_must_be_empty out\n+\t\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\t\ttest_line_count = 1 errors &&\n+\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n+\t\tgrep \"dangling blob $empty_blob\" out\n \t)\n '\n \n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437539","messageId":"patch-v9-17.17-8d926e41fc3-20210930T133300Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v9 17/17] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-30T13:37:22Z","receivedAt":"2021-09-30T13:38:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob >objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391\n $ git fsck\n error: garbage at end of loose object 'e69d[...]'\n error: unable to unpack contents of ./objects/e6/9d[...]\n error: e69d[...]: object corrupt or missing: ./objects/e6/9d[...]\n\nThere is currently some weird messaging in the edge case when the two\nare combined, i.e. because we're not explicitly passing along an error\nstate about this specific scenario from check_stream_oid() via\nread_loose_object() we'll end up printing the null OID if an object is\nof an unknown type *and* it can't be unpacked by zlib, e.g.:\n\n $ git hash-object --stdin -w -t garbage --literally >objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n $ /usr/bin/git fsck\n fatal: invalid object type\n $ ~/g/git/git fsck\n error: garbage at end of loose object '8315a83d2acc4c174aed59430f9a9c4ed926440f'\n error: unable to unpack contents of ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 8315a83d2acc4c174aed59430f9a9c4ed926440f: object corrupt or missing: ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 0000000000000000000000000000000000000000: object is of unknown type 'garbage': ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n [...]\n\nI think it's OK to leave that for future improvements, which would\ninvolve enum-ifying more error state as we've done with \"enum\nunpack_loose_header_result\" in preceding commits. In these\nincreasingly more obscure cases the worst that can happen is that\nwe'll get slightly nonsensical or inapplicable error messages.\n\nThere's other such potential edge cases, all of which might produce\nsome confusing messaging, but still be handled correctly as far as\npassing along errors goes. E.g. if check_object_signature() returns\nand oideq(real_oid, null_oid()) is true, which could happen if it\nreturns -1 due to the read_istream() call having failed.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 23 +++++++++++++++--------\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n cache.h | 3 ++-\n object-file.c | 21 ++++++++++-----------\n object-store.h | 1 +\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 11 files changed, 42 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 95e8e89e81f..8e2caf72819 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 623f8fc3194..980c26e3b25 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -598,23 +598,30 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct object *obj;\n \tenum object_type type;\n \tunsigned long size;\n-\tvoid *contents;\n+\tunsigned char *contents = NULL;\n \tint eaten;\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n-\tint err = 0;\n+\tstruct object_id real_oid = *null_oid();\n+\tint ret;\n \n \toi.type_name = &sb;\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n-\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n-\t\t\t oid_to_hex(oid), path);\n+\tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n+\tif (ret < 0) {\n+\t\tif (contents && !oideq(&real_oid, oid))\n+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\t}\n \tif (type < 0)\n-\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t\t oid_to_hex(oid), sb.buf, path);\n-\tif (err) {\n+\t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n+\tif (ret < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\tgoto cleanup;\n \t}\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7ce69c087ec..15ae406e6b7 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1415,7 +1415,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/cache.h b/cache.h\nindex 6c5f00c82d5..e2a203073ea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1361,7 +1361,8 @@ struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n \n int finalize_object_file(const char *tmpfile, const char *filename);\n \ndiff --git a/object-file.c b/object-file.c\nindex fe95285f405..49561e31551 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1084,9 +1084,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1094,8 +1096,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1120,9 +1122,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2572,6 +2574,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi)\n {\n@@ -2582,8 +2585,6 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n-\t*contents = NULL;\n-\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n@@ -2613,9 +2614,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex c90c41a07f7..17b072e5a19 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -253,6 +253,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \ndiff --git a/object.c b/object.c\nindex 4e85955a941..23a24e678a8 100644\n--- a/object.c\n+++ b/object.c\n@@ -279,7 +279,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -290,7 +290,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex c8e560d71ab..3f418e3a6af 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex a5e7401af8b..0f52ca9cc82 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -512,7 +512,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 167c319823a..eb0e772f098 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -54,6 +54,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcd hash-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -65,7 +66,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -75,6 +76,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcd hash-type-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -87,8 +89,8 @@ test_expect_success 'object with hash and type mismatch' '\n \n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.33.0.1374.g05459a61530\n\n"},{"id":"437568","messageId":"YVYKyzlYSbYrYy3J@nand.local","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"Re: [PATCH v9 00/17] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-30T19:06:51Z","receivedAt":"2021-09-30T19:06:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 30, 2021 at 03:37:05PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> The only change since v8 is the plugging of a memory leak introduced\n> in the previous 16/17. I've been doing integration of my local pending\n> patches using some follow-up work for the in-flight\n> ab/sanitize-leak-ci topic, which is already proving quite useful.\n\nGood catch, sorry that I missed it myself when reading the previous\nversion. The way you plugged the leak is sensible to me, so I'm happy\nwith this version, too.\n\nThanks,\nTaylor\n"},{"id":"437570","messageId":"78bab348-ba3a-7a27-e32e-6b75f91178db@gmail.com","threadId":"55404","inReplyTo":"patch-v9-01.17-520732612f7-20210930T133300Z-avarab@gmail.com","subject":"Re: [PATCH v9 01/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Andrei Rybak","fromEmail":"rybak.a.v@gmail.com","sentAt":"2021-09-30T19:22:53Z","receivedAt":"2021-09-30T19:22:59Z","isPatch":true,"sender":{"key":"rybak.a.v@gmail.com","avatar":"https://avatars.githubusercontent.com/u/624072?v=4"},"body":"On 30/09/2021 15:37, Ævar Arnfjörð Bjarmason wrote:\n> Fix a blindspot in the fsck tests by checking what we do when we\n> encounter an unknown \"garbage\" type produced with hash-object's\n> --literally option.\n> \n> This behavior needs to be improved, which'll be done in subsequent\n> patches, but for now let's test for the current behavior.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> t/t1450-fsck.sh | 17 +++++++++++++++++\n> 1 file changed, 17 insertions(+)\n> \n> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n> index 5071ac63a5b..969bfbbdd8f 100755\n> --- a/t/t1450-fsck.sh\n> +++ b/t/t1450-fsck.sh\n> @@ -865,4 +865,21 @@ test_expect_success 'detect corrupt index file in fsck' '\n> \ttest_i18ngrep \"bad index file\" errors\n> '\n> \n> +test_expect_success 'fsck hard errors on an invalid object type' '\n> +\tgit init --bare garbage-type &&\n> +\t(\n> +\t\tcd garbage-type &&\n> +\n> +\t\tempty=$(git hash-object --stdin -w -t blob +\t\tgarbage=$(git hash-object --stdin -w -t garbage --literally +\n> +\t\tcat >err.expect <<-\\EOF &&\n> +\t\tfatal: invalid object type\n> +\t\tEOF\n> +\t\ttest_must_fail git fsck >out 2>err &&\n> +\t\ttest_cmp err.expect err &&\n> +\t\ttest_must_be_empty out\n> +\t)\n> +'\n> +\n> test_done\n> \n\n"},{"id":"437578","messageId":"xmqqsfxlaicg.fsf@gitster.g","threadId":"55404","inReplyTo":"patch-v9-17.17-8d926e41fc3-20210930T133300Z-avarab@gmail.com","subject":"Re: [PATCH v9 17/17] fsck: report invalid object type-path combinations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-30T21:01:19Z","receivedAt":"2021-09-30T21:01:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 623f8fc3194..980c26e3b25 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -598,23 +598,30 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \tstruct object *obj;\n> \tenum object_type type;\n> \tunsigned long size;\n> -\tvoid *contents;\n> +\tunsigned char *contents = NULL;\n> \tint eaten;\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> -\tint err = 0;\n> +\tstruct object_id real_oid = *null_oid();\n> +\tint ret;\n> \n> \toi.type_name = &sb;\n> \toi.sizep = &size;\n> \toi.typep = &type;\n> \n> -\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n> -\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n> -\t\t\t oid_to_hex(oid), path);\n> +\tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n> +\tif (ret < 0) {\n> +\t\tif (contents && !oideq(&real_oid, oid))\n> +\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n> +\t\t\t oid_to_hex(&real_oid), path);\n> +\t\telse\n> +\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n> +\t\t\t oid_to_hex(oid), path);\n> +\t}\n> \tif (type < 0)\n> -\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n> -\t\t\t oid_to_hex(oid), sb.buf, path);\n> -\tif (err) {\n> +\t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n> +\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n> +\tif (ret < 0) {\n> \t\terrors_found |= ERROR_OBJECT;\n> \t\tgoto cleanup;\n> \t}\n\nThis is immediately touching up what 16/17 has introduced, which is\nmaking it a bit harder to follow than necessary, so let's take the\nwhole postimage of 16+17.\n\n> static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> {\n> \tstruct object *obj;\n> \tenum object_type type;\n> \tunsigned long size;\n> \tunsigned char *contents = NULL;\n> \tint eaten;\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> \tstruct object_id real_oid = *null_oid();\n> \tint ret;\n> \n> \toi.type_name = &sb;\n> \toi.sizep = &size;\n> \toi.typep = &type;\n> \n> \tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n> \tif (ret < 0) {\n> \t\tif (contents && !oideq(&real_oid, oid))\n> \t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n> \t\t\t oid_to_hex(&real_oid), path);\n> \t\telse\n> \t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n> \t\t\t oid_to_hex(oid), path);\n\nWe can emit an error() message from either one of these. contents\nmay or may not be NULL, ret is negative, and we continue. Do we\nknow anything about the value of type at this point? IOW, will we\nget into the body of the next \"if (type < 0)\" statement to overwrite\nret with -1?\n\n> \t}\n> \tif (type < 0)\n> \t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n> \t\t\t oid_to_hex(&real_oid), sb.buf, path);\n> \tif (ret < 0) {\n> \t\terrors_found |= ERROR_OBJECT;\n> \t\tgoto cleanup;\n\nIn any case, we'd jump to clean-up if any of the above hold, so we'd\navoid hittign the next BUG().\n\n> \t}\n> \n> \tif (!contents && type != OBJ_BLOB)\n> \t\tBUG(\"read_loose_object streamed a non-blob\");\n> \n> \tobj = parse_object_buffer(the_repository, oid, type, size,\n> \t\t\t\t contents, &eaten);\n> \n> \tif (!obj) {\n> \t\terrors_found |= ERROR_OBJECT;\n> \t\terror(_(\"%s: object could not be parsed: %s\"),\n> \t\t oid_to_hex(oid), path);\n> \t\tgoto cleanup_eaten;\n> \t}\n> \n> \tobj->flags &= ~(REACHABLE | SEEN);\n> \tobj->flags |= HAS_OBJ;\n> \tif (fsck_obj(obj, contents, size))\n> \t\terrors_found |= ERROR_OBJECT;\n> \n> cleanup_eaten:\n> \tif (!eaten)\n> \t\tfree(contents);\n> cleanup:\n> \tstrbuf_release(&sb);\n\nIn the \"goto cleanup\" error case above, we haven't done anything\nthat would have caused the object contents eaten, and contents may\neither point at an allocated memory or NULL (in the \"hash-path\nmismatch\" case, we may have contents allocated but nobody has freed\nit yet, leaking it).\n\nI am wondering if we initialized \"eaten\" to false, we can get rid of\none of the two labels we added in this series, which would fix this\nleak as well, no?\n\n> \treturn 0; /* keep checking other objects, even if we saw an error */\n> }\n"},{"id":"437625","messageId":"87bl49ru4t.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"78bab348-ba3a-7a27-e32e-6b75f91178db@gmail.com","subject":"Re: [PATCH v9 01/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:05:35Z","receivedAt":"2021-10-01T09:07:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 30 2021, Andrei Rybak wrote:\n\n> On 30/09/2021 15:37, Ævar Arnfjörð Bjarmason wrote:\n>> Fix a blindspot in the fsck tests by checking what we do when we\n>> encounter an unknown \"garbage\" type produced with hash-object's\n>> --literally option.\n>> This behavior needs to be improved, which'll be done in subsequent\n>> patches, but for now let's test for the current behavior.\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> t/t1450-fsck.sh | 17 +++++++++++++++++\n>> 1 file changed, 17 insertions(+)\n>> diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\n>> index 5071ac63a5b..969bfbbdd8f 100755\n>> --- a/t/t1450-fsck.sh\n>> +++ b/t/t1450-fsck.sh\n>> @@ -865,4 +865,21 @@ test_expect_success 'detect corrupt index file in fsck' '\n>> \ttest_i18ngrep \"bad index file\" errors\n>> '\n>> +test_expect_success 'fsck hard errors on an invalid object type'\n>> '\n>> +\tgit init --bare garbage-type &&\n>> +\t(\n>> +\t\tcd garbage-type &&\n>> +\n>> +\t\tempty=$(git hash-object --stdin -w -t blob > +\t\tgarbage=$(git hash-object --stdin -w -t garbage --literally \n> Patch 01/17 introduces two unused variables: \"garbage\" and \"empty\".\n> However, patch 16/17 introduces grep checks for \"garbage_blob\" and\n> \"empty_blob\". Aside from that, 't/test-lib.sh' already defines\n> $EMPTY_BLOB.\n\nWill fix in the v10 re-roll.\n\nI think this is from an earlier version where I used the $empty, FWIW\nyou do need it (or to write it) even with $EMPTY_BLOB since that's just\nthe OID, but doesn't give you the object. You can write the /dev/null\ninput and then use $EMPTY_BLOB, but I thought using the output of\nhash-object was less confusing.\n\nBut in any case it isn't needed herea as you point out, we just need to\nwrite the garbage object, we don't need either variable. Thanks!\n"},{"id":"437633","messageId":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com","subject":"[PATCH v10 00/17] fsck: lib-ify object-file.c & better fsck \"invalid object\" error reporting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:36Z","receivedAt":"2021-10-01T09:17:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This improves fsck error reporting, see the examples in the commit\nmessages of 16/17 and 17/17. To get there I've lib-ified more things\nin object-file.c and the general object APIs, i.e. now we'll return\nerror codes instead of calling die() in these cases.\n\nThis should fix the issues noted about v9[1]. I.e.:\n\n A. Junio's right in [2] that the \"type\" can't be trusted after a\n failed read_loose_object(). I.e. if we'll fail before we can parse\n it out it'll be uninitialized. It's now initialized to OBJ_NONE,\n we can trust it if it's set to something else.\n\n B. I re-arranged much of 16/17 and 17/17 to make the diff (but not\n the range-diff) smaller.\n\n C. We now share a single \"strbuf\" across the whole fsck_loose() walk\n for saving away the type name, instead of allocating a new one\n each time. This is both a better memory usage pattern, and makes\n fsck_loose() itself simpler.\n\n It also allows for using much of the pre-image as-is, i.e. the\n whole \"goto cleanup\" is gone. Likewise instead of \"ret\" and \"err\"\n we just have the \"err\" variable now.\n\n D. I fixed the redundant/left-over test setup noted by Andrei\n Rybak[3].\n\n1. http://lore.kernel.org/git/cover-v9-00.17-00000000000-20210930T133300Z-avarab@gmail.com\n2. https://lore.kernel.org/git/xmqqsfxlaicg.fsf@gitster.g/\n3. https://lore.kernel.org/git/78bab348-ba3a-7a27-e32e-6b75f91178db@gmail.com/\n\nÆvar Arnfjörð Bjarmason (17):\n fsck tests: add test for fsck-ing an unknown type\n fsck tests: refactor one test to use a sub-repo\n fsck tests: test current hash/type mismatch behavior\n fsck tests: test for garbage appended to a loose object\n cat-file tests: move bogus_* variable declarations earlier\n cat-file tests: test for missing/bogus object with -t, -s and -p\n cat-file tests: add corrupt loose object test\n cat-file tests: test for current --allow-unknown-type behavior\n object-file.c: don't set \"typep\" when returning non-zero\n object-file.c: return -1, not \"status\" from unpack_loose_header()\n object-file.c: make parse_loose_header_extended() public\n object-file.c: simplify unpack_loose_short_header()\n object-file.c: use \"enum\" return type for unpack_loose_header()\n object-file.c: return ULHR_TOO_LONG on \"header too long\"\n object-file.c: stop dying in parse_loose_header()\n fsck: don't hard die on invalid object types\n fsck: report invalid object type-path combinations\n\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 44 +++++++--\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 +-\n cache.h | 45 ++++++++-\n object-file.c | 176 +++++++++++++++------------------\n object-store.h | 7 +-\n object.c | 4 +-\n pack-check.c | 3 +-\n streaming.c | 27 +++--\n t/oid-info/oid | 2 +\n t/t1006-cat-file.sh | 223 +++++++++++++++++++++++++++++++++++++++---\n t/t1450-fsck.sh | 97 ++++++++++++++----\n 13 files changed, 476 insertions(+), 159 deletions(-)\n\nRange-diff against v9:\n 1: 520732612f7 ! 1: 00936435423 fsck tests: add test for fsck-ing an unknown type\n @@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n +\t(\n +\t\tcd garbage-type &&\n +\n -+\t\tempty=$(git hash-object --stdin -w -t blob err.expect <<-\\EOF &&\n +\t\tfatal: invalid object type\n 2: af7086623fe = 2: 32a2f9cc0c9 fsck tests: refactor one test to use a sub-repo\n 3: 102bc4f0176 = 3: 00d661a6032 fsck tests: test current hash/type mismatch behavior\n 4: ff7fc09d5a1 = 4: a527e3b262c fsck tests: test for garbage appended to a loose object\n 5: 278df093239 = 5: 7a63d30aef3 cat-file tests: move bogus_* variable declarations earlier\n 6: 290bf983590 = 6: a563c7efe1c cat-file tests: test for missing/bogus object with -t, -s and -p\n 7: a41b2c571e5 = 7: c5affb65b7e cat-file tests: add corrupt loose object test\n 8: cedeb117330 = 8: 76f9888a6f7 cat-file tests: test for current --allow-unknown-type behavior\n 9: 6f0673d38c8 = 9: 85a91f43634 object-file.c: don't set \"typep\" when returning non-zero\n10: 6637e8fd2ca = 10: 51eaa2e8479 object-file.c: return -1, not \"status\" from unpack_loose_header()\n11: 51db08ebbae = 11: 5cd2ba830e9 object-file.c: make parse_loose_header_extended() public\n12: dffe5581f6f = 12: 6899c6ec17a object-file.c: simplify unpack_loose_short_header()\n13: eb7c949c8b7 = 13: a3bdd53d296 object-file.c: use \"enum\" return type for unpack_loose_header()\n14: f4cc7271df7 = 14: 5a7c2855b50 object-file.c: return ULHR_TOO_LONG on \"header too long\"\n15: 25d6ec668d4 = 15: 3ec9fee7ee9 object-file.c: stop dying in parse_loose_header()\n16: 6ce0414b2b7 ! 16: 9b75ac7c8ed fsck: don't hard die on invalid object types\n @@ Commit message\n f6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\n for the introduction of read_loose_object().\n \n - Since we're now passing in a \"oi.type_name\" we'll have to clean up the\n - allocated \"strbuf sb\". That we're doing it right is asserted by\n - e.g. the \"fsck notices broken commit\" test added in 03818a4a94c\n - (split_ident: parse timestamp from end of line, 2013-10-14). To do\n - that switch to a \"goto cleanup\" pattern, and while we're at it factor\n - out the already duplicated free(content) to use that pattern.\n + Since we'll need a \"struct strbuf\" to hold the \"type_name\" let's pass\n + it to the for_each_loose_file_in_objdir() callback to avoid allocating\n + a new one for each loose object in the iteration. It also makes the\n + memory management simpler than sticking it in fsck_loose() itself, as\n + we'll only need to strbuf_reset() it, with no need to do a\n + strbuf_release() before each \"return\".\n +\n + Before this commit we'd never check the \"type\" if read_loose_object()\n + failed, but now we do. We therefore need to initialize it to OBJ_NONE\n + to be able to tell the difference between e.g. its\n + unpack_loose_header() having failed, and us getting past that and into\n + parse_loose_header().\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/fsck.c ##\n -@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n +@@ builtin/fsck.c: static void get_default_heads(void)\n + \t}\n + }\n + \n ++struct for_each_loose_cb\n ++{\n ++\tstruct progress *progress;\n ++\tstruct strbuf obj_type;\n ++};\n ++\n + static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n + {\n ++\tstruct for_each_loose_cb *cb_data = data;\n + \tstruct object *obj;\n +-\tenum object_type type;\n ++\tenum object_type type = OBJ_NONE;\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n -+\tstruct strbuf sb = STRBUF_INIT;\n +\tstruct object_info oi = OBJECT_INFO_INIT;\n +\tint err = 0;\n \n -\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n -+\toi.type_name = &sb;\n ++\tstrbuf_reset(&cb_data->obj_type);\n ++\toi.type_name = &cb_data->obj_type;\n +\toi.sizep = &size;\n +\toi.typep = &type;\n +\n +\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n +\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n +\t\t\t oid_to_hex(oid), path);\n -+\tif (type < 0)\n ++\tif (type != OBJ_NONE && type < 0)\n +\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n -+\t\t\t oid_to_hex(oid), sb.buf, path);\n -+\tif (err) {\n ++\t\t\t oid_to_hex(oid), cb_data->obj_type.buf, path);\n ++\tif (err < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n -\t\terror(_(\"%s: object corrupt or missing: %s\"),\n -\t\t oid_to_hex(oid), path);\n --\t\treturn 0; /* keep checking other objects */\n -+\t\tgoto cleanup;\n - \t}\n - \n - \tif (!contents && type != OBJ_BLOB)\n -@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n - \t\terrors_found |= ERROR_OBJECT;\n - \t\terror(_(\"%s: object could not be parsed: %s\"),\n - \t\t oid_to_hex(oid), path);\n --\t\tif (!eaten)\n --\t\t\tfree(contents);\n --\t\treturn 0; /* keep checking other objects */\n -+\t\tgoto cleanup_eaten;\n + \t\treturn 0; /* keep checking other objects */\n \t}\n \n - \tobj->flags &= ~(REACHABLE | SEEN);\n -@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n - \tif (fsck_obj(obj, contents, size))\n - \t\terrors_found |= ERROR_OBJECT;\n +@@ builtin/fsck.c: static int fsck_cruft(const char *basename, const char *path, void *data)\n + \treturn 0;\n + }\n \n -+cleanup_eaten:\n - \tif (!eaten)\n - \t\tfree(contents);\n -+cleanup:\n -+\tstrbuf_release(&sb);\n - \treturn 0; /* keep checking other objects, even if we saw an error */\n +-static int fsck_subdir(unsigned int nr, const char *path, void *progress)\n ++static int fsck_subdir(unsigned int nr, const char *path, void *data)\n + {\n ++\tstruct for_each_loose_cb *cb_data = data;\n ++\tstruct progress *progress = cb_data->progress;\n + \tdisplay_progress(progress, nr + 1);\n + \treturn 0;\n + }\n +@@ builtin/fsck.c: static int fsck_subdir(unsigned int nr, const char *path, void *progress)\n + static void fsck_object_dir(const char *path)\n + {\n + \tstruct progress *progress = NULL;\n ++\tstruct for_each_loose_cb cb_data = {\n ++\t\t.obj_type = STRBUF_INIT,\n ++\t\t.progress = progress,\n ++\t};\n + \n + \tif (verbose)\n + \t\tfprintf_ln(stderr, _(\"Checking object directory\"));\n +@@ builtin/fsck.c: static void fsck_object_dir(const char *path)\n + \t\tprogress = start_progress(_(\"Checking object directories\"), 256);\n + \n + \tfor_each_loose_file_in_objdir(path, fsck_loose, fsck_cruft, fsck_subdir,\n +-\t\t\t\t progress);\n ++\t\t\t\t &cb_data);\n + \tdisplay_progress(progress, 256);\n + \tstop_progress(&progress);\n ++\tstrbuf_release(&cb_data.obj_type);\n }\n \n + static int fsck_head_link(const char *head_ref_name,\n \n ## object-file.c ##\n @@ object-file.c: static int check_stream_oid(git_zstream *stream,\n @@ t/t1450-fsck.sh: test_expect_success 'detect corrupt index file in fsck' '\n \tgit init --bare garbage-type &&\n \t(\n \t\tcd garbage-type &&\n -@@ t/t1450-fsck.sh: test_expect_success 'fsck hard errors on an invalid object type' '\n + \n +-\t\tgit hash-object --stdin -w -t garbage --literally err.expect <<-\\EOF &&\n \t\tfatal: invalid object type\n \t\tEOF\n \t\ttest_must_fail git fsck >out 2>err &&\n @@ t/t1450-fsck.sh: test_expect_success 'fsck hard errors on an invalid object type\n -\t\ttest_must_be_empty out\n +\t\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n +\t\ttest_line_count = 1 errors &&\n -+\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err &&\n -+\t\tgrep \"dangling blob $empty_blob\" out\n ++\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err\n \t)\n '\n \n17: 8d926e41fc3 ! 17: 838df0a979b fsck: report invalid object type-path combinations\n @@ builtin/fast-export.c: static void export_blob(const struct object_id *oid)\n \n ## builtin/fsck.c ##\n @@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n - \tstruct object *obj;\n - \tenum object_type type;\n - \tunsigned long size;\n --\tvoid *contents;\n -+\tunsigned char *contents = NULL;\n + \tvoid *contents;\n \tint eaten;\n - \tstruct strbuf sb = STRBUF_INIT;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n --\tint err = 0;\n +\tstruct object_id real_oid = *null_oid();\n -+\tint ret;\n + \tint err = 0;\n \n - \toi.type_name = &sb;\n + \tstrbuf_reset(&cb_data->obj_type);\n +@@ builtin/fsck.c: static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \toi.sizep = &size;\n \toi.typep = &type;\n \n -\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n -\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n -\t\t\t oid_to_hex(oid), path);\n -+\tret = read_loose_object(path, oid, &real_oid, (void **)&contents, &oi);\n -+\tif (ret < 0) {\n ++\tif (read_loose_object(path, oid, &real_oid, &contents, &oi) < 0) {\n +\t\tif (contents && !oideq(&real_oid, oid))\n -+\t\t\terror(_(\"%s: hash-path mismatch, found at: %s\"),\n -+\t\t\t oid_to_hex(&real_oid), path);\n ++\t\t\terr = error(_(\"%s: hash-path mismatch, found at: %s\"),\n ++\t\t\t\t oid_to_hex(&real_oid), path);\n +\t\telse\n -+\t\t\terror(_(\"%s: object corrupt or missing: %s\"),\n -+\t\t\t oid_to_hex(oid), path);\n ++\t\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n ++\t\t\t\t oid_to_hex(oid), path);\n +\t}\n - \tif (type < 0)\n --\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n --\t\t\t oid_to_hex(oid), sb.buf, path);\n --\tif (err) {\n -+\t\tret = error(_(\"%s: object is of unknown type '%s': %s\"),\n -+\t\t\t oid_to_hex(&real_oid), sb.buf, path);\n -+\tif (ret < 0) {\n + \tif (type != OBJ_NONE && type < 0)\n + \t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n +-\t\t\t oid_to_hex(oid), cb_data->obj_type.buf, path);\n ++\t\t\t oid_to_hex(&real_oid), cb_data->obj_type.buf,\n ++\t\t\t path);\n + \tif (err < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n - \t\tgoto cleanup;\n - \t}\n + \t\treturn 0; /* keep checking other objects */\n \n ## builtin/index-pack.c ##\n @@ builtin/index-pack.c: static void fix_unresolved_deltas(struct hashfile *f)\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437634","messageId":"patch-v10-01.17-00936435423-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 01/17] fsck tests: add test for fsck-ing an unknown type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:37Z","receivedAt":"2021-10-01T09:17:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the fsck tests by checking what we do when we\nencounter an unknown \"garbage\" type produced with hash-object's\n--literally option.\n\nThis behavior needs to be improved, which'll be done in subsequent\npatches, but for now let's test for the current behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b..beb233e91b1 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -865,4 +865,20 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n+test_expect_success 'fsck hard errors on an invalid object type' '\n+\tgit init --bare garbage-type &&\n+\t(\n+\t\tcd garbage-type &&\n+\n+\t\tgit hash-object --stdin -w -t garbage --literally err.expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck >out 2>err &&\n+\t\ttest_cmp err.expect err &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437635","messageId":"patch-v10-02.17-32a2f9cc0c9-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 02/17] fsck tests: refactor one test to use a sub-repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:38Z","receivedAt":"2021-10-01T09:17:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor one of the fsck tests to use a throwaway repository. It's a\npervasive pattern in t1450-fsck.sh to spend a lot of effort on the\nteardown of a tests so we're not leaving corrupt content for the next\ntest.\n\nWe can instead use the pattern of creating a named sub-repository,\nthen we don't have to worry about cleaning up after ourselves, nobody\nwill care what state the broken \"hash-mismatch\" repository is after\nthis test runs.\n\nSee [1] for related discussion on various \"modern\" test patterns that\ncan be used to avoid verbosity and increase reliability.\n\n1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 35 ++++++++++++++++++-----------------\n 1 file changed, 18 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex beb233e91b1..b73bc2a2ec3 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -48,24 +48,25 @@ remove_object () {\n \trm \"$(sha1_file \"$1\")\"\n }\n \n-test_expect_success 'object with bad sha1' '\n-\tsha=$(echo blob | git hash-object -w --stdin) &&\n-\told=$(test_oid_to_path \"$sha\") &&\n-\tnew=$(dirname $old)/$(test_oid ff_2) &&\n-\tsha=\"$(dirname $new)$(basename $new)\" &&\n-\tmv .git/objects/$old .git/objects/$new &&\n-\ttest_when_finished \"remove_object $sha\" &&\n-\tgit update-index --add --cacheinfo 100644 $sha foo &&\n-\ttest_when_finished \"git read-tree -u --reset HEAD\" &&\n-\ttree=$(git write-tree) &&\n-\ttest_when_finished \"remove_object $tree\" &&\n-\tcmt=$(echo bogus | git commit-tree $tree) &&\n-\ttest_when_finished \"remove_object $cmt\" &&\n-\tgit update-ref refs/heads/bogus $cmt &&\n-\ttest_when_finished \"git update-ref -d refs/heads/bogus\" &&\n+test_expect_success 'object with hash mismatch' '\n+\tgit init --bare hash-mismatch &&\n+\t(\n+\t\tcd hash-mismatch &&\n \n-\ttest_must_fail git fsck 2>out &&\n-\ttest_i18ngrep \"$sha.*corrupt\" out\n+\t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"$oid.*corrupt\" out\n+\t)\n '\n \n test_expect_success 'branch pointing to non-commit' '\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437636","messageId":"patch-v10-03.17-00d661a6032-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 03/17] fsck tests: test current hash/type mismatch behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:39Z","receivedAt":"2021-10-01T09:17:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If fsck we move an object around between .git/objects/?? directories\nto simulate a hash mismatch \"git fsck\" will currently hard die() in\nobject-file.c. This behavior will be fixed in subsequent commits, but\nlet's test for it as-is for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex b73bc2a2ec3..f9cabcecd14 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -69,6 +69,30 @@ test_expect_success 'object with hash mismatch' '\n \t)\n '\n \n+test_expect_success 'object with hash and type mismatch' '\n+\tgit init --bare hash-type-mismatch &&\n+\t(\n+\t\tcd hash-type-mismatch &&\n+\n+\t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\told=$(test_oid_to_path \"$oid\") &&\n+\t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n+\t\toid=\"$(dirname $new)$(basename $new)\" &&\n+\n+\t\tmv objects/$old objects/$new &&\n+\t\tgit update-index --add --cacheinfo 100644 $oid foo &&\n+\t\ttree=$(git write-tree) &&\n+\t\tcmt=$(echo bogus | git commit-tree $tree) &&\n+\t\tgit update-ref refs/heads/bogus $cmt &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\tfatal: invalid object type\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437637","messageId":"patch-v10-05.17-7a63d30aef3-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 05/17] cat-file tests: move bogus_* variable declarations earlier","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:41Z","receivedAt":"2021-10-01T09:17:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the short/long bogus bogus object type variables into a form\nwhere the two sets can be used concurrently. This'll be used by\nsubsequently added tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 35 +++++++++++++++++++----------------\n 1 file changed, 19 insertions(+), 16 deletions(-)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 18b3779ccb6..ea6a53d425b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -315,36 +315,39 @@ test_expect_success '%(deltabase) reports packed delta bases' '\n \t}\n '\n \n-bogus_type=\"bogus\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n+test_expect_success 'setup bogus data' '\n+\tbogus_short_type=\"bogus\" &&\n+\tbogus_short_content=\"bogus\" &&\n+\tbogus_short_size=$(strlen \"$bogus_short_content\") &&\n+\tbogus_short_sha1=$(echo_without_newline \"$bogus_short_content\" | git hash-object -t $bogus_short_type --literally -w --stdin) &&\n+\n+\tbogus_long_type=\"abcdefghijklmnopqrstuvwxyz1234679\" &&\n+\tbogus_long_content=\"bogus\" &&\n+\tbogus_long_size=$(strlen \"$bogus_long_content\") &&\n+\tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n+'\n \n test_expect_success \"Type of broken object is correct\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of broken object is correct\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_short_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_short_sha1 >actual &&\n \ttest_cmp expect actual\n '\n-bogus_type=\"abcdefghijklmnopqrstuvwxyz1234679\"\n-bogus_content=\"bogus\"\n-bogus_size=$(strlen \"$bogus_content\")\n-bogus_sha1=$(echo_without_newline \"$bogus_content\" | git hash-object -t $bogus_type --literally -w --stdin)\n \n test_expect_success \"Type of broken object is correct when type is large\" '\n-\techo $bogus_type >expect &&\n-\tgit cat-file -t --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_type >expect &&\n+\tgit cat-file -t --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success \"Size of large broken object is correct when type is large\" '\n-\techo $bogus_size >expect &&\n-\tgit cat-file -s --allow-unknown-type $bogus_sha1 >actual &&\n+\techo $bogus_long_size >expect &&\n+\tgit cat-file -s --allow-unknown-type $bogus_long_sha1 >actual &&\n \ttest_cmp expect actual\n '\n \n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437638","messageId":"patch-v10-06.17-a563c7efe1c-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 06/17] cat-file tests: test for missing/bogus object with -t, -s and -p","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:42Z","receivedAt":"2021-10-01T09:17:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When we look up a missing object with cat_one_file() what error we\nprint out currently depends on whether we'll error out early in\nget_oid_with_context(), or if we'll get an error later from\noid_object_info_extended().\n\nThe --allow-unknown-type flag then changes whether we pass the\n\"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\" flag to get_oid_with_context() or\nnot.\n\nThe \"-p\" flag is yet another special-case in printing the same output\non the deadbeef OID as we'd emit on the deadbeef_short OID for the\n\"-s\" and \"-t\" options, it also doesn't support the\n\"--allow-unknown-type\" flag at all.\n\nLet's test the combination of the two sets of [-t, -s, -p] and\n[--{no-}allow-unknown-type] (the --no-allow-unknown-type is implicit\nin not supplying it), as well as a [missing,bogus] object pair.\n\nThis extends tests added in 3e370f9faf0 (t1006: add tests for git\ncat-file --allow-unknown-type, 2015-05-03).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/oid-info/oid | 2 ++\n t/t1006-cat-file.sh | 75 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 77 insertions(+)\n\ndiff --git a/t/oid-info/oid b/t/oid-info/oid\nindex a754970523c..7547d2c7903 100644\n--- a/t/oid-info/oid\n+++ b/t/oid-info/oid\n@@ -27,3 +27,5 @@ numeric\t\tsha1:0123456789012345678901234567890123456789\n numeric\t\tsha256:0123456789012345678901234567890123456789012345678901234567890123\n deadbeef\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n deadbeef\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef\n+deadbeef_short\tsha1:deadbeefdeadbeefdeadbeefdeadbeefdeadbee\n+deadbeef_short\tsha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbee\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex ea6a53d425b..abf57339a29 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -327,6 +327,81 @@ test_expect_success 'setup bogus data' '\n \tbogus_long_sha1=$(echo_without_newline \"$bogus_long_content\" | git hash-object -t $bogus_long_type --literally -w --stdin)\n '\n \n+for arg1 in '' --allow-unknown-type\n+do\n+\tfor arg2 in -s -t -p\n+\tdo\n+\t\tif test \"$arg1\" = \"--allow-unknown-type\" && test \"$arg2\" = \"-p\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus short OID\" '\n+\t\t\tcat >expect <<-\\EOF &&\n+\t\t\tfatal: invalid object type\n+\t\t\tEOF\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_short_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on bogus full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect <<-EOF\n+\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\n+\t\t\tif test \"$arg1\" = \"--allow-unknown-type\"\n+\t\t\tthen\n+\t\t\t\tgit cat-file $arg1 $arg2 $bogus_short_sha1\n+\t\t\telse\n+\t\t\t\ttest_must_fail git cat-file $arg1 $arg2 $bogus_long_sha1 >out 2>actual &&\n+\t\t\t\ttest_must_be_empty out &&\n+\t\t\t\ttest_cmp expect actual\n+\t\t\tfi\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing short OID\" '\n+\t\t\tcat >expect.err <<-EOF &&\n+\t\t\tfatal: Not a valid object name $(test_oid deadbeef_short)\n+\t\t\tEOF\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef_short) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out\n+\t\t'\n+\n+\t\ttest_expect_success \"cat-file $arg1 $arg2 error on missing full OID\" '\n+\t\t\tif test \"$arg2\" = \"-p\"\n+\t\t\tthen\n+\t\t\t\tcat >expect.err <<-EOF\n+\t\t\t\tfatal: Not a valid object name $(test_oid deadbeef)\n+\t\t\t\tEOF\n+\t\t\telse\n+\t\t\t\tcat >expect.err <<-\\EOF\n+\t\t\t\tfatal: git cat-file: could not get object info\n+\t\t\t\tEOF\n+\t\t\tfi &&\n+\t\t\ttest_must_fail git cat-file $arg1 $arg2 $(test_oid deadbeef) >out 2>err.actual &&\n+\t\t\ttest_must_be_empty out &&\n+\t\t\ttest_cmp expect.err err.actual\n+\t\t'\n+\tdone\n+done\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437639","messageId":"patch-v10-04.17-a527e3b262c-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 04/17] fsck tests: test for garbage appended to a loose object","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:40Z","receivedAt":"2021-10-01T09:17:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"There wasn't any output tests for this scenario, let's ensure that we\ndon't regress on it in the changes that come after this.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1450-fsck.sh | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex f9cabcecd14..281ff8bdd8e 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -93,6 +93,26 @@ test_expect_success 'object with hash and type mismatch' '\n \t)\n '\n \n+test_expect_success POSIXPERM 'zlib corrupt loose object output ' '\n+\tgit init --bare corrupt-loose-output &&\n+\t(\n+\t\tcd corrupt-loose-output &&\n+\t\toid=$(git hash-object -w --stdin --literally >$oidf &&\n+\n+\t\tcat >expect.error <<-EOF &&\n+\t\terror: garbage at end of loose object '\\''$oid'\\''\n+\t\terror: unable to unpack contents of ./$oidf\n+\t\terror: $oid: object corrupt or missing: ./$oidf\n+\t\tEOF\n+\t\ttest_must_fail git fsck 2>actual &&\n+\t\tgrep ^error: actual >error &&\n+\t\ttest_cmp expect.error error\n+\t)\n+'\n+\n test_expect_success 'branch pointing to non-commit' '\n \tgit rev-parse HEAD^{tree} >.git/refs/heads/invalid &&\n \ttest_when_finished \"git update-ref -d refs/heads/invalid\" &&\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437640","messageId":"patch-v10-07.17-c5affb65b7e-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 07/17] cat-file tests: add corrupt loose object test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:43Z","receivedAt":"2021-10-01T09:17:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a blindspot in the tests for \"cat-file\" (and by proxy, the guts of\nobject-file.c) by testing that when we can't decode a loose object\nwith zlib we'll emit an error from zlib.c.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex abf57339a29..15774979ad3 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -426,6 +426,58 @@ test_expect_success \"Size of large broken object is correct when type is large\"\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'cat-file -t and -s on corrupt loose object' '\n+\tgit init --bare corrupt-loose.git &&\n+\t(\n+\t\tcd corrupt-loose.git &&\n+\n+\t\t# Setup and create the empty blob and its path\n+\t\tempty_path=$(git rev-parse --git-path objects/$(test_oid_to_path \"$EMPTY_BLOB\")) &&\n+\t\tgit hash-object -w --stdin other.blob &&\n+\t\tother_blob=$(git hash-object -w --stdin out.expect <<-EOF &&\n+\t\t0\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# Swap the two to corrupt the repository\n+\t\tmv -f \"$other_path\" \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"hash mismatch\" err.fsck &&\n+\n+\t\t# confirm that cat-file is reading the new swapped-in\n+\t\t# blob...\n+\t\tcat >out.expect <<-EOF &&\n+\t\tblob\n+\t\tEOF\n+\t\tgit cat-file -t \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# ... since it has a different size now.\n+\t\tcat >out.expect <<-EOF &&\n+\t\t6\n+\t\tEOF\n+\t\tgit cat-file -s \"$EMPTY_BLOB\" >out.actual 2>err.actual &&\n+\t\ttest_must_be_empty err.actual &&\n+\t\ttest_cmp out.expect out.actual &&\n+\n+\t\t# So far \"cat-file\" has been happy to spew the found\n+\t\t# content out as-is. Try to make it zlib-invalid.\n+\t\tmv -f other.blob \"$empty_path\" &&\n+\t\ttest_must_fail git fsck 2>err.fsck &&\n+\t\tgrep \"^error: inflate: data stream error (\" err.fsck\n+\t)\n+'\n+\n # Tests for git cat-file --follow-symlinks\n test_expect_success 'prep for symlink tests' '\n \techo_without_newline \"$hello_content\" >morx &&\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437641","messageId":"patch-v10-08.17-76f9888a6f7-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 08/17] cat-file tests: test for current --allow-unknown-type behavior","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:44Z","receivedAt":"2021-10-01T09:17:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more tests for the current --allow-unknown-type behavior. As noted\nin [1] I don't think much of this makes sense, but let's test for it\nas-is so we can see if the behavior changes in the future.\n\n1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1006-cat-file.sh | 61 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 15774979ad3..5b16c69c286 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -402,6 +402,67 @@ do\n \tdone\n done\n \n+test_expect_success '-e is OK with a broken object without --allow-unknown-type' '\n+\tgit cat-file -e $bogus_short_sha1\n+'\n+\n+test_expect_success '-e can not be combined with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file -e --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success '-p cannot print a broken object even with --allow-unknown-type' '\n+\ttest_must_fail git cat-file -p $bogus_short_sha1 &&\n+\ttest_expect_code 128 git cat-file -p --allow-unknown-type $bogus_short_sha1\n+'\n+\n+test_expect_success ' does not work with objects of broken types' '\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type \"bogus\"\n+\tEOF\n+\ttest_must_fail git cat-file $bogus_short_type $bogus_short_sha1 2>err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'broken types combined with --batch and --batch-check' '\n+\techo $bogus_short_sha1 >bogus-oid &&\n+\n+\tcat >err.expect <<-\\EOF &&\n+\tfatal: invalid object type\n+\tEOF\n+\n+\ttest_must_fail git cat-file --batch err.actual &&\n+\ttest_cmp err.expect err.actual &&\n+\n+\ttest_must_fail git cat-file --batch-check err.actual &&\n+\ttest_cmp err.expect err.actual\n+'\n+\n+test_expect_success 'the --batch and --batch-check options do not combine with --allow-unknown-type' '\n+\ttest_expect_code 128 git cat-file --batch --allow-unknown-type expect <<-EOF &&\n+\t$bogus_short_type\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Create it manually, as \"git replace\" will die on bogus\n+\t# types.\n+\thead=$(git rev-parse --verify HEAD) &&\n+\ttest_when_finished \"rm -rf .git/refs/replace\" &&\n+\tmkdir -p .git/refs/replace &&\n+\techo $head >.git/refs/replace/$bogus_short_sha1 &&\n+\n+\tcat >expect <<-EOF &&\n+\tcommit\n+\tEOF\n+\tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \"Type of broken object is correct\" '\n \techo $bogus_short_type >expect &&\n \tgit cat-file -t --allow-unknown-type $bogus_short_sha1 >actual &&\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437642","messageId":"patch-v10-09.17-85a91f43634-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 09/17] object-file.c: don't set \"typep\" when returning non-zero","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:45Z","receivedAt":"2021-10-01T09:17:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"When the loose_object_info() function returns an error stop faking up\nthe \"oi->typep\" to OBJ_BAD. Let the return value of the function\nitself suffice. This code cleanup simplifies subsequent changes.\n\nThat we set this at all is a relic from the past. Before\n052fe5eaca9 (sha1_loose_object_info: make type lookup optional,\n2013-07-12) we would always return the type_from_string(type) via the\nparse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't\nparse it.\n\nThen in a combination of 46f034483eb (sha1_file: support reading from\na loose object of unknown type, 2015-05-03) and\nb3ea7dd32d6 (sha1_loose_object_info: handle errors from\nunpack_sha1_rest, 2017-10-05) our API drifted even further towards\nconflating the two again.\n\nHaving read the code paths involved carefully I think this is OK. We\nare just about to return -1, and we have only one caller:\ndo_oid_object_info_extended(). That function will in turn go on to\nreturn -1 when we return -1 here.\n\nThis might be introducing a subtle bug where a caller of\noid_object_info_extended() would inspect its \"typep\" and expect a\nmeaningful value if the function returned -1.\n\nSuch a problem would not occur for its simpler oid_object_info()\nsister function. That one always returns the \"enum object_type\", which\nin the case of -1 would be the OBJ_BAD.\n\nHaving read the code for all the callers of these functions I don't\nbelieve any such bug is being introduced here, and in any case we'd\nlikely already have such a bug for the \"sizep\" member (although\nblindly checking \"typep\" first would be a more common case).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex be4f94ecf3b..766ba88b851 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1525,8 +1525,6 @@ static int loose_object_info(struct repository *r,\n \t\tgit_inflate_end(&stream);\n \n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437643","messageId":"patch-v10-10.17-51eaa2e8479-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 10/17] object-file.c: return -1, not \"status\" from unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:46Z","receivedAt":"2021-10-01T09:17:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Return a -1 when git_inflate() fails instead of whatever Z_* status\nwe'd get from zlib.c. This makes no difference to any error we report,\nbut makes it more obvious that we don't care about the specific zlib\nerror codes here.\n\nSee d21f8426907 (unpack_sha1_header(): detect malformed object header,\n2016-09-25) for the commit that added the \"return status\" code. As far\nas I can tell there was never a real reason (e.g. different reporting)\nfor carrying down the \"status\" as opposed to \"-1\".\n\nAt the time that d21f8426907 was written there was a corresponding\n\"ret < Z_OK\" check right after the unpack_sha1_header() call (the\n\"unpack_sha1_header()\" function was later rename to our current\n\"unpack_loose_header()\").\n\nHowever, that check was removed in c84a1f3ed4d (sha1_file: refactor\nread_object, 2017-06-21) without changing the corresponding return\ncode.\n\nSo let's do the minor cleanup of also changing this function to return\na -1.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 766ba88b851..8475b128944 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1284,7 +1284,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t buffer, bufsiz);\n \n \tif (status < Z_OK)\n-\t\treturn status;\n+\t\treturn -1;\n \n \t/* Make sure we have the terminating NUL */\n \tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437644","messageId":"patch-v10-11.17-5cd2ba830e9-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 11/17] object-file.c: make parse_loose_header_extended() public","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:47Z","receivedAt":"2021-10-01T09:17:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the parse_loose_header_extended() function public and remove the\nparse_loose_header() wrapper. The only direct user of it outside of\nobject-file.c itself was in streaming.c, that caller can simply pass\nthe required \"struct object-info *\" instead.\n\nThis change is being done in preparation for teaching\nread_loose_object() to accept a flag to pass to\nparse_loose_header(). It isn't strictly necessary for that change, we\ncould simply use parse_loose_header_extended() there, but will leave\nthe API in a better end state.\n\nIt would be a better end-state to have already moved the declaration\nof these functions to object-store.h to avoid the forward declaration\nof \"struct object_info\" in cache.h, but let's leave that cleanup for\nsome other time.\n\n1. https://lore.kernel.org/git/patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 4 +++-\n object-file.c | 20 +++++++-------------\n streaming.c | 5 ++++-\n 3 files changed, 14 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..35f254dae4a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1320,7 +1320,9 @@ char *xdg_cache_home(const char *filename);\n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n-int parse_loose_header(const char *hdr, unsigned long *sizep);\n+struct object_info;\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 8475b128944..6b91c4edcf6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1385,8 +1385,8 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi,\n+\t\t unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1446,14 +1446,6 @@ static int parse_loose_header_extended(const char *hdr, struct object_info *oi,\n \treturn *hdr ? -1 : type;\n }\n \n-int parse_loose_header(const char *hdr, unsigned long *sizep)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_loose_header_extended(hdr, &oi, 0);\n-}\n-\n static int loose_object_info(struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t struct object_info *oi, int flags)\n@@ -1508,10 +1500,10 @@ static int loose_object_info(struct repository *r,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n \t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n@@ -2599,6 +2591,8 @@ int read_loose_object(const char *path,\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = size;\n \n \t*contents = NULL;\n \n@@ -2613,7 +2607,7 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, size);\n+\t*type = parse_loose_header(hdr, &oi, 0);\n \tif (*type < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\ndiff --git a/streaming.c b/streaming.c\nindex 5f480ad50c4..8beac62cbb7 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -223,6 +223,9 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t const struct object_id *oid,\n \t\t\t enum object_type *type)\n {\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.sizep = &st->size;\n+\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n@@ -231,7 +234,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n \t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &st->size) < 0)) {\n+\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n \t\treturn -1;\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437645","messageId":"patch-v10-12.17-6899c6ec17a-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 12/17] object-file.c: simplify unpack_loose_short_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:48Z","receivedAt":"2021-10-01T09:17:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the unpack_loose_short_header(),\nunpack_loose_header_to_strbuf() and unpack_loose_header() functions\ninto one.\n\nThe unpack_loose_header_to_strbuf() function was added in\n46f034483eb (sha1_file: support reading from a loose object of unknown\ntype, 2015-05-03).\n\nIts code was mostly copy/pasted between it and both of\nunpack_loose_header() and unpack_loose_short_header(). We now have a\nsingle unpack_loose_header() function which accepts an optional\n\"struct strbuf *\" instead.\n\nI think the remaining unpack_loose_header() function could be further\nsimplified, we're carrying some complexity just to be able to emit a\ngarbage type longer than MAX_HEADER_LEN, we could alternatively just\nsay \"we found a garbage type ...\" instead. But let's\nleave the current behavior in place for now.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 17 ++++++++++++++-\n object-file.c | 58 ++++++++++++++++++---------------------------------\n streaming.c | 3 ++-\n 3 files changed, 38 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 35f254dae4a..d7189aed8fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1319,7 +1319,22 @@ char *xdg_cache_home(const char *filename);\n \n int git_open_cloexec(const char *name, int flags);\n #define git_open(name) git_open_cloexec(name, O_RDONLY)\n-int unpack_loose_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n+\n+/**\n+ * unpack_loose_header() initializes the data stream needed to unpack\n+ * a loose object header.\n+ *\n+ * Returns 0 on success. Returns negative values on error.\n+ *\n+ * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n+ * \"hdrbuf\" argument is non-NULL. This is intended for use with\n+ * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n+ * reporting. The full header will be extracted to \"hdrbuf\" for use\n+ * with parse_loose_header().\n+ */\n+int unpack_loose_header(git_zstream *stream, unsigned char *map,\n+\t\t\tunsigned long mapsize, void *buffer,\n+\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 6b91c4edcf6..1327872cbf4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1255,11 +1255,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-static int unpack_loose_short_header(git_zstream *stream,\n-\t\t\t\t unsigned char *map, unsigned long mapsize,\n-\t\t\t\t void *buffer, unsigned long bufsiz)\n+int unpack_loose_header(git_zstream *stream,\n+\t\t\tunsigned char *map, unsigned long mapsize,\n+\t\t\tvoid *buffer, unsigned long bufsiz,\n+\t\t\tstruct strbuf *header)\n {\n-\tint ret;\n+\tint status;\n \n \t/* Get the data stream */\n \tmemset(stream, 0, sizeof(*stream));\n@@ -1270,35 +1271,8 @@ static int unpack_loose_short_header(git_zstream *stream,\n \n \tgit_inflate_init(stream);\n \tobj_read_unlock();\n-\tret = git_inflate(stream, 0);\n+\tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n-\n-\treturn ret;\n-}\n-\n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz)\n-{\n-\tint status = unpack_loose_short_header(stream, map, mapsize,\n-\t\t\t\t\t buffer, bufsiz);\n-\n-\tif (status < Z_OK)\n-\t\treturn -1;\n-\n-\t/* Make sure we have the terminating NUL */\n-\tif (!memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn -1;\n-\treturn 0;\n-}\n-\n-static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map,\n-\t\t\t\t\t unsigned long mapsize, void *buffer,\n-\t\t\t\t\t unsigned long bufsiz, struct strbuf *header)\n-{\n-\tint status;\n-\n-\tstatus = unpack_loose_short_header(stream, map, mapsize, buffer, bufsiz);\n \tif (status < Z_OK)\n \t\treturn -1;\n \n@@ -1308,6 +1282,14 @@ static int unpack_loose_header_to_strbuf(git_zstream *stream, unsigned char *map\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n \t\treturn 0;\n \n+\t/*\n+\t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n+\t * here is only non-NULL when we run \"cat-file\n+\t * --allow-unknown-type\".\n+\t */\n+\tif (!header)\n+\t\treturn -1;\n+\n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n \t * result out to header, and then append the result of further\n@@ -1457,6 +1439,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n \t\toidclr(oi->delta_base_oid);\n@@ -1490,11 +1473,9 @@ static int loose_object_info(struct repository *r,\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n-\t\tif (unpack_loose_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n-\t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n+\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \tif (status < 0)\n@@ -2602,7 +2583,8 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {\n+\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\tNULL) < 0) {\n \t\terror(_(\"unable to unpack header of %s\"), path);\n \t\tgoto out;\n \t}\ndiff --git a/streaming.c b/streaming.c\nindex 8beac62cbb7..cb3c3cf6ff6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -233,7 +233,8 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \t\t\t\t st->u.loose.mapped,\n \t\t\t\t st->u.loose.mapsize,\n \t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr)) < 0) ||\n+\t\t\t\t sizeof(st->u.loose.hdr),\n+\t\t\t\t NULL) < 0) ||\n \t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n \t\tgit_inflate_end(&st->z);\n \t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437646","messageId":"patch-v10-13.17-a3bdd53d296-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 13/17] object-file.c: use \"enum\" return type for unpack_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:49Z","receivedAt":"2021-10-01T09:17:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In a preceding commit we changed and documented unpack_loose_header()\nfrom its previous behavior of returning any negative value or zero, to\nonly -1 or 0.\n\nLet's add an \"enum unpack_loose_header_result\" type and use it for\nthese return values, and have the compiler assert that we're\nexhaustively covering all of them.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 19 +++++++++++++++----\n object-file.c | 34 +++++++++++++++++++++-------------\n streaming.c | 23 +++++++++++++----------\n 3 files changed, 49 insertions(+), 27 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d7189aed8fc..7239e20a625 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1324,7 +1324,10 @@ int git_open_cloexec(const char *name, int flags);\n * unpack_loose_header() initializes the data stream needed to unpack\n * a loose object header.\n *\n- * Returns 0 on success. Returns negative values on error.\n+ * Returns:\n+ *\n+ * - ULHR_OK on success\n+ * - ULHR_BAD on error\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n@@ -1332,9 +1335,17 @@ int git_open_cloexec(const char *name, int flags);\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n * with parse_loose_header().\n */\n-int unpack_loose_header(git_zstream *stream, unsigned char *map,\n-\t\t\tunsigned long mapsize, void *buffer,\n-\t\t\tunsigned long bufsiz, struct strbuf *hdrbuf);\n+enum unpack_loose_header_result {\n+\tULHR_OK,\n+\tULHR_BAD,\n+};\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *hdrbuf);\n+\n struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi,\n \t\t unsigned int flags);\ndiff --git a/object-file.c b/object-file.c\nindex 1327872cbf4..e0f508415dd 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1255,10 +1255,12 @@ void *map_loose_object(struct repository *r,\n \treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n-int unpack_loose_header(git_zstream *stream,\n-\t\t\tunsigned char *map, unsigned long mapsize,\n-\t\t\tvoid *buffer, unsigned long bufsiz,\n-\t\t\tstruct strbuf *header)\n+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n+\t\t\t\t\t\t unsigned char *map,\n+\t\t\t\t\t\t unsigned long mapsize,\n+\t\t\t\t\t\t void *buffer,\n+\t\t\t\t\t\t unsigned long bufsiz,\n+\t\t\t\t\t\t struct strbuf *header)\n {\n \tint status;\n \n@@ -1274,13 +1276,13 @@ int unpack_loose_header(git_zstream *stream,\n \tstatus = git_inflate(stream, 0);\n \tobj_read_lock();\n \tif (status < Z_OK)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * Check if entire header is unpacked in the first iteration.\n \t */\n \tif (memchr(buffer, '\\0', stream->next_out - (unsigned char *)buffer))\n-\t\treturn 0;\n+\t\treturn ULHR_OK;\n \n \t/*\n \t * We have a header longer than MAX_HEADER_LEN. The \"header\"\n@@ -1288,7 +1290,7 @@ int unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn -1;\n+\t\treturn ULHR_BAD;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1309,7 +1311,7 @@ int unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn -1;\n+\treturn ULHR_BAD;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1474,13 +1476,19 @@ static int loose_object_info(struct repository *r,\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \n-\tif (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n-\t\t\t\tallow_unknown ? &hdrbuf : NULL) < 0)\n+\tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n+\t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n-\tif (status < 0)\n-\t\t; /* Do nothing */\n-\telse if (hdrbuf.len) {\n+\t\tbreak;\n+\t}\n+\n+\tif (status < 0) {\n+\t\t/* Do nothing */\n+\t} else if (hdrbuf.len) {\n \t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n \t\t\t\t oid_to_hex(oid));\ndiff --git a/streaming.c b/streaming.c\nindex cb3c3cf6ff6..6df0247a4cb 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -229,17 +229,16 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n-\tif ((unpack_loose_header(&st->z,\n-\t\t\t\t st->u.loose.mapped,\n-\t\t\t\t st->u.loose.mapsize,\n-\t\t\t\t st->u.loose.hdr,\n-\t\t\t\t sizeof(st->u.loose.hdr),\n-\t\t\t\t NULL) < 0) ||\n-\t (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)) {\n-\t\tgit_inflate_end(&st->z);\n-\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n-\t\treturn -1;\n+\tswitch (unpack_loose_header(&st->z, st->u.loose.mapped,\n+\t\t\t\t st->u.loose.mapsize, st->u.loose.hdr,\n+\t\t\t\t sizeof(st->u.loose.hdr), NULL)) {\n+\tcase ULHR_OK:\n+\t\tbreak;\n+\tcase ULHR_BAD:\n+\t\tgoto error;\n \t}\n+\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n \tst->u.loose.hdr_avail = st->z.total_out;\n@@ -248,6 +247,10 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tst->read = read_istream_loose;\n \n \treturn 0;\n+error:\n+\tgit_inflate_end(&st->z);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn -1;\n }\n \n \n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437647","messageId":"patch-v10-14.17-5a7c2855b50-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 14/17] object-file.c: return ULHR_TOO_LONG on \"header too long\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:50Z","receivedAt":"2021-10-01T09:17:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the return code for \"header too long\" from the generic\nnegative return value unpack_loose_header() returns, and report via\nerror() if we exceed MAX_HEADER_LEN.\n\nAs a test added earlier in this series in t1006-cat-file.sh shows\nwe'll correctly emit zlib errors from zlib.c already in this case, so\nwe have no need to carry those return codes further down the\nstack. Let's instead just return ULHR_TOO_LONG saying we ran into the\nMAX_HEADER_LEN limit, or other negative values for \"unable to unpack\n header\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 5 ++++-\n object-file.c | 8 ++++++--\n streaming.c | 1 +\n t/t1006-cat-file.sh | 4 ++--\n 4 files changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7239e20a625..8e05392fda8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1328,16 +1328,19 @@ int git_open_cloexec(const char *name, int flags);\n *\n * - ULHR_OK on success\n * - ULHR_BAD on error\n+ * - ULHR_TOO_LONG if the header was too long\n *\n * It will only parse up to MAX_HEADER_LEN bytes unless an optional\n * \"hdrbuf\" argument is non-NULL. This is intended for use with\n * OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)\n * reporting. The full header will be extracted to \"hdrbuf\" for use\n- * with parse_loose_header().\n+ * with parse_loose_header(), ULHR_TOO_LONG will still be returned\n+ * from this function to indicate that the header was too long.\n */\n enum unpack_loose_header_result {\n \tULHR_OK,\n \tULHR_BAD,\n+\tULHR_TOO_LONG,\n };\n enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned char *map,\ndiff --git a/object-file.c b/object-file.c\nindex e0f508415dd..3589c5a2e33 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1290,7 +1290,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t * --allow-unknown-type\".\n \t */\n \tif (!header)\n-\t\treturn ULHR_BAD;\n+\t\treturn ULHR_TOO_LONG;\n \n \t/*\n \t * buffer[0..bufsiz] was not large enough. Copy the partial\n@@ -1311,7 +1311,7 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\tstream->next_out = buffer;\n \t\tstream->avail_out = bufsiz;\n \t} while (status != Z_STREAM_END);\n-\treturn ULHR_BAD;\n+\treturn ULHR_TOO_LONG;\n }\n \n static void *unpack_loose_rest(git_zstream *stream,\n@@ -1484,6 +1484,10 @@ static int loose_object_info(struct repository *r,\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n \t\t\t oid_to_hex(oid));\n \t\tbreak;\n+\tcase ULHR_TOO_LONG:\n+\t\tstatus = error(_(\"header for %s too long, exceeds %d bytes\"),\n+\t\t\t oid_to_hex(oid), MAX_HEADER_LEN);\n+\t\tbreak;\n \t}\n \n \tif (status < 0) {\ndiff --git a/streaming.c b/streaming.c\nindex 6df0247a4cb..bd89c50e7b3 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -235,6 +235,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_OK:\n \t\tbreak;\n \tcase ULHR_BAD:\n+\tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n \tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 5b16c69c286..a5e7401af8b 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -356,12 +356,12 @@ do\n \t\t\tif test \"$arg2\" = \"-p\"\n \t\t\tthen\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: Not a valid object name $bogus_long_sha1\n \t\t\t\tEOF\n \t\t\telse\n \t\t\t\tcat >expect <<-EOF\n-\t\t\t\terror: unable to unpack $bogus_long_sha1 header\n+\t\t\t\terror: header for $bogus_long_sha1 too long, exceeds 32 bytes\n \t\t\t\tfatal: git cat-file: could not get object info\n \t\t\t\tEOF\n \t\t\tfi &&\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437648","messageId":"patch-v10-15.17-3ec9fee7ee9-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 15/17] object-file.c: stop dying in parse_loose_header()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:51Z","receivedAt":"2021-10-01T09:17:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make parse_loose_header() return error codes and data instead of\ninvoking die() by itself.\n\nFor now we'll move the relevant die() call to loose_object_info() and\nread_loose_object() to keep this change smaller. In a subsequent\ncommit we'll make read_loose_object() return an error code instead of\ndying. We should also address the \"allow_unknown\" case (should be\nmoved to builtin/cat-file.c), but for now I'll be leaving it.\n\nFor making parse_loose_header() not die() change its prototype to\naccept a \"struct object_info *\" instead of the \"unsigned long *sizep\"\nit accepted before. Its callers can now check the populated populated\n\"oi->typep\".\n\nBecause of this we don't need to pass in the \"unsigned int flags\"\nwhich we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do\nthat check in loose_object_info().\n\nThis also refactors some confusing control flow around the \"status\"\nvariable. In some cases we set it to the return value of \"error()\",\ni.e. -1, and later checked if \"status < 0\" was true.\n\nSince 93cff9a978e (sha1_loose_object_info: return error for corrupted\nobjects, 2017-04-01) the return value of loose_object_info() (then\nnamed sha1_loose_object_info()) had been a \"status\" variable that be\nany negative value, as we were expecting to return the \"enum\nobject_type\".\n\nThe only negative type happens to be OBJ_BAD, but the code still\nassumed that more might be added. This was then used later in\ne.g. c84a1f3ed4d (sha1_file: refactor read_object, 2017-06-21). Now\nthat parse_loose_header() will return 0 on success instead of the\ntype (which it'll stick into the \"struct object_info\") we don't need\nto conflate these two cases in its callers.\n\nSince parse_loose_header() doesn't need to return an arbitrary\n\"status\" we only need to treat its \"ret < 0\" specially, but can\nidiomatically overwrite it with our own error() return. This along\nwith having made unpack_loose_header() return an \"enum\nunpack_loose_header_result\" in an earlier commit means that we can\nmove the previously nested if/else cases mostly into the \"ULHR_OK\"\nbranch of the \"switch\" statement.\n\nWe should be less silent if we reach that \"status = -1\" branch, which\nhappens if we've got trailing garbage in loose objects, see\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor a better way to handle it. For now let's punt on it, a subsequent\ncommit will address that edge case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n cache.h | 11 +++++++--\n object-file.c | 67 +++++++++++++++++++++++++--------------------------\n streaming.c | 3 ++-\n 3 files changed, 44 insertions(+), 37 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8e05392fda8..6c5f00c82d5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1349,9 +1349,16 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,\n \t\t\t\t\t\t unsigned long bufsiz,\n \t\t\t\t\t\t struct strbuf *hdrbuf);\n \n+/**\n+ * parse_loose_header() parses the starting \" \\0\" of an\n+ * object. If it doesn't follow that format -1 is returned. To check\n+ * the validity of the populate the \"typep\" in the \"struct\n+ * object_info\". It will be OBJ_BAD if the object type is unknown. The\n+ * parsed can be retrieved via \"oi->sizep\", and from there\n+ * passed to unpack_loose_rest().\n+ */\n struct object_info;\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags);\n+int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\t void *buf, unsigned long size, const char *type);\ndiff --git a/object-file.c b/object-file.c\nindex 3589c5a2e33..a70669700d0 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1369,8 +1369,7 @@ static void *unpack_loose_rest(git_zstream *stream,\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-int parse_loose_header(const char *hdr, struct object_info *oi,\n-\t\t unsigned int flags)\n+int parse_loose_header(const char *hdr, struct object_info *oi)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1392,15 +1391,6 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \ttype = type_from_string_gently(type_buf, type_len, 1);\n \tif (oi->type_name)\n \t\tstrbuf_add(oi->type_name, type_buf, type_len);\n-\t/*\n-\t * Set type to 0 if its an unknown object and\n-\t * we're obtaining the type using '--allow-unknown-type'\n-\t * option.\n-\t */\n-\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n-\t\ttype = 0;\n-\telse if (type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \tif (oi->typep)\n \t\t*oi->typep = type;\n \n@@ -1427,7 +1417,14 @@ int parse_loose_header(const char *hdr, struct object_info *oi,\n \t/*\n \t * The length must be followed by a zero byte\n \t */\n-\treturn *hdr ? -1 : type;\n+\tif (*hdr)\n+\t\treturn -1;\n+\n+\t/*\n+\t * The format is valid, but the type may still be bogus. The\n+\t * Caller needs to check its oi->typep.\n+\t */\n+\treturn 0;\n }\n \n static int loose_object_info(struct repository *r,\n@@ -1441,6 +1438,7 @@ static int loose_object_info(struct repository *r,\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \tunsigned long size_scratch;\n+\tenum object_type type_scratch;\n \tint allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (oi->delta_base_oid)\n@@ -1472,6 +1470,8 @@ static int loose_object_info(struct repository *r,\n \n \tif (!oi->sizep)\n \t\toi->sizep = &size_scratch;\n+\tif (!oi->typep)\n+\t\toi->typep = &type_scratch;\n \n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n@@ -1479,6 +1479,18 @@ static int loose_object_info(struct repository *r,\n \tswitch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr),\n \t\t\t\t allow_unknown ? &hdrbuf : NULL)) {\n \tcase ULHR_OK:\n+\t\tif (parse_loose_header(hdrbuf.len ? hdrbuf.buf : hdr, oi) < 0)\n+\t\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n+\t\telse if (!allow_unknown && *oi->typep < 0)\n+\t\t\tdie(_(\"invalid object type\"));\n+\n+\t\tif (!oi->contentp)\n+\t\t\tbreak;\n+\t\t*oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);\n+\t\tif (*oi->contentp)\n+\t\t\tgoto cleanup;\n+\n+\t\tstatus = -1;\n \t\tbreak;\n \tcase ULHR_BAD:\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n@@ -1490,31 +1502,16 @@ static int loose_object_info(struct repository *r,\n \t\tbreak;\n \t}\n \n-\tif (status < 0) {\n-\t\t/* Do nothing */\n-\t} else if (hdrbuf.len) {\n-\t\tif ((status = parse_loose_header(hdrbuf.buf, oi, flags)) < 0)\n-\t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t oid_to_hex(oid));\n-\t} else if ((status = parse_loose_header(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n-\n-\tif (status >= 0 && oi->contentp) {\n-\t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, oid);\n-\t\tif (!*oi->contentp) {\n-\t\t\tgit_inflate_end(&stream);\n-\t\t\tstatus = -1;\n-\t\t}\n-\t} else\n-\t\tgit_inflate_end(&stream);\n-\n+\tgit_inflate_end(&stream);\n+cleanup:\n \tmunmap(map, mapsize);\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\tif (oi->typep == &type_scratch)\n+\t\toi->typep = NULL;\n \toi->whence = OI_LOOSE;\n-\treturn (status < 0) ? status : 0;\n+\treturn status;\n }\n \n int obj_read_use_lock = 0;\n@@ -2585,6 +2582,7 @@ int read_loose_object(const char *path,\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\toi.typep = type;\n \toi.sizep = size;\n \n \t*contents = NULL;\n@@ -2601,12 +2599,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\t*type = parse_loose_header(hdr, &oi, 0);\n-\tif (*type < 0) {\n+\tif (parse_loose_header(hdr, &oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n+\tif (*type < 0)\n+\t\tdie(_(\"invalid object type\"));\n \n \tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\ndiff --git a/streaming.c b/streaming.c\nindex bd89c50e7b3..fe54665d86e 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -225,6 +225,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toi.sizep = &st->size;\n+\toi.typep = type;\n \n \tst->u.loose.mapped = map_loose_object(r, oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n@@ -238,7 +239,7 @@ static int open_istream_loose(struct git_istream *st, struct repository *r,\n \tcase ULHR_TOO_LONG:\n \t\tgoto error;\n \t}\n-\tif (parse_loose_header(st->u.loose.hdr, &oi, 0) < 0)\n+\tif (parse_loose_header(st->u.loose.hdr, &oi) < 0 || *type < 0)\n \t\tgoto error;\n \n \tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437649","messageId":"patch-v10-16.17-9b75ac7c8ed-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 16/17] fsck: don't hard die on invalid object types","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:52Z","receivedAt":"2021-10-01T09:17:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the error fsck emits on invalid object types, such as:\n\n $ git hash-object --stdin -w -t garbage --literally \n\nFrom the very ungraceful error of:\n\n $ git fsck\n fatal: invalid object type\n $\n\nTo:\n\n $ git fsck\n error: : object is of unknown type 'garbage': \n [ other fsck output ]\n\nWe'll still exit with non-zero, but now we'll finish the rest of the\ntraversal. The tests that's being added here asserts that we'll still\ncomplain about other fsck issues (e.g. an unrelated dangling blob).\n\nTo do this we need to pass down the \"OBJECT_INFO_ALLOW_UNKNOWN_TYPE\"\nflag from read_loose_object() through to parse_loose_header(). Since\nthe read_loose_object() function is only used in builtin/fsck.c we can\nsimply change it to accept a \"struct object_info\" (which contains the\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE in its flags). See\nf6371f92104 (sha1_file: add read_loose_object() function, 2017-01-13)\nfor the introduction of read_loose_object().\n\nSince we'll need a \"struct strbuf\" to hold the \"type_name\" let's pass\nit to the for_each_loose_file_in_objdir() callback to avoid allocating\na new one for each loose object in the iteration. It also makes the\nmemory management simpler than sticking it in fsck_loose() itself, as\nwe'll only need to strbuf_reset() it, with no need to do a\nstrbuf_release() before each \"return\".\n\nBefore this commit we'd never check the \"type\" if read_loose_object()\nfailed, but now we do. We therefore need to initialize it to OBJ_NONE\nto be able to tell the difference between e.g. its\nunpack_loose_header() having failed, and us getting past that and into\nparse_loose_header().\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 37 +++++++++++++++++++++++++++++++------\n object-file.c | 18 ++++++------------\n object-store.h | 6 +++---\n t/t1450-fsck.sh | 18 +++++++++---------\n 4 files changed, 49 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f7..260210bf8a1 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -593,18 +593,36 @@ static void get_default_heads(void)\n \t}\n }\n \n+struct for_each_loose_cb\n+{\n+\tstruct progress *progress;\n+\tstruct strbuf obj_type;\n+};\n+\n static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n {\n+\tstruct for_each_loose_cb *cb_data = data;\n \tstruct object *obj;\n-\tenum object_type type;\n+\tenum object_type type = OBJ_NONE;\n \tunsigned long size;\n \tvoid *contents;\n \tint eaten;\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tint err = 0;\n \n-\tif (read_loose_object(path, oid, &type, &size, &contents) < 0) {\n+\tstrbuf_reset(&cb_data->obj_type);\n+\toi.type_name = &cb_data->obj_type;\n+\toi.sizep = &size;\n+\toi.typep = &type;\n+\n+\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n+\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t oid_to_hex(oid), path);\n+\tif (type != OBJ_NONE && type < 0)\n+\t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n+\t\t\t oid_to_hex(oid), cb_data->obj_type.buf, path);\n+\tif (err < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n-\t\terror(_(\"%s: object corrupt or missing: %s\"),\n-\t\t oid_to_hex(oid), path);\n \t\treturn 0; /* keep checking other objects */\n \t}\n \n@@ -640,8 +658,10 @@ static int fsck_cruft(const char *basename, const char *path, void *data)\n \treturn 0;\n }\n \n-static int fsck_subdir(unsigned int nr, const char *path, void *progress)\n+static int fsck_subdir(unsigned int nr, const char *path, void *data)\n {\n+\tstruct for_each_loose_cb *cb_data = data;\n+\tstruct progress *progress = cb_data->progress;\n \tdisplay_progress(progress, nr + 1);\n \treturn 0;\n }\n@@ -649,6 +669,10 @@ static int fsck_subdir(unsigned int nr, const char *path, void *progress)\n static void fsck_object_dir(const char *path)\n {\n \tstruct progress *progress = NULL;\n+\tstruct for_each_loose_cb cb_data = {\n+\t\t.obj_type = STRBUF_INIT,\n+\t\t.progress = progress,\n+\t};\n \n \tif (verbose)\n \t\tfprintf_ln(stderr, _(\"Checking object directory\"));\n@@ -657,9 +681,10 @@ static void fsck_object_dir(const char *path)\n \t\tprogress = start_progress(_(\"Checking object directories\"), 256);\n \n \tfor_each_loose_file_in_objdir(path, fsck_loose, fsck_cruft, fsck_subdir,\n-\t\t\t\t progress);\n+\t\t\t\t &cb_data);\n \tdisplay_progress(progress, 256);\n \tstop_progress(&progress);\n+\tstrbuf_release(&cb_data.obj_type);\n }\n \n static int fsck_head_link(const char *head_ref_name,\ndiff --git a/object-file.c b/object-file.c\nindex a70669700d0..fe95285f405 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2572,18 +2572,15 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents)\n+\t\t void **contents,\n+\t\t struct object_info *oi)\n {\n \tint ret = -1;\n \tvoid *map = NULL;\n \tunsigned long mapsize;\n \tgit_zstream stream;\n \tchar hdr[MAX_HEADER_LEN];\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\toi.typep = type;\n-\toi.sizep = size;\n+\tunsigned long *size = oi->sizep;\n \n \t*contents = NULL;\n \n@@ -2599,15 +2596,13 @@ int read_loose_object(const char *path,\n \t\tgoto out;\n \t}\n \n-\tif (parse_loose_header(hdr, &oi) < 0) {\n+\tif (parse_loose_header(hdr, oi) < 0) {\n \t\terror(_(\"unable to parse header of %s\"), path);\n \t\tgit_inflate_end(&stream);\n \t\tgoto out;\n \t}\n-\tif (*type < 0)\n-\t\tdie(_(\"invalid object type\"));\n \n-\tif (*type == OBJ_BLOB && *size > big_file_threshold) {\n+\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \t\t\tgoto out;\n \t} else {\n@@ -2618,8 +2613,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size,\n-\t\t\t\t\t type_name(*type))) {\n+\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n \t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t\t oid_to_hex(expected_oid));\n \t\t\tfree(*contents);\ndiff --git a/object-store.h b/object-store.h\nindex c5130d8baea..c90c41a07f7 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -245,6 +245,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n \n /*\n * Open the loose object at path, check its hash, and return the contents,\n+ * use the \"oi\" argument to assert things about the object, or e.g. populate its\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\n * to allow streaming of large blobs.\n *\n@@ -252,9 +253,8 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n-\t\t enum object_type *type,\n-\t\t unsigned long *size,\n-\t\t void **contents);\n+\t\t void **contents,\n+\t\t struct object_info *oi);\n \n /* Retry packed storage after checking packed and loose storage */\n #define HAS_OBJECT_RECHECK_PACKED 1\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 281ff8bdd8e..faf0e98847b 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -85,11 +85,10 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcmt=$(echo bogus | git commit-tree $tree) &&\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n-\t\tcat >expect <<-\\EOF &&\n-\t\tfatal: invalid object type\n-\t\tEOF\n-\t\ttest_must_fail git fsck 2>actual &&\n-\t\ttest_cmp expect actual\n+\n+\t\ttest_must_fail git fsck 2>out &&\n+\t\tgrep \"^error: hash mismatch for \" out &&\n+\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n@@ -910,19 +909,20 @@ test_expect_success 'detect corrupt index file in fsck' '\n \ttest_i18ngrep \"bad index file\" errors\n '\n \n-test_expect_success 'fsck hard errors on an invalid object type' '\n+test_expect_success 'fsck error and recovery on invalid object type' '\n \tgit init --bare garbage-type &&\n \t(\n \t\tcd garbage-type &&\n \n-\t\tgit hash-object --stdin -w -t garbage --literally err.expect <<-\\EOF &&\n \t\tfatal: invalid object type\n \t\tEOF\n \t\ttest_must_fail git fsck >out 2>err &&\n-\t\ttest_cmp err.expect err &&\n-\t\ttest_must_be_empty out\n+\t\tgrep -e \"^error\" -e \"^fatal\" err >errors &&\n+\t\ttest_line_count = 1 errors &&\n+\t\tgrep \"$garbage_blob: object is of unknown type '\"'\"'garbage'\"'\"':\" err\n \t)\n '\n \n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437650","messageId":"patch-v10-17.17-838df0a979b-20211001T091051Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com","subject":"[PATCH v10 17/17] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T09:16:53Z","receivedAt":"2021-10-01T09:17:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the error that's emitted in cases where we find a loose object\nwe parse, but which isn't at the location we expect it to be.\n\nBefore this change we'd prefix the error with a not-a-OID derived from\nthe path at which the object was found, due to an emergent behavior in\nhow we'd end up with an \"OID\" in these codepaths.\n\nNow we'll instead say what object we hashed, and what path it was\nfound at. Before this patch series e.g.:\n\n $ git hash-object --stdin -w -t blob >objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391\n $ git fsck\n error: garbage at end of loose object 'e69d[...]'\n error: unable to unpack contents of ./objects/e6/9d[...]\n error: e69d[...]: object corrupt or missing: ./objects/e6/9d[...]\n\nThere is currently some weird messaging in the edge case when the two\nare combined, i.e. because we're not explicitly passing along an error\nstate about this specific scenario from check_stream_oid() via\nread_loose_object() we'll end up printing the null OID if an object is\nof an unknown type *and* it can't be unpacked by zlib, e.g.:\n\n $ git hash-object --stdin -w -t garbage --literally >objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n $ /usr/bin/git fsck\n fatal: invalid object type\n $ ~/g/git/git fsck\n error: garbage at end of loose object '8315a83d2acc4c174aed59430f9a9c4ed926440f'\n error: unable to unpack contents of ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 8315a83d2acc4c174aed59430f9a9c4ed926440f: object corrupt or missing: ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n error: 0000000000000000000000000000000000000000: object is of unknown type 'garbage': ./objects/83/15a83d2acc4c174aed59430f9a9c4ed926440f\n [...]\n\nI think it's OK to leave that for future improvements, which would\ninvolve enum-ifying more error state as we've done with \"enum\nunpack_loose_header_result\" in preceding commits. In these\nincreasingly more obscure cases the worst that can happen is that\nwe'll get slightly nonsensical or inapplicable error messages.\n\nThere's other such potential edge cases, all of which might produce\nsome confusing messaging, but still be handled correctly as far as\npassing along errors goes. E.g. if check_object_signature() returns\nand oideq(real_oid, null_oid()) is true, which could happen if it\nreturns -1 due to the read_istream() call having failed.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fast-export.c | 2 +-\n builtin/fsck.c | 15 +++++++++++----\n builtin/index-pack.c | 2 +-\n builtin/mktag.c | 3 ++-\n cache.h | 3 ++-\n object-file.c | 21 ++++++++++-----------\n object-store.h | 1 +\n object.c | 4 ++--\n pack-check.c | 3 ++-\n t/t1006-cat-file.sh | 2 +-\n t/t1450-fsck.sh | 8 +++++---\n 11 files changed, 38 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 95e8e89e81f..8e2caf72819 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -312,7 +312,7 @@ static void export_blob(const struct object_id *oid)\n \t\tif (!buf)\n \t\t\tdie(\"could not read blob %s\", oid_to_hex(oid));\n \t\tif (check_object_signature(the_repository, oid, buf, size,\n-\t\t\t\t\t type_name(type)) < 0)\n+\t\t\t\t\t type_name(type), NULL) < 0)\n \t\t\tdie(\"oid mismatch in blob %s\", oid_to_hex(oid));\n \t\tobject = parse_object_buffer(the_repository, oid, type,\n \t\t\t\t\t size, buf, &eaten);\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 260210bf8a1..30a516da29e 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -608,6 +608,7 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tvoid *contents;\n \tint eaten;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n+\tstruct object_id real_oid = *null_oid();\n \tint err = 0;\n \n \tstrbuf_reset(&cb_data->obj_type);\n@@ -615,12 +616,18 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \toi.sizep = &size;\n \toi.typep = &type;\n \n-\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n-\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n-\t\t\t oid_to_hex(oid), path);\n+\tif (read_loose_object(path, oid, &real_oid, &contents, &oi) < 0) {\n+\t\tif (contents && !oideq(&real_oid, oid))\n+\t\t\terr = error(_(\"%s: hash-path mismatch, found at: %s\"),\n+\t\t\t\t oid_to_hex(&real_oid), path);\n+\t\telse\n+\t\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n+\t\t\t\t oid_to_hex(oid), path);\n+\t}\n \tif (type != OBJ_NONE && type < 0)\n \t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n-\t\t\t oid_to_hex(oid), cb_data->obj_type.buf, path);\n+\t\t\t oid_to_hex(&real_oid), cb_data->obj_type.buf,\n+\t\t\t path);\n \tif (err < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7ce69c087ec..15ae406e6b7 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1415,7 +1415,7 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \n \t\tif (check_object_signature(the_repository, &d->oid,\n \t\t\t\t\t data, size,\n-\t\t\t\t\t type_name(type)))\n+\t\t\t\t\t type_name(type), NULL))\n \t\t\tdie(_(\"local object %s is corrupt\"), oid_to_hex(&d->oid));\n \n \t\t/*\ndiff --git a/builtin/mktag.c b/builtin/mktag.c\nindex dddcccdd368..3b2dbbb37e6 100644\n--- a/builtin/mktag.c\n+++ b/builtin/mktag.c\n@@ -62,7 +62,8 @@ static int verify_object_in_tag(struct object_id *tagged_oid, int *tagged_type)\n \n \trepl = lookup_replace_object(the_repository, tagged_oid);\n \tret = check_object_signature(the_repository, repl,\n-\t\t\t\t buffer, size, type_name(*tagged_type));\n+\t\t\t\t buffer, size, type_name(*tagged_type),\n+\t\t\t\t NULL);\n \tfree(buffer);\n \n \treturn ret;\ndiff --git a/cache.h b/cache.h\nindex 6c5f00c82d5..e2a203073ea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1361,7 +1361,8 @@ struct object_info;\n int parse_loose_header(const char *hdr, struct object_info *oi);\n \n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *buf, unsigned long size, const char *type);\n+\t\t\t void *buf, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp);\n \n int finalize_object_file(const char *tmpfile, const char *filename);\n \ndiff --git a/object-file.c b/object-file.c\nindex fe95285f405..49561e31551 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1084,9 +1084,11 @@ void *xmmap(void *start, size_t length,\n * the streaming interface and rehash it to do the same.\n */\n int check_object_signature(struct repository *r, const struct object_id *oid,\n-\t\t\t void *map, unsigned long size, const char *type)\n+\t\t\t void *map, unsigned long size, const char *type,\n+\t\t\t struct object_id *real_oidp)\n {\n-\tstruct object_id real_oid;\n+\tstruct object_id tmp;\n+\tstruct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n \tenum object_type obj_type;\n \tstruct git_istream *st;\n \tgit_hash_ctx c;\n@@ -1094,8 +1096,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \tint hdrlen;\n \n \tif (map) {\n-\t\thash_object_file(r->hash_algo, map, size, type, &real_oid);\n-\t\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\t\thash_object_file(r->hash_algo, map, size, type, real_oid);\n+\t\treturn !oideq(oid, real_oid) ? -1 : 0;\n \t}\n \n \tst = open_istream(r, oid, &obj_type, &size, NULL);\n@@ -1120,9 +1122,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n \t\t\tbreak;\n \t\tr->hash_algo->update_fn(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->final_oid_fn(real_oid, &c);\n \tclose_istream(st);\n-\treturn !oideq(oid, &real_oid) ? -1 : 0;\n+\treturn !oideq(oid, real_oid) ? -1 : 0;\n }\n \n int git_open_cloexec(const char *name, int flags)\n@@ -2572,6 +2574,7 @@ static int check_stream_oid(git_zstream *stream,\n \n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi)\n {\n@@ -2582,8 +2585,6 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n-\t*contents = NULL;\n-\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n@@ -2613,9 +2614,7 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf)) {\n-\t\t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n-\t\t\t oid_to_hex(expected_oid));\n+\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n \t\t\tfree(*contents);\n \t\t\tgoto out;\n \t\t}\ndiff --git a/object-store.h b/object-store.h\nindex c90c41a07f7..17b072e5a19 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -253,6 +253,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime);\n */\n int read_loose_object(const char *path,\n \t\t const struct object_id *expected_oid,\n+\t\t struct object_id *real_oid,\n \t\t void **contents,\n \t\t struct object_info *oi);\n \ndiff --git a/object.c b/object.c\nindex 4e85955a941..23a24e678a8 100644\n--- a/object.c\n+++ b/object.c\n@@ -279,7 +279,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tif ((obj && obj->type == OBJ_BLOB && repo_has_object_file(r, oid)) ||\n \t (!obj && repo_has_object_file(r, oid) &&\n \t oid_object_info(r, oid, NULL) == OBJ_BLOB)) {\n-\t\tif (check_object_signature(r, repl, NULL, 0, NULL) < 0) {\n+\t\tif (check_object_signature(r, repl, NULL, 0, NULL, NULL) < 0) {\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n \t\t\treturn NULL;\n \t\t}\n@@ -290,7 +290,7 @@ struct object *parse_object(struct repository *r, const struct object_id *oid)\n \tbuffer = repo_read_object_file(r, oid, &type, &size);\n \tif (buffer) {\n \t\tif (check_object_signature(r, repl, buffer, size,\n-\t\t\t\t\t type_name(type)) < 0) {\n+\t\t\t\t\t type_name(type), NULL) < 0) {\n \t\t\tfree(buffer);\n \t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(repl));\n \t\t\treturn NULL;\ndiff --git a/pack-check.c b/pack-check.c\nindex c8e560d71ab..3f418e3a6af 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -142,7 +142,8 @@ static int verify_packfile(struct repository *r,\n \t\t\terr = error(\"cannot unpack %s from %s at offset %\"PRIuMAX\"\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name,\n \t\t\t\t (uintmax_t)entries[i].offset);\n-\t\telse if (check_object_signature(r, &oid, data, size, type_name(type)))\n+\t\telse if (check_object_signature(r, &oid, data, size,\n+\t\t\t\t\t\ttype_name(type), NULL))\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t oid_to_hex(&oid), p->pack_name);\n \t\telse if (fn) {\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex a5e7401af8b..0f52ca9cc82 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -512,7 +512,7 @@ test_expect_success 'cat-file -t and -s on corrupt loose object' '\n \t\t# Swap the two to corrupt the repository\n \t\tmv -f \"$other_path\" \"$empty_path\" &&\n \t\ttest_must_fail git fsck 2>err.fsck &&\n-\t\tgrep \"hash mismatch\" err.fsck &&\n+\t\tgrep \"hash-path mismatch\" err.fsck &&\n \n \t\t# confirm that cat-file is reading the new swapped-in\n \t\t# blob...\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex faf0e98847b..6337236fd82 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -54,6 +54,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tcd hash-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -65,7 +66,7 @@ test_expect_success 'object with hash mismatch' '\n \t\tgit update-ref refs/heads/bogus $cmt &&\n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"$oid.*corrupt\" out\n+\t\tgrep \"$oldoid: hash-path mismatch, found at: .*$new\" out\n \t)\n '\n \n@@ -75,6 +76,7 @@ test_expect_success 'object with hash and type mismatch' '\n \t\tcd hash-type-mismatch &&\n \n \t\toid=$(echo blob | git hash-object -w --stdin -t garbage --literally) &&\n+\t\toldoid=$oid &&\n \t\told=$(test_oid_to_path \"$oid\") &&\n \t\tnew=$(dirname $old)/$(test_oid ff_2) &&\n \t\toid=\"$(dirname $new)$(basename $new)\" &&\n@@ -87,8 +89,8 @@ test_expect_success 'object with hash and type mismatch' '\n \n \n \t\ttest_must_fail git fsck 2>out &&\n-\t\tgrep \"^error: hash mismatch for \" out &&\n-\t\tgrep \"^error: $oid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n+\t\tgrep \"^error: $oldoid: hash-path mismatch, found at: .*$new\" out &&\n+\t\tgrep \"^error: $oldoid: object is of unknown type '\"'\"'garbage'\"'\"'\" out\n \t)\n '\n \n-- \n2.33.0.1375.g5eed55aa1b5\n\n"},{"id":"437740","messageId":"xmqqk0iw2y0m.fsf@gitster.g","threadId":"55404","inReplyTo":"patch-v10-17.17-838df0a979b-20211001T091051Z-avarab@gmail.com","subject":"Re: [PATCH v10 17/17] fsck: report invalid object type-path combinations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-01T22:14:33Z","receivedAt":"2021-10-01T22:14:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 260210bf8a1..30a516da29e 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -615,12 +616,18 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \toi.sizep = &size;\n> \toi.typep = &type;\n> \n> -\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n> -\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n> -\t\t\t oid_to_hex(oid), path);\n> +\tif (read_loose_object(path, oid, &real_oid, &contents, &oi) < 0) {\n> +\t\tif (contents && !oideq(&real_oid, oid))\n> +\t\t\terr = error(_(\"%s: hash-path mismatch, found at: %s\"),\n> +\t\t\t\t oid_to_hex(&real_oid), path);\n> +\t\telse\n> +\t\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n> +\t\t\t\t oid_to_hex(oid), path);\n> +\t}\n> \tif (type != OBJ_NONE && type < 0)\n> \t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n> -\t\t\t oid_to_hex(oid), cb_data->obj_type.buf, path);\n> +\t\t\t oid_to_hex(&real_oid), cb_data->obj_type.buf,\n> +\t\t\t path);\n> \tif (err < 0) {\n> \t\terrors_found |= ERROR_OBJECT;\n> \t\treturn 0; /* keep checking other objects */\n\nWhen we say \"hash-path mismatch\", we would have non-null contents,\npresumably obtained from read_loose_object(). err is made negative\nwhen we give that messge, and we come here to return. Did we forget\nto free \"contents\" in that case?\n\n"},{"id":"437744","messageId":"87sfxkpdqp.fsf@evledraar.gmail.com","threadId":"55404","inReplyTo":"xmqqk0iw2y0m.fsf@gitster.g","subject":"Re: [PATCH v10 17/17] fsck: report invalid object type-path combinations","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-01T22:33:40Z","receivedAt":"2021-10-01T22:44:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 01 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> diff --git a/builtin/fsck.c b/builtin/fsck.c\n>> index 260210bf8a1..30a516da29e 100644\n>> --- a/builtin/fsck.c\n>> +++ b/builtin/fsck.c\n>> @@ -615,12 +616,18 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n>> \toi.sizep = &size;\n>> \toi.typep = &type;\n>> \n>> -\tif (read_loose_object(path, oid, &contents, &oi) < 0)\n>> -\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n>> -\t\t\t oid_to_hex(oid), path);\n>> +\tif (read_loose_object(path, oid, &real_oid, &contents, &oi) < 0) {\n>> +\t\tif (contents && !oideq(&real_oid, oid))\n>> +\t\t\terr = error(_(\"%s: hash-path mismatch, found at: %s\"),\n>> +\t\t\t\t oid_to_hex(&real_oid), path);\n>> +\t\telse\n>> +\t\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n>> +\t\t\t\t oid_to_hex(oid), path);\n>> +\t}\n>> \tif (type != OBJ_NONE && type < 0)\n>> \t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n>> -\t\t\t oid_to_hex(oid), cb_data->obj_type.buf, path);\n>> +\t\t\t oid_to_hex(&real_oid), cb_data->obj_type.buf,\n>> +\t\t\t path);\n>> \tif (err < 0) {\n>> \t\terrors_found |= ERROR_OBJECT;\n>> \t\treturn 0; /* keep checking other objects */\n>\n> When we say \"hash-path mismatch\", we would have non-null contents,\n> presumably obtained from read_loose_object(). err is made negative\n> when we give that messge, and we come here to return. Did we forget\n> to free \"contents\" in that case?\n\nNo, e.g. the \"cat-file -t and -s on corrupt loose object\" test added in\nthis series doesn't error with SANITIZE=leak.\n\nThis is because as we go through read_loose_object() we'll make our way\nto unpack_loose_rest(), which will return that malloc'd buffer. So we\nwould leak it if we returned after that.\n\nExcept that in read_loose_object() we'll go on to call\ncheck_object_signature() right afterwards. The expecte OID is whatever\nwe inferred from the FS path, and the OID we saw is what we get from\nhashing. That call will return non-zero, and we'll free() the\ncontents. The buffer isn't NULL'd, but we can't use it.\n\nThis is all behavior that pre-dates this series. I think it's a bit\nstupid, and we should arguably do better about data recovery here, as\nalluded to at the end of the commit message.\n\nI.e. ideally we can use the information that we know we wanted OID A,\nwho cares if we found it at path B? It hashes to A and completes the\ngraph! Let's just re-write it to A. Or maybe it's not worth it. Or we'd\nwant to optionally log the content we *do* find on such failures,\ne.g. maybe the content is partial or whatever. I had some WIP work on\ntop of this that did that, e.g. to recover in cases where you append\ngarbage data at the end of an object (in which case we *do* have the\ncontent and can recover, we just need to stop reading at that byte once\nour OID matches, and re-write it out again).\n\nBut anyway, it doesn't work that way now, and this doesn't leak memory,\nor as far as I can tell do the wrong thing in these various edge cases,\nbecause \"content is bad\" is always synonymous with read_loose_object()\nitself calling free().\n\nThanks a lot for the careful checking!\n"},{"id":"440897","messageId":"20211111030302.75694-1-hanxin.hx@alibaba-inc.com","threadId":"55404","inReplyTo":"patch-v10-17.17-838df0a979b-20211001T091051Z-avarab@gmail.com","subject":"Re: [PATCH v2] receive-pack: not receive pack file with large object","fromName":"Han Xin","fromEmail":"chiyutianyi@gmail.com","sentAt":"2021-11-11T03:03:02Z","receivedAt":"2021-11-11T03:03:09Z","isPatch":true,"sender":{"key":"chiyutianyi@gmail.com","avatar":null},"body":"Ævar Arnfjörð Bjarmason   writes:\n...\n> diff --git a/object-file.c b/object-file.c\n> index fe95285f405..49561e31551 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1084,9 +1084,11 @@ void *xmmap(void *start, size_t length,\n> * the streaming interface and rehash it to do the same.\n> */\n> int check_object_signature(struct repository *r, const struct object_id *oid,\n> - void *map, unsigned long size, const char *type)\n> + void *map, unsigned long size, const char *type,\n> + struct object_id *real_oidp)\n> {\n> - struct object_id real_oid;\n> + struct object_id tmp;\n> + struct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n> enum object_type obj_type;\n> struct git_istream *st;\n> git_hash_ctx c;\n> @@ -1094,8 +1096,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n> int hdrlen;\n>\n> if (map) {\n> - hash_object_file(r->hash_algo, map, size, type, &real_oid);\n> - return !oideq(oid, &real_oid) ? -1 : 0;\n> + hash_object_file(r->hash_algo, map, size, type, real_oid);\n> + return !oideq(oid, real_oid) ? -1 : 0;\n> }\n>\n> st = open_istream(r, oid, &obj_type, &size, NULL);\n> @@ -1120,9 +1122,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n> break;\n> r->hash_algo->update_fn(&c, buf, readlen);\n> }\n> - r->hash_algo->final_oid_fn(&real_oid, &c);\n> + r->hash_algo->final_oid_fn(real_oid, &c);\n> close_istream(st);\n> - return !oideq(oid, &real_oid) ? -1 : 0;\n> + return !oideq(oid, real_oid) ? -1 : 0;\n> }\n>\n> int git_open_cloexec(const char *name, int flags)\n> @@ -2572,6 +2574,7 @@ static int check_stream_oid(git_zstream *stream,\n>\n> int read_loose_object(const char *path,\n> const struct object_id *expected_oid,\n> + struct object_id *real_oid,\n> void **contents,\n> struct object_info *oi)\n> {\n> @@ -2582,8 +2585,6 @@ int read_loose_object(const char *path,\n> char hdr[MAX_HEADER_LEN];\n> unsigned long *size = oi->sizep;\n>\n> - *contents = NULL;\n> -\n\nDeleting \"*contents = NULL;\" here will cause a memory free error.\nWhen reading a large loose blob ( large than big_file_threshold), it will enter the following block and *content will not be set:\n\n\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n\t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n\t\t\tgoto out;\n\t} else {\n\t\t...\n\t}\n\n\nThis test case can illustrate this problem:\n\ntest_expect_success 'fsck large loose blob' '\n\tblob=$(echo large | git hash-object -w --stdin) &&\n\tgit -c core.bigfilethreshold=4 fsck\n'\n\ngit(73697,0x1198f1e00) malloc: *** error for object 0x36: pointer being freed was not allocated\ngit(73697,0x1198f1e00) malloc: *** set a breakpoint in malloc_error_break to debug\n./test-lib.sh: line 947: 73697 Abort trap: 6 git -c core.bigfilethreshold=4 fsck\n\n> map = map_loose_object_1(the_repository, path, NULL, &mapsize);\n> if (!map) {\n> error_errno(_(\"unable to mmap %s\"), path);\n> @@ -2613,9 +2614,7 @@ int read_loose_object(const char *path,\n> goto out;\n> }\n> if (check_object_signature(the_repository, expected_oid,\n> - *contents, *size, oi->type_name->buf)) {\n> - error(_(\"hash mismatch for %s (expected %s)\"), path,\n> - oid_to_hex(expected_oid));\n> + *contents, *size, oi->type_name->buf, real_oid)) {\n> free(*contents);\n> goto out;\n> }\n...\n"},{"id":"440898","messageId":"20211111030532.75910-1-hanxin.hx@alibaba-inc.com","threadId":"55404","inReplyTo":"patch-v10-17.17-838df0a979b-20211001T091051Z-avarab@gmail.com","subject":"Re: [PATCH v10 17/17] fsck: report invalid object type-path combinations","fromName":"Han Xin","fromEmail":"chiyutianyi@gmail.com","sentAt":"2021-11-11T03:05:32Z","receivedAt":"2021-11-11T03:05:42Z","isPatch":true,"sender":{"key":"chiyutianyi@gmail.com","avatar":null},"body":"Ævar Arnfjörð Bjarmason   writes:\n...\n> diff --git a/object-file.c b/object-file.c\n> index fe95285f405..49561e31551 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1084,9 +1084,11 @@ void *xmmap(void *start, size_t length,\n> * the streaming interface and rehash it to do the same.\n> */\n> int check_object_signature(struct repository *r, const struct object_id *oid,\n> - void *map, unsigned long size, const char *type)\n> + void *map, unsigned long size, const char *type,\n> + struct object_id *real_oidp)\n> {\n> - struct object_id real_oid;\n> + struct object_id tmp;\n> + struct object_id *real_oid = real_oidp ? real_oidp : &tmp;\n> enum object_type obj_type;\n> struct git_istream *st;\n> git_hash_ctx c;\n> @@ -1094,8 +1096,8 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n> int hdrlen;\n>\n> if (map) {\n> - hash_object_file(r->hash_algo, map, size, type, &real_oid);\n> - return !oideq(oid, &real_oid) ? -1 : 0;\n> + hash_object_file(r->hash_algo, map, size, type, real_oid);\n> + return !oideq(oid, real_oid) ? -1 : 0;\n> }\n>\n> st = open_istream(r, oid, &obj_type, &size, NULL);\n> @@ -1120,9 +1122,9 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n> break;\n> r->hash_algo->update_fn(&c, buf, readlen);\n> }\n> - r->hash_algo->final_oid_fn(&real_oid, &c);\n> + r->hash_algo->final_oid_fn(real_oid, &c);\n> close_istream(st);\n> - return !oideq(oid, &real_oid) ? -1 : 0;\n> + return !oideq(oid, real_oid) ? -1 : 0;\n> }\n>\n> int git_open_cloexec(const char *name, int flags)\n> @@ -2572,6 +2574,7 @@ static int check_stream_oid(git_zstream *stream,\n>\n> int read_loose_object(const char *path,\n> const struct object_id *expected_oid,\n> + struct object_id *real_oid,\n> void **contents,\n> struct object_info *oi)\n> {\n> @@ -2582,8 +2585,6 @@ int read_loose_object(const char *path,\n> char hdr[MAX_HEADER_LEN];\n> unsigned long *size = oi->sizep;\n>\n> - *contents = NULL;\n> -\n\nDeleting \"*contents = NULL;\" here will cause a memory free error.\nWhen reading a large loose blob ( large than big_file_threshold), it will enter the following block and *content will not be set:\n\n\tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n\t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n\t\t\tgoto out;\n\t} else {\n\t\t...\n\t}\n\n\nThis test case can illustrate this problem:\n\ntest_expect_success 'fsck large loose blob' '\n\tblob=$(echo large | git hash-object -w --stdin) &&\n\tgit -c core.bigfilethreshold=4 fsck\n'\n\ngit(73697,0x1198f1e00) malloc: *** error for object 0x36: pointer being freed was not allocated\ngit(73697,0x1198f1e00) malloc: *** set a breakpoint in malloc_error_break to debug\n./test-lib.sh: line 947: 73697 Abort trap: 6 git -c core.bigfilethreshold=4 fsck\n\n> map = map_loose_object_1(the_repository, path, NULL, &mapsize);\n> if (!map) {\n> error_errno(_(\"unable to mmap %s\"), path);\n> @@ -2613,9 +2614,7 @@ int read_loose_object(const char *path,\n> goto out;\n> }\n> if (check_object_signature(the_repository, expected_oid,\n> - *contents, *size, oi->type_name->buf)) {\n> - error(_(\"hash mismatch for %s (expected %s)\"), path,\n> - oid_to_hex(expected_oid));\n> + *contents, *size, oi->type_name->buf, real_oid)) {\n> free(*contents);\n> goto out;\n> }\n...\n"},{"id":"440899","messageId":"cover-0.2-00000000000-20211111T051800Z-avarab@gmail.com","threadId":"55404","inReplyTo":"20211111030532.75910-1-hanxin.hx@alibaba-inc.com","subject":"[PATCH 0/2] v2.34.0-rc2 regression: free() of uninitialized in ab/fsck-unexpected-type","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-11T05:18:54Z","receivedAt":"2021-11-11T05:19:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Thu, Nov 11 2021, Han Xin wrote:\n> [...]\n> Deleting \"*contents = NULL;\" here will cause a memory free error.\n> When reading a large loose blob ( large than big_file_threshold), it will enter the following block and *content will not be set:\n>\n> if (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n> if (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n> goto out;\n> } else {\n> ...\n> }\n>\n>\n> This test case can illustrate this problem:\n>\n> test_expect_success 'fsck large loose blob' '\n> blob=$(echo large | git hash-object -w --stdin) &&\n> git -c core.bigfilethreshold=4 fsck\n> '\n>\n> git(73697,0x1198f1e00) malloc: *** error for object 0x36: pointer being freed was not allocated\n> git(73697,0x1198f1e00) malloc: *** set a breakpoint in malloc_error_break to debug\n> ./test-lib.sh: line 947: 73697 Abort trap: 6 git -c core.bigfilethreshold=4 fsck\n\nThanks a lot for the detailed report and test case. It looks like I've\ngot the dubious honor of most scary caught-by-rc bug so far.\n\nThis series:\n\nÆvar Arnfjörð Bjarmason (2):\n object-file: fix SEGV on free() regression in v2.34.0-rc2\n\nThis is the most minimal fix for this issue. So Junio, if you'd like\nto just pick this up for v2.34.0 you can peel just 1/2 off...\n\n object-file: free(*contents) only in read_loose_object() caller\n\n... a fix for a related issue. In ab/fsck-unexpected-type we stopped\ndie()-ing in the object-name.c, so per SANITIZE=leak's accounting we\nintroduced a memory leak with the same variable we dealt with in 1/2.\n\nBut IMO more importantly by changing this code so that only one\nfunction owns the free()-ing it's much easier to reason about this\ncode.\n\n builtin/fsck.c | 3 ++-\n object-file.c | 5 ++---\n t/t1050-large.sh | 8 ++++++++\n 3 files changed, 12 insertions(+), 4 deletions(-)\n\n-- \n2.34.0.rc2.795.g926201d1cc8\n\n"},{"id":"440900","messageId":"patch-1.2-811242178e4-20211111T051800Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.2-00000000000-20211111T051800Z-avarab@gmail.com","subject":"[PATCH 1/2] object-file: fix SEGV on free() regression in v2.34.0-rc2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-11T05:18:55Z","receivedAt":"2021-11-11T05:19:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a regression introduced in my 96e41f58fe1 (fsck: report invalid\nobject type-path combinations, 2021-10-01). When fsck-ing blobs larger\nthan core.bigFileThreshold we'd free() a pointer to uninitialized\nmemory.\n\nThis issue would have been caught by SANITIZE=address, but since it\ninvolves core.bigFileThreshold none of the existing tests in our test\nsuite covered it.\n\nRunning them with the \"big_file_threshold\" in \"environment.c\" changed\nto say \"6\" would have shown this failure, but let's add a dedicated\ntest for this scenario based on Han Xin's report[1].\n\nIt would be a good follow-up change to add a GIT_TEST_* mode to run\nall the tests with a low core.bigFileThreshold threshold.\n\nCurrently a lot of them fail (but none due to SANITIZE=address)\nbecause they make implicit assumptions about the current hardcoded\nsetting of core.bigFileThreshold.\n\nAround half the failures are due to us assuming that files larger than\nthat are binary, see 6bf3b813486 (diff --stat: mark any file larger\nthan core.bigfilethreshold binary, 2014-08-16) and the comment added\nin 12426e114b2 (diff: do not short-cut CHECK_SIZE_ONLY check in\ndiff_populate_filespec(), 2017-03-01). The rest seem to all be\npack/loose-related, i.e. they're assuming that something ends up as a\nloose object or in a pack.\n\nThe bug was introduced between v9 and v10[2] of the fsck series merged\nin 061a21d36d8 (Merge branch 'ab/fsck-unexpected-type', 2021-10-25).\n\n1. https://lore.kernel.org/git/20211111030302.75694-1-hanxin.hx@alibaba-inc.com/\n2. https://lore.kernel.org/git/cover-v10-00.17-00000000000-20211001T091051Z-avarab@gmail.com/\n\nReported-by: Han Xin \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n object-file.c | 2 ++\n t/t1050-large.sh | 8 ++++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/object-file.c b/object-file.c\nindex 02b79702748..ac476653a06 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2528,6 +2528,8 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n+\t*contents = NULL;\n+\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 4bab6a513c5..6bc1d76fb10 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -17,6 +17,14 @@ test_expect_success setup '\n \texport GIT_ALLOC_LIMIT\n '\n \n+test_expect_success 'enter \"large\" codepath, with small core.bigFileThreshold' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init --bare repo &&\n+\techo large | git -C repo hash-object -w --stdin &&\n+\tgit -C repo -c core.bigfilethreshold=4 fsck\n+'\n+\n # add a large file with different settings\n while read expect config\n do\n-- \n2.34.0.rc2.795.g926201d1cc8\n\n"},{"id":"440901","messageId":"patch-2.2-e2a5468e2e0-20211111T051800Z-avarab@gmail.com","threadId":"55404","inReplyTo":"cover-0.2-00000000000-20211111T051800Z-avarab@gmail.com","subject":"[PATCH 2/2] object-file: free(*contents) only in read_loose_object() caller","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-11T05:18:56Z","receivedAt":"2021-11-11T05:19:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In the preceding commit a free() of uninitialized memory regression in\n96e41f58fe1 (fsck: report invalid object type-path combinations,\n2021-10-01) was fixed, but we'd still have an issue with leaking\nmemory from fsck_loose(). Let's fix that issue too.\n\nThat issue was introduced in my 31deb28f5e0 (fsck: don't hard die on\ninvalid object types, 2021-10-01). It can be reproduced under\nSANITIZE=leak with the test I added in 093fffdfbec (fsck tests: add\ntest for fsck-ing an unknown type, 2021-10-01):\n\n ./t1450-fsck.sh --run=84 -vixd\n\nIn some sense it's not a problem, we lost the same amount of memory in\nterms of things malloc'd and not free'd. It just moved from the \"still\nreachable\" to \"definitely lost\" column in valgrind(1) nomenclature[1],\nsince we'd have die()'d before.\n\nBut now that we don't hard die() anymore in the library let's properly\nfree() it. Doing so makes this code much easier to follow, since we'll\nnow have one function owning the freeing of the \"contents\" variable,\nnot two.\n\nFor context on that memory management pattern the read_loose_object()\nfunction was added in f6371f92104 (sha1_file: add read_loose_object()\nfunction, 2017-01-13) and subsequently used in c68b489e564 (fsck:\nparse loose object paths directly, 2017-01-13). The pattern of it\nbeing the task of both sides to free() the memory has been there in\nthis form since its inception.\n\n1. https://valgrind.org/docs/manual/mc-manual.html#mc-manual.leaks\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/fsck.c | 3 ++-\n object-file.c | 7 ++-----\n 2 files changed, 4 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d87c28a1cc4..27b9e78094d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -605,7 +605,7 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \tstruct object *obj;\n \tenum object_type type = OBJ_NONE;\n \tunsigned long size;\n-\tvoid *contents;\n+\tvoid *contents = NULL;\n \tint eaten;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tstruct object_id real_oid = *null_oid();\n@@ -630,6 +630,7 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \t\t\t path);\n \tif (err < 0) {\n \t\terrors_found |= ERROR_OBJECT;\n+\t\tfree(contents);\n \t\treturn 0; /* keep checking other objects */\n \t}\n \ndiff --git a/object-file.c b/object-file.c\nindex ac476653a06..c3d866a287e 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2528,8 +2528,6 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n-\t*contents = NULL;\n-\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n@@ -2559,10 +2557,9 @@ int read_loose_object(const char *path,\n \t\t\tgoto out;\n \t\t}\n \t\tif (check_object_signature(the_repository, expected_oid,\n-\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n-\t\t\tfree(*contents);\n+\t\t\t\t\t *contents, *size,\n+\t\t\t\t\t oi->type_name->buf, real_oid))\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n \tret = 0; /* everything checks out */\n-- \n2.34.0.rc2.795.g926201d1cc8\n\n"},{"id":"440913","messageId":"YY00LfI8vHKMaxAs@coredump.intra.peff.net","threadId":"55404","inReplyTo":"patch-1.2-811242178e4-20211111T051800Z-avarab@gmail.com","subject":"Re: [PATCH 1/2] object-file: fix SEGV on free() regression in v2.34.0-rc2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-11-11T15:18:05Z","receivedAt":"2021-11-11T15:18:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 11, 2021 at 06:18:55AM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> diff --git a/object-file.c b/object-file.c\n> index 02b79702748..ac476653a06 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2528,6 +2528,8 @@ int read_loose_object(const char *path,\n> \tchar hdr[MAX_HEADER_LEN];\n> \tunsigned long *size = oi->sizep;\n> \n> +\t*contents = NULL;\n> +\n> \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n> \tif (!map) {\n> \t\terror_errno(_(\"unable to mmap %s\"), path);\n\nOK, I agree this fixes the segfault, and is the minimal fix.\n\nI do find the fact that fsck_loose() looks at \"contents\" after\nread_loose_object() returns an error to be a bit questionable. It's a\nrecipe for confusion about what has happened, and who is supposed to\nfree what. Your v2 addresses the leak, but by just shifting more burden\nto the caller. There's only one caller, so it's not too bad, but for a\npublic function, read_loose_object() has a lot of sharp edges.\n\nPlus I think it fails to work as intended for streaming blobs (we do not\nfill in \"contents\" at all in that case, so we can never say \"hash-path\nmismatch\").\n\nI understand you're trying to catch the case of \"we actually opened the\nfile and computed the sha1 of its contents\" from cases where we didn't\nget that far. But since you initialize real_oid, it seems like it would\nbe better to see if anything was written to that.\n\nI.e., something like:\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex d87c28a1cc..8f156ed9cd 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -617,18 +617,20 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \toi.typep = &type;\n \n \tif (read_loose_object(path, oid, &real_oid, &contents, &oi) < 0) {\n-\t\tif (contents && !oideq(&real_oid, oid))\n+\t\tif (!is_null_oid(&real_oid) && !oideq(&real_oid, oid))\n \t\t\terr = error(_(\"%s: hash-path mismatch, found at: %s\"),\n \t\t\t\t oid_to_hex(&real_oid), path);\n \t\telse\n \t\t\terr = error(_(\"%s: object corrupt or missing: %s\"),\n \t\t\t\t oid_to_hex(oid), path);\n+\t\terrors_found |= ERROR_OBJECT;\n+\t\treturn 0; /* keep checking other objects */\n \t}\n-\tif (type != OBJ_NONE && type < 0)\n+\tif (type != OBJ_NONE && type < 0) {\n \t\terr = error(_(\"%s: object is of unknown type '%s': %s\"),\n \t\t\t oid_to_hex(&real_oid), cb_data->obj_type.buf,\n \t\t\t path);\n-\tif (err < 0) {\n+\t\tfree(contents);\n \t\terrors_found |= ERROR_OBJECT;\n \t\treturn 0; /* keep checking other objects */\n \t}\n\n(the \"err\" variable is now superfluous, but I left it in to keep the\ndiff smaller). And then it would be safe to just set \"contents\" in\nread_loose_object() when we need it:\n\ndiff --git a/object-file.c b/object-file.c\nindex ac476653a0..5e8ff94fd4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2528,8 +2528,6 @@ int read_loose_object(const char *path,\n \tchar hdr[MAX_HEADER_LEN];\n \tunsigned long *size = oi->sizep;\n \n-\t*contents = NULL;\n-\n \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n@@ -2549,6 +2547,7 @@ int read_loose_object(const char *path,\n \t}\n \n \tif (*oi->typep == OBJ_BLOB && *size > big_file_threshold) {\n+\t\t*contents = NULL;\n \t\tif (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0)\n \t\t\tgoto out;\n \t} else {\n\nThat doesn't fix the hash-path mismatch problem for streaming, but it\nsets us up to do so, if check_stream_oid() returned the real_oid it\ncomputed.\n\nAll of this is much too large for an -rc fix, so we should take your\npatch as-is. These are just thoughts I had while trying to figure out\nif there were other problems caused by that same commit.\n\n-Peff\n"},{"id":"440928","messageId":"xmqqzgqaleux.fsf@gitster.g","threadId":"55404","inReplyTo":"20211111030302.75694-1-hanxin.hx@alibaba-inc.com","subject":"Re: [PATCH v2] receive-pack: not receive pack file with large object","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-11T18:35:50Z","receivedAt":"2021-11-11T18:35:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han Xin writes:\n\n>> @@ -2582,8 +2585,6 @@ int read_loose_object(const char *path,\n>> char hdr[MAX_HEADER_LEN];\n>> unsigned long *size = oi->sizep;\n>>\n>> - *contents = NULL;\n>> -\n>\n> Deleting \"*contents = NULL;\" here will cause a memory free error.\n\nGood find. I see in the discussion downthread we have a band-aid\nfor this issue already ;-)\n\nThanks.\n"},{"id":"440929","messageId":"xmqqtugilel1.fsf@gitster.g","threadId":"55404","inReplyTo":"patch-1.2-811242178e4-20211111T051800Z-avarab@gmail.com","subject":"Re: [PATCH 1/2] object-file: fix SEGV on free() regression in v2.34.0-rc2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-11T18:41:46Z","receivedAt":"2021-11-11T18:41:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Fix a regression introduced in my 96e41f58fe1 (fsck: report invalid\n> object type-path combinations, 2021-10-01). When fsck-ing blobs larger\n> than core.bigFileThreshold we'd free() a pointer to uninitialized\n> memory.\n\ns/d we'd/d, we'd/; no need to resend.\n\n> This issue would have been caught by SANITIZE=address, but since it\n> involves core.bigFileThreshold none of the existing tests in our test\n> suite covered it.\n\ns/d none/d, none/; likewise.\n\n> Running them with the \"big_file_threshold\" in \"environment.c\" changed\n> to say \"6\" would have shown this failure, but let's add a dedicated\n> test for this scenario based on Han Xin's report[1].\n\nYeah, it is a good and focused test. \n\nBy the way, I do not think changing big_file_threshold _blindly_ to\nsmaller values, instead of in a focused test like this, is a good\nidea in general. Some tests check if a file with a normal size that\nis smaller than the threshold correctly is treated as a binary file,\nand lowering threshold for them without understanding what they are\nmeant to test would trigger a \"bug\" that is not a bug at all, for\nexample.\n\n> It would be a good follow-up change to add a GIT_TEST_* mode to run\n> all the tests with a low core.bigFileThreshold threshold.\n\nSo, no, please don't do that.\n\n> object-file.c | 2 ++\n> t/t1050-large.sh | 8 ++++++++\n> 2 files changed, 10 insertions(+)\n>\n> diff --git a/object-file.c b/object-file.c\n> index 02b79702748..ac476653a06 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2528,6 +2528,8 @@ int read_loose_object(const char *path,\n> \tchar hdr[MAX_HEADER_LEN];\n> \tunsigned long *size = oi->sizep;\n> \n> +\t*contents = NULL;\n> +\n> \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n> \tif (!map) {\n> \t\terror_errno(_(\"unable to mmap %s\"), path);\n> diff --git a/t/t1050-large.sh b/t/t1050-large.sh\n> index 4bab6a513c5..6bc1d76fb10 100755\n> --- a/t/t1050-large.sh\n> +++ b/t/t1050-large.sh\n> @@ -17,6 +17,14 @@ test_expect_success setup '\n> \texport GIT_ALLOC_LIMIT\n> '\n> \n> +test_expect_success 'enter \"large\" codepath, with small core.bigFileThreshold' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\n> +\tgit init --bare repo &&\n> +\techo large | git -C repo hash-object -w --stdin &&\n> +\tgit -C repo -c core.bigfilethreshold=4 fsck\n> +'\n> +\n> # add a large file with different settings\n> while read expect config\n> do\n"},{"id":"440930","messageId":"xmqqk0hele05.fsf@gitster.g","threadId":"55404","inReplyTo":"patch-2.2-e2a5468e2e0-20211111T051800Z-avarab@gmail.com","subject":"Re: [PATCH 2/2] object-file: free(*contents) only in read_loose_object() caller","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-11T18:54:18Z","receivedAt":"2021-11-11T18:54:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index d87c28a1cc4..27b9e78094d 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -605,7 +605,7 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \tstruct object *obj;\n> \tenum object_type type = OBJ_NONE;\n> \tunsigned long size;\n> -\tvoid *contents;\n> +\tvoid *contents = NULL;\n> \tint eaten;\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> \tstruct object_id real_oid = *null_oid();\n> @@ -630,6 +630,7 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n> \t\t\t path);\n> \tif (err < 0) {\n> \t\terrors_found |= ERROR_OBJECT;\n> +\t\tfree(contents);\n> \t\treturn 0; /* keep checking other objects */\n> \t}\n> \n> diff --git a/object-file.c b/object-file.c\n> index ac476653a06..c3d866a287e 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2528,8 +2528,6 @@ int read_loose_object(const char *path,\n> \tchar hdr[MAX_HEADER_LEN];\n> \tunsigned long *size = oi->sizep;\n> \n> -\t*contents = NULL;\n> -\n> \tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n> \tif (!map) {\n> \t\terror_errno(_(\"unable to mmap %s\"), path);\n> @@ -2559,10 +2557,9 @@ int read_loose_object(const char *path,\n> \t\t\tgoto out;\n> \t\t}\n> \t\tif (check_object_signature(the_repository, expected_oid,\n> -\t\t\t\t\t *contents, *size, oi->type_name->buf, real_oid)) {\n> -\t\t\tfree(*contents);\n> +\t\t\t\t\t *contents, *size,\n> +\t\t\t\t\t oi->type_name->buf, real_oid))\n> \t\t\tgoto out;\n> -\t\t}\n> \t}\n\nYeah, I have to say that read_loose_object() that frees *contents\nwithout clearing *contents to NULL only because it wants to signal\nif the failure comes from check_object_signature() step is quite\nugly. Making the caller responsible for freeing (in other words,\nwhen caller's *contents is non-NULL after function returns, it\nalways has a valid piece of memory to be freed) makes the contract\neasier to explain.\n"},{"id":"441060","messageId":"211113.86y25sto9s.gmgdl@evledraar.gmail.com","threadId":"55404","inReplyTo":"xmqqtugilel1.fsf@gitster.g","subject":"Re: [PATCH 1/2] object-file: fix SEGV on free() regression in v2.34.0-rc2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-13T09:00:37Z","receivedAt":"2021-11-13T09:09:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 11 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> Fix a regression introduced in my 96e41f58fe1 (fsck: report invalid\n>> object type-path combinations, 2021-10-01). When fsck-ing blobs larger\n>> than core.bigFileThreshold we'd free() a pointer to uninitialized\n>> memory.\n>\n> s/d we'd/d, we'd/; no need to resend.\n>\n>> This issue would have been caught by SANITIZE=address, but since it\n>> involves core.bigFileThreshold none of the existing tests in our test\n>> suite covered it.\n>\n> s/d none/d, none/; likewise.\n>\n>> Running them with the \"big_file_threshold\" in \"environment.c\" changed\n>> to say \"6\" would have shown this failure, but let's add a dedicated\n>> test for this scenario based on Han Xin's report[1].\n>\n> Yeah, it is a good and focused test. \n>\n> By the way, I do not think changing big_file_threshold _blindly_ to\n> smaller values, instead of in a focused test like this, is a good\n> idea in general. Some tests check if a file with a normal size that\n> is smaller than the threshold correctly is treated as a binary file,\n> and lowering threshold for them without understanding what they are\n> meant to test would trigger a \"bug\" that is not a bug at all, for\n> example.\n>\n>> It would be a good follow-up change to add a GIT_TEST_* mode to run\n>> all the tests with a low core.bigFileThreshold threshold.\n>\n> So, no, please don't do that.\n\nYes it's probably not worth it, and I've got enough dragons to slay as\nit is.\n\nI took the commentary you added in 12426e114b2 (diff: do not short-cut\nCHECK_SIZE_ONLY check in diff_populate_filespec(), 2017-03-01) as a\nsuggestion that we might be conflating too many things in\ncore.bigFileThreshold, but maybe that's just projecting.\n\nI think that setting is probably too much of a kitchen sink grab bag of\nstuff for its own good. Any such GIT_TEST_* mode would I think need to\nintroduce another setting to not have it imply \"these files are binary\".\n\nWhich may be a good idea in general, and it might not. I.e. are there\nusers who mainly don't want to consider these for packing, but do want\nto have \"git diff\" work on them?\n\nAnyway, even if that were split up we'd still have the remaining tests\nthat are assuming that something ends up loose or in a pack.\n\nFixing those is probably a good idea either way, so poking at this might\nbe a useful canary for someone. I haven't looked in any detail, but a\npart of them are probably checking things manually on .git/objects and\ncould move to \"git rev-parse\" or whatever.\n\nThe other half likely really do care about whether something ends up\nloose or not, and would probably benefit from testing \"both sides\".\n\nNone of that's anything I'll pursue now, just idle thoughts from having\nlooked at these failures a bit, in case anyone's interested.\n"}]}