{"thread":{"id":"53458","subject":"[PATCH 01/44] t1050: match object ID paths in a hash-insensitive way","startedAt":"2020-05-13T00:54:40Z","lastAt":"2020-06-20T01:34:29Z","messageCount":175,"participants":["brian m. carlson","Martin Ågren","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":44},"messages":[{"id":"397649","messageId":"20200513005424.81369-2-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 01/44] t1050: match object ID paths in a hash-insensitive way","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:41Z","receivedAt":"2020-05-13T00:54:40Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The pattern here looking for failures is specific to SHA-1. Let's\ncreate a variable that matches the regex or glob pattern for a path\nwithin the objects directory.\n\nSigned-off-by: brian m. carlson \n---\n t/t1050-large.sh | 2 +-\n t/test-lib.sh | 1 +\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 184b479a21..7f88ea07c2 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -64,7 +64,7 @@ test_expect_success 'add a large file or two' '\n \ttest $count = 1 &&\n \tcnt=$(git show-index <\"$idx\" | wc -l) &&\n \ttest $cnt = 2 &&\n-\tfor l in .git/objects/??/??????????????????????????????????????\n+\tfor l in .git/objects/$OIDPATH_REGEX\n \tdo\n \t\ttest_path_is_file \"$l\" || continue\n \t\tbad=t\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex baf94546da..77e9a60fcb 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1428,6 +1428,7 @@ test_oid_init\n \n ZERO_OID=$(test_oid zero)\n OID_REGEX=$(echo $ZERO_OID | sed -e 's/0/[0-9a-f]/g')\n+OIDPATH_REGEX=$(test_oid_to_path $ZERO_OID | sed -e 's/0/[0-9a-f]/g')\n EMPTY_TREE=$(test_oid empty_tree)\n EMPTY_BLOB=$(test_oid empty_blob)\n _z40=$ZERO_OID\n"},{"id":"397650","messageId":"20200513005424.81369-3-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 02/44] Documentation: document v1 protocol object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:42Z","receivedAt":"2020-05-13T00:54:42Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Document a capability that indicates which hash algorithms are in use by\nboth sides of a remote connection. Use the term \"object-format\", since\nthis is the term used for the repository extension as well.\n\nSigned-off-by: brian m. carlson \n---\n .../technical/protocol-capabilities.txt | 16 +++++++++++++++-\n 1 file changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 2b267c0da6..026c42f86a 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -176,6 +176,21 @@ agent strings are purely informative for statistics and debugging\n purposes, and MUST NOT be used to programmatically assume the presence\n or absence of particular features.\n \n+object-format\n+-------------\n+\n+This capability, which takes a hash algorithm as an argument, indicates\n+that the server supports the given hash algorithms. It may be sent\n+multiple times; if so, the first one given is the one used in the ref\n+advertisement.\n+\n+When provided by the client, this indicates that it intends to use the\n+given hash algorithm to communicate. The algorithm provided must be one\n+that the server supports.\n+\n+If this capability is not provided, it is assumed that the only\n+supported algorithm is SHA-1.\n+\n symref\n ------\n \n@@ -189,7 +204,6 @@ refs being sent.\n \n Clients MAY use the parameters from this capability to select the proper initial\n branch when cloning a repository.\n-\n shallow\n -------\n \n"},{"id":"397651","messageId":"20200513005424.81369-5-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 04/44] wrapper: add function to compare strings with different NUL termination","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:44Z","receivedAt":"2020-05-13T00:54:42Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When parsing capabilities for the pack protocol, there are times we'll\nwant to compare the value of a capability to a NUL-terminated string.\nSince the data we're reading will be space-terminated, not\nNUL-terminated, we need a function that compares the two strings, but\nalso checks that they're the same length. Otherwise, if we used strncmp\nto compare these strings, we might accidentally accept a parameter that\nwas a prefix of the expected value.\n\nAdd a function, xstrncmpz, that takes a NUL-terminated string and a\nnon-NUL-terminated string, plus a length, and compares them, ensuring\nthat they are the same length.\n\nSigned-off-by: brian m. carlson \n---\n git-compat-util.h | 2 ++\n wrapper.c | 12 ++++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 8ba576e81e..6503deb171 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -868,6 +868,8 @@ char *xgetcwd(void);\n FILE *fopen_for_writing(const char *path);\n FILE *fopen_or_warn(const char *path, const char *mode);\n \n+int xstrncmpz(const char *s, const char *t, size_t len);\n+\n /*\n * FREE_AND_NULL(ptr) is like free(ptr) followed by ptr = NULL. Note\n * that ptr is used twice, so don't pass e.g. ptr++.\ndiff --git a/wrapper.c b/wrapper.c\nindex 3a1c0e0526..15a09740e7 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -430,6 +430,18 @@ int xmkstemp(char *filename_template)\n \treturn fd;\n }\n \n+/*\n+ * Like strncmp, but only return zero if s is NUL-terminated and exactly len\n+ * characters long. If it is not, consider it greater than t.\n+ */\n+int xstrncmpz(const char *s, const char *t, size_t len)\n+{\n+\tint res = strncmp(s, t, len);\n+\tif (res)\n+\t\treturn res;\n+\treturn s[len] == '\\0' ? 0 : 1;\n+}\n+\n /* Adapted from libiberty's mkstemp.c. */\n \n #undef TMP_MAX\n"},{"id":"397652","messageId":"20200513005424.81369-7-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 06/44] connect: add function to parse multiple v1 capability values","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:46Z","receivedAt":"2020-05-13T00:54:43Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In a capability response, we can have multiple symref entries. In the\nfuture, we will also allow for multiple hash algorithms to be specified.\nTo avoid duplication, expand the parse_feature_value function to take an\noptional offset where the parsing should begin next time. Add a wrapper\nfunction that allows us to query the next server feature value, and use\nit in the existing symref parsing code.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 30 +++++++++++++++++++++---------\n 1 file changed, 21 insertions(+), 9 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 641388a766..4027fd4677 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -18,7 +18,8 @@\n \n static char *server_capabilities_v1;\n static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n-static const char *parse_feature_value(const char *, const char *, int *);\n+static const char *parse_feature_value(const char *, const char *, int *, int *);\n+static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n \n static int check_ref(const char *name, unsigned int flags)\n {\n@@ -180,17 +181,16 @@ static void parse_one_symref_info(struct string_list *symref, const char *val, i\n static void annotate_refs_with_symref_info(struct ref *ref)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\tconst char *feature_list = server_capabilities_v1;\n+\tint offset = 0;\n \n-\twhile (feature_list) {\n+\twhile (1) {\n \t\tint len;\n \t\tconst char *val;\n \n-\t\tval = parse_feature_value(feature_list, \"symref\", &len);\n+\t\tval = next_server_feature_value(\"symref\", &len, &offset);\n \t\tif (!val)\n \t\t\tbreak;\n \t\tparse_one_symref_info(&symref, val, len);\n-\t\tfeature_list = val + 1;\n \t}\n \tstring_list_sort(&symref);\n \n@@ -452,7 +452,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \treturn list;\n }\n \n-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n+static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n {\n \tint len;\n \n@@ -460,6 +460,8 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \t\treturn NULL;\n \n \tlen = strlen(feature);\n+\tif (offset)\n+\t\tfeature_list += *offset;\n \twhile (*feature_list) {\n \t\tconst char *found = strstr(feature_list, feature);\n \t\tif (!found)\n@@ -474,9 +476,14 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \t\t\t}\n \t\t\t/* feature with a value (e.g., \"agent=git/1.2.3\") */\n \t\t\telse if (*value == '=') {\n+\t\t\t\tint end;\n+\n \t\t\t\tvalue++;\n+\t\t\t\tend = strcspn(value, \" \\t\\n\");\n \t\t\t\tif (lenp)\n-\t\t\t\t\t*lenp = strcspn(value, \" \\t\\n\");\n+\t\t\t\t\t*lenp = end;\n+\t\t\t\tif (offset)\n+\t\t\t\t\t*offset = value + end - feature_list;\n \t\t\t\treturn value;\n \t\t\t}\n \t\t\t/*\n@@ -491,12 +498,17 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \n int parse_feature_request(const char *feature_list, const char *feature)\n {\n-\treturn !!parse_feature_value(feature_list, feature, NULL);\n+\treturn !!parse_feature_value(feature_list, feature, NULL, NULL);\n+}\n+\n+static const char *next_server_feature_value(const char *feature, int *len, int *offset)\n+{\n+\treturn parse_feature_value(server_capabilities_v1, feature, len, offset);\n }\n \n const char *server_feature_value(const char *feature, int *len)\n {\n-\treturn parse_feature_value(server_capabilities_v1, feature, len);\n+\treturn parse_feature_value(server_capabilities_v1, feature, len, NULL);\n }\n \n int server_supports(const char *feature)\n"},{"id":"397653","messageId":"20200513005424.81369-8-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 07/44] connect: add function to fetch value of a v2 server capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:47Z","receivedAt":"2020-05-13T00:54:44Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"So far in protocol v2, all of our server capabilities that have values\nhave not had values that we've been interested in parsing. For example,\nwe receive but ignore the agent value.\n\nHowever, in a future commit, we're going to want to parse out the value\nof a server capability. To make this easy, add a function,\nserver_feature_v2, that can fetch the value provided as part of the\nserver capability.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 15 +++++++++++++++\n connect.h | 1 +\n 2 files changed, 16 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex 4027fd4677..4df9e77206 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -84,6 +84,21 @@ int server_supports_v2(const char *c, int die_on_error)\n \treturn 0;\n }\n \n+int server_feature_v2(const char *c, const char **v)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (*out == '=')) {\n+\t\t\t*v = out + 1;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n int server_supports_feature(const char *c, const char *feature,\n \t\t\t int die_on_error)\n {\ndiff --git a/connect.h b/connect.h\nindex 5f2382e018..4d76a6017d 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -19,6 +19,7 @@ struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_v2(const char *c, int die_on_error);\n+int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n \t\t\t int die_on_error);\n \n"},{"id":"397654","messageId":"20200513005424.81369-9-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 08/44] pkt-line: add a member for hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:48Z","receivedAt":"2020-05-13T00:54:45Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a member for the hash algorithm currently in use to the packet\nreader so it can parse references correctly.\n\nSigned-off-by: brian m. carlson \n---\n pkt-line.c | 1 +\n pkt-line.h | 3 +++\n 2 files changed, 4 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex a0e87b1e81..a4aea075de 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -479,6 +479,7 @@ void packet_reader_init(struct packet_reader *reader, int fd,\n \treader->buffer_size = sizeof(packet_buffer);\n \treader->options = options;\n \treader->me = \"git\";\n+\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n }\n \n enum packet_read_status packet_reader_read(struct packet_reader *reader)\ndiff --git a/pkt-line.h b/pkt-line.h\nindex fef3a0d792..4cd9435e9a 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -166,6 +166,9 @@ struct packet_reader {\n \n \tunsigned use_sideband : 1;\n \tconst char *me;\n+\n+\t/* hash algorithm in use */\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n /*\n"},{"id":"397655","messageId":"20200513005424.81369-11-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 10/44] connect: add function to detect supported v1 hash functions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:50Z","receivedAt":"2020-05-13T00:54:47Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a function, server_supports_hash, to see if the remote server\nsupports a particular hash algorithm when speaking protocol v1.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 24 ++++++++++++++++++++++++\n connect.h | 1 +\n 2 files changed, 25 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex 4df9e77206..cb69aafe2c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -511,6 +511,30 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \treturn NULL;\n }\n \n+int server_supports_hash(const char *desired, int *feature_supported)\n+{\n+\tint offset = 0;\n+\tint len, found = 0;\n+\tconst char *hash;\n+\n+\thash = next_server_feature_value(\"object-format\", &len, &offset);\n+\tif (feature_supported)\n+\t\t*feature_supported = !!hash;\n+\tif (!hash) {\n+\t\thash = hash_algos[GIT_HASH_SHA1].name;\n+\t\tlen = strlen(hash);\n+\t}\n+\twhile (hash) {\n+\t\tif (!xstrncmpz(desired, hash, len))\n+\t\t\tfound = 1;\n+\n+\t\tif (found)\n+\t\t\treturn 1;\n+\t\thash = next_server_feature_value(\"object-format\", &len, &offset);\n+\t}\n+\treturn 0;\n+}\n+\n int parse_feature_request(const char *feature_list, const char *feature)\n {\n \treturn !!parse_feature_value(feature_list, feature, NULL, NULL);\ndiff --git a/connect.h b/connect.h\nindex 4d76a6017d..fc75d6a457 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -18,6 +18,7 @@ int url_is_local_not_ssh(const char *url);\n struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n+int server_supports_hash(const char *desired, int *feature_supported);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n"},{"id":"397656","messageId":"20200513005424.81369-16-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 15/44] builtin/receive-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:55Z","receivedAt":"2020-05-13T00:54:51Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n builtin/receive-pack.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex a4159b559e..8755fa2463 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1624,6 +1624,8 @@ static struct command *read_head_info(struct packet_reader *reader,\n \t\tlinelen = strlen(reader->line);\n \t\tif (linelen < reader->pktlen) {\n \t\t\tconst char *feature_list = reader->line + linelen + 1;\n+\t\t\tconst char *hash;\n+\t\t\tint len = 0;\n \t\t\tif (parse_feature_request(feature_list, \"report-status\"))\n \t\t\t\treport_status = 1;\n \t\t\tif (parse_feature_request(feature_list, \"side-band-64k\"))\n@@ -1636,6 +1638,13 @@ static struct command *read_head_info(struct packet_reader *reader,\n \t\t\tif (advertise_push_options\n \t\t\t && parse_feature_request(feature_list, \"push-options\"))\n \t\t\t\tuse_push_options = 1;\n+\t\t\thash = parse_feature_value(feature_list, \"object-format\", &len, NULL);\n+\t\t\tif (!hash) {\n+\t\t\t\thash = hash_algos[GIT_HASH_SHA1].name;\n+\t\t\t\tlen = strlen(hash);\n+\t\t\t}\n+\t\t\tif (xstrncmpz(the_hash_algo->name, hash, len))\n+\t\t\t\tdie(\"error: unsupported object format '%s'\", hash);\n \t\t}\n \n \t\tif (!strcmp(reader->line, \"push-cert\")) {\n"},{"id":"397657","messageId":"20200513005424.81369-13-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 12/44] connect: make parse_feature_value extern","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:52Z","receivedAt":"2020-05-13T00:54:51Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We're going to be using this function in other files, so no longer mark\nthis function static.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 3 +--\n connect.h | 1 +\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex cb69aafe2c..511a069304 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -18,7 +18,6 @@\n \n static char *server_capabilities_v1;\n static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n-static const char *parse_feature_value(const char *, const char *, int *, int *);\n static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -467,7 +466,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \treturn list;\n }\n \n-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n+const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n {\n \tint len;\n \ndiff --git a/connect.h b/connect.h\nindex fc75d6a457..81935a0f2a 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -19,6 +19,7 @@ struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_hash(const char *desired, int *feature_supported);\n+const char *parse_feature_value(const char *, const char *, int *, int *);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n"},{"id":"397658","messageId":"20200513005424.81369-14-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 13/44] fetch-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:53Z","receivedAt":"2020-05-13T00:54:52Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n fetch-pack.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex f73a2ce6cb..1d277190e7 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1039,6 +1039,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tprint_verbose(args, _(\"Server supports %s\"), \"deepen-relative\");\n \telse if (args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n+\tif (!server_supports_hash(the_hash_algo->name, NULL))\n+\t\tdie(_(\"Server does not support this repository's object format\"));\n \n \tif (!args->no_dependents) {\n \t\tmark_complete_and_common_ref(negotiator, args, &ref);\n"},{"id":"397659","messageId":"20200513005424.81369-26-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 25/44] packfile: compute and use the index CRC offset","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:05Z","receivedAt":"2020-05-13T00:54:53Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Both v2 pack index files and the v3 format specified as part of the\nNewHash work have similar data starting at the CRC table. Much of the\nexisting code wants to read either this table or the offset entries\nfollowing it, and in doing so computes the offset each time.\n\nIn order to share as much code between v2 and v3, compute the offset of\nthe CRC table and store it when the pack is opened. Use this value to\ncompute offsets to not only the CRC table, but to the offset entries\nbeyond it.\n\nSigned-off-by: brian m. carlson \n---\n builtin/index-pack.c | 6 +-----\n object-store.h | 1 +\n packfile.c | 1 +\n 3 files changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f176dd28c8..7bea1fba52 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1555,13 +1555,9 @@ static void read_v2_anomalous_offsets(struct packed_git *p,\n {\n \tconst uint32_t *idx1, *idx2;\n \tuint32_t i;\n-\tconst uint32_t hashwords = the_hash_algo->rawsz / sizeof(uint32_t);\n \n \t/* The address of the 4-byte offset table */\n-\tidx1 = (((const uint32_t *)p->index_data)\n-\t\t+ 2 /* 8-byte header */\n-\t\t+ 256 /* fan out */\n-\t\t+ hashwords * p->num_objects /* object ID table */\n+\tidx1 = (((const uint32_t *)((const uint8_t *)p->index_data + p->crc_offset))\n \t\t+ p->num_objects /* CRC32 table */\n \t\t);\n \ndiff --git a/object-store.h b/object-store.h\nindex d1e490f203..f439d47af8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -70,6 +70,7 @@ struct packed_git {\n \tsize_t index_size;\n \tuint32_t num_objects;\n \tuint32_t num_bad_objects;\n+\tuint32_t crc_offset;\n \tunsigned char *bad_object_sha1;\n \tint index_version;\n \ttime_t mtime;\ndiff --git a/packfile.c b/packfile.c\nindex f4e752996d..6ab5233613 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -178,6 +178,7 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \t\t */\n \t\t (sizeof(off_t) <= 4))\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n+\t\tp->crc_offset = 8 + 4 * 256 + nr * hashsz;\n \t}\n \n \tp->index_version = version;\n"},{"id":"397660","messageId":"20200513005424.81369-25-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 24/44] t3200: mark assertion with SHA1 prerequisite","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:04Z","receivedAt":"2020-05-13T00:54:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"One of the test assertions in this test checks that git branch -m works\neven without a .git/config file. However, if the repository requires\nconfiguration extensions, such as because it uses a non-SHA-1 algorithm,\nthis assertion will fail. Mark the assertion as requiring SHA-1.\n\nSigned-off-by: brian m. carlson \n---\n t/t3200-branch.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\nindex 411a70b0ce..2a3fedc6b0 100755\n--- a/t/t3200-branch.sh\n+++ b/t/t3200-branch.sh\n@@ -402,7 +402,7 @@ EOF\n \n mv .git/config .git/config-saved\n \n-test_expect_success 'git branch -m q q2 without config should succeed' '\n+test_expect_success SHA1 'git branch -m q q2 without config should succeed' '\n \tgit branch -m q q2 &&\n \tgit branch -m q2 q\n '\n"},{"id":"397661","messageId":"20200513005424.81369-23-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 22/44] fetch-pack: parse and advertise the object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:02Z","receivedAt":"2020-05-13T00:54:56Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Parse the server's object-format capability and respond accordingly,\ndying if there is a mismatch.\n\nSigned-off-by: brian m. carlson \n---\n fetch-pack.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1d277190e7..3a48ed4b13 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1179,6 +1179,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t int sideband_all, int seen_ack)\n {\n \tint ret = 0;\n+\tconst char *hash_name;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n@@ -1193,6 +1194,17 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t\t\t args->server_options->items[i].string);\n \t}\n \n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n+\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n+\t\t\t the_hash_algo->name, hash_name);\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", the_hash_algo->name);\n+\t}\n+\telse if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1)\n+\t\tdie(_(\"the server does not support algorithm '%s'\"),\n+\t\t the_hash_algo->name);\n+\n \tpacket_buf_delim(&req_buf);\n \tif (args->use_thin_pack)\n \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n"},{"id":"397662","messageId":"20200513005424.81369-24-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 23/44] setup: set the_repository's hash algo when checking format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:03Z","receivedAt":"2020-05-13T00:54:58Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're checking the repository's format, set the hash algorithm at\nthe same time. This ensures that we perform a suitable initialization\nearly enough to avoid confusing any parts of the code. If we defer\nuntil later, we can end up with portions of the code which are confused\nabout the hash algorithm, resulting in segfaults.\n\nSigned-off-by: brian m. carlson \n---\n setup.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/setup.c b/setup.c\nindex 65fe5ecefb..019a1c6367 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1273,6 +1273,7 @@ void check_repository_format(struct repository_format *fmt)\n \t\tfmt = &repo_fmt;\n \tcheck_repository_format_gently(get_git_dir(), fmt, NULL);\n \tstartup_info->have_repository = 1;\n+\trepo_set_hash_algo(the_repository, fmt->hash_algo);\n \tclear_repository_format(&repo_fmt);\n }\n \n"},{"id":"397663","messageId":"20200513005424.81369-22-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 21/44] t5704: send object-format capability with SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:01Z","receivedAt":"2020-05-13T00:55:00Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we speak protocol v2 in this test, we must pass the object-format\nheader if the algorithm is not SHA-1. Otherwise, git upload-pack fails\nbecause the hash algorithm doesn't match and not because we've failed to\nspeak the protocol correctly. Pass the header so that our assertions\ntest what we're really interested in.\n\nSigned-off-by: brian m. carlson \n---\n t/t5704-protocol-violations.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/t5704-protocol-violations.sh b/t/t5704-protocol-violations.sh\nindex 950cfb21fe..47e78932b9 100755\n--- a/t/t5704-protocol-violations.sh\n+++ b/t/t5704-protocol-violations.sh\n@@ -6,9 +6,20 @@ communications if the other side says something unexpected. We are mostly\n making sure that we do not segfault or otherwise behave badly.'\n . ./test-lib.sh\n \n+# If we don't print the object format, we'll fail for a spurious reason: the\n+# mismatched object format.\n+print_object_format () {\n+\tlocal algo=$(test_oid algo) &&\n+\tif test \"$algo\" != \"sha1\"\n+\tthen\n+\t\tpacketize \"object-format=$algo\"\n+\tfi\n+}\n+\n test_expect_success 'extra delim packet in v2 ls-refs args' '\n \t{\n \t\tpacketize command=ls-refs &&\n+\t\tprint_object_format &&\n \t\tprintf 0001 &&\n \t\t# protocol expects 0000 flush here\n \t\tprintf 0001\n@@ -21,6 +32,7 @@ test_expect_success 'extra delim packet in v2 ls-refs args' '\n test_expect_success 'extra delim packet in v2 fetch args' '\n \t{\n \t\tpacketize command=fetch &&\n+\t\tprint_object_format &&\n \t\tprintf 0001 &&\n \t\t# protocol expects 0000 flush here\n \t\tprintf 0001\n"},{"id":"397664","messageId":"20200513005424.81369-19-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 18/44] remote-curl: implement object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:58Z","receivedAt":"2020-05-13T00:55:01Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Implement the object-format extensions that let us determine the hash\nalgorithm in use when pushing, pulling, and fetching.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 19 ++++++++++++++++++-\n 1 file changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 1c9aa3d0ab..3ed0dfec1b 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -41,7 +41,9 @@ struct options {\n \t\tdeepen_relative : 1,\n \t\tfrom_promisor : 1,\n \t\tno_dependents : 1,\n-\t\tatomic : 1;\n+\t\tatomic : 1,\n+\t\tobject_format : 1;\n+\tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n static struct string_list cas_options = STRING_LIST_INIT_DUP;\n@@ -190,6 +192,16 @@ static int set_option(const char *name, const char *value)\n \t} else if (!strcmp(name, \"filter\")) {\n \t\toptions.filter = xstrdup(value);\n \t\treturn 0;\n+\t} else if (!strcmp(name, \"object-format\")) {\n+\t\tint algo;\n+\t\toptions.object_format = 1;\n+\t\tif (strcmp(value, \"true\")) {\n+\t\t\talgo = hash_algo_by_name(value);\n+\t\t\tif (algo == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"unknown object format '%s'\", value);\n+\t\t\toptions.hash_algo = &hash_algos[algo];\n+\t\t}\n+\t\treturn 0;\n \t} else {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -231,6 +243,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n \t\t\t\t NULL, &heads->shallow);\n+\t\toptions.hash_algo = reader.hash_algo;\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n@@ -509,6 +522,9 @@ static struct ref *get_refs(int for_push)\n static void output_refs(struct ref *refs)\n {\n \tstruct ref *posn;\n+\tif (options.object_format && options.hash_algo) {\n+\t\tprintf(\":object-format %s\\n\", options.hash_algo->name);\n+\t}\n \tfor (posn = refs; posn; posn = posn->next) {\n \t\tif (posn->symref)\n \t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n@@ -1439,6 +1455,7 @@ int cmd_main(int argc, const char **argv)\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n+\t\t\tprintf(\"object-format\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else if (skip_prefix(buf.buf, \"stateless-connect \", &arg)) {\n"},{"id":"397665","messageId":"20200513005424.81369-31-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 30/44] connect: pass full packet reader when parsing v2 refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:10Z","receivedAt":"2020-05-13T00:55:02Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're parsing refs, we need to know not only what the line we're\nparsing is, but also the hash algorithm we should use to parse it, which\nis stored in the reader object. Pass the packet reader object through\nto the protocol v2 ref parsing function.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex a39b843589..bb4bc4ab7f 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -373,7 +373,7 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n }\n \n /* Returns 1 when a valid ref has been added to `list`, 0 otherwise */\n-static int process_ref_v2(const char *line, struct ref ***list)\n+static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n {\n \tint ret = 1;\n \tint i = 0;\n@@ -381,6 +381,7 @@ static int process_ref_v2(const char *line, struct ref ***list)\n \tstruct ref *ref;\n \tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n \tconst char *end;\n+\tconst char *line = reader->line;\n \n \t/*\n \t * Ref lines have a number of fields which are space deliminated. The\n@@ -466,9 +467,10 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t}\n \tpacket_flush(fd_out);\n \n+\n \t/* Process response from server */\n \twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n-\t\tif (!process_ref_v2(reader->line, &list))\n+\t\tif (!process_ref_v2(reader, &list))\n \t\t\tdie(_(\"invalid ls-refs response: %s\"), reader->line);\n \t}\n \n"},{"id":"397666","messageId":"20200513005424.81369-37-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 36/44] builtin/index-pack: add option to specify hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:16Z","receivedAt":"2020-05-13T00:55:04Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"git index-pack is usually run in a repository, but need not be. Since\npacks don't contains information on the algorithm in use, instead\nrelying on context, add an option to index-pack to tell it which one\nwe're using in case someone runs it outside of a repository.\n\nSigned-off-by: brian m. carlson \n---\n builtin/index-pack.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7bea1fba52..89f4962a00 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1760,6 +1760,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t} else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n \t\t\t\tmax_input_size = strtoumax(arg, NULL, 10);\n+\t\t\t} else if (skip_prefix(arg, \"--object-format=\", &arg)) {\n+\t\t\t\tint hash_algo = hash_algo_by_name(arg);\n+\t\t\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n+\t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n"},{"id":"397667","messageId":"20200513005424.81369-33-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 32/44] serve: advertise object-format capability for protocol v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:12Z","receivedAt":"2020-05-13T00:55:05Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In order to communicate the protocol supported by the server side, add\nsupport for advertising the object-format capability. We check that the\nclient side sends us an identical algorithm if it sends us its own\nobject-format capability, and assume it speaks SHA-1 if not.\n\nIn the test, when we're using an algorithm other than SHA-1, we need to\nspecify the algorithm in use so we don't get a failure with an \"unknown\nformat\" message. Add a wrapper function that specifies this header if\nrequired. Skip specifying this header for SHA-1 to test that it works\nboth with and without this header.\n\nSigned-off-by: brian m. carlson \n---\n serve.c | 27 +++++++++++++++++++++++++++\n t/t5701-git-serve.sh | 28 ++++++++++++++++++++--------\n 2 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/serve.c b/serve.c\nindex 317256c1a4..7ab7807fef 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -22,6 +22,14 @@ static int agent_advertise(struct repository *r,\n \treturn 1;\n }\n \n+static int object_format_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, r->hash_algo->name);\n+\treturn 1;\n+}\n+\n struct protocol_capability {\n \t/*\n \t * The name of the capability. The server uses this name when\n@@ -57,6 +65,7 @@ static struct protocol_capability capabilities[] = {\n \t{ \"ls-refs\", always_advertise, ls_refs },\n \t{ \"fetch\", upload_pack_advertise, upload_pack_v2 },\n \t{ \"server-option\", always_advertise, NULL },\n+\t{ \"object-format\", object_format_advertise, NULL },\n };\n \n static void advertise_capabilities(void)\n@@ -153,6 +162,22 @@ int has_capability(const struct argv_array *keys, const char *capability,\n \treturn 0;\n }\n \n+static void check_algorithm(struct repository *r, struct argv_array *keys)\n+{\n+\tint client = GIT_HASH_SHA1, server = hash_algo_by_ptr(r->hash_algo);\n+\tconst char *algo_name;\n+\n+\tif (has_capability(keys, \"object-format\", &algo_name)) {\n+\t\tclient = hash_algo_by_name(algo_name);\n+\t\tif (client == GIT_HASH_UNKNOWN)\n+\t\t\tdie(\"unknown object format '%s'\", algo_name);\n+\t}\n+\n+\tif (client != server)\n+\t\tdie(\"mismatched object format: server %s; client %s\\n\",\n+\t\t r->hash_algo->name, hash_algos[client].name);\n+}\n+\n enum request_state {\n \tPROCESS_REQUEST_KEYS,\n \tPROCESS_REQUEST_DONE,\n@@ -223,6 +248,8 @@ static int process_request(void)\n \tif (!command)\n \t\tdie(\"no command requested\");\n \n+\tcheck_algorithm(the_repository, &keys);\n+\n \tcommand->command(the_repository, &keys, &reader);\n \n \targv_array_clear(&keys);\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex ffb9613885..bcb6453ae3 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -4,13 +4,24 @@ test_description='test protocol v2 server commands'\n \n . ./test-lib.sh\n \n+write_command () {\n+\techo \"command=$1\"\n+\n+\tif test \"$(test_oid algo)\" != sha1\n+\tthen\n+\t\techo \"object-format=$(test_oid algo)\"\n+\tfi\n+}\n+\n test_expect_success 'test capability advertisement' '\n+\ttest_oid_init &&\n \tcat >expect <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n \tfetch=shallow\n \tserver-option\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \n@@ -45,6 +56,7 @@ test_expect_success 'request invalid capability' '\n test_expect_success 'request with no command' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tagent=git/test\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n-\tcommand=foo\n+\t$(write_command foo)\n \tagent=git/test\n \t0000\n \tEOF\n@@ -73,7 +85,7 @@ test_expect_success 'setup some refs and tags' '\n \n test_expect_success 'basics of ls-refs' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=ls-refs\n+\t$(write_command ls-refs)\n \t0000\n \tEOF\n \n@@ -95,7 +107,7 @@ test_expect_success 'basics of ls-refs' '\n \n test_expect_success 'basic ref-prefixes' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=ls-refs\n+\t$(write_command ls-refs)\n \t0001\n \tref-prefix refs/heads/master\n \tref-prefix refs/tags/one\n@@ -115,7 +127,7 @@ test_expect_success 'basic ref-prefixes' '\n \n test_expect_success 'refs/heads prefix' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=ls-refs\n+\t$(write_command ls-refs)\n \t0001\n \tref-prefix refs/heads/\n \t0000\n@@ -135,7 +147,7 @@ test_expect_success 'refs/heads prefix' '\n \n test_expect_success 'peel parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=ls-refs\n+\t$(write_command ls-refs)\n \t0001\n \tpeel\n \tref-prefix refs/tags/\n@@ -156,7 +168,7 @@ test_expect_success 'peel parameter' '\n \n test_expect_success 'symrefs parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=ls-refs\n+\t$(write_command ls-refs)\n \t0001\n \tsymrefs\n \tref-prefix refs/heads/\n@@ -177,7 +189,7 @@ test_expect_success 'symrefs parameter' '\n \n test_expect_success 'sending server-options' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=ls-refs\n+\t$(write_command ls-refs)\n \tserver-option=hello\n \tserver-option=world\n \t0001\n@@ -199,7 +211,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n \tgit init server &&\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tthis-is-not-a-command\n \t0000\n"},{"id":"397668","messageId":"20200513005424.81369-45-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 44/44] remote-testgit: adapt for object-format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:24Z","receivedAt":"2020-05-13T00:55:06Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using an algorithm other than SHA-1, we need the remote helper to\nadvertise support for the object-format extension and provide\ninformation back to us so that we can properly parse refs and return\ndata. Ensure that the test remote helper understands these extensions.\n\nSigned-off-by: brian m. carlson \n---\n t/t5801/git-remote-testgit | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/t/t5801/git-remote-testgit b/t/t5801/git-remote-testgit\nindex 6b9f0b5dc7..1544d6dc6b 100755\n--- a/t/t5801/git-remote-testgit\n+++ b/t/t5801/git-remote-testgit\n@@ -52,9 +52,11 @@ do\n \t\ttest -n \"$GIT_REMOTE_TESTGIT_SIGNED_TAGS\" && echo \"signed-tags\"\n \t\ttest -n \"$GIT_REMOTE_TESTGIT_NO_PRIVATE_UPDATE\" && echo \"no-private-update\"\n \t\techo 'option'\n+\t\techo 'object-format'\n \t\techo\n \t\t;;\n \tlist)\n+\t\techo \":object-format $(git rev-parse --show-object-format=storage)\"\n \t\tgit for-each-ref --format='? %(refname)' 'refs/heads/' 'refs/tags/'\n \t\thead=$(git symbolic-ref HEAD)\n \t\techo \"@$head HEAD\"\n@@ -139,6 +141,10 @@ do\n \t\t\ttest $val = \"true\" && force=\"true\" || force=\n \t\t\techo \"ok\"\n \t\t\t;;\n+\t\tobject-format)\n+\t\t\ttest $val = \"true\" && object_format=\"true\" || object_format=\n+\t\t\techo \"ok\"\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"unsupported\"\n \t\t\t;;\n"},{"id":"397669","messageId":"20200513005424.81369-21-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 20/44] t5562: pass object-format in synthesized test data","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:00Z","receivedAt":"2020-05-13T00:55:06Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Ensure that we pass the object-format capability in the synthesized test\ndata so that this test works with algorithms other than SHA-1.\n\nIn addition, add an additional test using the old data for when we're\nusing SHA-1 so that we can be sure that we preserve backwards\ncompatibility with servers not offering the object-format capability.\n\nSigned-off-by: brian m. carlson \n---\n t/t5562-http-backend-content-length.sh | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5562-http-backend-content-length.sh b/t/t5562-http-backend-content-length.sh\nindex 3f4ac71f83..f508d4d449 100755\n--- a/t/t5562-http-backend-content-length.sh\n+++ b/t/t5562-http-backend-content-length.sh\n@@ -46,6 +46,7 @@ ssize_b100dots() {\n }\n \n test_expect_success 'setup' '\n+\ttest_oid_init &&\n \tHTTP_CONTENT_ENCODING=\"identity\" &&\n \texport HTTP_CONTENT_ENCODING &&\n \tgit config http.receivepack true &&\n@@ -62,8 +63,8 @@ test_expect_success 'setup' '\n \ttest_copy_bytes 10 fetch_body.trunc &&\n \thash_next=$(git commit-tree -p HEAD -m next HEAD^{tree}) &&\n \t{\n-\t\tprintf \"%s %s refs/heads/newbranch\\\\0report-status\\\\n\" \\\n-\t\t\t\"$ZERO_OID\" \"$hash_next\" | packetize &&\n+\t\tprintf \"%s %s refs/heads/newbranch\\\\0report-status object-format=%s\\\\n\" \\\n+\t\t\t\"$ZERO_OID\" \"$hash_next\" \"$(test_oid algo)\" | packetize &&\n \t\tprintf 0000 &&\n \t\techo \"$hash_next\" | git pack-objects --stdout\n \t} >push_body &&\n@@ -117,6 +118,15 @@ test_expect_success GZIP 'push plain' '\n \ttest_cmp act.head exp.head\n '\n \n+test_expect_success GZIP 'push plain with SHA-1' '\n+\ttest_when_finished \"git branch -D newbranch\" &&\n+\ttest_http_env receive push_body &&\n+\tverify_http_result \"200 OK\" &&\n+\tgit rev-parse newbranch >act.head &&\n+\techo \"$hash_next\" >exp.head &&\n+\ttest_cmp act.head exp.head\n+'\n+\n test_expect_success 'push plain truncated' '\n \ttest_http_env receive push_body.trunc &&\n \t! verify_http_result \"200 OK\"\n"},{"id":"397670","messageId":"20200513005424.81369-44-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 43/44] bundle: detect hash algorithm when reading refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:23Z","receivedAt":"2020-05-13T00:55:08Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Much like with the dumb HTTP transport, there isn't a way to explicitly\nspecify the hash algorithm when dealing with a bundle, so detect the\nalgorithm based on the length of the object IDs in the prerequisites and\nref advertisements.\n\nSigned-off-by: brian m. carlson \n---\n bundle.c | 22 +++++++++++++++++++++-\n bundle.h | 1 +\n transport.c | 10 ++++++++--\n 3 files changed, 30 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 99439e07a1..2a0d744d3f 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -23,6 +23,17 @@ static void add_to_ref_list(const struct object_id *oid, const char *name,\n \tlist->nr++;\n }\n \n+static const struct git_hash_algo *detect_hash_algo(struct strbuf *buf)\n+{\n+\tsize_t len = strcspn(buf->buf, \" \\n\");\n+\tint algo;\n+\n+\talgo = hash_algo_by_length(len / 2);\n+\tif (algo == GIT_HASH_UNKNOWN)\n+\t\treturn NULL;\n+\treturn &hash_algos[algo];\n+}\n+\n static int parse_bundle_header(int fd, struct bundle_header *header,\n \t\t\t const char *report_path)\n {\n@@ -52,12 +63,21 @@ static int parse_bundle_header(int fd, struct bundle_header *header,\n \t\t}\n \t\tstrbuf_rtrim(&buf);\n \n+\t\tif (!header->hash_algo) {\n+\t\t\theader->hash_algo = detect_hash_algo(&buf);\n+\t\t\tif (!header->hash_algo) {\n+\t\t\t\terror(_(\"unknown hash algorithm length\"));\n+\t\t\t\tstatus = -1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n \t\t/*\n \t\t * Tip lines have object name, SP, and refname.\n \t\t * Prerequisites have object name that is optionally\n \t\t * followed by SP and subject line.\n \t\t */\n-\t\tif (parse_oid_hex(buf.buf, &oid, &p) ||\n+\t\tif (parse_oid_hex_algop(buf.buf, &oid, &p, header->hash_algo) ||\n \t\t (*p && !isspace(*p)) ||\n \t\t (!is_prereq && !*p)) {\n \t\t\tif (report_path)\ndiff --git a/bundle.h b/bundle.h\nindex ceab0c7475..2dc9442024 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -15,6 +15,7 @@ struct ref_list {\n struct bundle_header {\n \tstruct ref_list prerequisites;\n \tstruct ref_list references;\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n int is_bundle(const char *path, int quiet);\ndiff --git a/transport.c b/transport.c\nindex b43d985f90..38a432be69 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -143,6 +143,9 @@ static struct ref *get_refs_from_bundle(struct transport *transport,\n \tdata->fd = read_bundle_header(transport->url, &data->header);\n \tif (data->fd < 0)\n \t\tdie(_(\"could not read bundle '%s'\"), transport->url);\n+\n+\ttransport->hash_algo = data->header.hash_algo;\n+\n \tfor (i = 0; i < data->header.references.nr; i++) {\n \t\tstruct ref_list_entry *e = data->header.references.list + i;\n \t\tstruct ref *ref = alloc_ref(e->name);\n@@ -157,11 +160,14 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n \tstruct bundle_transport_data *data = transport->data;\n+\tint ret;\n \n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle(transport, 0, NULL);\n-\treturn unbundle(the_repository, &data->header, data->fd,\n-\t\t\ttransport->progress ? BUNDLE_VERBOSE : 0);\n+\tret = unbundle(the_repository, &data->header, data->fd,\n+\t\t\t transport->progress ? BUNDLE_VERBOSE : 0);\n+\ttransport->hash_algo = data->header.hash_algo;\n+\treturn ret;\n }\n \n static int close_bundle(struct transport *transport)\n"},{"id":"397671","messageId":"20200513005424.81369-42-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 41/44] t5703: use object-format serve option","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:21Z","receivedAt":"2020-05-13T00:55:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're using an algorithm other than SHA-1, we need to specify the\nalgorithm in use so we don't get a failure with an \"unknown format\"\nmessage. Add a wrapper function that specifies this header if required.\nSkip specifying this header for SHA-1 to test that it works both with an\nwithout this header.\n\nSigned-off-by: brian m. carlson \n---\n t/t5703-upload-pack-ref-in-want.sh | 19 ++++++++++++++-----\n 1 file changed, 14 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh\nindex a34460f7d8..afe7f7f919 100755\n--- a/t/t5703-upload-pack-ref-in-want.sh\n+++ b/t/t5703-upload-pack-ref-in-want.sh\n@@ -27,6 +27,15 @@ check_output () {\n \ttest_cmp sorted_commits actual_commits\n }\n \n+write_command () {\n+\techo \"command=$1\"\n+\n+\tif test \"$(test_oid algo)\" != sha1\n+\tthen\n+\t\techo \"object-format=$(test_oid algo)\"\n+\tfi\n+}\n+\n # c(o/foo) d(o/bar)\n # \\ /\n # b e(baz) f(master)\n@@ -62,7 +71,7 @@ test_expect_success 'config controls ref-in-want advertisement' '\n \n test_expect_success 'invalid want-ref line' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/non-existent\n@@ -83,7 +92,7 @@ test_expect_success 'basic want-ref' '\n \n \toid=$(git rev-parse a) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/master\n@@ -107,7 +116,7 @@ test_expect_success 'multiple want-ref lines' '\n \n \toid=$(git rev-parse b) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/o/foo\n@@ -129,7 +138,7 @@ test_expect_success 'mix want and want-ref' '\n \tgit rev-parse e f >expected_commits &&\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/master\n@@ -152,7 +161,7 @@ test_expect_success 'want-ref with ref we already have commit for' '\n \n \toid=$(git rev-parse c) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/o/foo\n"},{"id":"397672","messageId":"20200513005424.81369-43-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 42/44] t5300: pass --object-format to git index-pack","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:22Z","receivedAt":"2020-05-13T00:55:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"git index-pack by default reads the repository to determine the object\nformat. However, when outside of a repository, it's necessary to specify\nthe hash algorithm in use so that the pack can be properly indexed. Add\nan --object-format argument when invoking git index-pack outside of a\nrepository.\n\nSigned-off-by: brian m. carlson \n---\n t/t5300-pack-object.sh | 9 +++++----\n 1 file changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 410a09b0dd..746cdb626e 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -12,7 +12,8 @@ TRASH=$(pwd)\n \n test_expect_success \\\n 'setup' \\\n- 'rm -f .git/index* &&\n+ 'test_oid_init &&\n+ rm -f .git/index* &&\n perl -e \"print \\\"a\\\" x 4096;\" > a &&\n perl -e \"print \\\"b\\\" x 4096;\" > b &&\n perl -e \"print \\\"c\\\" x 4096;\" > c &&\n@@ -412,18 +413,18 @@ test_expect_success 'set up pack for non-repo tests' '\n '\n \n test_expect_success 'index-pack --stdin complains of non-repo' '\n-\tnongit test_must_fail git index-pack --stdin works in non-repo' '\n-\tnongit git index-pack ../foo.pack &&\n+\tnongit git index-pack --object-format=$(test_oid algo) ../foo.pack &&\n \ttest_path_is_file foo.idx\n '\n \n test_expect_success 'index-pack --strict works in non-repo' '\n \trm -f foo.idx &&\n-\tnongit git index-pack --strict ../foo.pack &&\n+\tnongit git index-pack --strict --object-format=$(test_oid algo) ../foo.pack &&\n \ttest_path_is_file foo.idx\n '\n \n"},{"id":"397673","messageId":"20200513005424.81369-41-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 40/44] t5702: offer an object-format capability in the test","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:20Z","receivedAt":"2020-05-13T00:55:11Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In order to make this test work with SHA-256, offer an object-format\ncapability so that both sides use the same algorithm.\n\nSigned-off-by: brian m. carlson \n---\n t/t5702-protocol-v2.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 5039e66dc4..116358b9ac 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -13,6 +13,7 @@ start_git_daemon --export-all --enable=receive-pack\n daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n \n test_expect_success 'create repo to be served by git-daemon' '\n+\ttest_oid_init &&\n \tgit init \"$daemon_parent\" &&\n \ttest_commit -C \"$daemon_parent\" one\n '\n@@ -394,6 +395,7 @@ test_expect_success 'even with handcrafted request, filter does not work if not\n \t# Custom request that tries to filter even though it is not advertised.\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=fetch\n+\tobject-format=$(test_oid algo)\n \t0001\n \twant $(git -C server rev-parse master)\n \tfilter blob:none\n"},{"id":"397674","messageId":"20200513005424.81369-40-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 39/44] t/helper: initialize the repository for test-sha1-array","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:19Z","receivedAt":"2020-05-13T00:55:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"test-sha1-array uses the_hash_algo under the hood. Since t0064 wants to\nuse the value that is correct for the hash algorithm that we're testing,\nmake sure the test helper initializes the repository to set\nthe_hash_algo correctly.\n\nSigned-off-by: brian m. carlson \n---\n t/helper/test-oid-array.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/helper/test-oid-array.c b/t/helper/test-oid-array.c\nindex ce9fd5f091..b16cd0b11b 100644\n--- a/t/helper/test-oid-array.c\n+++ b/t/helper/test-oid-array.c\n@@ -12,6 +12,9 @@ int cmd__oid_array(int argc, const char **argv)\n {\n \tstruct oid_array array = OID_ARRAY_INIT;\n \tstruct strbuf line = STRBUF_INIT;\n+\tint nongit_ok;\n+\n+\tsetup_git_directory_gently(&nongit_ok);\n \n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tconst char *arg;\n"},{"id":"397675","messageId":"20200513005424.81369-1-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":null,"subject":"[PATCH 00/44] SHA-256 part 2/3: protocol functionality","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:40Z","receivedAt":"2020-05-13T00:55:14Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This is part 2 of 3 of the SHA-256 work. This series, which is\nunfortunately longer than I'd like, adds all of the protocol logic to\nwork with SHA-256 repositories.\n\nIt was originally planned that we would not upgrade the protocol and\nwould use SHA-1 for all protocol functionality until some point in the\nfuture. However, doing that requires a huge amount of additional work\n(probably incorporating several hundred more patches which are not yet\nwritten) and it's not possible to get the test suite to even come close\nto passing without a way to fetch and push repositories. I therefore\ndecided that implementing an object-format extension was the best way\nforward.\n\nThis series provides object-format extensions for both the original and\nv2 protocols, including full documentation. Helpers, such as\ngit-remote-https, also learn capabilities to pass the object-format\nextension back and forth, and to query its state. The code is designed\nto allow multiple object-format extensions to be provided if the server\nsupports multiple algorithms for one repo and to default to SHA-1 if no\nobject-format extension is provided.\n\nThe other two cases are the dumb HTTP protocol and bundles, both of\nwhich have no object-format extension (because they provide no\ncapabilities) and are therefore distinguished solely by their hash\nlength. We will have problems if in the future we need to use another\n256-bit algorithm, but I plan to be improvident and hope that we'll move\nto longer algorithms in the future to cover ourselves for post-quantum\nsecurity.\n\nClone support is necessarily a little tricky because we are initializing\na repository and then fetching refs, at which point we learn what hash\nalgorithm the remote side supports. We work around this by calling the\ncode that updates the hash algorithm and repository version a second\ntime to rewrite that data once we know what version we're using. This\nis the most robust way I could approach this problem, but it is still a\nlittle ugly.\n\nAs mentioned, this series is longer than I'd like, but it is complete:\nthis is all the SHA-256 protocol work. Additional future series include\none last series of test fixes (28 patches) plus six final patches in the\nseries that enables SHA-256 support.\n\nbrian m. carlson (44):\n t1050: match object ID paths in a hash-insensitive way\n Documentation: document v1 protocol object-format capability\n connect: have ref processing code take struct packet_reader\n wrapper: add function to compare strings with different NUL\n termination\n remote: advertise the object-format capability on the server side\n connect: add function to parse multiple v1 capability values\n connect: add function to fetch value of a v2 server capability\n pkt-line: add a member for hash algorithm\n transport: add a hash algorithm member\n connect: add function to detect supported v1 hash functions\n send-pack: detect when the server doesn't support our hash\n connect: make parse_feature_value extern\n fetch-pack: detect when the server doesn't support our hash\n connect: detect algorithm when fetching refs\n builtin/receive-pack: detect when the server doesn't support our hash\n docs: update remote helper docs for object-format extensions\n transport-helper: implement object-format extensions\n remote-curl: implement object-format extensions\n builtin/clone: initialize hash algorithm properly\n t5562: pass object-format in synthesized test data\n t5704: send object-format capability with SHA-256\n fetch-pack: parse and advertise the object-format capability\n setup: set the_repository's hash algo when checking format\n t3200: mark assertion with SHA1 prerequisite\n packfile: compute and use the index CRC offset\n t5302: modernize test formatting\n builtin/show-index: provide options to determine hash algo\n t1302: expect repo format version 1 for SHA-256\n Documentation/technical: document object-format for protocol v2\n connect: pass full packet reader when parsing v2 refs\n connect: parse v2 refs with correct hash algorithm\n serve: advertise object-format capability for protocol v2\n t5500: make hash independent\n builtin/ls-remote: initialize repository based on fetch\n remote-curl: detect algorithm for dumb HTTP by size\n builtin/index-pack: add option to specify hash algorithm\n t1050: pass algorithm to index-pack when outside repo\n remote-curl: avoid truncating refs with ls-remote\n t/helper: initialize the repository for test-sha1-array\n t5702: offer an object-format capability in the test\n t5703: use object-format serve option\n t5300: pass --object-format to git index-pack\n bundle: detect hash algorithm when reading refs\n remote-testgit: adapt for object-format\n\n Documentation/gitremote-helpers.txt | 33 +-\n .../technical/protocol-capabilities.txt | 16 +-\n Documentation/technical/protocol-v2.txt | 9 +\n builtin/clone.c | 9 +\n builtin/index-pack.c | 11 +-\n builtin/ls-remote.c | 4 +\n builtin/receive-pack.c | 10 +\n builtin/show-index.c | 29 +-\n bundle.c | 22 +-\n bundle.h | 1 +\n connect.c | 136 +++++--\n connect.h | 3 +\n fetch-pack.c | 14 +\n git-compat-util.h | 2 +\n git.c | 2 +-\n object-store.h | 1 +\n packfile.c | 1 +\n pkt-line.c | 1 +\n pkt-line.h | 3 +\n remote-curl.c | 46 ++-\n send-pack.c | 6 +\n serve.c | 27 ++\n setup.c | 1 +\n t/helper/test-oid-array.c | 3 +\n t/t1050-large.sh | 6 +-\n t/t1302-repo-version.sh | 6 +-\n t/t3200-branch.sh | 2 +-\n t/t5300-pack-object.sh | 9 +-\n t/t5302-pack-index.sh | 360 +++++++++---------\n t/t5500-fetch-pack.sh | 5 +-\n t/t5562-http-backend-content-length.sh | 14 +-\n t/t5701-git-serve.sh | 28 +-\n t/t5702-protocol-v2.sh | 2 +\n t/t5703-upload-pack-ref-in-want.sh | 19 +-\n t/t5704-protocol-violations.sh | 12 +\n t/t5801/git-remote-testgit | 6 +\n t/test-lib.sh | 1 +\n transport-helper.c | 24 +-\n transport.c | 18 +-\n transport.h | 8 +\n upload-pack.c | 3 +-\n wrapper.c | 12 +\n 42 files changed, 670 insertions(+), 255 deletions(-)\n\n"},{"id":"397676","messageId":"20200513005424.81369-4-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 03/44] connect: have ref processing code take struct packet_reader","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:43Z","receivedAt":"2020-05-13T00:55:16Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In a future patch, we'll want to access multiple members from struct\npacket_reader when parsing references. Therefore, have the ref parsing\ncode take pointers to struct reader instead of having to pass multiple\narguments to each function.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 23 ++++++++++++++---------\n 1 file changed, 14 insertions(+), 9 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 23013c6344..641388a766 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -204,8 +204,9 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-static void process_capabilities(const char *line, int *len)\n+static void process_capabilities(struct packet_reader *reader, int *len)\n {\n+\tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n \tif (nul_location == *len)\n \t\treturn;\n@@ -213,8 +214,9 @@ static void process_capabilities(const char *line, int *len)\n \t*len = nul_location;\n }\n \n-static int process_dummy_ref(const char *line)\n+static int process_dummy_ref(const struct packet_reader *reader)\n {\n+\tconst char *line = reader->line;\n \tstruct object_id oid;\n \tconst char *name;\n \n@@ -234,9 +236,11 @@ static void check_no_capabilities(const char *line, int len)\n \t\t\tline + strlen(line));\n }\n \n-static int process_ref(const char *line, int len, struct ref ***list,\n-\t\t unsigned int flags, struct oid_array *extra_have)\n+static int process_ref(const struct packet_reader *reader, int len,\n+\t\t struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n {\n+\tconst char *line = reader->line;\n \tstruct object_id old_oid;\n \tconst char *name;\n \n@@ -260,9 +264,10 @@ static int process_ref(const char *line, int len, struct ref ***list,\n \treturn 1;\n }\n \n-static int process_shallow(const char *line, int len,\n+static int process_shallow(const struct packet_reader *reader, int len,\n \t\t\t struct oid_array *shallow_points)\n {\n+\tconst char *line = reader->line;\n \tconst char *arg;\n \tstruct object_id old_oid;\n \n@@ -315,20 +320,20 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \n \t\tswitch (state) {\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(reader->line, &len);\n-\t\t\tif (process_dummy_ref(reader->line)) {\n+\t\t\tprocess_capabilities(reader, &len);\n+\t\t\tif (process_dummy_ref(reader)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(reader->line, len, shallow_points))\n+\t\t\tif (process_shallow(reader, len, shallow_points))\n \t\t\t\tbreak;\n \t\t\tdie(_(\"protocol error: unexpected '%s'\"), reader->line);\n \t\tcase EXPECTING_DONE:\n"},{"id":"397677","messageId":"20200513005424.81369-10-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 09/44] transport: add a hash algorithm member","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:49Z","receivedAt":"2020-05-13T00:55:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When connecting to a remote system, we need to know what hash algorithm\nit will be using to talk to us. Add a hash_algo member to struct\ntransport and add a function to read this data from the transport\nobject.\n\nSigned-off-by: brian m. carlson \n---\n transport.c | 8 ++++++++\n transport.h | 8 ++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/transport.c b/transport.c\nindex 15f5ba4e8f..b43d985f90 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -311,6 +311,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \t\tBUG(\"unknown protocol version\");\n \t}\n \tdata->got_remote_heads = 1;\n+\ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n \t\tBUG(\"buffer must be empty at the end of handshake()\");\n@@ -996,9 +997,16 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\t\tret->smart_options->receivepack = remote->receivepack;\n \t}\n \n+\tret->hash_algo = &hash_algos[GIT_HASH_SHA1];\n+\n \treturn ret;\n }\n \n+const struct git_hash_algo *transport_get_hash_algo(struct transport *transport)\n+{\n+\treturn transport->hash_algo;\n+}\n+\n int transport_set_option(struct transport *transport,\n \t\t\t const char *name, const char *value)\n {\ndiff --git a/transport.h b/transport.h\nindex 4298c855be..2a9f96c05a 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -115,6 +115,8 @@ struct transport {\n \tstruct git_transport_options *smart_options;\n \n \tenum transport_family family;\n+\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n #define TRANSPORT_PUSH_ALL\t\t\t(1<<0)\n@@ -243,6 +245,12 @@ int transport_push(struct repository *repo,\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t const struct argv_array *ref_prefixes);\n \n+/*\n+ * Fetch the hash algorithm used by a remote.\n+ *\n+ * This can only be called after fetching the remote refs.\n+ */\n+const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n int transport_disconnect(struct transport *transport);\n"},{"id":"397678","messageId":"20200513005424.81369-17-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 16/44] docs: update remote helper docs for object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:56Z","receivedAt":"2020-05-13T00:55:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Update the remote helper docs to document the object-format extensions\nwe will implement in remote-curl and the transport helper code shortly.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitremote-helpers.txt | 33 +++++++++++++++++++++++++----\n 1 file changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex f48a031dc3..26f32e4421 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -238,6 +238,9 @@ the remote repository.\n \t`--signed-tags=verbatim` to linkgit:git-fast-export[1]. In the\n \tabsence of this capability, Git will use `--signed-tags=warn-strip`.\n \n+'object-format'::\n+\tThis indicates that the helper is able to interact with the remote\n+\tside using an explicit hash algorithm extension.\n \n \n COMMANDS\n@@ -257,12 +260,14 @@ Support for this command is mandatory.\n 'list'::\n \tLists the refs, one per line, in the format \" \n \t[ ...]\". The value may be a hex sha1 hash, \"@\" for\n-\ta symref, or \"?\" to indicate that the helper could not get the\n-\tvalue of the ref. A space-separated list of attributes follows\n-\tthe name; unrecognized attributes are ignored. The list ends\n-\twith a blank line.\n+\ta symref, \": \" for a key-value pair, or\n+\t\"?\" to indicate that the helper could not get the value of the\n+\tref. A space-separated list of attributes follows the name;\n+\tunrecognized attributes are ignored. The list ends with a\n+\tblank line.\n +\n See REF LIST ATTRIBUTES for a list of currently defined attributes.\n+See REF LIST KEYWORDS for a list of currently defined keywords.\n +\n Supported if the helper has the \"fetch\" or \"import\" capability.\n \n@@ -430,6 +435,18 @@ attributes are defined.\n \tThis ref is unchanged since the last import or fetch, although\n \tthe helper cannot necessarily determine what value that produced.\n \n+REF LIST KEYWORDS\n+-----------------\n+\n+The 'list' command may produce a list of key-value pairs.\n+The following keys are defined.\n+\n+'object-format'::\n+\tThe refs are using the given hash algorithm. This keyword is only\n+\tused if the server and client both support the object-format\n+\textension.\n+\n+\n OPTIONS\n -------\n \n@@ -514,6 +531,14 @@ set by Git if the remote helper has the 'option' capability.\n \ttransaction. If successful, all refs will be updated, or none will. If the\n \tremote side does not support this capability, the push will fail.\n \n+'option object-format' {'true'|algorithm}::\n+\tIf 'true', indicate that the caller wants hash algorithm information\n+\tto be passed back from the remote. This mode is used when fetching\n+\trefs.\n++\n+If set to an algorithm, indicate that the caller wants to interact with\n+the remote side using that algorithm.\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\n"},{"id":"397679","messageId":"20200513005424.81369-20-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 19/44] builtin/clone: initialize hash algorithm properly","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:59Z","receivedAt":"2020-05-13T00:55:21Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When performing a clone, we don't know what hash algorithm the other end\nwill support. Currently, we don't support fetching data belonging to a\ndifferent algorithm, so we must know what algorithm the remote side is\nusing in order to properly initialize the repository. We can know that\nonly after fetching the refs, so if the remote side has any references,\nuse that information to reinitialize the repository with the correct\nhash algorithm information.\n\nSigned-off-by: brian m. carlson \n---\n builtin/clone.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex cb48a291ca..f27d38bc8e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1217,6 +1217,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \trefs = transport_get_remote_refs(transport, &ref_prefixes);\n \n \tif (refs) {\n+\t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n+\n+\t\t/*\n+\t\t * Now that we know what algorithm the remote side is using,\n+\t\t * let's set ours to the same thing.\n+\t\t */\n+\t\tinitialize_repository_version(hash_algo);\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n"},{"id":"397680","messageId":"20200513005424.81369-15-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 14/44] connect: detect algorithm when fetching refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:54Z","receivedAt":"2020-05-13T00:55:21Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"If we're fetching refs, detect the hash algorithm and parse the refs\nusing that algorithm.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 511a069304..a39b843589 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -220,12 +220,24 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \n static void process_capabilities(struct packet_reader *reader, int *len)\n {\n+\tconst char *feat_val;\n+\tint feat_len;\n+\tint hash_algo;\n \tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n \tif (nul_location == *len)\n \t\treturn;\n \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n \t*len = nul_location;\n+\n+\tfeat_val = server_feature_value(\"object-format\", &feat_len);\n+\tif (feat_val) {\n+\t\tchar *hash_name = xstrndup(feat_val, feat_len);\n+\t\thash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo != GIT_HASH_UNKNOWN)\n+\t\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tfree(hash_name);\n+\t}\n }\n \n static int process_dummy_ref(const struct packet_reader *reader)\n@@ -234,7 +246,7 @@ static int process_dummy_ref(const struct packet_reader *reader)\n \tstruct object_id oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(line, &oid, &name))\n+\tif (parse_oid_hex_algop(line, &oid, &name, reader->hash_algo))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -258,7 +270,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \tstruct object_id old_oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(line, &old_oid, &name))\n+\tif (parse_oid_hex_algop(line, &old_oid, &name, reader->hash_algo))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -270,7 +282,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \t\tdie(_(\"protocol error: unexpected capabilities^{}\"));\n \t} else if (check_ref(name, flags)) {\n \t\tstruct ref *ref = alloc_ref(name);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n@@ -288,7 +300,7 @@ static int process_shallow(const struct packet_reader *reader, int len,\n \tif (!skip_prefix(line, \"shallow \", &arg))\n \t\treturn 0;\n \n-\tif (get_oid_hex(arg, &old_oid))\n+\tif (get_oid_hex_algop(arg, &old_oid, reader->hash_algo))\n \t\tdie(_(\"protocol error: expected shallow sha-1, got '%s'\"), arg);\n \tif (!shallow_points)\n \t\tdie(_(\"repository on the other end cannot be shallow\"));\n"},{"id":"397681","messageId":"20200513005424.81369-6-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 05/44] remote: advertise the object-format capability on the server side","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:45Z","receivedAt":"2020-05-13T00:55:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Advertise the current hash algorithm in use by using the object-format\ncapability as part of the ref advertisement.\n\nSigned-off-by: brian m. carlson \n---\n builtin/receive-pack.c | 1 +\n upload-pack.c | 3 ++-\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d37ab776b3..a4159b559e 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -248,6 +248,7 @@ static void show_ref(const char *path, const struct object_id *oid)\n \t\t\tstrbuf_addf(&cap, \" push-cert=%s\", push_cert_nonce);\n \t\tif (advertise_push_options)\n \t\t\tstrbuf_addstr(&cap, \" push-options\");\n+\t\tstrbuf_addf(&cap, \" object-format=%s\", the_hash_algo->name);\n \t\tstrbuf_addf(&cap, \" agent=%s\", git_user_agent_sanitized());\n \t\tpacket_write_fmt(1, \"%s %s%c%s\\n\",\n \t\t\t oid_to_hex(oid), path, 0, cap.buf);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 902d0ad5e1..df6cb51db7 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1005,7 +1005,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n \t\tformat_symref_info(&symref_info, cb_data);\n-\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s agent=%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s object-format=%s agent=%s\\n\",\n \t\t\t oid_to_hex(oid), refname_nons,\n \t\t\t 0, capabilities,\n \t\t\t (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n@@ -1015,6 +1015,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\t\t stateless_rpc ? \" no-done\" : \"\",\n \t\t\t symref_info.buf,\n \t\t\t allow_filter ? \" filter\" : \"\",\n+\t\t\t the_hash_algo->name,\n \t\t\t git_user_agent_sanitized());\n \t\tstrbuf_release(&symref_info);\n \t} else {\n"},{"id":"397682","messageId":"20200513005424.81369-28-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 27/44] builtin/show-index: provide options to determine hash algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:07Z","receivedAt":"2020-05-13T00:55:26Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"It's possible to use a variety of index formats with show-index, and we\nneed a way to indicate the hash algorithm which is in use for a\nparticular index we'd like to show. Default to using the value for the\nrepository we're in by calling setup_git_directory_gently, and allow\noverriding it by using a --hash argument.\n\nSigned-off-by: brian m. carlson \n---\n builtin/show-index.c | 29 ++++++++++++++++++++++++-----\n git.c | 2 +-\n 2 files changed, 25 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/show-index.c b/builtin/show-index.c\nindex 0826f6a5a2..ebfa2e9abd 100644\n--- a/builtin/show-index.c\n+++ b/builtin/show-index.c\n@@ -1,9 +1,12 @@\n #include \"builtin.h\"\n #include \"cache.h\"\n #include \"pack.h\"\n+#include \"parse-options.h\"\n \n-static const char show_index_usage[] =\n-\"git show-index\";\n+static const char *const show_index_usage[] = {\n+\t\"git show-index [--hash=HASH]\",\n+\tNULL\n+};\n \n int cmd_show_index(int argc, const char **argv, const char *prefix)\n {\n@@ -11,10 +14,26 @@ int cmd_show_index(int argc, const char **argv, const char *prefix)\n \tunsigned nr;\n \tunsigned int version;\n \tstatic unsigned int top_index[256];\n-\tconst unsigned hashsz = the_hash_algo->rawsz;\n+\tunsigned hashsz;\n+\tconst char *hash_name = NULL;\n+\tint hash_algo;\n+\tconst struct option show_index_options[] = {\n+\t\tOPT_STRING(0, \"hash\", &hash_name, N_(\"hash\"),\n+\t\t\t N_(\"specify the hash algorithm to use\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, show_index_options, show_index_usage, 0);\n+\n+\tif (hash_name) {\n+\t\thash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"Unknown hash algorithm\"));\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t}\n+\n+\thashsz = the_hash_algo->rawsz;\n \n-\tif (argc != 1)\n-\t\tusage(show_index_usage);\n \tif (fread(top_index, 2 * 4, 1, stdin) != 1)\n \t\tdie(\"unable to read header\");\n \tif (top_index[0] == htonl(PACK_IDX_SIGNATURE)) {\ndiff --git a/git.c b/git.c\nindex 2e4efb4ff0..e53e8159a2 100644\n--- a/git.c\n+++ b/git.c\n@@ -573,7 +573,7 @@ static struct cmd_struct commands[] = {\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n-\t{ \"show-index\", cmd_show_index },\n+\t{ \"show-index\", cmd_show_index, RUN_SETUP_GENTLY },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n \t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n"},{"id":"397683","messageId":"20200513005424.81369-34-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 33/44] t5500: make hash independent","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:13Z","receivedAt":"2020-05-13T00:55:26Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This test has hard-coded pkt-lines with object IDs. The pkt-line\nlengths necessarily differ between hash algorithms, so generate these\nlines with the packetize helper so they're always the right size. In\naddition, we will require an object-format capability for SHA-256, so\npass that capability on to the upload-pack process.\n\nSigned-off-by: brian m. carlson \n---\n t/t5500-fetch-pack.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 52dd1a688c..8fee99ecfb 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -871,9 +871,10 @@ test_expect_success 'shallow since with commit graph and already-seen commit' '\n \n \tGIT_PROTOCOL=version=2 git upload-pack . <<-EOF >/dev/null\n \t0012command=fetch\n+\t$(echo \"object-format=$(test_oid algo)\" | packetize)\n \t00010013deepen-since 1\n-\t0032want $(git rev-parse other)\n-\t0032have $(git rev-parse master)\n+\t$(echo \"want $(git rev-parse other)\" | packetize)\n+\t$(echo \"have $(git rev-parse master)\" | packetize)\n \t0000\n \tEOF\n \t)\n"},{"id":"397684","messageId":"20200513005424.81369-39-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 38/44] remote-curl: avoid truncating refs with ls-remote","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:18Z","receivedAt":"2020-05-13T00:55:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Normally, the remote-curl transport helper is aware of the hash\nalgorithm we're using because we're in a repo with the appropriate hash\nalgorithm set. However, when using git ls-remote outside of a\nrepository, we won't have initialized the hash algorithm properly, so\nuse hash_to_hex_algop to print the ref corresponding to the algorithm\nwe've detected.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 35275b42e9..9808e53182 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -548,7 +548,9 @@ static void output_refs(struct ref *refs)\n \t\tif (posn->symref)\n \t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n \t\telse\n-\t\t\tprintf(\"%s %s\\n\", oid_to_hex(&posn->old_oid), posn->name);\n+\t\t\tprintf(\"%s %s\\n\", hash_to_hex_algop(posn->old_oid.hash,\n+\t\t\t\t\t\t\t options.hash_algo),\n+\t\t\t\t\t posn->name);\n \t}\n \tprintf(\"\\n\");\n \tfflush(stdout);\n"},{"id":"397685","messageId":"20200513005424.81369-36-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 35/44] remote-curl: detect algorithm for dumb HTTP by size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:15Z","receivedAt":"2020-05-13T00:55:31Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When reading the info/refs file for a repository, we have no explicit\nway to detect which hash algorithm is in use because the file doesn't\nprovide one. Detect the hash algorithm in use by the size of the first\nobject ID.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 3ed0dfec1b..35275b42e9 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -252,6 +252,19 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \treturn list;\n }\n \n+static const struct git_hash_algo *detect_hash_algo(struct discovery *heads)\n+{\n+\tconst char *p = memchr(heads->buf, '\\t', heads->len);\n+\tint algo;\n+\tif (!p)\n+\t\treturn NULL;\n+\n+\talgo = hash_algo_by_length((p - heads->buf) / 2);\n+\tif (algo == GIT_HASH_UNKNOWN)\n+\t\treturn NULL;\n+\treturn &hash_algos[algo];\n+}\n+\n static struct ref *parse_info_refs(struct discovery *heads)\n {\n \tchar *data, *start, *mid;\n@@ -262,6 +275,12 @@ static struct ref *parse_info_refs(struct discovery *heads)\n \tstruct ref *ref = NULL;\n \tstruct ref *last_ref = NULL;\n \n+\toptions.hash_algo = detect_hash_algo(heads);\n+\tif (!options.hash_algo)\n+\t\tdie(\"%sinfo/refs not valid: could not determine hash algorithm; \"\n+\t\t \"is this a git repository?\",\n+\t\t url.buf);\n+\n \tdata = heads->buf;\n \tstart = NULL;\n \tmid = data;\n@@ -272,13 +291,13 @@ static struct ref *parse_info_refs(struct discovery *heads)\n \t\tif (data[i] == '\\t')\n \t\t\tmid = &data[i];\n \t\tif (data[i] == '\\n') {\n-\t\t\tif (mid - start != the_hash_algo->hexsz)\n+\t\t\tif (mid - start != options.hash_algo->hexsz)\n \t\t\t\tdie(_(\"%sinfo/refs not valid: is this a git repository?\"),\n \t\t\t\t transport_anonymize_url(url.buf));\n \t\t\tdata[i] = 0;\n \t\t\tref_name = mid + 1;\n \t\t\tref = alloc_ref(ref_name);\n-\t\t\tget_oid_hex(start, &ref->old_oid);\n+\t\t\tget_oid_hex_algop(start, &ref->old_oid, options.hash_algo);\n \t\t\tif (!refs)\n \t\t\t\trefs = ref;\n \t\t\tif (last_ref)\n"},{"id":"397686","messageId":"20200513005424.81369-38-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 37/44] t1050: pass algorithm to index-pack when outside repo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:17Z","receivedAt":"2020-05-13T00:55:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When outside a repository, git index-pack is unable to guess the hash\nalgorithm in use for a pack, since packs don't contain any information\non the algorithm in use. Pass an option to index-pack to help it out in\nthis test.\n\nSigned-off-by: brian m. carlson \n---\n t/t1050-large.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 7f88ea07c2..6a56d1ca24 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -12,6 +12,7 @@ file_size () {\n }\n \n test_expect_success setup '\n+\ttest_oid_init &&\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n@@ -177,7 +178,8 @@ test_expect_success 'git-show a large file' '\n \n test_expect_success 'index-pack' '\n \tgit clone file://\"$(pwd)\"/.git foo &&\n-\tGIT_DIR=non-existent git index-pack --strict --verify foo/.git/objects/pack/*.pack\n+\tGIT_DIR=non-existent git index-pack --object-format=$(test_oid algo) \\\n+\t\t--strict --verify foo/.git/objects/pack/*.pack\n '\n \n test_expect_success 'repack' '\n"},{"id":"397687","messageId":"20200513005424.81369-35-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 34/44] builtin/ls-remote: initialize repository based on fetch","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:14Z","receivedAt":"2020-05-13T00:55:36Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"ls-remote may or may not operate within a repository, and as such will\nnot have been initialized with the repository's hash algorithm. Even if\nit were, the remote side could be using a different algorithm and we\nwould still want to display those refs properly. Find the hash\nalgorithm used by the remote side by querying the transport object and\nset our hash algorithm accordingly.\n\nWithout this change, if the remote side is using SHA-256, we truncate\nthe refs to 40 hex characters, since that's the length of the default\nhash algorithm (SHA-1).\n\nSigned-off-by: brian m. carlson \n---\n builtin/ls-remote.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex 6ef519514b..3a4dd12903 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -118,6 +118,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\ttransport->server_options = &server_options;\n \n \tref = transport_get_remote_refs(transport, &ref_prefixes);\n+\tif (ref) {\n+\t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t}\n \tif (transport_disconnect(transport)) {\n \t\tUNLEAK(sorting);\n \t\treturn 1;\n"},{"id":"397688","messageId":"20200513005424.81369-32-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 31/44] connect: parse v2 refs with correct hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:11Z","receivedAt":"2020-05-13T00:55:59Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using protocol v2, we need to know what hash algorithm is used by\nthe remote end. See if the server has sent us an object-format\ncapability, and if so, use it to determine the hash algorithm in use and\nset that value in the packet reader. Parse the refs using this\nalgorithm.\n\nNote that we use memcpy instead of oidcpy for copying values, since\noidcpy is intentionally limited to the current hash algorithm length,\nand the copy will be too short if the server side uses SHA-256 but the\nclient side has not had a repository set up (and therefore defaults to\nSHA-1).\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 19 +++++++++++++++----\n 1 file changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex bb4bc4ab7f..4e6462e52f 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -394,7 +394,7 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\tgoto out;\n \t}\n \n-\tif (parse_oid_hex(line_sections.items[i++].string, &old_oid, &end) ||\n+\tif (parse_oid_hex_algop(line_sections.items[i++].string, &old_oid, &end, reader->hash_algo) ||\n \t *end) {\n \t\tret = 0;\n \t\tgoto out;\n@@ -402,7 +402,7 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \n \tref = alloc_ref(line_sections.items[i++].string);\n \n-\toidcpy(&ref->old_oid, &old_oid);\n+\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n \t**list = ref;\n \t*list = &ref->next;\n \n@@ -415,7 +415,8 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\t\tstruct object_id peeled_oid;\n \t\t\tchar *peeled_name;\n \t\t\tstruct ref *peeled;\n-\t\t\tif (parse_oid_hex(arg, &peeled_oid, &end) || *end) {\n+\t\t\tif (parse_oid_hex_algop(arg, &peeled_oid, &end,\n+\t\t\t\t\t\treader->hash_algo) || *end) {\n \t\t\t\tret = 0;\n \t\t\t\tgoto out;\n \t\t\t}\n@@ -423,7 +424,8 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n \t\t\tpeeled = alloc_ref(peeled_name);\n \n-\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n+\t\t\tmemcpy(peeled->old_oid.hash, peeled_oid.hash,\n+\t\t\t reader->hash_algo->rawsz);\n \t\t\t**list = peeled;\n \t\t\t*list = &peeled->next;\n \n@@ -442,6 +444,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options)\n {\n \tint i;\n+\tconst char *hash_name;\n \t*list = NULL;\n \n \tif (server_supports_v2(\"ls-refs\", 1))\n@@ -450,6 +453,14 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \tif (server_supports_v2(\"agent\", 0))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n \n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n+\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n+\t}\n+\n \tif (server_options && server_options->nr &&\n \t server_supports_v2(\"server-option\", 1))\n \t\tfor (i = 0; i < server_options->nr; i++)\n"},{"id":"397689","messageId":"20200513005424.81369-12-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 11/44] send-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:51Z","receivedAt":"2020-05-13T00:56:02Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n send-pack.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex d1b7edc995..fb037568a9 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -362,6 +362,7 @@ int send_pack(struct send_pack_args *args,\n \tint atomic_supported = 0;\n \tint use_push_options = 0;\n \tint push_options_supported = 0;\n+\tint object_format_supported = 0;\n \tunsigned cmds_sent = 0;\n \tint ret;\n \tstruct async demux;\n@@ -388,6 +389,9 @@ int send_pack(struct send_pack_args *args,\n \tif (server_supports(\"push-options\"))\n \t\tpush_options_supported = 1;\n \n+\tif (!server_supports_hash(the_hash_algo->name, &object_format_supported))\n+\t\tdie(_(\"the receiving end does not support this repository's hash algorithm\"));\n+\n \tif (args->push_cert != SEND_PACK_PUSH_CERT_NEVER) {\n \t\tint len;\n \t\tpush_cert_nonce = server_feature_value(\"push-cert\", &len);\n@@ -428,6 +432,8 @@ int send_pack(struct send_pack_args *args,\n \t\tstrbuf_addstr(&cap_buf, \" atomic\");\n \tif (use_push_options)\n \t\tstrbuf_addstr(&cap_buf, \" push-options\");\n+\tif (object_format_supported)\n+\t\tstrbuf_addf(&cap_buf, \" object-format=%s\", the_hash_algo->name);\n \tif (agent_supported)\n \t\tstrbuf_addf(&cap_buf, \" agent=%s\", git_user_agent_sanitized());\n \n"},{"id":"397690","messageId":"20200513005424.81369-29-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 28/44] t1302: expect repo format version 1 for SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:08Z","receivedAt":"2020-05-13T00:56:03Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using SHA-256, we need to take advantage of the extensions section\nin the config file, so we need to use repository format version 1.\nUpdate the test to look for the correct value.\n\nNote that test_oid produces a value without a trailing newline, so use\necho to ensure we print a trailing newline to compare it correctly\nagainst the actual results.\n\nSigned-off-by: brian m. carlson \n---\n t/t1302-repo-version.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh\nindex ce4cff13bb..d60c042ce8 100755\n--- a/t/t1302-repo-version.sh\n+++ b/t/t1302-repo-version.sh\n@@ -8,6 +8,10 @@ test_description='Test repository version check'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n+\ttest_oid_cache <<-\\EOF &&\n+\tversion sha1:0\n+\tversion sha256:1\n+\tEOF\n \tcat >test.patch <<-\\EOF &&\n \tdiff --git a/test.txt b/test.txt\n \tnew file mode 100644\n@@ -23,7 +27,7 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'gitdir selection on normal repos' '\n-\techo 0 >expect &&\n+\techo $(test_oid version) >expect &&\n \tgit config core.repositoryformatversion >actual &&\n \tgit -C test config core.repositoryformatversion >actual2 &&\n \ttest_cmp expect actual &&\n"},{"id":"397691","messageId":"20200513005424.81369-27-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 26/44] t5302: modernize test formatting","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:06Z","receivedAt":"2020-05-13T00:56:04Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Our style these days is to place the description and the opening quote\nof the body on the same line as test_expect_success (if it fits), to\nplace the trailing quote on a line by itself after the body, and to use\ntabs. Since we're going to be making several significant changes to\nthis test, modernize the style to aid in readability of the subsequent\npatches.\n\nThis patch should have no functional change.\n\nSigned-off-by: brian m. carlson \n---\n t/t5302-pack-index.sh | 360 +++++++++++++++++++++---------------------\n 1 file changed, 184 insertions(+), 176 deletions(-)\n\ndiff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\nindex ad07f2f7fc..8981c9b90e 100755\n--- a/t/t5302-pack-index.sh\n+++ b/t/t5302-pack-index.sh\n@@ -7,65 +7,65 @@ test_description='pack index with 64-bit offsets and object CRC'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n- test_oid_init &&\n- rawsz=$(test_oid rawsz) &&\n- rm -rf .git &&\n- git init &&\n- git config pack.threads 1 &&\n- i=1 &&\n- while test $i -le 100\n- do\n- iii=$(printf '%03i' $i)\n-\t test-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n-\t test-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n-\t test-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n-\t test-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n-\t test-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n- echo $iii >file_$iii &&\n-\t test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n- git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n- i=$(expr $i + 1) || return 1\n- done &&\n- { echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n- git update-index --add file_101 &&\n- tree=$(git write-tree) &&\n- commit=$(git commit-tree $tree obj-list &&\n- git update-ref HEAD $commit\n+\ttest_oid_init &&\n+\trawsz=$(test_oid rawsz) &&\n+\trm -rf .git &&\n+\tgit init &&\n+\tgit config pack.threads 1 &&\n+\ti=1 &&\n+\twhile test $i -le 100\n+\tdo\n+\t\tiii=$(printf '%03i' $i)\n+\t\ttest-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree obj-list &&\n+\tgit update-ref HEAD $commit\n '\n \n-test_expect_success \\\n- 'pack-objects with index version 1' \\\n- 'pack1=$(git pack-objects --index-version=1 test-1 &1) ||\n \t! (echo \"$msg\" | grep \"pack too large .* off_t\")\n@@ -91,21 +91,21 @@ else\n \tsay \"# skipping tests concerning 64-bit offsets\"\n fi\n \n-test_expect_success OFF64_T \\\n- 'index v2: verify a pack with some 64-bit offsets' \\\n- 'git verify-pack -v \"test-3-${pack3}.pack\"'\n+test_expect_success OFF64_T 'index v2: verify a pack with some 64-bit offsets' '\n+\tgit verify-pack -v \"test-3-${pack3}.pack\"\n+'\n \n-test_expect_success OFF64_T \\\n- '64-bit offsets: should be different from previous index v2 results' \\\n- '! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"'\n+test_expect_success OFF64_T '64-bit offsets: should be different from previous index v2 results' '\n+\t! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"\n+'\n \n-test_expect_success OFF64_T \\\n- 'index v2: force some 64-bit offsets with index-pack' \\\n- 'git index-pack --index-version=2,0x40000 -o 3.idx \"test-1-${pack1}.pack\"'\n+test_expect_success OFF64_T 'index v2: force some 64-bit offsets with index-pack' '\n+\tgit index-pack --index-version=2,0x40000 -o 3.idx \"test-1-${pack1}.pack\"\n+'\n \n-test_expect_success OFF64_T \\\n- '64-bit offsets: index-pack result should match pack-objects one' \\\n- 'cmp \"test-3-${pack3}.idx\" \"3.idx\"'\n+test_expect_success OFF64_T '64-bit offsets: index-pack result should match pack-objects one' '\n+\tcmp \"test-3-${pack3}.idx\" \"3.idx\"\n+'\n \n test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2 (cheat)' '\n \t# This cheats by knowing which lower offset should still be encoded\n@@ -120,135 +120,143 @@ test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '\n # returns the object number for given object in given pack index\n index_obj_nr()\n {\n- idx_file=$1\n- object_sha1=$2\n- nr=0\n- git show-index < $idx_file |\n- while read offs sha1 extra\n- do\n- nr=$(($nr + 1))\n- test \"$sha1\" = \"$object_sha1\" || continue\n- echo \"$(($nr - 1))\"\n- break\n- done\n+\tidx_file=$1\n+\tobject_sha1=$2\n+\tnr=0\n+\tgit show-index < $idx_file |\n+\twhile read offs sha1 extra\n+\tdo\n+\t nr=$(($nr + 1))\n+\t test \"$sha1\" = \"$object_sha1\" || continue\n+\t echo \"$(($nr - 1))\"\n+\t break\n+\tdone\n }\n \n # returns the pack offset for given object as found in given pack index\n index_obj_offset()\n {\n- idx_file=$1\n- object_sha1=$2\n- git show-index < $idx_file | grep $object_sha1 |\n- ( read offs extra && echo \"$offs\" )\n+\tidx_file=$1\n+\tobject_sha1=$2\n+\tgit show-index < $idx_file | grep $object_sha1 |\n+\t( read offs extra && echo \"$offs\" )\n }\n \n-test_expect_success \\\n- '[index v1] 1) stream pack to repository' \\\n- 'git index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n- git prune-packed &&\n- git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n- cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n- cmp \"test-1-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n+test_expect_success '[index v1] 1) stream pack to repository' '\n+\tgit index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n+\tgit prune-packed &&\n+\tgit count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n+\tcmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n+\tcmp \"test-1-${pack1}.idx\"\t\".git/objects/pack/pack-${pack1}.idx\"\n+'\n \n test_expect_success \\\n- '[index v1] 2) create a stealth corruption in a delta base reference' \\\n- '# This test assumes file_101 is a delta smaller than 16 bytes.\n- # It should be against file_100 but we substitute its base for file_099\n- sha1_101=$(git hash-object file_101) &&\n- sha1_099=$(git hash-object file_099) &&\n- offs_101=$(index_obj_offset 1.idx $sha1_101) &&\n- nr_099=$(index_obj_nr 1.idx $sha1_099) &&\n- chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n- recordsz=$((rawsz + 4)) &&\n- dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n- if=\".git/objects/pack/pack-${pack1}.idx\" \\\n- skip=$((4 + 256 * 4 + $nr_099 * recordsz)) \\\n- bs=1 count=$rawsz conv=notrunc &&\n- git cat-file blob $sha1_101 > file_101_foo1'\n+\t'[index v1] 2) create a stealth corruption in a delta base reference' '\n+\t# This test assumes file_101 is a delta smaller than 16 bytes.\n+\t# It should be against file_100 but we substitute its base for file_099\n+\tsha1_101=$(git hash-object file_101) &&\n+\tsha1_099=$(git hash-object file_099) &&\n+\toffs_101=$(index_obj_offset 1.idx $sha1_101) &&\n+\tnr_099=$(index_obj_nr 1.idx $sha1_099) &&\n+\tchmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n+\trecordsz=$((rawsz + 4)) &&\n+\tdd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n+\t if=\".git/objects/pack/pack-${pack1}.idx\" \\\n+\t skip=$((4 + 256 * 4 + $nr_099 * recordsz)) \\\n+\t bs=1 count=$rawsz conv=notrunc &&\n+\tgit cat-file blob $sha1_101 > file_101_foo1\n+'\n \n test_expect_success \\\n- '[index v1] 3) corrupted delta happily returned wrong data' \\\n- 'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'\n+\t'[index v1] 3) corrupted delta happily returned wrong data' '\n+\ttest -f file_101_foo1 && ! cmp file_101 file_101_foo1\n+'\n \n test_expect_success \\\n- '[index v1] 4) confirm that the pack is actually corrupted' \\\n- 'test_must_fail git fsck --full $commit'\n+\t'[index v1] 4) confirm that the pack is actually corrupted' '\n+\ttest_must_fail git fsck --full $commit\n+'\n \n test_expect_success \\\n- '[index v1] 5) pack-objects happily reuses corrupted data' \\\n- 'pack4=$(git pack-objects test-4 file_101_foo2\n+'\n \n test_expect_success \\\n- '[index v2] 1) stream pack to repository' \\\n- 'rm -f .git/objects/pack/* &&\n- git index-pack --index-version=2 --stdin < \"test-1-${pack1}.pack\" &&\n- git prune-packed &&\n- git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n- cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n- cmp \"test-2-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n+\t'[index v2] 3) corrupted delta happily returned wrong data' '\n+\ttest -f file_101_foo2 && ! cmp file_101 file_101_foo2\n+'\n \n test_expect_success \\\n- '[index v2] 2) create a stealth corruption in a delta base reference' \\\n- '# This test assumes file_101 is a delta smaller than 16 bytes.\n- # It should be against file_100 but we substitute its base for file_099\n- sha1_101=$(git hash-object file_101) &&\n- sha1_099=$(git hash-object file_099) &&\n- offs_101=$(index_obj_offset 1.idx $sha1_101) &&\n- nr_099=$(index_obj_nr 1.idx $sha1_099) &&\n- chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n- dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n- if=\".git/objects/pack/pack-${pack1}.idx\" \\\n- skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \\\n- bs=1 count=$rawsz conv=notrunc &&\n- git cat-file blob $sha1_101 > file_101_foo2'\n+\t'[index v2] 4) confirm that the pack is actually corrupted' '\n+\ttest_must_fail git fsck --full $commit\n+'\n \n test_expect_success \\\n- '[index v2] 3) corrupted delta happily returned wrong data' \\\n- 'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'\n+\t'[index v2] 5) pack-objects refuses to reuse corrupted data' '\n+\ttest_must_fail git pack-objects test-5 /dev/null || exit 1\n- done /dev/null || exit 1\n+\t done wrong-tag <wrong-tag <err &&\n- grep \"^warning:.* expected .tagger. line\" err\n+\ttag=$(git hash-object -t tag -w --stdin err &&\n+\tgrep \"^warning:.* expected .tagger. line\" err\n '\n \n test_expect_success 'index-pack --fsck-objects also warns upon missing tagger in tag' '\n- git index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&\n- grep \"^warning:.* expected .tagger. line\" err\n+\tgit index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&\n+\tgrep \"^warning:.* expected .tagger. line\" err\n '\n \n test_done\n"},{"id":"397692","messageId":"20200513005424.81369-30-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 29/44] Documentation/technical: document object-format for protocol v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:54:09Z","receivedAt":"2020-05-13T00:56:06Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Document the object-format extension for protocol v2.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/protocol-v2.txt | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 7e3766cafb..107e421fb7 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -453,3 +453,12 @@ included in a request. This is done by sending each option as a\n a request.\n \n The provided options must not contain a NUL or LF character.\n+\n+ object-format\n+~~~~~~~~~~~~~~~\n+\n+The server can advertise the `object-format` capability with a value `X` (in the\n+form `object-format=X`) to notify the client that the server is able to deal\n+with objects using hash algorithm X. If not specified, the server is assumed to\n+only handle SHA-1. If the client would like to use a hash algorithm other than\n+SHA-1, it should specify its object-format string.\n"},{"id":"397693","messageId":"20200513005424.81369-18-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH 17/44] transport-helper: implement object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T00:53:57Z","receivedAt":"2020-05-13T00:56:09Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Implement the object-format extensions that let us determine the hash\nalgorithm in use when pushing or pulling data.\n\nSigned-off-by: brian m. carlson \n---\n transport-helper.c | 24 ++++++++++++++++++++++--\n 1 file changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex a46afcb69d..ae33b0eea7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -32,7 +32,8 @@ struct helper_data {\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n-\t\tno_private_update : 1;\n+\t\tno_private_update : 1,\n+\t\tobject_format : 1;\n \n \t/*\n \t * As an optimization, the transport code may invoke fetch before\n@@ -207,6 +208,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->import_marks = xstrdup(arg);\n \t\t} else if (starts_with(capname, \"no-private-update\")) {\n \t\t\tdata->no_private_update = 1;\n+\t\t} else if (starts_with(capname, \"object-format\")) {\n+\t\t\tdata->object_format = 1;\n \t\t} else if (mandatory) {\n \t\t\tdie(_(\"unknown mandatory capability %s; this remote \"\n \t\t\t \"helper probably needs newer version of Git\"),\n@@ -1103,6 +1106,12 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \tdata->get_refs_list_called = 1;\n \thelper = get_helper(transport);\n \n+\tif (data->object_format) {\n+\t\twrite_str_in_full(helper->in, \"option object-format\\n\");\n+\t\tif (recvline(data, &buf) || strcmp(buf.buf, \"ok\"))\n+\t\t\texit(128);\n+\t}\n+\n \tif (data->push && for_push)\n \t\twrite_str_in_full(helper->in, \"list for-push\\n\");\n \telse\n@@ -1115,6 +1124,17 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \n \t\tif (!*buf.buf)\n \t\t\tbreak;\n+\t\telse if (buf.buf[0] == ':') {\n+\t\t\tconst char *value;\n+\t\t\tif (skip_prefix(buf.buf, \":object-format \", &value)) {\n+\t\t\t\tint algo = hash_algo_by_name(value);\n+\t\t\t\tif (algo == GIT_HASH_UNKNOWN)\n+\t\t\t\t\tdie(_(\"unsupported object format '%s'\"),\n+\t\t\t\t\t value);\n+\t\t\t\ttransport->hash_algo = &hash_algos[algo];\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n \n \t\teov = strchr(buf.buf, ' ');\n \t\tif (!eov)\n@@ -1127,7 +1147,7 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \t\tif (buf.buf[0] == '@')\n \t\t\t(*tail)->symref = xstrdup(buf.buf + 1);\n \t\telse if (buf.buf[0] != '?')\n-\t\t\tget_oid_hex(buf.buf, &(*tail)->old_oid);\n+\t\t\tget_oid_hex_algop(buf.buf, &(*tail)->old_oid, transport->hash_algo);\n \t\tif (eon) {\n \t\t\tif (has_attribute(eon + 1, \"unchanged\")) {\n \t\t\t\t(*tail)->status |= REF_STATUS_UPTODATE;\n"},{"id":"397739","messageId":"CAN0heSqYsTZDnTG+roZ9pydUcEXJBayWXnijOcAZqb3-Rqegjg@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-3-sandals@crustytoothpaste.net","subject":"Re: [PATCH 02/44] Documentation: document v1 protocol object-format capability","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:28:46Z","receivedAt":"2020-05-13T19:29:00Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n\n> @@ -189,7 +204,6 @@ refs being sent.\n>\n> Clients MAY use the parameters from this capability to select the proper initial\n> branch when cloning a repository.\n> -\n> shallow\n> -------\n\nLooks like a spurious line deletion snuck in.\n\n\nMartin\n"},{"id":"397741","messageId":"CAN0heSrtV15dxhKMeePbbD42FzWbnw2kRe+nKwuzzJJszvwi0A@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-4-sandals@crustytoothpaste.net","subject":"Re: [PATCH 03/44] connect: have ref processing code take struct packet_reader","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:30:23Z","receivedAt":"2020-05-13T19:30:37Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> In a future patch, we'll want to access multiple members from struct\n> packet_reader when parsing references. Therefore, have the ref parsing\n> code take pointers to struct reader instead of having to pass multiple\n> arguments to each function.\n\nMakes sense.\n\n> -static void process_capabilities(const char *line, int *len)\n> +static void process_capabilities(struct packet_reader *reader, int *len)\n> {\n> + const char *line = reader->line;\n> int nul_location = strlen(line);\n> if (nul_location == *len)\n> return;\n\n\"line+len\" made it pretty obvious that they belonged together.\n\"reader+len\" not so much. Your patch does minimize the change. Would\ns/len/linelen/ be worth the extra churn? Possibly not. Right now, at\nleast we're pretty consistent about using \"len\" -- if this ends up as a\nmixture of \"linelen\" and \"len\" I think it's worse, overall.\n\n\nMartin\n"},{"id":"397742","messageId":"CAN0heSqBeW86yyH6vUATX+Az=4qG0gW_QhgsbWnTKXX=Q9hiig@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-5-sandals@crustytoothpaste.net","subject":"Re: [PATCH 04/44] wrapper: add function to compare strings with different NUL termination","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:32:38Z","receivedAt":"2020-05-13T19:32:52Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index 8ba576e81e..6503deb171 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -868,6 +868,8 @@ char *xgetcwd(void);\n> FILE *fopen_for_writing(const char *path);\n> FILE *fopen_or_warn(const char *path, const char *mode);\n>\n> +int xstrncmpz(const char *s, const char *t, size_t len);\n> +\n> /*\n> * FREE_AND_NULL(ptr) is like free(ptr) followed by ptr = NULL. Note\n> * that ptr is used twice, so don't pass e.g. ptr++.\n> diff --git a/wrapper.c b/wrapper.c\n> index 3a1c0e0526..15a09740e7 100644\n> --- a/wrapper.c\n> +++ b/wrapper.c\n> @@ -430,6 +430,18 @@ int xmkstemp(char *filename_template)\n> return fd;\n> }\n>\n> +/*\n> + * Like strncmp, but only return zero if s is NUL-terminated and exactly len\n> + * characters long. If it is not, consider it greater than t.\n> + */\n\nI think this comment would be easier to find in the .h file.\n\nAnd since I'm already commenting...\n\n> +int xstrncmpz(const char *s, const char *t, size_t len)\n> +{\n> + int res = strncmp(s, t, len);\n> + if (res)\n> + return res;\n> + return s[len] == '\\0' ? 0 : 1;\n> +}\n> +\n> /* Adapted from libiberty's mkstemp.c. */\n>\n> #undef TMP_MAX\n\nIt's not entirely obvious from the context, but this function is\ninserted between some \"tmp\" stuff and some other \"tmp\" stuff. I don't\nthink we need to bikeshed its exact home, but maybe \"close to other\nstring stuff\", or at least not in the middle of the \"tmp\" section.\n\n\nMartin\n"},{"id":"397744","messageId":"CAN0heSq1ULyJE6FLsNMc-V+J1ke8ZqyVF-MghK_d+0i7SSfTDA@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-8-sandals@crustytoothpaste.net","subject":"Re: [PATCH 07/44] connect: add function to fetch value of a v2 server capability","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:37:17Z","receivedAt":"2020-05-13T19:37:30Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:58, brian m. carlson\n wrote:\n> +int server_feature_v2(const char *c, const char **v)\n> +{\n> + int i;\n> +\n> + for (i = 0; i < server_capabilities_v2.argc; i++) {\n> + const char *out;\n> + if (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n> + (*out == '=')) {\n> + *v = out + 1;\n> + return 1;\n> + }\n> + }\n> + return 0;\n> +}\n> +\n\nThis looks like it was based on `server_supports_feature()`, which\nexplains the \"1 means yup got it, 0 means no match\". The name of\n`server_supports_feature()` does suggest the boolean nature of return\nvalue. For this new function, I would perhaps have expected \"0 means\nsuccess, negative means error\". That said, I'm not familiar with\nconnect.c. Let's see how this is used...\n\n> int server_supports_feature(const char *c, const char *feature,\n> int die_on_error)\n> {\n\nJust a thought:\n\nMaybe this existing function could learn to take a pointer (or NULL) and\nassign to it if we have a '=' (possibly even requiring a '=' if this new\npointer is non-NULL). I dunno, maybe two similar functions are better\nafter all than having one with modes like that.\n\n\nMartin\n"},{"id":"397745","messageId":"CAN0heSosiP=vCcd1QBmWNsVUuHTmRJbjGrWV9tU0=BSU1ENmDQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-11-sandals@crustytoothpaste.net","subject":"Re: [PATCH 10/44] connect: add function to detect supported v1 hash functions","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:39:41Z","receivedAt":"2020-05-13T19:39:54Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n> Add a function, server_supports_hash, to see if the remote server\n> supports a particular hash algorithm when speaking protocol v1.\n\n> +int server_supports_hash(const char *desired, int *feature_supported)\n> +{\n> + int offset = 0;\n> + int len, found = 0;\n> + const char *hash;\n> +\n> + hash = next_server_feature_value(\"object-format\", &len, &offset);\n> + if (feature_supported)\n> + *feature_supported = !!hash;\n\nIf we got something, anything, the server supports this feature. It just\nremains to see if it supports the exact algorithm we're after.\n\n> + if (!hash) {\n> + hash = hash_algos[GIT_HASH_SHA1].name;\n> + len = strlen(hash);\n> + }\n\nOK, if the server doesn't say anything, we fall back to SHA-1. If it's\nthe desired one, we'll return 1 accordingly below.\n\n> + while (hash) {\n> + if (!xstrncmpz(desired, hash, len))\n> + found = 1;\n> +\n> + if (found)\n> + return 1;\n\nI first thought this structure was because this loop body would learn to\ndo something else later in the series. But this is it. This looks like\nit could just be \"if (!xstrncmpz(...)) return 1;\" and drop \"found\".\n\n\n\n> + hash = next_server_feature_value(\"object-format\", &len, &offset);\n> + }\n> + return 0;\n> +}\n\nMartin\n"},{"id":"397746","messageId":"CAN0heSqvy_y7tNkfs_499vfYL6iOvE6nH5s_42WOHAT-5iSgUQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-12-sandals@crustytoothpaste.net","subject":"Re: [PATCH 11/44] send-pack: detect when the server doesn't support our hash","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:41:15Z","receivedAt":"2020-05-13T19:41:28Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:58, brian m. carlson\n wrote:\n>\n> Detect when the server doesn't support our hash algorithm and abort.\n\n> + if (!server_supports_hash(the_hash_algo->name, &object_format_supported))\n> + die(_(\"the receiving end does not support this repository's hash algorithm\"));\n\nI suppose this isn't the long-term wanted behavior? Would this be where\nwe would later learn to realize that \"oh, crap, we need to\nconvert/translate on the fly\"?\n\n> @@ -428,6 +432,8 @@ int send_pack(struct send_pack_args *args,\n> strbuf_addstr(&cap_buf, \" atomic\");\n> if (use_push_options)\n> strbuf_addstr(&cap_buf, \" push-options\");\n> + if (object_format_supported)\n> + strbuf_addf(&cap_buf, \" object-format=%s\", the_hash_algo->name);\n\nThis isn't advertised in the log message: If we do detect support, go on\nto reply with our choice of object format / hash algo name.\n\n\nMartin\n"},{"id":"397749","messageId":"CAN0heSrS+xMqhcUTZjhzq_EX0Zwm-PO9Yu6ue3q9dc_Z1VF3tQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-13-sandals@crustytoothpaste.net","subject":"Re: [PATCH 12/44] connect: make parse_feature_value extern","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-13T19:48:20Z","receivedAt":"2020-05-13T19:48:34Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> We're going to be using this function in other files, so no longer mark\n> this function static.\n>\n> static char *server_capabilities_v1;\n> static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n> -static const char *parse_feature_value(const char *, const char *, int *, int *);\n> static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n\n> -static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n> +const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n> {\n\n> --- a/connect.h\n> +++ b/connect.h\n> +const char *parse_feature_value(const char *, const char *, int *, int *);\n\nThis \"char *, int *\" comes from the forward-declaration above, which is\nnow dropped. Now that this is a header file for everyone to use, I think\nthese parameters should be named, at least, but even better would be\nsome documentation. ;-)\n\nI'll stop reading here. I'm not familiar with the technical details here\n(i.e., where you'd be most interested in review), so I've just left some\nmore or less superficial comments.\n\nOne thing I've noticed is that there are relatively few tests so far. I\nsuppose it could be hard to trigger things before everything is properly\nplugged through. But maybe at least various error paths could be\nexercised already at this point, such as in the previous patch I\ncommented on.\n\nSo far I feel like I'm following along ok and I have a feeling I know\nwhere this is leading up to. Nicely done so far.\n\n\nMartin\n"},{"id":"397789","messageId":"20200513224912.GE6605@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSosiP=vCcd1QBmWNsVUuHTmRJbjGrWV9tU0=BSU1ENmDQ@mail.gmail.com","subject":"Re: [PATCH 10/44] connect: add function to detect supported v1 hash functions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T22:49:12Z","receivedAt":"2020-05-13T22:49:49Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-13 at 19:39:41, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:56, brian m. carlson\n> wrote:\n> > + while (hash) {\n> > + if (!xstrncmpz(desired, hash, len))\n> > + found = 1;\n> > +\n> > + if (found)\n> > + return 1;\n> \n> I first thought this structure was because this loop body would learn to\n> do something else later in the series. But this is it. This looks like\n> it could just be \"if (!xstrncmpz(...)) return 1;\" and drop \"found\".\n\nYeah, I think it could. I originally didn't have the helper and the\ncode was pretty hideous, so I probably forgot to simplify when I used\nthe helper again.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397790","messageId":"20200513225221.GF6605@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSqvy_y7tNkfs_499vfYL6iOvE6nH5s_42WOHAT-5iSgUQ@mail.gmail.com","subject":"Re: [PATCH 11/44] send-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-13T22:52:21Z","receivedAt":"2020-05-13T22:52:29Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-13 at 19:41:15, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:58, brian m. carlson\n> wrote:\n> >\n> > Detect when the server doesn't support our hash algorithm and abort.\n> \n> > + if (!server_supports_hash(the_hash_algo->name, &object_format_supported))\n> > + die(_(\"the receiving end does not support this repository's hash algorithm\"));\n> \n> I suppose this isn't the long-term wanted behavior? Would this be where\n> we would later learn to realize that \"oh, crap, we need to\n> convert/translate on the fly\"?\n\nYes, this would be the point at which we'd decide whether we could\nsupport the remote side's algorithm and decide to rewrite objects. We\nmight still fail, such as if we're SHA-256 only without a lookup table\nand the remote side is SHA-1, but theoretically we'd do the conversion\nhere.\n\n> > @@ -428,6 +432,8 @@ int send_pack(struct send_pack_args *args,\n> > strbuf_addstr(&cap_buf, \" atomic\");\n> > if (use_push_options)\n> > strbuf_addstr(&cap_buf, \" push-options\");\n> > + if (object_format_supported)\n> > + strbuf_addf(&cap_buf, \" object-format=%s\", the_hash_algo->name);\n> \n> This isn't advertised in the log message: If we do detect support, go on\n> to reply with our choice of object format / hash algo name.\n\nI'll update the message.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397800","messageId":"xmqqh7wjqpzg.fsf@gitster.c.googlers.com","threadId":"53458","inReplyTo":"CAN0heSqYsTZDnTG+roZ9pydUcEXJBayWXnijOcAZqb3-Rqegjg@mail.gmail.com","subject":"Re: [PATCH 02/44] Documentation: document v1 protocol object-format capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-14T01:12:19Z","receivedAt":"2020-05-14T01:12:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n> On Wed, 13 May 2020 at 02:56, brian m. carlson\n> wrote:\n>\n>> @@ -189,7 +204,6 @@ refs being sent.\n>>\n>> Clients MAY use the parameters from this capability to select the proper initial\n>> branch when cloning a repository.\n>> -\n>> shallow\n>> -------\n>\n> Looks like a spurious line deletion snuck in.\n\nIndeed. I wonder if that is why our documentation build fails near\nthe tip of 'pu'.\n\n"},{"id":"397948","messageId":"20200515232212.GB6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"xmqqh7wjqpzg.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 02/44] Documentation: document v1 protocol object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-15T23:22:12Z","receivedAt":"2020-05-15T23:22:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-14 at 01:12:19, Junio C Hamano wrote:\n> Martin Ågren writes:\n> \n> > On Wed, 13 May 2020 at 02:56, brian m. carlson\n> > wrote:\n> >\n> >> @@ -189,7 +204,6 @@ refs being sent.\n> >>\n> >> Clients MAY use the parameters from this capability to select the proper initial\n> >> branch when cloning a repository.\n> >> -\n> >> shallow\n> >> -------\n> >\n> > Looks like a spurious line deletion snuck in.\n> \n> Indeed. I wonder if that is why our documentation build fails near\n> the tip of 'pu'.\n\nI'll definitely do a reroll this weekend and kick a basic doc build off\nbefore I send it out.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397950","messageId":"xmqq5zcw9275.fsf@gitster.c.googlers.com","threadId":"53458","inReplyTo":"20200515232212.GB6362@camp.crustytoothpaste.net","subject":"Re: [PATCH 02/44] Documentation: document v1 protocol object-format capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-16T00:02:38Z","receivedAt":"2020-05-16T00:02:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On 2020-05-14 at 01:12:19, Junio C Hamano wrote:\n>> Martin Ågren writes:\n>> \n>> > On Wed, 13 May 2020 at 02:56, brian m. carlson\n>> > wrote:\n>> >\n>> >> @@ -189,7 +204,6 @@ refs being sent.\n>> >>\n>> >> Clients MAY use the parameters from this capability to select the proper initial\n>> >> branch when cloning a repository.\n>> >> -\n>> >> shallow\n>> >> -------\n>> >\n>> > Looks like a spurious line deletion snuck in.\n>> \n>> Indeed. I wonder if that is why our documentation build fails near\n>> the tip of 'pu'.\n>\n> I'll definitely do a reroll this weekend and kick a basic doc build off\n> before I send it out.\n\nFWIW, I've fixed it up on my end so the documentation build of 'pu'\nhas been working OK.\n"},{"id":"397960","messageId":"CAN0heSo=xUGwb8FLXYpcFXO+e9dqvpzR=h2Bs+HC_ubQdTQuWw@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-15-sandals@crustytoothpaste.net","subject":"Re: [PATCH 14/44] connect: detect algorithm when fetching refs","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T10:40:11Z","receivedAt":"2020-05-16T10:40:30Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:57, brian m. carlson\n wrote:\n>\n> If we're fetching refs, detect the hash algorithm and parse the refs\n> using that algorithm.\n\nAs the added documentation from patch 2 says, if there are multiple\n\"object-format\" capabilities, \"the first one given is the one used in\nthe ref advertisement\". And that's what you implement below.\n\nExplaining that in this commit message and/or referring to \"a recent\ncommit\" (patch 2) and/or adding that documentation here, not back then,\nwould have avoided some confusion on my part, and perhaps also for\nfuture readers.\n\nI don't have a strong opinion on which of those is better, I just think\nyou could somehow make that a bit clearer here.\n\n> static void process_capabilities(struct packet_reader *reader, int *len)\n> {\n> + const char *feat_val;\n> + int feat_len;\n> + int hash_algo;\n\nYou could reduce the scope of `hash_algo`.\n\n> const char *line = reader->line;\n> int nul_location = strlen(line);\n> if (nul_location == *len)\n> return;\n> server_capabilities_v1 = xstrdup(line + nul_location + 1);\n> *len = nul_location;\n> +\n> + feat_val = server_feature_value(\"object-format\", &feat_len);\n> + if (feat_val) {\n> + char *hash_name = xstrndup(feat_val, feat_len);\n> + hash_algo = hash_algo_by_name(hash_name);\n> + if (hash_algo != GIT_HASH_UNKNOWN)\n> + reader->hash_algo = &hash_algos[hash_algo];\n> + free(hash_name);\n> + }\n> }\n\nxstrndup is needed because we're not guaranteed a terminating NUL. You\nremember to call free afterwards. Ok.\n\nIf we don't get any \"object-format\", we do basically nothing here and\n`reader->hash_algo` will remain as whatever it already is. The docs from\npatch 2 promise that this will be handled as \"SHA-1\" -- would it be more\nrobust if we did a similar fallback dance as you do elsewhere?\n\n feat_val = ...;\n if (!feat_val) {\n feat_val = hash_algos[GIT_HASH_SHA1].name;\n feat_len = strlen(feat_val);\n }\n char *hash_name = ...\n ...\n\nYou do initialize `reader->hash_algo` in patch 8, so I don't think this\nchanges anything now. Maybe it's just premature future-proofing (if such\na thing exists).\n\n\n\nMartin\n"},{"id":"397961","messageId":"CAN0heSr3eAZrXnp8OoA6m1Hmfcgno7Dr2VK8m4jQqGHmgYySLA@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-16-sandals@crustytoothpaste.net","subject":"Re: [PATCH 15/44] builtin/receive-pack: detect when the server doesn't support our hash","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T10:41:02Z","receivedAt":"2020-05-16T10:41:15Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n> + const char *hash;\n> + int len = 0;\n\nMicronit: These new variables are used in tandem. I could see these as\n\"NULL, 0\" or both uninitialized, but this is a mixture. That's a really\nsmall nit, of course. (Maybe you needed to fight a compiler warning?)\n\n> + hash = parse_feature_value(feature_list, \"object-format\", &len, NULL);\n> + if (!hash) {\n> + hash = hash_algos[GIT_HASH_SHA1].name;\n> + len = strlen(hash);\n> + }\n> + if (xstrncmpz(the_hash_algo->name, hash, len))\n> + die(\"error: unsupported object format '%s'\", hash);\n\nOk, this is a familiar pattern by now: if we get nothing, behave as if\nwe got SHA-1.\n\n\nMartin\n"},{"id":"397962","messageId":"CAN0heSquFyhHQMiiGkkLfcZVFZs1SNV7QDhTjPY38rjy7sejXg@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-20-sandals@crustytoothpaste.net","subject":"Re: [PATCH 19/44] builtin/clone: initialize hash algorithm properly","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T10:48:45Z","receivedAt":"2020-05-16T10:48:59Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:57, brian m. carlson\n wrote:\n>\n> When performing a clone, we don't know what hash algorithm the other end\n> will support. Currently, we don't support fetching data belonging to a\n> different algorithm, so we must know what algorithm the remote side is\n> using in order to properly initialize the repository. We can know that\n> only after fetching the refs, so if the remote side has any references,\n> use that information to reinitialize the repository with the correct\n> hash algorithm information.\n>\n> Signed-off-by: brian m. carlson \n> ---\n> builtin/clone.c | 9 +++++++++\n> 1 file changed, 9 insertions(+)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index cb48a291ca..f27d38bc8e 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -1217,6 +1217,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> refs = transport_get_remote_refs(transport, &ref_prefixes);\n>\n> if (refs) {\n> + int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n> +\n> + /*\n> + * Now that we know what algorithm the remote side is using,\n> + * let's set ours to the same thing.\n> + */\n> + initialize_repository_version(hash_algo);\n\nThis made me go \"huh\". It's not really new in this series, it's just\nthat from `initialize_repository_version(int)` I would have expected the\nargument to be the, well, repository version, not a hash algo\nidentifier. But it all makes sense once you realize that the function is\n\"please initialize the repository version based on this stuff that I\ngive you\" where, currently, the only input is a hash algo. (I see that\nHan-Wen's reftable series adds another parameter here.)\n\n> + repo_set_hash_algo(the_repository, hash_algo);\n\nI first wondered whether all calls to `repo_set_hash_algo()` would want\nto be preceded by `initialize_repository_version()`, which might call\nfor the latter being called by the former. But I guess not. Various\nusers of `repo_set_hash_algo()` -- not that there would be a lot of them\n-- might want to do similar updating and/or sanity checks, but the exact\ndetails would differ.\n\n\nMartin\n"},{"id":"397964","messageId":"CAN0heSqVrOTpSpptqe+zu+9-qQVYAL=jcx7tas3pNRcSVH+auQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-21-sandals@crustytoothpaste.net","subject":"Re: [PATCH 20/44] t5562: pass object-format in synthesized test data","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T10:55:33Z","receivedAt":"2020-05-16T10:55:46Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> Ensure that we pass the object-format capability in the synthesized test\n> data so that this test works with algorithms other than SHA-1.\n\nRight.\n\n> In addition, add an additional test using the old data for when we're\n> using SHA-1 so that we can be sure that we preserve backwards\n> compatibility with servers not offering the object-format capability.\n\nI'll have some questions on this below.\n\n> @@ -62,8 +63,8 @@ test_expect_success 'setup' '\n> test_copy_bytes 10 fetch_body.trunc &&\n> hash_next=$(git commit-tree -p HEAD -m next HEAD^{tree}) &&\n> {\n> - printf \"%s %s refs/heads/newbranch\\\\0report-status\\\\n\" \\\n> - \"$ZERO_OID\" \"$hash_next\" | packetize &&\n> + printf \"%s %s refs/heads/newbranch\\\\0report-status object-format=%s\\\\n\" \\\n> + \"$ZERO_OID\" \"$hash_next\" \"$(test_oid algo)\" | packetize &&\n> printf 0000 &&\n> echo \"$hash_next\" | git pack-objects --stdout\n> } >push_body &&\n\nMakes sense.\n\n> @@ -117,6 +118,15 @@ test_expect_success GZIP 'push plain' '\n> test_cmp act.head exp.head\n> '\n>\n> +test_expect_success GZIP 'push plain with SHA-1' '\n> + test_when_finished \"git branch -D newbranch\" &&\n> + test_http_env receive push_body &&\n> + verify_http_result \"200 OK\" &&\n> + git rev-parse newbranch >act.head &&\n> + echo \"$hash_next\" >exp.head &&\n> + test_cmp act.head exp.head\n> +'\n> +\n\nHmmm. Isn't this an exact copy of the 'push plain' test immediately\npreceding it? The commit message talks about using the \"old data\"\n(i.e., without \"object-format=%s\"?). Should this test use a variant of\npush_body where we're not adding \"object-format\"? I'm not sure I grok\nwhat exactly we want to test here.. And does it really belong in\nt/t*-content-length.sh?\n\n\nMartin\n"},{"id":"397965","messageId":"CAN0heSqf8E9cXvJcDJmx_YU5Eiwnqre-g3MwU3UMe40SioCNPA@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-22-sandals@crustytoothpaste.net","subject":"Re: [PATCH 21/44] t5704: send object-format capability with SHA-256","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:02:48Z","receivedAt":"2020-05-16T11:03:01Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> When we speak protocol v2 in this test, we must pass the object-format\n> header if the algorithm is not SHA-1. Otherwise, git upload-pack fails\n> because the hash algorithm doesn't match and not because we've failed to\n> speak the protocol correctly. Pass the header so that our assertions\n> test what we're really interested in.\n\n> +# If we don't print the object format, we'll fail for a spurious reason: the\n> +# mismatched object format.\n> +print_object_format () {\n> + local algo=$(test_oid algo) &&\n> + if test \"$algo\" != \"sha1\"\n> + then\n> + packetize \"object-format=$algo\"\n> + fi\n> +}\n> +\n> test_expect_success 'extra delim packet in v2 ls-refs args' '\n> {\n> packetize command=ls-refs &&\n> + print_object_format &&\n> printf 0001 &&\n> # protocol expects 0000 flush here\n> printf 0001\n> @@ -21,6 +32,7 @@ test_expect_success 'extra delim packet in v2 ls-refs args' '\n> test_expect_success 'extra delim packet in v2 fetch args' '\n> {\n> packetize command=fetch &&\n> + print_object_format &&\n> printf 0001 &&\n> # protocol expects 0000 flush here\n> printf 0001\n\nSo we need to pass this capability for the SHA-256 tests to run ok. But\nif we start passing \"object-format=sha1\" unconditionally at this point\nin the series, the tests will fail:\n\n error: 'grep expected flush after ls-refs arguments err' didn't find\na match in:\n fatal: unknown capability 'object-format=sha1'\n\nThat is, we don't yet actually implement \"object-format\" handling. So\nthis will still fail with SHA-256 (\"unknown capability\"), just that once\nthe implementation is in place, the SHA-256 tests will pass (as will the\nnormal SHA-1 runs). Do I understand that correctly?\n\nOr put differently, by the end of the series, we can do this:\n\ndiff --git a/t/t5704-protocol-violations.sh b/t/t5704-protocol-violations.sh\nindex 47e78932b9..22993812e2 100755\n--- a/t/t5704-protocol-violations.sh\n+++ b/t/t5704-protocol-violations.sh\n@@ -6,14 +6,11 @@ communications if the other side says something\nunexpected. We are mostly\n making sure that we do not segfault or otherwise behave badly.'\n . ./test-lib.sh\n\n-# If we don't print the object format, we'll fail for a spurious reason: the\n-# mismatched object format.\n+# If we don't print the object format, we might fail for a spurious reason:\n+# the mismatched object format.\n print_object_format () {\n local algo=$(test_oid algo) &&\n- if test \"$algo\" != \"sha1\"\n- then\n- packetize \"object-format=$algo\"\n- fi\n+ packetize \"object-format=$algo\"\n }\n\n test_expect_success 'extra delim packet in v2 ls-refs args' '\n\nShould we? (And if we do, we might as well drop this function and inline\nthe whole thing, IMHO.)\n\n\nMartin\n"},{"id":"397966","messageId":"CAN0heSrGJV4u2JJW98AbmQum-=KgDeBsDvRUU_RzbPHNfbcaUg@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-23-sandals@crustytoothpaste.net","subject":"Re: [PATCH 22/44] fetch-pack: parse and advertise the object-format capability","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:03:10Z","receivedAt":"2020-05-16T11:03:23Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> Parse the server's object-format capability and respond accordingly,\n> dying if there is a mismatch.\n\n> + if (server_feature_v2(\"object-format\", &hash_name)) {\n> + int hash_algo = hash_algo_by_name(hash_name);\n> + if (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n> + die(_(\"mismatched algorithms: client %s; server %s\"),\n> + the_hash_algo->name, hash_name);\n> + packet_write_fmt(fd_out, \"object-format=%s\", the_hash_algo->name);\n> + }\n> + else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1)\n> + die(_(\"the server does not support algorithm '%s'\"),\n> + the_hash_algo->name);\n\nMicronit: \"} else if (...) {\", i.e., join to one line and add braces.\n\n\nMartin\n"},{"id":"397967","messageId":"CAN0heSrMrXE2g4LaZ4A71CxGteVfE2RfsT6WyGzmfFSHFKjATQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-24-sandals@crustytoothpaste.net","subject":"Re: [PATCH 23/44] setup: set the_repository's hash algo when checking format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:03:56Z","receivedAt":"2020-05-16T11:04:09Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> When we're checking the repository's format, set the hash algorithm at\n> the same time. This ensures that we perform a suitable initialization\n> early enough to avoid confusing any parts of the code. If we defer\n> until later, we can end up with portions of the code which are confused\n> about the hash algorithm, resulting in segfaults.\n\nThis doesn't make a difference as long as you just use SHA-1, right?\nThat is, this isn't a bug in the first half of this series nor in\nv2.27-rc0 as long as you stick to SHA-1?\n\n\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -1273,6 +1273,7 @@ void check_repository_format(struct repository_format *fmt)\n> fmt = &repo_fmt;\n> check_repository_format_gently(get_git_dir(), fmt, NULL);\n> startup_info->have_repository = 1;\n> + repo_set_hash_algo(the_repository, fmt->hash_algo);\n> clear_repository_format(&repo_fmt);\n> }\n\nMartin\n"},{"id":"397968","messageId":"CAN0heSpKG_2F_WFP6c7WoppEwoQ7Pjbik152+8_hcz70C3i2MQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-25-sandals@crustytoothpaste.net","subject":"Re: [PATCH 24/44] t3200: mark assertion with SHA1 prerequisite","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:04:56Z","receivedAt":"2020-05-16T11:05:10Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> One of the test assertions in this test checks that git branch -m works\n> even without a .git/config file. However, if the repository requires\n> configuration extensions, such as because it uses a non-SHA-1 algorithm,\n> this assertion will fail. Mark the assertion as requiring SHA-1.\n\nMakes sense.\n\n> -test_expect_success 'git branch -m q q2 without config should succeed' '\n> +test_expect_success SHA1 'git branch -m q q2 without config should succeed' '\n> git branch -m q q2 &&\n> git branch -m q2 q\n> '\n\nGoing forward, we might need config files for other reasons (reftable?),\nmeaning this would become \"SHA1,!REFTABLE\". So maybe this should be\n\"!CONFIG_EXTENSIONS\" or \"CONFIG_LESS\". I think this is ok for now,\nthough. When/if someone needs to make another fix like this here -- or\nat the very least the *third* time around -- that's when we should think\na bit bigger.\n\n\nMartin\n"},{"id":"397970","messageId":"CAN0heSoozYnTJpiz3VZGq6XvrpmXenaf-5FiF=uts4Jhfb5c4g@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-26-sandals@crustytoothpaste.net","subject":"Re: [PATCH 25/44] packfile: compute and use the index CRC offset","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:12:26Z","receivedAt":"2020-05-16T11:12:39Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> Both v2 pack index files and the v3 format specified as part of the\n> NewHash work have similar data starting at the CRC table. Much of the\n> existing code wants to read either this table or the offset entries\n> following it, and in doing so computes the offset each time.\n>\n> In order to share as much code between v2 and v3, compute the offset of\n> the CRC table and store it when the pack is opened. Use this value to\n> compute offsets to not only the CRC table, but to the offset entries\n> beyond it.\n\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1555,13 +1555,9 @@ static void read_v2_anomalous_offsets(struct packed_git *p,\n> {\n> const uint32_t *idx1, *idx2;\n> uint32_t i;\n> - const uint32_t hashwords = the_hash_algo->rawsz / sizeof(uint32_t);\n>\n> /* The address of the 4-byte offset table */\n> - idx1 = (((const uint32_t *)p->index_data)\n> - + 2 /* 8-byte header */\n> - + 256 /* fan out */\n> - + hashwords * p->num_objects /* object ID table */\n> + idx1 = (((const uint32_t *)((const uint8_t *)p->index_data + p->crc_offset))\n> + p->num_objects /* CRC32 table */\n> );\n\nThis counts in four-byte words (so `+ 2` skips ahead 8B as the comment\nnotes). And that's why we need to use \"rawsz/4\".\n\nNot new in this patch, but that outer pair of parenthesis just makes\nthis harder to read, IMHO. I keep scanning back and forth wondering,\n\"where is this whole thing going to get multiplied or something?\"\n\n idx1 = (const uint32_t *)((const uint8_t *)p->index_data + p->crc_offset)\n + p->num_objects /* CRC32 table */;\n\nThe double-casting can be avoided with something like this, but I'm not\nsure it's really any better:\n\n idx1 = (const uint32_t *)p->index_data\n + p->crc_offset/sizeof(uint32_t)\n + p->num_objects /* CRC32 table */;\n\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -178,6 +178,7 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n> */\n> (sizeof(off_t) <= 4))\n> return error(\"pack too large for current definition of off_t in %s\", path);\n> + p->crc_offset = 8 + 4 * 256 + nr * hashsz;\n> }\n>\n> p->index_version = version;\n\nIt doesn't fit in the context, but `nr` will be assigned to\n`p->num_objects`. And now we can just use `hashsz` without dividing by\n4, so this does the same calculation as the old one above.\n\n\nMartin\n"},{"id":"397971","messageId":"CAN0heSqoGVAbfnDAG+PKnuCa81F=euGCBnf6Vk=a7wjvNc+i-A@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-31-sandals@crustytoothpaste.net","subject":"Re: [PATCH 30/44] connect: pass full packet reader when parsing v2 refs","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:13:39Z","receivedAt":"2020-05-16T11:13:53Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n> @@ -466,9 +467,10 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> }\n> packet_flush(fd_out);\n>\n> +\n\nAh, this is where that line from patch 2 went. ;-)\n\n\n> /* Process response from server */\n> while (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> - if (!process_ref_v2(reader->line, &list))\n> + if (!process_ref_v2(reader, &list))\n> die(_(\"invalid ls-refs response: %s\"), reader->line);\n> }\n\nMartin\n"},{"id":"397972","messageId":"CAN0heSpq-t1E6CBsLiVDUE8NYB2ES916MR=d99eug9U=vdOLHQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-32-sandals@crustytoothpaste.net","subject":"Re: [PATCH 31/44] connect: parse v2 refs with correct hash algorithm","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:14:16Z","receivedAt":"2020-05-16T11:14:29Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:58, brian m. carlson\n wrote:\n>\n> When using protocol v2, we need to know what hash algorithm is used by\n> the remote end. See if the server has sent us an object-format\n> capability, and if so, use it to determine the hash algorithm in use and\n> set that value in the packet reader. Parse the refs using this\n> algorithm.\n>\n> Note that we use memcpy instead of oidcpy for copying values, since\n> oidcpy is intentionally limited to the current hash algorithm length,\n> and the copy will be too short if the server side uses SHA-256 but the\n> client side has not had a repository set up (and therefore defaults to\n> SHA-1).\n\n> - oidcpy(&ref->old_oid, &old_oid);\n> + memcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n\nMight an `oidcpy_algop()` prove useful over time?\n\n oidcpy_algop(&ref->old_oid, &old_oid, reader->hash_algo);\n\n> @@ -442,6 +444,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> const struct string_list *server_options)\n> {\n> int i;\n> + const char *hash_name;\n> *list = NULL;\n>\n> if (server_supports_v2(\"ls-refs\", 1))\n> @@ -450,6 +453,14 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> if (server_supports_v2(\"agent\", 0))\n> packet_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n>\n> + if (server_feature_v2(\"object-format\", &hash_name)) {\n> + int hash_algo = hash_algo_by_name(hash_name);\n> + if (hash_algo == GIT_HASH_UNKNOWN)\n> + die(_(\"unknown object format '%s' specified by server\"), hash_name);\n> + reader->hash_algo = &hash_algos[hash_algo];\n> + packet_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n> + }\n> +\n\n(Similar to an earlier comment I made, if we don't see any\n\"object-format\", we rely on `reader->hash_algo` to have been properly\nset up (which it has) and to not have been modified since (which we\ncould probably rely on, hmm?).)\n\n\n\nMartin\n"},{"id":"397973","messageId":"CAN0heSpNp1f=3p_ef-13547Sf6qkDXG1-i+0tDDX3VBhP72dgQ@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-33-sandals@crustytoothpaste.net","subject":"Re: [PATCH 32/44] serve: advertise object-format capability for protocol v2","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:15:25Z","receivedAt":"2020-05-16T11:15:39Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> In order to communicate the protocol supported by the server side, add\n> support for advertising the object-format capability. We check that the\n> client side sends us an identical algorithm if it sends us its own\n> object-format capability, and assume it speaks SHA-1 if not.\n>\n> In the test, when we're using an algorithm other than SHA-1, we need to\n> specify the algorithm in use so we don't get a failure with an \"unknown\n> format\" message. Add a wrapper function that specifies this header if\n> required. Skip specifying this header for SHA-1 to test that it works\n> both with and without this header.\n\nThis last sentence sort of answers an earlier question I made: should we\nstop special-casing in the test and just always write the capability? I\ncan see your point here, but it only applies if you actually go to the\ntrouble of running the tests both with SHA-1 and SHA-256, right?\n\nThat is, I wonder if we shouldn't always pass the \"object-format\"\ncapability in the tests and, if we have the SHA-1 prereq, execute a\ndedicated test where we do not pass it and verify that we default\ncorrectly. Hmm?\n\n> +write_command () {\n> + echo \"command=$1\"\n> +\n> + if test \"$(test_oid algo)\" != sha1\n> + then\n> + echo \"object-format=$(test_oid algo)\"\n> + fi\n> +}\n> +\n> test_expect_success 'test capability advertisement' '\n> + test_oid_init &&\n> cat >expect <<-EOF &&\n> version 2\n> agent=git/$(git version | cut -d\" \" -f3)\n> ls-refs\n> fetch=shallow\n> server-option\n> + object-format=$(test_oid algo)\n> 0000\n> EOF\n>\n> @@ -45,6 +56,7 @@ test_expect_success 'request invalid capability' '\n> test_expect_success 'request with no command' '\n> test-tool pkt-line pack >in <<-EOF &&\n> agent=git/test\n> + object-format=$(test_oid algo)\n> 0000\n> EOF\n> test_must_fail test-tool serve-v2 --stateless-rpc 2>err wrote:\n>\n> ls-remote may or may not operate within a repository, and as such will\n> not have been initialized with the repository's hash algorithm. Even if\n> it were, the remote side could be using a different algorithm and we\n> would still want to display those refs properly. Find the hash\n> algorithm used by the remote side by querying the transport object and\n> set our hash algorithm accordingly.\n>\n> Without this change, if the remote side is using SHA-256, we truncate\n> the refs to 40 hex characters, since that's the length of the default\n> hash algorithm (SHA-1).\n\nCould we add a test that passes now but would have failed before?\n\n> ref = transport_get_remote_refs(transport, &ref_prefixes);\n> + if (ref) {\n> + int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n> + repo_set_hash_algo(the_repository, hash_algo);\n> + }\n\nThis will modify `the_hash_algo`. Quoting commit 78a6766802 (\"Integrate\nhash algorithm support with repo setup\", 2017-11-12):\n\n Add a constant, the_hash_algo, which points to the hash_algo structure\n pointer in the repository global. Note that this is the hash which is\n used to serialize data to disk, not the hash which is used to display\n items to the user. The transition plan anticipates that these may be\n different. We can add an additional element in the future (say,\n ui_hash_algo) to provide for this case.\n\nDon't we violate that here? Is it mostly luck that we can go on to list\nwhat we want to list and that we will never write to disk based on\n`the_hash_algo` being \"wrong\"(?)? Or am I missing something?\n\n\nMartin\n"},{"id":"397975","messageId":"CAN0heSqcv6cTJcptOo=BgTwxcTkdL+mm-HKgX4UBnTscECJJTg@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-36-sandals@crustytoothpaste.net","subject":"Re: [PATCH 35/44] remote-curl: detect algorithm for dumb HTTP by size","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:17:06Z","receivedAt":"2020-05-16T11:17:19Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:57, brian m. carlson\n wrote:\n\n> + options.hash_algo = detect_hash_algo(heads);\n> + if (!options.hash_algo)\n> + die(\"%sinfo/refs not valid: could not determine hash algorithm; \"\n> + \"is this a git repository?\",\n> + url.buf);\n\nShould this use `transport_anonymize_url()`?\n\n> if (data[i] == '\\n') {\n> - if (mid - start != the_hash_algo->hexsz)\n> + if (mid - start != options.hash_algo->hexsz)\n> die(_(\"%sinfo/refs not valid: is this a git repository?\"),\n> transport_anonymize_url(url.buf));\n\nLike here and elsewhere.\n\n> data[i] = 0;\n> ref_name = mid + 1;\n> ref = alloc_ref(ref_name);\n> - get_oid_hex(start, &ref->old_oid);\n> + get_oid_hex_algop(start, &ref->old_oid, options.hash_algo);\n\nOther than that, looks ok.\n\n\n\nMartin\n"},{"id":"397976","messageId":"CAN0heSqpqAyKyaz+Er-SppR8k5W=zfw31rLR=Z8yZzqu=BCnTA@mail.gmail.com","threadId":"53458","inReplyTo":"20200513005424.81369-37-sandals@crustytoothpaste.net","subject":"Re: [PATCH 36/44] builtin/index-pack: add option to specify hash algorithm","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-16T11:18:12Z","receivedAt":"2020-05-16T11:18:27Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 13 May 2020 at 02:56, brian m. carlson\n wrote:\n>\n> git index-pack is usually run in a repository, but need not be. Since\n> packs don't contains information on the algorithm in use, instead\n> relying on context, add an option to index-pack to tell it which one\n> we're using in case someone runs it outside of a repository.\n>\n> Signed-off-by: brian m. carlson \n> ---\n> builtin/index-pack.c | 5 +++++\n> 1 file changed, 5 insertions(+)\n>\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index 7bea1fba52..89f4962a00 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1760,6 +1760,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n> die(_(\"bad %s\"), arg);\n> } else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n> max_input_size = strtoumax(arg, NULL, 10);\n> + } else if (skip_prefix(arg, \"--object-format=\", &arg)) {\n> + int hash_algo = hash_algo_by_name(arg);\n> + if (hash_algo == GIT_HASH_UNKNOWN)\n> + die(_(\"unknown hash algorithm '%s'\"), arg);\n> + repo_set_hash_algo(the_repository, hash_algo);\n> } else\n\nPatch 27 added `--hash` to `git show-index` and I almost commented on\n\"hash\" vs \"object-format\". In the end I figured the object format was a\nmore technical (protocol) term. But now I wonder. Should we try to align\nsuch options from the start? Or is there perhaps a reason for those\ndifferent approaches?\n\nSimilar to an earlier patch where we modify `the_hash_algo` like this, I\nfeel a bit nervous. What happens if you pass in a \"wrong\" algo here,\ni.e., SHA-1 in a SHA-256 repo? Or, given the motivation in the commit\nmessage, should this only be allowed if we really *are* outside a repo?\n\n\nMartin\n"},{"id":"397984","messageId":"20200516191452.GD6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSqf8E9cXvJcDJmx_YU5Eiwnqre-g3MwU3UMe40SioCNPA@mail.gmail.com","subject":"Re: [PATCH 21/44] t5704: send object-format capability with SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-16T19:14:52Z","receivedAt":"2020-05-16T19:16:51Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 11:02:48, Martin Ågren wrote:\n> So we need to pass this capability for the SHA-256 tests to run ok. But\n> if we start passing \"object-format=sha1\" unconditionally at this point\n> in the series, the tests will fail:\n> \n> error: 'grep expected flush after ls-refs arguments err' didn't find\n> a match in:\n> fatal: unknown capability 'object-format=sha1'\n> \n> That is, we don't yet actually implement \"object-format\" handling. So\n> this will still fail with SHA-256 (\"unknown capability\"), just that once\n> the implementation is in place, the SHA-256 tests will pass (as will the\n> normal SHA-1 runs). Do I understand that correctly?\n\nYes, that's correct.\n\n> Or put differently, by the end of the series, we can do this:\n> \n> diff --git a/t/t5704-protocol-violations.sh b/t/t5704-protocol-violations.sh\n> index 47e78932b9..22993812e2 100755\n> --- a/t/t5704-protocol-violations.sh\n> +++ b/t/t5704-protocol-violations.sh\n> @@ -6,14 +6,11 @@ communications if the other side says something\n> unexpected. We are mostly\n> making sure that we do not segfault or otherwise behave badly.'\n> . ./test-lib.sh\n> \n> -# If we don't print the object format, we'll fail for a spurious reason: the\n> -# mismatched object format.\n> +# If we don't print the object format, we might fail for a spurious reason:\n> +# the mismatched object format.\n> print_object_format () {\n> local algo=$(test_oid algo) &&\n> - if test \"$algo\" != \"sha1\"\n> - then\n> - packetize \"object-format=$algo\"\n> - fi\n> + packetize \"object-format=$algo\"\n> }\n> \n> test_expect_success 'extra delim packet in v2 ls-refs args' '\n> \n> Should we? (And if we do, we might as well drop this function and inline\n> the whole thing, IMHO.)\n\nWe certainly can. I'll move this later on in the series so that we\ncan simplify the code.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397985","messageId":"20200516192955.GE6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSrMrXE2g4LaZ4A71CxGteVfE2RfsT6WyGzmfFSHFKjATQ@mail.gmail.com","subject":"Re: [PATCH 23/44] setup: set the_repository's hash algo when checking format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-16T19:29:55Z","receivedAt":"2020-05-16T19:30:35Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 11:03:56, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:56, brian m. carlson\n> wrote:\n> >\n> > When we're checking the repository's format, set the hash algorithm at\n> > the same time. This ensures that we perform a suitable initialization\n> > early enough to avoid confusing any parts of the code. If we defer\n> > until later, we can end up with portions of the code which are confused\n> > about the hash algorithm, resulting in segfaults.\n> \n> This doesn't make a difference as long as you just use SHA-1, right?\n> That is, this isn't a bug in the first half of this series nor in\n> v2.27-rc0 as long as you stick to SHA-1?\n\nCorrect, because the default is SHA-1 if no algorithm is specified.\nI'll update the commit message to reflect that this affects only\nSHA-256.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397987","messageId":"20200516195001.GF6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSqVrOTpSpptqe+zu+9-qQVYAL=jcx7tas3pNRcSVH+auQ@mail.gmail.com","subject":"Re: [PATCH 20/44] t5562: pass object-format in synthesized test data","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-16T19:50:01Z","receivedAt":"2020-05-16T19:50:39Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 10:55:33, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:56, brian m. carlson\n> wrote:\n> >\n> > Ensure that we pass the object-format capability in the synthesized test\n> > data so that this test works with algorithms other than SHA-1.\n> \n> Right.\n> \n> > In addition, add an additional test using the old data for when we're\n> > using SHA-1 so that we can be sure that we preserve backwards\n> > compatibility with servers not offering the object-format capability.\n> \n> I'll have some questions on this below.\n\nI think this got dropped in the rebase.\n\n> Hmmm. Isn't this an exact copy of the 'push plain' test immediately\n> preceding it? The commit message talks about using the \"old data\"\n> (i.e., without \"object-format=%s\"?). Should this test use a variant of\n> push_body where we're not adding \"object-format\"? I'm not sure I grok\n> what exactly we want to test here.. And does it really belong in\n> t/t*-content-length.sh?\n\nIt is. I'll probably drop this part of the patch.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397988","messageId":"20200516195941.GG6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSo=xUGwb8FLXYpcFXO+e9dqvpzR=h2Bs+HC_ubQdTQuWw@mail.gmail.com","subject":"Re: [PATCH 14/44] connect: detect algorithm when fetching refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-16T19:59:41Z","receivedAt":"2020-05-16T20:01:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 10:40:11, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:57, brian m. carlson\n> wrote:\n> >\n> > If we're fetching refs, detect the hash algorithm and parse the refs\n> > using that algorithm.\n> \n> As the added documentation from patch 2 says, if there are multiple\n> \"object-format\" capabilities, \"the first one given is the one used in\n> the ref advertisement\". And that's what you implement below.\n> \n> Explaining that in this commit message and/or referring to \"a recent\n> commit\" (patch 2) and/or adding that documentation here, not back then,\n> would have avoided some confusion on my part, and perhaps also for\n> future readers.\n\nI'll try to reword to improve things.\n\n> > static void process_capabilities(struct packet_reader *reader, int *len)\n> > {\n> > + const char *feat_val;\n> > + int feat_len;\n> > + int hash_algo;\n> \n> You could reduce the scope of `hash_algo`.\n\nCan do.\n\n> > const char *line = reader->line;\n> > int nul_location = strlen(line);\n> > if (nul_location == *len)\n> > return;\n> > server_capabilities_v1 = xstrdup(line + nul_location + 1);\n> > *len = nul_location;\n> > +\n> > + feat_val = server_feature_value(\"object-format\", &feat_len);\n> > + if (feat_val) {\n> > + char *hash_name = xstrndup(feat_val, feat_len);\n> > + hash_algo = hash_algo_by_name(hash_name);\n> > + if (hash_algo != GIT_HASH_UNKNOWN)\n> > + reader->hash_algo = &hash_algos[hash_algo];\n> > + free(hash_name);\n> > + }\n> > }\n> \n> xstrndup is needed because we're not guaranteed a terminating NUL. You\n> remember to call free afterwards. Ok.\n> \n> If we don't get any \"object-format\", we do basically nothing here and\n> `reader->hash_algo` will remain as whatever it already is. The docs from\n> patch 2 promise that this will be handled as \"SHA-1\" -- would it be more\n> robust if we did a similar fallback dance as you do elsewhere?\n> \n> feat_val = ...;\n> if (!feat_val) {\n> feat_val = hash_algos[GIT_HASH_SHA1].name;\n> feat_len = strlen(feat_val);\n> }\n> char *hash_name = ...\n> ...\n\nYeah, I can do that.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397989","messageId":"20200516202855.GH6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSrLxatUNbJPZDmMC=NtF4R_AZF8gF-pUwJ+uQFMp3A6-w@mail.gmail.com","subject":"Re: [PATCH 34/44] builtin/ls-remote: initialize repository based on fetch","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-16T20:28:55Z","receivedAt":"2020-05-16T20:29:03Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 11:16:46, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:58, brian m. carlson\n> wrote:\n> >\n> > ls-remote may or may not operate within a repository, and as such will\n> > not have been initialized with the repository's hash algorithm. Even if\n> > it were, the remote side could be using a different algorithm and we\n> > would still want to display those refs properly. Find the hash\n> > algorithm used by the remote side by querying the transport object and\n> > set our hash algorithm accordingly.\n> >\n> > Without this change, if the remote side is using SHA-256, we truncate\n> > the refs to 40 hex characters, since that's the length of the default\n> > hash algorithm (SHA-1).\n> \n> Could we add a test that passes now but would have failed before?\n\nThe existing tests that call \"git ls-remote\" actually fail with SHA-256\nif we don't do this, specifically \"ls-remote works outside repository\"\nin t5512. That's the thing with a lot of this series: our existing test\nsuite is enormously effective at catching these things, but writing a\nnew test is hard because we can't actually instantiate a SHA-256\nrepository (because then users could, and it's broken until the end of\nthe series). Perhaps unsurprisingly, that's how I found this problem.\n\nSo while I would love to write a test for this case, I can't without\nallowing users to corrupt and destroy their data in the mean time (or\ntacking the final six commits to this series).\n\n> > ref = transport_get_remote_refs(transport, &ref_prefixes);\n> > + if (ref) {\n> > + int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n> > + repo_set_hash_algo(the_repository, hash_algo);\n> > + }\n> \n> This will modify `the_hash_algo`. Quoting commit 78a6766802 (\"Integrate\n> hash algorithm support with repo setup\", 2017-11-12):\n> \n> Add a constant, the_hash_algo, which points to the hash_algo structure\n> pointer in the repository global. Note that this is the hash which is\n> used to serialize data to disk, not the hash which is used to display\n> items to the user. The transition plan anticipates that these may be\n> different. We can add an additional element in the future (say,\n> ui_hash_algo) to provide for this case.\n> \n> Don't we violate that here? Is it mostly luck that we can go on to list\n> what we want to list and that we will never write to disk based on\n> `the_hash_algo` being \"wrong\"(?)? Or am I missing something?\n\nWe do violate that and we also rely on it never having any effect on our\ncurrent repository. Unfortunately, as things stand now, we don't\nsupport multiple hash algorithms in the same running binary, and we\ncan't until we allow a member of struct object_id to vary based on the\nhash algorithm. That work is coming in a future series (after we have a\nfully functioning SHA-256 stage 4 implementation), but at this point,\nI'm still working through all of the crashes we get from random places\nwhere we make assumptions about initializing things, so it's not a\nstraightforward fix.\n\nFor now, I think this is the best we can do without major additional\nsurgery to the codebase. I'm fine with stating that git ls-remote can\nread the repository (to parse remotes) but can't write to it, since\nthat's the behavior users will expect anyway. I'll update the commit\nmessage to reflect that wart and assumption, since it would be good to\ndocument it.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"397990","messageId":"20200516204710.GI6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSqpqAyKyaz+Er-SppR8k5W=zfw31rLR=Z8yZzqu=BCnTA@mail.gmail.com","subject":"Re: [PATCH 36/44] builtin/index-pack: add option to specify hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-16T20:47:10Z","receivedAt":"2020-05-16T20:47:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 11:18:12, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:56, brian m. carlson\n> wrote:\n> >\n> > git index-pack is usually run in a repository, but need not be. Since\n> > packs don't contains information on the algorithm in use, instead\n> > relying on context, add an option to index-pack to tell it which one\n> > we're using in case someone runs it outside of a repository.\n> >\n> > Signed-off-by: brian m. carlson \n> > ---\n> > builtin/index-pack.c | 5 +++++\n> > 1 file changed, 5 insertions(+)\n> >\n> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> > index 7bea1fba52..89f4962a00 100644\n> > --- a/builtin/index-pack.c\n> > +++ b/builtin/index-pack.c\n> > @@ -1760,6 +1760,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n> > die(_(\"bad %s\"), arg);\n> > } else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n> > max_input_size = strtoumax(arg, NULL, 10);\n> > + } else if (skip_prefix(arg, \"--object-format=\", &arg)) {\n> > + int hash_algo = hash_algo_by_name(arg);\n> > + if (hash_algo == GIT_HASH_UNKNOWN)\n> > + die(_(\"unknown hash algorithm '%s'\"), arg);\n> > + repo_set_hash_algo(the_repository, hash_algo);\n> > } else\n> \n> Patch 27 added `--hash` to `git show-index` and I almost commented on\n> \"hash\" vs \"object-format\". In the end I figured the object format was a\n> more technical (protocol) term. But now I wonder. Should we try to align\n> such options from the start? Or is there perhaps a reason for those\n> different approaches?\n\nI'll bring them into sync.\n\n> Similar to an earlier patch where we modify `the_hash_algo` like this, I\n> feel a bit nervous. What happens if you pass in a \"wrong\" algo here,\n> i.e., SHA-1 in a SHA-256 repo? Or, given the motivation in the commit\n> message, should this only be allowed if we really *are* outside a repo?\n\nUnfortunately, we can't prevent the user from being inside repository A,\nwhich is SHA-1, while invoking git index-pack on repository B, which is\nSHA-256. That is valid without --stdin, if uncommon, and it needs to be\nsupported. I can prevent it from being used with --stdin, though.\n\nIf you pass in a wrong algorithm, we usually blow up with an inflate\nerror because we consume more bytes than expected with our ref deltas.\nI'm not aware of any cases where we segfault or access invalid memory;\nwe just blow up in a nonobvious way. That's true, too, if you manually\ntamper with the algorithm in extensions.objectformat; usually we blow up\n(but not segfault) because the index is \"corrupt\".\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"398015","messageId":"CAN0heSqXSPXG38aqQggxA6yjkg_+PVVdh3M01RQKJM0gO0wAPA@mail.gmail.com","threadId":"53458","inReplyTo":"20200516204710.GI6362@camp.crustytoothpaste.net","subject":"Re: [PATCH 36/44] builtin/index-pack: add option to specify hash algorithm","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-05-17T18:16:37Z","receivedAt":"2020-05-17T18:16:52Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Sat, 16 May 2020 at 22:47, brian m. carlson\n wrote:\n>\n> On 2020-05-16 at 11:18:12, Martin Ågren wrote:\n> > On Wed, 13 May 2020 at 02:56, brian m. carlson\n> > wrote:\n> > >\n> > > git index-pack is usually run in a repository, but need not be. Since\n> > > packs don't contains information on the algorithm in use, instead\n> > > relying on context, add an option to index-pack to tell it which one\n> > > we're using in case someone runs it outside of a repository.\n>\n> > Similar to an earlier patch where we modify `the_hash_algo` like this, I\n> > feel a bit nervous. What happens if you pass in a \"wrong\" algo here,\n> > i.e., SHA-1 in a SHA-256 repo? Or, given the motivation in the commit\n> > message, should this only be allowed if we really *are* outside a repo?\n>\n> Unfortunately, we can't prevent the user from being inside repository A,\n> which is SHA-1, while invoking git index-pack on repository B, which is\n> SHA-256.\n\nAh, I see.\n\n> That is valid without --stdin, if uncommon, and it needs to be\n> supported. I can prevent it from being used with --stdin, though.\n\nHmm, that might make sense. I suppose it could quickly get out of\ncontrol with bug reports coming in along the lines of \"if I do this\nreally crazy git index-pack invocation, I manage to mess things up\". The\neasiest way to address this might be through documentation, i.e., \"don't\nuse this option\", \"for internal use\" or even \"to be used by the test\nsuite only\" for which there is even precedence in git-index-pack(1).\n\nOn the other hand, if we need to detect such hash mismatch even once the\nSHA-256 work is 100% complete, then I suppose we really should try a\nbit to catch bad invocations.\n\nAs a tangent, I see that v2.27.0 will come with `git init\n--object-format=` and `GIT_DEFAULT_HASH_ALGORITHM`. The docs for\nthe former mentions \"(if enabled)\". Should we add something more scary\nto those to make it clear that they shouldn't be used and that you\nbasically shouldn't even try to figure out how to enable them? I can\nalready see the tweets and blog posts a few weeks from now about how you\ncan build Git from source setting a single switch, run\n\n git init --object-format=sha256\n\nand you're in the future! Which will just lead to pain some days or\nweeks later.... \"I've done lots of work. How do I convert my repo to\nSHA-1 so I can share it?\"...\n\nWe've added \"experimental\" things before and tried to document the\nexperimental nature. Maybe here we're not even \"experimental\" -- more\nlike \"if you use this in production, you *will* suffer\"?\n\n> If you pass in a wrong algorithm, we usually blow up with an inflate\n> error because we consume more bytes than expected with our ref deltas.\n> I'm not aware of any cases where we segfault or access invalid memory;\n> we just blow up in a nonobvious way. That's true, too, if you manually\n> tamper with the algorithm in extensions.objectformat; usually we blow up\n> (but not segfault) because the index is \"corrupt\".\n\nOk, I see. I suppose \"some time\", we could tweak error messages to hint\nabout an object-format mismatch, but I don't think that needs to block\nyour work here now.\n\nMartin\n"},{"id":"398030","messageId":"20200517205245.GJ6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSqXSPXG38aqQggxA6yjkg_+PVVdh3M01RQKJM0gO0wAPA@mail.gmail.com","subject":"Re: [PATCH 36/44] builtin/index-pack: add option to specify hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-17T20:52:45Z","receivedAt":"2020-05-17T20:52:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-17 at 18:16:37, Martin Ågren wrote:\n> On Sat, 16 May 2020 at 22:47, brian m. carlson\n> wrote:\n> > That is valid without --stdin, if uncommon, and it needs to be\n> > supported. I can prevent it from being used with --stdin, though.\n> \n> Hmm, that might make sense. I suppose it could quickly get out of\n> control with bug reports coming in along the lines of \"if I do this\n> really crazy git index-pack invocation, I manage to mess things up\". The\n> easiest way to address this might be through documentation, i.e., \"don't\n> use this option\", \"for internal use\" or even \"to be used by the test\n> suite only\" for which there is even precedence in git-index-pack(1).\n> \n> On the other hand, if we need to detect such hash mismatch even once the\n> SHA-256 work is 100% complete, then I suppose we really should try a\n> bit to catch bad invocations.\n\nI can add documentation and a warning there.\n\nIf we actually verified the checksum at the end of the pack first, then\nwe'd be able to distinguish the two cases, because we'd try to compute a\nclearly invalid hash over the body, and the likelihood of it matching\nwould be very small. We don't at the moment, for reasons I'm unclear\nabout, but it's probably performance.\n\n> As a tangent, I see that v2.27.0 will come with `git init\n> --object-format=` and `GIT_DEFAULT_HASH_ALGORITHM`. The docs for\n> the former mentions \"(if enabled)\". Should we add something more scary\n> to those to make it clear that they shouldn't be used and that you\n> basically shouldn't even try to figure out how to enable them? I can\n> already see the tweets and blog posts a few weeks from now about how you\n> can build Git from source setting a single switch, run\n> \n> git init --object-format=sha256\n> \n> and you're in the future! Which will just lead to pain some days or\n> weeks later.... \"I've done lots of work. How do I convert my repo to\n> SHA-1 so I can share it?\"...\n> \n> We've added \"experimental\" things before and tried to document the\n> experimental nature. Maybe here we're not even \"experimental\" -- more\n> like \"if you use this in production, you *will* suffer\"?\n\nWell, the option is there, but it produces the following:\n\n % git init --object-format=sha256\n fatal: The hash algorithm sha256 is not supported in this build.\n\nwhich can be distinguished from this:\n\n % git init --object-format=blake2b\n fatal: unknown hash algorithm 'blake2b'\n\nRight now it's pretty broken without this series, so you can't use it.\nI mean, you have the source and can remove the check, but it doesn't\nwork as it stands, so I'm not too worried about people trying to do that\nat the moment. I'll sneak in some documentation for the end product,\nthough.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"398033","messageId":"20200517223717.GL6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"CAN0heSpq-t1E6CBsLiVDUE8NYB2ES916MR=d99eug9U=vdOLHQ@mail.gmail.com","subject":"Re: [PATCH 31/44] connect: parse v2 refs with correct hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-17T22:37:17Z","receivedAt":"2020-05-17T22:37:57Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-16 at 11:14:16, Martin Ågren wrote:\n> On Wed, 13 May 2020 at 02:58, brian m. carlson\n> wrote:\n> >\n> > When using protocol v2, we need to know what hash algorithm is used by\n> > the remote end. See if the server has sent us an object-format\n> > capability, and if so, use it to determine the hash algorithm in use and\n> > set that value in the packet reader. Parse the refs using this\n> > algorithm.\n> >\n> > Note that we use memcpy instead of oidcpy for copying values, since\n> > oidcpy is intentionally limited to the current hash algorithm length,\n> > and the copy will be too short if the server side uses SHA-256 but the\n> > client side has not had a repository set up (and therefore defaults to\n> > SHA-1).\n> \n> > - oidcpy(&ref->old_oid, &old_oid);\n> > + memcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n> \n> Might an `oidcpy_algop()` prove useful over time?\n> \n> oidcpy_algop(&ref->old_oid, &old_oid, reader->hash_algo);\n\nI think I can just omit this chunk, because oidcpy now copies the entire\nstruct for speed.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"398092","messageId":"xmqqimgt5i61.fsf@gitster.c.googlers.com","threadId":"53458","inReplyTo":"20200513005424.81369-28-sandals@crustytoothpaste.net","subject":"Re: [PATCH 27/44] builtin/show-index: provide options to determine hash algo","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-18T16:20:22Z","receivedAt":"2020-05-18T16:20:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> It's possible to use a variety of index formats with show-index, and we\n> need a way to indicate the hash algorithm which is in use for a\n> particular index we'd like to show. Default to using the value for the\n> repository we're in by calling setup_git_directory_gently, and allow\n> overriding it by using a --hash argument.\n\nI think you meant to say that \"show-index\" does not autodetect what\nhash algorithm is used from its input, and the new argument is a way\nfor the user to help the command when the hash algorithm is\ndifferent from what is used in the current repository?\n\nI ask because I found that your version can be read to say that\n\"show-index\" can show the contents of a given pack index using any\nhash algorithm we support, and the user can specify --hash=SHA-256\nwhen running the command on a pack .idx that uses SHA-1 object names\nto auto-convert it, and readers wouldn't be able to guess which was\nmeant with only the above five lines.\n\n> Signed-off-by: brian m. carlson \n> ---\n> builtin/show-index.c | 29 ++++++++++++++++++++++++-----\n> git.c | 2 +-\n> 2 files changed, 25 insertions(+), 6 deletions(-)\n>\n> diff --git a/builtin/show-index.c b/builtin/show-index.c\n> index 0826f6a5a2..ebfa2e9abd 100644\n> --- a/builtin/show-index.c\n> +++ b/builtin/show-index.c\n> @@ -1,9 +1,12 @@\n> #include \"builtin.h\"\n> #include \"cache.h\"\n> #include \"pack.h\"\n> +#include \"parse-options.h\"\n> \n> -static const char show_index_usage[] =\n> -\"git show-index\";\n> +static const char *const show_index_usage[] = {\n> +\t\"git show-index [--hash=HASH]\",\n> +\tNULL\n> +};\n\nDo we say --hash=SHA-1 etc. or --hash-algo=SHA-256 in other places?\nWould the word \"hash\" alone clear enough that it does not refer to\na specific \"hash\" value but the name of an algorithm?\n\nThe generating side seems to use \"index-pack --object-format=\"\nand the transport seems to use a capability \"object-format=\",\nneither of which is directly visible to the end users, but I think\nthey follow \"git init --object-format=\", so we are consistent\nthere.\n\nPerhaps we should follow suit here, too?\n\n> int cmd_show_index(int argc, const char **argv, const char *prefix)\n> {\n> @@ -11,10 +14,26 @@ int cmd_show_index(int argc, const char **argv, const char *prefix)\n> \tunsigned nr;\n> \tunsigned int version;\n> \tstatic unsigned int top_index[256];\n> -\tconst unsigned hashsz = the_hash_algo->rawsz;\n> +\tunsigned hashsz;\n> +\tconst char *hash_name = NULL;\n> +\tint hash_algo;\n> +\tconst struct option show_index_options[] = {\n> +\t\tOPT_STRING(0, \"hash\", &hash_name, N_(\"hash\"),\n> +\t\t\t N_(\"specify the hash algorithm to use\")),\n\ninit-db has an entry identical to this except for the second token\ngiven to the macro is \"object-format\" instead of \"hash\". Both may\nwant to change what's inside N_() to \"hash algorithm\".\n\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, prefix, show_index_options, show_index_usage, 0);\n> +\n> +\tif (hash_name) {\n> +\t\thash_algo = hash_algo_by_name(hash_name);\n> +\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n> +\t\t\tdie(_(\"Unknown hash algorithm\"));\n> +\t\trepo_set_hash_algo(the_repository, hash_algo);\n> +\t}\n> +\n> +\thashsz = the_hash_algo->rawsz;\n> \n> -\tif (argc != 1)\n> -\t\tusage(show_index_usage);\n> \tif (fread(top_index, 2 * 4, 1, stdin) != 1)\n> \t\tdie(\"unable to read header\");\n> \tif (top_index[0] == htonl(PACK_IDX_SIGNATURE)) {\n> diff --git a/git.c b/git.c\n> index 2e4efb4ff0..e53e8159a2 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -573,7 +573,7 @@ static struct cmd_struct commands[] = {\n> \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n> \t{ \"show\", cmd_show, RUN_SETUP },\n> \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n> -\t{ \"show-index\", cmd_show_index },\n> +\t{ \"show-index\", cmd_show_index, RUN_SETUP_GENTLY },\n\nHmph, this is not necessary to support peeking an .idx file in\nanother repository that uses a different hash algorithm than ours\n(we do need the --hash= override to tell that the algo is\ndifferent from what we read from our repository settings). Is this\nabsolutely necessary?\n\nAh, I am misreading the patch. We didn't even do setup but we now\noptionally do, in order to see if we are in a repository and what\nobject format it uses to give the default value to --hash=\nwhen the argument is not given. The need for RUN_SETUP_GENTLY\nis understandable.\n\nAs we do not take any path argument on the command line, the other\nside effect of setup_git_directory() that takes us up to the top\nlevel of the working tree does not hurt us, either, so this is a\ngood change, I think.\n\nThanks.\n\n\n\n> \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n> \t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n> \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n"},{"id":"398170","messageId":"20200519003100.GN6362@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"xmqqimgt5i61.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 27/44] builtin/show-index: provide options to determine hash algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-19T00:31:00Z","receivedAt":"2020-05-19T00:31:07Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-05-18 at 16:20:22, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > It's possible to use a variety of index formats with show-index, and we\n> > need a way to indicate the hash algorithm which is in use for a\n> > particular index we'd like to show. Default to using the value for the\n> > repository we're in by calling setup_git_directory_gently, and allow\n> > overriding it by using a --hash argument.\n> \n> I think you meant to say that \"show-index\" does not autodetect what\n> hash algorithm is used from its input, and the new argument is a way\n> for the user to help the command when the hash algorithm is\n> different from what is used in the current repository?\n\nCorrect.\n\n> I ask because I found that your version can be read to say that\n> \"show-index\" can show the contents of a given pack index using any\n> hash algorithm we support, and the user can specify --hash=SHA-256\n> when running the command on a pack .idx that uses SHA-1 object names\n> to auto-convert it, and readers wouldn't be able to guess which was\n> meant with only the above five lines.\n\nNo, that's definitely not what I meant. I'll adjust the commit message\nto make this clearer.\n\n> Do we say --hash=SHA-1 etc. or --hash-algo=SHA-256 in other places?\n> Would the word \"hash\" alone clear enough that it does not refer to\n> a specific \"hash\" value but the name of an algorithm?\n> \n> The generating side seems to use \"index-pack --object-format=\"\n> and the transport seems to use a capability \"object-format=\",\n> neither of which is directly visible to the end users, but I think\n> they follow \"git init --object-format=\", so we are consistent\n> there.\n> \n> Perhaps we should follow suit here, too?\n\nYeah, as I mentioned to Martin elsewhere in the thread, I'm going to\nmake this consistent and use --object-formta.\n\n> > diff --git a/git.c b/git.c\n> > index 2e4efb4ff0..e53e8159a2 100644\n> > --- a/git.c\n> > +++ b/git.c\n> > @@ -573,7 +573,7 @@ static struct cmd_struct commands[] = {\n> > \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n> > \t{ \"show\", cmd_show, RUN_SETUP },\n> > \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n> > -\t{ \"show-index\", cmd_show_index },\n> > +\t{ \"show-index\", cmd_show_index, RUN_SETUP_GENTLY },\n> \n> Hmph, this is not necessary to support peeking an .idx file in\n> another repository that uses a different hash algorithm than ours\n> (we do need the --hash= override to tell that the algo is\n> different from what we read from our repository settings). Is this\n> absolutely necessary?\n> \n> Ah, I am misreading the patch. We didn't even do setup but we now\n> optionally do, in order to see if we are in a repository and what\n> object format it uses to give the default value to --hash=\n> when the argument is not given. The need for RUN_SETUP_GENTLY\n> is understandable.\n\nYes, this is designed to make us do the right thing when we're in a\nrepository (e.g., with --stdin) by autodetecting the algorithm in use\nbut not fail when we're outside of a repository. I'll update the commit\nmessage to make this a lot clearer, since I clearly omitted a lot of\nthings that were in my head when writing this.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"398505","messageId":"20200525195930.309665-2-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 01/44] t1050: match object ID paths in a hash-insensitive way","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:47Z","receivedAt":"2020-05-25T19:59:48Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The pattern here looking for failures is specific to SHA-1. Let's\ncreate a variable that matches the regex or glob pattern for a path\nwithin the objects directory.\n\nSigned-off-by: brian m. carlson \n---\n t/t1050-large.sh | 2 +-\n t/test-lib.sh | 1 +\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 184b479a21..7f88ea07c2 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -64,7 +64,7 @@ test_expect_success 'add a large file or two' '\n \ttest $count = 1 &&\n \tcnt=$(git show-index <\"$idx\" | wc -l) &&\n \ttest $cnt = 2 &&\n-\tfor l in .git/objects/??/??????????????????????????????????????\n+\tfor l in .git/objects/$OIDPATH_REGEX\n \tdo\n \t\ttest_path_is_file \"$l\" || continue\n \t\tbad=t\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d36b6ddc62..5c65c3e26c 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1414,6 +1414,7 @@ test_oid_init\n \n ZERO_OID=$(test_oid zero)\n OID_REGEX=$(echo $ZERO_OID | sed -e 's/0/[0-9a-f]/g')\n+OIDPATH_REGEX=$(test_oid_to_path $ZERO_OID | sed -e 's/0/[0-9a-f]/g')\n EMPTY_TREE=$(test_oid empty_tree)\n EMPTY_BLOB=$(test_oid empty_blob)\n _z40=$ZERO_OID\n"},{"id":"398506","messageId":"20200525195930.309665-3-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 02/44] Documentation: document v1 protocol object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:48Z","receivedAt":"2020-05-25T19:59:49Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Document a capability that indicates which hash algorithms are in use by\nboth sides of a remote connection. Use the term \"object-format\", since\nthis is the term used for the repository extension as well.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/protocol-capabilities.txt | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 2b267c0da6..36ccd14f97 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -176,6 +176,21 @@ agent strings are purely informative for statistics and debugging\n purposes, and MUST NOT be used to programmatically assume the presence\n or absence of particular features.\n \n+object-format\n+-------------\n+\n+This capability, which takes a hash algorithm as an argument, indicates\n+that the server supports the given hash algorithms. It may be sent\n+multiple times; if so, the first one given is the one used in the ref\n+advertisement.\n+\n+When provided by the client, this indicates that it intends to use the\n+given hash algorithm to communicate. The algorithm provided must be one\n+that the server supports.\n+\n+If this capability is not provided, it is assumed that the only\n+supported algorithm is SHA-1.\n+\n symref\n ------\n \n"},{"id":"398507","messageId":"20200525195930.309665-1-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 00/44] SHA-256 part 2/3: protocol functionality","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:46Z","receivedAt":"2020-05-25T19:59:50Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This is part 2 of 3 of the SHA-256 work. This series adds all of the\nprotocol logic to work with SHA-256 repositories.\n\nChanges from v1:\n* Fix spurious line additions and deletions.\n* Rename len to linelen for easier understanding.\n* Move the documentation comment for xstrncmpz to the header.\n* Drop a useless variable (found).\n* Update several commit messages to better explain things as suggested\n by Junio and Martin.\n* Name the parameters for parse_feature_value for better documentation.\n* Reduce the scope of variables when possible.\n* Add explicit handling for missing object-format capabilities.\n* Rename all new options to --object-format.\n* Use oidcpy where possible.\n* Test more failure cases.\n* Have index-pack fail if --stdin and --object-format are both\n specified.\n* Move and simplify t5704.\n* Other miscellaneous cleanups to respond to review feedback.\n\nRange-diff below.\n\nbrian m. carlson (44):\n t1050: match object ID paths in a hash-insensitive way\n Documentation: document v1 protocol object-format capability\n connect: have ref processing code take struct packet_reader\n wrapper: add function to compare strings with different NUL\n termination\n remote: advertise the object-format capability on the server side\n connect: add function to parse multiple v1 capability values\n connect: add function to fetch value of a v2 server capability\n pkt-line: add a member for hash algorithm\n transport: add a hash algorithm member\n connect: add function to detect supported v1 hash functions\n send-pack: detect when the server doesn't support our hash\n connect: make parse_feature_value extern\n fetch-pack: detect when the server doesn't support our hash\n connect: detect algorithm when fetching refs\n builtin/receive-pack: detect when the server doesn't support our hash\n docs: update remote helper docs for object-format extensions\n transport-helper: implement object-format extensions\n remote-curl: implement object-format extensions\n builtin/clone: initialize hash algorithm properly\n t5562: pass object-format in synthesized test data\n fetch-pack: parse and advertise the object-format capability\n setup: set the_repository's hash algo when checking format\n t3200: mark assertion with SHA1 prerequisite\n packfile: compute and use the index CRC offset\n t5302: modernize test formatting\n builtin/show-index: provide options to determine hash algo\n t1302: expect repo format version 1 for SHA-256\n Documentation/technical: document object-format for protocol v2\n connect: pass full packet reader when parsing v2 refs\n connect: parse v2 refs with correct hash algorithm\n serve: advertise object-format capability for protocol v2\n t5500: make hash independent\n builtin/ls-remote: initialize repository based on fetch\n remote-curl: detect algorithm for dumb HTTP by size\n builtin/index-pack: add option to specify hash algorithm\n t1050: pass algorithm to index-pack when outside repo\n remote-curl: avoid truncating refs with ls-remote\n t/helper: initialize the repository for test-sha1-array\n t5702: offer an object-format capability in the test\n t5703: use object-format serve option\n t5704: send object-format capability with SHA-256\n t5300: pass --object-format to git index-pack\n bundle: detect hash algorithm when reading refs\n remote-testgit: adapt for object-format\n\n Documentation/git-index-pack.txt | 8 +\n Documentation/git-show-index.txt | 11 +-\n Documentation/gitremote-helpers.txt | 33 +-\n .../technical/protocol-capabilities.txt | 15 +\n Documentation/technical/protocol-v2.txt | 9 +\n builtin/clone.c | 9 +\n builtin/index-pack.c | 14 +-\n builtin/ls-remote.c | 4 +\n builtin/receive-pack.c | 10 +\n builtin/show-index.c | 29 +-\n bundle.c | 22 +-\n bundle.h | 1 +\n connect.c | 138 +++++--\n connect.h | 3 +\n fetch-pack.c | 14 +\n git-compat-util.h | 6 +\n git.c | 2 +-\n object-store.h | 1 +\n packfile.c | 1 +\n pkt-line.c | 1 +\n pkt-line.h | 3 +\n remote-curl.c | 46 ++-\n send-pack.c | 6 +\n serve.c | 27 ++\n setup.c | 1 +\n t/helper/test-oid-array.c | 3 +\n t/t1050-large.sh | 6 +-\n t/t1302-repo-version.sh | 6 +-\n t/t3200-branch.sh | 2 +-\n t/t5300-pack-object.sh | 9 +-\n t/t5302-pack-index.sh | 360 +++++++++---------\n t/t5500-fetch-pack.sh | 5 +-\n t/t5562-http-backend-content-length.sh | 5 +-\n t/t5701-git-serve.sh | 25 ++\n t/t5702-protocol-v2.sh | 2 +\n t/t5703-upload-pack-ref-in-want.sh | 19 +-\n t/t5704-protocol-violations.sh | 2 +\n t/t5801/git-remote-testgit | 6 +\n t/test-lib.sh | 1 +\n transport-helper.c | 24 +-\n transport.c | 18 +-\n transport.h | 8 +\n upload-pack.c | 3 +-\n wrapper.c | 8 +\n 44 files changed, 678 insertions(+), 248 deletions(-)\n\nRange-diff against v1:\n 1: 82a0a5beae = 1: 5878fe6a98 t1050: match object ID paths in a hash-insensitive way\n 2: 95e84f6457 ! 2: 402864eaa3 Documentation: document v1 protocol object-format capability\n @@ Documentation/technical/protocol-capabilities.txt: agent strings are purely info\n symref\n ------\n \n -@@ Documentation/technical/protocol-capabilities.txt: refs being sent.\n - \n - Clients MAY use the parameters from this capability to select the proper initial\n - branch when cloning a repository.\n --\n - shallow\n - -------\n - \n 3: 7c82e91a11 ! 3: d124692e2f connect: have ref processing code take struct packet_reader\n @@ Commit message\n code take pointers to struct reader instead of having to pass multiple\n arguments to each function.\n \n + Rename the len variable to \"linelen\" to make it clearer what the\n + variable does in light of the variable change.\n +\n Signed-off-by: brian m. carlson \n \n ## connect.c ##\n @@ connect.c: static void annotate_refs_with_symref_info(struct ref *ref)\n }\n \n -static void process_capabilities(const char *line, int *len)\n -+static void process_capabilities(struct packet_reader *reader, int *len)\n ++static void process_capabilities(struct packet_reader *reader, int *linelen)\n {\n +\tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n - \tif (nul_location == *len)\n +-\tif (nul_location == *len)\n ++\tif (nul_location == *linelen)\n \t\treturn;\n -@@ connect.c: static void process_capabilities(const char *line, int *len)\n - \t*len = nul_location;\n + \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n +-\t*len = nul_location;\n ++\t*linelen = nul_location;\n }\n \n -static int process_dummy_ref(const char *line)\n 4: a78234de04 < -: ---------- wrapper: add function to compare strings with different NUL termination\n -: ---------- > 4: cce29662b4 wrapper: add function to compare strings with different NUL termination\n 5: 628ecec99a = 5: 3b207e304b remote: advertise the object-format capability on the server side\n 6: 9990767072 = 6: 235d7f5b8f connect: add function to parse multiple v1 capability values\n 7: 5ce2b7afde = 7: 0324e126b1 connect: add function to fetch value of a v2 server capability\n 8: e5d58b48f3 = 8: cdba3122ce pkt-line: add a member for hash algorithm\n 9: bce9ba0538 = 9: c8233c3b42 transport: add a hash algorithm member\n10: 2d016e3870 ! 10: b9273c4021 connect: add function to detect supported v1 hash functions\n @@ connect.c: static const char *parse_feature_value(const char *feature_list, cons\n +int server_supports_hash(const char *desired, int *feature_supported)\n +{\n +\tint offset = 0;\n -+\tint len, found = 0;\n ++\tint len;\n +\tconst char *hash;\n +\n +\thash = next_server_feature_value(\"object-format\", &len, &offset);\n @@ connect.c: static const char *parse_feature_value(const char *feature_list, cons\n +\t}\n +\twhile (hash) {\n +\t\tif (!xstrncmpz(desired, hash, len))\n -+\t\t\tfound = 1;\n -+\n -+\t\tif (found)\n +\t\t\treturn 1;\n ++\n +\t\thash = next_server_feature_value(\"object-format\", &len, &offset);\n +\t}\n +\treturn 0;\n11: 9fdc67b825 ! 11: e2d37b75c8 send-pack: detect when the server doesn't support our hash\n @@ Commit message\n send-pack: detect when the server doesn't support our hash\n \n Detect when the server doesn't support our hash algorithm and abort.\n + If the server does support our hash, advertise it as part of our\n + capabilities.\n \n Signed-off-by: brian m. carlson \n \n12: 91a1fb0a7d ! 12: 602734cbbb connect: make parse_feature_value extern\n @@ connect.h: struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_hash(const char *desired, int *feature_supported);\n -+const char *parse_feature_value(const char *, const char *, int *, int *);\n ++const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n13: fd82e5f755 = 13: d97fa2c8aa fetch-pack: detect when the server doesn't support our hash\n14: b62f751fe4 ! 14: ba052f1da7 connect: detect algorithm when fetching refs\n @@ Commit message\n If we're fetching refs, detect the hash algorithm and parse the refs\n using that algorithm.\n \n + As mentioned in the documentation, if multiple versions of the\n + object-format capability are provided, we use the first. No known\n + implementation supports multiple algorithms now, but they may in the\n + future.\n +\n Signed-off-by: brian m. carlson \n \n ## connect.c ##\n @@ connect.c: static void annotate_refs_with_symref_info(struct ref *ref)\n \n - static void process_capabilities(struct packet_reader *reader, int *len)\n + static void process_capabilities(struct packet_reader *reader, int *linelen)\n {\n +\tconst char *feat_val;\n +\tint feat_len;\n -+\tint hash_algo;\n \tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n - \tif (nul_location == *len)\n + \tif (nul_location == *linelen)\n \t\treturn;\n \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n - \t*len = nul_location;\n + \t*linelen = nul_location;\n +\n +\tfeat_val = server_feature_value(\"object-format\", &feat_len);\n +\tif (feat_val) {\n +\t\tchar *hash_name = xstrndup(feat_val, feat_len);\n -+\t\thash_algo = hash_algo_by_name(hash_name);\n ++\t\tint hash_algo = hash_algo_by_name(hash_name);\n +\t\tif (hash_algo != GIT_HASH_UNKNOWN)\n +\t\t\treader->hash_algo = &hash_algos[hash_algo];\n +\t\tfree(hash_name);\n ++\t} else {\n ++\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n +\t}\n }\n \n15: 29b4219411 ! 15: 661d94d4de builtin/receive-pack: detect when the server doesn't support our hash\n @@ builtin/receive-pack.c: static struct command *read_head_info(struct packet_read\n \t\tlinelen = strlen(reader->line);\n \t\tif (linelen < reader->pktlen) {\n \t\t\tconst char *feature_list = reader->line + linelen + 1;\n -+\t\t\tconst char *hash;\n ++\t\t\tconst char *hash = NULL;\n +\t\t\tint len = 0;\n \t\t\tif (parse_feature_request(feature_list, \"report-status\"))\n \t\t\t\treport_status = 1;\n16: f8eb8c96f8 = 16: fd8b85390c docs: update remote helper docs for object-format extensions\n17: 93bf7005a8 = 17: 32285e611f transport-helper: implement object-format extensions\n18: ed75c102a3 = 18: a33d1ed9a0 remote-curl: implement object-format extensions\n19: bf16872e73 = 19: fffdf0780d builtin/clone: initialize hash algorithm properly\n20: ce77713343 ! 20: f616f85b4b t5562: pass object-format in synthesized test data\n @@ t/t5562-http-backend-content-length.sh: test_expect_success 'setup' '\n \t\tprintf 0000 &&\n \t\techo \"$hash_next\" | git pack-objects --stdout\n \t} >push_body &&\n -@@ t/t5562-http-backend-content-length.sh: test_expect_success GZIP 'push plain' '\n - \ttest_cmp act.head exp.head\n - '\n - \n -+test_expect_success GZIP 'push plain with SHA-1' '\n -+\ttest_when_finished \"git branch -D newbranch\" &&\n -+\ttest_http_env receive push_body &&\n -+\tverify_http_result \"200 OK\" &&\n -+\tgit rev-parse newbranch >act.head &&\n -+\techo \"$hash_next\" >exp.head &&\n -+\ttest_cmp act.head exp.head\n -+'\n -+\n - test_expect_success 'push plain truncated' '\n - \ttest_http_env receive push_body.trunc &&\n - \t! verify_http_result \"200 OK\"\n21: e4dd90fa9d < -: ---------- t5704: send object-format capability with SHA-256\n22: 626d6e9018 ! 21: eca43da42e fetch-pack: parse and advertise the object-format capability\n @@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator,\n +\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n +\t\t\t the_hash_algo->name, hash_name);\n +\t\tpacket_write_fmt(fd_out, \"object-format=%s\", the_hash_algo->name);\n -+\t}\n -+\telse if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1)\n ++\t} else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n +\t\tdie(_(\"the server does not support algorithm '%s'\"),\n +\t\t the_hash_algo->name);\n ++\t}\n +\n \tpacket_buf_delim(&req_buf);\n \tif (args->use_thin_pack)\n23: 8c675b5117 ! 22: 22c1a62e10 setup: set the_repository's hash algo when checking format\n @@ Commit message\n the same time. This ensures that we perform a suitable initialization\n early enough to avoid confusing any parts of the code. If we defer\n until later, we can end up with portions of the code which are confused\n - about the hash algorithm, resulting in segfaults.\n + about the hash algorithm, resulting in segfaults when working with\n + SHA-256 repositories.\n \n Signed-off-by: brian m. carlson \n \n24: b714d4accc = 23: 7c7f2263d5 t3200: mark assertion with SHA1 prerequisite\n25: eacde58fda = 24: ee8a71a926 packfile: compute and use the index CRC offset\n26: 81bb8cdb18 = 25: 6afecf0b09 t5302: modernize test formatting\n27: 8623f21715 ! 26: 99a847ba4e builtin/show-index: provide options to determine hash algo\n @@ Metadata\n ## Commit message ##\n builtin/show-index: provide options to determine hash algo\n \n - It's possible to use a variety of index formats with show-index, and we\n - need a way to indicate the hash algorithm which is in use for a\n - particular index we'd like to show. Default to using the value for the\n - repository we're in by calling setup_git_directory_gently, and allow\n - overriding it by using a --hash argument.\n + show-index is capable of reading any possible index file whether or not\n + the index is inside a repository. However, because our index files lack\n + metadata about the hash algorithm in use, it's not possible to\n + autodetect the algorithm that a particular index file is using.\n +\n + In order to allow us to read index files of any algorithm, let's set up\n + the .git directory gently so that we default to the algorithm for the\n + current repository, and add an --object-format option to allow users to\n + override this setting and continue to run show-index outside of a\n + repository altogether. Let's also document this new option so that\n + people can find it and use it.\n \n Signed-off-by: brian m. carlson \n \n + ## Documentation/git-show-index.txt ##\n +@@ Documentation/git-show-index.txt: git-show-index - Show packed archive index\n + SYNOPSIS\n + --------\n + [verse]\n +-'git show-index'\n ++'git show-index' [--object-format=]\n + \n + \n + DESCRIPTION\n +@@ Documentation/git-show-index.txt: Note that you can get more information on a packfile by calling\n + linkgit:git-verify-pack[1]. However, as this command considers only the\n + index file itself, it's both faster and more flexible.\n + \n ++OPTIONS\n ++-------\n ++\n ++--object-format=::\n ++\tSpecify the given object format (hash algorithm) for the index file. The\n ++\tvalid values are 'sha1' and (if enabled) 'sha256'. The default is the\n ++\talgorithm for the current repository (set by `extensions.objectFormat`), or\n ++\t'sha1' if no value is set or outside a repository..\n ++\n + GIT\n + ---\n + Part of the linkgit:git[1] suite\n +\n ## builtin/show-index.c ##\n @@\n #include \"builtin.h\"\n @@ builtin/show-index.c\n -static const char show_index_usage[] =\n -\"git show-index\";\n +static const char *const show_index_usage[] = {\n -+\t\"git show-index [--hash=HASH]\",\n ++\t\"git show-index [--object-format=]\",\n +\tNULL\n +};\n \n @@ builtin/show-index.c: int cmd_show_index(int argc, const char **argv, const char\n +\tconst char *hash_name = NULL;\n +\tint hash_algo;\n +\tconst struct option show_index_options[] = {\n -+\t\tOPT_STRING(0, \"hash\", &hash_name, N_(\"hash\"),\n ++\t\tOPT_STRING(0, \"object-format\", &hash_name, N_(\"hash-algorithm\"),\n +\t\t\t N_(\"specify the hash algorithm to use\")),\n +\t\tOPT_END()\n +\t};\n28: bb3d2f566a = 27: 9f7c7bafaf t1302: expect repo format version 1 for SHA-256\n29: cc25069cb6 = 28: d0ea597d63 Documentation/technical: document object-format for protocol v2\n30: efdac6383f ! 29: 51848df542 connect: pass full packet reader when parsing v2 refs\n @@ connect.c: static int process_ref_v2(const char *line, struct ref ***list)\n \t/*\n \t * Ref lines have a number of fields which are space deliminated. The\n @@ connect.c: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n - \t}\n - \tpacket_flush(fd_out);\n \n -+\n \t/* Process response from server */\n \twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n -\t\tif (!process_ref_v2(reader->line, &list))\n31: 602405e436 ! 30: b57361f3b8 connect: parse v2 refs with correct hash algorithm\n @@ Commit message\n set that value in the packet reader. Parse the refs using this\n algorithm.\n \n - Note that we use memcpy instead of oidcpy for copying values, since\n - oidcpy is intentionally limited to the current hash algorithm length,\n - and the copy will be too short if the server side uses SHA-256 but the\n - client side has not had a repository set up (and therefore defaults to\n - SHA-1).\n -\n Signed-off-by: brian m. carlson \n \n ## connect.c ##\n +@@ connect.c: static int process_ref(const struct packet_reader *reader, int len,\n + \t\tdie(_(\"protocol error: unexpected capabilities^{}\"));\n + \t} else if (check_ref(name, flags)) {\n + \t\tstruct ref *ref = alloc_ref(name);\n +-\t\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n ++\t\toidcpy(&ref->old_oid, &old_oid);\n + \t\t**list = ref;\n + \t\t*list = &ref->next;\n + \t}\n @@ connect.c: static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\tgoto out;\n \t}\n32: 96236ac9ae ! 31: a0c0f0f7a3 serve: advertise object-format capability for protocol v2\n @@ Commit message\n \n In the test, when we're using an algorithm other than SHA-1, we need to\n specify the algorithm in use so we don't get a failure with an \"unknown\n - format\" message. Add a wrapper function that specifies this header if\n - required. Skip specifying this header for SHA-1 to test that it works\n - both with and without this header.\n + format\" message. Add a test that we handle a mismatched algorithm.\n + Remove the test_oid_init call since it's no longer necessary.\n \n Signed-off-by: brian m. carlson \n \n + ## connect.c ##\n +@@ connect.c: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n + \t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n + \t\treader->hash_algo = &hash_algos[hash_algo];\n + \t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n ++\t} else {\n ++\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n + \t}\n + \n + \tif (server_options && server_options->nr &&\n +\n ## serve.c ##\n @@ serve.c: static int agent_advertise(struct repository *r,\n \treturn 1;\n @@ serve.c: static int process_request(void)\n \n ## t/t5701-git-serve.sh ##\n @@ t/t5701-git-serve.sh: test_description='test protocol v2 server commands'\n - \n . ./test-lib.sh\n \n -+write_command () {\n -+\techo \"command=$1\"\n -+\n -+\tif test \"$(test_oid algo)\" != sha1\n -+\tthen\n -+\t\techo \"object-format=$(test_oid algo)\"\n -+\tfi\n -+}\n -+\n test_expect_success 'test capability advertisement' '\n -+\ttest_oid_init &&\n ++\ttest_oid_cache <<-EOF &&\n ++\twrong_algo sha1:sha256\n ++\twrong_algo sha256:sha1\n ++\tEOF\n \tcat >expect <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n @@ t/t5701-git-serve.sh: test_expect_success 'request invalid capability' '\n \tEOF\n \ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n --\tcommand=foo\n -+\t$(write_command foo)\n + \tcommand=foo\n ++\tobject-format=$(test_oid algo)\n \tagent=git/test\n \t0000\n \tEOF\n -@@ t/t5701-git-serve.sh: test_expect_success 'setup some refs and tags' '\n +@@ t/t5701-git-serve.sh: test_expect_success 'request invalid command' '\n + \ttest_i18ngrep \"invalid command\" err\n + '\n \n ++test_expect_success 'wrong object-format' '\n ++\ttest-tool pkt-line pack >in <<-EOF &&\n ++\tcommand=fetch\n ++\tagent=git/test\n ++\tobject-format=$(test_oid wrong_algo)\n ++\t0000\n ++\tEOF\n ++\ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n --\tcommand=ls-refs\n -+\t$(write_command ls-refs)\n + \tcommand=ls-refs\n ++\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \n @@ t/t5701-git-serve.sh: test_expect_success 'basics of ls-refs' '\n - \n test_expect_success 'basic ref-prefixes' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n --\tcommand=ls-refs\n -+\t$(write_command ls-refs)\n + \tcommand=ls-refs\n ++\tobject-format=$(test_oid algo)\n \t0001\n \tref-prefix refs/heads/master\n \tref-prefix refs/tags/one\n @@ t/t5701-git-serve.sh: test_expect_success 'basic ref-prefixes' '\n - \n test_expect_success 'refs/heads prefix' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n --\tcommand=ls-refs\n -+\t$(write_command ls-refs)\n + \tcommand=ls-refs\n ++\tobject-format=$(test_oid algo)\n \t0001\n \tref-prefix refs/heads/\n \t0000\n @@ t/t5701-git-serve.sh: test_expect_success 'refs/heads prefix' '\n - \n test_expect_success 'peel parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n --\tcommand=ls-refs\n -+\t$(write_command ls-refs)\n + \tcommand=ls-refs\n ++\tobject-format=$(test_oid algo)\n \t0001\n \tpeel\n \tref-prefix refs/tags/\n @@ t/t5701-git-serve.sh: test_expect_success 'peel parameter' '\n - \n test_expect_success 'symrefs parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n --\tcommand=ls-refs\n -+\t$(write_command ls-refs)\n + \tcommand=ls-refs\n ++\tobject-format=$(test_oid algo)\n \t0001\n \tsymrefs\n \tref-prefix refs/heads/\n @@ t/t5701-git-serve.sh: test_expect_success 'symrefs parameter' '\n - \n test_expect_success 'sending server-options' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n --\tcommand=ls-refs\n -+\t$(write_command ls-refs)\n + \tcommand=ls-refs\n ++\tobject-format=$(test_oid algo)\n \tserver-option=hello\n \tserver-option=world\n \t0001\n @@ t/t5701-git-serve.sh: test_expect_success 'unexpected lines are not allowed in fetch request' '\n - \tgit init server &&\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n --\tcommand=fetch\n -+\t$(write_command fetch)\n + \tcommand=fetch\n ++\tobject-format=$(test_oid algo)\n \t0001\n \tthis-is-not-a-command\n \t0000\n33: 57f3bbb709 = 32: 1694f3f838 t5500: make hash independent\n34: 8242e65747 ! 33: 902b394667 builtin/ls-remote: initialize repository based on fetch\n @@ Commit message\n the refs to 40 hex characters, since that's the length of the default\n hash algorithm (SHA-1).\n \n + Note that technically this is not a correct setting of the repository\n + hash algorithm since, if we are in a repository, it might be one of a\n + different hash algorithm from the remote side. However, our current\n + code paths don't handle multiple algorithms and won't for some time, so\n + this is the best we can do. We rely on the fact that ls-remote never\n + modifies the current repository, which is a reasonable assumption to\n + make.\n +\n Signed-off-by: brian m. carlson \n \n ## builtin/ls-remote.c ##\n35: c5664b646f ! 34: cc12b9b51f remote-curl: detect algorithm for dumb HTTP by size\n @@ Commit message\n provide one. Detect the hash algorithm in use by the size of the first\n object ID.\n \n + We anonymize the URL like elsewhere in the function in case the user has\n + decided to include a secret in the URL.\n +\n Signed-off-by: brian m. carlson \n \n ## remote-curl.c ##\n @@ remote-curl.c: static struct ref *parse_info_refs(struct discovery *heads)\n +\tif (!options.hash_algo)\n +\t\tdie(\"%sinfo/refs not valid: could not determine hash algorithm; \"\n +\t\t \"is this a git repository?\",\n -+\t\t url.buf);\n ++\t\t transport_anonymize_url(url.buf));\n +\n \tdata = heads->buf;\n \tstart = NULL;\n36: 31cd59a221 < -: ---------- builtin/index-pack: add option to specify hash algorithm\n -: ---------- > 35: b5425c9f54 builtin/index-pack: add option to specify hash algorithm\n37: 658b787e8c = 36: 5c70c24d7a t1050: pass algorithm to index-pack when outside repo\n38: 64429337ba = 37: 460d6008e8 remote-curl: avoid truncating refs with ls-remote\n39: cde2128520 = 38: 60a98d9b53 t/helper: initialize the repository for test-sha1-array\n40: 0af00a7681 = 39: b66c3ead37 t5702: offer an object-format capability in the test\n41: 74278d4c1c = 40: af43274a1f t5703: use object-format serve option\n -: ---------- > 41: f5085b1f3f t5704: send object-format capability with SHA-256\n42: 4f735c8bb5 = 42: a1b01babda t5300: pass --object-format to git index-pack\n43: 3854f70427 = 43: dbb5f7195e bundle: detect hash algorithm when reading refs\n44: 103be1f4d6 = 44: 6c823bbe68 remote-testgit: adapt for object-format\n"},{"id":"398508","messageId":"20200525195930.309665-5-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 04/44] wrapper: add function to compare strings with different NUL termination","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:50Z","receivedAt":"2020-05-25T19:59:50Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When parsing capabilities for the pack protocol, there are times we'll\nwant to compare the value of a capability to a NUL-terminated string.\nSince the data we're reading will be space-terminated, not\nNUL-terminated, we need a function that compares the two strings, but\nalso checks that they're the same length. Otherwise, if we used strncmp\nto compare these strings, we might accidentally accept a parameter that\nwas a prefix of the expected value.\n\nAdd a function, xstrncmpz, that takes a NUL-terminated string and a\nnon-NUL-terminated string, plus a length, and compares them, ensuring\nthat they are the same length.\n\nSigned-off-by: brian m. carlson \n---\n git-compat-util.h | 6 ++++++\n wrapper.c | 8 ++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a73632e8e4..5637114b8d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -868,6 +868,12 @@ char *xgetcwd(void);\n FILE *fopen_for_writing(const char *path);\n FILE *fopen_or_warn(const char *path, const char *mode);\n \n+/*\n+ * Like strncmp, but only return zero if s is NUL-terminated and exactly len\n+ * characters long. If it is not, consider it greater than t.\n+ */\n+int xstrncmpz(const char *s, const char *t, size_t len);\n+\n /*\n * FREE_AND_NULL(ptr) is like free(ptr) followed by ptr = NULL. Note\n * that ptr is used twice, so don't pass e.g. ptr++.\ndiff --git a/wrapper.c b/wrapper.c\nindex 3a1c0e0526..4ff4a9c3db 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -105,6 +105,14 @@ char *xstrndup(const char *str, size_t len)\n \treturn xmemdupz(str, p ? p - str : len);\n }\n \n+int xstrncmpz(const char *s, const char *t, size_t len)\n+{\n+\tint res = strncmp(s, t, len);\n+\tif (res)\n+\t\treturn res;\n+\treturn s[len] == '\\0' ? 0 : 1;\n+}\n+\n void *xrealloc(void *ptr, size_t size)\n {\n \tvoid *ret;\n"},{"id":"398509","messageId":"20200525195930.309665-4-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 03/44] connect: have ref processing code take struct packet_reader","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:49Z","receivedAt":"2020-05-25T19:59:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In a future patch, we'll want to access multiple members from struct\npacket_reader when parsing references. Therefore, have the ref parsing\ncode take pointers to struct reader instead of having to pass multiple\narguments to each function.\n\nRename the len variable to \"linelen\" to make it clearer what the\nvariable does in light of the variable change.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 27 ++++++++++++++++-----------\n 1 file changed, 16 insertions(+), 11 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 23013c6344..ccc5274189 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -204,17 +204,19 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-static void process_capabilities(const char *line, int *len)\n+static void process_capabilities(struct packet_reader *reader, int *linelen)\n {\n+\tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n-\tif (nul_location == *len)\n+\tif (nul_location == *linelen)\n \t\treturn;\n \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n-\t*len = nul_location;\n+\t*linelen = nul_location;\n }\n \n-static int process_dummy_ref(const char *line)\n+static int process_dummy_ref(const struct packet_reader *reader)\n {\n+\tconst char *line = reader->line;\n \tstruct object_id oid;\n \tconst char *name;\n \n@@ -234,9 +236,11 @@ static void check_no_capabilities(const char *line, int len)\n \t\t\tline + strlen(line));\n }\n \n-static int process_ref(const char *line, int len, struct ref ***list,\n-\t\t unsigned int flags, struct oid_array *extra_have)\n+static int process_ref(const struct packet_reader *reader, int len,\n+\t\t struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n {\n+\tconst char *line = reader->line;\n \tstruct object_id old_oid;\n \tconst char *name;\n \n@@ -260,9 +264,10 @@ static int process_ref(const char *line, int len, struct ref ***list,\n \treturn 1;\n }\n \n-static int process_shallow(const char *line, int len,\n+static int process_shallow(const struct packet_reader *reader, int len,\n \t\t\t struct oid_array *shallow_points)\n {\n+\tconst char *line = reader->line;\n \tconst char *arg;\n \tstruct object_id old_oid;\n \n@@ -315,20 +320,20 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \n \t\tswitch (state) {\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(reader->line, &len);\n-\t\t\tif (process_dummy_ref(reader->line)) {\n+\t\t\tprocess_capabilities(reader, &len);\n+\t\t\tif (process_dummy_ref(reader)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(reader->line, len, shallow_points))\n+\t\t\tif (process_shallow(reader, len, shallow_points))\n \t\t\t\tbreak;\n \t\t\tdie(_(\"protocol error: unexpected '%s'\"), reader->line);\n \t\tcase EXPECTING_DONE:\n"},{"id":"398510","messageId":"20200525195930.309665-15-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 14/44] connect: detect algorithm when fetching refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:00Z","receivedAt":"2020-05-25T19:59:55Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"If we're fetching refs, detect the hash algorithm and parse the refs\nusing that algorithm.\n\nAs mentioned in the documentation, if multiple versions of the\nobject-format capability are provided, we use the first. No known\nimplementation supports multiple algorithms now, but they may in the\nfuture.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 397fad7e32..915f1736a0 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -220,12 +220,25 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \n static void process_capabilities(struct packet_reader *reader, int *linelen)\n {\n+\tconst char *feat_val;\n+\tint feat_len;\n \tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n \tif (nul_location == *linelen)\n \t\treturn;\n \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n \t*linelen = nul_location;\n+\n+\tfeat_val = server_feature_value(\"object-format\", &feat_len);\n+\tif (feat_val) {\n+\t\tchar *hash_name = xstrndup(feat_val, feat_len);\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo != GIT_HASH_UNKNOWN)\n+\t\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tfree(hash_name);\n+\t} else {\n+\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n+\t}\n }\n \n static int process_dummy_ref(const struct packet_reader *reader)\n@@ -234,7 +247,7 @@ static int process_dummy_ref(const struct packet_reader *reader)\n \tstruct object_id oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(line, &oid, &name))\n+\tif (parse_oid_hex_algop(line, &oid, &name, reader->hash_algo))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -258,7 +271,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \tstruct object_id old_oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(line, &old_oid, &name))\n+\tif (parse_oid_hex_algop(line, &old_oid, &name, reader->hash_algo))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -270,7 +283,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \t\tdie(_(\"protocol error: unexpected capabilities^{}\"));\n \t} else if (check_ref(name, flags)) {\n \t\tstruct ref *ref = alloc_ref(name);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n@@ -288,7 +301,7 @@ static int process_shallow(const struct packet_reader *reader, int len,\n \tif (!skip_prefix(line, \"shallow \", &arg))\n \t\treturn 0;\n \n-\tif (get_oid_hex(arg, &old_oid))\n+\tif (get_oid_hex_algop(arg, &old_oid, reader->hash_algo))\n \t\tdie(_(\"protocol error: expected shallow sha-1, got '%s'\"), arg);\n \tif (!shallow_points)\n \t\tdie(_(\"repository on the other end cannot be shallow\"));\n"},{"id":"398511","messageId":"20200525195930.309665-14-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 13/44] fetch-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:59Z","receivedAt":"2020-05-25T19:59:57Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n fetch-pack.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7eaa19d7c1..34c339a5fe 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1040,6 +1040,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tprint_verbose(args, _(\"Server supports %s\"), \"deepen-relative\");\n \telse if (args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n+\tif (!server_supports_hash(the_hash_algo->name, NULL))\n+\t\tdie(_(\"Server does not support this repository's object format\"));\n \n \tif (!args->no_dependents) {\n \t\tmark_complete_and_common_ref(negotiator, args, &ref);\n"},{"id":"398512","messageId":"20200525195930.309665-13-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 12/44] connect: make parse_feature_value extern","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:58Z","receivedAt":"2020-05-25T19:59:58Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We're going to be using this function in other files, so no longer mark\nthis function static.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 3 +--\n connect.h | 1 +\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex eaa13b41bb..397fad7e32 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -18,7 +18,6 @@\n \n static char *server_capabilities_v1;\n static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n-static const char *parse_feature_value(const char *, const char *, int *, int *);\n static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -467,7 +466,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \treturn list;\n }\n \n-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n+const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n {\n \tint len;\n \ndiff --git a/connect.h b/connect.h\nindex fc75d6a457..ace074dcb6 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -19,6 +19,7 @@ struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_hash(const char *desired, int *feature_supported);\n+const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n"},{"id":"398513","messageId":"20200525195930.309665-17-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 16/44] docs: update remote helper docs for object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:02Z","receivedAt":"2020-05-25T19:59:58Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Update the remote helper docs to document the object-format extensions\nwe will implement in remote-curl and the transport helper code shortly.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitremote-helpers.txt | 33 +++++++++++++++++++++++++----\n 1 file changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex f48a031dc3..26f32e4421 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -238,6 +238,9 @@ the remote repository.\n \t`--signed-tags=verbatim` to linkgit:git-fast-export[1]. In the\n \tabsence of this capability, Git will use `--signed-tags=warn-strip`.\n \n+'object-format'::\n+\tThis indicates that the helper is able to interact with the remote\n+\tside using an explicit hash algorithm extension.\n \n \n COMMANDS\n@@ -257,12 +260,14 @@ Support for this command is mandatory.\n 'list'::\n \tLists the refs, one per line, in the format \" \n \t[ ...]\". The value may be a hex sha1 hash, \"@\" for\n-\ta symref, or \"?\" to indicate that the helper could not get the\n-\tvalue of the ref. A space-separated list of attributes follows\n-\tthe name; unrecognized attributes are ignored. The list ends\n-\twith a blank line.\n+\ta symref, \": \" for a key-value pair, or\n+\t\"?\" to indicate that the helper could not get the value of the\n+\tref. A space-separated list of attributes follows the name;\n+\tunrecognized attributes are ignored. The list ends with a\n+\tblank line.\n +\n See REF LIST ATTRIBUTES for a list of currently defined attributes.\n+See REF LIST KEYWORDS for a list of currently defined keywords.\n +\n Supported if the helper has the \"fetch\" or \"import\" capability.\n \n@@ -430,6 +435,18 @@ attributes are defined.\n \tThis ref is unchanged since the last import or fetch, although\n \tthe helper cannot necessarily determine what value that produced.\n \n+REF LIST KEYWORDS\n+-----------------\n+\n+The 'list' command may produce a list of key-value pairs.\n+The following keys are defined.\n+\n+'object-format'::\n+\tThe refs are using the given hash algorithm. This keyword is only\n+\tused if the server and client both support the object-format\n+\textension.\n+\n+\n OPTIONS\n -------\n \n@@ -514,6 +531,14 @@ set by Git if the remote helper has the 'option' capability.\n \ttransaction. If successful, all refs will be updated, or none will. If the\n \tremote side does not support this capability, the push will fail.\n \n+'option object-format' {'true'|algorithm}::\n+\tIf 'true', indicate that the caller wants hash algorithm information\n+\tto be passed back from the remote. This mode is used when fetching\n+\trefs.\n++\n+If set to an algorithm, indicate that the caller wants to interact with\n+the remote side using that algorithm.\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\n"},{"id":"398514","messageId":"20200525195930.309665-12-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 11/44] send-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:57Z","receivedAt":"2020-05-25T19:59:59Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\nIf the server does support our hash, advertise it as part of our\ncapabilities.\n\nSigned-off-by: brian m. carlson \n---\n send-pack.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 0abee22283..02aefcb08e 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -363,6 +363,7 @@ int send_pack(struct send_pack_args *args,\n \tint atomic_supported = 0;\n \tint use_push_options = 0;\n \tint push_options_supported = 0;\n+\tint object_format_supported = 0;\n \tunsigned cmds_sent = 0;\n \tint ret;\n \tstruct async demux;\n@@ -389,6 +390,9 @@ int send_pack(struct send_pack_args *args,\n \tif (server_supports(\"push-options\"))\n \t\tpush_options_supported = 1;\n \n+\tif (!server_supports_hash(the_hash_algo->name, &object_format_supported))\n+\t\tdie(_(\"the receiving end does not support this repository's hash algorithm\"));\n+\n \tif (args->push_cert != SEND_PACK_PUSH_CERT_NEVER) {\n \t\tint len;\n \t\tpush_cert_nonce = server_feature_value(\"push-cert\", &len);\n@@ -429,6 +433,8 @@ int send_pack(struct send_pack_args *args,\n \t\tstrbuf_addstr(&cap_buf, \" atomic\");\n \tif (use_push_options)\n \t\tstrbuf_addstr(&cap_buf, \" push-options\");\n+\tif (object_format_supported)\n+\t\tstrbuf_addf(&cap_buf, \" object-format=%s\", the_hash_algo->name);\n \tif (agent_supported)\n \t\tstrbuf_addf(&cap_buf, \" agent=%s\", git_user_agent_sanitized());\n \n"},{"id":"398515","messageId":"20200525195930.309665-28-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 27/44] t1302: expect repo format version 1 for SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:13Z","receivedAt":"2020-05-25T20:00:02Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using SHA-256, we need to take advantage of the extensions section\nin the config file, so we need to use repository format version 1.\nUpdate the test to look for the correct value.\n\nNote that test_oid produces a value without a trailing newline, so use\necho to ensure we print a trailing newline to compare it correctly\nagainst the actual results.\n\nSigned-off-by: brian m. carlson \n---\n t/t1302-repo-version.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh\nindex ce4cff13bb..d60c042ce8 100755\n--- a/t/t1302-repo-version.sh\n+++ b/t/t1302-repo-version.sh\n@@ -8,6 +8,10 @@ test_description='Test repository version check'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n+\ttest_oid_cache <<-\\EOF &&\n+\tversion sha1:0\n+\tversion sha256:1\n+\tEOF\n \tcat >test.patch <<-\\EOF &&\n \tdiff --git a/test.txt b/test.txt\n \tnew file mode 100644\n@@ -23,7 +27,7 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'gitdir selection on normal repos' '\n-\techo 0 >expect &&\n+\techo $(test_oid version) >expect &&\n \tgit config core.repositoryformatversion >actual &&\n \tgit -C test config core.repositoryformatversion >actual2 &&\n \ttest_cmp expect actual &&\n"},{"id":"398516","messageId":"20200525195930.309665-26-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 25/44] t5302: modernize test formatting","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:11Z","receivedAt":"2020-05-25T20:00:03Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Our style these days is to place the description and the opening quote\nof the body on the same line as test_expect_success (if it fits), to\nplace the trailing quote on a line by itself after the body, and to use\ntabs. Since we're going to be making several significant changes to\nthis test, modernize the style to aid in readability of the subsequent\npatches.\n\nThis patch should have no functional change.\n\nSigned-off-by: brian m. carlson \n---\n t/t5302-pack-index.sh | 360 +++++++++++++++++++++---------------------\n 1 file changed, 184 insertions(+), 176 deletions(-)\n\ndiff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\nindex ad07f2f7fc..8981c9b90e 100755\n--- a/t/t5302-pack-index.sh\n+++ b/t/t5302-pack-index.sh\n@@ -7,65 +7,65 @@ test_description='pack index with 64-bit offsets and object CRC'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n- test_oid_init &&\n- rawsz=$(test_oid rawsz) &&\n- rm -rf .git &&\n- git init &&\n- git config pack.threads 1 &&\n- i=1 &&\n- while test $i -le 100\n- do\n- iii=$(printf '%03i' $i)\n-\t test-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n-\t test-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n-\t test-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n-\t test-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n-\t test-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n- echo $iii >file_$iii &&\n-\t test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n- git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n- i=$(expr $i + 1) || return 1\n- done &&\n- { echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n- git update-index --add file_101 &&\n- tree=$(git write-tree) &&\n- commit=$(git commit-tree $tree obj-list &&\n- git update-ref HEAD $commit\n+\ttest_oid_init &&\n+\trawsz=$(test_oid rawsz) &&\n+\trm -rf .git &&\n+\tgit init &&\n+\tgit config pack.threads 1 &&\n+\ti=1 &&\n+\twhile test $i -le 100\n+\tdo\n+\t\tiii=$(printf '%03i' $i)\n+\t\ttest-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree obj-list &&\n+\tgit update-ref HEAD $commit\n '\n \n-test_expect_success \\\n- 'pack-objects with index version 1' \\\n- 'pack1=$(git pack-objects --index-version=1 test-1 &1) ||\n \t! (echo \"$msg\" | grep \"pack too large .* off_t\")\n@@ -91,21 +91,21 @@ else\n \tsay \"# skipping tests concerning 64-bit offsets\"\n fi\n \n-test_expect_success OFF64_T \\\n- 'index v2: verify a pack with some 64-bit offsets' \\\n- 'git verify-pack -v \"test-3-${pack3}.pack\"'\n+test_expect_success OFF64_T 'index v2: verify a pack with some 64-bit offsets' '\n+\tgit verify-pack -v \"test-3-${pack3}.pack\"\n+'\n \n-test_expect_success OFF64_T \\\n- '64-bit offsets: should be different from previous index v2 results' \\\n- '! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"'\n+test_expect_success OFF64_T '64-bit offsets: should be different from previous index v2 results' '\n+\t! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"\n+'\n \n-test_expect_success OFF64_T \\\n- 'index v2: force some 64-bit offsets with index-pack' \\\n- 'git index-pack --index-version=2,0x40000 -o 3.idx \"test-1-${pack1}.pack\"'\n+test_expect_success OFF64_T 'index v2: force some 64-bit offsets with index-pack' '\n+\tgit index-pack --index-version=2,0x40000 -o 3.idx \"test-1-${pack1}.pack\"\n+'\n \n-test_expect_success OFF64_T \\\n- '64-bit offsets: index-pack result should match pack-objects one' \\\n- 'cmp \"test-3-${pack3}.idx\" \"3.idx\"'\n+test_expect_success OFF64_T '64-bit offsets: index-pack result should match pack-objects one' '\n+\tcmp \"test-3-${pack3}.idx\" \"3.idx\"\n+'\n \n test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2 (cheat)' '\n \t# This cheats by knowing which lower offset should still be encoded\n@@ -120,135 +120,143 @@ test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '\n # returns the object number for given object in given pack index\n index_obj_nr()\n {\n- idx_file=$1\n- object_sha1=$2\n- nr=0\n- git show-index < $idx_file |\n- while read offs sha1 extra\n- do\n- nr=$(($nr + 1))\n- test \"$sha1\" = \"$object_sha1\" || continue\n- echo \"$(($nr - 1))\"\n- break\n- done\n+\tidx_file=$1\n+\tobject_sha1=$2\n+\tnr=0\n+\tgit show-index < $idx_file |\n+\twhile read offs sha1 extra\n+\tdo\n+\t nr=$(($nr + 1))\n+\t test \"$sha1\" = \"$object_sha1\" || continue\n+\t echo \"$(($nr - 1))\"\n+\t break\n+\tdone\n }\n \n # returns the pack offset for given object as found in given pack index\n index_obj_offset()\n {\n- idx_file=$1\n- object_sha1=$2\n- git show-index < $idx_file | grep $object_sha1 |\n- ( read offs extra && echo \"$offs\" )\n+\tidx_file=$1\n+\tobject_sha1=$2\n+\tgit show-index < $idx_file | grep $object_sha1 |\n+\t( read offs extra && echo \"$offs\" )\n }\n \n-test_expect_success \\\n- '[index v1] 1) stream pack to repository' \\\n- 'git index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n- git prune-packed &&\n- git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n- cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n- cmp \"test-1-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n+test_expect_success '[index v1] 1) stream pack to repository' '\n+\tgit index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n+\tgit prune-packed &&\n+\tgit count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n+\tcmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n+\tcmp \"test-1-${pack1}.idx\"\t\".git/objects/pack/pack-${pack1}.idx\"\n+'\n \n test_expect_success \\\n- '[index v1] 2) create a stealth corruption in a delta base reference' \\\n- '# This test assumes file_101 is a delta smaller than 16 bytes.\n- # It should be against file_100 but we substitute its base for file_099\n- sha1_101=$(git hash-object file_101) &&\n- sha1_099=$(git hash-object file_099) &&\n- offs_101=$(index_obj_offset 1.idx $sha1_101) &&\n- nr_099=$(index_obj_nr 1.idx $sha1_099) &&\n- chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n- recordsz=$((rawsz + 4)) &&\n- dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n- if=\".git/objects/pack/pack-${pack1}.idx\" \\\n- skip=$((4 + 256 * 4 + $nr_099 * recordsz)) \\\n- bs=1 count=$rawsz conv=notrunc &&\n- git cat-file blob $sha1_101 > file_101_foo1'\n+\t'[index v1] 2) create a stealth corruption in a delta base reference' '\n+\t# This test assumes file_101 is a delta smaller than 16 bytes.\n+\t# It should be against file_100 but we substitute its base for file_099\n+\tsha1_101=$(git hash-object file_101) &&\n+\tsha1_099=$(git hash-object file_099) &&\n+\toffs_101=$(index_obj_offset 1.idx $sha1_101) &&\n+\tnr_099=$(index_obj_nr 1.idx $sha1_099) &&\n+\tchmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n+\trecordsz=$((rawsz + 4)) &&\n+\tdd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n+\t if=\".git/objects/pack/pack-${pack1}.idx\" \\\n+\t skip=$((4 + 256 * 4 + $nr_099 * recordsz)) \\\n+\t bs=1 count=$rawsz conv=notrunc &&\n+\tgit cat-file blob $sha1_101 > file_101_foo1\n+'\n \n test_expect_success \\\n- '[index v1] 3) corrupted delta happily returned wrong data' \\\n- 'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'\n+\t'[index v1] 3) corrupted delta happily returned wrong data' '\n+\ttest -f file_101_foo1 && ! cmp file_101 file_101_foo1\n+'\n \n test_expect_success \\\n- '[index v1] 4) confirm that the pack is actually corrupted' \\\n- 'test_must_fail git fsck --full $commit'\n+\t'[index v1] 4) confirm that the pack is actually corrupted' '\n+\ttest_must_fail git fsck --full $commit\n+'\n \n test_expect_success \\\n- '[index v1] 5) pack-objects happily reuses corrupted data' \\\n- 'pack4=$(git pack-objects test-4 file_101_foo2\n+'\n \n test_expect_success \\\n- '[index v2] 1) stream pack to repository' \\\n- 'rm -f .git/objects/pack/* &&\n- git index-pack --index-version=2 --stdin < \"test-1-${pack1}.pack\" &&\n- git prune-packed &&\n- git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n- cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n- cmp \"test-2-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n+\t'[index v2] 3) corrupted delta happily returned wrong data' '\n+\ttest -f file_101_foo2 && ! cmp file_101 file_101_foo2\n+'\n \n test_expect_success \\\n- '[index v2] 2) create a stealth corruption in a delta base reference' \\\n- '# This test assumes file_101 is a delta smaller than 16 bytes.\n- # It should be against file_100 but we substitute its base for file_099\n- sha1_101=$(git hash-object file_101) &&\n- sha1_099=$(git hash-object file_099) &&\n- offs_101=$(index_obj_offset 1.idx $sha1_101) &&\n- nr_099=$(index_obj_nr 1.idx $sha1_099) &&\n- chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n- dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n- if=\".git/objects/pack/pack-${pack1}.idx\" \\\n- skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \\\n- bs=1 count=$rawsz conv=notrunc &&\n- git cat-file blob $sha1_101 > file_101_foo2'\n+\t'[index v2] 4) confirm that the pack is actually corrupted' '\n+\ttest_must_fail git fsck --full $commit\n+'\n \n test_expect_success \\\n- '[index v2] 3) corrupted delta happily returned wrong data' \\\n- 'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'\n+\t'[index v2] 5) pack-objects refuses to reuse corrupted data' '\n+\ttest_must_fail git pack-objects test-5 /dev/null || exit 1\n- done /dev/null || exit 1\n+\t done wrong-tag <wrong-tag <err &&\n- grep \"^warning:.* expected .tagger. line\" err\n+\ttag=$(git hash-object -t tag -w --stdin err &&\n+\tgrep \"^warning:.* expected .tagger. line\" err\n '\n \n test_expect_success 'index-pack --fsck-objects also warns upon missing tagger in tag' '\n- git index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&\n- grep \"^warning:.* expected .tagger. line\" err\n+\tgit index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&\n+\tgrep \"^warning:.* expected .tagger. line\" err\n '\n \n test_done\n"},{"id":"398517","messageId":"20200525195930.309665-33-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 32/44] t5500: make hash independent","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:18Z","receivedAt":"2020-05-25T20:00:04Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This test has hard-coded pkt-lines with object IDs. The pkt-line\nlengths necessarily differ between hash algorithms, so generate these\nlines with the packetize helper so they're always the right size. In\naddition, we will require an object-format capability for SHA-256, so\npass that capability on to the upload-pack process.\n\nSigned-off-by: brian m. carlson \n---\n t/t5500-fetch-pack.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 8c54e34ef1..dfed113247 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -871,9 +871,10 @@ test_expect_success 'shallow since with commit graph and already-seen commit' '\n \n \tGIT_PROTOCOL=version=2 git upload-pack . <<-EOF >/dev/null\n \t0012command=fetch\n+\t$(echo \"object-format=$(test_oid algo)\" | packetize)\n \t00010013deepen-since 1\n-\t0032want $(git rev-parse other)\n-\t0032have $(git rev-parse master)\n+\t$(echo \"want $(git rev-parse other)\" | packetize)\n+\t$(echo \"have $(git rev-parse master)\" | packetize)\n \t0000\n \tEOF\n \t)\n"},{"id":"398518","messageId":"20200525195930.309665-35-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 34/44] remote-curl: detect algorithm for dumb HTTP by size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:20Z","receivedAt":"2020-05-25T20:00:05Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When reading the info/refs file for a repository, we have no explicit\nway to detect which hash algorithm is in use because the file doesn't\nprovide one. Detect the hash algorithm in use by the size of the first\nobject ID.\n\nWe anonymize the URL like elsewhere in the function in case the user has\ndecided to include a secret in the URL.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 3ed0dfec1b..617edfedf4 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -252,6 +252,19 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \treturn list;\n }\n \n+static const struct git_hash_algo *detect_hash_algo(struct discovery *heads)\n+{\n+\tconst char *p = memchr(heads->buf, '\\t', heads->len);\n+\tint algo;\n+\tif (!p)\n+\t\treturn NULL;\n+\n+\talgo = hash_algo_by_length((p - heads->buf) / 2);\n+\tif (algo == GIT_HASH_UNKNOWN)\n+\t\treturn NULL;\n+\treturn &hash_algos[algo];\n+}\n+\n static struct ref *parse_info_refs(struct discovery *heads)\n {\n \tchar *data, *start, *mid;\n@@ -262,6 +275,12 @@ static struct ref *parse_info_refs(struct discovery *heads)\n \tstruct ref *ref = NULL;\n \tstruct ref *last_ref = NULL;\n \n+\toptions.hash_algo = detect_hash_algo(heads);\n+\tif (!options.hash_algo)\n+\t\tdie(\"%sinfo/refs not valid: could not determine hash algorithm; \"\n+\t\t \"is this a git repository?\",\n+\t\t transport_anonymize_url(url.buf));\n+\n \tdata = heads->buf;\n \tstart = NULL;\n \tmid = data;\n@@ -272,13 +291,13 @@ static struct ref *parse_info_refs(struct discovery *heads)\n \t\tif (data[i] == '\\t')\n \t\t\tmid = &data[i];\n \t\tif (data[i] == '\\n') {\n-\t\t\tif (mid - start != the_hash_algo->hexsz)\n+\t\t\tif (mid - start != options.hash_algo->hexsz)\n \t\t\t\tdie(_(\"%sinfo/refs not valid: is this a git repository?\"),\n \t\t\t\t transport_anonymize_url(url.buf));\n \t\t\tdata[i] = 0;\n \t\t\tref_name = mid + 1;\n \t\t\tref = alloc_ref(ref_name);\n-\t\t\tget_oid_hex(start, &ref->old_oid);\n+\t\t\tget_oid_hex_algop(start, &ref->old_oid, options.hash_algo);\n \t\t\tif (!refs)\n \t\t\t\trefs = ref;\n \t\t\tif (last_ref)\n"},{"id":"398519","messageId":"20200525195930.309665-36-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 35/44] builtin/index-pack: add option to specify hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:21Z","receivedAt":"2020-05-25T20:00:07Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"git index-pack is usually run in a repository, but need not be. Since\npacks don't contains information on the algorithm in use, instead\nrelying on context, add an option to index-pack to tell it which one\nwe're using in case someone runs it outside of a repository. Since\nusing --stdin necessarily implies a repository, don't allow specifying\nan object format if it's provided to prevent users from passing an\noption that won't work. Add documentation for this option.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-index-pack.txt | 8 ++++++++\n builtin/index-pack.c | 8 ++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex d5b7560bfe..9316d9a80b 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -93,6 +93,14 @@ OPTIONS\n --max-input-size=::\n \tDie, if the pack is larger than .\n \n+--object-format=::\n+\tSpecify the given object format (hash algorithm) for the pack. The valid\n+\tvalues are 'sha1' and (if enabled) 'sha256'. The default is the algorithm for\n+\tthe current repository (set by `extensions.objectFormat`), or 'sha1' if no\n+\tvalue is set or outside a repository.\n++\n+This option cannot be used with --stdin.\n+\n NOTES\n -----\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7bea1fba52..f865666db9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1667,6 +1667,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tunsigned char pack_hash[GIT_MAX_RAWSZ];\n \tunsigned foreign_nr = 1;\t/* zero is a \"good\" value, assume bad */\n \tint report_end_of_input = 0;\n+\tint hash_algo = 0;\n \n \t/*\n \t * index-pack never needs to fetch missing objects except when\n@@ -1760,6 +1761,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t} else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n \t\t\t\tmax_input_size = strtoumax(arg, NULL, 10);\n+\t\t\t} else if (skip_prefix(arg, \"--object-format=\", &arg)) {\n+\t\t\t\thash_algo = hash_algo_by_name(arg);\n+\t\t\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n+\t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -1776,6 +1782,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"--fix-thin cannot be used without --stdin\"));\n \tif (from_stdin && !startup_info->have_repository)\n \t\tdie(_(\"--stdin requires a git repository\"));\n+\tif (from_stdin && hash_algo)\n+\t\tdie(_(\"--object-format cannot be used with --stdin\"));\n \tif (!index_name && pack_name)\n \t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n \n"},{"id":"398520","messageId":"20200525195930.309665-44-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 43/44] bundle: detect hash algorithm when reading refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:29Z","receivedAt":"2020-05-25T20:00:08Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Much like with the dumb HTTP transport, there isn't a way to explicitly\nspecify the hash algorithm when dealing with a bundle, so detect the\nalgorithm based on the length of the object IDs in the prerequisites and\nref advertisements.\n\nSigned-off-by: brian m. carlson \n---\n bundle.c | 22 +++++++++++++++++++++-\n bundle.h | 1 +\n transport.c | 10 ++++++++--\n 3 files changed, 30 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 99439e07a1..2a0d744d3f 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -23,6 +23,17 @@ static void add_to_ref_list(const struct object_id *oid, const char *name,\n \tlist->nr++;\n }\n \n+static const struct git_hash_algo *detect_hash_algo(struct strbuf *buf)\n+{\n+\tsize_t len = strcspn(buf->buf, \" \\n\");\n+\tint algo;\n+\n+\talgo = hash_algo_by_length(len / 2);\n+\tif (algo == GIT_HASH_UNKNOWN)\n+\t\treturn NULL;\n+\treturn &hash_algos[algo];\n+}\n+\n static int parse_bundle_header(int fd, struct bundle_header *header,\n \t\t\t const char *report_path)\n {\n@@ -52,12 +63,21 @@ static int parse_bundle_header(int fd, struct bundle_header *header,\n \t\t}\n \t\tstrbuf_rtrim(&buf);\n \n+\t\tif (!header->hash_algo) {\n+\t\t\theader->hash_algo = detect_hash_algo(&buf);\n+\t\t\tif (!header->hash_algo) {\n+\t\t\t\terror(_(\"unknown hash algorithm length\"));\n+\t\t\t\tstatus = -1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n \t\t/*\n \t\t * Tip lines have object name, SP, and refname.\n \t\t * Prerequisites have object name that is optionally\n \t\t * followed by SP and subject line.\n \t\t */\n-\t\tif (parse_oid_hex(buf.buf, &oid, &p) ||\n+\t\tif (parse_oid_hex_algop(buf.buf, &oid, &p, header->hash_algo) ||\n \t\t (*p && !isspace(*p)) ||\n \t\t (!is_prereq && !*p)) {\n \t\t\tif (report_path)\ndiff --git a/bundle.h b/bundle.h\nindex ceab0c7475..2dc9442024 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -15,6 +15,7 @@ struct ref_list {\n struct bundle_header {\n \tstruct ref_list prerequisites;\n \tstruct ref_list references;\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n int is_bundle(const char *path, int quiet);\ndiff --git a/transport.c b/transport.c\nindex b43d985f90..38a432be69 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -143,6 +143,9 @@ static struct ref *get_refs_from_bundle(struct transport *transport,\n \tdata->fd = read_bundle_header(transport->url, &data->header);\n \tif (data->fd < 0)\n \t\tdie(_(\"could not read bundle '%s'\"), transport->url);\n+\n+\ttransport->hash_algo = data->header.hash_algo;\n+\n \tfor (i = 0; i < data->header.references.nr; i++) {\n \t\tstruct ref_list_entry *e = data->header.references.list + i;\n \t\tstruct ref *ref = alloc_ref(e->name);\n@@ -157,11 +160,14 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n \tstruct bundle_transport_data *data = transport->data;\n+\tint ret;\n \n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle(transport, 0, NULL);\n-\treturn unbundle(the_repository, &data->header, data->fd,\n-\t\t\ttransport->progress ? BUNDLE_VERBOSE : 0);\n+\tret = unbundle(the_repository, &data->header, data->fd,\n+\t\t\t transport->progress ? BUNDLE_VERBOSE : 0);\n+\ttransport->hash_algo = data->header.hash_algo;\n+\treturn ret;\n }\n \n static int close_bundle(struct transport *transport)\n"},{"id":"398521","messageId":"20200525195930.309665-40-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 39/44] t5702: offer an object-format capability in the test","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:25Z","receivedAt":"2020-05-25T20:00:09Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In order to make this test work with SHA-256, offer an object-format\ncapability so that both sides use the same algorithm.\n\nSigned-off-by: brian m. carlson \n---\n t/t5702-protocol-v2.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 5039e66dc4..116358b9ac 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -13,6 +13,7 @@ start_git_daemon --export-all --enable=receive-pack\n daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n \n test_expect_success 'create repo to be served by git-daemon' '\n+\ttest_oid_init &&\n \tgit init \"$daemon_parent\" &&\n \ttest_commit -C \"$daemon_parent\" one\n '\n@@ -394,6 +395,7 @@ test_expect_success 'even with handcrafted request, filter does not work if not\n \t# Custom request that tries to filter even though it is not advertised.\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=fetch\n+\tobject-format=$(test_oid algo)\n \t0001\n \twant $(git -C server rev-parse master)\n \tfilter blob:none\n"},{"id":"398522","messageId":"20200525195930.309665-45-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 44/44] remote-testgit: adapt for object-format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:30Z","receivedAt":"2020-05-25T20:00:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using an algorithm other than SHA-1, we need the remote helper to\nadvertise support for the object-format extension and provide\ninformation back to us so that we can properly parse refs and return\ndata. Ensure that the test remote helper understands these extensions.\n\nSigned-off-by: brian m. carlson \n---\n t/t5801/git-remote-testgit | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/t/t5801/git-remote-testgit b/t/t5801/git-remote-testgit\nindex 6b9f0b5dc7..1544d6dc6b 100755\n--- a/t/t5801/git-remote-testgit\n+++ b/t/t5801/git-remote-testgit\n@@ -52,9 +52,11 @@ do\n \t\ttest -n \"$GIT_REMOTE_TESTGIT_SIGNED_TAGS\" && echo \"signed-tags\"\n \t\ttest -n \"$GIT_REMOTE_TESTGIT_NO_PRIVATE_UPDATE\" && echo \"no-private-update\"\n \t\techo 'option'\n+\t\techo 'object-format'\n \t\techo\n \t\t;;\n \tlist)\n+\t\techo \":object-format $(git rev-parse --show-object-format=storage)\"\n \t\tgit for-each-ref --format='? %(refname)' 'refs/heads/' 'refs/tags/'\n \t\thead=$(git symbolic-ref HEAD)\n \t\techo \"@$head HEAD\"\n@@ -139,6 +141,10 @@ do\n \t\t\ttest $val = \"true\" && force=\"true\" || force=\n \t\t\techo \"ok\"\n \t\t\t;;\n+\t\tobject-format)\n+\t\t\ttest $val = \"true\" && object_format=\"true\" || object_format=\n+\t\t\techo \"ok\"\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"unsupported\"\n \t\t\t;;\n"},{"id":"398523","messageId":"20200525195930.309665-39-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 38/44] t/helper: initialize the repository for test-sha1-array","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:24Z","receivedAt":"2020-05-25T20:00:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"test-sha1-array uses the_hash_algo under the hood. Since t0064 wants to\nuse the value that is correct for the hash algorithm that we're testing,\nmake sure the test helper initializes the repository to set\nthe_hash_algo correctly.\n\nSigned-off-by: brian m. carlson \n---\n t/helper/test-oid-array.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/helper/test-oid-array.c b/t/helper/test-oid-array.c\nindex ce9fd5f091..b16cd0b11b 100644\n--- a/t/helper/test-oid-array.c\n+++ b/t/helper/test-oid-array.c\n@@ -12,6 +12,9 @@ int cmd__oid_array(int argc, const char **argv)\n {\n \tstruct oid_array array = OID_ARRAY_INIT;\n \tstruct strbuf line = STRBUF_INIT;\n+\tint nongit_ok;\n+\n+\tsetup_git_directory_gently(&nongit_ok);\n \n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tconst char *arg;\n"},{"id":"398524","messageId":"20200525195930.309665-38-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 37/44] remote-curl: avoid truncating refs with ls-remote","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:23Z","receivedAt":"2020-05-25T20:00:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Normally, the remote-curl transport helper is aware of the hash\nalgorithm we're using because we're in a repo with the appropriate hash\nalgorithm set. However, when using git ls-remote outside of a\nrepository, we won't have initialized the hash algorithm properly, so\nuse hash_to_hex_algop to print the ref corresponding to the algorithm\nwe've detected.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 617edfedf4..c077323008 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -548,7 +548,9 @@ static void output_refs(struct ref *refs)\n \t\tif (posn->symref)\n \t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n \t\telse\n-\t\t\tprintf(\"%s %s\\n\", oid_to_hex(&posn->old_oid), posn->name);\n+\t\t\tprintf(\"%s %s\\n\", hash_to_hex_algop(posn->old_oid.hash,\n+\t\t\t\t\t\t\t options.hash_algo),\n+\t\t\t\t\t posn->name);\n \t}\n \tprintf(\"\\n\");\n \tfflush(stdout);\n"},{"id":"398525","messageId":"20200525195930.309665-37-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 36/44] t1050: pass algorithm to index-pack when outside repo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:22Z","receivedAt":"2020-05-25T20:00:15Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When outside a repository, git index-pack is unable to guess the hash\nalgorithm in use for a pack, since packs don't contain any information\non the algorithm in use. Pass an option to index-pack to help it out in\nthis test.\n\nSigned-off-by: brian m. carlson \n---\n t/t1050-large.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 7f88ea07c2..6a56d1ca24 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -12,6 +12,7 @@ file_size () {\n }\n \n test_expect_success setup '\n+\ttest_oid_init &&\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n@@ -177,7 +178,8 @@ test_expect_success 'git-show a large file' '\n \n test_expect_success 'index-pack' '\n \tgit clone file://\"$(pwd)\"/.git foo &&\n-\tGIT_DIR=non-existent git index-pack --strict --verify foo/.git/objects/pack/*.pack\n+\tGIT_DIR=non-existent git index-pack --object-format=$(test_oid algo) \\\n+\t\t--strict --verify foo/.git/objects/pack/*.pack\n '\n \n test_expect_success 'repack' '\n"},{"id":"398526","messageId":"20200525195930.309665-34-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 33/44] builtin/ls-remote: initialize repository based on fetch","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:19Z","receivedAt":"2020-05-25T20:00:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"ls-remote may or may not operate within a repository, and as such will\nnot have been initialized with the repository's hash algorithm. Even if\nit were, the remote side could be using a different algorithm and we\nwould still want to display those refs properly. Find the hash\nalgorithm used by the remote side by querying the transport object and\nset our hash algorithm accordingly.\n\nWithout this change, if the remote side is using SHA-256, we truncate\nthe refs to 40 hex characters, since that's the length of the default\nhash algorithm (SHA-1).\n\nNote that technically this is not a correct setting of the repository\nhash algorithm since, if we are in a repository, it might be one of a\ndifferent hash algorithm from the remote side. However, our current\ncode paths don't handle multiple algorithms and won't for some time, so\nthis is the best we can do. We rely on the fact that ls-remote never\nmodifies the current repository, which is a reasonable assumption to\nmake.\n\nSigned-off-by: brian m. carlson \n---\n builtin/ls-remote.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex 6ef519514b..3a4dd12903 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -118,6 +118,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\ttransport->server_options = &server_options;\n \n \tref = transport_get_remote_refs(transport, &ref_prefixes);\n+\tif (ref) {\n+\t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t}\n \tif (transport_disconnect(transport)) {\n \t\tUNLEAK(sorting);\n \t\treturn 1;\n"},{"id":"398527","messageId":"20200525195930.309665-31-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 30/44] connect: parse v2 refs with correct hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:16Z","receivedAt":"2020-05-25T20:00:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using protocol v2, we need to know what hash algorithm is used by\nthe remote end. See if the server has sent us an object-format\ncapability, and if so, use it to determine the hash algorithm in use and\nset that value in the packet reader. Parse the refs using this\nalgorithm.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 21 ++++++++++++++++-----\n 1 file changed, 16 insertions(+), 5 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 1d05bc56ed..66650ff2d3 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -283,7 +283,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \t\tdie(_(\"protocol error: unexpected capabilities^{}\"));\n \t} else if (check_ref(name, flags)) {\n \t\tstruct ref *ref = alloc_ref(name);\n-\t\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n@@ -395,7 +395,7 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\tgoto out;\n \t}\n \n-\tif (parse_oid_hex(line_sections.items[i++].string, &old_oid, &end) ||\n+\tif (parse_oid_hex_algop(line_sections.items[i++].string, &old_oid, &end, reader->hash_algo) ||\n \t *end) {\n \t\tret = 0;\n \t\tgoto out;\n@@ -403,7 +403,7 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \n \tref = alloc_ref(line_sections.items[i++].string);\n \n-\toidcpy(&ref->old_oid, &old_oid);\n+\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n \t**list = ref;\n \t*list = &ref->next;\n \n@@ -416,7 +416,8 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\t\tstruct object_id peeled_oid;\n \t\t\tchar *peeled_name;\n \t\t\tstruct ref *peeled;\n-\t\t\tif (parse_oid_hex(arg, &peeled_oid, &end) || *end) {\n+\t\t\tif (parse_oid_hex_algop(arg, &peeled_oid, &end,\n+\t\t\t\t\t\treader->hash_algo) || *end) {\n \t\t\t\tret = 0;\n \t\t\t\tgoto out;\n \t\t\t}\n@@ -424,7 +425,8 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n \t\t\tpeeled = alloc_ref(peeled_name);\n \n-\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n+\t\t\tmemcpy(peeled->old_oid.hash, peeled_oid.hash,\n+\t\t\t reader->hash_algo->rawsz);\n \t\t\t**list = peeled;\n \t\t\t*list = &peeled->next;\n \n@@ -443,6 +445,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t const struct string_list *server_options)\n {\n \tint i;\n+\tconst char *hash_name;\n \t*list = NULL;\n \n \tif (server_supports_v2(\"ls-refs\", 1))\n@@ -451,6 +454,14 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \tif (server_supports_v2(\"agent\", 0))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n \n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n+\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n+\t}\n+\n \tif (server_options && server_options->nr &&\n \t server_supports_v2(\"server-option\", 1))\n \t\tfor (i = 0; i < server_options->nr; i++)\n"},{"id":"398528","messageId":"20200525195930.309665-6-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 05/44] remote: advertise the object-format capability on the server side","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:51Z","receivedAt":"2020-05-25T20:00:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Advertise the current hash algorithm in use by using the object-format\ncapability as part of the ref advertisement.\n\nSigned-off-by: brian m. carlson \n---\n builtin/receive-pack.c | 1 +\n upload-pack.c | 3 ++-\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex ea3d0f01af..4ffa501dce 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -249,6 +249,7 @@ static void show_ref(const char *path, const struct object_id *oid)\n \t\t\tstrbuf_addf(&cap, \" push-cert=%s\", push_cert_nonce);\n \t\tif (advertise_push_options)\n \t\t\tstrbuf_addstr(&cap, \" push-options\");\n+\t\tstrbuf_addf(&cap, \" object-format=%s\", the_hash_algo->name);\n \t\tstrbuf_addf(&cap, \" agent=%s\", git_user_agent_sanitized());\n \t\tpacket_write_fmt(1, \"%s %s%c%s\\n\",\n \t\t\t oid_to_hex(oid), path, 0, cap.buf);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0478bff3e7..636911fec5 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1008,7 +1008,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n \t\tformat_symref_info(&symref_info, cb_data);\n-\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s agent=%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s object-format=%s agent=%s\\n\",\n \t\t\t oid_to_hex(oid), refname_nons,\n \t\t\t 0, capabilities,\n \t\t\t (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n@@ -1018,6 +1018,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\t\t stateless_rpc ? \" no-done\" : \"\",\n \t\t\t symref_info.buf,\n \t\t\t allow_filter ? \" filter\" : \"\",\n+\t\t\t the_hash_algo->name,\n \t\t\t git_user_agent_sanitized());\n \t\tstrbuf_release(&symref_info);\n \t} else {\n"},{"id":"398529","messageId":"20200525195930.309665-24-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 23/44] t3200: mark assertion with SHA1 prerequisite","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:09Z","receivedAt":"2020-05-25T20:00:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"One of the test assertions in this test checks that git branch -m works\neven without a .git/config file. However, if the repository requires\nconfiguration extensions, such as because it uses a non-SHA-1 algorithm,\nthis assertion will fail. Mark the assertion as requiring SHA-1.\n\nSigned-off-by: brian m. carlson \n---\n t/t3200-branch.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\nindex 411a70b0ce..2a3fedc6b0 100755\n--- a/t/t3200-branch.sh\n+++ b/t/t3200-branch.sh\n@@ -402,7 +402,7 @@ EOF\n \n mv .git/config .git/config-saved\n \n-test_expect_success 'git branch -m q q2 without config should succeed' '\n+test_expect_success SHA1 'git branch -m q q2 without config should succeed' '\n \tgit branch -m q q2 &&\n \tgit branch -m q2 q\n '\n"},{"id":"398530","messageId":"20200525195930.309665-30-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 29/44] connect: pass full packet reader when parsing v2 refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:15Z","receivedAt":"2020-05-25T20:00:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're parsing refs, we need to know not only what the line we're\nparsing is, but also the hash algorithm we should use to parse it, which\nis stored in the reader object. Pass the packet reader object through\nto the protocol v2 ref parsing function.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 915f1736a0..1d05bc56ed 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -374,7 +374,7 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n }\n \n /* Returns 1 when a valid ref has been added to `list`, 0 otherwise */\n-static int process_ref_v2(const char *line, struct ref ***list)\n+static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n {\n \tint ret = 1;\n \tint i = 0;\n@@ -382,6 +382,7 @@ static int process_ref_v2(const char *line, struct ref ***list)\n \tstruct ref *ref;\n \tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n \tconst char *end;\n+\tconst char *line = reader->line;\n \n \t/*\n \t * Ref lines have a number of fields which are space deliminated. The\n@@ -469,7 +470,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \n \t/* Process response from server */\n \twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n-\t\tif (!process_ref_v2(reader->line, &list))\n+\t\tif (!process_ref_v2(reader, &list))\n \t\t\tdie(_(\"invalid ls-refs response: %s\"), reader->line);\n \t}\n \n"},{"id":"398531","messageId":"20200525195930.309665-10-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 09/44] transport: add a hash algorithm member","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:55Z","receivedAt":"2020-05-25T20:00:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When connecting to a remote system, we need to know what hash algorithm\nit will be using to talk to us. Add a hash_algo member to struct\ntransport and add a function to read this data from the transport\nobject.\n\nSigned-off-by: brian m. carlson \n---\n transport.c | 8 ++++++++\n transport.h | 8 ++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/transport.c b/transport.c\nindex 15f5ba4e8f..b43d985f90 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -311,6 +311,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \t\tBUG(\"unknown protocol version\");\n \t}\n \tdata->got_remote_heads = 1;\n+\ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n \t\tBUG(\"buffer must be empty at the end of handshake()\");\n@@ -996,9 +997,16 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\t\tret->smart_options->receivepack = remote->receivepack;\n \t}\n \n+\tret->hash_algo = &hash_algos[GIT_HASH_SHA1];\n+\n \treturn ret;\n }\n \n+const struct git_hash_algo *transport_get_hash_algo(struct transport *transport)\n+{\n+\treturn transport->hash_algo;\n+}\n+\n int transport_set_option(struct transport *transport,\n \t\t\t const char *name, const char *value)\n {\ndiff --git a/transport.h b/transport.h\nindex 4298c855be..2a9f96c05a 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -115,6 +115,8 @@ struct transport {\n \tstruct git_transport_options *smart_options;\n \n \tenum transport_family family;\n+\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n #define TRANSPORT_PUSH_ALL\t\t\t(1<<0)\n@@ -243,6 +245,12 @@ int transport_push(struct repository *repo,\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t const struct argv_array *ref_prefixes);\n \n+/*\n+ * Fetch the hash algorithm used by a remote.\n+ *\n+ * This can only be called after fetching the remote refs.\n+ */\n+const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n int transport_disconnect(struct transport *transport);\n"},{"id":"398532","messageId":"20200525195930.309665-11-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 10/44] connect: add function to detect supported v1 hash functions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:56Z","receivedAt":"2020-05-25T20:00:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a function, server_supports_hash, to see if the remote server\nsupports a particular hash algorithm when speaking protocol v1.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 22 ++++++++++++++++++++++\n connect.h | 1 +\n 2 files changed, 23 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex ad0e4e8e56..eaa13b41bb 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -511,6 +511,28 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \treturn NULL;\n }\n \n+int server_supports_hash(const char *desired, int *feature_supported)\n+{\n+\tint offset = 0;\n+\tint len;\n+\tconst char *hash;\n+\n+\thash = next_server_feature_value(\"object-format\", &len, &offset);\n+\tif (feature_supported)\n+\t\t*feature_supported = !!hash;\n+\tif (!hash) {\n+\t\thash = hash_algos[GIT_HASH_SHA1].name;\n+\t\tlen = strlen(hash);\n+\t}\n+\twhile (hash) {\n+\t\tif (!xstrncmpz(desired, hash, len))\n+\t\t\treturn 1;\n+\n+\t\thash = next_server_feature_value(\"object-format\", &len, &offset);\n+\t}\n+\treturn 0;\n+}\n+\n int parse_feature_request(const char *feature_list, const char *feature)\n {\n \treturn !!parse_feature_value(feature_list, feature, NULL, NULL);\ndiff --git a/connect.h b/connect.h\nindex 4d76a6017d..fc75d6a457 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -18,6 +18,7 @@ int url_is_local_not_ssh(const char *url);\n struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n+int server_supports_hash(const char *desired, int *feature_supported);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n"},{"id":"398533","messageId":"20200525195930.309665-8-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 07/44] connect: add function to fetch value of a v2 server capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:53Z","receivedAt":"2020-05-25T20:00:29Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"So far in protocol v2, all of our server capabilities that have values\nhave not had values that we've been interested in parsing. For example,\nwe receive but ignore the agent value.\n\nHowever, in a future commit, we're going to want to parse out the value\nof a server capability. To make this easy, add a function,\nserver_feature_v2, that can fetch the value provided as part of the\nserver capability.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 15 +++++++++++++++\n connect.h | 1 +\n 2 files changed, 16 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex 2b55a32d4d..ad0e4e8e56 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -84,6 +84,21 @@ int server_supports_v2(const char *c, int die_on_error)\n \treturn 0;\n }\n \n+int server_feature_v2(const char *c, const char **v)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (*out == '=')) {\n+\t\t\t*v = out + 1;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n int server_supports_feature(const char *c, const char *feature,\n \t\t\t int die_on_error)\n {\ndiff --git a/connect.h b/connect.h\nindex 5f2382e018..4d76a6017d 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -19,6 +19,7 @@ struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_v2(const char *c, int die_on_error);\n+int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n \t\t\t int die_on_error);\n \n"},{"id":"398534","messageId":"20200525195930.309665-9-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 08/44] pkt-line: add a member for hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:54Z","receivedAt":"2020-05-25T20:00:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a member for the hash algorithm currently in use to the packet\nreader so it can parse references correctly.\n\nSigned-off-by: brian m. carlson \n---\n pkt-line.c | 1 +\n pkt-line.h | 3 +++\n 2 files changed, 4 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex a0e87b1e81..a4aea075de 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -479,6 +479,7 @@ void packet_reader_init(struct packet_reader *reader, int fd,\n \treader->buffer_size = sizeof(packet_buffer);\n \treader->options = options;\n \treader->me = \"git\";\n+\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n }\n \n enum packet_read_status packet_reader_read(struct packet_reader *reader)\ndiff --git a/pkt-line.h b/pkt-line.h\nindex fef3a0d792..4cd9435e9a 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -166,6 +166,9 @@ struct packet_reader {\n \n \tunsigned use_sideband : 1;\n \tconst char *me;\n+\n+\t/* hash algorithm in use */\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n /*\n"},{"id":"398535","messageId":"20200525195930.309665-7-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 06/44] connect: add function to parse multiple v1 capability values","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:58:52Z","receivedAt":"2020-05-25T20:00:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In a capability response, we can have multiple symref entries. In the\nfuture, we will also allow for multiple hash algorithms to be specified.\nTo avoid duplication, expand the parse_feature_value function to take an\noptional offset where the parsing should begin next time. Add a wrapper\nfunction that allows us to query the next server feature value, and use\nit in the existing symref parsing code.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 30 +++++++++++++++++++++---------\n 1 file changed, 21 insertions(+), 9 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex ccc5274189..2b55a32d4d 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -18,7 +18,8 @@\n \n static char *server_capabilities_v1;\n static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n-static const char *parse_feature_value(const char *, const char *, int *);\n+static const char *parse_feature_value(const char *, const char *, int *, int *);\n+static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n \n static int check_ref(const char *name, unsigned int flags)\n {\n@@ -180,17 +181,16 @@ static void parse_one_symref_info(struct string_list *symref, const char *val, i\n static void annotate_refs_with_symref_info(struct ref *ref)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\tconst char *feature_list = server_capabilities_v1;\n+\tint offset = 0;\n \n-\twhile (feature_list) {\n+\twhile (1) {\n \t\tint len;\n \t\tconst char *val;\n \n-\t\tval = parse_feature_value(feature_list, \"symref\", &len);\n+\t\tval = next_server_feature_value(\"symref\", &len, &offset);\n \t\tif (!val)\n \t\t\tbreak;\n \t\tparse_one_symref_info(&symref, val, len);\n-\t\tfeature_list = val + 1;\n \t}\n \tstring_list_sort(&symref);\n \n@@ -452,7 +452,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \treturn list;\n }\n \n-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n+static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n {\n \tint len;\n \n@@ -460,6 +460,8 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \t\treturn NULL;\n \n \tlen = strlen(feature);\n+\tif (offset)\n+\t\tfeature_list += *offset;\n \twhile (*feature_list) {\n \t\tconst char *found = strstr(feature_list, feature);\n \t\tif (!found)\n@@ -474,9 +476,14 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \t\t\t}\n \t\t\t/* feature with a value (e.g., \"agent=git/1.2.3\") */\n \t\t\telse if (*value == '=') {\n+\t\t\t\tint end;\n+\n \t\t\t\tvalue++;\n+\t\t\t\tend = strcspn(value, \" \\t\\n\");\n \t\t\t\tif (lenp)\n-\t\t\t\t\t*lenp = strcspn(value, \" \\t\\n\");\n+\t\t\t\t\t*lenp = end;\n+\t\t\t\tif (offset)\n+\t\t\t\t\t*offset = value + end - feature_list;\n \t\t\t\treturn value;\n \t\t\t}\n \t\t\t/*\n@@ -491,12 +498,17 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \n int parse_feature_request(const char *feature_list, const char *feature)\n {\n-\treturn !!parse_feature_value(feature_list, feature, NULL);\n+\treturn !!parse_feature_value(feature_list, feature, NULL, NULL);\n+}\n+\n+static const char *next_server_feature_value(const char *feature, int *len, int *offset)\n+{\n+\treturn parse_feature_value(server_capabilities_v1, feature, len, offset);\n }\n \n const char *server_feature_value(const char *feature, int *len)\n {\n-\treturn parse_feature_value(server_capabilities_v1, feature, len);\n+\treturn parse_feature_value(server_capabilities_v1, feature, len, NULL);\n }\n \n int server_supports(const char *feature)\n"},{"id":"398536","messageId":"20200525195930.309665-22-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 21/44] fetch-pack: parse and advertise the object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:07Z","receivedAt":"2020-05-25T20:00:33Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Parse the server's object-format capability and respond accordingly,\ndying if there is a mismatch.\n\nSigned-off-by: brian m. carlson \n---\n fetch-pack.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 34c339a5fe..ecfb8c3cf5 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1180,6 +1180,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t int sideband_all, int seen_ack)\n {\n \tint ret = 0;\n+\tconst char *hash_name;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n@@ -1194,6 +1195,17 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t\t\t args->server_options->items[i].string);\n \t}\n \n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n+\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n+\t\t\t the_hash_algo->name, hash_name);\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", the_hash_algo->name);\n+\t} else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n+\t\tdie(_(\"the server does not support algorithm '%s'\"),\n+\t\t the_hash_algo->name);\n+\t}\n+\n \tpacket_buf_delim(&req_buf);\n \tif (args->use_thin_pack)\n \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n"},{"id":"398537","messageId":"20200525195930.309665-20-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 19/44] builtin/clone: initialize hash algorithm properly","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:05Z","receivedAt":"2020-05-25T20:00:34Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When performing a clone, we don't know what hash algorithm the other end\nwill support. Currently, we don't support fetching data belonging to a\ndifferent algorithm, so we must know what algorithm the remote side is\nusing in order to properly initialize the repository. We can know that\nonly after fetching the refs, so if the remote side has any references,\nuse that information to reinitialize the repository with the correct\nhash algorithm information.\n\nSigned-off-by: brian m. carlson \n---\n builtin/clone.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex cb48a291ca..f27d38bc8e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1217,6 +1217,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \trefs = transport_get_remote_refs(transport, &ref_prefixes);\n \n \tif (refs) {\n+\t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n+\n+\t\t/*\n+\t\t * Now that we know what algorithm the remote side is using,\n+\t\t * let's set ours to the same thing.\n+\t\t */\n+\t\tinitialize_repository_version(hash_algo);\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n"},{"id":"398538","messageId":"20200525195930.309665-21-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 20/44] t5562: pass object-format in synthesized test data","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:06Z","receivedAt":"2020-05-25T20:00:35Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Ensure that we pass the object-format capability in the synthesized test\ndata so that this test works with algorithms other than SHA-1.\n\nIn addition, add an additional test using the old data for when we're\nusing SHA-1 so that we can be sure that we preserve backwards\ncompatibility with servers not offering the object-format capability.\n\nSigned-off-by: brian m. carlson \n---\n t/t5562-http-backend-content-length.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5562-http-backend-content-length.sh b/t/t5562-http-backend-content-length.sh\nindex 3f4ac71f83..c6ec625497 100755\n--- a/t/t5562-http-backend-content-length.sh\n+++ b/t/t5562-http-backend-content-length.sh\n@@ -46,6 +46,7 @@ ssize_b100dots() {\n }\n \n test_expect_success 'setup' '\n+\ttest_oid_init &&\n \tHTTP_CONTENT_ENCODING=\"identity\" &&\n \texport HTTP_CONTENT_ENCODING &&\n \tgit config http.receivepack true &&\n@@ -62,8 +63,8 @@ test_expect_success 'setup' '\n \ttest_copy_bytes 10 fetch_body.trunc &&\n \thash_next=$(git commit-tree -p HEAD -m next HEAD^{tree}) &&\n \t{\n-\t\tprintf \"%s %s refs/heads/newbranch\\\\0report-status\\\\n\" \\\n-\t\t\t\"$ZERO_OID\" \"$hash_next\" | packetize &&\n+\t\tprintf \"%s %s refs/heads/newbranch\\\\0report-status object-format=%s\\\\n\" \\\n+\t\t\t\"$ZERO_OID\" \"$hash_next\" \"$(test_oid algo)\" | packetize &&\n \t\tprintf 0000 &&\n \t\techo \"$hash_next\" | git pack-objects --stdout\n \t} >push_body &&\n"},{"id":"398539","messageId":"20200525195930.309665-41-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 40/44] t5703: use object-format serve option","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:26Z","receivedAt":"2020-05-25T20:00:36Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're using an algorithm other than SHA-1, we need to specify the\nalgorithm in use so we don't get a failure with an \"unknown format\"\nmessage. Add a wrapper function that specifies this header if required.\nSkip specifying this header for SHA-1 to test that it works both with an\nwithout this header.\n\nSigned-off-by: brian m. carlson \n---\n t/t5703-upload-pack-ref-in-want.sh | 19 ++++++++++++++-----\n 1 file changed, 14 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh\nindex a34460f7d8..afe7f7f919 100755\n--- a/t/t5703-upload-pack-ref-in-want.sh\n+++ b/t/t5703-upload-pack-ref-in-want.sh\n@@ -27,6 +27,15 @@ check_output () {\n \ttest_cmp sorted_commits actual_commits\n }\n \n+write_command () {\n+\techo \"command=$1\"\n+\n+\tif test \"$(test_oid algo)\" != sha1\n+\tthen\n+\t\techo \"object-format=$(test_oid algo)\"\n+\tfi\n+}\n+\n # c(o/foo) d(o/bar)\n # \\ /\n # b e(baz) f(master)\n@@ -62,7 +71,7 @@ test_expect_success 'config controls ref-in-want advertisement' '\n \n test_expect_success 'invalid want-ref line' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/non-existent\n@@ -83,7 +92,7 @@ test_expect_success 'basic want-ref' '\n \n \toid=$(git rev-parse a) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/master\n@@ -107,7 +116,7 @@ test_expect_success 'multiple want-ref lines' '\n \n \toid=$(git rev-parse b) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/o/foo\n@@ -129,7 +138,7 @@ test_expect_success 'mix want and want-ref' '\n \tgit rev-parse e f >expected_commits &&\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/master\n@@ -152,7 +161,7 @@ test_expect_success 'want-ref with ref we already have commit for' '\n \n \toid=$(git rev-parse c) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/o/foo\n"},{"id":"398540","messageId":"20200525195930.309665-42-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 41/44] t5704: send object-format capability with SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:27Z","receivedAt":"2020-05-25T20:00:37Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we speak protocol v2 in this test, we must pass the object-format\nheader if the algorithm is not SHA-1. Otherwise, git upload-pack fails\nbecause the hash algorithm doesn't match and not because we've failed to\nspeak the protocol correctly. Pass the header so that our assertions\ntest what we're really interested in.\n\nSigned-off-by: brian m. carlson \n---\n t/t5704-protocol-violations.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t5704-protocol-violations.sh b/t/t5704-protocol-violations.sh\nindex 950cfb21fe..5c941949b9 100755\n--- a/t/t5704-protocol-violations.sh\n+++ b/t/t5704-protocol-violations.sh\n@@ -9,6 +9,7 @@ making sure that we do not segfault or otherwise behave badly.'\n test_expect_success 'extra delim packet in v2 ls-refs args' '\n \t{\n \t\tpacketize command=ls-refs &&\n+\t\tpacketize \"object-format=$(test_oid algo)\" &&\n \t\tprintf 0001 &&\n \t\t# protocol expects 0000 flush here\n \t\tprintf 0001\n@@ -21,6 +22,7 @@ test_expect_success 'extra delim packet in v2 ls-refs args' '\n test_expect_success 'extra delim packet in v2 fetch args' '\n \t{\n \t\tpacketize command=fetch &&\n+\t\tpacketize \"object-format=$(test_oid algo)\" &&\n \t\tprintf 0001 &&\n \t\t# protocol expects 0000 flush here\n \t\tprintf 0001\n"},{"id":"398541","messageId":"20200525195930.309665-43-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 42/44] t5300: pass --object-format to git index-pack","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:28Z","receivedAt":"2020-05-25T20:00:39Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"git index-pack by default reads the repository to determine the object\nformat. However, when outside of a repository, it's necessary to specify\nthe hash algorithm in use so that the pack can be properly indexed. Add\nan --object-format argument when invoking git index-pack outside of a\nrepository.\n\nSigned-off-by: brian m. carlson \n---\n t/t5300-pack-object.sh | 9 +++++----\n 1 file changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 410a09b0dd..746cdb626e 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -12,7 +12,8 @@ TRASH=$(pwd)\n \n test_expect_success \\\n 'setup' \\\n- 'rm -f .git/index* &&\n+ 'test_oid_init &&\n+ rm -f .git/index* &&\n perl -e \"print \\\"a\\\" x 4096;\" > a &&\n perl -e \"print \\\"b\\\" x 4096;\" > b &&\n perl -e \"print \\\"c\\\" x 4096;\" > c &&\n@@ -412,18 +413,18 @@ test_expect_success 'set up pack for non-repo tests' '\n '\n \n test_expect_success 'index-pack --stdin complains of non-repo' '\n-\tnongit test_must_fail git index-pack --stdin works in non-repo' '\n-\tnongit git index-pack ../foo.pack &&\n+\tnongit git index-pack --object-format=$(test_oid algo) ../foo.pack &&\n \ttest_path_is_file foo.idx\n '\n \n test_expect_success 'index-pack --strict works in non-repo' '\n \trm -f foo.idx &&\n-\tnongit git index-pack --strict ../foo.pack &&\n+\tnongit git index-pack --strict --object-format=$(test_oid algo) ../foo.pack &&\n \ttest_path_is_file foo.idx\n '\n \n"},{"id":"398542","messageId":"20200525195930.309665-32-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 31/44] serve: advertise object-format capability for protocol v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:17Z","receivedAt":"2020-05-25T20:00:40Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In order to communicate the protocol supported by the server side, add\nsupport for advertising the object-format capability. We check that the\nclient side sends us an identical algorithm if it sends us its own\nobject-format capability, and assume it speaks SHA-1 if not.\n\nIn the test, when we're using an algorithm other than SHA-1, we need to\nspecify the algorithm in use so we don't get a failure with an \"unknown\nformat\" message. Add a test that we handle a mismatched algorithm.\nRemove the test_oid_init call since it's no longer necessary.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 2 ++\n serve.c | 27 +++++++++++++++++++++++++++\n t/t5701-git-serve.sh | 25 +++++++++++++++++++++++++\n 3 files changed, 54 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex 66650ff2d3..2ada5b5161 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -460,6 +460,8 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n \t\treader->hash_algo = &hash_algos[hash_algo];\n \t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n+\t} else {\n+\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n \n \tif (server_options && server_options->nr &&\ndiff --git a/serve.c b/serve.c\nindex 317256c1a4..7ab7807fef 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -22,6 +22,14 @@ static int agent_advertise(struct repository *r,\n \treturn 1;\n }\n \n+static int object_format_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, r->hash_algo->name);\n+\treturn 1;\n+}\n+\n struct protocol_capability {\n \t/*\n \t * The name of the capability. The server uses this name when\n@@ -57,6 +65,7 @@ static struct protocol_capability capabilities[] = {\n \t{ \"ls-refs\", always_advertise, ls_refs },\n \t{ \"fetch\", upload_pack_advertise, upload_pack_v2 },\n \t{ \"server-option\", always_advertise, NULL },\n+\t{ \"object-format\", object_format_advertise, NULL },\n };\n \n static void advertise_capabilities(void)\n@@ -153,6 +162,22 @@ int has_capability(const struct argv_array *keys, const char *capability,\n \treturn 0;\n }\n \n+static void check_algorithm(struct repository *r, struct argv_array *keys)\n+{\n+\tint client = GIT_HASH_SHA1, server = hash_algo_by_ptr(r->hash_algo);\n+\tconst char *algo_name;\n+\n+\tif (has_capability(keys, \"object-format\", &algo_name)) {\n+\t\tclient = hash_algo_by_name(algo_name);\n+\t\tif (client == GIT_HASH_UNKNOWN)\n+\t\t\tdie(\"unknown object format '%s'\", algo_name);\n+\t}\n+\n+\tif (client != server)\n+\t\tdie(\"mismatched object format: server %s; client %s\\n\",\n+\t\t r->hash_algo->name, hash_algos[client].name);\n+}\n+\n enum request_state {\n \tPROCESS_REQUEST_KEYS,\n \tPROCESS_REQUEST_DONE,\n@@ -223,6 +248,8 @@ static int process_request(void)\n \tif (!command)\n \t\tdie(\"no command requested\");\n \n+\tcheck_algorithm(the_repository, &keys);\n+\n \tcommand->command(the_repository, &keys, &reader);\n \n \targv_array_clear(&keys);\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex ffb9613885..a1f5fdc9fd 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -5,12 +5,17 @@ test_description='test protocol v2 server commands'\n . ./test-lib.sh\n \n test_expect_success 'test capability advertisement' '\n+\ttest_oid_cache <<-EOF &&\n+\twrong_algo sha1:sha256\n+\twrong_algo sha256:sha1\n+\tEOF\n \tcat >expect <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n \tfetch=shallow\n \tserver-option\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \n@@ -45,6 +50,7 @@ test_expect_success 'request invalid capability' '\n test_expect_success 'request with no command' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tagent=git/test\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n \tcommand=foo\n+\tobject-format=$(test_oid algo)\n \tagent=git/test\n \t0000\n \tEOF\n@@ -61,6 +68,17 @@ test_expect_success 'request invalid command' '\n \ttest_i18ngrep \"invalid command\" err\n '\n \n+test_expect_success 'wrong object-format' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=fetch\n+\tagent=git/test\n+\tobject-format=$(test_oid wrong_algo)\n+\t0000\n+\tEOF\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \n@@ -96,6 +115,7 @@ test_expect_success 'basics of ls-refs' '\n test_expect_success 'basic ref-prefixes' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tref-prefix refs/heads/master\n \tref-prefix refs/tags/one\n@@ -116,6 +136,7 @@ test_expect_success 'basic ref-prefixes' '\n test_expect_success 'refs/heads prefix' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tref-prefix refs/heads/\n \t0000\n@@ -136,6 +157,7 @@ test_expect_success 'refs/heads prefix' '\n test_expect_success 'peel parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tpeel\n \tref-prefix refs/tags/\n@@ -157,6 +179,7 @@ test_expect_success 'peel parameter' '\n test_expect_success 'symrefs parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tsymrefs\n \tref-prefix refs/heads/\n@@ -178,6 +201,7 @@ test_expect_success 'symrefs parameter' '\n test_expect_success 'sending server-options' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \tserver-option=hello\n \tserver-option=world\n \t0001\n@@ -200,6 +224,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=fetch\n+\tobject-format=$(test_oid algo)\n \t0001\n \tthis-is-not-a-command\n \t0000\n"},{"id":"398543","messageId":"20200525195930.309665-29-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 28/44] Documentation/technical: document object-format for protocol v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:14Z","receivedAt":"2020-05-25T20:00:42Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Document the object-format extension for protocol v2.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/protocol-v2.txt | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 7e3766cafb..107e421fb7 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -453,3 +453,12 @@ included in a request. This is done by sending each option as a\n a request.\n \n The provided options must not contain a NUL or LF character.\n+\n+ object-format\n+~~~~~~~~~~~~~~~\n+\n+The server can advertise the `object-format` capability with a value `X` (in the\n+form `object-format=X`) to notify the client that the server is able to deal\n+with objects using hash algorithm X. If not specified, the server is assumed to\n+only handle SHA-1. If the client would like to use a hash algorithm other than\n+SHA-1, it should specify its object-format string.\n"},{"id":"398544","messageId":"20200525195930.309665-27-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 26/44] builtin/show-index: provide options to determine hash algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:12Z","receivedAt":"2020-05-25T20:00:43Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"show-index is capable of reading any possible index file whether or not\nthe index is inside a repository. However, because our index files lack\nmetadata about the hash algorithm in use, it's not possible to\nautodetect the algorithm that a particular index file is using.\n\nIn order to allow us to read index files of any algorithm, let's set up\nthe .git directory gently so that we default to the algorithm for the\ncurrent repository, and add an --object-format option to allow users to\noverride this setting and continue to run show-index outside of a\nrepository altogether. Let's also document this new option so that\npeople can find it and use it.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-show-index.txt | 11 ++++++++++-\n builtin/show-index.c | 29 ++++++++++++++++++++++++-----\n git.c | 2 +-\n 3 files changed, 35 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt\nindex 424e4ba84c..39b1d8eaa1 100644\n--- a/Documentation/git-show-index.txt\n+++ b/Documentation/git-show-index.txt\n@@ -9,7 +9,7 @@ git-show-index - Show packed archive index\n SYNOPSIS\n --------\n [verse]\n-'git show-index'\n+'git show-index' [--object-format=]\n \n \n DESCRIPTION\n@@ -36,6 +36,15 @@ Note that you can get more information on a packfile by calling\n linkgit:git-verify-pack[1]. However, as this command considers only the\n index file itself, it's both faster and more flexible.\n \n+OPTIONS\n+-------\n+\n+--object-format=::\n+\tSpecify the given object format (hash algorithm) for the index file. The\n+\tvalid values are 'sha1' and (if enabled) 'sha256'. The default is the\n+\talgorithm for the current repository (set by `extensions.objectFormat`), or\n+\t'sha1' if no value is set or outside a repository..\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/show-index.c b/builtin/show-index.c\nindex 0826f6a5a2..8106b03a6b 100644\n--- a/builtin/show-index.c\n+++ b/builtin/show-index.c\n@@ -1,9 +1,12 @@\n #include \"builtin.h\"\n #include \"cache.h\"\n #include \"pack.h\"\n+#include \"parse-options.h\"\n \n-static const char show_index_usage[] =\n-\"git show-index\";\n+static const char *const show_index_usage[] = {\n+\t\"git show-index [--object-format=]\",\n+\tNULL\n+};\n \n int cmd_show_index(int argc, const char **argv, const char *prefix)\n {\n@@ -11,10 +14,26 @@ int cmd_show_index(int argc, const char **argv, const char *prefix)\n \tunsigned nr;\n \tunsigned int version;\n \tstatic unsigned int top_index[256];\n-\tconst unsigned hashsz = the_hash_algo->rawsz;\n+\tunsigned hashsz;\n+\tconst char *hash_name = NULL;\n+\tint hash_algo;\n+\tconst struct option show_index_options[] = {\n+\t\tOPT_STRING(0, \"object-format\", &hash_name, N_(\"hash-algorithm\"),\n+\t\t\t N_(\"specify the hash algorithm to use\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, show_index_options, show_index_usage, 0);\n+\n+\tif (hash_name) {\n+\t\thash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"Unknown hash algorithm\"));\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t}\n+\n+\thashsz = the_hash_algo->rawsz;\n \n-\tif (argc != 1)\n-\t\tusage(show_index_usage);\n \tif (fread(top_index, 2 * 4, 1, stdin) != 1)\n \t\tdie(\"unable to read header\");\n \tif (top_index[0] == htonl(PACK_IDX_SIGNATURE)) {\ndiff --git a/git.c b/git.c\nindex a2d337eed7..2f021b97f3 100644\n--- a/git.c\n+++ b/git.c\n@@ -574,7 +574,7 @@ static struct cmd_struct commands[] = {\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n-\t{ \"show-index\", cmd_show_index },\n+\t{ \"show-index\", cmd_show_index, RUN_SETUP_GENTLY },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n \t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n"},{"id":"398545","messageId":"20200525195930.309665-25-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 24/44] packfile: compute and use the index CRC offset","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:10Z","receivedAt":"2020-05-25T20:00:45Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Both v2 pack index files and the v3 format specified as part of the\nNewHash work have similar data starting at the CRC table. Much of the\nexisting code wants to read either this table or the offset entries\nfollowing it, and in doing so computes the offset each time.\n\nIn order to share as much code between v2 and v3, compute the offset of\nthe CRC table and store it when the pack is opened. Use this value to\ncompute offsets to not only the CRC table, but to the offset entries\nbeyond it.\n\nSigned-off-by: brian m. carlson \n---\n builtin/index-pack.c | 6 +-----\n object-store.h | 1 +\n packfile.c | 1 +\n 3 files changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f176dd28c8..7bea1fba52 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1555,13 +1555,9 @@ static void read_v2_anomalous_offsets(struct packed_git *p,\n {\n \tconst uint32_t *idx1, *idx2;\n \tuint32_t i;\n-\tconst uint32_t hashwords = the_hash_algo->rawsz / sizeof(uint32_t);\n \n \t/* The address of the 4-byte offset table */\n-\tidx1 = (((const uint32_t *)p->index_data)\n-\t\t+ 2 /* 8-byte header */\n-\t\t+ 256 /* fan out */\n-\t\t+ hashwords * p->num_objects /* object ID table */\n+\tidx1 = (((const uint32_t *)((const uint8_t *)p->index_data + p->crc_offset))\n \t\t+ p->num_objects /* CRC32 table */\n \t\t);\n \ndiff --git a/object-store.h b/object-store.h\nindex d1e490f203..f439d47af8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -70,6 +70,7 @@ struct packed_git {\n \tsize_t index_size;\n \tuint32_t num_objects;\n \tuint32_t num_bad_objects;\n+\tuint32_t crc_offset;\n \tunsigned char *bad_object_sha1;\n \tint index_version;\n \ttime_t mtime;\ndiff --git a/packfile.c b/packfile.c\nindex f4e752996d..6ab5233613 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -178,6 +178,7 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \t\t */\n \t\t (sizeof(off_t) <= 4))\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n+\t\tp->crc_offset = 8 + 4 * 256 + nr * hashsz;\n \t}\n \n \tp->index_version = version;\n"},{"id":"398546","messageId":"20200525195930.309665-23-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 22/44] setup: set the_repository's hash algo when checking format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:08Z","receivedAt":"2020-05-25T20:00:46Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're checking the repository's format, set the hash algorithm at\nthe same time. This ensures that we perform a suitable initialization\nearly enough to avoid confusing any parts of the code. If we defer\nuntil later, we can end up with portions of the code which are confused\nabout the hash algorithm, resulting in segfaults when working with\nSHA-256 repositories.\n\nSigned-off-by: brian m. carlson \n---\n setup.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/setup.c b/setup.c\nindex 65fe5ecefb..019a1c6367 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1273,6 +1273,7 @@ void check_repository_format(struct repository_format *fmt)\n \t\tfmt = &repo_fmt;\n \tcheck_repository_format_gently(get_git_dir(), fmt, NULL);\n \tstartup_info->have_repository = 1;\n+\trepo_set_hash_algo(the_repository, fmt->hash_algo);\n \tclear_repository_format(&repo_fmt);\n }\n \n"},{"id":"398547","messageId":"20200525195930.309665-16-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 15/44] builtin/receive-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:01Z","receivedAt":"2020-05-25T20:00:47Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n builtin/receive-pack.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 4ffa501dce..d43663bb0a 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1625,6 +1625,8 @@ static struct command *read_head_info(struct packet_reader *reader,\n \t\tlinelen = strlen(reader->line);\n \t\tif (linelen < reader->pktlen) {\n \t\t\tconst char *feature_list = reader->line + linelen + 1;\n+\t\t\tconst char *hash = NULL;\n+\t\t\tint len = 0;\n \t\t\tif (parse_feature_request(feature_list, \"report-status\"))\n \t\t\t\treport_status = 1;\n \t\t\tif (parse_feature_request(feature_list, \"side-band-64k\"))\n@@ -1637,6 +1639,13 @@ static struct command *read_head_info(struct packet_reader *reader,\n \t\t\tif (advertise_push_options\n \t\t\t && parse_feature_request(feature_list, \"push-options\"))\n \t\t\t\tuse_push_options = 1;\n+\t\t\thash = parse_feature_value(feature_list, \"object-format\", &len, NULL);\n+\t\t\tif (!hash) {\n+\t\t\t\thash = hash_algos[GIT_HASH_SHA1].name;\n+\t\t\t\tlen = strlen(hash);\n+\t\t\t}\n+\t\t\tif (xstrncmpz(the_hash_algo->name, hash, len))\n+\t\t\t\tdie(\"error: unsupported object format '%s'\", hash);\n \t\t}\n \n \t\tif (!strcmp(reader->line, \"push-cert\")) {\n"},{"id":"398548","messageId":"20200525195930.309665-19-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 18/44] remote-curl: implement object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:04Z","receivedAt":"2020-05-25T20:00:48Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Implement the object-format extensions that let us determine the hash\nalgorithm in use when pushing, pulling, and fetching.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 19 ++++++++++++++++++-\n 1 file changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 1c9aa3d0ab..3ed0dfec1b 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -41,7 +41,9 @@ struct options {\n \t\tdeepen_relative : 1,\n \t\tfrom_promisor : 1,\n \t\tno_dependents : 1,\n-\t\tatomic : 1;\n+\t\tatomic : 1,\n+\t\tobject_format : 1;\n+\tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n static struct string_list cas_options = STRING_LIST_INIT_DUP;\n@@ -190,6 +192,16 @@ static int set_option(const char *name, const char *value)\n \t} else if (!strcmp(name, \"filter\")) {\n \t\toptions.filter = xstrdup(value);\n \t\treturn 0;\n+\t} else if (!strcmp(name, \"object-format\")) {\n+\t\tint algo;\n+\t\toptions.object_format = 1;\n+\t\tif (strcmp(value, \"true\")) {\n+\t\t\talgo = hash_algo_by_name(value);\n+\t\t\tif (algo == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"unknown object format '%s'\", value);\n+\t\t\toptions.hash_algo = &hash_algos[algo];\n+\t\t}\n+\t\treturn 0;\n \t} else {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -231,6 +243,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n \t\t\t\t NULL, &heads->shallow);\n+\t\toptions.hash_algo = reader.hash_algo;\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n@@ -509,6 +522,9 @@ static struct ref *get_refs(int for_push)\n static void output_refs(struct ref *refs)\n {\n \tstruct ref *posn;\n+\tif (options.object_format && options.hash_algo) {\n+\t\tprintf(\":object-format %s\\n\", options.hash_algo->name);\n+\t}\n \tfor (posn = refs; posn; posn = posn->next) {\n \t\tif (posn->symref)\n \t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n@@ -1439,6 +1455,7 @@ int cmd_main(int argc, const char **argv)\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n+\t\t\tprintf(\"object-format\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else if (skip_prefix(buf.buf, \"stateless-connect \", &arg)) {\n"},{"id":"398549","messageId":"20200525195930.309665-18-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200525195930.309665-1-sandals@crustytoothpaste.net","subject":"[PATCH v2 17/44] transport-helper: implement object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-05-25T19:59:03Z","receivedAt":"2020-05-25T20:00:50Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Implement the object-format extensions that let us determine the hash\nalgorithm in use when pushing or pulling data.\n\nSigned-off-by: brian m. carlson \n---\n transport-helper.c | 24 ++++++++++++++++++++++--\n 1 file changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex a46afcb69d..ae33b0eea7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -32,7 +32,8 @@ struct helper_data {\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n-\t\tno_private_update : 1;\n+\t\tno_private_update : 1,\n+\t\tobject_format : 1;\n \n \t/*\n \t * As an optimization, the transport code may invoke fetch before\n@@ -207,6 +208,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->import_marks = xstrdup(arg);\n \t\t} else if (starts_with(capname, \"no-private-update\")) {\n \t\t\tdata->no_private_update = 1;\n+\t\t} else if (starts_with(capname, \"object-format\")) {\n+\t\t\tdata->object_format = 1;\n \t\t} else if (mandatory) {\n \t\t\tdie(_(\"unknown mandatory capability %s; this remote \"\n \t\t\t \"helper probably needs newer version of Git\"),\n@@ -1103,6 +1106,12 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \tdata->get_refs_list_called = 1;\n \thelper = get_helper(transport);\n \n+\tif (data->object_format) {\n+\t\twrite_str_in_full(helper->in, \"option object-format\\n\");\n+\t\tif (recvline(data, &buf) || strcmp(buf.buf, \"ok\"))\n+\t\t\texit(128);\n+\t}\n+\n \tif (data->push && for_push)\n \t\twrite_str_in_full(helper->in, \"list for-push\\n\");\n \telse\n@@ -1115,6 +1124,17 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \n \t\tif (!*buf.buf)\n \t\t\tbreak;\n+\t\telse if (buf.buf[0] == ':') {\n+\t\t\tconst char *value;\n+\t\t\tif (skip_prefix(buf.buf, \":object-format \", &value)) {\n+\t\t\t\tint algo = hash_algo_by_name(value);\n+\t\t\t\tif (algo == GIT_HASH_UNKNOWN)\n+\t\t\t\t\tdie(_(\"unsupported object format '%s'\"),\n+\t\t\t\t\t value);\n+\t\t\t\ttransport->hash_algo = &hash_algos[algo];\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n \n \t\teov = strchr(buf.buf, ' ');\n \t\tif (!eov)\n@@ -1127,7 +1147,7 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \t\tif (buf.buf[0] == '@')\n \t\t\t(*tail)->symref = xstrdup(buf.buf + 1);\n \t\telse if (buf.buf[0] != '?')\n-\t\t\tget_oid_hex(buf.buf, &(*tail)->old_oid);\n+\t\t\tget_oid_hex_algop(buf.buf, &(*tail)->old_oid, transport->hash_algo);\n \t\tif (eon) {\n \t\t\tif (has_attribute(eon + 1, \"unchanged\")) {\n \t\t\t\t(*tail)->status |= REF_STATUS_UPTODATE;\n"},{"id":"400157","messageId":"20200619175601.569856-3-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 02/44] Documentation: document v1 protocol object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:19Z","receivedAt":"2020-06-19T17:56:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Document a capability that indicates which hash algorithms are in use by\nboth sides of a remote connection. Use the term \"object-format\", since\nthis is the term used for the repository extension as well.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/protocol-capabilities.txt | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 2b267c0da6..36ccd14f97 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -176,6 +176,21 @@ agent strings are purely informative for statistics and debugging\n purposes, and MUST NOT be used to programmatically assume the presence\n or absence of particular features.\n \n+object-format\n+-------------\n+\n+This capability, which takes a hash algorithm as an argument, indicates\n+that the server supports the given hash algorithms. It may be sent\n+multiple times; if so, the first one given is the one used in the ref\n+advertisement.\n+\n+When provided by the client, this indicates that it intends to use the\n+given hash algorithm to communicate. The algorithm provided must be one\n+that the server supports.\n+\n+If this capability is not provided, it is assumed that the only\n+supported algorithm is SHA-1.\n+\n symref\n ------\n \n"},{"id":"400158","messageId":"20200619175601.569856-1-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200513005424.81369-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 00/44] SHA-256 part 2/3: protocol functionality","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:17Z","receivedAt":"2020-06-19T17:56:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This is part 2 of 3 of the SHA-256 work. This series adds all of the\nprotocol logic to work with SHA-256 repositories.\n\nv3 fixes a bug in patch 34 which prevented cloning an empty repository\nwith the dumb HTTP protocol. We look up the hash algorithm by length of\nthe data in the info/refs file and if we have no refs, we have no\nentries.\n\nPreviously, we just failed and complained, which isn't really helpful,\nnor is it backward compatible. So now we use whatever the default is\nfor the current repository. That means we honor GIT_DEFAULT_HASH or git\nclone -c, and default to SHA-1 otherwise. Users are encouraged to\nswitch to the smart protocol if they need to distinguish the remote\nside's hash algorithm when the repository is empty.\n\nThere are tests for the default hash behavior, but not for git clone -c,\nbecause the extensions.objectformat option doesn't exist yet. I have\ntested that it does indeed work, though.\n\nOtherwise, this series is the same as v2 except for a rebase (for my\nconvenience and Junio's).\n\nChanges from v2:\n* Rebase onto master.\n* Fix cloning an empty repository with the dumb HTTP protocol.\n\nChanges from v1:\n* Fix spurious line additions and deletions.\n* Rename len to linelen for easier understanding.\n* Move the documentation comment for xstrncmpz to the header.\n* Drop a useless variable (found).\n* Update several commit messages to better explain things as suggested\n by Junio and Martin.\n* Name the parameters for parse_feature_value for better documentation.\n* Reduce the scope of variables when possible.\n* Add explicit handling for missing object-format capabilities.\n* Rename all new options to --object-format.\n* Use oidcpy where possible.\n* Test more failure cases.\n* Have index-pack fail if --stdin and --object-format are both\n specified.\n* Move and simplify t5704.\n* Other miscellaneous cleanups to respond to review feedback.\n\nRange-diff below.\n\nbrian m. carlson (44):\n t1050: match object ID paths in a hash-insensitive way\n Documentation: document v1 protocol object-format capability\n connect: have ref processing code take struct packet_reader\n wrapper: add function to compare strings with different NUL\n termination\n remote: advertise the object-format capability on the server side\n connect: add function to parse multiple v1 capability values\n connect: add function to fetch value of a v2 server capability\n pkt-line: add a member for hash algorithm\n transport: add a hash algorithm member\n connect: add function to detect supported v1 hash functions\n send-pack: detect when the server doesn't support our hash\n connect: make parse_feature_value extern\n fetch-pack: detect when the server doesn't support our hash\n connect: detect algorithm when fetching refs\n builtin/receive-pack: detect when the server doesn't support our hash\n docs: update remote helper docs for object-format extensions\n transport-helper: implement object-format extensions\n remote-curl: implement object-format extensions\n builtin/clone: initialize hash algorithm properly\n t5562: pass object-format in synthesized test data\n fetch-pack: parse and advertise the object-format capability\n setup: set the_repository's hash algo when checking format\n t3200: mark assertion with SHA1 prerequisite\n packfile: compute and use the index CRC offset\n t5302: modernize test formatting\n builtin/show-index: provide options to determine hash algo\n t1302: expect repo format version 1 for SHA-256\n Documentation/technical: document object-format for protocol v2\n connect: pass full packet reader when parsing v2 refs\n connect: parse v2 refs with correct hash algorithm\n serve: advertise object-format capability for protocol v2\n t5500: make hash independent\n builtin/ls-remote: initialize repository based on fetch\n remote-curl: detect algorithm for dumb HTTP by size\n builtin/index-pack: add option to specify hash algorithm\n t1050: pass algorithm to index-pack when outside repo\n remote-curl: avoid truncating refs with ls-remote\n t/helper: initialize the repository for test-sha1-array\n t5702: offer an object-format capability in the test\n t5703: use object-format serve option\n t5704: send object-format capability with SHA-256\n t5300: pass --object-format to git index-pack\n bundle: detect hash algorithm when reading refs\n remote-testgit: adapt for object-format\n\n Documentation/git-index-pack.txt | 8 +\n Documentation/git-show-index.txt | 11 +-\n Documentation/gitremote-helpers.txt | 33 +-\n .../technical/protocol-capabilities.txt | 15 +\n Documentation/technical/protocol-v2.txt | 9 +\n builtin/clone.c | 9 +\n builtin/index-pack.c | 14 +-\n builtin/ls-remote.c | 4 +\n builtin/receive-pack.c | 10 +\n builtin/show-index.c | 29 +-\n bundle.c | 22 +-\n bundle.h | 1 +\n connect.c | 138 +++++--\n connect.h | 3 +\n fetch-pack.c | 14 +\n git-compat-util.h | 6 +\n git.c | 2 +-\n object-store.h | 1 +\n packfile.c | 1 +\n pkt-line.c | 1 +\n pkt-line.h | 3 +\n remote-curl.c | 46 ++-\n send-pack.c | 6 +\n serve.c | 27 ++\n setup.c | 1 +\n t/helper/test-oid-array.c | 3 +\n t/t1050-large.sh | 6 +-\n t/t1302-repo-version.sh | 6 +-\n t/t3200-branch.sh | 2 +-\n t/t5300-pack-object.sh | 9 +-\n t/t5302-pack-index.sh | 360 +++++++++---------\n t/t5500-fetch-pack.sh | 5 +-\n t/t5550-http-fetch-dumb.sh | 18 +\n t/t5562-http-backend-content-length.sh | 5 +-\n t/t5701-git-serve.sh | 25 ++\n t/t5702-protocol-v2.sh | 2 +\n t/t5703-upload-pack-ref-in-want.sh | 19 +-\n t/t5704-protocol-violations.sh | 2 +\n t/t5801/git-remote-testgit | 6 +\n t/test-lib.sh | 1 +\n transport-helper.c | 24 +-\n transport.c | 18 +-\n transport.h | 8 +\n upload-pack.c | 3 +-\n wrapper.c | 8 +\n 45 files changed, 696 insertions(+), 248 deletions(-)\n\nRange-diff against v2:\n 1: 5878fe6a98 = 1: 3504602e31 t1050: match object ID paths in a hash-insensitive way\n 2: 402864eaa3 = 2: 150ccddb98 Documentation: document v1 protocol object-format capability\n 3: d124692e2f = 3: b86ec9fffe connect: have ref processing code take struct packet_reader\n 4: cce29662b4 = 4: f048e638e5 wrapper: add function to compare strings with different NUL termination\n 5: 3b207e304b ! 5: 99261e8221 remote: advertise the object-format capability on the server side\n @@ upload-pack.c\n @@ upload-pack.c: static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n - \t\tformat_symref_info(&symref_info, cb_data);\n + \t\tformat_symref_info(&symref_info, &data->symref);\n -\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s agent=%s\\n\",\n +\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s object-format=%s agent=%s\\n\",\n \t\t\t oid_to_hex(oid), refname_nons,\n \t\t\t 0, capabilities,\n \t\t\t (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n @@ upload-pack.c: static int send_ref(const char *refname, const struct object_id *oid,\n - \t\t\t stateless_rpc ? \" no-done\" : \"\",\n + \t\t\t data->stateless_rpc ? \" no-done\" : \"\",\n \t\t\t symref_info.buf,\n \t\t\t allow_filter ? \" filter\" : \"\",\n +\t\t\t the_hash_algo->name,\n 6: 235d7f5b8f = 6: 5504199a26 connect: add function to parse multiple v1 capability values\n 7: 0324e126b1 = 7: 59d1b463bf connect: add function to fetch value of a v2 server capability\n 8: cdba3122ce = 8: 1b2789cab7 pkt-line: add a member for hash algorithm\n 9: c8233c3b42 = 9: 971d05e2c7 transport: add a hash algorithm member\n10: b9273c4021 = 10: 7b90abd41a connect: add function to detect supported v1 hash functions\n11: e2d37b75c8 = 11: 578676762d send-pack: detect when the server doesn't support our hash\n12: 602734cbbb = 12: 131e98603a connect: make parse_feature_value extern\n13: d97fa2c8aa = 13: a786478005 fetch-pack: detect when the server doesn't support our hash\n14: ba052f1da7 = 14: 3436a6db7b connect: detect algorithm when fetching refs\n15: 661d94d4de = 15: c8d0760e3f builtin/receive-pack: detect when the server doesn't support our hash\n16: fd8b85390c = 16: 944bf6ab9a docs: update remote helper docs for object-format extensions\n17: 32285e611f = 17: 9f072d34dc transport-helper: implement object-format extensions\n18: a33d1ed9a0 = 18: 2bdc53a8d9 remote-curl: implement object-format extensions\n19: fffdf0780d = 19: 87c6cd32f7 builtin/clone: initialize hash algorithm properly\n20: f616f85b4b = 20: e2cc4d34fe t5562: pass object-format in synthesized test data\n21: eca43da42e = 21: 34b712a983 fetch-pack: parse and advertise the object-format capability\n22: 22c1a62e10 = 22: 3e43a7d314 setup: set the_repository's hash algo when checking format\n23: 7c7f2263d5 = 23: fcf0ef64f0 t3200: mark assertion with SHA1 prerequisite\n24: ee8a71a926 = 24: 192578b5dd packfile: compute and use the index CRC offset\n25: 6afecf0b09 = 25: a560ed3194 t5302: modernize test formatting\n26: 99a847ba4e = 26: 9a079f99a0 builtin/show-index: provide options to determine hash algo\n27: 9f7c7bafaf = 27: 56d97c9c94 t1302: expect repo format version 1 for SHA-256\n28: d0ea597d63 = 28: 8c25898ded Documentation/technical: document object-format for protocol v2\n29: 51848df542 = 29: 39af318fb0 connect: pass full packet reader when parsing v2 refs\n30: b57361f3b8 ! 30: 5b334cb9b9 connect: parse v2 refs with correct hash algorithm\n @@ connect.c: static int process_ref_v2(struct packet_reader *reader, struct ref **\n \t\t\t*list = &peeled->next;\n \n @@ connect.c: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n - \t\t\t const struct string_list *server_options)\n + \t\t\t int stateless_rpc)\n {\n \tint i;\n +\tconst char *hash_name;\n31: a0c0f0f7a3 = 31: 955ea0b4cb serve: advertise object-format capability for protocol v2\n32: 1694f3f838 = 32: 5172e56115 t5500: make hash independent\n33: 902b394667 = 33: 79b576238f builtin/ls-remote: initialize repository based on fetch\n34: cc12b9b51f ! 34: 0ecf5c1a1f remote-curl: detect algorithm for dumb HTTP by size\n @@ Commit message\n provide one. Detect the hash algorithm in use by the size of the first\n object ID.\n \n + If we have an empty repository, we don't know what the hash algorithm is\n + on the remote side, so default to whatever the local side has\n + configured. Without doing this, we cannot clone an empty repository\n + since we don't know its hash algorithm. Test this case appropriately,\n + since we currently have no tests for cloning an empty repository with\n + the dumb HTTP protocol.\n +\n We anonymize the URL like elsewhere in the function in case the user has\n decided to include a secret in the URL.\n \n @@ remote-curl.c: static struct ref *parse_git_refs(struct discovery *heads, int fo\n +\tconst char *p = memchr(heads->buf, '\\t', heads->len);\n +\tint algo;\n +\tif (!p)\n -+\t\treturn NULL;\n ++\t\treturn the_hash_algo;\n +\n +\talgo = hash_algo_by_length((p - heads->buf) / 2);\n +\tif (algo == GIT_HASH_UNKNOWN)\n @@ remote-curl.c: static struct ref *parse_info_refs(struct discovery *heads)\n \t\t\tif (!refs)\n \t\t\t\trefs = ref;\n \t\t\tif (last_ref)\n +\n + ## t/t5550-http-fetch-dumb.sh ##\n +@@ t/t5550-http-fetch-dumb.sh: test_expect_success 'create password-protected repository' '\n + \t \"$HTTPD_DOCUMENT_ROOT_PATH/auth/dumb/repo.git\"\n + '\n + \n ++test_expect_success 'create empty remote repository' '\n ++\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/empty.git\" &&\n ++\t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/empty.git\" &&\n ++\t mkdir -p hooks &&\n ++\t write_script \"hooks/post-update\" <<-\\EOF &&\n ++\t exec git update-server-info\n ++\tEOF\n ++\t hooks/post-update\n ++\t)\n ++'\n ++\n ++test_expect_success 'empty dumb HTTP repository has default hash algorithm' '\n ++\ttest_when_finished \"rm -fr clone-empty\" &&\n ++\tgit clone $HTTPD_URL/dumb/empty.git clone-empty &&\n ++\tgit -C clone-empty rev-parse --show-object-format >empty-format &&\n ++\ttest \"$(cat empty-format)\" = \"$(test_oid algo)\"\n ++'\n ++\n + setup_askpass_helper\n + \n + test_expect_success 'cloning password-protected repository can fail' '\n35: b5425c9f54 = 35: 95bc1ef6fb builtin/index-pack: add option to specify hash algorithm\n36: 5c70c24d7a = 36: 89c389100e t1050: pass algorithm to index-pack when outside repo\n37: 460d6008e8 = 37: bd93aabe6b remote-curl: avoid truncating refs with ls-remote\n38: 60a98d9b53 = 38: a8e31600cc t/helper: initialize the repository for test-sha1-array\n39: b66c3ead37 = 39: d217758a83 t5702: offer an object-format capability in the test\n40: af43274a1f = 40: 6d1964142c t5703: use object-format serve option\n41: f5085b1f3f = 41: 84cad9b3ba t5704: send object-format capability with SHA-256\n42: a1b01babda = 42: 1e96dbf3dc t5300: pass --object-format to git index-pack\n43: dbb5f7195e = 43: 14cb067334 bundle: detect hash algorithm when reading refs\n44: 6c823bbe68 = 44: 816d08eb2e remote-testgit: adapt for object-format\n"},{"id":"400159","messageId":"20200619175601.569856-4-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 03/44] connect: have ref processing code take struct packet_reader","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:20Z","receivedAt":"2020-06-19T17:56:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In a future patch, we'll want to access multiple members from struct\npacket_reader when parsing references. Therefore, have the ref parsing\ncode take pointers to struct reader instead of having to pass multiple\narguments to each function.\n\nRename the len variable to \"linelen\" to make it clearer what the\nvariable does in light of the variable change.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 27 ++++++++++++++++-----------\n 1 file changed, 16 insertions(+), 11 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 0df45a1108..e66e779ebd 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -205,17 +205,19 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-static void process_capabilities(const char *line, int *len)\n+static void process_capabilities(struct packet_reader *reader, int *linelen)\n {\n+\tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n-\tif (nul_location == *len)\n+\tif (nul_location == *linelen)\n \t\treturn;\n \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n-\t*len = nul_location;\n+\t*linelen = nul_location;\n }\n \n-static int process_dummy_ref(const char *line)\n+static int process_dummy_ref(const struct packet_reader *reader)\n {\n+\tconst char *line = reader->line;\n \tstruct object_id oid;\n \tconst char *name;\n \n@@ -235,9 +237,11 @@ static void check_no_capabilities(const char *line, int len)\n \t\t\tline + strlen(line));\n }\n \n-static int process_ref(const char *line, int len, struct ref ***list,\n-\t\t unsigned int flags, struct oid_array *extra_have)\n+static int process_ref(const struct packet_reader *reader, int len,\n+\t\t struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n {\n+\tconst char *line = reader->line;\n \tstruct object_id old_oid;\n \tconst char *name;\n \n@@ -261,9 +265,10 @@ static int process_ref(const char *line, int len, struct ref ***list,\n \treturn 1;\n }\n \n-static int process_shallow(const char *line, int len,\n+static int process_shallow(const struct packet_reader *reader, int len,\n \t\t\t struct oid_array *shallow_points)\n {\n+\tconst char *line = reader->line;\n \tconst char *arg;\n \tstruct object_id old_oid;\n \n@@ -317,20 +322,20 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \n \t\tswitch (state) {\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(reader->line, &len);\n-\t\t\tif (process_dummy_ref(reader->line)) {\n+\t\t\tprocess_capabilities(reader, &len);\n+\t\t\tif (process_dummy_ref(reader)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(reader->line, len, shallow_points))\n+\t\t\tif (process_shallow(reader, len, shallow_points))\n \t\t\t\tbreak;\n \t\t\tdie(_(\"protocol error: unexpected '%s'\"), reader->line);\n \t\tcase EXPECTING_DONE:\n"},{"id":"400160","messageId":"20200619175601.569856-11-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 10/44] connect: add function to detect supported v1 hash functions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:27Z","receivedAt":"2020-06-19T17:56:26Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a function, server_supports_hash, to see if the remote server\nsupports a particular hash algorithm when speaking protocol v1.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 22 ++++++++++++++++++++++\n connect.h | 1 +\n 2 files changed, 23 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex 2cded78b0a..a52b038865 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -527,6 +527,28 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \treturn NULL;\n }\n \n+int server_supports_hash(const char *desired, int *feature_supported)\n+{\n+\tint offset = 0;\n+\tint len;\n+\tconst char *hash;\n+\n+\thash = next_server_feature_value(\"object-format\", &len, &offset);\n+\tif (feature_supported)\n+\t\t*feature_supported = !!hash;\n+\tif (!hash) {\n+\t\thash = hash_algos[GIT_HASH_SHA1].name;\n+\t\tlen = strlen(hash);\n+\t}\n+\twhile (hash) {\n+\t\tif (!xstrncmpz(desired, hash, len))\n+\t\t\treturn 1;\n+\n+\t\thash = next_server_feature_value(\"object-format\", &len, &offset);\n+\t}\n+\treturn 0;\n+}\n+\n int parse_feature_request(const char *feature_list, const char *feature)\n {\n \treturn !!parse_feature_value(feature_list, feature, NULL, NULL);\ndiff --git a/connect.h b/connect.h\nindex 88702fdd17..c53976f7ec 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -18,6 +18,7 @@ int url_is_local_not_ssh(const char *url);\n struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n+int server_supports_hash(const char *desired, int *feature_supported);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n"},{"id":"400161","messageId":"20200619175601.569856-18-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 17/44] transport-helper: implement object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:34Z","receivedAt":"2020-06-19T17:56:36Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Implement the object-format extensions that let us determine the hash\nalgorithm in use when pushing or pulling data.\n\nSigned-off-by: brian m. carlson \n---\n transport-helper.c | 24 ++++++++++++++++++++++--\n 1 file changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex a46afcb69d..ae33b0eea7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -32,7 +32,8 @@ struct helper_data {\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n-\t\tno_private_update : 1;\n+\t\tno_private_update : 1,\n+\t\tobject_format : 1;\n \n \t/*\n \t * As an optimization, the transport code may invoke fetch before\n@@ -207,6 +208,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->import_marks = xstrdup(arg);\n \t\t} else if (starts_with(capname, \"no-private-update\")) {\n \t\t\tdata->no_private_update = 1;\n+\t\t} else if (starts_with(capname, \"object-format\")) {\n+\t\t\tdata->object_format = 1;\n \t\t} else if (mandatory) {\n \t\t\tdie(_(\"unknown mandatory capability %s; this remote \"\n \t\t\t \"helper probably needs newer version of Git\"),\n@@ -1103,6 +1106,12 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \tdata->get_refs_list_called = 1;\n \thelper = get_helper(transport);\n \n+\tif (data->object_format) {\n+\t\twrite_str_in_full(helper->in, \"option object-format\\n\");\n+\t\tif (recvline(data, &buf) || strcmp(buf.buf, \"ok\"))\n+\t\t\texit(128);\n+\t}\n+\n \tif (data->push && for_push)\n \t\twrite_str_in_full(helper->in, \"list for-push\\n\");\n \telse\n@@ -1115,6 +1124,17 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \n \t\tif (!*buf.buf)\n \t\t\tbreak;\n+\t\telse if (buf.buf[0] == ':') {\n+\t\t\tconst char *value;\n+\t\t\tif (skip_prefix(buf.buf, \":object-format \", &value)) {\n+\t\t\t\tint algo = hash_algo_by_name(value);\n+\t\t\t\tif (algo == GIT_HASH_UNKNOWN)\n+\t\t\t\t\tdie(_(\"unsupported object format '%s'\"),\n+\t\t\t\t\t value);\n+\t\t\t\ttransport->hash_algo = &hash_algos[algo];\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n \n \t\teov = strchr(buf.buf, ' ');\n \t\tif (!eov)\n@@ -1127,7 +1147,7 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n \t\tif (buf.buf[0] == '@')\n \t\t\t(*tail)->symref = xstrdup(buf.buf + 1);\n \t\telse if (buf.buf[0] != '?')\n-\t\t\tget_oid_hex(buf.buf, &(*tail)->old_oid);\n+\t\t\tget_oid_hex_algop(buf.buf, &(*tail)->old_oid, transport->hash_algo);\n \t\tif (eon) {\n \t\t\tif (has_attribute(eon + 1, \"unchanged\")) {\n \t\t\t\t(*tail)->status |= REF_STATUS_UPTODATE;\n"},{"id":"400162","messageId":"20200619175601.569856-23-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 22/44] setup: set the_repository's hash algo when checking format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:39Z","receivedAt":"2020-06-19T17:56:40Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're checking the repository's format, set the hash algorithm at\nthe same time. This ensures that we perform a suitable initialization\nearly enough to avoid confusing any parts of the code. If we defer\nuntil later, we can end up with portions of the code which are confused\nabout the hash algorithm, resulting in segfaults when working with\nSHA-256 repositories.\n\nSigned-off-by: brian m. carlson \n---\n setup.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/setup.c b/setup.c\nindex 65fe5ecefb..019a1c6367 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1273,6 +1273,7 @@ void check_repository_format(struct repository_format *fmt)\n \t\tfmt = &repo_fmt;\n \tcheck_repository_format_gently(get_git_dir(), fmt, NULL);\n \tstartup_info->have_repository = 1;\n+\trepo_set_hash_algo(the_repository, fmt->hash_algo);\n \tclear_repository_format(&repo_fmt);\n }\n \n"},{"id":"400163","messageId":"20200619175601.569856-22-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 21/44] fetch-pack: parse and advertise the object-format capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:38Z","receivedAt":"2020-06-19T17:56:41Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Parse the server's object-format capability and respond accordingly,\ndying if there is a mismatch.\n\nSigned-off-by: brian m. carlson \n---\n fetch-pack.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex c090030680..7e58f295f5 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1180,6 +1180,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t int sideband_all, int seen_ack)\n {\n \tint ret = 0;\n+\tconst char *hash_name;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n@@ -1194,6 +1195,17 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t\t\t args->server_options->items[i].string);\n \t}\n \n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n+\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n+\t\t\t the_hash_algo->name, hash_name);\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", the_hash_algo->name);\n+\t} else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n+\t\tdie(_(\"the server does not support algorithm '%s'\"),\n+\t\t the_hash_algo->name);\n+\t}\n+\n \tpacket_buf_delim(&req_buf);\n \tif (args->use_thin_pack)\n \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n"},{"id":"400164","messageId":"20200619175601.569856-27-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 26/44] builtin/show-index: provide options to determine hash algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:43Z","receivedAt":"2020-06-19T17:56:44Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"show-index is capable of reading any possible index file whether or not\nthe index is inside a repository. However, because our index files lack\nmetadata about the hash algorithm in use, it's not possible to\nautodetect the algorithm that a particular index file is using.\n\nIn order to allow us to read index files of any algorithm, let's set up\nthe .git directory gently so that we default to the algorithm for the\ncurrent repository, and add an --object-format option to allow users to\noverride this setting and continue to run show-index outside of a\nrepository altogether. Let's also document this new option so that\npeople can find it and use it.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-show-index.txt | 11 ++++++++++-\n builtin/show-index.c | 29 ++++++++++++++++++++++++-----\n git.c | 2 +-\n 3 files changed, 35 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt\nindex 424e4ba84c..39b1d8eaa1 100644\n--- a/Documentation/git-show-index.txt\n+++ b/Documentation/git-show-index.txt\n@@ -9,7 +9,7 @@ git-show-index - Show packed archive index\n SYNOPSIS\n --------\n [verse]\n-'git show-index'\n+'git show-index' [--object-format=]\n \n \n DESCRIPTION\n@@ -36,6 +36,15 @@ Note that you can get more information on a packfile by calling\n linkgit:git-verify-pack[1]. However, as this command considers only the\n index file itself, it's both faster and more flexible.\n \n+OPTIONS\n+-------\n+\n+--object-format=::\n+\tSpecify the given object format (hash algorithm) for the index file. The\n+\tvalid values are 'sha1' and (if enabled) 'sha256'. The default is the\n+\talgorithm for the current repository (set by `extensions.objectFormat`), or\n+\t'sha1' if no value is set or outside a repository..\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/show-index.c b/builtin/show-index.c\nindex 0826f6a5a2..8106b03a6b 100644\n--- a/builtin/show-index.c\n+++ b/builtin/show-index.c\n@@ -1,9 +1,12 @@\n #include \"builtin.h\"\n #include \"cache.h\"\n #include \"pack.h\"\n+#include \"parse-options.h\"\n \n-static const char show_index_usage[] =\n-\"git show-index\";\n+static const char *const show_index_usage[] = {\n+\t\"git show-index [--object-format=]\",\n+\tNULL\n+};\n \n int cmd_show_index(int argc, const char **argv, const char *prefix)\n {\n@@ -11,10 +14,26 @@ int cmd_show_index(int argc, const char **argv, const char *prefix)\n \tunsigned nr;\n \tunsigned int version;\n \tstatic unsigned int top_index[256];\n-\tconst unsigned hashsz = the_hash_algo->rawsz;\n+\tunsigned hashsz;\n+\tconst char *hash_name = NULL;\n+\tint hash_algo;\n+\tconst struct option show_index_options[] = {\n+\t\tOPT_STRING(0, \"object-format\", &hash_name, N_(\"hash-algorithm\"),\n+\t\t\t N_(\"specify the hash algorithm to use\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, show_index_options, show_index_usage, 0);\n+\n+\tif (hash_name) {\n+\t\thash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"Unknown hash algorithm\"));\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t}\n+\n+\thashsz = the_hash_algo->rawsz;\n \n-\tif (argc != 1)\n-\t\tusage(show_index_usage);\n \tif (fread(top_index, 2 * 4, 1, stdin) != 1)\n \t\tdie(\"unable to read header\");\n \tif (top_index[0] == htonl(PACK_IDX_SIGNATURE)) {\ndiff --git a/git.c b/git.c\nindex a2d337eed7..2f021b97f3 100644\n--- a/git.c\n+++ b/git.c\n@@ -574,7 +574,7 @@ static struct cmd_struct commands[] = {\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n-\t{ \"show-index\", cmd_show_index },\n+\t{ \"show-index\", cmd_show_index, RUN_SETUP_GENTLY },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n \t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n"},{"id":"400165","messageId":"20200619175601.569856-26-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 25/44] t5302: modernize test formatting","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:42Z","receivedAt":"2020-06-19T17:56:45Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Our style these days is to place the description and the opening quote\nof the body on the same line as test_expect_success (if it fits), to\nplace the trailing quote on a line by itself after the body, and to use\ntabs. Since we're going to be making several significant changes to\nthis test, modernize the style to aid in readability of the subsequent\npatches.\n\nThis patch should have no functional change.\n\nSigned-off-by: brian m. carlson \n---\n t/t5302-pack-index.sh | 360 +++++++++++++++++++++---------------------\n 1 file changed, 184 insertions(+), 176 deletions(-)\n\ndiff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\nindex ad07f2f7fc..8981c9b90e 100755\n--- a/t/t5302-pack-index.sh\n+++ b/t/t5302-pack-index.sh\n@@ -7,65 +7,65 @@ test_description='pack index with 64-bit offsets and object CRC'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n- test_oid_init &&\n- rawsz=$(test_oid rawsz) &&\n- rm -rf .git &&\n- git init &&\n- git config pack.threads 1 &&\n- i=1 &&\n- while test $i -le 100\n- do\n- iii=$(printf '%03i' $i)\n-\t test-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n-\t test-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n-\t test-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n-\t test-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n-\t test-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n- echo $iii >file_$iii &&\n-\t test-tool genrandom \"$iii\" 8192 >>file_$iii &&\n- git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n- i=$(expr $i + 1) || return 1\n- done &&\n- { echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n- git update-index --add file_101 &&\n- tree=$(git write-tree) &&\n- commit=$(git commit-tree $tree obj-list &&\n- git update-ref HEAD $commit\n+\ttest_oid_init &&\n+\trawsz=$(test_oid rawsz) &&\n+\trm -rf .git &&\n+\tgit init &&\n+\tgit config pack.threads 1 &&\n+\ti=1 &&\n+\twhile test $i -le 100\n+\tdo\n+\t\tiii=$(printf '%03i' $i)\n+\t\ttest-tool genrandom \"bar\" 200 > wide_delta_$iii &&\n+\t\ttest-tool genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$i 100 > deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 1) 100 >> deep_delta_$iii &&\n+\t\ttest-tool genrandom \"foo\"$(expr $i + 2) 100 >> deep_delta_$iii &&\n+\t\techo $iii >file_$iii &&\n+\t\ttest-tool genrandom \"$iii\" 8192 >>file_$iii &&\n+\t\tgit update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n+\t\ti=$(expr $i + 1) || return 1\n+\tdone &&\n+\t{ echo 101 && test-tool genrandom 100 8192; } >file_101 &&\n+\tgit update-index --add file_101 &&\n+\ttree=$(git write-tree) &&\n+\tcommit=$(git commit-tree $tree obj-list &&\n+\tgit update-ref HEAD $commit\n '\n \n-test_expect_success \\\n- 'pack-objects with index version 1' \\\n- 'pack1=$(git pack-objects --index-version=1 test-1 &1) ||\n \t! (echo \"$msg\" | grep \"pack too large .* off_t\")\n@@ -91,21 +91,21 @@ else\n \tsay \"# skipping tests concerning 64-bit offsets\"\n fi\n \n-test_expect_success OFF64_T \\\n- 'index v2: verify a pack with some 64-bit offsets' \\\n- 'git verify-pack -v \"test-3-${pack3}.pack\"'\n+test_expect_success OFF64_T 'index v2: verify a pack with some 64-bit offsets' '\n+\tgit verify-pack -v \"test-3-${pack3}.pack\"\n+'\n \n-test_expect_success OFF64_T \\\n- '64-bit offsets: should be different from previous index v2 results' \\\n- '! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"'\n+test_expect_success OFF64_T '64-bit offsets: should be different from previous index v2 results' '\n+\t! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"\n+'\n \n-test_expect_success OFF64_T \\\n- 'index v2: force some 64-bit offsets with index-pack' \\\n- 'git index-pack --index-version=2,0x40000 -o 3.idx \"test-1-${pack1}.pack\"'\n+test_expect_success OFF64_T 'index v2: force some 64-bit offsets with index-pack' '\n+\tgit index-pack --index-version=2,0x40000 -o 3.idx \"test-1-${pack1}.pack\"\n+'\n \n-test_expect_success OFF64_T \\\n- '64-bit offsets: index-pack result should match pack-objects one' \\\n- 'cmp \"test-3-${pack3}.idx\" \"3.idx\"'\n+test_expect_success OFF64_T '64-bit offsets: index-pack result should match pack-objects one' '\n+\tcmp \"test-3-${pack3}.idx\" \"3.idx\"\n+'\n \n test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2 (cheat)' '\n \t# This cheats by knowing which lower offset should still be encoded\n@@ -120,135 +120,143 @@ test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '\n # returns the object number for given object in given pack index\n index_obj_nr()\n {\n- idx_file=$1\n- object_sha1=$2\n- nr=0\n- git show-index < $idx_file |\n- while read offs sha1 extra\n- do\n- nr=$(($nr + 1))\n- test \"$sha1\" = \"$object_sha1\" || continue\n- echo \"$(($nr - 1))\"\n- break\n- done\n+\tidx_file=$1\n+\tobject_sha1=$2\n+\tnr=0\n+\tgit show-index < $idx_file |\n+\twhile read offs sha1 extra\n+\tdo\n+\t nr=$(($nr + 1))\n+\t test \"$sha1\" = \"$object_sha1\" || continue\n+\t echo \"$(($nr - 1))\"\n+\t break\n+\tdone\n }\n \n # returns the pack offset for given object as found in given pack index\n index_obj_offset()\n {\n- idx_file=$1\n- object_sha1=$2\n- git show-index < $idx_file | grep $object_sha1 |\n- ( read offs extra && echo \"$offs\" )\n+\tidx_file=$1\n+\tobject_sha1=$2\n+\tgit show-index < $idx_file | grep $object_sha1 |\n+\t( read offs extra && echo \"$offs\" )\n }\n \n-test_expect_success \\\n- '[index v1] 1) stream pack to repository' \\\n- 'git index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n- git prune-packed &&\n- git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n- cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n- cmp \"test-1-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n+test_expect_success '[index v1] 1) stream pack to repository' '\n+\tgit index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n+\tgit prune-packed &&\n+\tgit count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n+\tcmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n+\tcmp \"test-1-${pack1}.idx\"\t\".git/objects/pack/pack-${pack1}.idx\"\n+'\n \n test_expect_success \\\n- '[index v1] 2) create a stealth corruption in a delta base reference' \\\n- '# This test assumes file_101 is a delta smaller than 16 bytes.\n- # It should be against file_100 but we substitute its base for file_099\n- sha1_101=$(git hash-object file_101) &&\n- sha1_099=$(git hash-object file_099) &&\n- offs_101=$(index_obj_offset 1.idx $sha1_101) &&\n- nr_099=$(index_obj_nr 1.idx $sha1_099) &&\n- chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n- recordsz=$((rawsz + 4)) &&\n- dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n- if=\".git/objects/pack/pack-${pack1}.idx\" \\\n- skip=$((4 + 256 * 4 + $nr_099 * recordsz)) \\\n- bs=1 count=$rawsz conv=notrunc &&\n- git cat-file blob $sha1_101 > file_101_foo1'\n+\t'[index v1] 2) create a stealth corruption in a delta base reference' '\n+\t# This test assumes file_101 is a delta smaller than 16 bytes.\n+\t# It should be against file_100 but we substitute its base for file_099\n+\tsha1_101=$(git hash-object file_101) &&\n+\tsha1_099=$(git hash-object file_099) &&\n+\toffs_101=$(index_obj_offset 1.idx $sha1_101) &&\n+\tnr_099=$(index_obj_nr 1.idx $sha1_099) &&\n+\tchmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n+\trecordsz=$((rawsz + 4)) &&\n+\tdd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n+\t if=\".git/objects/pack/pack-${pack1}.idx\" \\\n+\t skip=$((4 + 256 * 4 + $nr_099 * recordsz)) \\\n+\t bs=1 count=$rawsz conv=notrunc &&\n+\tgit cat-file blob $sha1_101 > file_101_foo1\n+'\n \n test_expect_success \\\n- '[index v1] 3) corrupted delta happily returned wrong data' \\\n- 'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'\n+\t'[index v1] 3) corrupted delta happily returned wrong data' '\n+\ttest -f file_101_foo1 && ! cmp file_101 file_101_foo1\n+'\n \n test_expect_success \\\n- '[index v1] 4) confirm that the pack is actually corrupted' \\\n- 'test_must_fail git fsck --full $commit'\n+\t'[index v1] 4) confirm that the pack is actually corrupted' '\n+\ttest_must_fail git fsck --full $commit\n+'\n \n test_expect_success \\\n- '[index v1] 5) pack-objects happily reuses corrupted data' \\\n- 'pack4=$(git pack-objects test-4 file_101_foo2\n+'\n \n test_expect_success \\\n- '[index v2] 1) stream pack to repository' \\\n- 'rm -f .git/objects/pack/* &&\n- git index-pack --index-version=2 --stdin < \"test-1-${pack1}.pack\" &&\n- git prune-packed &&\n- git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n- cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n- cmp \"test-2-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n+\t'[index v2] 3) corrupted delta happily returned wrong data' '\n+\ttest -f file_101_foo2 && ! cmp file_101 file_101_foo2\n+'\n \n test_expect_success \\\n- '[index v2] 2) create a stealth corruption in a delta base reference' \\\n- '# This test assumes file_101 is a delta smaller than 16 bytes.\n- # It should be against file_100 but we substitute its base for file_099\n- sha1_101=$(git hash-object file_101) &&\n- sha1_099=$(git hash-object file_099) &&\n- offs_101=$(index_obj_offset 1.idx $sha1_101) &&\n- nr_099=$(index_obj_nr 1.idx $sha1_099) &&\n- chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n- dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n- if=\".git/objects/pack/pack-${pack1}.idx\" \\\n- skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \\\n- bs=1 count=$rawsz conv=notrunc &&\n- git cat-file blob $sha1_101 > file_101_foo2'\n+\t'[index v2] 4) confirm that the pack is actually corrupted' '\n+\ttest_must_fail git fsck --full $commit\n+'\n \n test_expect_success \\\n- '[index v2] 3) corrupted delta happily returned wrong data' \\\n- 'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'\n+\t'[index v2] 5) pack-objects refuses to reuse corrupted data' '\n+\ttest_must_fail git pack-objects test-5 /dev/null || exit 1\n- done /dev/null || exit 1\n+\t done wrong-tag <wrong-tag <err &&\n- grep \"^warning:.* expected .tagger. line\" err\n+\ttag=$(git hash-object -t tag -w --stdin err &&\n+\tgrep \"^warning:.* expected .tagger. line\" err\n '\n \n test_expect_success 'index-pack --fsck-objects also warns upon missing tagger in tag' '\n- git index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&\n- grep \"^warning:.* expected .tagger. line\" err\n+\tgit index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&\n+\tgrep \"^warning:.* expected .tagger. line\" err\n '\n \n test_done\n"},{"id":"400166","messageId":"20200619175601.569856-31-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 30/44] connect: parse v2 refs with correct hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:47Z","receivedAt":"2020-06-19T17:56:48Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using protocol v2, we need to know what hash algorithm is used by\nthe remote end. See if the server has sent us an object-format\ncapability, and if so, use it to determine the hash algorithm in use and\nset that value in the packet reader. Parse the refs using this\nalgorithm.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 21 ++++++++++++++++-----\n 1 file changed, 16 insertions(+), 5 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 320cc2016d..e6cf2f8dc4 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -284,7 +284,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \t\tdie(_(\"protocol error: unexpected capabilities^{}\"));\n \t} else if (check_ref(name, flags)) {\n \t\tstruct ref *ref = alloc_ref(name);\n-\t\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n@@ -397,7 +397,7 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\tgoto out;\n \t}\n \n-\tif (parse_oid_hex(line_sections.items[i++].string, &old_oid, &end) ||\n+\tif (parse_oid_hex_algop(line_sections.items[i++].string, &old_oid, &end, reader->hash_algo) ||\n \t *end) {\n \t\tret = 0;\n \t\tgoto out;\n@@ -405,7 +405,7 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \n \tref = alloc_ref(line_sections.items[i++].string);\n \n-\toidcpy(&ref->old_oid, &old_oid);\n+\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n \t**list = ref;\n \t*list = &ref->next;\n \n@@ -418,7 +418,8 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\t\tstruct object_id peeled_oid;\n \t\t\tchar *peeled_name;\n \t\t\tstruct ref *peeled;\n-\t\t\tif (parse_oid_hex(arg, &peeled_oid, &end) || *end) {\n+\t\t\tif (parse_oid_hex_algop(arg, &peeled_oid, &end,\n+\t\t\t\t\t\treader->hash_algo) || *end) {\n \t\t\t\tret = 0;\n \t\t\t\tgoto out;\n \t\t\t}\n@@ -426,7 +427,8 @@ static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n \t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n \t\t\tpeeled = alloc_ref(peeled_name);\n \n-\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n+\t\t\tmemcpy(peeled->old_oid.hash, peeled_oid.hash,\n+\t\t\t reader->hash_algo->rawsz);\n \t\t\t**list = peeled;\n \t\t\t*list = &peeled->next;\n \n@@ -456,6 +458,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\t int stateless_rpc)\n {\n \tint i;\n+\tconst char *hash_name;\n \t*list = NULL;\n \n \tif (server_supports_v2(\"ls-refs\", 1))\n@@ -464,6 +467,14 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \tif (server_supports_v2(\"agent\", 0))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n \n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n+\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n+\t}\n+\n \tif (server_options && server_options->nr &&\n \t server_supports_v2(\"server-option\", 1))\n \t\tfor (i = 0; i < server_options->nr; i++)\n"},{"id":"400170","messageId":"20200619175601.569856-35-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 34/44] remote-curl: detect algorithm for dumb HTTP by size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:51Z","receivedAt":"2020-06-19T17:56:48Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When reading the info/refs file for a repository, we have no explicit\nway to detect which hash algorithm is in use because the file doesn't\nprovide one. Detect the hash algorithm in use by the size of the first\nobject ID.\n\nIf we have an empty repository, we don't know what the hash algorithm is\non the remote side, so default to whatever the local side has\nconfigured. Without doing this, we cannot clone an empty repository\nsince we don't know its hash algorithm. Test this case appropriately,\nsince we currently have no tests for cloning an empty repository with\nthe dumb HTTP protocol.\n\nWe anonymize the URL like elsewhere in the function in case the user has\ndecided to include a secret in the URL.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 23 +++++++++++++++++++++--\n t/t5550-http-fetch-dumb.sh | 18 ++++++++++++++++++\n 2 files changed, 39 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex f0203547c5..e666845d9d 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -252,6 +252,19 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \treturn list;\n }\n \n+static const struct git_hash_algo *detect_hash_algo(struct discovery *heads)\n+{\n+\tconst char *p = memchr(heads->buf, '\\t', heads->len);\n+\tint algo;\n+\tif (!p)\n+\t\treturn the_hash_algo;\n+\n+\talgo = hash_algo_by_length((p - heads->buf) / 2);\n+\tif (algo == GIT_HASH_UNKNOWN)\n+\t\treturn NULL;\n+\treturn &hash_algos[algo];\n+}\n+\n static struct ref *parse_info_refs(struct discovery *heads)\n {\n \tchar *data, *start, *mid;\n@@ -262,6 +275,12 @@ static struct ref *parse_info_refs(struct discovery *heads)\n \tstruct ref *ref = NULL;\n \tstruct ref *last_ref = NULL;\n \n+\toptions.hash_algo = detect_hash_algo(heads);\n+\tif (!options.hash_algo)\n+\t\tdie(\"%sinfo/refs not valid: could not determine hash algorithm; \"\n+\t\t \"is this a git repository?\",\n+\t\t transport_anonymize_url(url.buf));\n+\n \tdata = heads->buf;\n \tstart = NULL;\n \tmid = data;\n@@ -272,13 +291,13 @@ static struct ref *parse_info_refs(struct discovery *heads)\n \t\tif (data[i] == '\\t')\n \t\t\tmid = &data[i];\n \t\tif (data[i] == '\\n') {\n-\t\t\tif (mid - start != the_hash_algo->hexsz)\n+\t\t\tif (mid - start != options.hash_algo->hexsz)\n \t\t\t\tdie(_(\"%sinfo/refs not valid: is this a git repository?\"),\n \t\t\t\t transport_anonymize_url(url.buf));\n \t\t\tdata[i] = 0;\n \t\t\tref_name = mid + 1;\n \t\t\tref = alloc_ref(ref_name);\n-\t\t\tget_oid_hex(start, &ref->old_oid);\n+\t\t\tget_oid_hex_algop(start, &ref->old_oid, options.hash_algo);\n \t\t\tif (!refs)\n \t\t\t\trefs = ref;\n \t\t\tif (last_ref)\ndiff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh\nindex 50485300eb..e57716bacd 100755\n--- a/t/t5550-http-fetch-dumb.sh\n+++ b/t/t5550-http-fetch-dumb.sh\n@@ -50,6 +50,24 @@ test_expect_success 'create password-protected repository' '\n \t \"$HTTPD_DOCUMENT_ROOT_PATH/auth/dumb/repo.git\"\n '\n \n+test_expect_success 'create empty remote repository' '\n+\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/empty.git\" &&\n+\t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/empty.git\" &&\n+\t mkdir -p hooks &&\n+\t write_script \"hooks/post-update\" <<-\\EOF &&\n+\t exec git update-server-info\n+\tEOF\n+\t hooks/post-update\n+\t)\n+'\n+\n+test_expect_success 'empty dumb HTTP repository has default hash algorithm' '\n+\ttest_when_finished \"rm -fr clone-empty\" &&\n+\tgit clone $HTTPD_URL/dumb/empty.git clone-empty &&\n+\tgit -C clone-empty rev-parse --show-object-format >empty-format &&\n+\ttest \"$(cat empty-format)\" = \"$(test_oid algo)\"\n+'\n+\n setup_askpass_helper\n \n test_expect_success 'cloning password-protected repository can fail' '\n"},{"id":"400167","messageId":"20200619175601.569856-30-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 29/44] connect: pass full packet reader when parsing v2 refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:46Z","receivedAt":"2020-06-19T17:56:49Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're parsing refs, we need to know not only what the line we're\nparsing is, but also the hash algorithm we should use to parse it, which\nis stored in the reader object. Pass the packet reader object through\nto the protocol v2 ref parsing function.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex b6e110cb24..320cc2016d 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -376,7 +376,7 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n }\n \n /* Returns 1 when a valid ref has been added to `list`, 0 otherwise */\n-static int process_ref_v2(const char *line, struct ref ***list)\n+static int process_ref_v2(struct packet_reader *reader, struct ref ***list)\n {\n \tint ret = 1;\n \tint i = 0;\n@@ -384,6 +384,7 @@ static int process_ref_v2(const char *line, struct ref ***list)\n \tstruct ref *ref;\n \tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n \tconst char *end;\n+\tconst char *line = reader->line;\n \n \t/*\n \t * Ref lines have a number of fields which are space deliminated. The\n@@ -482,7 +483,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \n \t/* Process response from server */\n \twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n-\t\tif (!process_ref_v2(reader->line, &list))\n+\t\tif (!process_ref_v2(reader, &list))\n \t\t\tdie(_(\"invalid ls-refs response: %s\"), reader->line);\n \t}\n \n"},{"id":"400168","messageId":"20200619175601.569856-32-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 31/44] serve: advertise object-format capability for protocol v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:48Z","receivedAt":"2020-06-19T17:56:50Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In order to communicate the protocol supported by the server side, add\nsupport for advertising the object-format capability. We check that the\nclient side sends us an identical algorithm if it sends us its own\nobject-format capability, and assume it speaks SHA-1 if not.\n\nIn the test, when we're using an algorithm other than SHA-1, we need to\nspecify the algorithm in use so we don't get a failure with an \"unknown\nformat\" message. Add a test that we handle a mismatched algorithm.\nRemove the test_oid_init call since it's no longer necessary.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 2 ++\n serve.c | 27 +++++++++++++++++++++++++++\n t/t5701-git-serve.sh | 25 +++++++++++++++++++++++++\n 3 files changed, 54 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex e6cf2f8dc4..e0d5b9fee0 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -473,6 +473,8 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \t\t\tdie(_(\"unknown object format '%s' specified by server\"), hash_name);\n \t\treader->hash_algo = &hash_algos[hash_algo];\n \t\tpacket_write_fmt(fd_out, \"object-format=%s\", reader->hash_algo->name);\n+\t} else {\n+\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n \n \tif (server_options && server_options->nr &&\ndiff --git a/serve.c b/serve.c\nindex c046926ba1..fbd2fcdfb5 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -22,6 +22,14 @@ static int agent_advertise(struct repository *r,\n \treturn 1;\n }\n \n+static int object_format_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, r->hash_algo->name);\n+\treturn 1;\n+}\n+\n struct protocol_capability {\n \t/*\n \t * The name of the capability. The server uses this name when\n@@ -57,6 +65,7 @@ static struct protocol_capability capabilities[] = {\n \t{ \"ls-refs\", always_advertise, ls_refs },\n \t{ \"fetch\", upload_pack_advertise, upload_pack_v2 },\n \t{ \"server-option\", always_advertise, NULL },\n+\t{ \"object-format\", object_format_advertise, NULL },\n };\n \n static void advertise_capabilities(void)\n@@ -153,6 +162,22 @@ int has_capability(const struct argv_array *keys, const char *capability,\n \treturn 0;\n }\n \n+static void check_algorithm(struct repository *r, struct argv_array *keys)\n+{\n+\tint client = GIT_HASH_SHA1, server = hash_algo_by_ptr(r->hash_algo);\n+\tconst char *algo_name;\n+\n+\tif (has_capability(keys, \"object-format\", &algo_name)) {\n+\t\tclient = hash_algo_by_name(algo_name);\n+\t\tif (client == GIT_HASH_UNKNOWN)\n+\t\t\tdie(\"unknown object format '%s'\", algo_name);\n+\t}\n+\n+\tif (client != server)\n+\t\tdie(\"mismatched object format: server %s; client %s\\n\",\n+\t\t r->hash_algo->name, hash_algos[client].name);\n+}\n+\n enum request_state {\n \tPROCESS_REQUEST_KEYS,\n \tPROCESS_REQUEST_DONE,\n@@ -225,6 +250,8 @@ static int process_request(void)\n \tif (!command)\n \t\tdie(\"no command requested\");\n \n+\tcheck_algorithm(the_repository, &keys);\n+\n \tcommand->command(the_repository, &keys, &reader);\n \n \targv_array_clear(&keys);\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex ffb9613885..a1f5fdc9fd 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -5,12 +5,17 @@ test_description='test protocol v2 server commands'\n . ./test-lib.sh\n \n test_expect_success 'test capability advertisement' '\n+\ttest_oid_cache <<-EOF &&\n+\twrong_algo sha1:sha256\n+\twrong_algo sha256:sha1\n+\tEOF\n \tcat >expect <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n \tfetch=shallow\n \tserver-option\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \n@@ -45,6 +50,7 @@ test_expect_success 'request invalid capability' '\n test_expect_success 'request with no command' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tagent=git/test\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n \tcommand=foo\n+\tobject-format=$(test_oid algo)\n \tagent=git/test\n \t0000\n \tEOF\n@@ -61,6 +68,17 @@ test_expect_success 'request invalid command' '\n \ttest_i18ngrep \"invalid command\" err\n '\n \n+test_expect_success 'wrong object-format' '\n+\ttest-tool pkt-line pack >in <<-EOF &&\n+\tcommand=fetch\n+\tagent=git/test\n+\tobject-format=$(test_oid wrong_algo)\n+\t0000\n+\tEOF\n+\ttest_must_fail test-tool serve-v2 --stateless-rpc 2>err in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0000\n \tEOF\n \n@@ -96,6 +115,7 @@ test_expect_success 'basics of ls-refs' '\n test_expect_success 'basic ref-prefixes' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tref-prefix refs/heads/master\n \tref-prefix refs/tags/one\n@@ -116,6 +136,7 @@ test_expect_success 'basic ref-prefixes' '\n test_expect_success 'refs/heads prefix' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tref-prefix refs/heads/\n \t0000\n@@ -136,6 +157,7 @@ test_expect_success 'refs/heads prefix' '\n test_expect_success 'peel parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tpeel\n \tref-prefix refs/tags/\n@@ -157,6 +179,7 @@ test_expect_success 'peel parameter' '\n test_expect_success 'symrefs parameter' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \t0001\n \tsymrefs\n \tref-prefix refs/heads/\n@@ -178,6 +201,7 @@ test_expect_success 'symrefs parameter' '\n test_expect_success 'sending server-options' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=ls-refs\n+\tobject-format=$(test_oid algo)\n \tserver-option=hello\n \tserver-option=world\n \t0001\n@@ -200,6 +224,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' '\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=fetch\n+\tobject-format=$(test_oid algo)\n \t0001\n \tthis-is-not-a-command\n \t0000\n"},{"id":"400169","messageId":"20200619175601.569856-38-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 37/44] remote-curl: avoid truncating refs with ls-remote","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:54Z","receivedAt":"2020-06-19T17:56:51Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Normally, the remote-curl transport helper is aware of the hash\nalgorithm we're using because we're in a repo with the appropriate hash\nalgorithm set. However, when using git ls-remote outside of a\nrepository, we won't have initialized the hash algorithm properly, so\nuse hash_to_hex_algop to print the ref corresponding to the algorithm\nwe've detected.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex e666845d9d..5cbc6e5002 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -548,7 +548,9 @@ static void output_refs(struct ref *refs)\n \t\tif (posn->symref)\n \t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n \t\telse\n-\t\t\tprintf(\"%s %s\\n\", oid_to_hex(&posn->old_oid), posn->name);\n+\t\t\tprintf(\"%s %s\\n\", hash_to_hex_algop(posn->old_oid.hash,\n+\t\t\t\t\t\t\t options.hash_algo),\n+\t\t\t\t\t posn->name);\n \t}\n \tprintf(\"\\n\");\n \tfflush(stdout);\n"},{"id":"400171","messageId":"20200619175601.569856-34-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 33/44] builtin/ls-remote: initialize repository based on fetch","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:50Z","receivedAt":"2020-06-19T17:56:55Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"ls-remote may or may not operate within a repository, and as such will\nnot have been initialized with the repository's hash algorithm. Even if\nit were, the remote side could be using a different algorithm and we\nwould still want to display those refs properly. Find the hash\nalgorithm used by the remote side by querying the transport object and\nset our hash algorithm accordingly.\n\nWithout this change, if the remote side is using SHA-256, we truncate\nthe refs to 40 hex characters, since that's the length of the default\nhash algorithm (SHA-1).\n\nNote that technically this is not a correct setting of the repository\nhash algorithm since, if we are in a repository, it might be one of a\ndifferent hash algorithm from the remote side. However, our current\ncode paths don't handle multiple algorithms and won't for some time, so\nthis is the best we can do. We rely on the fact that ls-remote never\nmodifies the current repository, which is a reasonable assumption to\nmake.\n\nSigned-off-by: brian m. carlson \n---\n builtin/ls-remote.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex 6ef519514b..3a4dd12903 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -118,6 +118,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\ttransport->server_options = &server_options;\n \n \tref = transport_get_remote_refs(transport, &ref_prefixes);\n+\tif (ref) {\n+\t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t}\n \tif (transport_disconnect(transport)) {\n \t\tUNLEAK(sorting);\n \t\treturn 1;\n"},{"id":"400172","messageId":"20200619175601.569856-29-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 28/44] Documentation/technical: document object-format for protocol v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:45Z","receivedAt":"2020-06-19T17:56:58Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Document the object-format extension for protocol v2.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/protocol-v2.txt | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 3996d70891..b288df7ed7 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -455,3 +455,12 @@ included in a request. This is done by sending each option as a\n a request.\n \n The provided options must not contain a NUL or LF character.\n+\n+ object-format\n+~~~~~~~~~~~~~~~\n+\n+The server can advertise the `object-format` capability with a value `X` (in the\n+form `object-format=X`) to notify the client that the server is able to deal\n+with objects using hash algorithm X. If not specified, the server is assumed to\n+only handle SHA-1. If the client would like to use a hash algorithm other than\n+SHA-1, it should specify its object-format string.\n"},{"id":"400173","messageId":"20200619175601.569856-40-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 39/44] t5702: offer an object-format capability in the test","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:56Z","receivedAt":"2020-06-19T17:56:59Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In order to make this test work with SHA-256, offer an object-format\ncapability so that both sides use the same algorithm.\n\nSigned-off-by: brian m. carlson \n---\n t/t5702-protocol-v2.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 8da65e60de..63f425bbad 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -13,6 +13,7 @@ start_git_daemon --export-all --enable=receive-pack\n daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n \n test_expect_success 'create repo to be served by git-daemon' '\n+\ttest_oid_init &&\n \tgit init \"$daemon_parent\" &&\n \ttest_commit -C \"$daemon_parent\" one\n '\n@@ -394,6 +395,7 @@ test_expect_success 'even with handcrafted request, filter does not work if not\n \t# Custom request that tries to filter even though it is not advertised.\n \ttest-tool pkt-line pack >in <<-EOF &&\n \tcommand=fetch\n+\tobject-format=$(test_oid algo)\n \t0001\n \twant $(git -C server rev-parse master)\n \tfilter blob:none\n"},{"id":"400174","messageId":"20200619175601.569856-42-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 41/44] t5704: send object-format capability with SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:58Z","receivedAt":"2020-06-19T17:57:01Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we speak protocol v2 in this test, we must pass the object-format\nheader if the algorithm is not SHA-1. Otherwise, git upload-pack fails\nbecause the hash algorithm doesn't match and not because we've failed to\nspeak the protocol correctly. Pass the header so that our assertions\ntest what we're really interested in.\n\nSigned-off-by: brian m. carlson \n---\n t/t5704-protocol-violations.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t5704-protocol-violations.sh b/t/t5704-protocol-violations.sh\nindex 950cfb21fe..5c941949b9 100755\n--- a/t/t5704-protocol-violations.sh\n+++ b/t/t5704-protocol-violations.sh\n@@ -9,6 +9,7 @@ making sure that we do not segfault or otherwise behave badly.'\n test_expect_success 'extra delim packet in v2 ls-refs args' '\n \t{\n \t\tpacketize command=ls-refs &&\n+\t\tpacketize \"object-format=$(test_oid algo)\" &&\n \t\tprintf 0001 &&\n \t\t# protocol expects 0000 flush here\n \t\tprintf 0001\n@@ -21,6 +22,7 @@ test_expect_success 'extra delim packet in v2 ls-refs args' '\n test_expect_success 'extra delim packet in v2 fetch args' '\n \t{\n \t\tpacketize command=fetch &&\n+\t\tpacketize \"object-format=$(test_oid algo)\" &&\n \t\tprintf 0001 &&\n \t\t# protocol expects 0000 flush here\n \t\tprintf 0001\n"},{"id":"400175","messageId":"20200619175601.569856-2-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 01/44] t1050: match object ID paths in a hash-insensitive way","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:18Z","receivedAt":"2020-06-19T17:57:03Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The pattern here looking for failures is specific to SHA-1. Let's\ncreate a variable that matches the regex or glob pattern for a path\nwithin the objects directory.\n\nSigned-off-by: brian m. carlson \n---\n t/t1050-large.sh | 2 +-\n t/test-lib.sh | 1 +\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 184b479a21..7f88ea07c2 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -64,7 +64,7 @@ test_expect_success 'add a large file or two' '\n \ttest $count = 1 &&\n \tcnt=$(git show-index <\"$idx\" | wc -l) &&\n \ttest $cnt = 2 &&\n-\tfor l in .git/objects/??/??????????????????????????????????????\n+\tfor l in .git/objects/$OIDPATH_REGEX\n \tdo\n \t\ttest_path_is_file \"$l\" || continue\n \t\tbad=t\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex dbc027ff26..618a7c8d5b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1414,6 +1414,7 @@ test_oid_init\n \n ZERO_OID=$(test_oid zero)\n OID_REGEX=$(echo $ZERO_OID | sed -e 's/0/[0-9a-f]/g')\n+OIDPATH_REGEX=$(test_oid_to_path $ZERO_OID | sed -e 's/0/[0-9a-f]/g')\n EMPTY_TREE=$(test_oid empty_tree)\n EMPTY_BLOB=$(test_oid empty_blob)\n _z40=$ZERO_OID\n"},{"id":"400176","messageId":"20200619175601.569856-33-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 32/44] t5500: make hash independent","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:49Z","receivedAt":"2020-06-19T17:57:05Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This test has hard-coded pkt-lines with object IDs. The pkt-line\nlengths necessarily differ between hash algorithms, so generate these\nlines with the packetize helper so they're always the right size. In\naddition, we will require an object-format capability for SHA-256, so\npass that capability on to the upload-pack process.\n\nSigned-off-by: brian m. carlson \n---\n t/t5500-fetch-pack.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 8c54e34ef1..dfed113247 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -871,9 +871,10 @@ test_expect_success 'shallow since with commit graph and already-seen commit' '\n \n \tGIT_PROTOCOL=version=2 git upload-pack . <<-EOF >/dev/null\n \t0012command=fetch\n+\t$(echo \"object-format=$(test_oid algo)\" | packetize)\n \t00010013deepen-since 1\n-\t0032want $(git rev-parse other)\n-\t0032have $(git rev-parse master)\n+\t$(echo \"want $(git rev-parse other)\" | packetize)\n+\t$(echo \"have $(git rev-parse master)\" | packetize)\n \t0000\n \tEOF\n \t)\n"},{"id":"400177","messageId":"20200619175601.569856-17-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 16/44] docs: update remote helper docs for object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:33Z","receivedAt":"2020-06-19T17:57:06Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Update the remote helper docs to document the object-format extensions\nwe will implement in remote-curl and the transport helper code shortly.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/gitremote-helpers.txt | 33 +++++++++++++++++++++++++----\n 1 file changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex 93baeeb029..6f1e269ae4 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -238,6 +238,9 @@ the remote repository.\n \t`--signed-tags=verbatim` to linkgit:git-fast-export[1]. In the\n \tabsence of this capability, Git will use `--signed-tags=warn-strip`.\n \n+'object-format'::\n+\tThis indicates that the helper is able to interact with the remote\n+\tside using an explicit hash algorithm extension.\n \n \n COMMANDS\n@@ -257,12 +260,14 @@ Support for this command is mandatory.\n 'list'::\n \tLists the refs, one per line, in the format \" \n \t[ ...]\". The value may be a hex sha1 hash, \"@\" for\n-\ta symref, or \"?\" to indicate that the helper could not get the\n-\tvalue of the ref. A space-separated list of attributes follows\n-\tthe name; unrecognized attributes are ignored. The list ends\n-\twith a blank line.\n+\ta symref, \": \" for a key-value pair, or\n+\t\"?\" to indicate that the helper could not get the value of the\n+\tref. A space-separated list of attributes follows the name;\n+\tunrecognized attributes are ignored. The list ends with a\n+\tblank line.\n +\n See REF LIST ATTRIBUTES for a list of currently defined attributes.\n+See REF LIST KEYWORDS for a list of currently defined keywords.\n +\n Supported if the helper has the \"fetch\" or \"import\" capability.\n \n@@ -432,6 +437,18 @@ attributes are defined.\n \tThis ref is unchanged since the last import or fetch, although\n \tthe helper cannot necessarily determine what value that produced.\n \n+REF LIST KEYWORDS\n+-----------------\n+\n+The 'list' command may produce a list of key-value pairs.\n+The following keys are defined.\n+\n+'object-format'::\n+\tThe refs are using the given hash algorithm. This keyword is only\n+\tused if the server and client both support the object-format\n+\textension.\n+\n+\n OPTIONS\n -------\n \n@@ -516,6 +533,14 @@ set by Git if the remote helper has the 'option' capability.\n \ttransaction. If successful, all refs will be updated, or none will. If the\n \tremote side does not support this capability, the push will fail.\n \n+'option object-format' {'true'|algorithm}::\n+\tIf 'true', indicate that the caller wants hash algorithm information\n+\tto be passed back from the remote. This mode is used when fetching\n+\trefs.\n++\n+If set to an algorithm, indicate that the caller wants to interact with\n+the remote side using that algorithm.\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\n"},{"id":"400178","messageId":"20200619175601.569856-10-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 09/44] transport: add a hash algorithm member","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:26Z","receivedAt":"2020-06-19T17:57:15Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When connecting to a remote system, we need to know what hash algorithm\nit will be using to talk to us. Add a hash_algo member to struct\ntransport and add a function to read this data from the transport\nobject.\n\nSigned-off-by: brian m. carlson \n---\n transport.c | 8 ++++++++\n transport.h | 8 ++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/transport.c b/transport.c\nindex 7d50c502ad..a016f41702 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -312,6 +312,7 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \t\tBUG(\"unknown protocol version\");\n \t}\n \tdata->got_remote_heads = 1;\n+\ttransport->hash_algo = reader.hash_algo;\n \n \tif (reader.line_peeked)\n \t\tBUG(\"buffer must be empty at the end of handshake()\");\n@@ -988,9 +989,16 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\t\tret->smart_options->receivepack = remote->receivepack;\n \t}\n \n+\tret->hash_algo = &hash_algos[GIT_HASH_SHA1];\n+\n \treturn ret;\n }\n \n+const struct git_hash_algo *transport_get_hash_algo(struct transport *transport)\n+{\n+\treturn transport->hash_algo;\n+}\n+\n int transport_set_option(struct transport *transport,\n \t\t\t const char *name, const char *value)\n {\ndiff --git a/transport.h b/transport.h\nindex 4298c855be..2a9f96c05a 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -115,6 +115,8 @@ struct transport {\n \tstruct git_transport_options *smart_options;\n \n \tenum transport_family family;\n+\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n #define TRANSPORT_PUSH_ALL\t\t\t(1<<0)\n@@ -243,6 +245,12 @@ int transport_push(struct repository *repo,\n const struct ref *transport_get_remote_refs(struct transport *transport,\n \t\t\t\t\t const struct argv_array *ref_prefixes);\n \n+/*\n+ * Fetch the hash algorithm used by a remote.\n+ *\n+ * This can only be called after fetching the remote refs.\n+ */\n+const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n int transport_disconnect(struct transport *transport);\n"},{"id":"400179","messageId":"20200619175601.569856-15-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 14/44] connect: detect algorithm when fetching refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:31Z","receivedAt":"2020-06-19T17:57:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"If we're fetching refs, detect the hash algorithm and parse the refs\nusing that algorithm.\n\nAs mentioned in the documentation, if multiple versions of the\nobject-format capability are provided, we use the first. No known\nimplementation supports multiple algorithms now, but they may in the\nfuture.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 8f70a05699..b6e110cb24 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -221,12 +221,25 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \n static void process_capabilities(struct packet_reader *reader, int *linelen)\n {\n+\tconst char *feat_val;\n+\tint feat_len;\n \tconst char *line = reader->line;\n \tint nul_location = strlen(line);\n \tif (nul_location == *linelen)\n \t\treturn;\n \tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n \t*linelen = nul_location;\n+\n+\tfeat_val = server_feature_value(\"object-format\", &feat_len);\n+\tif (feat_val) {\n+\t\tchar *hash_name = xstrndup(feat_val, feat_len);\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo != GIT_HASH_UNKNOWN)\n+\t\t\treader->hash_algo = &hash_algos[hash_algo];\n+\t\tfree(hash_name);\n+\t} else {\n+\t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n+\t}\n }\n \n static int process_dummy_ref(const struct packet_reader *reader)\n@@ -235,7 +248,7 @@ static int process_dummy_ref(const struct packet_reader *reader)\n \tstruct object_id oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(line, &oid, &name))\n+\tif (parse_oid_hex_algop(line, &oid, &name, reader->hash_algo))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -259,7 +272,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \tstruct object_id old_oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(line, &old_oid, &name))\n+\tif (parse_oid_hex_algop(line, &old_oid, &name, reader->hash_algo))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -271,7 +284,7 @@ static int process_ref(const struct packet_reader *reader, int len,\n \t\tdie(_(\"protocol error: unexpected capabilities^{}\"));\n \t} else if (check_ref(name, flags)) {\n \t\tstruct ref *ref = alloc_ref(name);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\tmemcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n@@ -289,7 +302,7 @@ static int process_shallow(const struct packet_reader *reader, int len,\n \tif (!skip_prefix(line, \"shallow \", &arg))\n \t\treturn 0;\n \n-\tif (get_oid_hex(arg, &old_oid))\n+\tif (get_oid_hex_algop(arg, &old_oid, reader->hash_algo))\n \t\tdie(_(\"protocol error: expected shallow sha-1, got '%s'\"), arg);\n \tif (!shallow_points)\n \t\tdie(_(\"repository on the other end cannot be shallow\"));\n"},{"id":"400180","messageId":"20200619175601.569856-8-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 07/44] connect: add function to fetch value of a v2 server capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:24Z","receivedAt":"2020-06-19T17:57:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"So far in protocol v2, all of our server capabilities that have values\nhave not had values that we've been interested in parsing. For example,\nwe receive but ignore the agent value.\n\nHowever, in a future commit, we're going to want to parse out the value\nof a server capability. To make this easy, add a function,\nserver_feature_v2, that can fetch the value provided as part of the\nserver capability.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 15 +++++++++++++++\n connect.h | 1 +\n 2 files changed, 16 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex d30c637ab3..2cded78b0a 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -84,6 +84,21 @@ int server_supports_v2(const char *c, int die_on_error)\n \treturn 0;\n }\n \n+int server_feature_v2(const char *c, const char **v)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (*out == '=')) {\n+\t\t\t*v = out + 1;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n int server_supports_feature(const char *c, const char *feature,\n \t\t\t int die_on_error)\n {\ndiff --git a/connect.h b/connect.h\nindex 235bc66254..88702fdd17 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -19,6 +19,7 @@ struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_v2(const char *c, int die_on_error);\n+int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n \t\t\t int die_on_error);\n \n"},{"id":"400181","messageId":"20200619175601.569856-6-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 05/44] remote: advertise the object-format capability on the server side","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:22Z","receivedAt":"2020-06-19T17:57:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Advertise the current hash algorithm in use by using the object-format\ncapability as part of the ref advertisement.\n\nSigned-off-by: brian m. carlson \n---\n builtin/receive-pack.c | 1 +\n upload-pack.c | 3 ++-\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex ea3d0f01af..4ffa501dce 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -249,6 +249,7 @@ static void show_ref(const char *path, const struct object_id *oid)\n \t\t\tstrbuf_addf(&cap, \" push-cert=%s\", push_cert_nonce);\n \t\tif (advertise_push_options)\n \t\t\tstrbuf_addstr(&cap, \" push-options\");\n+\t\tstrbuf_addf(&cap, \" object-format=%s\", the_hash_algo->name);\n \t\tstrbuf_addf(&cap, \" agent=%s\", git_user_agent_sanitized());\n \t\tpacket_write_fmt(1, \"%s %s%c%s\\n\",\n \t\t\t oid_to_hex(oid), path, 0, cap.buf);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 401c9e6c4b..a72eef9d14 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1063,7 +1063,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n \t\tformat_symref_info(&symref_info, &data->symref);\n-\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s agent=%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s object-format=%s agent=%s\\n\",\n \t\t\t oid_to_hex(oid), refname_nons,\n \t\t\t 0, capabilities,\n \t\t\t (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n@@ -1073,6 +1073,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\t\t data->stateless_rpc ? \" no-done\" : \"\",\n \t\t\t symref_info.buf,\n \t\t\t allow_filter ? \" filter\" : \"\",\n+\t\t\t the_hash_algo->name,\n \t\t\t git_user_agent_sanitized());\n \t\tstrbuf_release(&symref_info);\n \t} else {\n"},{"id":"400182","messageId":"20200619175601.569856-7-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 06/44] connect: add function to parse multiple v1 capability values","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:23Z","receivedAt":"2020-06-19T17:57:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In a capability response, we can have multiple symref entries. In the\nfuture, we will also allow for multiple hash algorithms to be specified.\nTo avoid duplication, expand the parse_feature_value function to take an\noptional offset where the parsing should begin next time. Add a wrapper\nfunction that allows us to query the next server feature value, and use\nit in the existing symref parsing code.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 30 +++++++++++++++++++++---------\n 1 file changed, 21 insertions(+), 9 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex e66e779ebd..d30c637ab3 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -18,7 +18,8 @@\n \n static char *server_capabilities_v1;\n static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n-static const char *parse_feature_value(const char *, const char *, int *);\n+static const char *parse_feature_value(const char *, const char *, int *, int *);\n+static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n \n static int check_ref(const char *name, unsigned int flags)\n {\n@@ -181,17 +182,16 @@ static void parse_one_symref_info(struct string_list *symref, const char *val, i\n static void annotate_refs_with_symref_info(struct ref *ref)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\tconst char *feature_list = server_capabilities_v1;\n+\tint offset = 0;\n \n-\twhile (feature_list) {\n+\twhile (1) {\n \t\tint len;\n \t\tconst char *val;\n \n-\t\tval = parse_feature_value(feature_list, \"symref\", &len);\n+\t\tval = next_server_feature_value(\"symref\", &len, &offset);\n \t\tif (!val)\n \t\t\tbreak;\n \t\tparse_one_symref_info(&symref, val, len);\n-\t\tfeature_list = val + 1;\n \t}\n \tstring_list_sort(&symref);\n \n@@ -468,7 +468,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \treturn list;\n }\n \n-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n+static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n {\n \tint len;\n \n@@ -476,6 +476,8 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \t\treturn NULL;\n \n \tlen = strlen(feature);\n+\tif (offset)\n+\t\tfeature_list += *offset;\n \twhile (*feature_list) {\n \t\tconst char *found = strstr(feature_list, feature);\n \t\tif (!found)\n@@ -490,9 +492,14 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \t\t\t}\n \t\t\t/* feature with a value (e.g., \"agent=git/1.2.3\") */\n \t\t\telse if (*value == '=') {\n+\t\t\t\tint end;\n+\n \t\t\t\tvalue++;\n+\t\t\t\tend = strcspn(value, \" \\t\\n\");\n \t\t\t\tif (lenp)\n-\t\t\t\t\t*lenp = strcspn(value, \" \\t\\n\");\n+\t\t\t\t\t*lenp = end;\n+\t\t\t\tif (offset)\n+\t\t\t\t\t*offset = value + end - feature_list;\n \t\t\t\treturn value;\n \t\t\t}\n \t\t\t/*\n@@ -507,12 +514,17 @@ static const char *parse_feature_value(const char *feature_list, const char *fea\n \n int parse_feature_request(const char *feature_list, const char *feature)\n {\n-\treturn !!parse_feature_value(feature_list, feature, NULL);\n+\treturn !!parse_feature_value(feature_list, feature, NULL, NULL);\n+}\n+\n+static const char *next_server_feature_value(const char *feature, int *len, int *offset)\n+{\n+\treturn parse_feature_value(server_capabilities_v1, feature, len, offset);\n }\n \n const char *server_feature_value(const char *feature, int *len)\n {\n-\treturn parse_feature_value(server_capabilities_v1, feature, len);\n+\treturn parse_feature_value(server_capabilities_v1, feature, len, NULL);\n }\n \n int server_supports(const char *feature)\n"},{"id":"400183","messageId":"20200619175601.569856-45-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 44/44] remote-testgit: adapt for object-format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:56:01Z","receivedAt":"2020-06-19T17:57:26Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using an algorithm other than SHA-1, we need the remote helper to\nadvertise support for the object-format extension and provide\ninformation back to us so that we can properly parse refs and return\ndata. Ensure that the test remote helper understands these extensions.\n\nSigned-off-by: brian m. carlson \n---\n t/t5801/git-remote-testgit | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/t/t5801/git-remote-testgit b/t/t5801/git-remote-testgit\nindex 6b9f0b5dc7..1544d6dc6b 100755\n--- a/t/t5801/git-remote-testgit\n+++ b/t/t5801/git-remote-testgit\n@@ -52,9 +52,11 @@ do\n \t\ttest -n \"$GIT_REMOTE_TESTGIT_SIGNED_TAGS\" && echo \"signed-tags\"\n \t\ttest -n \"$GIT_REMOTE_TESTGIT_NO_PRIVATE_UPDATE\" && echo \"no-private-update\"\n \t\techo 'option'\n+\t\techo 'object-format'\n \t\techo\n \t\t;;\n \tlist)\n+\t\techo \":object-format $(git rev-parse --show-object-format=storage)\"\n \t\tgit for-each-ref --format='? %(refname)' 'refs/heads/' 'refs/tags/'\n \t\thead=$(git symbolic-ref HEAD)\n \t\techo \"@$head HEAD\"\n@@ -139,6 +141,10 @@ do\n \t\t\ttest $val = \"true\" && force=\"true\" || force=\n \t\t\techo \"ok\"\n \t\t\t;;\n+\t\tobject-format)\n+\t\t\ttest $val = \"true\" && object_format=\"true\" || object_format=\n+\t\t\techo \"ok\"\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"unsupported\"\n \t\t\t;;\n"},{"id":"400184","messageId":"20200619175601.569856-44-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 43/44] bundle: detect hash algorithm when reading refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:56:00Z","receivedAt":"2020-06-19T17:57:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Much like with the dumb HTTP transport, there isn't a way to explicitly\nspecify the hash algorithm when dealing with a bundle, so detect the\nalgorithm based on the length of the object IDs in the prerequisites and\nref advertisements.\n\nSigned-off-by: brian m. carlson \n---\n bundle.c | 22 +++++++++++++++++++++-\n bundle.h | 1 +\n transport.c | 10 ++++++++--\n 3 files changed, 30 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 99439e07a1..2a0d744d3f 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -23,6 +23,17 @@ static void add_to_ref_list(const struct object_id *oid, const char *name,\n \tlist->nr++;\n }\n \n+static const struct git_hash_algo *detect_hash_algo(struct strbuf *buf)\n+{\n+\tsize_t len = strcspn(buf->buf, \" \\n\");\n+\tint algo;\n+\n+\talgo = hash_algo_by_length(len / 2);\n+\tif (algo == GIT_HASH_UNKNOWN)\n+\t\treturn NULL;\n+\treturn &hash_algos[algo];\n+}\n+\n static int parse_bundle_header(int fd, struct bundle_header *header,\n \t\t\t const char *report_path)\n {\n@@ -52,12 +63,21 @@ static int parse_bundle_header(int fd, struct bundle_header *header,\n \t\t}\n \t\tstrbuf_rtrim(&buf);\n \n+\t\tif (!header->hash_algo) {\n+\t\t\theader->hash_algo = detect_hash_algo(&buf);\n+\t\t\tif (!header->hash_algo) {\n+\t\t\t\terror(_(\"unknown hash algorithm length\"));\n+\t\t\t\tstatus = -1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n \t\t/*\n \t\t * Tip lines have object name, SP, and refname.\n \t\t * Prerequisites have object name that is optionally\n \t\t * followed by SP and subject line.\n \t\t */\n-\t\tif (parse_oid_hex(buf.buf, &oid, &p) ||\n+\t\tif (parse_oid_hex_algop(buf.buf, &oid, &p, header->hash_algo) ||\n \t\t (*p && !isspace(*p)) ||\n \t\t (!is_prereq && !*p)) {\n \t\t\tif (report_path)\ndiff --git a/bundle.h b/bundle.h\nindex ceab0c7475..2dc9442024 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -15,6 +15,7 @@ struct ref_list {\n struct bundle_header {\n \tstruct ref_list prerequisites;\n \tstruct ref_list references;\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n int is_bundle(const char *path, int quiet);\ndiff --git a/transport.c b/transport.c\nindex a016f41702..b255f123c0 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -143,6 +143,9 @@ static struct ref *get_refs_from_bundle(struct transport *transport,\n \tdata->fd = read_bundle_header(transport->url, &data->header);\n \tif (data->fd < 0)\n \t\tdie(_(\"could not read bundle '%s'\"), transport->url);\n+\n+\ttransport->hash_algo = data->header.hash_algo;\n+\n \tfor (i = 0; i < data->header.references.nr; i++) {\n \t\tstruct ref_list_entry *e = data->header.references.list + i;\n \t\tstruct ref *ref = alloc_ref(e->name);\n@@ -157,11 +160,14 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n \tstruct bundle_transport_data *data = transport->data;\n+\tint ret;\n \n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle(transport, 0, NULL);\n-\treturn unbundle(the_repository, &data->header, data->fd,\n-\t\t\ttransport->progress ? BUNDLE_VERBOSE : 0);\n+\tret = unbundle(the_repository, &data->header, data->fd,\n+\t\t\t transport->progress ? BUNDLE_VERBOSE : 0);\n+\ttransport->hash_algo = data->header.hash_algo;\n+\treturn ret;\n }\n \n static int close_bundle(struct transport *transport)\n"},{"id":"400185","messageId":"20200619175601.569856-5-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 04/44] wrapper: add function to compare strings with different NUL termination","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:21Z","receivedAt":"2020-06-19T17:57:31Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When parsing capabilities for the pack protocol, there are times we'll\nwant to compare the value of a capability to a NUL-terminated string.\nSince the data we're reading will be space-terminated, not\nNUL-terminated, we need a function that compares the two strings, but\nalso checks that they're the same length. Otherwise, if we used strncmp\nto compare these strings, we might accidentally accept a parameter that\nwas a prefix of the expected value.\n\nAdd a function, xstrncmpz, that takes a NUL-terminated string and a\nnon-NUL-terminated string, plus a length, and compares them, ensuring\nthat they are the same length.\n\nSigned-off-by: brian m. carlson \n---\n git-compat-util.h | 6 ++++++\n wrapper.c | 8 ++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a73632e8e4..5637114b8d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -868,6 +868,12 @@ char *xgetcwd(void);\n FILE *fopen_for_writing(const char *path);\n FILE *fopen_or_warn(const char *path, const char *mode);\n \n+/*\n+ * Like strncmp, but only return zero if s is NUL-terminated and exactly len\n+ * characters long. If it is not, consider it greater than t.\n+ */\n+int xstrncmpz(const char *s, const char *t, size_t len);\n+\n /*\n * FREE_AND_NULL(ptr) is like free(ptr) followed by ptr = NULL. Note\n * that ptr is used twice, so don't pass e.g. ptr++.\ndiff --git a/wrapper.c b/wrapper.c\nindex 3a1c0e0526..4ff4a9c3db 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -105,6 +105,14 @@ char *xstrndup(const char *str, size_t len)\n \treturn xmemdupz(str, p ? p - str : len);\n }\n \n+int xstrncmpz(const char *s, const char *t, size_t len)\n+{\n+\tint res = strncmp(s, t, len);\n+\tif (res)\n+\t\treturn res;\n+\treturn s[len] == '\\0' ? 0 : 1;\n+}\n+\n void *xrealloc(void *ptr, size_t size)\n {\n \tvoid *ret;\n"},{"id":"400186","messageId":"20200619175601.569856-36-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 35/44] builtin/index-pack: add option to specify hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:52Z","receivedAt":"2020-06-19T17:57:36Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"git index-pack is usually run in a repository, but need not be. Since\npacks don't contains information on the algorithm in use, instead\nrelying on context, add an option to index-pack to tell it which one\nwe're using in case someone runs it outside of a repository. Since\nusing --stdin necessarily implies a repository, don't allow specifying\nan object format if it's provided to prevent users from passing an\noption that won't work. Add documentation for this option.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-index-pack.txt | 8 ++++++++\n builtin/index-pack.c | 8 ++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex d5b7560bfe..9316d9a80b 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -93,6 +93,14 @@ OPTIONS\n --max-input-size=::\n \tDie, if the pack is larger than .\n \n+--object-format=::\n+\tSpecify the given object format (hash algorithm) for the pack. The valid\n+\tvalues are 'sha1' and (if enabled) 'sha256'. The default is the algorithm for\n+\tthe current repository (set by `extensions.objectFormat`), or 'sha1' if no\n+\tvalue is set or outside a repository.\n++\n+This option cannot be used with --stdin.\n+\n NOTES\n -----\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7bea1fba52..f865666db9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1667,6 +1667,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tunsigned char pack_hash[GIT_MAX_RAWSZ];\n \tunsigned foreign_nr = 1;\t/* zero is a \"good\" value, assume bad */\n \tint report_end_of_input = 0;\n+\tint hash_algo = 0;\n \n \t/*\n \t * index-pack never needs to fetch missing objects except when\n@@ -1760,6 +1761,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t} else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n \t\t\t\tmax_input_size = strtoumax(arg, NULL, 10);\n+\t\t\t} else if (skip_prefix(arg, \"--object-format=\", &arg)) {\n+\t\t\t\thash_algo = hash_algo_by_name(arg);\n+\t\t\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n+\t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n+\t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -1776,6 +1782,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"--fix-thin cannot be used without --stdin\"));\n \tif (from_stdin && !startup_info->have_repository)\n \t\tdie(_(\"--stdin requires a git repository\"));\n+\tif (from_stdin && hash_algo)\n+\t\tdie(_(\"--object-format cannot be used with --stdin\"));\n \tif (!index_name && pack_name)\n \t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n \n"},{"id":"400187","messageId":"20200619175601.569856-28-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 27/44] t1302: expect repo format version 1 for SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:44Z","receivedAt":"2020-06-19T17:57:37Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When using SHA-256, we need to take advantage of the extensions section\nin the config file, so we need to use repository format version 1.\nUpdate the test to look for the correct value.\n\nNote that test_oid produces a value without a trailing newline, so use\necho to ensure we print a trailing newline to compare it correctly\nagainst the actual results.\n\nSigned-off-by: brian m. carlson \n---\n t/t1302-repo-version.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh\nindex ce4cff13bb..d60c042ce8 100755\n--- a/t/t1302-repo-version.sh\n+++ b/t/t1302-repo-version.sh\n@@ -8,6 +8,10 @@ test_description='Test repository version check'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n+\ttest_oid_cache <<-\\EOF &&\n+\tversion sha1:0\n+\tversion sha256:1\n+\tEOF\n \tcat >test.patch <<-\\EOF &&\n \tdiff --git a/test.txt b/test.txt\n \tnew file mode 100644\n@@ -23,7 +27,7 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'gitdir selection on normal repos' '\n-\techo 0 >expect &&\n+\techo $(test_oid version) >expect &&\n \tgit config core.repositoryformatversion >actual &&\n \tgit -C test config core.repositoryformatversion >actual2 &&\n \ttest_cmp expect actual &&\n"},{"id":"400188","messageId":"20200619175601.569856-41-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 40/44] t5703: use object-format serve option","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:57Z","receivedAt":"2020-06-19T17:57:38Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When we're using an algorithm other than SHA-1, we need to specify the\nalgorithm in use so we don't get a failure with an \"unknown format\"\nmessage. Add a wrapper function that specifies this header if required.\nSkip specifying this header for SHA-1 to test that it works both with an\nwithout this header.\n\nSigned-off-by: brian m. carlson \n---\n t/t5703-upload-pack-ref-in-want.sh | 19 ++++++++++++++-----\n 1 file changed, 14 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh\nindex 92ad5eeec0..748282f058 100755\n--- a/t/t5703-upload-pack-ref-in-want.sh\n+++ b/t/t5703-upload-pack-ref-in-want.sh\n@@ -27,6 +27,15 @@ check_output () {\n \ttest_cmp sorted_commits actual_commits\n }\n \n+write_command () {\n+\techo \"command=$1\"\n+\n+\tif test \"$(test_oid algo)\" != sha1\n+\tthen\n+\t\techo \"object-format=$(test_oid algo)\"\n+\tfi\n+}\n+\n # c(o/foo) d(o/bar)\n # \\ /\n # b e(baz) f(master)\n@@ -65,7 +74,7 @@ test_expect_success 'config controls ref-in-want advertisement' '\n \n test_expect_success 'invalid want-ref line' '\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/non-existent\n@@ -86,7 +95,7 @@ test_expect_success 'basic want-ref' '\n \n \toid=$(git rev-parse a) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/master\n@@ -110,7 +119,7 @@ test_expect_success 'multiple want-ref lines' '\n \n \toid=$(git rev-parse b) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/o/foo\n@@ -132,7 +141,7 @@ test_expect_success 'mix want and want-ref' '\n \tgit rev-parse e f >expected_commits &&\n \n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/master\n@@ -155,7 +164,7 @@ test_expect_success 'want-ref with ref we already have commit for' '\n \n \toid=$(git rev-parse c) &&\n \ttest-tool pkt-line pack >in <<-EOF &&\n-\tcommand=fetch\n+\t$(write_command fetch)\n \t0001\n \tno-progress\n \twant-ref refs/heads/o/foo\n"},{"id":"400189","messageId":"20200619175601.569856-39-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 38/44] t/helper: initialize the repository for test-sha1-array","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:55Z","receivedAt":"2020-06-19T17:57:38Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"test-sha1-array uses the_hash_algo under the hood. Since t0064 wants to\nuse the value that is correct for the hash algorithm that we're testing,\nmake sure the test helper initializes the repository to set\nthe_hash_algo correctly.\n\nSigned-off-by: brian m. carlson \n---\n t/helper/test-oid-array.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/t/helper/test-oid-array.c b/t/helper/test-oid-array.c\nindex ce9fd5f091..b16cd0b11b 100644\n--- a/t/helper/test-oid-array.c\n+++ b/t/helper/test-oid-array.c\n@@ -12,6 +12,9 @@ int cmd__oid_array(int argc, const char **argv)\n {\n \tstruct oid_array array = OID_ARRAY_INIT;\n \tstruct strbuf line = STRBUF_INIT;\n+\tint nongit_ok;\n+\n+\tsetup_git_directory_gently(&nongit_ok);\n \n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tconst char *arg;\n"},{"id":"400190","messageId":"20200619175601.569856-37-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 36/44] t1050: pass algorithm to index-pack when outside repo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:53Z","receivedAt":"2020-06-19T17:57:39Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When outside a repository, git index-pack is unable to guess the hash\nalgorithm in use for a pack, since packs don't contain any information\non the algorithm in use. Pass an option to index-pack to help it out in\nthis test.\n\nSigned-off-by: brian m. carlson \n---\n t/t1050-large.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 7f88ea07c2..6a56d1ca24 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -12,6 +12,7 @@ file_size () {\n }\n \n test_expect_success setup '\n+\ttest_oid_init &&\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n@@ -177,7 +178,8 @@ test_expect_success 'git-show a large file' '\n \n test_expect_success 'index-pack' '\n \tgit clone file://\"$(pwd)\"/.git foo &&\n-\tGIT_DIR=non-existent git index-pack --strict --verify foo/.git/objects/pack/*.pack\n+\tGIT_DIR=non-existent git index-pack --object-format=$(test_oid algo) \\\n+\t\t--strict --verify foo/.git/objects/pack/*.pack\n '\n \n test_expect_success 'repack' '\n"},{"id":"400191","messageId":"20200619175601.569856-43-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 42/44] t5300: pass --object-format to git index-pack","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:59Z","receivedAt":"2020-06-19T17:57:39Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"git index-pack by default reads the repository to determine the object\nformat. However, when outside of a repository, it's necessary to specify\nthe hash algorithm in use so that the pack can be properly indexed. Add\nan --object-format argument when invoking git index-pack outside of a\nrepository.\n\nSigned-off-by: brian m. carlson \n---\n t/t5300-pack-object.sh | 9 +++++----\n 1 file changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 410a09b0dd..746cdb626e 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -12,7 +12,8 @@ TRASH=$(pwd)\n \n test_expect_success \\\n 'setup' \\\n- 'rm -f .git/index* &&\n+ 'test_oid_init &&\n+ rm -f .git/index* &&\n perl -e \"print \\\"a\\\" x 4096;\" > a &&\n perl -e \"print \\\"b\\\" x 4096;\" > b &&\n perl -e \"print \\\"c\\\" x 4096;\" > c &&\n@@ -412,18 +413,18 @@ test_expect_success 'set up pack for non-repo tests' '\n '\n \n test_expect_success 'index-pack --stdin complains of non-repo' '\n-\tnongit test_must_fail git index-pack --stdin works in non-repo' '\n-\tnongit git index-pack ../foo.pack &&\n+\tnongit git index-pack --object-format=$(test_oid algo) ../foo.pack &&\n \ttest_path_is_file foo.idx\n '\n \n test_expect_success 'index-pack --strict works in non-repo' '\n \trm -f foo.idx &&\n-\tnongit git index-pack --strict ../foo.pack &&\n+\tnongit git index-pack --strict --object-format=$(test_oid algo) ../foo.pack &&\n \ttest_path_is_file foo.idx\n '\n \n"},{"id":"400192","messageId":"20200619175601.569856-16-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 15/44] builtin/receive-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:32Z","receivedAt":"2020-06-19T17:57:40Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n builtin/receive-pack.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 4ffa501dce..d43663bb0a 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1625,6 +1625,8 @@ static struct command *read_head_info(struct packet_reader *reader,\n \t\tlinelen = strlen(reader->line);\n \t\tif (linelen < reader->pktlen) {\n \t\t\tconst char *feature_list = reader->line + linelen + 1;\n+\t\t\tconst char *hash = NULL;\n+\t\t\tint len = 0;\n \t\t\tif (parse_feature_request(feature_list, \"report-status\"))\n \t\t\t\treport_status = 1;\n \t\t\tif (parse_feature_request(feature_list, \"side-band-64k\"))\n@@ -1637,6 +1639,13 @@ static struct command *read_head_info(struct packet_reader *reader,\n \t\t\tif (advertise_push_options\n \t\t\t && parse_feature_request(feature_list, \"push-options\"))\n \t\t\t\tuse_push_options = 1;\n+\t\t\thash = parse_feature_value(feature_list, \"object-format\", &len, NULL);\n+\t\t\tif (!hash) {\n+\t\t\t\thash = hash_algos[GIT_HASH_SHA1].name;\n+\t\t\t\tlen = strlen(hash);\n+\t\t\t}\n+\t\t\tif (xstrncmpz(the_hash_algo->name, hash, len))\n+\t\t\t\tdie(\"error: unsupported object format '%s'\", hash);\n \t\t}\n \n \t\tif (!strcmp(reader->line, \"push-cert\")) {\n"},{"id":"400193","messageId":"20200619175601.569856-19-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 18/44] remote-curl: implement object-format extensions","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:35Z","receivedAt":"2020-06-19T17:57:41Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Implement the object-format extensions that let us determine the hash\nalgorithm in use when pushing, pulling, and fetching.\n\nSigned-off-by: brian m. carlson \n---\n remote-curl.c | 19 ++++++++++++++++++-\n 1 file changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 75532a8bae..f0203547c5 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -41,7 +41,9 @@ struct options {\n \t\tdeepen_relative : 1,\n \t\tfrom_promisor : 1,\n \t\tno_dependents : 1,\n-\t\tatomic : 1;\n+\t\tatomic : 1,\n+\t\tobject_format : 1;\n+\tconst struct git_hash_algo *hash_algo;\n };\n static struct options options;\n static struct string_list cas_options = STRING_LIST_INIT_DUP;\n@@ -190,6 +192,16 @@ static int set_option(const char *name, const char *value)\n \t} else if (!strcmp(name, \"filter\")) {\n \t\toptions.filter = xstrdup(value);\n \t\treturn 0;\n+\t} else if (!strcmp(name, \"object-format\")) {\n+\t\tint algo;\n+\t\toptions.object_format = 1;\n+\t\tif (strcmp(value, \"true\")) {\n+\t\t\talgo = hash_algo_by_name(value);\n+\t\t\tif (algo == GIT_HASH_UNKNOWN)\n+\t\t\t\tdie(\"unknown object format '%s'\", value);\n+\t\t\toptions.hash_algo = &hash_algos[algo];\n+\t\t}\n+\t\treturn 0;\n \t} else {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -231,6 +243,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n \t\t\t\t NULL, &heads->shallow);\n+\t\toptions.hash_algo = reader.hash_algo;\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n@@ -509,6 +522,9 @@ static struct ref *get_refs(int for_push)\n static void output_refs(struct ref *refs)\n {\n \tstruct ref *posn;\n+\tif (options.object_format && options.hash_algo) {\n+\t\tprintf(\":object-format %s\\n\", options.hash_algo->name);\n+\t}\n \tfor (posn = refs; posn; posn = posn->next) {\n \t\tif (posn->symref)\n \t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n@@ -1499,6 +1515,7 @@ int cmd_main(int argc, const char **argv)\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n+\t\t\tprintf(\"object-format\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else if (skip_prefix(buf.buf, \"stateless-connect \", &arg)) {\n"},{"id":"400194","messageId":"20200619175601.569856-24-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 23/44] t3200: mark assertion with SHA1 prerequisite","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:40Z","receivedAt":"2020-06-19T17:57:42Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"One of the test assertions in this test checks that git branch -m works\neven without a .git/config file. However, if the repository requires\nconfiguration extensions, such as because it uses a non-SHA-1 algorithm,\nthis assertion will fail. Mark the assertion as requiring SHA-1.\n\nSigned-off-by: brian m. carlson \n---\n t/t3200-branch.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\nindex 411a70b0ce..2a3fedc6b0 100755\n--- a/t/t3200-branch.sh\n+++ b/t/t3200-branch.sh\n@@ -402,7 +402,7 @@ EOF\n \n mv .git/config .git/config-saved\n \n-test_expect_success 'git branch -m q q2 without config should succeed' '\n+test_expect_success SHA1 'git branch -m q q2 without config should succeed' '\n \tgit branch -m q q2 &&\n \tgit branch -m q2 q\n '\n"},{"id":"400195","messageId":"20200619175601.569856-25-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 24/44] packfile: compute and use the index CRC offset","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:41Z","receivedAt":"2020-06-19T17:57:43Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Both v2 pack index files and the v3 format specified as part of the\nNewHash work have similar data starting at the CRC table. Much of the\nexisting code wants to read either this table or the offset entries\nfollowing it, and in doing so computes the offset each time.\n\nIn order to share as much code between v2 and v3, compute the offset of\nthe CRC table and store it when the pack is opened. Use this value to\ncompute offsets to not only the CRC table, but to the offset entries\nbeyond it.\n\nSigned-off-by: brian m. carlson \n---\n builtin/index-pack.c | 6 +-----\n object-store.h | 1 +\n packfile.c | 1 +\n 3 files changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f176dd28c8..7bea1fba52 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1555,13 +1555,9 @@ static void read_v2_anomalous_offsets(struct packed_git *p,\n {\n \tconst uint32_t *idx1, *idx2;\n \tuint32_t i;\n-\tconst uint32_t hashwords = the_hash_algo->rawsz / sizeof(uint32_t);\n \n \t/* The address of the 4-byte offset table */\n-\tidx1 = (((const uint32_t *)p->index_data)\n-\t\t+ 2 /* 8-byte header */\n-\t\t+ 256 /* fan out */\n-\t\t+ hashwords * p->num_objects /* object ID table */\n+\tidx1 = (((const uint32_t *)((const uint8_t *)p->index_data + p->crc_offset))\n \t\t+ p->num_objects /* CRC32 table */\n \t\t);\n \ndiff --git a/object-store.h b/object-store.h\nindex d1e490f203..f439d47af8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -70,6 +70,7 @@ struct packed_git {\n \tsize_t index_size;\n \tuint32_t num_objects;\n \tuint32_t num_bad_objects;\n+\tuint32_t crc_offset;\n \tunsigned char *bad_object_sha1;\n \tint index_version;\n \ttime_t mtime;\ndiff --git a/packfile.c b/packfile.c\nindex f4e752996d..6ab5233613 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -178,6 +178,7 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,\n \t\t */\n \t\t (sizeof(off_t) <= 4))\n \t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n+\t\tp->crc_offset = 8 + 4 * 256 + nr * hashsz;\n \t}\n \n \tp->index_version = version;\n"},{"id":"400196","messageId":"20200619175601.569856-12-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 11/44] send-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:28Z","receivedAt":"2020-06-19T17:57:44Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\nIf the server does support our hash, advertise it as part of our\ncapabilities.\n\nSigned-off-by: brian m. carlson \n---\n send-pack.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 0abee22283..02aefcb08e 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -363,6 +363,7 @@ int send_pack(struct send_pack_args *args,\n \tint atomic_supported = 0;\n \tint use_push_options = 0;\n \tint push_options_supported = 0;\n+\tint object_format_supported = 0;\n \tunsigned cmds_sent = 0;\n \tint ret;\n \tstruct async demux;\n@@ -389,6 +390,9 @@ int send_pack(struct send_pack_args *args,\n \tif (server_supports(\"push-options\"))\n \t\tpush_options_supported = 1;\n \n+\tif (!server_supports_hash(the_hash_algo->name, &object_format_supported))\n+\t\tdie(_(\"the receiving end does not support this repository's hash algorithm\"));\n+\n \tif (args->push_cert != SEND_PACK_PUSH_CERT_NEVER) {\n \t\tint len;\n \t\tpush_cert_nonce = server_feature_value(\"push-cert\", &len);\n@@ -429,6 +433,8 @@ int send_pack(struct send_pack_args *args,\n \t\tstrbuf_addstr(&cap_buf, \" atomic\");\n \tif (use_push_options)\n \t\tstrbuf_addstr(&cap_buf, \" push-options\");\n+\tif (object_format_supported)\n+\t\tstrbuf_addf(&cap_buf, \" object-format=%s\", the_hash_algo->name);\n \tif (agent_supported)\n \t\tstrbuf_addf(&cap_buf, \" agent=%s\", git_user_agent_sanitized());\n \n"},{"id":"400197","messageId":"20200619175601.569856-21-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 20/44] t5562: pass object-format in synthesized test data","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:37Z","receivedAt":"2020-06-19T17:57:45Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Ensure that we pass the object-format capability in the synthesized test\ndata so that this test works with algorithms other than SHA-1.\n\nIn addition, add an additional test using the old data for when we're\nusing SHA-1 so that we can be sure that we preserve backwards\ncompatibility with servers not offering the object-format capability.\n\nSigned-off-by: brian m. carlson \n---\n t/t5562-http-backend-content-length.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5562-http-backend-content-length.sh b/t/t5562-http-backend-content-length.sh\nindex 3f4ac71f83..c6ec625497 100755\n--- a/t/t5562-http-backend-content-length.sh\n+++ b/t/t5562-http-backend-content-length.sh\n@@ -46,6 +46,7 @@ ssize_b100dots() {\n }\n \n test_expect_success 'setup' '\n+\ttest_oid_init &&\n \tHTTP_CONTENT_ENCODING=\"identity\" &&\n \texport HTTP_CONTENT_ENCODING &&\n \tgit config http.receivepack true &&\n@@ -62,8 +63,8 @@ test_expect_success 'setup' '\n \ttest_copy_bytes 10 fetch_body.trunc &&\n \thash_next=$(git commit-tree -p HEAD -m next HEAD^{tree}) &&\n \t{\n-\t\tprintf \"%s %s refs/heads/newbranch\\\\0report-status\\\\n\" \\\n-\t\t\t\"$ZERO_OID\" \"$hash_next\" | packetize &&\n+\t\tprintf \"%s %s refs/heads/newbranch\\\\0report-status object-format=%s\\\\n\" \\\n+\t\t\t\"$ZERO_OID\" \"$hash_next\" \"$(test_oid algo)\" | packetize &&\n \t\tprintf 0000 &&\n \t\techo \"$hash_next\" | git pack-objects --stdout\n \t} >push_body &&\n"},{"id":"400198","messageId":"20200619175601.569856-20-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 19/44] builtin/clone: initialize hash algorithm properly","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:36Z","receivedAt":"2020-06-19T17:57:48Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When performing a clone, we don't know what hash algorithm the other end\nwill support. Currently, we don't support fetching data belonging to a\ndifferent algorithm, so we must know what algorithm the remote side is\nusing in order to properly initialize the repository. We can know that\nonly after fetching the refs, so if the remote side has any references,\nuse that information to reinitialize the repository with the correct\nhash algorithm information.\n\nSigned-off-by: brian m. carlson \n---\n builtin/clone.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 2a8e3aaaed..e3519a8355 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1220,6 +1220,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \trefs = transport_get_remote_refs(transport, &ref_prefixes);\n \n \tif (refs) {\n+\t\tint hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));\n+\n+\t\t/*\n+\t\t * Now that we know what algorithm the remote side is using,\n+\t\t * let's set ours to the same thing.\n+\t\t */\n+\t\tinitialize_repository_version(hash_algo);\n+\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n"},{"id":"400199","messageId":"20200619175601.569856-14-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 13/44] fetch-pack: detect when the server doesn't support our hash","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:30Z","receivedAt":"2020-06-19T17:57:54Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Detect when the server doesn't support our hash algorithm and abort.\n\nSigned-off-by: brian m. carlson \n---\n fetch-pack.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex d8bbf45ee2..c090030680 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1040,6 +1040,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tprint_verbose(args, _(\"Server supports %s\"), \"deepen-relative\");\n \telse if (args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n+\tif (!server_supports_hash(the_hash_algo->name, NULL))\n+\t\tdie(_(\"Server does not support this repository's object format\"));\n \n \tif (!args->no_dependents) {\n \t\tmark_complete_and_common_ref(negotiator, args, &ref);\n"},{"id":"400200","messageId":"20200619175601.569856-9-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 08/44] pkt-line: add a member for hash algorithm","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:25Z","receivedAt":"2020-06-19T17:57:56Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a member for the hash algorithm currently in use to the packet\nreader so it can parse references correctly.\n\nSigned-off-by: brian m. carlson \n---\n pkt-line.c | 1 +\n pkt-line.h | 3 +++\n 2 files changed, 4 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 8f9bc68ee2..844c253ccd 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -490,6 +490,7 @@ void packet_reader_init(struct packet_reader *reader, int fd,\n \treader->buffer_size = sizeof(packet_buffer);\n \treader->options = options;\n \treader->me = \"git\";\n+\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n }\n \n enum packet_read_status packet_reader_read(struct packet_reader *reader)\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 5b373fe4cd..8c90daa59e 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -177,6 +177,9 @@ struct packet_reader {\n \n \tunsigned use_sideband : 1;\n \tconst char *me;\n+\n+\t/* hash algorithm in use */\n+\tconst struct git_hash_algo *hash_algo;\n };\n \n /*\n"},{"id":"400201","messageId":"20200619175601.569856-13-sandals@crustytoothpaste.net","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"[PATCH v3 12/44] connect: make parse_feature_value extern","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-19T17:55:29Z","receivedAt":"2020-06-19T17:57:57Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We're going to be using this function in other files, so no longer mark\nthis function static.\n\nSigned-off-by: brian m. carlson \n---\n connect.c | 3 +--\n connect.h | 1 +\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex a52b038865..8f70a05699 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -18,7 +18,6 @@\n \n static char *server_capabilities_v1;\n static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n-static const char *parse_feature_value(const char *, const char *, int *, int *);\n static const char *next_server_feature_value(const char *feature, int *len, int *offset);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -483,7 +482,7 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n \treturn list;\n }\n \n-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n+const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset)\n {\n \tint len;\n \ndiff --git a/connect.h b/connect.h\nindex c53976f7ec..c53586e929 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -19,6 +19,7 @@ struct packet_reader;\n enum protocol_version discover_version(struct packet_reader *reader);\n \n int server_supports_hash(const char *desired, int *feature_supported);\n+const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset);\n int server_supports_v2(const char *c, int die_on_error);\n int server_feature_v2(const char *c, const char **v);\n int server_supports_feature(const char *c, const char *feature,\n"},{"id":"400224","messageId":"xmqqk102g3ua.fsf@gitster.c.googlers.com","threadId":"53458","inReplyTo":"20200619175601.569856-1-sandals@crustytoothpaste.net","subject":"Re: [PATCH v3 00/44] SHA-256 part 2/3: protocol functionality","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-06-19T21:09:33Z","receivedAt":"2020-06-19T21:09:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> This is part 2 of 3 of the SHA-256 work. This series adds all of the\n> protocol logic to work with SHA-256 repositories.\n>\n> v3 fixes a bug in patch 34 which prevented cloning an empty repository\n> with the dumb HTTP protocol. We look up the hash algorithm by length of\n> the data in the info/refs file and if we have no refs, we have no\n> entries.\n>\n> Previously, we just failed and complained, which isn't really helpful,\n> nor is it backward compatible. So now we use whatever the default is\n> for the current repository. That means we honor GIT_DEFAULT_HASH or git\n> clone -c, and default to SHA-1 otherwise. Users are encouraged to\n> switch to the smart protocol if they need to distinguish the remote\n> side's hash algorithm when the repository is empty.\n>\n> There are tests for the default hash behavior, but not for git clone -c,\n> because the extensions.objectformat option doesn't exist yet. I have\n> tested that it does indeed work, though.\n>\n> Otherwise, this series is the same as v2 except for a rebase (for my\n> convenience and Junio's).\n\nNot mine, though. Keeping the same base is easier to see the\nincremental difference.\n\nIt wasn't too cumbersome to rebase back on the same base as what was\nqueued (and the making sure the result, when merged to 'master',\nmatches the result of applying all these patches directly on top of\n'master'), though ;-)\n\nIn any case, the updated step 34 made sense to me. Thanks.\n\n"},{"id":"400248","messageId":"20200620013350.GK6531@camp.crustytoothpaste.net","threadId":"53458","inReplyTo":"xmqqk102g3ua.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 00/44] SHA-256 part 2/3: protocol functionality","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-06-20T01:33:50Z","receivedAt":"2020-06-20T01:34:29Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-06-19 at 21:09:33, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > Otherwise, this series is the same as v2 except for a rebase (for my\n> > convenience and Junio's).\n> \n> Not mine, though. Keeping the same base is easier to see the\n> incremental difference.\n\nOkay, sorry about that. It does make it more convenient for me\neventually (since I get to resolve conflicts more incrementally), but I\ndon't usually have to worry about that until the series hits master, so\nI can hold off. I'm not rebasing as many patches anymore, so it's less\nof a problem for me.\n\n> It wasn't too cumbersome to rebase back on the same base as what was\n> queued (and the making sure the result, when merged to 'master',\n> matches the result of applying all these patches directly on top of\n> 'master'), though ;-)\n> \n> In any case, the updated step 34 made sense to me. Thanks.\n\nYeah, I discovered it the other day when updating another project to\ndeal with a SHA-256 Git, and I happen to be on vacation today, so I\nthought I'd send out a quick fix. I was surprised to learn that we had\nno tests for cloning empty repositories, but here we are.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"}]}