{"thread":{"id":"57388","subject":"[PATCH] fetch —object-info-format: client option for object-info","startedAt":"2022-02-08T23:19:26Z","lastAt":"2022-09-30T23:24:24Z","messageCount":103,"participants":["Calvin Wan","Philip Oakley","John Cai","Jonathan Tan","Junio C Hamano","Taylor Blau","Josh Steadmon","Ævar Arnfjörð Bjarmason","Phillip Wood","ZheNing Hu"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"448012","messageId":"20220208231911.725273-1-calvinwan@google.com","threadId":"57388","inReplyTo":null,"subject":"[PATCH] fetch —object-info-format: client option for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-02-08T23:19:10Z","receivedAt":"2022-02-08T23:19:26Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add ‘—object-info-format’ option to fetch. This option allows\nthe client to make an object-info [1] command request to a server\nthat supports protocol v2.\n\nThe transport implementation uses vtables [2], similar to how Git\nfetches refs, to determine whether a process needs to be taken over\nbefore sending the object-info request. Different protocols\nrequire different setups for making requests.\n\n[1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n[2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.1513287544.git.jonathantanmy@google.com/\n\nHelped-by: Jonathan Tan \nSigned-off-by: Calvin Wan \n\n---\n builtin/fetch.c | 80 ++++++++++++++++++++----------\n fetch-pack.c | 53 ++++++++++++++++++++\n fetch-pack.h | 7 +++\n t/t5583-fetch-object-info.sh | 95 ++++++++++++++++++++++++++++++++++++\n transport-helper.c | 12 +++++\n transport-internal.h | 1 +\n transport.c | 63 ++++++++++++++++++++++++\n transport.h | 1 +\n 8 files changed, 286 insertions(+), 26 deletions(-)\n create mode 100755 t/t5583-fetch-object-info.sh\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5f06b21f8e..991063072b 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,9 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"protocol.h\"\n+#include \"pkt-line.h\"\n+#include \"connect.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] \"),\n \tN_(\"git fetch --multiple [] [( | )...]\"),\n \tN_(\"git fetch --all []\"),\n+\tN_(\"git fetch --object-info-format=[] []\"),\n \tNULL\n };\n \n@@ -85,6 +89,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n static int fetch_write_commit_graph = -1;\n static int stdin_refspecs = 0;\n static int negotiate_only;\n+static struct string_list object_info_format = STRING_LIST_INIT_NODUP;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -220,6 +225,8 @@ static struct option builtin_fetch_options[] = {\n \t\t N_(\"write the commit-graph after fetching\")),\n \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n \t\t N_(\"accept refspecs from stdin\")),\n+\tOPT_STRING_LIST(0, \"object-info-format\", &object_info_format, N_(\"option\"),\n+\t\t N_(\"command request arguments\")),\n \tOPT_END()\n };\n \n@@ -2000,6 +2007,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tstruct remote *remote = NULL;\n \tint result = 0;\n \tint prune_tags_ok = 1;\n+\tstruct oid_array oids = OID_ARRAY_INIT;\n+\tstruct object_id oid;\n \n \tpacket_trace_identity(\"fetch\");\n \n@@ -2019,6 +2028,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\n+\n \tif (recurse_submodules != RECURSE_SUBMODULES_OFF) {\n \t\tint *sfjc = submodule_fetch_jobs_config == -1\n \t\t\t ? &submodule_fetch_jobs_config : NULL;\n@@ -2057,34 +2067,52 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n-\tif (all) {\n-\t\tif (argc == 1)\n-\t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n-\t\telse if (argc > 1)\n-\t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n-\t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n-\t} else if (argc == 0) {\n-\t\t/* No arguments -- use default remote */\n-\t\tremote = remote_get(NULL);\n-\t} else if (multiple) {\n-\t\t/* All arguments are assumed to be remotes or groups */\n-\t\tfor (i = 0; i < argc; i++)\n-\t\t\tif (!add_remote_or_group(argv[i], &list))\n-\t\t\t\tdie(_(\"no such remote or remote group: %s\"),\n-\t\t\t\t argv[i]);\n-\t} else {\n-\t\t/* Single remote or group */\n-\t\t(void) add_remote_or_group(argv[0], &list);\n-\t\tif (list.nr > 1) {\n-\t\t\t/* More than one remote */\n-\t\t\tif (argc > 1)\n-\t\t\t\tdie(_(\"fetching a group and specifying refspecs does not make sense\"));\n+\tif (object_info_format.nr > 0) {\n+\t\tif (argc == 0 || argc == 1) {\n+\t\t\tdie(_(\"must supply remote and object ids when using --object-info-format\"));\n \t\t} else {\n-\t\t\t/* Zero or one remotes */\n \t\t\tremote = remote_get(argv[0]);\n-\t\t\tprune_tags_ok = (argc == 1);\n-\t\t\targc--;\n-\t\t\targv++;\n+\t\t\tfor (i = 1; i < argc; i++) {\n+\t\t\t\tif (get_oid(argv[i], &oid))\n+\t\t\t\t\treturn error(_(\"malformed object name '%s'\"), argv[i]);\n+\t\t\t\toid_array_append(&oids, &oid);\n+\t\t\t}\n+\t\t}\n+\t\tgtransport = prepare_transport(remote, 0);\n+\t\tgtransport->server_options = &object_info_format;\n+\t\tresult = transport_fetch_object_info(gtransport, &oids);\n+\n+\t\treturn result;\t\n+\t} else {\n+\t\tif (all) {\n+\t\t\tif (argc == 1)\n+\t\t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n+\t\t\telse if (argc > 1)\n+\t\t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n+\t\t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n+\t\t} else if (argc == 0) {\n+\t\t\t/* No arguments -- use default remote */\n+\t\t\tremote = remote_get(NULL);\n+\t\t} else if (multiple) {\n+\t\t\t/* All arguments are assumed to be remotes or groups */\n+\t\t\tfor (i = 0; i < argc; i++)\n+\t\t\t\tif (!add_remote_or_group(argv[i], &list))\n+\t\t\t\t\tdie(_(\"no such remote or remote group: %s\"),\n+\t\t\t\t\t argv[i]);\n+\t\t} else {\n+\t\t\t/* Single remote or group */\n+\t\t\t(void) add_remote_or_group(argv[0], &list);\n+\t\t\tif (list.nr > 1) {\n+\t\t\t\t/* More than one remote */\n+\t\t\t\tif (argc > 1)\n+\t\t\t\t\tdie(_(\"fetching a group and specifying refspecs does not make sense\"));\n+\t\t\t} else {\n+\t\t\t\t/* Zero or one remotes */\n+\t\t\t\tremote = remote_get(argv[0]);\n+\t\t\t\tprune_tags_ok = (argc == 1);\n+\t\t\t\targc--;\n+\t\t\t\targv++;\n+\t\t\t}\n \t\t}\n \t}\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dd6ec449f2..d1c5254aa8 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1265,6 +1265,59 @@ static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n \tpacket_buf_delim(req_buf);\n }\n \n+static void write_object_info_command_and_capabilities(struct strbuf *req_buf,\n+\t\t\t\t\t\t const struct string_list *server_options)\n+{\n+\tconst char *hash_name;\n+\n+\tif (server_supports_v2(\"object-info\", 1))\n+\t\tpacket_buf_write(req_buf, \"command=object-info\");\n+\tif (server_supports_v2(\"agent\", 0))\n+\t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n+\tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n+\t\tpacket_buf_write(req_buf, \"session-id=%s\", trace2_session_id());\n+\tif (server_options && server_options->nr &&\n+\t server_supports_v2(\"server-option\", 1)) {\n+\t\tint i;\n+\t\tfor (i = 0; i < server_options->nr; i++)\n+\t\t\tpacket_buf_write(req_buf, \"server-option=%s\",\n+\t\t\t\t\t server_options->items[i].string);\n+\t}\n+\n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n+\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n+\t\t\t the_hash_algo->name, hash_name);\n+\t\tpacket_buf_write(req_buf, \"object-format=%s\", the_hash_algo->name);\n+\t} else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n+\t\tdie(_(\"the server does not support algorithm '%s'\"),\n+\t\t the_hash_algo->name);\n+\t}\n+\tpacket_buf_delim(req_buf);\n+}\n+\n+void send_object_info_request(int fd_out, struct object_info_args *args)\n+{\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\tint i;\n+\n+\twrite_object_info_command_and_capabilities(&req_buf, args->server_options);\n+\n+\tif (string_list_has_string(args->server_options, \"size\"))\n+\t\tpacket_buf_write(&req_buf, \"size\");\n+\n+\tfor (i = 0; i < args->oids->nr; i++) {\n+\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n+\t}\n+\n+\tpacket_buf_flush(&req_buf);\n+\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n+\t\tdie_errno(_(\"unable to write request to remote\"));\n+\n+\tstrbuf_release(&req_buf);\n+}\n+\n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 7f94a2a583..2ad5ec5c64 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -68,6 +68,11 @@ struct fetch_pack_args {\n \tunsigned connectivity_checked:1;\n };\n \n+struct object_info_args {\n+\tconst struct string_list *server_options;\n+\tconst struct oid_array *oids;\n+};\n+\n /*\n * sought represents remote references that should be updated from.\n * On return, the names that were found on the remote will have been\n@@ -101,4 +106,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+void send_object_info_request(int fd_out, struct object_info_args *args);\n+\n #endif\ndiff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\nnew file mode 100755\nindex 0000000000..93b09da3b1\n--- /dev/null\n+++ b/t/t5583-fetch-object-info.sh\n@@ -0,0 +1,95 @@\n+#!/bin/sh\n+\n+test_description='test git fetch object-info version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+initial_directory=$pwd\n+\n+# Test fetch object-info with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" message1 a.txt\n+'\n+\n+test_expect_success 'fetch object-info with git:// using protocol v2' '\n+\ttest_when_finished \"cd '$initial_directory'\" &&\n+\n+\tcd \"$daemon_parent\" &&\n+\n+\tcat >actual <<-EOF &&\n+\t$(GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info-format=size \"$GIT_DAEMON_URL/parent\" $(git rev-parse message1:a.txt))\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse message1:a.txt) $(wc -c actual <<-EOF &&\n+\t$(git -c protocol.version=2 fetch --object-info-format=size \"$HTTPD_URL/smart/http_parent\" $(git rev-parse message1:a.txt))\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse message1:a.txt) $(wc -c actual <<-EOF &&\n+\t$(git fetch --object-info-format=size \"file://$(pwd)\" $(git rev-parse message1:a.txt))\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse message1:a.txt) $(wc -c vtable->fetch_object_info(transport, oids);\n+\t}\n+\treturn -1;\n+}\n+\n static int fetch_refs(struct transport *transport,\n \t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -1269,6 +1280,7 @@ static struct transport_vtable vtable = {\n \t.get_refs_list\t= get_refs_list,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n+\t.fetch_object_info = fetch_object_info,\n \t.connect\t= connect_helper,\n \t.disconnect\t= release_helper\n };\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733a..04fa015011 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -59,6 +59,7 @@ struct transport_vtable {\n \t * use. disconnect() releases these resources.\n \t **/\n \tint (*disconnect)(struct transport *connection);\n+\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n };\n \n #endif\ndiff --git a/transport.c b/transport.c\nindex 2a3e324154..0e1bf3c4cd 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -445,6 +445,62 @@ static int fetch_refs_via_pack(struct transport *transport,\n \treturn ret;\n }\n \n+static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n+{\n+\tint ret = 0;\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct object_info_args args;\n+\tstruct packet_reader reader;\n+\n+\tmemset(&args, 0, sizeof(args));\n+\targs.server_options = transport->server_options;\n+\targs.oids = oids;\n+\n+\tconnect_setup(transport, 0);\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\tPACKET_READ_CHOMP_NEWLINE |\n+\t\t\tPACKET_READ_GENTLE_ON_EOF |\n+\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n+\tdata->version = discover_version(&reader);\n+\n+\tif (data->version == protocol_unknown_version)\n+\t\tBUG(\"unknown protocol version\");\n+\telse if (data->version <= protocol_v1)\n+\t\tdie_if_server_options(transport);\n+\n+\tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tsend_object_info_request(data->fd[1], &args);\n+\t\tbreak;\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tdie(_(\"wrong protocol version. expected v2\"));\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n+\t\tdie(_(\"error reading object info header\"));\n+\t}\n+\tif (strcmp(reader.line, \"size\")) {\n+\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n+\t}\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\t\tprintf(\"%s\\n\", reader.line);\n+\t}\n+\t\n+\tclose(data->fd[0]);\n+\n+\tif (data->fd[1] >= 0) {\n+\t\tclose(data->fd[1]);\n+\t}\n+\tif (finish_connect(data->conn))\n+\t\tret = -1;\n+\tdata->conn = NULL;\n+\n+\treturn ret;\n+}\n+\n static int push_had_errors(struct ref *ref)\n {\n \tfor (; ref; ref = ref->next) {\n@@ -890,6 +946,7 @@ static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n+\t.fetch_object_info = fetch_object_info,\n \t.disconnect\t= disconnect_git\n };\n \n@@ -1043,6 +1100,7 @@ static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n+\t.fetch_object_info = fetch_object_info,\n \t.connect\t= connect_git,\n \t.disconnect\t= disconnect_git\n };\n@@ -1420,6 +1478,11 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n \treturn transport->remote_refs;\n }\n \n+int transport_fetch_object_info(struct transport *transport, struct oid_array *oids) {\n+\ttransport->vtable->fetch_object_info(transport, oids);\n+\treturn 1;\n+}\n+\n int transport_fetch_refs(struct transport *transport, struct ref *refs)\n {\n \tint rc;\ndiff --git a/transport.h b/transport.h\nindex 3f16e50c19..1c807591de 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -278,6 +278,7 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n * This can only be called after fetching the remote refs.\n */\n const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n+int transport_fetch_object_info(struct transport *transport, struct oid_array *oids);\n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n \n /*\n\nbase-commit: b23dac905bde28da47543484320db16312c87551\n-- \n2.33.0.664.g0785eb7698\n\n"},{"id":"448014","messageId":"20220208235631.732466-1-calvinwan@google.com","threadId":"57388","inReplyTo":"20220208231911.725273-1-calvinwan@google.com","subject":"[PATCH v2] fetch —object-info-format: client option for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-02-08T23:56:30Z","receivedAt":"2022-02-08T23:56:39Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add ‘—object-info-format’ option to fetch. This option allows\nthe client to make an object-info [1] command request to a server\nthat supports protocol v2.\n\nThe transport implementation uses vtables [2], similar to how Git\nfetches refs, to determine whether a process needs to be taken over\nbefore sending the object-info request. Different protocols\nrequire different setups for making requests.\n\n[1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n[2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.15=\n13287544.git.jonathantanmy@google.com/\n\nHelped-by: Jonathan Tan \nSigned-off-by: Calvin Wan \n\n---\nPlease ignore the patch above. It was sent with a stale patch message.\n builtin/fetch.c | 26 ++++++++++++\n fetch-pack.c | 53 +++++++++++++++++++++++\n fetch-pack.h | 7 ++++\n t/t5583-fetch-object-info.sh | 81 ++++++++++++++++++++++++++++++++++++\n transport-helper.c | 12 ++++++\n transport-internal.h | 1 +\n transport.c | 59 ++++++++++++++++++++++++++\n transport.h | 1 +\n 8 files changed, 240 insertions(+)\n create mode 100755 t/t5583-fetch-object-info.sh\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5f06b21f8e..b48d9e93d0 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,9 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"protocol.h\"\n+#include \"pkt-line.h\"\n+#include \"connect.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] \"),\n \tN_(\"git fetch --multiple [] [( | )...]\"),\n \tN_(\"git fetch --all []\"),\n+\tN_(\"git fetch --object-info-format=[] []\"),\n \tNULL\n };\n \n@@ -85,6 +89,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n static int fetch_write_commit_graph = -1;\n static int stdin_refspecs = 0;\n static int negotiate_only;\n+static struct string_list object_info_format = STRING_LIST_INIT_NODUP;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -220,6 +225,8 @@ static struct option builtin_fetch_options[] = {\n \t\t N_(\"write the commit-graph after fetching\")),\n \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n \t\t N_(\"accept refspecs from stdin\")),\n+\tOPT_STRING_LIST(0, \"object-info-format\", &object_info_format, N_(\"option\"),\n+\t\t N_(\"command request arguments\")),\n \tOPT_END()\n };\n \n@@ -2000,6 +2007,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tstruct remote *remote = NULL;\n \tint result = 0;\n \tint prune_tags_ok = 1;\n+\tstruct oid_array oids = OID_ARRAY_INIT;\n+\tstruct object_id oid;\n \n \tpacket_trace_identity(\"fetch\");\n \n@@ -2057,6 +2066,23 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n+\tif (object_info_format.nr > 0) {\n+\t\tif (argc == 0 || argc == 1) {\n+\t\t\tdie(_(\"must supply remote and object ids when using --object-info-format\"));\n+\t\t} else {\n+\t\t\tremote = remote_get(argv[0]);\n+\t\t\tfor (i = 1; i < argc; i++) {\n+\t\t\t\tif (get_oid(argv[i], &oid))\n+\t\t\t\t\treturn error(_(\"malformed object name '%s'\"), argv[i]);\n+\t\t\t\toid_array_append(&oids, &oid);\n+\t\t\t}\n+\t\t}\n+\t\tgtransport = prepare_transport(remote, 0);\n+\t\tgtransport->server_options = &object_info_format;\n+\t\tresult = transport_fetch_object_info(gtransport, &oids);\n+\t\treturn result;\n+\t}\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dd6ec449f2..d1c5254aa8 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1265,6 +1265,59 @@ static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n \tpacket_buf_delim(req_buf);\n }\n \n+static void write_object_info_command_and_capabilities(struct strbuf *req_buf,\n+\t\t\t\t\t\t const struct string_list *server_options)\n+{\n+\tconst char *hash_name;\n+\n+\tif (server_supports_v2(\"object-info\", 1))\n+\t\tpacket_buf_write(req_buf, \"command=object-info\");\n+\tif (server_supports_v2(\"agent\", 0))\n+\t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n+\tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n+\t\tpacket_buf_write(req_buf, \"session-id=%s\", trace2_session_id());\n+\tif (server_options && server_options->nr &&\n+\t server_supports_v2(\"server-option\", 1)) {\n+\t\tint i;\n+\t\tfor (i = 0; i < server_options->nr; i++)\n+\t\t\tpacket_buf_write(req_buf, \"server-option=%s\",\n+\t\t\t\t\t server_options->items[i].string);\n+\t}\n+\n+\tif (server_feature_v2(\"object-format\", &hash_name)) {\n+\t\tint hash_algo = hash_algo_by_name(hash_name);\n+\t\tif (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n+\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n+\t\t\t the_hash_algo->name, hash_name);\n+\t\tpacket_buf_write(req_buf, \"object-format=%s\", the_hash_algo->name);\n+\t} else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n+\t\tdie(_(\"the server does not support algorithm '%s'\"),\n+\t\t the_hash_algo->name);\n+\t}\n+\tpacket_buf_delim(req_buf);\n+}\n+\n+void send_object_info_request(int fd_out, struct object_info_args *args)\n+{\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\tint i;\n+\n+\twrite_object_info_command_and_capabilities(&req_buf, args->server_options);\n+\n+\tif (string_list_has_string(args->server_options, \"size\"))\n+\t\tpacket_buf_write(&req_buf, \"size\");\n+\n+\tfor (i = 0; i < args->oids->nr; i++) {\n+\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n+\t}\n+\n+\tpacket_buf_flush(&req_buf);\n+\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n+\t\tdie_errno(_(\"unable to write request to remote\"));\n+\n+\tstrbuf_release(&req_buf);\n+}\n+\n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 7f94a2a583..2ad5ec5c64 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -68,6 +68,11 @@ struct fetch_pack_args {\n \tunsigned connectivity_checked:1;\n };\n \n+struct object_info_args {\n+\tconst struct string_list *server_options;\n+\tconst struct oid_array *oids;\n+};\n+\n /*\n * sought represents remote references that should be updated from.\n * On return, the names that were found on the remote will have been\n@@ -101,4 +106,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+void send_object_info_request(int fd_out, struct object_info_args *args);\n+\n #endif\ndiff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\nnew file mode 100755\nindex 0000000000..942426b3ca\n--- /dev/null\n+++ b/t/t5583-fetch-object-info.sh\n@@ -0,0 +1,81 @@\n+#!/bin/sh\n+\n+test_description='test git fetch object-info version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test fetch object-info with 'git://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" message1 a.txt\n+'\n+\n+test_expect_success 'fetch object-info with git:// using protocol v2' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit -c protocol.version=2 fetch --object-info-format=size \"$GIT_DAEMON_URL/parent\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+stop_git_daemon\n+\n+# Test protocol v2 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+ \n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message1 a.txt\n+'\n+\n+test_expect_success 'fetch object-info with http:// using protocol v2' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit -c protocol.version=2 fetch --object-info-format=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+# Test fetch object-info with 'file://' transport\n+#\n+\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init server &&\n+\ttest_commit -C server message1 a.txt &&\n+\tgit -C server config protocol.version 2\n+'\n+\n+test_expect_success 'fetch object-info with file:// using protocol v2' '\n+\t(\n+\t\tcd server &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit fetch --object-info-format=size \"file://$(pwd)\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex a0297b0986..9ecda03dde 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -671,6 +671,17 @@ static int connect_helper(struct transport *transport, const char *name,\n static struct ref *get_refs_list_using_list(struct transport *transport,\n \t\t\t\t\t int for_push);\n \n+static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, 0)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->fetch_object_info(transport, oids);\n+\t}\n+\treturn -1;\n+}\n+\n static int fetch_refs(struct transport *transport,\n \t\t int nr_heads, struct ref **to_fetch)\n {\n@@ -1269,6 +1280,7 @@ static struct transport_vtable vtable = {\n \t.get_refs_list\t= get_refs_list,\n \t.fetch_refs\t= fetch_refs,\n \t.push_refs\t= push_refs,\n+\t.fetch_object_info = fetch_object_info,\n \t.connect\t= connect_helper,\n \t.disconnect\t= release_helper\n };\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733a..04fa015011 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -59,6 +59,7 @@ struct transport_vtable {\n \t * use. disconnect() releases these resources.\n \t **/\n \tint (*disconnect)(struct transport *connection);\n+\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n };\n \n #endif\ndiff --git a/transport.c b/transport.c\nindex 2a3e324154..87cd14c99f 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -445,6 +445,59 @@ static int fetch_refs_via_pack(struct transport *transport,\n \treturn ret;\n }\n \n+static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n+{\n+\tint ret = 0;\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct object_info_args args;\n+\tstruct packet_reader reader;\n+\n+\tmemset(&args, 0, sizeof(args));\n+\targs.server_options = transport->server_options;\n+\targs.oids = oids;\n+\n+\tconnect_setup(transport, 0);\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\tPACKET_READ_CHOMP_NEWLINE |\n+\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n+\tdata->version = discover_version(&reader);\n+\n+\tif (data->version == protocol_unknown_version)\n+\t\tBUG(\"unknown protocol version\");\n+\telse if (data->version <= protocol_v1)\n+\t\tdie_if_server_options(transport);\n+\n+\tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tsend_object_info_request(data->fd[1], &args);\n+\t\tbreak;\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tdie(_(\"wrong protocol version. expected v2\"));\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n+\t\tdie(_(\"error reading object info header\"));\n+\t}\n+\tif (strcmp(reader.line, \"size\")) {\n+\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n+\t}\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\t\tprintf(\"%s\\n\", reader.line);\n+\t}\n+\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\tclose(data->fd[0]);\n+\tif (data->fd[1] >= 0)\n+\t\tclose(data->fd[1]);\n+\tif (finish_connect(data->conn))\n+\t\tret = -1;\n+\tdata->conn = NULL;\n+\n+\treturn ret;\n+}\n+\n static int push_had_errors(struct ref *ref)\n {\n \tfor (; ref; ref = ref->next) {\n@@ -890,6 +943,7 @@ static struct transport_vtable taken_over_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n+\t.fetch_object_info = fetch_object_info,\n \t.disconnect\t= disconnect_git\n };\n \n@@ -1043,6 +1097,7 @@ static struct transport_vtable builtin_smart_vtable = {\n \t.get_refs_list\t= get_refs_via_connect,\n \t.fetch_refs\t= fetch_refs_via_pack,\n \t.push_refs\t= git_transport_push,\n+\t.fetch_object_info = fetch_object_info,\n \t.connect\t= connect_git,\n \t.disconnect\t= disconnect_git\n };\n@@ -1420,6 +1475,10 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n \treturn transport->remote_refs;\n }\n \n+int transport_fetch_object_info(struct transport *transport, struct oid_array *oids) {\n+\treturn transport->vtable->fetch_object_info(transport, oids);\n+}\n+\n int transport_fetch_refs(struct transport *transport, struct ref *refs)\n {\n \tint rc;\ndiff --git a/transport.h b/transport.h\nindex 3f16e50c19..1c807591de 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -278,6 +278,7 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n * This can only be called after fetching the remote refs.\n */\n const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n+int transport_fetch_object_info(struct transport *transport, struct oid_array *oids);\n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n \n /*\n\nbase-commit: b23dac905bde28da47543484320db16312c87551\n-- \n2.33.0.664.g0785eb7698\n\n"},{"id":"448045","messageId":"e21c4f77-061c-d125-97cb-1a8cb7692e56@iee.email","threadId":"57388","inReplyTo":"20220208235631.732466-1-calvinwan@google.com","subject":"Re: [PATCH v2] fetch —object-info-format: client option for object-info","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2022-02-09T12:48:15Z","receivedAt":"2022-02-09T12:48:23Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 08/02/2022 23:56, Calvin Wan wrote:\n> Add ‘—object-info-format’ option to fetch. This option allows\n> the client to make an object-info [1] command request to a server\n> that supports protocol v2.\n>\n> The transport implementation uses vtables [2], similar to how Git\n> fetches refs, to determine whether a process needs to be taken over\n> before sending the object-info request. Different protocols\n> require different setups for making requests.\n>\n> [1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n> [2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.15=\n> 13287544.git.jonathantanmy@google.com/\n>\n> Helped-by: Jonathan Tan \n> Signed-off-by: Calvin Wan \n>\n> ---\n> Please ignore the patch above. It was sent with a stale patch message.\n> builtin/fetch.c | 26 ++++++++++++\n> fetch-pack.c | 53 +++++++++++++++++++++++\n> fetch-pack.h | 7 ++++\n> t/t5583-fetch-object-info.sh | 81 ++++++++++++++++++++++++++++++++++++\n> transport-helper.c | 12 ++++++\n> transport-internal.h | 1 +\n> transport.c | 59 ++++++++++++++++++++++++++\n> transport.h | 1 +\n> 8 files changed, 240 insertions(+)\n> create mode 100755 t/t5583-fetch-object-info.sh\n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 5f06b21f8e..b48d9e93d0 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -29,6 +29,9 @@\n> #include \"commit-graph.h\"\n> #include \"shallow.h\"\n> #include \"worktree.h\"\n> +#include \"protocol.h\"\n> +#include \"pkt-line.h\"\n> +#include \"connect.h\"\n> \n> #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n> \n> @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> \tN_(\"git fetch [] \"),\n> \tN_(\"git fetch --multiple [] [( | )...]\"),\n> \tN_(\"git fetch --all []\"),\n> +\tN_(\"git fetch --object-info-format=[] []\"),\n> \tNULL\n> };\n\nDoesn't this also need a matching Documentarian update for the option?\n--\nPhilip\n"},{"id":"448068","messageId":"2B98F5EC-5F69-4DE1-89BF-33FCD1B03573@gmail.com","threadId":"57388","inReplyTo":"20220208231911.725273-1-calvinwan@google.com","subject":"Re: [PATCH] fetch —object-info-format: client option for object-info","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-02-09T19:09:48Z","receivedAt":"2022-02-09T19:10:06Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"Hi Calvin,\n\nOn 8 Feb 2022, at 18:19, Calvin Wan wrote:\n\n> Add ‘—object-info-format’ option to fetch. This option allows\n> the client to make an object-info [1] command request to a server\n> that supports protocol v2.\n>\n> The transport implementation uses vtables [2], similar to how Git\n> fetches refs, to determine whether a process needs to be taken over\n> before sending the object-info request. Different protocols\n> require different setups for making requests.\n>\n> [1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n> [2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.1513287544.git.jonathantanmy@google.com/\n>\n> Helped-by: Jonathan Tan \n> Signed-off-by: Calvin Wan \n>\n> ---\n> builtin/fetch.c | 80 ++++++++++++++++++++----------\n> fetch-pack.c | 53 ++++++++++++++++++++\n> fetch-pack.h | 7 +++\n> t/t5583-fetch-object-info.sh | 95 ++++++++++++++++++++++++++++++++++++\n> transport-helper.c | 12 +++++\n> transport-internal.h | 1 +\n> transport.c | 63 ++++++++++++++++++++++++\n> transport.h | 1 +\n> 8 files changed, 286 insertions(+), 26 deletions(-)\n> create mode 100755 t/t5583-fetch-object-info.sh\n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 5f06b21f8e..991063072b 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -29,6 +29,9 @@\n> #include \"commit-graph.h\"\n> #include \"shallow.h\"\n> #include \"worktree.h\"\n> +#include \"protocol.h\"\n> +#include \"pkt-line.h\"\n> +#include \"connect.h\"\n>\n> #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n>\n> @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> \tN_(\"git fetch [] \"),\n> \tN_(\"git fetch --multiple [] [( | )...]\"),\n> \tN_(\"git fetch --all []\"),\n> +\tN_(\"git fetch --object-info-format=[] []\"),\n\n--object-info-format seems to be more like a command, rather than a formatting\noption. When a flag ends with \"format\", it feels like it is specifying how to\ntransform the output kind of like git for-each-ref --format. So --object-info-format looks\nfeels like it's meant to transform the object-info output. Maybe just --object-info\nand then provide a different flag for the perhaps?\n\nWould be interested in hearing what others think from a UI perspective.\n\n> \tNULL\n> };\n>\n> @@ -85,6 +89,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n> static int fetch_write_commit_graph = -1;\n> static int stdin_refspecs = 0;\n> static int negotiate_only;\n> +static struct string_list object_info_format = STRING_LIST_INIT_NODUP;\n\nshould object_info_format be cleared at some point?\n\n>\n> static int git_fetch_config(const char *k, const char *v, void *cb)\n> {\n> @@ -220,6 +225,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t N_(\"write the commit-graph after fetching\")),\n> \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> \t\t N_(\"accept refspecs from stdin\")),\n> +\tOPT_STRING_LIST(0, \"object-info-format\", &object_info_format, N_(\"option\"),\n> +\t\t N_(\"command request arguments\")),\n> \tOPT_END()\n> };\n>\n> @@ -2000,6 +2007,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> \tint prune_tags_ok = 1;\n> +\tstruct oid_array oids = OID_ARRAY_INIT;\n> +\tstruct object_id oid;\n>\n> \tpacket_trace_identity(\"fetch\");\n>\n> @@ -2019,6 +2028,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n>\n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\n> +\n> \tif (recurse_submodules != RECURSE_SUBMODULES_OFF) {\n> \t\tint *sfjc = submodule_fetch_jobs_config == -1\n> \t\t\t ? &submodule_fetch_jobs_config : NULL;\n> @@ -2057,34 +2067,52 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n>\n> -\tif (all) {\n> -\t\tif (argc == 1)\n> -\t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> -\t\telse if (argc > 1)\n> -\t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n> -\t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n> -\t} else if (argc == 0) {\n> -\t\t/* No arguments -- use default remote */\n> -\t\tremote = remote_get(NULL);\n> -\t} else if (multiple) {\n> -\t\t/* All arguments are assumed to be remotes or groups */\n> -\t\tfor (i = 0; i < argc; i++)\n> -\t\t\tif (!add_remote_or_group(argv[i], &list))\n> -\t\t\t\tdie(_(\"no such remote or remote group: %s\"),\n> -\t\t\t\t argv[i]);\n> -\t} else {\n> -\t\t/* Single remote or group */\n> -\t\t(void) add_remote_or_group(argv[0], &list);\n> -\t\tif (list.nr > 1) {\n> -\t\t\t/* More than one remote */\n> -\t\t\tif (argc > 1)\n> -\t\t\t\tdie(_(\"fetching a group and specifying refspecs does not make sense\"));\n> +\tif (object_info_format.nr > 0) {\n> +\t\tif (argc == 0 || argc == 1) {\n> +\t\t\tdie(_(\"must supply remote and object ids when using --object-info-format\"));\n> \t\t} else {\n> -\t\t\t/* Zero or one remotes */\n> \t\t\tremote = remote_get(argv[0]);\n> -\t\t\tprune_tags_ok = (argc == 1);\n> -\t\t\targc--;\n> -\t\t\targv++;\n> +\t\t\tfor (i = 1; i < argc; i++) {\n> +\t\t\t\tif (get_oid(argv[i], &oid))\n> +\t\t\t\t\treturn error(_(\"malformed object name '%s'\"), argv[i]);\n> +\t\t\t\toid_array_append(&oids, &oid);\n> +\t\t\t}\n> +\t\t}\n> +\t\tgtransport = prepare_transport(remote, 0);\n> +\t\tgtransport->server_options = &object_info_format;\n> +\t\tresult = transport_fetch_object_info(gtransport, &oids);\n> +\n> +\t\treturn result;\t\n> +\t} else {\n> +\t\tif (all) {\n> +\t\t\tif (argc == 1)\n> +\t\t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> +\t\t\telse if (argc > 1)\n> +\t\t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n> +\t\t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n> +\t\t} else if (argc == 0) {\n> +\t\t\t/* No arguments -- use default remote */\n> +\t\t\tremote = remote_get(NULL);\n> +\t\t} else if (multiple) {\n> +\t\t\t/* All arguments are assumed to be remotes or groups */\n> +\t\t\tfor (i = 0; i < argc; i++)\n> +\t\t\t\tif (!add_remote_or_group(argv[i], &list))\n> +\t\t\t\t\tdie(_(\"no such remote or remote group: %s\"),\n> +\t\t\t\t\t argv[i]);\n> +\t\t} else {\n> +\t\t\t/* Single remote or group */\n> +\t\t\t(void) add_remote_or_group(argv[0], &list);\n> +\t\t\tif (list.nr > 1) {\n> +\t\t\t\t/* More than one remote */\n> +\t\t\t\tif (argc > 1)\n> +\t\t\t\t\tdie(_(\"fetching a group and specifying refspecs does not make sense\"));\n> +\t\t\t} else {\n> +\t\t\t\t/* Zero or one remotes */\n> +\t\t\t\tremote = remote_get(argv[0]);\n> +\t\t\t\tprune_tags_ok = (argc == 1);\n> +\t\t\t\targc--;\n> +\t\t\t\targv++;\n> +\t\t\t}\n> \t\t}\n> \t}\n>\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index dd6ec449f2..d1c5254aa8 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1265,6 +1265,59 @@ static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n> \tpacket_buf_delim(req_buf);\n> }\n>\n> +static void write_object_info_command_and_capabilities(struct strbuf *req_buf,\n> +\t\t\t\t\t\t const struct string_list *server_options)\n> +{\n> +\tconst char *hash_name;\n> +\n> +\tif (server_supports_v2(\"object-info\", 1))\n> +\t\tpacket_buf_write(req_buf, \"command=object-info\");\n> +\tif (server_supports_v2(\"agent\", 0))\n> +\t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n> +\tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n> +\t\tpacket_buf_write(req_buf, \"session-id=%s\", trace2_session_id());\n> +\tif (server_options && server_options->nr &&\n> +\t server_supports_v2(\"server-option\", 1)) {\n> +\t\tint i;\n> +\t\tfor (i = 0; i < server_options->nr; i++)\n> +\t\t\tpacket_buf_write(req_buf, \"server-option=%s\",\n> +\t\t\t\t\t server_options->items[i].string);\n> +\t}\n> +\n> +\tif (server_feature_v2(\"object-format\", &hash_name)) {\n> +\t\tint hash_algo = hash_algo_by_name(hash_name);\n> +\t\tif (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n> +\t\t\tdie(_(\"mismatched algorithms: client %s; server %s\"),\n> +\t\t\t the_hash_algo->name, hash_name);\n> +\t\tpacket_buf_write(req_buf, \"object-format=%s\", the_hash_algo->name);\n> +\t} else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n> +\t\tdie(_(\"the server does not support algorithm '%s'\"),\n> +\t\t the_hash_algo->name);\n> +\t}\n> +\tpacket_buf_delim(req_buf);\n> +}\n> +\n> +void send_object_info_request(int fd_out, struct object_info_args *args)\n> +{\n> +\tstruct strbuf req_buf = STRBUF_INIT;\n> +\tint i;\n> +\n> +\twrite_object_info_command_and_capabilities(&req_buf, args->server_options);\n> +\n> +\tif (string_list_has_string(args->server_options, \"size\"))\n> +\t\tpacket_buf_write(&req_buf, \"size\");\n> +\n> +\tfor (i = 0; i < args->oids->nr; i++) {\n> +\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> +\t}\n> +\n> +\tpacket_buf_flush(&req_buf);\n> +\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n> +\t\tdie_errno(_(\"unable to write request to remote\"));\n> +\n> +\tstrbuf_release(&req_buf);\n> +}\n> +\n> static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> \t\t\t struct fetch_pack_args *args,\n> \t\t\t const struct ref *wants, struct oidset *common,\n> diff --git a/fetch-pack.h b/fetch-pack.h\n> index 7f94a2a583..2ad5ec5c64 100644\n> --- a/fetch-pack.h\n> +++ b/fetch-pack.h\n> @@ -68,6 +68,11 @@ struct fetch_pack_args {\n> \tunsigned connectivity_checked:1;\n> };\n>\n> +struct object_info_args {\n> +\tconst struct string_list *server_options;\n> +\tconst struct oid_array *oids;\n> +};\n> +\n> /*\n> * sought represents remote references that should be updated from.\n> * On return, the names that were found on the remote will have been\n> @@ -101,4 +106,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n> */\n> int report_unmatched_refs(struct ref **sought, int nr_sought);\n>\n> +void send_object_info_request(int fd_out, struct object_info_args *args);\n> +\n> #endif\n> diff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\n> new file mode 100755\n> index 0000000000..93b09da3b1\n> --- /dev/null\n> +++ b/t/t5583-fetch-object-info.sh\n> @@ -0,0 +1,95 @@\n> +#!/bin/sh\n> +\n> +test_description='test git fetch object-info version 2'\n> +\n> +TEST_NO_CREATE_REPO=1\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +\n> +. ./test-lib.sh\n> +\n> +initial_directory=$pwd\n> +\n> +# Test fetch object-info with 'git://' transport\n> +#\n> +. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n> +start_git_daemon --export-all --enable=receive-pack\n> +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n> +\n> +\n> +test_expect_success 'create repo to be served by git-daemon' '\n> +\tgit init \"$daemon_parent\" &&\n> +\ttest_commit -C \"$daemon_parent\" message1 a.txt\n> +'\n> +\n> +test_expect_success 'fetch object-info with git:// using protocol v2' '\n> +\ttest_when_finished \"cd '$initial_directory'\" &&\n> +\n> +\tcd \"$daemon_parent\" &&\n> +\n> +\tcat >actual <<-EOF &&\n> +\t$(GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info-format=size \"$GIT_DAEMON_URL/parent\" $(git rev-parse message1:a.txt))\n> +\tEOF\n> +\n> +\tcat >expect <<-EOF &&\n> +\t$(git rev-parse message1:a.txt) $(wc -c +\tEOF\n> +\n> +\ttest_cmp expect actual\n> +'\n> +stop_git_daemon\n> +\n> +# Test protocol v2 with 'http://' transport\n> +#\n> +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> +start_httpd\n> +\n> +test_expect_success 'create repo to be served by http:// transport' '\n> +\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n> +\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message1 a.txt\n> +'\n> +\n> +test_expect_success 'fetch object-info with http:// using protocol v2' '\n> +\ttest_when_finished \"cd '$initial_directory'\" &&\n> +\n> +\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\n> +\tcat >actual <<-EOF &&\n> +\t$(git -c protocol.version=2 fetch --object-info-format=size \"$HTTPD_URL/smart/http_parent\" $(git rev-parse message1:a.txt))\n> +\tEOF\n> +\n> +\tcat >expect <<-EOF &&\n> +\t$(git rev-parse message1:a.txt) $(wc -c +\tEOF\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +# Test fetch object-info with 'file://' transport\n> +#\n> +\n> +test_expect_success 'create repo to be serbed by file:// transport' '\n> +\tgit init server &&\n> +\ttest_commit -C server message1 a.txt &&\n> +\tgit -C server config protocol.version 2\n> +'\n> +\n> +test_expect_success 'fetch object-info with file:// using protocol v2' '\n> +\ttest_when_finished \"cd '$initial_directory'\" &&\n> +\n> +\tcd server &&\n> +\n> +\tcat >actual <<-EOF &&\n> +\t$(git fetch --object-info-format=size \"file://$(pwd)\" $(git rev-parse message1:a.txt))\n> +\tEOF\n> +\n> +\tcat >expect <<-EOF &&\n> +\t$(git rev-parse message1:a.txt) $(wc -c +\tEOF\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n> \\ No newline at end of file\n> diff --git a/transport-helper.c b/transport-helper.c\n> index a0297b0986..9ecda03dde 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -671,6 +671,17 @@ static int connect_helper(struct transport *transport, const char *name,\n> static struct ref *get_refs_list_using_list(struct transport *transport,\n> \t\t\t\t\t int for_push);\n>\n> +static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n> +{\n> +\tget_helper(transport);\n> +\n> +\tif (process_connect(transport, 0)) {\n> +\t\tdo_take_over(transport);\n> +\t\treturn transport->vtable->fetch_object_info(transport, oids);\n> +\t}\n> +\treturn -1;\n> +}\n> +\n> static int fetch_refs(struct transport *transport,\n> \t\t int nr_heads, struct ref **to_fetch)\n> {\n> @@ -1269,6 +1280,7 @@ static struct transport_vtable vtable = {\n> \t.get_refs_list\t= get_refs_list,\n> \t.fetch_refs\t= fetch_refs,\n> \t.push_refs\t= push_refs,\n> +\t.fetch_object_info = fetch_object_info,\n> \t.connect\t= connect_helper,\n> \t.disconnect\t= release_helper\n> };\n> diff --git a/transport-internal.h b/transport-internal.h\n> index c4ca0b733a..04fa015011 100644\n> --- a/transport-internal.h\n> +++ b/transport-internal.h\n> @@ -59,6 +59,7 @@ struct transport_vtable {\n> \t * use. disconnect() releases these resources.\n> \t **/\n> \tint (*disconnect)(struct transport *connection);\n> +\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n> };\n>\n> #endif\n> diff --git a/transport.c b/transport.c\n> index 2a3e324154..0e1bf3c4cd 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -445,6 +445,62 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \treturn ret;\n> }\n>\n> +static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n> +{\n> +\tint ret = 0;\n> +\tstruct git_transport_data *data = transport->data;\n> +\tstruct object_info_args args;\n> +\tstruct packet_reader reader;\n> +\n> +\tmemset(&args, 0, sizeof(args));\n> +\targs.server_options = transport->server_options;\n> +\targs.oids = oids;\n> +\n> +\tconnect_setup(transport, 0);\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\tPACKET_READ_CHOMP_NEWLINE |\n> +\t\t\tPACKET_READ_GENTLE_ON_EOF |\n> +\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n> +\tdata->version = discover_version(&reader);\n> +\n> +\tif (data->version == protocol_unknown_version)\n> +\t\tBUG(\"unknown protocol version\");\n> +\telse if (data->version <= protocol_v1)\n> +\t\tdie_if_server_options(transport);\n> +\n> +\tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tsend_object_info_request(data->fd[1], &args);\n> +\t\tbreak;\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tdie(_(\"wrong protocol version. expected v2\"));\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> +\n> +\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> +\t\tdie(_(\"error reading object info header\"));\n> +\t}\n> +\tif (strcmp(reader.line, \"size\")) {\n> +\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n> +\t}\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> +\t\tprintf(\"%s\\n\", reader.line);\n> +\t}\n> +\t\n> +\tclose(data->fd[0]);\n> +\n> +\tif (data->fd[1] >= 0) {\n> +\t\tclose(data->fd[1]);\n> +\t}\n> +\tif (finish_connect(data->conn))\n> +\t\tret = -1;\n> +\tdata->conn = NULL;\n> +\n> +\treturn ret;\n> +}\n> +\n> static int push_had_errors(struct ref *ref)\n> {\n> \tfor (; ref; ref = ref->next) {\n> @@ -890,6 +946,7 @@ static struct transport_vtable taken_over_vtable = {\n> \t.get_refs_list\t= get_refs_via_connect,\n> \t.fetch_refs\t= fetch_refs_via_pack,\n> \t.push_refs\t= git_transport_push,\n> +\t.fetch_object_info = fetch_object_info,\n> \t.disconnect\t= disconnect_git\n> };\n>\n> @@ -1043,6 +1100,7 @@ static struct transport_vtable builtin_smart_vtable = {\n> \t.get_refs_list\t= get_refs_via_connect,\n> \t.fetch_refs\t= fetch_refs_via_pack,\n> \t.push_refs\t= git_transport_push,\n> +\t.fetch_object_info = fetch_object_info,\n> \t.connect\t= connect_git,\n> \t.disconnect\t= disconnect_git\n> };\n> @@ -1420,6 +1478,11 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n> \treturn transport->remote_refs;\n> }\n>\n> +int transport_fetch_object_info(struct transport *transport, struct oid_array *oids) {\n> +\ttransport->vtable->fetch_object_info(transport, oids);\n> +\treturn 1;\n> +}\n> +\n> int transport_fetch_refs(struct transport *transport, struct ref *refs)\n> {\n> \tint rc;\n> diff --git a/transport.h b/transport.h\n> index 3f16e50c19..1c807591de 100644\n> --- a/transport.h\n> +++ b/transport.h\n> @@ -278,6 +278,7 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n> * This can only be called after fetching the remote refs.\n> */\n> const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n> +int transport_fetch_object_info(struct transport *transport, struct oid_array *oids);\n> int transport_fetch_refs(struct transport *transport, struct ref *refs);\n>\n> /*\n>\n> base-commit: b23dac905bde28da47543484320db16312c87551\n> -- \n> 2.33.0.664.g0785eb7698\n"},{"id":"448072","messageId":"20220209204153.481122-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220208235631.732466-1-calvinwan@google.com","subject":"Re: [PATCH v2] fetch object-info-format: client option for object-info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-02-09T20:41:53Z","receivedAt":"2022-02-09T20:42:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Calvin Wan writes:\n> Add ‘—object-info-format’ option to fetch. This option allows\n> the client to make an object-info [1] command request to a server\n> that supports protocol v2.\n\nAvoid using characters above 127 in commit messages (unless, say, as part of\nsomeone's name). They make it hard to search for, and in this case, whatever\ndash is there is wrong (it should be \"--\").\n\n> The transport implementation uses vtables [2], similar to how Git\n> fetches refs,\n\nYou should be explicit that you're adding a new function to the vtable.\n(Whether that is what we should do is another issue: let me look at the\npatch to see.)\n\n> to determine whether a process needs to be taken over\n> before sending the object-info request. \n\nThe vtable is not to determine whether a process needs to be taken over,\nbut so that we support multiple protocols (HTTP, SSH, etc.). In any\ncase, this detail is probably not relevant.\n\n> Different protocols\n> require different setups for making requests.\n\nThis is true, but I don't see the relevance.\n\n> [1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n> [2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.15=\n> 13287544.git.jonathantanmy@google.com/\n\nFor merged code, quote the commit, not the email.\n\n> @@ -220,6 +225,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t N_(\"write the commit-graph after fetching\")),\n> \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> \t\t N_(\"accept refspecs from stdin\")),\n> +\tOPT_STRING_LIST(0, \"object-info-format\", &object_info_format, N_(\"option\"),\n> +\t\t N_(\"command request arguments\")),\n\nI would have expected a parameter named \"format\" to take a format\nstring, but taking a string list of the fields we need might work too.\nIn any case, maybe rename it to \"--object-info\" or similar.\n\n> @@ -2000,6 +2007,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> \tint prune_tags_ok = 1;\n> +\tstruct oid_array oids = OID_ARRAY_INIT;\n> +\tstruct object_id oid;\n\nThe \"oids\" at function level needs a more descriptive name (e.g.\n\"oids_for_object_info\"). The name of \"oid\" is fine, since it's just used\nas a temporary variable, but since it is temporary, it should be\ndeclared in the block where it's used. (Same for \"oids\", actually:\ndeclare it in the block it's used, and in that case you can keep the\nname since it's more tightly scoped.)\n\n> @@ -2057,6 +2066,23 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n> \n> +\tif (object_info_format.nr > 0) {\n> +\t\tif (argc == 0 || argc == 1) {\n> +\t\t\tdie(_(\"must supply remote and object ids when using --object-info-format\"));\n> +\t\t} else {\n> +\t\t\tremote = remote_get(argv[0]);\n> +\t\t\tfor (i = 1; i < argc; i++) {\n> +\t\t\t\tif (get_oid(argv[i], &oid))\n> +\t\t\t\t\treturn error(_(\"malformed object name '%s'\"), argv[i]);\n> +\t\t\t\toid_array_append(&oids, &oid);\n> +\t\t\t}\n> +\t\t}\n> +\t\tgtransport = prepare_transport(remote, 0);\n> +\t\tgtransport->server_options = &object_info_format;\n> +\t\tresult = transport_fetch_object_info(gtransport, &oids);\n> +\t\treturn result;\n> +\t}\n\nI was thinking that this should reuse the refspec parsing mechanism\n(which also supports stdin), but upon more thought, using the refspec\nparser means that we would also need to check that all refspecs are\nexact OIDs (because we wouldn't know what to do with them otherwise).\nOK, parsing the objects by ourselves looks reasonable.\n\nThe assignment of object_info_format to server_options is probably not a\ngood idea, though, since readers of server_options would expect server\noptions, not what you're assigning. The best place to put this\ninformation is in smart_options. (See the negotiate_only code.)\n\n> +static void write_object_info_command_and_capabilities(struct strbuf *req_buf,\n> +\t\t\t\t\t\t const struct string_list *server_options)\n> +{\n\n[snip contents]\n\nThis code is very similar to code in fetch-pack.c. If you stick to\ncrafting the request in builtin/fetch.c, you should refactor\nfetch-pack.{c,h} to expose this functionality (in a preparatory commit)\nand then use that function from here.\n\n> +void send_object_info_request(int fd_out, struct object_info_args *args)\n> +{\n> +\tstruct strbuf req_buf = STRBUF_INIT;\n> +\tint i;\n> +\n> +\twrite_object_info_command_and_capabilities(&req_buf, args->server_options);\n> +\n> +\tif (string_list_has_string(args->server_options, \"size\"))\n> +\t\tpacket_buf_write(&req_buf, \"size\");\n\nWhat happens if \"size\" is not in the list?\n\n> +\t\tprintf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n\nYou can just write \"echo $object_id $length >expect\". Also, test the\nfollowing:\n - more than one OID\n - an OID that's not on the remote\n - a malformed OID\n - a server that doesn't support protocol v2\n - a server that supports protocol v2 but not object-format\n\n(You don't have to do this for all protocols; just pick one. I prefer\nHTTP, since that's the most complex.)\n\nOther than that, the tests look good. Thanks for testing the different\nprotocols.\n\n> @@ -1269,6 +1280,7 @@ static struct transport_vtable vtable = {\n> \t.get_refs_list\t= get_refs_list,\n> \t.fetch_refs\t= fetch_refs,\n> \t.push_refs\t= push_refs,\n> +\t.fetch_object_info = fetch_object_info,\n> \t.connect\t= connect_helper,\n> \t.disconnect\t= release_helper\n> };\n\nAdding a function to the transport vtable is not so disruptive since we\ndon't have many transport vtables, but better if we can avoid this\ndisruption. In this case, I think it's better to reuse fetch_refs.\nMainly, the plumbing from transport_fetch_refs() to all the helper\nfunctions in fetch-pack.c already exists, so reusing fetch_refs would\nallow us to reuse that plumbing.\n\nThis also means that we don't have to expose the protocol functionality\nin fetch-pack.c that you copied over to builtin/fetch.c in this patch,\nwhich is an added bonus.\n\n> +static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n> +{\n\n[snip contents]\n\nAnd reusing the plumbing might mean that we don't need this function\ntoo.\n\nTaking a step back, there also needs to be a fallback mechanism for when\nthe server doesn't support object-info.\n\nIf you generally agree with my review comments, I would say that your\nnext steps are:\n - investigate if we can reuse the transport_fetch_pack -> fetch-pack.c\n machinery\n - make a fallback for when the server doesn't support object-info\n (might be easier when we use the machinery, so I would start with\n that first)\n"},{"id":"448193","messageId":"CAFySSZAAq4Q-BXeKHBozcX1byxhma6oKT0JnvOScYJkQ0fFi0w@mail.gmail.com","threadId":"57388","inReplyTo":"e21c4f77-061c-d125-97cb-1a8cb7692e56@iee.email","subject":"Re: [PATCH v2] fetch —object-info-format: client option for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-02-10T22:32:45Z","receivedAt":"2022-02-10T22:32:58Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Yes it does, thank you!\n\n\nOn Wed, Feb 9, 2022 at 4:48 AM Philip Oakley wrote:\n>\n> On 08/02/2022 23:56, Calvin Wan wrote:\n> > Add ‘—object-info-format’ option to fetch. This option allows\n> > the client to make an object-info [1] command request to a server\n> > that supports protocol v2.\n> >\n> > The transport implementation uses vtables [2], similar to how Git\n> > fetches refs, to determine whether a process needs to be taken over\n> > before sending the object-info request. Different protocols\n> > require different setups for making requests.\n> >\n> > [1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n> > [2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.15=\n> > 13287544.git.jonathantanmy@google.com/\n> >\n> > Helped-by: Jonathan Tan \n> > Signed-off-by: Calvin Wan \n> >\n> > ---\n> > Please ignore the patch above. It was sent with a stale patch message.\n> > builtin/fetch.c | 26 ++++++++++++\n> > fetch-pack.c | 53 +++++++++++++++++++++++\n> > fetch-pack.h | 7 ++++\n> > t/t5583-fetch-object-info.sh | 81 ++++++++++++++++++++++++++++++++++++\n> > transport-helper.c | 12 ++++++\n> > transport-internal.h | 1 +\n> > transport.c | 59 ++++++++++++++++++++++++++\n> > transport.h | 1 +\n> > 8 files changed, 240 insertions(+)\n> > create mode 100755 t/t5583-fetch-object-info.sh\n> >\n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 5f06b21f8e..b48d9e93d0 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -29,6 +29,9 @@\n> > #include \"commit-graph.h\"\n> > #include \"shallow.h\"\n> > #include \"worktree.h\"\n> > +#include \"protocol.h\"\n> > +#include \"pkt-line.h\"\n> > +#include \"connect.h\"\n> >\n> > #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n> >\n> > @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> > N_(\"git fetch [] \"),\n> > N_(\"git fetch --multiple [] [( | )...]\"),\n> > N_(\"git fetch --all []\"),\n> > + N_(\"git fetch --object-info-format=[] []\"),\n> > NULL\n> > };\n>\n> Doesn't this also need a matching Documentarian update for the option?\n> --\n> Philip\n"},{"id":"448196","messageId":"CAFySSZCZfekrdBH1NMArgPLdF4o1KQVY251BLyFgRxp=pDgEOA@mail.gmail.com","threadId":"57388","inReplyTo":"2B98F5EC-5F69-4DE1-89BF-33FCD1B03573@gmail.com","subject":"Re: [PATCH] fetch —object-info-format: client option for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-02-10T22:49:34Z","receivedAt":"2022-02-10T22:49:51Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> Maybe just --object-info and then provide a different flag for the perhaps?\nI agree. --object-info is a better name for the option. And with\n, there is also probably a better name -- will look into\nthe codebase and see what other options do.\n\n> should object_info_format be cleared at some point?\nI don't see why it needs to be cleared. The command only uses this\nvariable once in object-info.\n\nThank you for your input!\n\nOn Wed, Feb 9, 2022 at 11:09 AM John Cai wrote:\n>\n> Hi Calvin,\n>\n> On 8 Feb 2022, at 18:19, Calvin Wan wrote:\n>\n> > Add ‘—object-info-format’ option to fetch. This option allows\n> > the client to make an object-info [1] command request to a server\n> > that supports protocol v2.\n> >\n> > The transport implementation uses vtables [2], similar to how Git\n> > fetches refs, to determine whether a process needs to be taken over\n> > before sending the object-info request. Different protocols\n> > require different setups for making requests.\n> >\n> > [1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n> > [2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.1513287544.git.jonathantanmy@google.com/\n> >\n> > Helped-by: Jonathan Tan \n> > Signed-off-by: Calvin Wan \n> >\n> > ---\n> > builtin/fetch.c | 80 ++++++++++++++++++++----------\n> > fetch-pack.c | 53 ++++++++++++++++++++\n> > fetch-pack.h | 7 +++\n> > t/t5583-fetch-object-info.sh | 95 ++++++++++++++++++++++++++++++++++++\n> > transport-helper.c | 12 +++++\n> > transport-internal.h | 1 +\n> > transport.c | 63 ++++++++++++++++++++++++\n> > transport.h | 1 +\n> > 8 files changed, 286 insertions(+), 26 deletions(-)\n> > create mode 100755 t/t5583-fetch-object-info.sh\n> >\n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 5f06b21f8e..991063072b 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -29,6 +29,9 @@\n> > #include \"commit-graph.h\"\n> > #include \"shallow.h\"\n> > #include \"worktree.h\"\n> > +#include \"protocol.h\"\n> > +#include \"pkt-line.h\"\n> > +#include \"connect.h\"\n> >\n> > #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n> >\n> > @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> > N_(\"git fetch [] \"),\n> > N_(\"git fetch --multiple [] [( | )...]\"),\n> > N_(\"git fetch --all []\"),\n> > + N_(\"git fetch --object-info-format=[] []\"),\n>\n> --object-info-format seems to be more like a command, rather than a formatting\n> option. When a flag ends with \"format\", it feels like it is specifying how to\n> transform the output kind of like git for-each-ref --format. So --object-info-format looks\n> feels like it's meant to transform the object-info output. Maybe just --object-info\n> and then provide a different flag for the perhaps?\n>\n> Would be interested in hearing what others think from a UI perspective.\n>\n> > NULL\n> > };\n> >\n> > @@ -85,6 +89,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n> > static int fetch_write_commit_graph = -1;\n> > static int stdin_refspecs = 0;\n> > static int negotiate_only;\n> > +static struct string_list object_info_format = STRING_LIST_INIT_NODUP;\n>\n> should object_info_format be cleared at some point?\n>\n> >\n> > static int git_fetch_config(const char *k, const char *v, void *cb)\n> > {\n> > @@ -220,6 +225,8 @@ static struct option builtin_fetch_options[] = {\n> > N_(\"write the commit-graph after fetching\")),\n> > OPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> > N_(\"accept refspecs from stdin\")),\n> > + OPT_STRING_LIST(0, \"object-info-format\", &object_info_format, N_(\"option\"),\n> > + N_(\"command request arguments\")),\n> > OPT_END()\n> > };\n> >\n> > @@ -2000,6 +2007,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > struct remote *remote = NULL;\n> > int result = 0;\n> > int prune_tags_ok = 1;\n> > + struct oid_array oids = OID_ARRAY_INIT;\n> > + struct object_id oid;\n> >\n> > packet_trace_identity(\"fetch\");\n> >\n> > @@ -2019,6 +2028,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> >\n> > argc = parse_options(argc, argv, prefix,\n> > builtin_fetch_options, builtin_fetch_usage, 0);\n> > +\n> > if (recurse_submodules != RECURSE_SUBMODULES_OFF) {\n> > int *sfjc = submodule_fetch_jobs_config == -1\n> > ? &submodule_fetch_jobs_config : NULL;\n> > @@ -2057,34 +2067,52 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > if (dry_run)\n> > write_fetch_head = 0;\n> >\n> > - if (all) {\n> > - if (argc == 1)\n> > - die(_(\"fetch --all does not take a repository argument\"));\n> > - else if (argc > 1)\n> > - die(_(\"fetch --all does not make sense with refspecs\"));\n> > - (void) for_each_remote(get_one_remote_for_fetch, &list);\n> > - } else if (argc == 0) {\n> > - /* No arguments -- use default remote */\n> > - remote = remote_get(NULL);\n> > - } else if (multiple) {\n> > - /* All arguments are assumed to be remotes or groups */\n> > - for (i = 0; i < argc; i++)\n> > - if (!add_remote_or_group(argv[i], &list))\n> > - die(_(\"no such remote or remote group: %s\"),\n> > - argv[i]);\n> > - } else {\n> > - /* Single remote or group */\n> > - (void) add_remote_or_group(argv[0], &list);\n> > - if (list.nr > 1) {\n> > - /* More than one remote */\n> > - if (argc > 1)\n> > - die(_(\"fetching a group and specifying refspecs does not make sense\"));\n> > + if (object_info_format.nr > 0) {\n> > + if (argc == 0 || argc == 1) {\n> > + die(_(\"must supply remote and object ids when using --object-info-format\"));\n> > } else {\n> > - /* Zero or one remotes */\n> > remote = remote_get(argv[0]);\n> > - prune_tags_ok = (argc == 1);\n> > - argc--;\n> > - argv++;\n> > + for (i = 1; i < argc; i++) {\n> > + if (get_oid(argv[i], &oid))\n> > + return error(_(\"malformed object name '%s'\"), argv[i]);\n> > + oid_array_append(&oids, &oid);\n> > + }\n> > + }\n> > + gtransport = prepare_transport(remote, 0);\n> > + gtransport->server_options = &object_info_format;\n> > + result = transport_fetch_object_info(gtransport, &oids);\n> > +\n> > + return result;\n> > + } else {\n> > + if (all) {\n> > + if (argc == 1)\n> > + die(_(\"fetch --all does not take a repository argument\"));\n> > + else if (argc > 1)\n> > + die(_(\"fetch --all does not make sense with refspecs\"));\n> > + (void) for_each_remote(get_one_remote_for_fetch, &list);\n> > + } else if (argc == 0) {\n> > + /* No arguments -- use default remote */\n> > + remote = remote_get(NULL);\n> > + } else if (multiple) {\n> > + /* All arguments are assumed to be remotes or groups */\n> > + for (i = 0; i < argc; i++)\n> > + if (!add_remote_or_group(argv[i], &list))\n> > + die(_(\"no such remote or remote group: %s\"),\n> > + argv[i]);\n> > + } else {\n> > + /* Single remote or group */\n> > + (void) add_remote_or_group(argv[0], &list);\n> > + if (list.nr > 1) {\n> > + /* More than one remote */\n> > + if (argc > 1)\n> > + die(_(\"fetching a group and specifying refspecs does not make sense\"));\n> > + } else {\n> > + /* Zero or one remotes */\n> > + remote = remote_get(argv[0]);\n> > + prune_tags_ok = (argc == 1);\n> > + argc--;\n> > + argv++;\n> > + }\n> > }\n> > }\n> >\n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index dd6ec449f2..d1c5254aa8 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> > @@ -1265,6 +1265,59 @@ static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n> > packet_buf_delim(req_buf);\n> > }\n> >\n> > +static void write_object_info_command_and_capabilities(struct strbuf *req_buf,\n> > + const struct string_list *server_options)\n> > +{\n> > + const char *hash_name;\n> > +\n> > + if (server_supports_v2(\"object-info\", 1))\n> > + packet_buf_write(req_buf, \"command=object-info\");\n> > + if (server_supports_v2(\"agent\", 0))\n> > + packet_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n> > + if (advertise_sid && server_supports_v2(\"session-id\", 0))\n> > + packet_buf_write(req_buf, \"session-id=%s\", trace2_session_id());\n> > + if (server_options && server_options->nr &&\n> > + server_supports_v2(\"server-option\", 1)) {\n> > + int i;\n> > + for (i = 0; i < server_options->nr; i++)\n> > + packet_buf_write(req_buf, \"server-option=%s\",\n> > + server_options->items[i].string);\n> > + }\n> > +\n> > + if (server_feature_v2(\"object-format\", &hash_name)) {\n> > + int hash_algo = hash_algo_by_name(hash_name);\n> > + if (hash_algo_by_ptr(the_hash_algo) != hash_algo)\n> > + die(_(\"mismatched algorithms: client %s; server %s\"),\n> > + the_hash_algo->name, hash_name);\n> > + packet_buf_write(req_buf, \"object-format=%s\", the_hash_algo->name);\n> > + } else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {\n> > + die(_(\"the server does not support algorithm '%s'\"),\n> > + the_hash_algo->name);\n> > + }\n> > + packet_buf_delim(req_buf);\n> > +}\n> > +\n> > +void send_object_info_request(int fd_out, struct object_info_args *args)\n> > +{\n> > + struct strbuf req_buf = STRBUF_INIT;\n> > + int i;\n> > +\n> > + write_object_info_command_and_capabilities(&req_buf, args->server_options);\n> > +\n> > + if (string_list_has_string(args->server_options, \"size\"))\n> > + packet_buf_write(&req_buf, \"size\");\n> > +\n> > + for (i = 0; i < args->oids->nr; i++) {\n> > + packet_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> > + }\n> > +\n> > + packet_buf_flush(&req_buf);\n> > + if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n> > + die_errno(_(\"unable to write request to remote\"));\n> > +\n> > + strbuf_release(&req_buf);\n> > +}\n> > +\n> > static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> > struct fetch_pack_args *args,\n> > const struct ref *wants, struct oidset *common,\n> > diff --git a/fetch-pack.h b/fetch-pack.h\n> > index 7f94a2a583..2ad5ec5c64 100644\n> > --- a/fetch-pack.h\n> > +++ b/fetch-pack.h\n> > @@ -68,6 +68,11 @@ struct fetch_pack_args {\n> > unsigned connectivity_checked:1;\n> > };\n> >\n> > +struct object_info_args {\n> > + const struct string_list *server_options;\n> > + const struct oid_array *oids;\n> > +};\n> > +\n> > /*\n> > * sought represents remote references that should be updated from.\n> > * On return, the names that were found on the remote will have been\n> > @@ -101,4 +106,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n> > */\n> > int report_unmatched_refs(struct ref **sought, int nr_sought);\n> >\n> > +void send_object_info_request(int fd_out, struct object_info_args *args);\n> > +\n> > #endif\n> > diff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\n> > new file mode 100755\n> > index 0000000000..93b09da3b1\n> > --- /dev/null\n> > +++ b/t/t5583-fetch-object-info.sh\n> > @@ -0,0 +1,95 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='test git fetch object-info version 2'\n> > +\n> > +TEST_NO_CREATE_REPO=1\n> > +\n> > +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> > +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +initial_directory=$pwd\n> > +\n> > +# Test fetch object-info with 'git://' transport\n> > +#\n> > +. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n> > +start_git_daemon --export-all --enable=receive-pack\n> > +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n> > +\n> > +\n> > +test_expect_success 'create repo to be served by git-daemon' '\n> > + git init \"$daemon_parent\" &&\n> > + test_commit -C \"$daemon_parent\" message1 a.txt\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info with git:// using protocol v2' '\n> > + test_when_finished \"cd '$initial_directory'\" &&\n> > +\n> > + cd \"$daemon_parent\" &&\n> > +\n> > + cat >actual <<-EOF &&\n> > + $(GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info-format=size \"$GIT_DAEMON_URL/parent\" $(git rev-parse message1:a.txt))\n> > + EOF\n> > +\n> > + cat >expect <<-EOF &&\n> > + $(git rev-parse message1:a.txt) $(wc -c > + EOF\n> > +\n> > + test_cmp expect actual\n> > +'\n> > +stop_git_daemon\n> > +\n> > +# Test protocol v2 with 'http://' transport\n> > +#\n> > +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> > +start_httpd\n> > +\n> > +test_expect_success 'create repo to be served by http:// transport' '\n> > + git init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + git -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n> > + test_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message1 a.txt\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info with http:// using protocol v2' '\n> > + test_when_finished \"cd '$initial_directory'\" &&\n> > +\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > +\n> > + cat >actual <<-EOF &&\n> > + $(git -c protocol.version=2 fetch --object-info-format=size \"$HTTPD_URL/smart/http_parent\" $(git rev-parse message1:a.txt))\n> > + EOF\n> > +\n> > + cat >expect <<-EOF &&\n> > + $(git rev-parse message1:a.txt) $(wc -c > + EOF\n> > +\n> > + test_cmp expect actual\n> > +'\n> > +\n> > +# Test fetch object-info with 'file://' transport\n> > +#\n> > +\n> > +test_expect_success 'create repo to be serbed by file:// transport' '\n> > + git init server &&\n> > + test_commit -C server message1 a.txt &&\n> > + git -C server config protocol.version 2\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info with file:// using protocol v2' '\n> > + test_when_finished \"cd '$initial_directory'\" &&\n> > +\n> > + cd server &&\n> > +\n> > + cat >actual <<-EOF &&\n> > + $(git fetch --object-info-format=size \"file://$(pwd)\" $(git rev-parse message1:a.txt))\n> > + EOF\n> > +\n> > + cat >expect <<-EOF &&\n> > + $(git rev-parse message1:a.txt) $(wc -c > + EOF\n> > +\n> > + test_cmp expect actual\n> > +'\n> > +\n> > +test_done\n> > \\ No newline at end of file\n> > diff --git a/transport-helper.c b/transport-helper.c\n> > index a0297b0986..9ecda03dde 100644\n> > --- a/transport-helper.c\n> > +++ b/transport-helper.c\n> > @@ -671,6 +671,17 @@ static int connect_helper(struct transport *transport, const char *name,\n> > static struct ref *get_refs_list_using_list(struct transport *transport,\n> > int for_push);\n> >\n> > +static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n> > +{\n> > + get_helper(transport);\n> > +\n> > + if (process_connect(transport, 0)) {\n> > + do_take_over(transport);\n> > + return transport->vtable->fetch_object_info(transport, oids);\n> > + }\n> > + return -1;\n> > +}\n> > +\n> > static int fetch_refs(struct transport *transport,\n> > int nr_heads, struct ref **to_fetch)\n> > {\n> > @@ -1269,6 +1280,7 @@ static struct transport_vtable vtable = {\n> > .get_refs_list = get_refs_list,\n> > .fetch_refs = fetch_refs,\n> > .push_refs = push_refs,\n> > + .fetch_object_info = fetch_object_info,\n> > .connect = connect_helper,\n> > .disconnect = release_helper\n> > };\n> > diff --git a/transport-internal.h b/transport-internal.h\n> > index c4ca0b733a..04fa015011 100644\n> > --- a/transport-internal.h\n> > +++ b/transport-internal.h\n> > @@ -59,6 +59,7 @@ struct transport_vtable {\n> > * use. disconnect() releases these resources.\n> > **/\n> > int (*disconnect)(struct transport *connection);\n> > + int (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n> > };\n> >\n> > #endif\n> > diff --git a/transport.c b/transport.c\n> > index 2a3e324154..0e1bf3c4cd 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -445,6 +445,62 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > return ret;\n> > }\n> >\n> > +static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n> > +{\n> > + int ret = 0;\n> > + struct git_transport_data *data = transport->data;\n> > + struct object_info_args args;\n> > + struct packet_reader reader;\n> > +\n> > + memset(&args, 0, sizeof(args));\n> > + args.server_options = transport->server_options;\n> > + args.oids = oids;\n> > +\n> > + connect_setup(transport, 0);\n> > + packet_reader_init(&reader, data->fd[0], NULL, 0,\n> > + PACKET_READ_CHOMP_NEWLINE |\n> > + PACKET_READ_GENTLE_ON_EOF |\n> > + PACKET_READ_DIE_ON_ERR_PACKET);\n> > + data->version = discover_version(&reader);\n> > +\n> > + if (data->version == protocol_unknown_version)\n> > + BUG(\"unknown protocol version\");\n> > + else if (data->version <= protocol_v1)\n> > + die_if_server_options(transport);\n> > +\n> > + switch (data->version) {\n> > + case protocol_v2:\n> > + send_object_info_request(data->fd[1], &args);\n> > + break;\n> > + case protocol_v1:\n> > + case protocol_v0:\n> > + die(_(\"wrong protocol version. expected v2\"));\n> > + case protocol_unknown_version:\n> > + BUG(\"unknown protocol version\");\n> > + }\n> > +\n> > + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> > + die(_(\"error reading object info header\"));\n> > + }\n> > + if (strcmp(reader.line, \"size\")) {\n> > + die(_(\"expected 'size', received '%s'\"), reader.line);\n> > + }\n> > + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> > + printf(\"%s\\n\", reader.line);\n> > + }\n> > +\n> > + close(data->fd[0]);\n> > +\n> > + if (data->fd[1] >= 0) {\n> > + close(data->fd[1]);\n> > + }\n> > + if (finish_connect(data->conn))\n> > + ret = -1;\n> > + data->conn = NULL;\n> > +\n> > + return ret;\n> > +}\n> > +\n> > static int push_had_errors(struct ref *ref)\n> > {\n> > for (; ref; ref = ref->next) {\n> > @@ -890,6 +946,7 @@ static struct transport_vtable taken_over_vtable = {\n> > .get_refs_list = get_refs_via_connect,\n> > .fetch_refs = fetch_refs_via_pack,\n> > .push_refs = git_transport_push,\n> > + .fetch_object_info = fetch_object_info,\n> > .disconnect = disconnect_git\n> > };\n> >\n> > @@ -1043,6 +1100,7 @@ static struct transport_vtable builtin_smart_vtable = {\n> > .get_refs_list = get_refs_via_connect,\n> > .fetch_refs = fetch_refs_via_pack,\n> > .push_refs = git_transport_push,\n> > + .fetch_object_info = fetch_object_info,\n> > .connect = connect_git,\n> > .disconnect = disconnect_git\n> > };\n> > @@ -1420,6 +1478,11 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n> > return transport->remote_refs;\n> > }\n> >\n> > +int transport_fetch_object_info(struct transport *transport, struct oid_array *oids) {\n> > + transport->vtable->fetch_object_info(transport, oids);\n> > + return 1;\n> > +}\n> > +\n> > int transport_fetch_refs(struct transport *transport, struct ref *refs)\n> > {\n> > int rc;\n> > diff --git a/transport.h b/transport.h\n> > index 3f16e50c19..1c807591de 100644\n> > --- a/transport.h\n> > +++ b/transport.h\n> > @@ -278,6 +278,7 @@ const struct ref *transport_get_remote_refs(struct transport *transport,\n> > * This can only be called after fetching the remote refs.\n> > */\n> > const struct git_hash_algo *transport_get_hash_algo(struct transport *transport);\n> > +int transport_fetch_object_info(struct transport *transport, struct oid_array *oids);\n> > int transport_fetch_refs(struct transport *transport, struct ref *refs);\n> >\n> > /*\n> >\n> > base-commit: b23dac905bde28da47543484320db16312c87551\n> > --\n> > 2.33.0.664.g0785eb7698\n"},{"id":"448200","messageId":"CAFySSZAyyEOLMXhJNMmJTV1AJRdnc9AER1n20PhYaLZs1BZPdg@mail.gmail.com","threadId":"57388","inReplyTo":"20220209204153.481122-1-jonathantanmy@google.com","subject":"Re: [PATCH v2] fetch object-info-format: client option for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-02-10T22:58:40Z","receivedAt":"2022-02-10T22:58:55Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> If you generally agree with my review comments\nEverything you've said makes sense. Thanks for taking an in depth look\nat this! I'll make the changes you suggested and then start working on\nthe next steps.\n\nOn Wed, Feb 9, 2022 at 12:42 PM Jonathan Tan wrote:\n>\n> Calvin Wan writes:\n> > Add ‘—object-info-format’ option to fetch. This option allows\n> > the client to make an object-info [1] command request to a server\n> > that supports protocol v2.\n>\n> Avoid using characters above 127 in commit messages (unless, say, as part of\n> someone's name). They make it hard to search for, and in this case, whatever\n> dash is there is wrong (it should be \"--\").\n>\n> > The transport implementation uses vtables [2], similar to how Git\n> > fetches refs,\n>\n> You should be explicit that you're adding a new function to the vtable.\n> (Whether that is what we should do is another issue: let me look at the\n> patch to see.)\n>\n> > to determine whether a process needs to be taken over\n> > before sending the object-info request.\n>\n> The vtable is not to determine whether a process needs to be taken over,\n> but so that we support multiple protocols (HTTP, SSH, etc.). In any\n> case, this detail is probably not relevant.\n>\n> > Different protocols\n> > require different setups for making requests.\n>\n> This is true, but I don't see the relevance.\n>\n> > [1] https://lore.kernel.org/git/20210420233830.2181153-1-bga@google.com/\n> > [2] https://lore.kernel.org/git/26f276956001a120cc9105b0071762c2fd4a45c5.15=\n> > 13287544.git.jonathantanmy@google.com/\n>\n> For merged code, quote the commit, not the email.\n>\n> > @@ -220,6 +225,8 @@ static struct option builtin_fetch_options[] = {\n> > N_(\"write the commit-graph after fetching\")),\n> > OPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> > N_(\"accept refspecs from stdin\")),\n> > + OPT_STRING_LIST(0, \"object-info-format\", &object_info_format, N_(\"option\"),\n> > + N_(\"command request arguments\")),\n>\n> I would have expected a parameter named \"format\" to take a format\n> string, but taking a string list of the fields we need might work too.\n> In any case, maybe rename it to \"--object-info\" or similar.\n>\n> > @@ -2000,6 +2007,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > struct remote *remote = NULL;\n> > int result = 0;\n> > int prune_tags_ok = 1;\n> > + struct oid_array oids = OID_ARRAY_INIT;\n> > + struct object_id oid;\n>\n> The \"oids\" at function level needs a more descriptive name (e.g.\n> \"oids_for_object_info\"). The name of \"oid\" is fine, since it's just used\n> as a temporary variable, but since it is temporary, it should be\n> declared in the block where it's used. (Same for \"oids\", actually:\n> declare it in the block it's used, and in that case you can keep the\n> name since it's more tightly scoped.)\n>\n> > @@ -2057,6 +2066,23 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > if (dry_run)\n> > write_fetch_head = 0;\n> >\n> > + if (object_info_format.nr > 0) {\n> > + if (argc == 0 || argc == 1) {\n> > + die(_(\"must supply remote and object ids when using --object-info-format\"));\n> > + } else {\n> > + remote = remote_get(argv[0]);\n> > + for (i = 1; i < argc; i++) {\n> > + if (get_oid(argv[i], &oid))\n> > + return error(_(\"malformed object name '%s'\"), argv[i]);\n> > + oid_array_append(&oids, &oid);\n> > + }\n> > + }\n> > + gtransport = prepare_transport(remote, 0);\n> > + gtransport->server_options = &object_info_format;\n> > + result = transport_fetch_object_info(gtransport, &oids);\n> > + return result;\n> > + }\n>\n> I was thinking that this should reuse the refspec parsing mechanism\n> (which also supports stdin), but upon more thought, using the refspec\n> parser means that we would also need to check that all refspecs are\n> exact OIDs (because we wouldn't know what to do with them otherwise).\n> OK, parsing the objects by ourselves looks reasonable.\n>\n> The assignment of object_info_format to server_options is probably not a\n> good idea, though, since readers of server_options would expect server\n> options, not what you're assigning. The best place to put this\n> information is in smart_options. (See the negotiate_only code.)\n>\n> > +static void write_object_info_command_and_capabilities(struct strbuf *req_buf,\n> > + const struct string_list *server_options)\n> > +{\n>\n> [snip contents]\n>\n> This code is very similar to code in fetch-pack.c. If you stick to\n> crafting the request in builtin/fetch.c, you should refactor\n> fetch-pack.{c,h} to expose this functionality (in a preparatory commit)\n> and then use that function from here.\n>\n> > +void send_object_info_request(int fd_out, struct object_info_args *args)\n> > +{\n> > + struct strbuf req_buf = STRBUF_INIT;\n> > + int i;\n> > +\n> > + write_object_info_command_and_capabilities(&req_buf, args->server_options);\n> > +\n> > + if (string_list_has_string(args->server_options, \"size\"))\n> > + packet_buf_write(&req_buf, \"size\");\n>\n> What happens if \"size\" is not in the list?\n>\n> > + printf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n>\n> You can just write \"echo $object_id $length >expect\". Also, test the\n> following:\n> - more than one OID\n> - an OID that's not on the remote\n> - a malformed OID\n> - a server that doesn't support protocol v2\n> - a server that supports protocol v2 but not object-format\n>\n> (You don't have to do this for all protocols; just pick one. I prefer\n> HTTP, since that's the most complex.)\n>\n> Other than that, the tests look good. Thanks for testing the different\n> protocols.\n>\n> > @@ -1269,6 +1280,7 @@ static struct transport_vtable vtable = {\n> > .get_refs_list = get_refs_list,\n> > .fetch_refs = fetch_refs,\n> > .push_refs = push_refs,\n> > + .fetch_object_info = fetch_object_info,\n> > .connect = connect_helper,\n> > .disconnect = release_helper\n> > };\n>\n> Adding a function to the transport vtable is not so disruptive since we\n> don't have many transport vtables, but better if we can avoid this\n> disruption. In this case, I think it's better to reuse fetch_refs.\n> Mainly, the plumbing from transport_fetch_refs() to all the helper\n> functions in fetch-pack.c already exists, so reusing fetch_refs would\n> allow us to reuse that plumbing.\n>\n> This also means that we don't have to expose the protocol functionality\n> in fetch-pack.c that you copied over to builtin/fetch.c in this patch,\n> which is an added bonus.\n>\n> > +static int fetch_object_info(struct transport *transport, struct oid_array *oids)\n> > +{\n>\n> [snip contents]\n>\n> And reusing the plumbing might mean that we don't need this function\n> too.\n>\n> Taking a step back, there also needs to be a fallback mechanism for when\n> the server doesn't support object-info.\n>\n> If you generally agree with my review comments, I would say that your\n> next steps are:\n> - investigate if we can reuse the transport_fetch_pack -> fetch-pack.c\n> machinery\n> - make a fallback for when the server doesn't support object-info\n> (might be easier when we use the machinery, so I would start with\n> that first)\n"},{"id":"452495","messageId":"20220328191112.3092139-1-calvinwan@google.com","threadId":"57388","inReplyTo":"20220208235631.732466-1-calvinwan@google.com","subject":"[PATCH v3 0/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-28T19:11:09Z","receivedAt":"2022-03-28T19:11:27Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Sometimes it is useful to get information about an object without\nhaving to download it completely. The server logic has already been\nimplemented as “a2ba162cda (object-info: support for retrieving object\ninfo, 2021-04-20)”. This patch implements the client option for it.\n\nAdd ‘--object-info’ option to fetch. This option allows the client to\nmake an object-info command request to a server that supports protocol\nv2. If the server is v2, but does not allow for the object-info\ncommand request, the entire object is fetched instead and the relevant\ninformation is returned.\n\nPatch 1 is a small refactor. Patch 2 adds a config option for\nadvertisement of object-info. Patch 3 is the main implementation of\nfetch object-info.\n\nMajor changes since v2:\n * Added a fallback for servers that do not\n allow object-info\n * Refactored the fetch object-info code by moving it to fetch_refs in\n the vtable rather than having its own function in the vtable. This\n allows for the code to easily transition to the fallback if\n object-info fails.\n\nCalvin Wan (3):\n fetch-pack: refactor how packet is written to be more generic\n objectinfo.advertise: add config option to advertise object-info\n object-info: add option for retrieving object info\n\n Documentation/config/transfer.txt | 4 +\n Documentation/fetch-options.txt | 5 ++\n Documentation/git-fetch.txt | 1 +\n builtin/fetch.c | 36 +++++++-\n fetch-pack.c | 66 ++++++++++----\n fetch-pack.h | 9 ++\n protocol-caps.c | 11 +++\n protocol-caps.h | 6 ++\n serve.c | 2 +-\n t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++\n transport-helper.c | 8 +-\n transport-internal.h | 1 +\n transport.c | 75 +++++++++++++++-\n transport.h | 9 ++\n 14 files changed, 349 insertions(+), 22 deletions(-)\n create mode 100755 t/t5583-fetch-object-info.sh\n\n\nbase-commit: f01e51a7cfd75131b7266131b1f7540ce0a8e5c1\n-- \n2.33.0.664.g0785eb7698\n\n"},{"id":"452497","messageId":"20220328191112.3092139-2-calvinwan@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-1-calvinwan@google.com","subject":"[PATCH v3 1/3] fetch-pack: refactor packet writing and fetch options","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-28T19:11:10Z","receivedAt":"2022-03-28T19:11:55Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A subsequent commit will need to write capabilities for another\ncommand, so refactor write_fetch_command_and_capabilities() to be able\nto used by both fetch and future command.\n\nMove fetch options set in `FETCH_CHECK_LOCAL` from the fetch state\nmachine to above the state machine so it is set by default. The\ninitial state of the state machine is always `FETCH_CHECK_LOCAL` so\nthis does not affect any current functionality. This change prepares\nfor a subsequent commit that doesn't need to check the local state, but\nstill requires those options to be set before sending the fetch request.\n\n---\n fetch-pack.c | 24 ++++++++++++------------\n 1 file changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 87657907e7..b709a61baf 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1237,13 +1237,13 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \treturn haves_added;\n }\n \n-static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n-\t\t\t\t\t\t const struct string_list *server_options)\n+static void write_command_and_capabilities(struct strbuf *req_buf,\n+\t\t\t\t\t\t const struct string_list *server_options, const char* command)\n {\n \tconst char *hash_name;\n \n-\tif (server_supports_v2(\"fetch\", 1))\n-\t\tpacket_buf_write(req_buf, \"command=fetch\");\n+\tif (server_supports_v2(command, 1))\n+\t\tpacket_buf_write(req_buf, \"command=%s\", command);\n \tif (server_supports_v2(\"agent\", 0))\n \t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n \tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n@@ -1279,7 +1279,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \tint done_sent = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n-\twrite_fetch_command_and_capabilities(&req_buf, args->server_options);\n+\twrite_command_and_capabilities(&req_buf, args->server_options, \"fetch\");\n \n \tif (args->use_thin_pack)\n \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n@@ -1598,18 +1598,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\treader.me = \"fetch-pack\";\n \t}\n \n+\t/* v2 supports these by default */\n+\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\tuse_sideband = 2;\n+\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n+\t\targs->deepen = 1;\n+\n \twhile (state != FETCH_DONE) {\n \t\tswitch (state) {\n \t\tcase FETCH_CHECK_LOCAL:\n \t\t\tsort_ref_list(&ref, ref_compare_name);\n \t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n \n-\t\t\t/* v2 supports these by default */\n-\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n-\t\t\tuse_sideband = 2;\n-\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n-\t\t\t\targs->deepen = 1;\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n@@ -2060,7 +2060,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n \t\tint received_ready = 0;\n \n \t\tstrbuf_reset(&req_buf);\n-\t\twrite_fetch_command_and_capabilities(&req_buf, server_options);\n+\t\twrite_command_and_capabilities(&req_buf, server_options, \"fetch\");\n \n \t\tpacket_buf_write(&req_buf, \"wait-for-done\");\n \n-- \n2.33.0.664.g0785eb7698\n\n"},{"id":"452496","messageId":"20220328191112.3092139-3-calvinwan@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-1-calvinwan@google.com","subject":"[PATCH v3 2/3] transfer.advertiseObjectInfo: add object-info config","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-28T19:11:11Z","receivedAt":"2022-03-28T19:11:57Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Currently, object-info is always advertised by the server. Add a\nconfig option to optionally advertise it. A subsequent patch uses this\noption for testing.\n\n---\n Documentation/config/transfer.txt | 4 ++++\n protocol-caps.c | 11 +++++++++++\n protocol-caps.h | 6 ++++++\n serve.c | 2 +-\n 4 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\nindex b49429eb4d..14892a3155 100644\n--- a/Documentation/config/transfer.txt\n+++ b/Documentation/config/transfer.txt\n@@ -77,3 +77,7 @@ transfer.unpackLimit::\n transfer.advertiseSID::\n \tBoolean. When true, client and server processes will advertise their\n \tunique session IDs to their remote counterpart. Defaults to false.\n+\n+transfer.advertiseObjectInfo::\n+\tBoolean. When true or not set, server processes will advertise its\n+\tability to accept `object-info` command requests\n\\ No newline at end of file\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex bbde91810a..f290e3cca2 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -8,6 +8,9 @@\n #include \"object-store.h\"\n #include \"string-list.h\"\n #include \"strbuf.h\"\n+#include \"config.h\"\n+\n+static int advertise_object_info = -1;\n \n struct requested_info {\n \tunsigned size : 1;\n@@ -111,3 +114,11 @@ int cap_object_info(struct repository *r, struct packet_reader *request)\n \n \treturn 0;\n }\n+\n+int object_info_advertise(struct repository *r, struct strbuf *value)\n+{\n+\tif (advertise_object_info == -1 &&\n+\t\tgit_config_get_bool(\"transfer.advertiseObjectInfo\", &advertise_object_info))\n+\t\tadvertise_object_info = 0;\n+\treturn advertise_object_info;\n+}\ndiff --git a/protocol-caps.h b/protocol-caps.h\nindex 15c4550360..36f7a46b37 100644\n--- a/protocol-caps.h\n+++ b/protocol-caps.h\n@@ -5,4 +5,10 @@ struct repository;\n struct packet_reader;\n int cap_object_info(struct repository *r, struct packet_reader *request);\n \n+/*\n+ * Advertises object-info capability if \"objectinfo.advertise\" is either\n+ * set to true or not set\n+ */\n+int object_info_advertise(struct repository *r, struct strbuf *value);\n+\n #endif /* PROTOCOL_CAPS_H */\ndiff --git a/serve.c b/serve.c\nindex b3fe9b5126..fb4ad367a7 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -133,7 +133,7 @@ static struct protocol_capability capabilities[] = {\n \t},\n \t{\n \t\t.name = \"object-info\",\n-\t\t.advertise = always_advertise,\n+\t\t.advertise = object_info_advertise,\n \t\t.command = cap_object_info,\n \t},\n };\n-- \n2.33.0.664.g0785eb7698\n\n"},{"id":"452498","messageId":"20220328191112.3092139-4-calvinwan@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-1-calvinwan@google.com","subject":"[PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-28T19:11:12Z","receivedAt":"2022-03-28T19:12:01Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Sometimes it is useful to get information about an object without\nhaving to download it completely. The server logic has already been\nimplemented as “a2ba162cda (object-info: support for retrieving object\ninfo, 2021-04-20)”. This patch implements the client option for\nit. Currently, only 'size' is supported, however future patches can\nimplement additional options.\n\nAdd ‘--object-info’ option to fetch. This option allows the client to\nmake an object-info command request to a server that supports protocol\nv2. If the server is v2, but does not allow for the object-info\ncommand request, fetch the objects as if it were a standard fetch\n(however without changing any refs). Therefore, hook `fetch\nobject-info` into transport_fetch_refs() to easily fallback if needed.\n\nA previous patch added the `transfer.advertiseObjectInfo` config\noption, of which this patch utilizes to test the fallback.\n\n---\n Documentation/fetch-options.txt | 5 ++\n Documentation/git-fetch.txt | 1 +\n builtin/fetch.c | 36 ++++++++-\n fetch-pack.c | 42 +++++++++-\n fetch-pack.h | 9 +++\n t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n transport-helper.c | 8 +-\n transport-internal.h | 1 +\n transport.c | 75 ++++++++++++++++-\n transport.h | 9 +++\n 10 files changed, 315 insertions(+), 9 deletions(-)\n create mode 100755 t/t5583-fetch-object-info.sh\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex f903683189..f1ca95c32d 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -299,3 +299,8 @@ endif::git-pull[]\n -6::\n --ipv6::\n \tUse IPv6 addresses only, ignoring IPv4 addresses.\n+\n+--object-info=[]::\n+\tFetches only the relevant information passed in arguments from a\n+\tspecific remote and set of object ids. Object 'size' is currently\n+\tthe only supported argument.\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca61..a13d2ba7ad 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -13,6 +13,7 @@ SYNOPSIS\n 'git fetch' [] \n 'git fetch' --multiple [] [( | )...]\n 'git fetch' --all []\n+'git fetch' --object-info=[] []\n \n \n DESCRIPTION\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4d12c2fd4d..0b21bebfca 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -29,6 +29,9 @@\n #include \"commit-graph.h\"\n #include \"shallow.h\"\n #include \"worktree.h\"\n+#include \"protocol.h\"\n+#include \"pkt-line.h\"\n+#include \"connect.h\"\n \n #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n \n@@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] \"),\n \tN_(\"git fetch --multiple [] [( | )...]\"),\n \tN_(\"git fetch --all []\"),\n+\tN_(\"git fetch --object-info=[] []\"),\n \tNULL\n };\n \n@@ -86,6 +90,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n static int fetch_write_commit_graph = -1;\n static int stdin_refspecs = 0;\n static int negotiate_only;\n+static struct string_list object_info_options = STRING_LIST_INIT_NODUP;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -221,6 +226,8 @@ static struct option builtin_fetch_options[] = {\n \t\t N_(\"write the commit-graph after fetching\")),\n \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n \t\t N_(\"accept refspecs from stdin\")),\n+\tOPT_STRING_LIST(0, \"object-info\", &object_info_options, N_(\"option\"),\n+\t\t N_(\"command request arguments\")),\n \tOPT_END()\n };\n \n@@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tstruct remote *remote = NULL;\n \tint result = 0;\n \tint prune_tags_ok = 1;\n+\tstruct oid_array object_info_oids = OID_ARRAY_INIT;\n \n \tpacket_trace_identity(\"fetch\");\n \n@@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (dry_run)\n \t\twrite_fetch_head = 0;\n \n-\tif (all) {\n+\tif (object_info_options.nr > 0) {\n+\t\tif (argc == 0 || argc == 1) {\n+\t\t\tdie(_(\"must supply remote and object ids when using --object-info\"));\n+\t\t} else {\n+\t\t\tstruct object_id oid;\n+\t\t\tremote = remote_get(argv[0]);\n+\t\t\tfor (i = 1; i < argc; i++) {\n+\t\t\t\tif (get_oid(argv[i], &oid))\n+\t\t\t\t\treturn error(_(\"malformed object id '%s'\"), argv[i]);\n+\t\t\t\toid_array_append(&object_info_oids, &oid);\n+\t\t\t}\n+\t\t}\n+\t} else if (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n \t\telse if (argc > 1)\n@@ -2199,7 +2219,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (negotiate_only) {\n+\tif (object_info_options.nr > 0) {\n+\t\tif (!remote)\n+\t\t\tdie(_(\"must supply remote when using --object-info\"));\n+\t\tgtransport = prepare_transport(remote, 1);\n+\t\tif (gtransport->smart_options) {\n+\t\t\tgtransport->smart_options->object_info = 1;\n+\t\t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n+\t\t\tgtransport->smart_options->object_info_options = &object_info_options;\n+\t\t}\n+\t\tif (server_options.nr)\n+\t\t\tgtransport->server_options = &server_options;\n+\t\tresult = transport_fetch_refs(gtransport, NULL);\n+\t} else if (negotiate_only) {\n \t\tstruct oidset acked_commits = OIDSET_INIT;\n \t\tstruct oidset_iter iter;\n \t\tconst struct object_id *oid;\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex b709a61baf..36e3d1c5d0 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n \tpacket_buf_delim(req_buf);\n }\n \n+void send_object_info_request(int fd_out, struct object_info_args *args)\n+{\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\tint i;\n+\n+\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n+\n+\tif (string_list_has_string(args->object_info_options, \"size\"))\n+\t\tpacket_buf_write(&req_buf, \"size\");\n+\n+\tfor (i = 0; i < args->oids->nr; i++) {\n+\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n+\t}\n+\n+\tpacket_buf_flush(&req_buf);\n+\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n+\t\tdie_errno(_(\"unable to write request to remote\"));\n+\n+\tstrbuf_release(&req_buf);\n+}\n+\n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n@@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\targs->deepen = 1;\n \n+\tif (args->object_info) {\n+\t\tstate = FETCH_SEND_REQUEST;\n+\t}\n+\n \twhile (state != FETCH_DONE) {\n \t\tswitch (state) {\n \t\tcase FETCH_CHECK_LOCAL:\n@@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n-\t\t\tif (everything_local(args, &ref))\n+\t\t\tif (!args->object_info && everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n@@ -1999,8 +2024,19 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t}\n \t\targs->connectivity_checked = 1;\n \t}\n-\n-\tupdate_shallow(args, sought, nr_sought, &si);\n+\t\n+\tif (args->object_info) {\n+\t\tstruct ref *ref_cpy_reader = ref_cpy;\n+\t\tunsigned long size = 0;\n+\t\twhile (ref_cpy_reader) {\n+\t\t\toid_object_info(the_repository, &(ref_cpy_reader->old_oid), &size);\n+\t\t\tprintf(\"%s %li\\n\", oid_to_hex(&(ref_cpy_reader->old_oid)), size);\n+\t\t\tref_cpy_reader = ref_cpy_reader->next;\n+\t\t}\n+\t}\n+\telse {\n+\t\tupdate_shallow(args, sought, nr_sought, &si);\n+\t}\n cleanup:\n \tclear_shallow_info(&si);\n \toid_array_clear(&shallows_scratch);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 7f94a2a583..23e69bb218 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -42,6 +42,7 @@ struct fetch_pack_args {\n \tunsigned update_shallow:1;\n \tunsigned reject_shallow_remote:1;\n \tunsigned deepen:1;\n+\tunsigned object_info:1;\n \n \t/*\n \t * Indicate that the remote of this request is a promisor remote. The\n@@ -68,6 +69,12 @@ struct fetch_pack_args {\n \tunsigned connectivity_checked:1;\n };\n \n+struct object_info_args {\n+\tconst struct string_list *object_info_options;\n+\tconst struct string_list *server_options;\n+\tconst struct oid_array *oids;\n+};\n+\n /*\n * sought represents remote references that should be updated from.\n * On return, the names that were found on the remote will have been\n@@ -101,4 +108,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+void send_object_info_request(int fd_out, struct object_info_args *args);\n+\n #endif\ndiff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\nnew file mode 100755\nindex 0000000000..e5db6d953c\n--- /dev/null\n+++ b/t/t5583-fetch-object-info.sh\n@@ -0,0 +1,138 @@\n+#!/bin/sh\n+\n+test_description='test git fetch object-info version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+# Test fetch object-info with 'git://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" message1 a.txt\n+'\n+\n+test_expect_success 'fetch object-info with git:// using protocol v2' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit -c protocol.version=2 fetch --object-info=size \"$GIT_DAEMON_URL/parent\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+stop_git_daemon\n+\n+# Test protocol v2 with 'http://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message1 a.txt &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message2 b.txt\n+'\n+\n+test_expect_success 'fetch object-info with http:// using protocol v2' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch object-info for multiple objects' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tobject_id1=$(git rev-parse message1:a.txt) &&\n+\t\tobject_id2=$(git rev-parse message2:b.txt) &&\n+\t\tlength1=$(wc -c expect &&\n+\t\tprintf \"%s %d\\n\" \"$object_id2\" \"$length2\" >>expect &&\n+\t\tgit -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id1\" \"$object_id2\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch object-info fallbacks to standard fetch if object-info is not supported by the server' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit config objectinfo.advertise false &&\n+\t\tgit -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch object-info fails on server with legacy protocol' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\ttest_must_fail git -c protocol.version=0 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" 2>err &&\n+\t\ttest_i18ngrep \"object-info requires protocol v2\" err\n+\t)\n+'\n+\n+test_expect_success 'fetch object-info fails on malformed OID' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tmalformed_object_id=\"this_id_is_not_valid\" &&\n+\t\ttest_must_fail git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$malformed_object_id\" 2>err &&\n+\t\ttest_i18ngrep \"malformed object id '$malformed_object_id'\" err\n+\t)\n+'\n+\n+test_expect_success 'fetch object-info fails on missing OID' '\n+\tgit clone \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" missing_oid_repo &&\n+\ttest_commit -C missing_oid_repo message3 c.txt &&\n+\t(\n+\t\tcd missing_oid_repo &&\n+\t\tobject_id=$(git rev-parse message3:c.txt) &&\n+\t\ttest_must_fail env GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" 2>err &&\n+\t\ttest_i18ngrep \"fatal: remote error: upload-pack: not our ref $object_id\" err\n+\t)\n+'\n+\n+# Test fetch object-info with 'file://' transport\n+\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init server &&\n+\ttest_commit -C server message1 a.txt &&\n+\tgit -C server config protocol.version 2\n+'\n+\n+test_expect_success 'fetch object-info with file:// using protocol v2' '\n+\t(\n+\t\tcd server &&\n+\t\tobject_id=$(git rev-parse message1:a.txt) &&\n+\t\tlength=$(wc -c expect &&\n+\t\tgit -c protocol.version=2 fetch --object-info=size \"file://$(pwd)\" \"$object_id\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex a0297b0986..64c175e904 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -686,13 +686,17 @@ static int fetch_refs(struct transport *transport,\n \n \t/*\n \t * If we reach here, then the server, the client, and/or the transport\n-\t * helper does not support protocol v2. --negotiate-only requires\n-\t * protocol v2.\n+\t * helper does not support protocol v2. --negotiate-only and --object-info\n+\t * require protocol v2.\n \t */\n \tif (data->transport_options.acked_commits) {\n \t\twarning(_(\"--negotiate-only requires protocol v2\"));\n \t\treturn -1;\n \t}\n+\tif (transport->smart_options->object_info) {\n+\t\twarning(_(\"--object-info requires protocol v2\"));\n+\t\treturn -1;\n+\t}\n \n \tif (!data->get_refs_list_called)\n \t\tget_refs_list_using_list(transport, 0);\ndiff --git a/transport-internal.h b/transport-internal.h\nindex c4ca0b733a..04fa015011 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -59,6 +59,7 @@ struct transport_vtable {\n \t * use. disconnect() releases these resources.\n \t **/\n \tint (*disconnect)(struct transport *connection);\n+\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n };\n \n #endif\ndiff --git a/transport.c b/transport.c\nindex 70e9840a90..65a1b1fdb3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \treturn refs;\n }\n \n+/*\n+ * Fetches object info if server supports object-info command\n+ * Make sure to fallback to normal fetch if this fails\n+ */\n+static int fetch_object_info(struct transport *transport)\n+{\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct object_info_args args;\n+\tstruct packet_reader reader;\n+\n+\tmemset(&args, 0, sizeof(args));\n+\targs.server_options = transport->server_options;\n+\targs.object_info_options = transport->smart_options->object_info_options;\n+\targs.oids = transport->smart_options->object_info_oids;\n+\n+\tconnect_setup(transport, 0);\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\tPACKET_READ_CHOMP_NEWLINE |\n+\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n+\tdata->version = discover_version(&reader);\n+\n+\ttransport->hash_algo = reader.hash_algo;\n+\n+\tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tif (!server_supports_v2(\"object-info\", 0))\n+\t\t\treturn 0;\n+\t\tsend_object_info_request(data->fd[1], &args);\n+\t\tbreak;\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tdie(_(\"wrong protocol version. expected v2\"));\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n+\t\tdie(_(\"error reading object info header\"));\n+\t}\n+\tif (strcmp(reader.line, \"size\")) {\n+\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n+\t}\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\t\tprintf(\"%s\\n\", reader.line);\n+\t}\n+\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\n+\treturn 1;\n+}\n+\n+static int append_oid_to_ref(const struct object_id *oid, void *data)\n+{\n+\tstruct ref *ref = data;\n+\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n+\ttemp_ref->old_oid = *oid;\n+\ttemp_ref->exact_oid = 1;\n+\tref->next = temp_ref;\n+\tref = ref->next;\n+\treturn 0;\n+}\n+\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n \t\t\t\t\tstruct transport_ls_refs_options *options)\n {\n@@ -364,6 +425,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tstruct ref *refs = NULL;\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n+\tstruct ref *wanted_refs = xcalloc(1, sizeof (struct ref));\n \n \tmemset(&args, 0, sizeof(args));\n \targs.uploadpack = data->options.uploadpack;\n@@ -388,8 +450,17 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.server_options = transport->server_options;\n \targs.negotiation_tips = data->options.negotiation_tips;\n \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n+\targs.object_info = transport->smart_options->object_info;\n \n-\tif (!data->got_remote_heads) {\n+\tif (transport->smart_options && transport->smart_options->object_info) {\n+\t\tstruct ref *temp_ref = wanted_refs;\n+\t\tif (fetch_object_info(transport)) {\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t\toid_array_for_each(transport->smart_options->object_info_oids, append_oid_to_ref, temp_ref);\n+\t\twanted_refs = wanted_refs->next;\n+\t\ttransport->remote_refs = wanted_refs;\n+\t} else if (!data->got_remote_heads) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n \t\tfor (i = 0; i < nr_heads; i++) {\n@@ -406,7 +477,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \telse if (data->version <= protocol_v1)\n \t\tdie_if_server_options(transport);\n \n-\tif (data->options.acked_commits) {\n+\tif (data->options.acked_commits && !transport->smart_options->object_info) {\n \t\tif (data->version < protocol_v2) {\n \t\t\twarning(_(\"--negotiate-only requires protocol v2\"));\n \t\t\tret = -1;\ndiff --git a/transport.h b/transport.h\nindex a0bc6a1e9e..3bba16f86e 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -30,6 +30,12 @@ struct git_transport_options {\n \t */\n \tunsigned connectivity_checked:1;\n \n+\t/*\n+\t * Transport will attempt to pull only object-info. Fallbacks\n+\t * to pulling entire object if object-info is not supported\n+\t */\n+\tunsigned object_info : 1;\n+\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n@@ -53,6 +59,9 @@ struct git_transport_options {\n \t * common commits to this oidset instead of fetching any packfiles.\n \t */\n \tstruct oidset *acked_commits;\n+\n+\tstruct oid_array *object_info_oids;\n+\tstruct string_list *object_info_options;\n };\n \n enum transport_family {\n-- \n2.33.0.664.g0785eb7698\n\n"},{"id":"452600","messageId":"xmqq7d8cttup.fsf@gitster.g","threadId":"57388","inReplyTo":"20220328191112.3092139-4-calvinwan@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-29T19:57:18Z","receivedAt":"2022-03-29T19:57:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> diff --git a/transport-internal.h b/transport-internal.h\n> index c4ca0b733a..04fa015011 100644\n> --- a/transport-internal.h\n> +++ b/transport-internal.h\n> @@ -59,6 +59,7 @@ struct transport_vtable {\n> \t * use. disconnect() releases these resources.\n> \t **/\n> \tint (*disconnect)(struct transport *connection);\n> +\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n> };\n> \n> #endif\n\nWe need something like this squashed in in order to pass \"make hdr-check\".\n\n transport-internal.h | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/transport-internal.h b/transport-internal.h\nindex 04fa015011..1416c9197b 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -5,6 +5,7 @@ struct ref;\n struct transport;\n struct strvec;\n struct transport_ls_refs_options;\n+struct oid_array;\n \n struct transport_vtable {\n \t/**\n-- \n2.35.1-887-g427ebd35eb\n\n"},{"id":"452606","messageId":"xmqq35j0ts2j.fsf@gitster.g","threadId":"57388","inReplyTo":"20220328191112.3092139-1-calvinwan@google.com","subject":"Re: [PATCH v3 0/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-29T20:35:48Z","receivedAt":"2022-03-29T20:35:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Sometimes it is useful to get information about an object without\n> having to download it completely. The server logic has already been\n> implemented as “a2ba162cda (object-info: support for retrieving object\n> info, 2021-04-20)”. This patch implements the client option for it.\n>\n> Add ‘--object-info’ option to fetch. This option allows the client to\n> make an object-info command request to a server that supports protocol\n> v2. If the server is v2, but does not allow for the object-info\n> command request, the entire object is fetched instead and the relevant\n> information is returned.\n>\n> Patch 1 is a small refactor. Patch 2 adds a config option for\n> advertisement of object-info. Patch 3 is the main implementation of\n> fetch object-info.\n>\n> Major changes since v2:\n> * Added a fallback for servers that do not\n> allow object-info\n> * Refactored the fetch object-info code by moving it to fetch_refs in\n> the vtable rather than having its own function in the vtable. This\n> allows for the code to easily transition to the fallback if\n> object-info fails.\n>\n> Calvin Wan (3):\n> fetch-pack: refactor how packet is written to be more generic\n> objectinfo.advertise: add config option to advertise object-info\n> object-info: add option for retrieving object info\n>\n> Documentation/config/transfer.txt | 4 +\n> Documentation/fetch-options.txt | 5 ++\n> Documentation/git-fetch.txt | 1 +\n> builtin/fetch.c | 36 +++++++-\n> fetch-pack.c | 66 ++++++++++----\n> fetch-pack.h | 9 ++\n> protocol-caps.c | 11 +++\n> protocol-caps.h | 6 ++\n> serve.c | 2 +-\n> t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++\n> transport-helper.c | 8 +-\n> transport-internal.h | 1 +\n> transport.c | 75 +++++++++++++++-\n> transport.h | 9 ++\n> 14 files changed, 349 insertions(+), 22 deletions(-)\n> create mode 100755 t/t5583-fetch-object-info.sh\n>\n>\n> base-commit: f01e51a7cfd75131b7266131b1f7540ce0a8e5c1\n\nI am getting the following failures when they are queued on this\nbase, or on a more recent tip of 'master/main'.\n\nTest Summary Report\n-------------------\nt5555-http-smart-common.sh (Wstat: 256 Tests: 10 Failed: 1)\n Failed test: 9\n Non-zero exit status: 1\nt5583-fetch-object-info.sh (Wstat: 256 Tests: 11 Failed: 1)\n Failed test: 5\n Non-zero exit status: 1\nt5701-git-serve.sh (Wstat: 256 Tests: 20 Failed: 2)\n Failed tests: 1, 20\n Non-zero exit status: 1\n"},{"id":"452614","messageId":"xmqqy20sqtfe.fsf@gitster.g","threadId":"57388","inReplyTo":"20220328191112.3092139-3-calvinwan@google.com","subject":"Re: [PATCH v3 2/3] transfer.advertiseObjectInfo: add object-info config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-29T22:34:45Z","receivedAt":"2022-03-29T22:34:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Currently, object-info is always advertised by the server. Add a\n> config option to optionally advertise it. A subsequent patch uses this\n> option for testing.\n>\n> ---\n> Documentation/config/transfer.txt | 4 ++++\n> protocol-caps.c | 11 +++++++++++\n> protocol-caps.h | 6 ++++++\n> serve.c | 2 +-\n> 4 files changed, 22 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\n> index b49429eb4d..14892a3155 100644\n> --- a/Documentation/config/transfer.txt\n> +++ b/Documentation/config/transfer.txt\n> @@ -77,3 +77,7 @@ transfer.unpackLimit::\n> transfer.advertiseSID::\n> \tBoolean. When true, client and server processes will advertise their\n> \tunique session IDs to their remote counterpart. Defaults to false.\n> +\n> +transfer.advertiseObjectInfo::\n> +\tBoolean. When true or not set, server processes will advertise its\n> +\tability to accept `object-info` command requests\n> \\ No newline at end of file\n\nCarefully proofread before sending your changes out ;-)\n\nBy mimicking the style of the previous item more closely, i.e.\n\n\tBoolean. When true, ... requests. Defaults to true.\n\nit would make it easier to read, I suspect.\n\n> diff --git a/protocol-caps.c b/protocol-caps.c\n> index bbde91810a..f290e3cca2 100644\n> --- a/protocol-caps.c\n> +++ b/protocol-caps.c\n> @@ -8,6 +8,9 @@\n> #include \"object-store.h\"\n> #include \"string-list.h\"\n> #include \"strbuf.h\"\n> +#include \"config.h\"\n> +\n> +static int advertise_object_info = -1;\n\nIt strikes me somewhat odd that this is the _first_ such capability\nthat needs a variable here to control if it is advertised or not\ndepending on the configuration file settings.\n\nIn other words, \"Why doesn't transfer.advertiseSID also have a\nsimilar thing in this file?\" should be the first question that comes\nto any reviewers' mind.\n\nPerhaps we just invented a better way to handle such conditional\ncapability, in which case we may want to port the existing ones that\ndo not use the same arrangement over to the new way. Even though it\nwould be outside the scope of the series, it would be a good way to\nmake sure we won't accumulate too much technical debt to leave some\nhint somewhere near here or in protoco-caps.h where this patch\ntouches. I cannot quite tell what is going on.\n\n> int object_info_advertise(struct repository *r, struct strbuf *value)\n> {\n> \tif (advertise_object_info == -1 &&\n> \t git_config_get_bool(\"transfer.advertiseObjectInfo\", &advertise_object_info))\n> \t\tadvertise_object_info = 0;\n> \treturn advertise_object_info;\n> }\n\n\"If unset(yet), try to get_bool() and if it fails, set it to 0\"\n\nSo advertise_object_info becomes true only if transfer.advertiseObjectInfo\nis set to true?\n\nIt is inconsistent with what we read in the documentation, isn't it?\n\n> diff --git a/protocol-caps.h b/protocol-caps.h\n> index 15c4550360..36f7a46b37 100644\n> --- a/protocol-caps.h\n> +++ b/protocol-caps.h\n> @@ -5,4 +5,10 @@ struct repository;\n> struct packet_reader;\n> int cap_object_info(struct repository *r, struct packet_reader *request);\n> \n> +/*\n> + * Advertises object-info capability if \"objectinfo.advertise\" is either\n> + * set to true or not set\n> + */\n> +int object_info_advertise(struct repository *r, struct strbuf *value);\n> +\n> #endif /* PROTOCOL_CAPS_H */\n> diff --git a/serve.c b/serve.c\n> index b3fe9b5126..fb4ad367a7 100644\n> --- a/serve.c\n> +++ b/serve.c\n> @@ -133,7 +133,7 @@ static struct protocol_capability capabilities[] = {\n> \t},\n> \t{\n> \t\t.name = \"object-info\",\n> -\t\t.advertise = always_advertise,\n> +\t\t.advertise = object_info_advertise,\n> \t\t.command = cap_object_info,\n> \t},\n> };\n\nThis is a good step to add a test to see if the server does honor\nthe (1) lack of (2) true set to and (3) false set to the\nconfiguration variable and sends (or does not send) the capability.\n"},{"id":"452616","messageId":"CAFySSZDs0k-DVfmTQPnrgwhn3f+57rdc0tq-c-kGh5_KJjXx1Q@mail.gmail.com","threadId":"57388","inReplyTo":"xmqq35j0ts2j.fsf@gitster.g","subject":"Re: [PATCH v3 0/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-29T22:40:29Z","receivedAt":"2022-03-29T22:40:44Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"I seem to have introduced a small bug with the advertisement while I\nwas making my final changes. Will re-roll later today\n\nOn Tue, Mar 29, 2022 at 1:35 PM Junio C Hamano wrote:\n>\n> Calvin Wan writes:\n>\n> > Sometimes it is useful to get information about an object without\n> > having to download it completely. The server logic has already been\n> > implemented as “a2ba162cda (object-info: support for retrieving object\n> > info, 2021-04-20)”. This patch implements the client option for it.\n> >\n> > Add ‘--object-info’ option to fetch. This option allows the client to\n> > make an object-info command request to a server that supports protocol\n> > v2. If the server is v2, but does not allow for the object-info\n> > command request, the entire object is fetched instead and the relevant\n> > information is returned.\n> >\n> > Patch 1 is a small refactor. Patch 2 adds a config option for\n> > advertisement of object-info. Patch 3 is the main implementation of\n> > fetch object-info.\n> >\n> > Major changes since v2:\n> > * Added a fallback for servers that do not\n> > allow object-info\n> > * Refactored the fetch object-info code by moving it to fetch_refs in\n> > the vtable rather than having its own function in the vtable. This\n> > allows for the code to easily transition to the fallback if\n> > object-info fails.\n> >\n> > Calvin Wan (3):\n> > fetch-pack: refactor how packet is written to be more generic\n> > objectinfo.advertise: add config option to advertise object-info\n> > object-info: add option for retrieving object info\n> >\n> > Documentation/config/transfer.txt | 4 +\n> > Documentation/fetch-options.txt | 5 ++\n> > Documentation/git-fetch.txt | 1 +\n> > builtin/fetch.c | 36 +++++++-\n> > fetch-pack.c | 66 ++++++++++----\n> > fetch-pack.h | 9 ++\n> > protocol-caps.c | 11 +++\n> > protocol-caps.h | 6 ++\n> > serve.c | 2 +-\n> > t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++\n> > transport-helper.c | 8 +-\n> > transport-internal.h | 1 +\n> > transport.c | 75 +++++++++++++++-\n> > transport.h | 9 ++\n> > 14 files changed, 349 insertions(+), 22 deletions(-)\n> > create mode 100755 t/t5583-fetch-object-info.sh\n> >\n> >\n> > base-commit: f01e51a7cfd75131b7266131b1f7540ce0a8e5c1\n>\n> I am getting the following failures when they are queued on this\n> base, or on a more recent tip of 'master/main'.\n>\n> Test Summary Report\n> -------------------\n> t5555-http-smart-common.sh (Wstat: 256 Tests: 10 Failed: 1)\n> Failed test: 9\n> Non-zero exit status: 1\n> t5583-fetch-object-info.sh (Wstat: 256 Tests: 11 Failed: 1)\n> Failed test: 5\n> Non-zero exit status: 1\n> t5701-git-serve.sh (Wstat: 256 Tests: 20 Failed: 2)\n> Failed tests: 1, 20\n> Non-zero exit status: 1\n"},{"id":"452623","messageId":"CAFySSZB5aTAsYn=Oqd_hUCQu+X1+GCgpvscLYqk+qzGQc3sAVw@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqy20sqtfe.fsf@gitster.g","subject":"Re: [PATCH v3 2/3] transfer.advertiseObjectInfo: add object-info config","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-29T22:48:28Z","receivedAt":"2022-03-29T22:48:43Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> It strikes me somewhat odd that this is the _first_ such capability\n> that needs a variable here to control if it is advertised or not\n> depending on the configuration file settings.\n\n> In other words, \"Why doesn't transfer.advertiseSID also have a\n> similar thing in this file?\" should be the first question that comes\n> to any reviewers' mind.\nIt doesn't exist in protocol-caps.c, but instead in serve.c. Would be\na good discussion for whether the advertise function should be serve.c\nor protocol-caps.c\n\n> \"If unset(yet), try to get_bool() and if it fails, set it to 0\"\n\n> So advertise_object_info becomes true only if transfer.advertiseObjectInfo\n> is set to true?\nThis was the bug I introduced and will reroll\n\nOn Tue, Mar 29, 2022 at 3:34 PM Junio C Hamano wrote:\n>\n> Calvin Wan writes:\n>\n> > Currently, object-info is always advertised by the server. Add a\n> > config option to optionally advertise it. A subsequent patch uses this\n> > option for testing.\n> >\n> > ---\n> > Documentation/config/transfer.txt | 4 ++++\n> > protocol-caps.c | 11 +++++++++++\n> > protocol-caps.h | 6 ++++++\n> > serve.c | 2 +-\n> > 4 files changed, 22 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt\n> > index b49429eb4d..14892a3155 100644\n> > --- a/Documentation/config/transfer.txt\n> > +++ b/Documentation/config/transfer.txt\n> > @@ -77,3 +77,7 @@ transfer.unpackLimit::\n> > transfer.advertiseSID::\n> > Boolean. When true, client and server processes will advertise their\n> > unique session IDs to their remote counterpart. Defaults to false.\n> > +\n> > +transfer.advertiseObjectInfo::\n> > + Boolean. When true or not set, server processes will advertise its\n> > + ability to accept `object-info` command requests\n> > \\ No newline at end of file\n>\n> Carefully proofread before sending your changes out ;-)\n>\n> By mimicking the style of the previous item more closely, i.e.\n>\n> Boolean. When true, ... requests. Defaults to true.\n>\n> it would make it easier to read, I suspect.\n>\n> > diff --git a/protocol-caps.c b/protocol-caps.c\n> > index bbde91810a..f290e3cca2 100644\n> > --- a/protocol-caps.c\n> > +++ b/protocol-caps.c\n> > @@ -8,6 +8,9 @@\n> > #include \"object-store.h\"\n> > #include \"string-list.h\"\n> > #include \"strbuf.h\"\n> > +#include \"config.h\"\n> > +\n> > +static int advertise_object_info = -1;\n>\n> It strikes me somewhat odd that this is the _first_ such capability\n> that needs a variable here to control if it is advertised or not\n> depending on the configuration file settings.\n>\n> In other words, \"Why doesn't transfer.advertiseSID also have a\n> similar thing in this file?\" should be the first question that comes\n> to any reviewers' mind.\n>\n> Perhaps we just invented a better way to handle such conditional\n> capability, in which case we may want to port the existing ones that\n> do not use the same arrangement over to the new way. Even though it\n> would be outside the scope of the series, it would be a good way to\n> make sure we won't accumulate too much technical debt to leave some\n> hint somewhere near here or in protoco-caps.h where this patch\n> touches. I cannot quite tell what is going on.\n>\n> > int object_info_advertise(struct repository *r, struct strbuf *value)\n> > {\n> > if (advertise_object_info == -1 &&\n> > git_config_get_bool(\"transfer.advertiseObjectInfo\", &advertise_object_info))\n> > advertise_object_info = 0;\n> > return advertise_object_info;\n> > }\n>\n> \"If unset(yet), try to get_bool() and if it fails, set it to 0\"\n>\n> So advertise_object_info becomes true only if transfer.advertiseObjectInfo\n> is set to true?\n>\n> It is inconsistent with what we read in the documentation, isn't it?\n>\n> > diff --git a/protocol-caps.h b/protocol-caps.h\n> > index 15c4550360..36f7a46b37 100644\n> > --- a/protocol-caps.h\n> > +++ b/protocol-caps.h\n> > @@ -5,4 +5,10 @@ struct repository;\n> > struct packet_reader;\n> > int cap_object_info(struct repository *r, struct packet_reader *request);\n> >\n> > +/*\n> > + * Advertises object-info capability if \"objectinfo.advertise\" is either\n> > + * set to true or not set\n> > + */\n> > +int object_info_advertise(struct repository *r, struct strbuf *value);\n> > +\n> > #endif /* PROTOCOL_CAPS_H */\n> > diff --git a/serve.c b/serve.c\n> > index b3fe9b5126..fb4ad367a7 100644\n> > --- a/serve.c\n> > +++ b/serve.c\n> > @@ -133,7 +133,7 @@ static struct protocol_capability capabilities[] = {\n> > },\n> > {\n> > .name = \"object-info\",\n> > - .advertise = always_advertise,\n> > + .advertise = object_info_advertise,\n> > .command = cap_object_info,\n> > },\n> > };\n>\n> This is a good step to add a test to see if the server does honor\n> the (1) lack of (2) true set to and (3) false set to the\n> configuration variable and sends (or does not send) the capability.\n"},{"id":"452625","messageId":"xmqqmth8qsiv.fsf@gitster.g","threadId":"57388","inReplyTo":"20220328191112.3092139-4-calvinwan@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-29T22:54:16Z","receivedAt":"2022-03-29T22:54:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Add ‘--object-info’ option to fetch. This option allows the client to\n> make an object-info command request to a server that supports protocol\n> v2. If the server is v2, but does not allow for the object-info\n> command request, fetch the objects as if it were a standard fetch\n> (however without changing any refs). Therefore, hook `fetch\n> object-info` into transport_fetch_refs() to easily fallback if needed.\n\nSorry, but I do not see anything common between \"git fetch\" that\nfetches history and associated objects and \"retrieving only object\ninfo\". Other than \"the way I happened to have chosen to implement,\nwhat is used to implement fetch has the most commonality\".\n\nIf we were to add more \"server offloading\", say \"run blame on the\nserver side\", are we going to piggy back over fetch-pack, too?\n\nIt is not necessarily bad way to experiment, to reuse the code to\nestablish connection, check if the other side is capable to serve\nus, throw args at them and then retrieve the result, but exposing\nthat implemention detail makes a HORRIBLE UX to force users to say\n\"git fetch --blame='master Makefile' origin\".\n\nShouldn't we be making the part that we truly need to reuse into a\nseparate API out of fetch-pack and either (1) allow new commands be\neasily written reusing the code to create \"git remote-object-info\"\nand \"git remote-blame\", or (2) come up with a single \"do things on\nremote\" command with various subcommands, i.e. \"git over-there\nobject-info\" and \"git over-there blame\"?\n\n> A previous patch added the `transfer.advertiseObjectInfo` config\n> option, of which this patch utilizes to test the fallback.\n>\n> ---\n\nMissing sign-off.\n\n> diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\n> index 550c16ca61..a13d2ba7ad 100644\n> --- a/Documentation/git-fetch.txt\n> +++ b/Documentation/git-fetch.txt\n> @@ -13,6 +13,7 @@ SYNOPSIS\n> 'git fetch' [] \n> 'git fetch' --multiple [] [( | )...]\n> 'git fetch' --all []\n> +'git fetch' --object-info=[] []\n\nThis is a start of slippery slope of making \"fetch\" a kitchen sink\nthat does not have much to do with \"git fetch\". Let's not go this\nroute.\n\n> @@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> \tint prune_tags_ok = 1;\n> +\tstruct oid_array object_info_oids = OID_ARRAY_INIT;\n> \n> \tpacket_trace_identity(\"fetch\");\n> \n> @@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n> \n> -\tif (all) {\n> +\tif (object_info_options.nr > 0) {\n> +\t\tif (argc == 0 || argc == 1) {\n> +\t\t\tdie(_(\"must supply remote and object ids when using --object-info\"));\n> +\t\t} else {\n> +\t\t\tstruct object_id oid;\n> +\t\t\tremote = remote_get(argv[0]);\n> +\t\t\tfor (i = 1; i < argc; i++) {\n> +\t\t\t\tif (get_oid(argv[i], &oid))\n> +\t\t\t\t\treturn error(_(\"malformed object id '%s'\"), argv[i]);\n> +\t\t\t\toid_array_append(&object_info_oids, &oid);\n> +\t\t\t}\n> +\t\t}\n> +\t} else if (all) {\n> \t\tif (argc == 1)\n> \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> \t\telse if (argc > 1)\n\nYuck. Let's read this again, realize how little commonality we have\nin processing args and barf. That's all because we tried to\npiggy-back on \"fetch\" because the patch wanted to reuse the parts\nthat are not shown in the patch text. The right way to do so would\nbe to factor that \"part that do not appear in the patch, because\nthey are common\" out into callable from new code. That way, we do\nnot have to contaminate this if/else conditional, both arms of which\nis still about running \"fetch\" to retrieve history and connected\nobjects, and has nothing to do with \"fetch object info\". The way\nthis patch changes this part of the code is an unnecessary risk to\nbreak the normal \"git fetch\", and it does not have to be.\n\nI like the idea to give client side a way to ask server to do random\nthings. I do not think I want to see unrelated things piggy-backing\non existing client programs only because that is an expedient way to\nimplement it.\n"},{"id":"452626","messageId":"xmqqfsn0qsi4.fsf@gitster.g","threadId":"57388","inReplyTo":"20220328191112.3092139-2-calvinwan@google.com","subject":"Re: [PATCH v3 1/3] fetch-pack: refactor packet writing and fetch options","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-29T22:54:43Z","receivedAt":"2022-03-29T22:54:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> -static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n> -\t\t\t\t\t\t const struct string_list *server_options)\n> +static void write_command_and_capabilities(struct strbuf *req_buf,\n> +\t\t\t\t\t\t const struct string_list *server_options, const char* command)\n\nReindent the second line to take advantage of the fact that the\nfunction name is now slightly shorter?\n\nIt is named \"command\" and \"capabilities\", but the parameters it\ntakes has the \"command\" at the end. We probably want to swap them,\ni.e.\n\nstatic void write_command_and_capabilities(struct strbuf *req_buf,\n\t\t\t\t\t const char *command,\n\t\t\t\t\t const struct string_list *server_options)\n\t\nNote that asterisk sticks to identifier, not to type, in this code base.\n\n> {\n> \tconst char *hash_name;\n> \n> -\tif (server_supports_v2(\"fetch\", 1))\n> -\t\tpacket_buf_write(req_buf, \"command=fetch\");\n> +\tif (server_supports_v2(command, 1))\n> +\t\tpacket_buf_write(req_buf, \"command=%s\", command);\n> \tif (server_supports_v2(\"agent\", 0))\n> \t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n> \tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n> @@ -1279,7 +1279,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> \tint done_sent = 0;\n> \tstruct strbuf req_buf = STRBUF_INIT;\n> \n> -\twrite_fetch_command_and_capabilities(&req_buf, args->server_options);\n> +\twrite_command_and_capabilities(&req_buf, args->server_options, \"fetch\");\n> \n> \tif (args->use_thin_pack)\n> \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n\nThe above two hunks (and there is another one at the end) are\ntrivial no-op refactoring, and the first paragraph of the proposed\nlog message clearly explains it.\n\n> @@ -1598,18 +1598,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\treader.me = \"fetch-pack\";\n> \t}\n> \n> +\t/* v2 supports these by default */\n> +\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n> +\tuse_sideband = 2;\n> +\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> +\t\targs->deepen = 1;\n> +\n> \twhile (state != FETCH_DONE) {\n> \t\tswitch (state) {\n> \t\tcase FETCH_CHECK_LOCAL:\n> \t\t\tsort_ref_list(&ref, ref_compare_name);\n> \t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n> \n> -\t\t\t/* v2 supports these by default */\n> -\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n> -\t\t\tuse_sideband = 2;\n> -\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> -\t\t\t\targs->deepen = 1;\n> -\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n\nThe second paragraph of the proposed log message touched briefly\nabout the existence of this change. It is a no-op for the current\ncode if two conditions are met: (1) the initial state is CHECK_LOCAL,\n(2) we only enter CHECK_LOCAL once and never return to it. And they\nboth hold true, so this is a no-op change.\n\nBut I am not sure if it is a good idea to have this in this patch.\nPossibilities are (1) this patch as-is is OK, (2) move it in a\nseparate patch as it has nothing to do with the other refactoring,\nor (3) do it as part of the change that wants this change.\n\nI may form an opinion once I see the step that wants this change,\nbut not yet.\n\n> @@ -2060,7 +2060,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n> \t\tint received_ready = 0;\n> \n> \t\tstrbuf_reset(&req_buf);\n> -\t\twrite_fetch_command_and_capabilities(&req_buf, server_options);\n> +\t\twrite_command_and_capabilities(&req_buf, server_options, \"fetch\");\n> \n> \t\tpacket_buf_write(&req_buf, \"wait-for-done\");\n"},{"id":"452627","messageId":"YkOPyc9tUfe2Tozx@nand.local","threadId":"57388","inReplyTo":"20220328191112.3092139-2-calvinwan@google.com","subject":"Re: [PATCH v3 1/3] fetch-pack: refactor packet writing and fetch options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-03-29T23:01:29Z","receivedAt":"2022-03-29T23:02:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 28, 2022 at 07:11:10PM +0000, Calvin Wan wrote:\n> A subsequent commit will need to write capabilities for another\n> command, so refactor write_fetch_command_and_capabilities() to be able\n> to used by both fetch and future command.\n\nMakes obvious sense, and this was something that jumped out to me when I\nlooked at the first and second versions of this patch. I'm glad that\nthis is getting factored out.\n\n> Move fetch options set in `FETCH_CHECK_LOCAL` from the fetch state\n> machine to above the state machine so it is set by default. The\n> initial state of the state machine is always `FETCH_CHECK_LOCAL` so\n> this does not affect any current functionality. This change prepares\n> for a subsequent commit that doesn't need to check the local state, but\n> still requires those options to be set before sending the fetch request.\n\nThis took a little more thinking, but I agree that this is behaviorally\nequivalent. The key points are (and here I'm basically restating what\nyou already wrote):\n\n - when the state machine in do_fetch_pack_v2() is in\n FETCH_CHECK_LOCAL, we set a few v2-specific defaults\n\n - it will be helpful for a future patch to have these defaults set\n _before_ we enter the main loop\n\n - and we can safely make that change, since the initial state is\n always FETCH_CHECK_LOCAL anyway, so we're guaranteed to hit that\n code whether it's in the loop or not.\n\nSo this seems safe to me, too. I'll have to see why this is necessary,\nbut I suspect we'll find out in subsequent patches, so let's read on...\n\nThanks,\nTaylor\n"},{"id":"452629","messageId":"YkORvY3duS7dQRRY@nand.local","threadId":"57388","inReplyTo":"CAFySSZB5aTAsYn=Oqd_hUCQu+X1+GCgpvscLYqk+qzGQc3sAVw@mail.gmail.com","subject":"Re: [PATCH v3 2/3] transfer.advertiseObjectInfo: add object-info config","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-03-29T23:09:49Z","receivedAt":"2022-03-29T23:09:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 29, 2022 at 03:48:28PM -0700, Calvin Wan wrote:\n> > It strikes me somewhat odd that this is the _first_ such capability\n> > that needs a variable here to control if it is advertised or not\n> > depending on the configuration file settings.\n>\n> > In other words, \"Why doesn't transfer.advertiseSID also have a\n> > similar thing in this file?\" should be the first question that comes\n> > to any reviewers' mind.\n>\n> It doesn't exist in protocol-caps.c, but instead in serve.c. Would be\n> a good discussion for whether the advertise function should be serve.c\n> or protocol-caps.c\n\nMost of the others are defined statically within serve.c, but there are\na couple of exceptions:\n\n - `ls_refs_advertise()` is defined in ls-refs.c, and\n - `upload_pack_advertise()` is defined in upload-pack.c.\n\nEven though cap_object_info() is defined in protocol-caps.c, I think we\nshould probably keep the definition of `object_info_advertise()` local\nto serve.c.\n\nIf there were a object-info.c, it would make sense to put it there, but\nsince there isn't, it makes more sense to keep it local to serve.c,\nwhich avoids us having to declare the function in a header file.\n\nThanks,\nTaylor\n"},{"id":"452630","messageId":"YkOT/TCKPK3LT9gh@nand.local","threadId":"57388","inReplyTo":"20220328191112.3092139-4-calvinwan@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-03-29T23:19:25Z","receivedAt":"2022-03-29T23:19:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Calvin,\n\nJunio took a deeper look below, but here are a few small code-hygiene\nissues that I noticed while taking a quicker look through this patch.\n\nOn Mon, Mar 28, 2022 at 07:11:12PM +0000, Calvin Wan wrote:\n> Sometimes it is useful to get information about an object without\n> having to download it completely. The server logic has already been\n> implemented as “a2ba162cda (object-info: support for retrieving object\n> info, 2021-04-20)”. This patch implements the client option for\n> it. Currently, only 'size' is supported, however future patches can\n> implement additional options.\n>\n> Add ‘--object-info’ option to fetch. This option allows the client to\n\nThe single-quotes here look like smart quotes. I don't think we forbid\nthese explicitly within commit messages, but using the standard '\n(ASCII 0x27) character is more common.\n\n> make an object-info command request to a server that supports protocol\n> v2. If the server is v2, but does not allow for the object-info\n> command request, fetch the objects as if it were a standard fetch\n> (however without changing any refs). Therefore, hook `fetch\n> object-info` into transport_fetch_refs() to easily fallback if needed.\n>\n> A previous patch added the `transfer.advertiseObjectInfo` config\n> option, of which this patch utilizes to test the fallback.\n>\n> ---\n> Documentation/fetch-options.txt | 5 ++\n> Documentation/git-fetch.txt | 1 +\n> builtin/fetch.c | 36 ++++++++-\n> fetch-pack.c | 42 +++++++++-\n> fetch-pack.h | 9 +++\n> t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n> transport-helper.c | 8 +-\n> transport-internal.h | 1 +\n> transport.c | 75 ++++++++++++++++-\n> transport.h | 9 +++\n> 10 files changed, 315 insertions(+), 9 deletions(-)\n> create mode 100755 t/t5583-fetch-object-info.sh\n>\n> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> index f903683189..f1ca95c32d 100644\n> --- a/Documentation/fetch-options.txt\n> +++ b/Documentation/fetch-options.txt\n> @@ -299,3 +299,8 @@ endif::git-pull[]\n> -6::\n> --ipv6::\n> \tUse IPv6 addresses only, ignoring IPv4 addresses.\n> +\n> +--object-info=[]::\n> +\tFetches only the relevant information passed in arguments from a\n> +\tspecific remote and set of object ids. Object 'size' is currently\n> +\tthe only supported argument.\n> diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\n> index 550c16ca61..a13d2ba7ad 100644\n> --- a/Documentation/git-fetch.txt\n> +++ b/Documentation/git-fetch.txt\n> @@ -13,6 +13,7 @@ SYNOPSIS\n> 'git fetch' [] \n> 'git fetch' --multiple [] [( | )...]\n> 'git fetch' --all []\n> +'git fetch' --object-info=[] []\n>\n>\n> DESCRIPTION\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 4d12c2fd4d..0b21bebfca 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -29,6 +29,9 @@\n> #include \"commit-graph.h\"\n> #include \"shallow.h\"\n> #include \"worktree.h\"\n> +#include \"protocol.h\"\n> +#include \"pkt-line.h\"\n> +#include \"connect.h\"\n>\n> #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n>\n> @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> \tN_(\"git fetch [] \"),\n> \tN_(\"git fetch --multiple [] [( | )...]\"),\n> \tN_(\"git fetch --all []\"),\n> +\tN_(\"git fetch --object-info=[] []\"),\n> \tNULL\n> };\n>\n> @@ -86,6 +90,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n> static int fetch_write_commit_graph = -1;\n> static int stdin_refspecs = 0;\n> static int negotiate_only;\n> +static struct string_list object_info_options = STRING_LIST_INIT_NODUP;\n>\n> static int git_fetch_config(const char *k, const char *v, void *cb)\n> {\n> @@ -221,6 +226,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t N_(\"write the commit-graph after fetching\")),\n> \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> \t\t N_(\"accept refspecs from stdin\")),\n> +\tOPT_STRING_LIST(0, \"object-info\", &object_info_options, N_(\"option\"),\n> +\t\t N_(\"command request arguments\")),\n> \tOPT_END()\n> };\n>\n> @@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> \tint prune_tags_ok = 1;\n> +\tstruct oid_array object_info_oids = OID_ARRAY_INIT;\n>\n> \tpacket_trace_identity(\"fetch\");\n>\n> @@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n>\n> -\tif (all) {\n> +\tif (object_info_options.nr > 0) {\n\nSmall nit; since object_info_options.nr should never be negative, we\nshould instead write \"if (object_info_options)\".\n\n> +\t\tif (argc == 0 || argc == 1) {\n> +\t\t\tdie(_(\"must supply remote and object ids when using --object-info\"));\n> +\t\t} else {\n> +\t\t\tstruct object_id oid;\n> +\t\t\tremote = remote_get(argv[0]);\n> +\t\t\tfor (i = 1; i < argc; i++) {\n> +\t\t\t\tif (get_oid(argv[i], &oid))\n> +\t\t\t\t\treturn error(_(\"malformed object id '%s'\"), argv[i]);\n> +\t\t\t\toid_array_append(&object_info_oids, &oid);\n> +\t\t\t}\n> +\t\t}\n> +\t} else if (all) {\n> \t\tif (argc == 1)\n> \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> \t\telse if (argc > 1)\n> @@ -2199,7 +2219,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \t\t}\n> \t}\n>\n> -\tif (negotiate_only) {\n> +\tif (object_info_options.nr > 0) {\n> +\t\tif (!remote)\n> +\t\t\tdie(_(\"must supply remote when using --object-info\"));\n> +\t\tgtransport = prepare_transport(remote, 1);\n> +\t\tif (gtransport->smart_options) {\n> +\t\t\tgtransport->smart_options->object_info = 1;\n> +\t\t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n> +\t\t\tgtransport->smart_options->object_info_options = &object_info_options;\n> +\t\t}\n> +\t\tif (server_options.nr)\n> +\t\t\tgtransport->server_options = &server_options;\n> +\t\tresult = transport_fetch_refs(gtransport, NULL);\n> +\t} else if (negotiate_only) {\n> \t\tstruct oidset acked_commits = OIDSET_INIT;\n> \t\tstruct oidset_iter iter;\n> \t\tconst struct object_id *oid;\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index b709a61baf..36e3d1c5d0 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n> \tpacket_buf_delim(req_buf);\n> }\n>\n> +void send_object_info_request(int fd_out, struct object_info_args *args)\n> +{\n> +\tstruct strbuf req_buf = STRBUF_INIT;\n> +\tint i;\n\nI don't think this would ever be practically exploit-able, since we'd\nrun past the argument limit long before we hit INT_MAX, but this should\nbe a size_t instead of an int to match the type of args->oid->nr.\n\n> +\n> +\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> +\n> +\tif (string_list_has_string(args->object_info_options, \"size\"))\n> +\t\tpacket_buf_write(&req_buf, \"size\");\n> +\n> +\tfor (i = 0; i < args->oids->nr; i++) {\n> +\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> +\t}\n\nSmall nit, the braces around this for-loop are not required.\n\n> static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> \t\t\t struct fetch_pack_args *args,\n> \t\t\t const struct ref *wants, struct oidset *common,\n> @@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> \t\targs->deepen = 1;\n>\n> +\tif (args->object_info) {\n> +\t\tstate = FETCH_SEND_REQUEST;\n> +\t}\n> +\n\nDitto here.\n\n> \twhile (state != FETCH_DONE) {\n> \t\tswitch (state) {\n> \t\tcase FETCH_CHECK_LOCAL:\n> @@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n> -\t\t\tif (everything_local(args, &ref))\n> +\t\t\tif (!args->object_info && everything_local(args, &ref))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n> @@ -1999,8 +2024,19 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> \t\t}\n> \t\targs->connectivity_checked = 1;\n> \t}\n> -\n> -\tupdate_shallow(args, sought, nr_sought, &si);\n> +\n> +\tif (args->object_info) {\n> +\t\tstruct ref *ref_cpy_reader = ref_cpy;\n> +\t\tunsigned long size = 0;\n> +\t\twhile (ref_cpy_reader) {\n> +\t\t\toid_object_info(the_repository, &(ref_cpy_reader->old_oid), &size);\n> +\t\t\tprintf(\"%s %li\\n\", oid_to_hex(&(ref_cpy_reader->old_oid)), size);\n\nBoth expressions like \"&(x)\" can be written instead as \"&x\" without the\nouter parenthesis.\n\nLikewise, we do not use %li, that line should instead read:\n\n printf(\"%s %\"PRIuMAX\"\\n\", oid_to_hex(&ref_cpy_reader->old_oid), (uintmax_t)size);\n\nThanks,\nTaylor\n"},{"id":"452632","messageId":"xmqq4k3gqn7n.fsf@gitster.g","threadId":"57388","inReplyTo":"xmqqmth8qsiv.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-30T00:49:00Z","receivedAt":"2022-03-30T00:49:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Shouldn't we be making the part that we truly need to reuse into a\n> separate API out of fetch-pack and either (1) allow new commands be\n> easily written reusing the code to create \"git remote-object-info\"\n> and \"git remote-blame\", or (2) come up with a single \"do things on\n> remote\" command with various subcommands, i.e. \"git over-there\n> object-info\" and \"git over-there blame\"?\n\nFor completeness, we could also make the \"git archive\" the gold\nstandard. If we want \"git blame\" that is mostly executed on the\nremote site, we could teach \"git blame --remote \", which may\nbe where a user, who does not know nor care about how things are\nimplemented, would expect to find the feature. Is \"object-info\" in\nprinciple like executing \"cat-file --batch-check\" remotely?\n\nI think that \"Object 'size' is currently the only one the server\nknows to emit\", combined with the lack of any client side support so\nfar, means that it is not too late to rework the object-info thing\nand possibly reuse more from \"cat-file --batch-check\".\n\nIf we can create a system where running things locally and remotely\nare comparable in feature, that would be great, but I do not think\nthat is a realistic goal to aim for.\n\nThoughts?\n\n\n\n"},{"id":"452708","messageId":"20220330215522.2302571-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-2-calvinwan@google.com","subject":"Re: [PATCH v3 1/3] fetch-pack: refactor packet writing and fetch options","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-30T21:55:22Z","receivedAt":"2022-03-30T21:55:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Calvin Wan writes:\n> A subsequent commit will need to write capabilities for another\n> command, so refactor write_fetch_command_and_capabilities() to be able\n> to used by both fetch and future command.\n> \n> Move fetch options set in `FETCH_CHECK_LOCAL` from the fetch state\n> machine to above the state machine so it is set by default. The\n> initial state of the state machine is always `FETCH_CHECK_LOCAL` so\n> this does not affect any current functionality. This change prepares\n> for a subsequent commit that doesn't need to check the local state, but\n> still requires those options to be set before sending the fetch request.\n\nI think it would be clearer to rephrase \"that doesn't need to check the\nlocal state\" to \"that will skip FETCH_CHECK_LOCAL\". It does literally\nmean the same thing, but I had a mental block with \"initial state...is\nalways FETCH_CHECK_LOCAL\" in the preceding sentence.\n\nI agree with some of the others who said that this change should go into\nits own patch.\n\nOtherwise, this patch makes sense.\n"},{"id":"452709","messageId":"15354CB8-E1DD-476C-B6E0-52F9451108A6@gmail.com","threadId":"57388","inReplyTo":"20220328191112.3092139-4-calvinwan@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"John Cai","fromEmail":"johncai86@gmail.com","sentAt":"2022-03-30T22:06:35Z","receivedAt":"2022-03-30T22:06:40Z","isPatch":true,"sender":{"key":"johncai86@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2354211?v=4"},"body":"\n\nOn 28 Mar 2022, at 15:11, Calvin Wan wrote:\n\n> Sometimes it is useful to get information about an object without\n> having to download it completely. The server logic has already been\n> implemented as “a2ba162cda (object-info: support for retrieving object\n> info, 2021-04-20)”. This patch implements the client option for\n> it. Currently, only 'size' is supported, however future patches can\n> implement additional options.\n>\n> Add ‘--object-info’ option to fetch. This option allows the client to\n> make an object-info command request to a server that supports protocol\n> v2. If the server is v2, but does not allow for the object-info\n> command request, fetch the objects as if it were a standard fetch\n> (however without changing any refs). Therefore, hook `fetch\n> object-info` into transport_fetch_refs() to easily fallback if needed.\n>\n> A previous patch added the `transfer.advertiseObjectInfo` config\n> option, of which this patch utilizes to test the fallback.\n>\n> ---\n> Documentation/fetch-options.txt | 5 ++\n> Documentation/git-fetch.txt | 1 +\n> builtin/fetch.c | 36 ++++++++-\n> fetch-pack.c | 42 +++++++++-\n> fetch-pack.h | 9 +++\n> t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n> transport-helper.c | 8 +-\n> transport-internal.h | 1 +\n> transport.c | 75 ++++++++++++++++-\n> transport.h | 9 +++\n> 10 files changed, 315 insertions(+), 9 deletions(-)\n> create mode 100755 t/t5583-fetch-object-info.sh\n>\n> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> index f903683189..f1ca95c32d 100644\n> --- a/Documentation/fetch-options.txt\n> +++ b/Documentation/fetch-options.txt\n> @@ -299,3 +299,8 @@ endif::git-pull[]\n> -6::\n> --ipv6::\n> \tUse IPv6 addresses only, ignoring IPv4 addresses.\n> +\n> +--object-info=[]::\n> +\tFetches only the relevant information passed in arguments from a\n> +\tspecific remote and set of object ids. Object 'size' is currently\n> +\tthe only supported argument.\n> diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\n> index 550c16ca61..a13d2ba7ad 100644\n> --- a/Documentation/git-fetch.txt\n> +++ b/Documentation/git-fetch.txt\n> @@ -13,6 +13,7 @@ SYNOPSIS\n> 'git fetch' [] \n> 'git fetch' --multiple [] [( | )...]\n> 'git fetch' --all []\n> +'git fetch' --object-info=[] []\n>\n>\n> DESCRIPTION\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 4d12c2fd4d..0b21bebfca 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -29,6 +29,9 @@\n> #include \"commit-graph.h\"\n> #include \"shallow.h\"\n> #include \"worktree.h\"\n> +#include \"protocol.h\"\n> +#include \"pkt-line.h\"\n> +#include \"connect.h\"\n>\n> #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n>\n> @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> \tN_(\"git fetch [] \"),\n> \tN_(\"git fetch --multiple [] [( | )...]\"),\n> \tN_(\"git fetch --all []\"),\n> +\tN_(\"git fetch --object-info=[] []\"),\n> \tNULL\n> };\n>\n> @@ -86,6 +90,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n> static int fetch_write_commit_graph = -1;\n> static int stdin_refspecs = 0;\n> static int negotiate_only;\n> +static struct string_list object_info_options = STRING_LIST_INIT_NODUP;\n>\n> static int git_fetch_config(const char *k, const char *v, void *cb)\n> {\n> @@ -221,6 +226,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t N_(\"write the commit-graph after fetching\")),\n> \tOPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> \t\t N_(\"accept refspecs from stdin\")),\n> +\tOPT_STRING_LIST(0, \"object-info\", &object_info_options, N_(\"option\"),\n> +\t\t N_(\"command request arguments\")),\n> \tOPT_END()\n> };\n>\n> @@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tstruct remote *remote = NULL;\n> \tint result = 0;\n> \tint prune_tags_ok = 1;\n> +\tstruct oid_array object_info_oids = OID_ARRAY_INIT;\n>\n> \tpacket_trace_identity(\"fetch\");\n>\n> @@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (dry_run)\n> \t\twrite_fetch_head = 0;\n>\n> -\tif (all) {\n> +\tif (object_info_options.nr > 0) {\n> +\t\tif (argc == 0 || argc == 1) {\n> +\t\t\tdie(_(\"must supply remote and object ids when using --object-info\"));\n> +\t\t} else {\n> +\t\t\tstruct object_id oid;\n> +\t\t\tremote = remote_get(argv[0]);\n> +\t\t\tfor (i = 1; i < argc; i++) {\n> +\t\t\t\tif (get_oid(argv[i], &oid))\n> +\t\t\t\t\treturn error(_(\"malformed object id '%s'\"), argv[i]);\n> +\t\t\t\toid_array_append(&object_info_oids, &oid);\n> +\t\t\t}\n> +\t\t}\n> +\t} else if (all) {\n> \t\tif (argc == 1)\n> \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> \t\telse if (argc > 1)\n> @@ -2199,7 +2219,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \t\t}\n> \t}\n>\n> -\tif (negotiate_only) {\n> +\tif (object_info_options.nr > 0) {\n> +\t\tif (!remote)\n> +\t\t\tdie(_(\"must supply remote when using --object-info\"));\n> +\t\tgtransport = prepare_transport(remote, 1);\n> +\t\tif (gtransport->smart_options) {\n> +\t\t\tgtransport->smart_options->object_info = 1;\n> +\t\t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n> +\t\t\tgtransport->smart_options->object_info_options = &object_info_options;\n> +\t\t}\n> +\t\tif (server_options.nr)\n> +\t\t\tgtransport->server_options = &server_options;\n> +\t\tresult = transport_fetch_refs(gtransport, NULL);\n> +\t} else if (negotiate_only) {\n> \t\tstruct oidset acked_commits = OIDSET_INIT;\n> \t\tstruct oidset_iter iter;\n> \t\tconst struct object_id *oid;\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index b709a61baf..36e3d1c5d0 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n> \tpacket_buf_delim(req_buf);\n> }\n>\n> +void send_object_info_request(int fd_out, struct object_info_args *args)\n> +{\n> +\tstruct strbuf req_buf = STRBUF_INIT;\n> +\tint i;\n> +\n> +\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> +\n> +\tif (string_list_has_string(args->object_info_options, \"size\"))\n> +\t\tpacket_buf_write(&req_buf, \"size\");\n> +\n> +\tfor (i = 0; i < args->oids->nr; i++) {\n> +\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> +\t}\n> +\n> +\tpacket_buf_flush(&req_buf);\n> +\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n> +\t\tdie_errno(_(\"unable to write request to remote\"));\n> +\n> +\tstrbuf_release(&req_buf);\n> +}\n> +\n> static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> \t\t\t struct fetch_pack_args *args,\n> \t\t\t const struct ref *wants, struct oidset *common,\n> @@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> \t\targs->deepen = 1;\n>\n> +\tif (args->object_info) {\n> +\t\tstate = FETCH_SEND_REQUEST;\n> +\t}\n> +\n> \twhile (state != FETCH_DONE) {\n> \t\tswitch (state) {\n> \t\tcase FETCH_CHECK_LOCAL:\n> @@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n> -\t\t\tif (everything_local(args, &ref))\n> +\t\t\tif (!args->object_info && everything_local(args, &ref))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n> @@ -1999,8 +2024,19 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> \t\t}\n> \t\targs->connectivity_checked = 1;\n> \t}\n> -\n> -\tupdate_shallow(args, sought, nr_sought, &si);\n> +\t\n> +\tif (args->object_info) {\n> +\t\tstruct ref *ref_cpy_reader = ref_cpy;\n> +\t\tunsigned long size = 0;\n> +\t\twhile (ref_cpy_reader) {\n> +\t\t\toid_object_info(the_repository, &(ref_cpy_reader->old_oid), &size);\n> +\t\t\tprintf(\"%s %li\\n\", oid_to_hex(&(ref_cpy_reader->old_oid)), size);\n> +\t\t\tref_cpy_reader = ref_cpy_reader->next;\n> +\t\t}\n> +\t}\n> +\telse {\n> +\t\tupdate_shallow(args, sought, nr_sought, &si);\n> +\t}\n> cleanup:\n> \tclear_shallow_info(&si);\n> \toid_array_clear(&shallows_scratch);\n> diff --git a/fetch-pack.h b/fetch-pack.h\n> index 7f94a2a583..23e69bb218 100644\n> --- a/fetch-pack.h\n> +++ b/fetch-pack.h\n> @@ -42,6 +42,7 @@ struct fetch_pack_args {\n> \tunsigned update_shallow:1;\n> \tunsigned reject_shallow_remote:1;\n> \tunsigned deepen:1;\n> +\tunsigned object_info:1;\n>\n> \t/*\n> \t * Indicate that the remote of this request is a promisor remote. The\n> @@ -68,6 +69,12 @@ struct fetch_pack_args {\n> \tunsigned connectivity_checked:1;\n> };\n>\n> +struct object_info_args {\n> +\tconst struct string_list *object_info_options;\n> +\tconst struct string_list *server_options;\n> +\tconst struct oid_array *oids;\n> +};\n> +\n> /*\n> * sought represents remote references that should be updated from.\n> * On return, the names that were found on the remote will have been\n> @@ -101,4 +108,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n> */\n> int report_unmatched_refs(struct ref **sought, int nr_sought);\n>\n> +void send_object_info_request(int fd_out, struct object_info_args *args);\n> +\n> #endif\n> diff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\n> new file mode 100755\n> index 0000000000..e5db6d953c\n> --- /dev/null\n> +++ b/t/t5583-fetch-object-info.sh\n> @@ -0,0 +1,138 @@\n> +#!/bin/sh\n> +\n> +test_description='test git fetch object-info version 2'\n> +\n> +TEST_NO_CREATE_REPO=1\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +\n> +. ./test-lib.sh\n> +\n> +# Test fetch object-info with 'git://' transport\n> +\n> +. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n> +start_git_daemon --export-all --enable=receive-pack\n> +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n> +\n> +\n> +test_expect_success 'create repo to be served by git-daemon' '\n> +\tgit init \"$daemon_parent\" &&\n> +\ttest_commit -C \"$daemon_parent\" message1 a.txt\n> +'\n> +\n> +test_expect_success 'fetch object-info with git:// using protocol v2' '\n> +\t(\n> +\t\tcd \"$daemon_parent\" &&\n> +\t\tobject_id=$(git rev-parse message1:a.txt) &&\n> +\t\tlength=$(wc -c +\n> +\t\tprintf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> +\t\tgit -c protocol.version=2 fetch --object-info=size \"$GIT_DAEMON_URL/parent\" \"$object_id\" >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +stop_git_daemon\n> +\n> +# Test protocol v2 with 'http://' transport\n> +\n> +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> +start_httpd\n> +\n> +test_expect_success 'create repo to be served by http:// transport' '\n> +\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n> +\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message1 a.txt &&\n> +\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message2 b.txt\n> +'\n> +\n> +test_expect_success 'fetch object-info with http:// using protocol v2' '\n> +\t(\n> +\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\t\tobject_id=$(git rev-parse message1:a.txt) &&\n> +\t\tlength=$(wc -c +\n> +\t\tprintf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> +\t\tgit -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'fetch object-info for multiple objects' '\n> +\t(\n> +\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\t\tobject_id1=$(git rev-parse message1:a.txt) &&\n> +\t\tobject_id2=$(git rev-parse message2:b.txt) &&\n> +\t\tlength1=$(wc -c +\t\tlength2=$(wc -c +\n> +\t\tprintf \"%s %d\\n\" \"$object_id1\" \"$length1\" >expect &&\n> +\t\tprintf \"%s %d\\n\" \"$object_id2\" \"$length2\" >>expect &&\n> +\t\tgit -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id1\" \"$object_id2\" >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'fetch object-info fallbacks to standard fetch if object-info is not supported by the server' '\n> +\t(\n> +\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\t\tobject_id=$(git rev-parse message1:a.txt) &&\n> +\t\tlength=$(wc -c +\n> +\t\tprintf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> +\t\tgit config objectinfo.advertise false &&\n> +\t\tgit -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n\nI noticed this test doesn't confirm that standard fetch was actually used. Would\nit be useful to check for the existence of the object to prove it got fetched?\n\nAlso, is seems the way this test is set up, it's fetching from itself.\nThis is logically fine in most cases that client == server, but it was a little\nconfusing at first because it looked like the invocation of git config was on\nthe client rather than the server, until I realized they are the same.\n\nSince the server is being used as the client, we wouldn't be able to confirm that\nthe object got fetched. I realize for most of these tests, it doesn't really matter\nto conflate client with server. I think this is the one exception.\n\nBut, as a larger question for the list, I'm wondering if it's best practice to separate client from\nserver in tests, or if it's an accepted style to just use one repository in cases\nwhen it doesn't really matter?\n\nThanks\nJohn\n\n> +'\n> +\n> +test_expect_success 'fetch object-info fails on server with legacy protocol' '\n> +\t(\n> +\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\t\tobject_id=$(git rev-parse message1:a.txt) &&\n> +\t\ttest_must_fail git -c protocol.version=0 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" 2>err &&\n> +\t\ttest_i18ngrep \"object-info requires protocol v2\" err\n> +\t)\n> +'\n> +\n> +test_expect_success 'fetch object-info fails on malformed OID' '\n> +\t(\n> +\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\t\tmalformed_object_id=\"this_id_is_not_valid\" &&\n> +\t\ttest_must_fail git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$malformed_object_id\" 2>err &&\n> +\t\ttest_i18ngrep \"malformed object id '$malformed_object_id'\" err\n> +\t)\n> +'\n> +\n> +test_expect_success 'fetch object-info fails on missing OID' '\n> +\tgit clone \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" missing_oid_repo &&\n> +\ttest_commit -C missing_oid_repo message3 c.txt &&\n> +\t(\n> +\t\tcd missing_oid_repo &&\n> +\t\tobject_id=$(git rev-parse message3:c.txt) &&\n> +\t\ttest_must_fail env GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" 2>err &&\n> +\t\ttest_i18ngrep \"fatal: remote error: upload-pack: not our ref $object_id\" err\n> +\t)\n> +'\n> +\n> +# Test fetch object-info with 'file://' transport\n> +\n> +test_expect_success 'create repo to be served by file:// transport' '\n> +\tgit init server &&\n> +\ttest_commit -C server message1 a.txt &&\n> +\tgit -C server config protocol.version 2\n> +'\n> +\n> +test_expect_success 'fetch object-info with file:// using protocol v2' '\n> +\t(\n> +\t\tcd server &&\n> +\t\tobject_id=$(git rev-parse message1:a.txt) &&\n> +\t\tlength=$(wc -c +\n> +\t\tprintf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> +\t\tgit -c protocol.version=2 fetch --object-info=size \"file://$(pwd)\" \"$object_id\" >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_done\n> diff --git a/transport-helper.c b/transport-helper.c\n> index a0297b0986..64c175e904 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -686,13 +686,17 @@ static int fetch_refs(struct transport *transport,\n>\n> \t/*\n> \t * If we reach here, then the server, the client, and/or the transport\n> -\t * helper does not support protocol v2. --negotiate-only requires\n> -\t * protocol v2.\n> +\t * helper does not support protocol v2. --negotiate-only and --object-info\n> +\t * require protocol v2.\n> \t */\n> \tif (data->transport_options.acked_commits) {\n> \t\twarning(_(\"--negotiate-only requires protocol v2\"));\n> \t\treturn -1;\n> \t}\n> +\tif (transport->smart_options->object_info) {\n> +\t\twarning(_(\"--object-info requires protocol v2\"));\n> +\t\treturn -1;\n> +\t}\n>\n> \tif (!data->get_refs_list_called)\n> \t\tget_refs_list_using_list(transport, 0);\n> diff --git a/transport-internal.h b/transport-internal.h\n> index c4ca0b733a..04fa015011 100644\n> --- a/transport-internal.h\n> +++ b/transport-internal.h\n> @@ -59,6 +59,7 @@ struct transport_vtable {\n> \t * use. disconnect() releases these resources.\n> \t **/\n> \tint (*disconnect)(struct transport *connection);\n> +\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n> };\n>\n> #endif\n> diff --git a/transport.c b/transport.c\n> index 70e9840a90..65a1b1fdb3 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> \treturn refs;\n> }\n>\n> +/*\n> + * Fetches object info if server supports object-info command\n> + * Make sure to fallback to normal fetch if this fails\n> + */\n> +static int fetch_object_info(struct transport *transport)\n> +{\n> +\tstruct git_transport_data *data = transport->data;\n> +\tstruct object_info_args args;\n> +\tstruct packet_reader reader;\n> +\n> +\tmemset(&args, 0, sizeof(args));\n> +\targs.server_options = transport->server_options;\n> +\targs.object_info_options = transport->smart_options->object_info_options;\n> +\targs.oids = transport->smart_options->object_info_oids;\n> +\n> +\tconnect_setup(transport, 0);\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\tPACKET_READ_CHOMP_NEWLINE |\n> +\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n> +\tdata->version = discover_version(&reader);\n> +\n> +\ttransport->hash_algo = reader.hash_algo;\n> +\n> +\tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tif (!server_supports_v2(\"object-info\", 0))\n> +\t\t\treturn 0;\n> +\t\tsend_object_info_request(data->fd[1], &args);\n> +\t\tbreak;\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tdie(_(\"wrong protocol version. expected v2\"));\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> +\n> +\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> +\t\tdie(_(\"error reading object info header\"));\n> +\t}\n> +\tif (strcmp(reader.line, \"size\")) {\n> +\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n> +\t}\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> +\t\tprintf(\"%s\\n\", reader.line);\n> +\t}\n> +\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> +\n> +\treturn 1;\n> +}\n> +\n> +static int append_oid_to_ref(const struct object_id *oid, void *data)\n> +{\n> +\tstruct ref *ref = data;\n> +\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n> +\ttemp_ref->old_oid = *oid;\n> +\ttemp_ref->exact_oid = 1;\n> +\tref->next = temp_ref;\n> +\tref = ref->next;\n> +\treturn 0;\n> +}\n> +\n> static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n> \t\t\t\t\tstruct transport_ls_refs_options *options)\n> {\n> @@ -364,6 +425,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \tstruct ref *refs = NULL;\n> \tstruct fetch_pack_args args;\n> \tstruct ref *refs_tmp = NULL;\n> +\tstruct ref *wanted_refs = xcalloc(1, sizeof (struct ref));\n>\n> \tmemset(&args, 0, sizeof(args));\n> \targs.uploadpack = data->options.uploadpack;\n> @@ -388,8 +450,17 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \targs.server_options = transport->server_options;\n> \targs.negotiation_tips = data->options.negotiation_tips;\n> \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n> +\targs.object_info = transport->smart_options->object_info;\n>\n> -\tif (!data->got_remote_heads) {\n> +\tif (transport->smart_options && transport->smart_options->object_info) {\n> +\t\tstruct ref *temp_ref = wanted_refs;\n> +\t\tif (fetch_object_info(transport)) {\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\t\toid_array_for_each(transport->smart_options->object_info_oids, append_oid_to_ref, temp_ref);\n> +\t\twanted_refs = wanted_refs->next;\n> +\t\ttransport->remote_refs = wanted_refs;\n> +\t} else if (!data->got_remote_heads) {\n> \t\tint i;\n> \t\tint must_list_refs = 0;\n> \t\tfor (i = 0; i < nr_heads; i++) {\n> @@ -406,7 +477,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \telse if (data->version <= protocol_v1)\n> \t\tdie_if_server_options(transport);\n>\n> -\tif (data->options.acked_commits) {\n> +\tif (data->options.acked_commits && !transport->smart_options->object_info) {\n> \t\tif (data->version < protocol_v2) {\n> \t\t\twarning(_(\"--negotiate-only requires protocol v2\"));\n> \t\t\tret = -1;\n> diff --git a/transport.h b/transport.h\n> index a0bc6a1e9e..3bba16f86e 100644\n> --- a/transport.h\n> +++ b/transport.h\n> @@ -30,6 +30,12 @@ struct git_transport_options {\n> \t */\n> \tunsigned connectivity_checked:1;\n>\n> +\t/*\n> +\t * Transport will attempt to pull only object-info. Fallbacks\n> +\t * to pulling entire object if object-info is not supported\n> +\t */\n> +\tunsigned object_info : 1;\n> +\n> \tint depth;\n> \tconst char *deepen_since;\n> \tconst struct string_list *deepen_not;\n> @@ -53,6 +59,9 @@ struct git_transport_options {\n> \t * common commits to this oidset instead of fetching any packfiles.\n> \t */\n> \tstruct oidset *acked_commits;\n> +\n> +\tstruct oid_array *object_info_oids;\n> +\tstruct string_list *object_info_options;\n> };\n>\n> enum transport_family {\n> -- \n> 2.33.0.664.g0785eb7698\n"},{"id":"452710","messageId":"20220330220703.2304827-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-4-calvinwan@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-30T22:07:03Z","receivedAt":"2022-03-30T22:07:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Calvin Wan writes:\n> @@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> \t\targs->deepen = 1;\n> \n> +\tif (args->object_info) {\n> +\t\tstate = FETCH_SEND_REQUEST;\n> +\t}\n> +\n> \twhile (state != FETCH_DONE) {\n> \t\tswitch (state) {\n> \t\tcase FETCH_CHECK_LOCAL:\n> @@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n> -\t\t\tif (everything_local(args, &ref))\n> +\t\t\tif (!args->object_info && everything_local(args, &ref))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n\nI haven't looked into this deeply, but one thing to consider is to do\nthis in a separate function (instead of reusing do_fetch_pack_v2())\nbecause the object-info request and response has almost nothing in\ncommon with fetch.\n\n> diff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\n\n[skip tests]\n\nAlso test when: the user gives an invalid --object-info (e.g.\n\"--object-info=foo\") and when the user gives two parameters, one valid\nand one invalid. (Both should fail.)\n\n> diff --git a/transport-internal.h b/transport-internal.h\n> index c4ca0b733a..04fa015011 100644\n> --- a/transport-internal.h\n> +++ b/transport-internal.h\n> @@ -59,6 +59,7 @@ struct transport_vtable {\n> \t * use. disconnect() releases these resources.\n> \t **/\n> \tint (*disconnect)(struct transport *connection);\n> +\tint (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n> };\n\nDo we need another entry in the vtable, or can we do without? (As it is,\nI don't think we need it, as evidenced by the fact that this patch did\nnot add any entries to the transport_vtable instances.)\n\n> diff --git a/transport.c b/transport.c\n> index 70e9840a90..65a1b1fdb3 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> \treturn refs;\n> }\n> \n> +/*\n> + * Fetches object info if server supports object-info command\n> + * Make sure to fallback to normal fetch if this fails\n> + */\n> +static int fetch_object_info(struct transport *transport)\n> +{\n> +\tstruct git_transport_data *data = transport->data;\n> +\tstruct object_info_args args;\n> +\tstruct packet_reader reader;\n> +\n> +\tmemset(&args, 0, sizeof(args));\n> +\targs.server_options = transport->server_options;\n> +\targs.object_info_options = transport->smart_options->object_info_options;\n> +\targs.oids = transport->smart_options->object_info_oids;\n\nI think we can avoid code duplication in this function too. So from\nhere (the connect_setup line below)...\n\n> +\tconnect_setup(transport, 0);\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\tPACKET_READ_CHOMP_NEWLINE |\n> +\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n> +\tdata->version = discover_version(&reader);\n> +\n> +\ttransport->hash_algo = reader.hash_algo;\n> +\n> +\tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tif (!server_supports_v2(\"object-info\", 0))\n> +\t\t\treturn 0;\n> +\t\tsend_object_info_request(data->fd[1], &args);\n> +\t\tbreak;\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tdie(_(\"wrong protocol version. expected v2\"));\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n\n...up to here can be a function (except that we need to take out\nsend_object_info_request() and maybe parameterize the server_supports_v2\ncall).\n\nFrom here...\n\n> +\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> +\t\tdie(_(\"error reading object info header\"));\n> +\t}\n> +\tif (strcmp(reader.line, \"size\")) {\n> +\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n> +\t}\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> +\t\tprintf(\"%s\\n\", reader.line);\n> +\t}\n\n...until here is object-info-specific, yes.\n\nAnd this...\n\n> +\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> +\n> +\treturn 1;\n> +}\n\nis common code once again.\n\nAlso instead of returning 0 from this function if the server doesn't\nsupport object-info, it's probably clearer to have one function that\nchecks what the server supports, and then have an \"if\" block which\neither does the object-info part or the fetch part depending on what the\nserver says.\n"},{"id":"452712","messageId":"YkTV19+eCQNq/C5s@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-4-calvinwan@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2022-03-30T22:12:39Z","receivedAt":"2022-03-30T22:12:51Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2022.03.28 19:11, Calvin Wan wrote:\n> Sometimes it is useful to get information about an object without\n> having to download it completely. The server logic has already been\n> implemented as “a2ba162cda (object-info: support for retrieving object\n> info, 2021-04-20)”. This patch implements the client option for\n> it. Currently, only 'size' is supported, however future patches can\n> implement additional options.\n> \n> Add ‘--object-info’ option to fetch. This option allows the client to\n> make an object-info command request to a server that supports protocol\n> v2. If the server is v2, but does not allow for the object-info\n> command request, fetch the objects as if it were a standard fetch\n> (however without changing any refs). Therefore, hook `fetch\n> object-info` into transport_fetch_refs() to easily fallback if needed.\n> \n> A previous patch added the `transfer.advertiseObjectInfo` config\n> option, of which this patch utilizes to test the fallback.\n> \n> ---\n> Documentation/fetch-options.txt | 5 ++\n> Documentation/git-fetch.txt | 1 +\n> builtin/fetch.c | 36 ++++++++-\n> fetch-pack.c | 42 +++++++++-\n> fetch-pack.h | 9 +++\n> t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n> transport-helper.c | 8 +-\n> transport-internal.h | 1 +\n> transport.c | 75 ++++++++++++++++-\n> transport.h | 9 +++\n> 10 files changed, 315 insertions(+), 9 deletions(-)\n> create mode 100755 t/t5583-fetch-object-info.sh\n\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index b709a61baf..36e3d1c5d0 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n> \tpacket_buf_delim(req_buf);\n> }\n> \n> +void send_object_info_request(int fd_out, struct object_info_args *args)\n> +{\n> +\tstruct strbuf req_buf = STRBUF_INIT;\n> +\tint i;\n> +\n> +\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> +\n> +\tif (string_list_has_string(args->object_info_options, \"size\"))\n> +\t\tpacket_buf_write(&req_buf, \"size\");\n\nDo we need a newline after \"size\" here? It's not clear to me that it's\nnecessary; Documentation/technical/protocol-v2.txt just says\n\"command-specific-args are packet line framed arguments defined by each\nindividual command\", but the oid arguments below have a newline, so we\nshould be consistent one way or the other.\n\nIt looks like the server-side implementation wants just a bare \"size\"\nstring (no newline), but I suspect that either is OK because the\npacket_reader is probably using PACKET_READ_CHOMP_NEWLINE.\n\n\n> +\tfor (i = 0; i < args->oids->nr; i++) {\n> +\t\tpacket_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> +\t}\n> +\n> +\tpacket_buf_flush(&req_buf);\n> +\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n> +\t\tdie_errno(_(\"unable to write request to remote\"));\n> +\n> +\tstrbuf_release(&req_buf);\n> +}\n> +\n\n> diff --git a/transport.c b/transport.c\n> index 70e9840a90..65a1b1fdb3 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> \treturn refs;\n> }\n> \n> +/*\n> + * Fetches object info if server supports object-info command\n> + * Make sure to fallback to normal fetch if this fails\n> + */\n> +static int fetch_object_info(struct transport *transport)\n> +{\n> +\tstruct git_transport_data *data = transport->data;\n> +\tstruct object_info_args args;\n> +\tstruct packet_reader reader;\n> +\n> +\tmemset(&args, 0, sizeof(args));\n> +\targs.server_options = transport->server_options;\n> +\targs.object_info_options = transport->smart_options->object_info_options;\n> +\targs.oids = transport->smart_options->object_info_oids;\n> +\n> +\tconnect_setup(transport, 0);\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\tPACKET_READ_CHOMP_NEWLINE |\n> +\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n> +\tdata->version = discover_version(&reader);\n> +\n> +\ttransport->hash_algo = reader.hash_algo;\n> +\n> +\tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tif (!server_supports_v2(\"object-info\", 0))\n> +\t\t\treturn 0;\n> +\t\tsend_object_info_request(data->fd[1], &args);\n> +\t\tbreak;\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tdie(_(\"wrong protocol version. expected v2\"));\n\nThe comment at the top of this function says that callers should be\nprepared to fallback to normal fetch if this function fails. The only\nway it can currently fail is if we are using protocol V2 but the server\ndoesn't support object-info. Rather than die() if we're on protocol V1\nor V0, can we also return a failure here and let callers fallback to\nfetch?\n\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> +\n> +\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> +\t\tdie(_(\"error reading object info header\"));\n> +\t}\n> +\tif (strcmp(reader.line, \"size\")) {\n> +\t\tdie(_(\"expected 'size', received '%s'\"), reader.line);\n> +\t}\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> +\t\tprintf(\"%s\\n\", reader.line);\n> +\t}\n> +\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> +\n> +\treturn 1;\n> +}\n> +\n"},{"id":"452714","messageId":"CAFySSZB3hAhUF2ODK5YaPP8VcPa=_LovD82zOm0BHcpQzk_T-Q@mail.gmail.com","threadId":"57388","inReplyTo":"xmqq4k3gqn7n.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-30T22:31:24Z","receivedAt":"2022-03-30T22:31:38Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Overall, I agree that this option does not belong in fetch. I wouldn't\nsay I chose to put it there for the sake of implementation\nconvenience, but because \"it does what fetch is supposed to do minus\nthe object itself\". This logic no longer makes any sense to me, and I\nregret not seeing the problems with it as I was coming up with\nworkarounds.\n\n> I think that \"Object 'size' is currently the only one the server\n> knows to emit\", combined with the lack of any client side support so\n> far, means that it is not too late to rework the object-info thing\n> and possibly reuse more from \"cat-file --batch-check\".\n\nI talked this over with Jonathan Tan and we came to the conclusion\nthat this is the next option I should look into, rather than\nseparating it out into its own command.\n\nThank you for the review Junio! I appreciate how straightforward all\nof your comments are.\n\nOn Tue, Mar 29, 2022 at 5:49 PM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > Shouldn't we be making the part that we truly need to reuse into a\n> > separate API out of fetch-pack and either (1) allow new commands be\n> > easily written reusing the code to create \"git remote-object-info\"\n> > and \"git remote-blame\", or (2) come up with a single \"do things on\n> > remote\" command with various subcommands, i.e. \"git over-there\n> > object-info\" and \"git over-there blame\"?\n>\n> For completeness, we could also make the \"git archive\" the gold\n> standard. If we want \"git blame\" that is mostly executed on the\n> remote site, we could teach \"git blame --remote \", which may\n> be where a user, who does not know nor care about how things are\n> implemented, would expect to find the feature. Is \"object-info\" in\n> principle like executing \"cat-file --batch-check\" remotely?\n>\n> I think that \"Object 'size' is currently the only one the server\n> knows to emit\", combined with the lack of any client side support so\n> far, means that it is not too late to rework the object-info thing\n> and possibly reuse more from \"cat-file --batch-check\".\n>\n> If we can create a system where running things locally and remotely\n> are comparable in feature, that would be great, but I do not think\n> that is a realistic goal to aim for.\n>\n> Thoughts?\n>\n>\n>\n"},{"id":"452716","messageId":"20220330224314.2310534-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"xmqqmth8qsiv.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-03-30T22:43:14Z","receivedAt":"2022-03-30T22:43:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano writes:\n> Calvin Wan writes:\n> \n> > Add ‘--object-info’ option to fetch. This option allows the client to\n> > make an object-info command request to a server that supports protocol\n> > v2. If the server is v2, but does not allow for the object-info\n> > command request, fetch the objects as if it were a standard fetch\n> > (however without changing any refs). Therefore, hook `fetch\n> > object-info` into transport_fetch_refs() to easily fallback if needed.\n> \n> Sorry, but I do not see anything common between \"git fetch\" that\n> fetches history and associated objects and \"retrieving only object\n> info\". Other than \"the way I happened to have chosen to implement,\n> what is used to implement fetch has the most commonality\".\n\nCalvin and I discussed details of implementation before he wrote these\npatches, and my thinking at that time was that to get the relevant\ninformation, we're prepared to fetch the objects themselves from the\nremote if need be, but prefer to fetch only metadata if possible, so\nputting it in \"fetch\" seemed reasonable to me. But I agree that it's\nprobably better to put it somewhere else.\n"},{"id":"452717","messageId":"CAFySSZDWYN6sKZgm=XKcZUb0OgYBKPwn9OX92m4Ybgr+XZ_8LQ@mail.gmail.com","threadId":"57388","inReplyTo":"YkTV19+eCQNq/C5s@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-30T22:46:27Z","receivedAt":"2022-03-30T22:46:42Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> but the oid arguments below have a newline, so we\n> should be consistent one way or the other.\n\nTo match what the other packet functions do, I will get rid of the new\nline for the oid loop.\n\n> Rather than die() if we're on protocol V1\n> or V0, can we also return a failure here and let callers fallback to\n> fetch?\n\nSounds reasonable. Will look into it.\n\nOn Wed, Mar 30, 2022 at 3:12 PM Josh Steadmon wrote:\n>\n> On 2022.03.28 19:11, Calvin Wan wrote:\n> > Sometimes it is useful to get information about an object without\n> > having to download it completely. The server logic has already been\n> > implemented as “a2ba162cda (object-info: support for retrieving object\n> > info, 2021-04-20)”. This patch implements the client option for\n> > it. Currently, only 'size' is supported, however future patches can\n> > implement additional options.\n> >\n> > Add ‘--object-info’ option to fetch. This option allows the client to\n> > make an object-info command request to a server that supports protocol\n> > v2. If the server is v2, but does not allow for the object-info\n> > command request, fetch the objects as if it were a standard fetch\n> > (however without changing any refs). Therefore, hook `fetch\n> > object-info` into transport_fetch_refs() to easily fallback if needed.\n> >\n> > A previous patch added the `transfer.advertiseObjectInfo` config\n> > option, of which this patch utilizes to test the fallback.\n> >\n> > ---\n> > Documentation/fetch-options.txt | 5 ++\n> > Documentation/git-fetch.txt | 1 +\n> > builtin/fetch.c | 36 ++++++++-\n> > fetch-pack.c | 42 +++++++++-\n> > fetch-pack.h | 9 +++\n> > t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n> > transport-helper.c | 8 +-\n> > transport-internal.h | 1 +\n> > transport.c | 75 ++++++++++++++++-\n> > transport.h | 9 +++\n> > 10 files changed, 315 insertions(+), 9 deletions(-)\n> > create mode 100755 t/t5583-fetch-object-info.sh\n>\n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index b709a61baf..36e3d1c5d0 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> > @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n> > packet_buf_delim(req_buf);\n> > }\n> >\n> > +void send_object_info_request(int fd_out, struct object_info_args *args)\n> > +{\n> > + struct strbuf req_buf = STRBUF_INIT;\n> > + int i;\n> > +\n> > + write_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> > +\n> > + if (string_list_has_string(args->object_info_options, \"size\"))\n> > + packet_buf_write(&req_buf, \"size\");\n>\n> Do we need a newline after \"size\" here? It's not clear to me that it's\n> necessary; Documentation/technical/protocol-v2.txt just says\n> \"command-specific-args are packet line framed arguments defined by each\n> individual command\", but the oid arguments below have a newline, so we\n> should be consistent one way or the other.\n>\n> It looks like the server-side implementation wants just a bare \"size\"\n> string (no newline), but I suspect that either is OK because the\n> packet_reader is probably using PACKET_READ_CHOMP_NEWLINE.\n>\n>\n> > + for (i = 0; i < args->oids->nr; i++) {\n> > + packet_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> > + }\n> > +\n> > + packet_buf_flush(&req_buf);\n> > + if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n> > + die_errno(_(\"unable to write request to remote\"));\n> > +\n> > + strbuf_release(&req_buf);\n> > +}\n> > +\n>\n> > diff --git a/transport.c b/transport.c\n> > index 70e9840a90..65a1b1fdb3 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> > return refs;\n> > }\n> >\n> > +/*\n> > + * Fetches object info if server supports object-info command\n> > + * Make sure to fallback to normal fetch if this fails\n> > + */\n> > +static int fetch_object_info(struct transport *transport)\n> > +{\n> > + struct git_transport_data *data = transport->data;\n> > + struct object_info_args args;\n> > + struct packet_reader reader;\n> > +\n> > + memset(&args, 0, sizeof(args));\n> > + args.server_options = transport->server_options;\n> > + args.object_info_options = transport->smart_options->object_info_options;\n> > + args.oids = transport->smart_options->object_info_oids;\n> > +\n> > + connect_setup(transport, 0);\n> > + packet_reader_init(&reader, data->fd[0], NULL, 0,\n> > + PACKET_READ_CHOMP_NEWLINE |\n> > + PACKET_READ_DIE_ON_ERR_PACKET);\n> > + data->version = discover_version(&reader);\n> > +\n> > + transport->hash_algo = reader.hash_algo;\n> > +\n> > + switch (data->version) {\n> > + case protocol_v2:\n> > + if (!server_supports_v2(\"object-info\", 0))\n> > + return 0;\n> > + send_object_info_request(data->fd[1], &args);\n> > + break;\n> > + case protocol_v1:\n> > + case protocol_v0:\n> > + die(_(\"wrong protocol version. expected v2\"));\n>\n> The comment at the top of this function says that callers should be\n> prepared to fallback to normal fetch if this function fails. The only\n> way it can currently fail is if we are using protocol V2 but the server\n> doesn't support object-info. Rather than die() if we're on protocol V1\n> or V0, can we also return a failure here and let callers fallback to\n> fetch?\n>\n> > + case protocol_unknown_version:\n> > + BUG(\"unknown protocol version\");\n> > + }\n> > +\n> > + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> > + die(_(\"error reading object info header\"));\n> > + }\n> > + if (strcmp(reader.line, \"size\")) {\n> > + die(_(\"expected 'size', received '%s'\"), reader.line);\n> > + }\n> > + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> > + printf(\"%s\\n\", reader.line);\n> > + }\n> > + check_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> > +\n> > + return 1;\n> > +}\n> > +\n"},{"id":"452718","messageId":"CAFySSZDD4VsQckvuubjZbVwisEbHbCpZK2sA_FQQj_voVF94Pw@mail.gmail.com","threadId":"57388","inReplyTo":"YkOT/TCKPK3LT9gh@nand.local","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-30T22:47:23Z","receivedAt":"2022-03-30T22:47:40Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Thank you for the code hygiene tips, Taylor! All of these sounds like\ngood ideas and will follow through\n\nOn Tue, Mar 29, 2022 at 4:19 PM Taylor Blau wrote:\n>\n> Hi Calvin,\n>\n> Junio took a deeper look below, but here are a few small code-hygiene\n> issues that I noticed while taking a quicker look through this patch.\n>\n> On Mon, Mar 28, 2022 at 07:11:12PM +0000, Calvin Wan wrote:\n> > Sometimes it is useful to get information about an object without\n> > having to download it completely. The server logic has already been\n> > implemented as “a2ba162cda (object-info: support for retrieving object\n> > info, 2021-04-20)”. This patch implements the client option for\n> > it. Currently, only 'size' is supported, however future patches can\n> > implement additional options.\n> >\n> > Add ‘--object-info’ option to fetch. This option allows the client to\n>\n> The single-quotes here look like smart quotes. I don't think we forbid\n> these explicitly within commit messages, but using the standard '\n> (ASCII 0x27) character is more common.\n>\n> > make an object-info command request to a server that supports protocol\n> > v2. If the server is v2, but does not allow for the object-info\n> > command request, fetch the objects as if it were a standard fetch\n> > (however without changing any refs). Therefore, hook `fetch\n> > object-info` into transport_fetch_refs() to easily fallback if needed.\n> >\n> > A previous patch added the `transfer.advertiseObjectInfo` config\n> > option, of which this patch utilizes to test the fallback.\n> >\n> > ---\n> > Documentation/fetch-options.txt | 5 ++\n> > Documentation/git-fetch.txt | 1 +\n> > builtin/fetch.c | 36 ++++++++-\n> > fetch-pack.c | 42 +++++++++-\n> > fetch-pack.h | 9 +++\n> > t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n> > transport-helper.c | 8 +-\n> > transport-internal.h | 1 +\n> > transport.c | 75 ++++++++++++++++-\n> > transport.h | 9 +++\n> > 10 files changed, 315 insertions(+), 9 deletions(-)\n> > create mode 100755 t/t5583-fetch-object-info.sh\n> >\n> > diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> > index f903683189..f1ca95c32d 100644\n> > --- a/Documentation/fetch-options.txt\n> > +++ b/Documentation/fetch-options.txt\n> > @@ -299,3 +299,8 @@ endif::git-pull[]\n> > -6::\n> > --ipv6::\n> > Use IPv6 addresses only, ignoring IPv4 addresses.\n> > +\n> > +--object-info=[]::\n> > + Fetches only the relevant information passed in arguments from a\n> > + specific remote and set of object ids. Object 'size' is currently\n> > + the only supported argument.\n> > diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\n> > index 550c16ca61..a13d2ba7ad 100644\n> > --- a/Documentation/git-fetch.txt\n> > +++ b/Documentation/git-fetch.txt\n> > @@ -13,6 +13,7 @@ SYNOPSIS\n> > 'git fetch' [] \n> > 'git fetch' --multiple [] [( | )...]\n> > 'git fetch' --all []\n> > +'git fetch' --object-info=[] []\n> >\n> >\n> > DESCRIPTION\n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 4d12c2fd4d..0b21bebfca 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -29,6 +29,9 @@\n> > #include \"commit-graph.h\"\n> > #include \"shallow.h\"\n> > #include \"worktree.h\"\n> > +#include \"protocol.h\"\n> > +#include \"pkt-line.h\"\n> > +#include \"connect.h\"\n> >\n> > #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n> >\n> > @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> > N_(\"git fetch [] \"),\n> > N_(\"git fetch --multiple [] [( | )...]\"),\n> > N_(\"git fetch --all []\"),\n> > + N_(\"git fetch --object-info=[] []\"),\n> > NULL\n> > };\n> >\n> > @@ -86,6 +90,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n> > static int fetch_write_commit_graph = -1;\n> > static int stdin_refspecs = 0;\n> > static int negotiate_only;\n> > +static struct string_list object_info_options = STRING_LIST_INIT_NODUP;\n> >\n> > static int git_fetch_config(const char *k, const char *v, void *cb)\n> > {\n> > @@ -221,6 +226,8 @@ static struct option builtin_fetch_options[] = {\n> > N_(\"write the commit-graph after fetching\")),\n> > OPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> > N_(\"accept refspecs from stdin\")),\n> > + OPT_STRING_LIST(0, \"object-info\", &object_info_options, N_(\"option\"),\n> > + N_(\"command request arguments\")),\n> > OPT_END()\n> > };\n> >\n> > @@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > struct remote *remote = NULL;\n> > int result = 0;\n> > int prune_tags_ok = 1;\n> > + struct oid_array object_info_oids = OID_ARRAY_INIT;\n> >\n> > packet_trace_identity(\"fetch\");\n> >\n> > @@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > if (dry_run)\n> > write_fetch_head = 0;\n> >\n> > - if (all) {\n> > + if (object_info_options.nr > 0) {\n>\n> Small nit; since object_info_options.nr should never be negative, we\n> should instead write \"if (object_info_options)\".\n>\n> > + if (argc == 0 || argc == 1) {\n> > + die(_(\"must supply remote and object ids when using --object-info\"));\n> > + } else {\n> > + struct object_id oid;\n> > + remote = remote_get(argv[0]);\n> > + for (i = 1; i < argc; i++) {\n> > + if (get_oid(argv[i], &oid))\n> > + return error(_(\"malformed object id '%s'\"), argv[i]);\n> > + oid_array_append(&object_info_oids, &oid);\n> > + }\n> > + }\n> > + } else if (all) {\n> > if (argc == 1)\n> > die(_(\"fetch --all does not take a repository argument\"));\n> > else if (argc > 1)\n> > @@ -2199,7 +2219,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > }\n> > }\n> >\n> > - if (negotiate_only) {\n> > + if (object_info_options.nr > 0) {\n> > + if (!remote)\n> > + die(_(\"must supply remote when using --object-info\"));\n> > + gtransport = prepare_transport(remote, 1);\n> > + if (gtransport->smart_options) {\n> > + gtransport->smart_options->object_info = 1;\n> > + gtransport->smart_options->object_info_oids = &object_info_oids;\n> > + gtransport->smart_options->object_info_options = &object_info_options;\n> > + }\n> > + if (server_options.nr)\n> > + gtransport->server_options = &server_options;\n> > + result = transport_fetch_refs(gtransport, NULL);\n> > + } else if (negotiate_only) {\n> > struct oidset acked_commits = OIDSET_INIT;\n> > struct oidset_iter iter;\n> > const struct object_id *oid;\n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index b709a61baf..36e3d1c5d0 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> > @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n> > packet_buf_delim(req_buf);\n> > }\n> >\n> > +void send_object_info_request(int fd_out, struct object_info_args *args)\n> > +{\n> > + struct strbuf req_buf = STRBUF_INIT;\n> > + int i;\n>\n> I don't think this would ever be practically exploit-able, since we'd\n> run past the argument limit long before we hit INT_MAX, but this should\n> be a size_t instead of an int to match the type of args->oid->nr.\n>\n> > +\n> > + write_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> > +\n> > + if (string_list_has_string(args->object_info_options, \"size\"))\n> > + packet_buf_write(&req_buf, \"size\");\n> > +\n> > + for (i = 0; i < args->oids->nr; i++) {\n> > + packet_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> > + }\n>\n> Small nit, the braces around this for-loop are not required.\n>\n> > static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> > struct fetch_pack_args *args,\n> > const struct ref *wants, struct oidset *common,\n> > @@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> > if (args->depth > 0 || args->deepen_since || args->deepen_not)\n> > args->deepen = 1;\n> >\n> > + if (args->object_info) {\n> > + state = FETCH_SEND_REQUEST;\n> > + }\n> > +\n>\n> Ditto here.\n>\n> > while (state != FETCH_DONE) {\n> > switch (state) {\n> > case FETCH_CHECK_LOCAL:\n> > @@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> > /* Filter 'ref' by 'sought' and those that aren't local */\n> > mark_complete_and_common_ref(negotiator, args, &ref);\n> > filter_refs(args, &ref, sought, nr_sought);\n> > - if (everything_local(args, &ref))\n> > + if (!args->object_info && everything_local(args, &ref))\n> > state = FETCH_DONE;\n> > else\n> > state = FETCH_SEND_REQUEST;\n> > @@ -1999,8 +2024,19 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> > }\n> > args->connectivity_checked = 1;\n> > }\n> > -\n> > - update_shallow(args, sought, nr_sought, &si);\n> > +\n> > + if (args->object_info) {\n> > + struct ref *ref_cpy_reader = ref_cpy;\n> > + unsigned long size = 0;\n> > + while (ref_cpy_reader) {\n> > + oid_object_info(the_repository, &(ref_cpy_reader->old_oid), &size);\n> > + printf(\"%s %li\\n\", oid_to_hex(&(ref_cpy_reader->old_oid)), size);\n>\n> Both expressions like \"&(x)\" can be written instead as \"&x\" without the\n> outer parenthesis.\n>\n> Likewise, we do not use %li, that line should instead read:\n>\n> printf(\"%s %\"PRIuMAX\"\\n\", oid_to_hex(&ref_cpy_reader->old_oid), (uintmax_t)size);\n>\n> Thanks,\n> Taylor\n"},{"id":"452723","messageId":"xmqqpmm357p2.fsf@gitster.g","threadId":"57388","inReplyTo":"20220330224314.2310534-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-30T23:42:01Z","receivedAt":"2022-03-30T23:42:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Calvin and I discussed details of implementation before he wrote these\n> patches, and my thinking at that time was that to get the relevant\n> information, we're prepared to fetch the objects themselves from the\n> remote if need be, but prefer to fetch only metadata if possible, so\n> putting it in \"fetch\" seemed reasonable to me. But I agree that it's\n> probably better to put it somewhere else.\n\nYeah, I am sympathetic that every \"learning something from remote\"\nlooks like a \"fetch\". After all, a remote blame may be \"fetching\nthe result of running blame\". But it may not be a good way to\ndecide what the feature should be called.\n\nThanks.\n\n"},{"id":"452763","messageId":"xmqqk0ca51rc.fsf@gitster.g","threadId":"57388","inReplyTo":"20220328191112.3092139-1-calvinwan@google.com","subject":"Re: [PATCH v3 0/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-31T01:50:15Z","receivedAt":"2022-03-31T01:50:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Sometimes it is useful to get information about an object without\n> having to download it completely. The server logic has already been\n> implemented as “a2ba162cda (object-info: support for retrieving object\n> info, 2021-04-20)”. This patch implements the client option for it.\n\nMerged to 'seen', this seems to break a test in t5583.\ne.g. https://github.com/git/git/runs/5763421702?check_suite_focus=true\n\nThanks.\n"},{"id":"452830","messageId":"CAFySSZAwXJc6BG8Q9P3h9njyMGz9YsOuCWSwTp+33vafiJ4L_A@mail.gmail.com","threadId":"57388","inReplyTo":"15354CB8-E1DD-476C-B6E0-52F9451108A6@gmail.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-03-31T19:56:14Z","receivedAt":"2022-03-31T19:56:29Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"(Just a small nit before I respond. If you could reply with only the\ncode you're referring to, that would be much appreciated).\n\n> Would it be useful to check for the existence of the object to prove it got fetched?\nThe object is not saved into the filesystem after fetching for only\nobject-info so checking for its existence wouldn't prove anything.\n\n> Since the server is being used as the client, we wouldn't be able to confirm that\n> the object got fetched\nI agree that this test case can be written better to confirm that the\nobject was fetched through the standard channel. I think I could use\nGIT_TRACE_PACKET=1 with the output piped to a log file and then\ngrepping that log file for packet information resembling a standard\nfetch to confirm. (See t5702-protocol-v2.sh as a reference)\n\n> But, as a larger question for the list, I'm wondering if it's best practice to separate client from\n> server in tests, or if it's an accepted style to just use one repository in cases\n> when it doesn't really matter?\nI am as well!\n\nOn Wed, Mar 30, 2022 at 3:06 PM John Cai wrote:\n>\n>\n>\n> On 28 Mar 2022, at 15:11, Calvin Wan wrote:\n>\n> > Sometimes it is useful to get information about an object without\n> > having to download it completely. The server logic has already been\n> > implemented as “a2ba162cda (object-info: support for retrieving object\n> > info, 2021-04-20)”. This patch implements the client option for\n> > it. Currently, only 'size' is supported, however future patches can\n> > implement additional options.\n> >\n> > Add ‘--object-info’ option to fetch. This option allows the client to\n> > make an object-info command request to a server that supports protocol\n> > v2. If the server is v2, but does not allow for the object-info\n> > command request, fetch the objects as if it were a standard fetch\n> > (however without changing any refs). Therefore, hook `fetch\n> > object-info` into transport_fetch_refs() to easily fallback if needed.\n> >\n> > A previous patch added the `transfer.advertiseObjectInfo` config\n> > option, of which this patch utilizes to test the fallback.\n> >\n> > ---\n> > Documentation/fetch-options.txt | 5 ++\n> > Documentation/git-fetch.txt | 1 +\n> > builtin/fetch.c | 36 ++++++++-\n> > fetch-pack.c | 42 +++++++++-\n> > fetch-pack.h | 9 +++\n> > t/t5583-fetch-object-info.sh | 138 ++++++++++++++++++++++++++++++++\n> > transport-helper.c | 8 +-\n> > transport-internal.h | 1 +\n> > transport.c | 75 ++++++++++++++++-\n> > transport.h | 9 +++\n> > 10 files changed, 315 insertions(+), 9 deletions(-)\n> > create mode 100755 t/t5583-fetch-object-info.sh\n> >\n> > diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> > index f903683189..f1ca95c32d 100644\n> > --- a/Documentation/fetch-options.txt\n> > +++ b/Documentation/fetch-options.txt\n> > @@ -299,3 +299,8 @@ endif::git-pull[]\n> > -6::\n> > --ipv6::\n> > Use IPv6 addresses only, ignoring IPv4 addresses.\n> > +\n> > +--object-info=[]::\n> > + Fetches only the relevant information passed in arguments from a\n> > + specific remote and set of object ids. Object 'size' is currently\n> > + the only supported argument.\n> > diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\n> > index 550c16ca61..a13d2ba7ad 100644\n> > --- a/Documentation/git-fetch.txt\n> > +++ b/Documentation/git-fetch.txt\n> > @@ -13,6 +13,7 @@ SYNOPSIS\n> > 'git fetch' [] \n> > 'git fetch' --multiple [] [( | )...]\n> > 'git fetch' --all []\n> > +'git fetch' --object-info=[] []\n> >\n> >\n> > DESCRIPTION\n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 4d12c2fd4d..0b21bebfca 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -29,6 +29,9 @@\n> > #include \"commit-graph.h\"\n> > #include \"shallow.h\"\n> > #include \"worktree.h\"\n> > +#include \"protocol.h\"\n> > +#include \"pkt-line.h\"\n> > +#include \"connect.h\"\n> >\n> > #define FORCED_UPDATES_DELAY_WARNING_IN_MS (10 * 1000)\n> >\n> > @@ -37,6 +40,7 @@ static const char * const builtin_fetch_usage[] = {\n> > N_(\"git fetch [] \"),\n> > N_(\"git fetch --multiple [] [( | )...]\"),\n> > N_(\"git fetch --all []\"),\n> > + N_(\"git fetch --object-info=[] []\"),\n> > NULL\n> > };\n> >\n> > @@ -86,6 +90,7 @@ static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n> > static int fetch_write_commit_graph = -1;\n> > static int stdin_refspecs = 0;\n> > static int negotiate_only;\n> > +static struct string_list object_info_options = STRING_LIST_INIT_NODUP;\n> >\n> > static int git_fetch_config(const char *k, const char *v, void *cb)\n> > {\n> > @@ -221,6 +226,8 @@ static struct option builtin_fetch_options[] = {\n> > N_(\"write the commit-graph after fetching\")),\n> > OPT_BOOL(0, \"stdin\", &stdin_refspecs,\n> > N_(\"accept refspecs from stdin\")),\n> > + OPT_STRING_LIST(0, \"object-info\", &object_info_options, N_(\"option\"),\n> > + N_(\"command request arguments\")),\n> > OPT_END()\n> > };\n> >\n> > @@ -2087,6 +2094,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > struct remote *remote = NULL;\n> > int result = 0;\n> > int prune_tags_ok = 1;\n> > + struct oid_array object_info_oids = OID_ARRAY_INIT;\n> >\n> > packet_trace_identity(\"fetch\");\n> >\n> > @@ -2168,7 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > if (dry_run)\n> > write_fetch_head = 0;\n> >\n> > - if (all) {\n> > + if (object_info_options.nr > 0) {\n> > + if (argc == 0 || argc == 1) {\n> > + die(_(\"must supply remote and object ids when using --object-info\"));\n> > + } else {\n> > + struct object_id oid;\n> > + remote = remote_get(argv[0]);\n> > + for (i = 1; i < argc; i++) {\n> > + if (get_oid(argv[i], &oid))\n> > + return error(_(\"malformed object id '%s'\"), argv[i]);\n> > + oid_array_append(&object_info_oids, &oid);\n> > + }\n> > + }\n> > + } else if (all) {\n> > if (argc == 1)\n> > die(_(\"fetch --all does not take a repository argument\"));\n> > else if (argc > 1)\n> > @@ -2199,7 +2219,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> > }\n> > }\n> >\n> > - if (negotiate_only) {\n> > + if (object_info_options.nr > 0) {\n> > + if (!remote)\n> > + die(_(\"must supply remote when using --object-info\"));\n> > + gtransport = prepare_transport(remote, 1);\n> > + if (gtransport->smart_options) {\n> > + gtransport->smart_options->object_info = 1;\n> > + gtransport->smart_options->object_info_oids = &object_info_oids;\n> > + gtransport->smart_options->object_info_options = &object_info_options;\n> > + }\n> > + if (server_options.nr)\n> > + gtransport->server_options = &server_options;\n> > + result = transport_fetch_refs(gtransport, NULL);\n> > + } else if (negotiate_only) {\n> > struct oidset acked_commits = OIDSET_INIT;\n> > struct oidset_iter iter;\n> > const struct object_id *oid;\n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index b709a61baf..36e3d1c5d0 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> > @@ -1269,6 +1269,27 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n> > packet_buf_delim(req_buf);\n> > }\n> >\n> > +void send_object_info_request(int fd_out, struct object_info_args *args)\n> > +{\n> > + struct strbuf req_buf = STRBUF_INIT;\n> > + int i;\n> > +\n> > + write_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> > +\n> > + if (string_list_has_string(args->object_info_options, \"size\"))\n> > + packet_buf_write(&req_buf, \"size\");\n> > +\n> > + for (i = 0; i < args->oids->nr; i++) {\n> > + packet_buf_write(&req_buf, \"oid %s\\n\", oid_to_hex(&args->oids->oid[i]));\n> > + }\n> > +\n> > + packet_buf_flush(&req_buf);\n> > + if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n> > + die_errno(_(\"unable to write request to remote\"));\n> > +\n> > + strbuf_release(&req_buf);\n> > +}\n> > +\n> > static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n> > struct fetch_pack_args *args,\n> > const struct ref *wants, struct oidset *common,\n> > @@ -1604,6 +1625,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> > if (args->depth > 0 || args->deepen_since || args->deepen_not)\n> > args->deepen = 1;\n> >\n> > + if (args->object_info) {\n> > + state = FETCH_SEND_REQUEST;\n> > + }\n> > +\n> > while (state != FETCH_DONE) {\n> > switch (state) {\n> > case FETCH_CHECK_LOCAL:\n> > @@ -1613,7 +1638,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> > /* Filter 'ref' by 'sought' and those that aren't local */\n> > mark_complete_and_common_ref(negotiator, args, &ref);\n> > filter_refs(args, &ref, sought, nr_sought);\n> > - if (everything_local(args, &ref))\n> > + if (!args->object_info && everything_local(args, &ref))\n> > state = FETCH_DONE;\n> > else\n> > state = FETCH_SEND_REQUEST;\n> > @@ -1999,8 +2024,19 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> > }\n> > args->connectivity_checked = 1;\n> > }\n> > -\n> > - update_shallow(args, sought, nr_sought, &si);\n> > +\n> > + if (args->object_info) {\n> > + struct ref *ref_cpy_reader = ref_cpy;\n> > + unsigned long size = 0;\n> > + while (ref_cpy_reader) {\n> > + oid_object_info(the_repository, &(ref_cpy_reader->old_oid), &size);\n> > + printf(\"%s %li\\n\", oid_to_hex(&(ref_cpy_reader->old_oid)), size);\n> > + ref_cpy_reader = ref_cpy_reader->next;\n> > + }\n> > + }\n> > + else {\n> > + update_shallow(args, sought, nr_sought, &si);\n> > + }\n> > cleanup:\n> > clear_shallow_info(&si);\n> > oid_array_clear(&shallows_scratch);\n> > diff --git a/fetch-pack.h b/fetch-pack.h\n> > index 7f94a2a583..23e69bb218 100644\n> > --- a/fetch-pack.h\n> > +++ b/fetch-pack.h\n> > @@ -42,6 +42,7 @@ struct fetch_pack_args {\n> > unsigned update_shallow:1;\n> > unsigned reject_shallow_remote:1;\n> > unsigned deepen:1;\n> > + unsigned object_info:1;\n> >\n> > /*\n> > * Indicate that the remote of this request is a promisor remote. The\n> > @@ -68,6 +69,12 @@ struct fetch_pack_args {\n> > unsigned connectivity_checked:1;\n> > };\n> >\n> > +struct object_info_args {\n> > + const struct string_list *object_info_options;\n> > + const struct string_list *server_options;\n> > + const struct oid_array *oids;\n> > +};\n> > +\n> > /*\n> > * sought represents remote references that should be updated from.\n> > * On return, the names that were found on the remote will have been\n> > @@ -101,4 +108,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n> > */\n> > int report_unmatched_refs(struct ref **sought, int nr_sought);\n> >\n> > +void send_object_info_request(int fd_out, struct object_info_args *args);\n> > +\n> > #endif\n> > diff --git a/t/t5583-fetch-object-info.sh b/t/t5583-fetch-object-info.sh\n> > new file mode 100755\n> > index 0000000000..e5db6d953c\n> > --- /dev/null\n> > +++ b/t/t5583-fetch-object-info.sh\n> > @@ -0,0 +1,138 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='test git fetch object-info version 2'\n> > +\n> > +TEST_NO_CREATE_REPO=1\n> > +\n> > +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> > +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +# Test fetch object-info with 'git://' transport\n> > +\n> > +. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n> > +start_git_daemon --export-all --enable=receive-pack\n> > +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n> > +\n> > +\n> > +test_expect_success 'create repo to be served by git-daemon' '\n> > + git init \"$daemon_parent\" &&\n> > + test_commit -C \"$daemon_parent\" message1 a.txt\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info with git:// using protocol v2' '\n> > + (\n> > + cd \"$daemon_parent\" &&\n> > + object_id=$(git rev-parse message1:a.txt) &&\n> > + length=$(wc -c > +\n> > + printf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> > + git -c protocol.version=2 fetch --object-info=size \"$GIT_DAEMON_URL/parent\" \"$object_id\" >actual &&\n> > + test_cmp expect actual\n> > + )\n> > +'\n> > +stop_git_daemon\n> > +\n> > +# Test protocol v2 with 'http://' transport\n> > +\n> > +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> > +start_httpd\n> > +\n> > +test_expect_success 'create repo to be served by http:// transport' '\n> > + git init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + git -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n> > + test_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message1 a.txt &&\n> > + test_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" message2 b.txt\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info with http:// using protocol v2' '\n> > + (\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + object_id=$(git rev-parse message1:a.txt) &&\n> > + length=$(wc -c > +\n> > + printf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> > + git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n> > + test_cmp expect actual\n> > + )\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info for multiple objects' '\n> > + (\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + object_id1=$(git rev-parse message1:a.txt) &&\n> > + object_id2=$(git rev-parse message2:b.txt) &&\n> > + length1=$(wc -c > + length2=$(wc -c > +\n> > + printf \"%s %d\\n\" \"$object_id1\" \"$length1\" >expect &&\n> > + printf \"%s %d\\n\" \"$object_id2\" \"$length2\" >>expect &&\n> > + git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id1\" \"$object_id2\" >actual &&\n> > + test_cmp expect actual\n> > + )\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info fallbacks to standard fetch if object-info is not supported by the server' '\n> > + (\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + object_id=$(git rev-parse message1:a.txt) &&\n> > + length=$(wc -c > +\n> > + printf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> > + git config objectinfo.advertise false &&\n> > + git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" >actual &&\n> > + test_cmp expect actual\n> > + )\n>\n> I noticed this test doesn't confirm that standard fetch was actually used. Would\n> it be useful to check for the existence of the object to prove it got fetched?\n>\n> Also, is seems the way this test is set up, it's fetching from itself.\n> This is logically fine in most cases that client == server, but it was a little\n> confusing at first because it looked like the invocation of git config was on\n> the client rather than the server, until I realized they are the same.\n>\n> Since the server is being used as the client, we wouldn't be able to confirm that\n> the object got fetched. I realize for most of these tests, it doesn't really matter\n> to conflate client with server. I think this is the one exception.\n>\n> But, as a larger question for the list, I'm wondering if it's best practice to separate client from\n> server in tests, or if it's an accepted style to just use one repository in cases\n> when it doesn't really matter?\n>\n> Thanks\n> John\n>\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info fails on server with legacy protocol' '\n> > + (\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + object_id=$(git rev-parse message1:a.txt) &&\n> > + test_must_fail git -c protocol.version=0 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" 2>err &&\n> > + test_i18ngrep \"object-info requires protocol v2\" err\n> > + )\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info fails on malformed OID' '\n> > + (\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > + malformed_object_id=\"this_id_is_not_valid\" &&\n> > + test_must_fail git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$malformed_object_id\" 2>err &&\n> > + test_i18ngrep \"malformed object id '$malformed_object_id'\" err\n> > + )\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info fails on missing OID' '\n> > + git clone \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" missing_oid_repo &&\n> > + test_commit -C missing_oid_repo message3 c.txt &&\n> > + (\n> > + cd missing_oid_repo &&\n> > + object_id=$(git rev-parse message3:c.txt) &&\n> > + test_must_fail env GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch --object-info=size \"$HTTPD_URL/smart/http_parent\" \"$object_id\" 2>err &&\n> > + test_i18ngrep \"fatal: remote error: upload-pack: not our ref $object_id\" err\n> > + )\n> > +'\n> > +\n> > +# Test fetch object-info with 'file://' transport\n> > +\n> > +test_expect_success 'create repo to be served by file:// transport' '\n> > + git init server &&\n> > + test_commit -C server message1 a.txt &&\n> > + git -C server config protocol.version 2\n> > +'\n> > +\n> > +test_expect_success 'fetch object-info with file:// using protocol v2' '\n> > + (\n> > + cd server &&\n> > + object_id=$(git rev-parse message1:a.txt) &&\n> > + length=$(wc -c > +\n> > + printf \"%s %d\\n\" \"$object_id\" \"$length\" >expect &&\n> > + git -c protocol.version=2 fetch --object-info=size \"file://$(pwd)\" \"$object_id\" >actual &&\n> > + test_cmp expect actual\n> > + )\n> > +'\n> > +\n> > +test_done\n> > diff --git a/transport-helper.c b/transport-helper.c\n> > index a0297b0986..64c175e904 100644\n> > --- a/transport-helper.c\n> > +++ b/transport-helper.c\n> > @@ -686,13 +686,17 @@ static int fetch_refs(struct transport *transport,\n> >\n> > /*\n> > * If we reach here, then the server, the client, and/or the transport\n> > - * helper does not support protocol v2. --negotiate-only requires\n> > - * protocol v2.\n> > + * helper does not support protocol v2. --negotiate-only and --object-info\n> > + * require protocol v2.\n> > */\n> > if (data->transport_options.acked_commits) {\n> > warning(_(\"--negotiate-only requires protocol v2\"));\n> > return -1;\n> > }\n> > + if (transport->smart_options->object_info) {\n> > + warning(_(\"--object-info requires protocol v2\"));\n> > + return -1;\n> > + }\n> >\n> > if (!data->get_refs_list_called)\n> > get_refs_list_using_list(transport, 0);\n> > diff --git a/transport-internal.h b/transport-internal.h\n> > index c4ca0b733a..04fa015011 100644\n> > --- a/transport-internal.h\n> > +++ b/transport-internal.h\n> > @@ -59,6 +59,7 @@ struct transport_vtable {\n> > * use. disconnect() releases these resources.\n> > **/\n> > int (*disconnect)(struct transport *connection);\n> > + int (*fetch_object_info)(struct transport *transport, struct oid_array *oids);\n> > };\n> >\n> > #endif\n> > diff --git a/transport.c b/transport.c\n> > index 70e9840a90..65a1b1fdb3 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -350,6 +350,67 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> > return refs;\n> > }\n> >\n> > +/*\n> > + * Fetches object info if server supports object-info command\n> > + * Make sure to fallback to normal fetch if this fails\n> > + */\n> > +static int fetch_object_info(struct transport *transport)\n> > +{\n> > + struct git_transport_data *data = transport->data;\n> > + struct object_info_args args;\n> > + struct packet_reader reader;\n> > +\n> > + memset(&args, 0, sizeof(args));\n> > + args.server_options = transport->server_options;\n> > + args.object_info_options = transport->smart_options->object_info_options;\n> > + args.oids = transport->smart_options->object_info_oids;\n> > +\n> > + connect_setup(transport, 0);\n> > + packet_reader_init(&reader, data->fd[0], NULL, 0,\n> > + PACKET_READ_CHOMP_NEWLINE |\n> > + PACKET_READ_DIE_ON_ERR_PACKET);\n> > + data->version = discover_version(&reader);\n> > +\n> > + transport->hash_algo = reader.hash_algo;\n> > +\n> > + switch (data->version) {\n> > + case protocol_v2:\n> > + if (!server_supports_v2(\"object-info\", 0))\n> > + return 0;\n> > + send_object_info_request(data->fd[1], &args);\n> > + break;\n> > + case protocol_v1:\n> > + case protocol_v0:\n> > + die(_(\"wrong protocol version. expected v2\"));\n> > + case protocol_unknown_version:\n> > + BUG(\"unknown protocol version\");\n> > + }\n> > +\n> > + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> > + die(_(\"error reading object info header\"));\n> > + }\n> > + if (strcmp(reader.line, \"size\")) {\n> > + die(_(\"expected 'size', received '%s'\"), reader.line);\n> > + }\n> > + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> > + printf(\"%s\\n\", reader.line);\n> > + }\n> > + check_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> > +\n> > + return 1;\n> > +}\n> > +\n> > +static int append_oid_to_ref(const struct object_id *oid, void *data)\n> > +{\n> > + struct ref *ref = data;\n> > + struct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n> > + temp_ref->old_oid = *oid;\n> > + temp_ref->exact_oid = 1;\n> > + ref->next = temp_ref;\n> > + ref = ref->next;\n> > + return 0;\n> > +}\n> > +\n> > static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n> > struct transport_ls_refs_options *options)\n> > {\n> > @@ -364,6 +425,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > struct ref *refs = NULL;\n> > struct fetch_pack_args args;\n> > struct ref *refs_tmp = NULL;\n> > + struct ref *wanted_refs = xcalloc(1, sizeof (struct ref));\n> >\n> > memset(&args, 0, sizeof(args));\n> > args.uploadpack = data->options.uploadpack;\n> > @@ -388,8 +450,17 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > args.server_options = transport->server_options;\n> > args.negotiation_tips = data->options.negotiation_tips;\n> > args.reject_shallow_remote = transport->smart_options->reject_shallow;\n> > + args.object_info = transport->smart_options->object_info;\n> >\n> > - if (!data->got_remote_heads) {\n> > + if (transport->smart_options && transport->smart_options->object_info) {\n> > + struct ref *temp_ref = wanted_refs;\n> > + if (fetch_object_info(transport)) {\n> > + goto cleanup;\n> > + }\n> > + oid_array_for_each(transport->smart_options->object_info_oids, append_oid_to_ref, temp_ref);\n> > + wanted_refs = wanted_refs->next;\n> > + transport->remote_refs = wanted_refs;\n> > + } else if (!data->got_remote_heads) {\n> > int i;\n> > int must_list_refs = 0;\n> > for (i = 0; i < nr_heads; i++) {\n> > @@ -406,7 +477,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > else if (data->version <= protocol_v1)\n> > die_if_server_options(transport);\n> >\n> > - if (data->options.acked_commits) {\n> > + if (data->options.acked_commits && !transport->smart_options->object_info) {\n> > if (data->version < protocol_v2) {\n> > warning(_(\"--negotiate-only requires protocol v2\"));\n> > ret = -1;\n> > diff --git a/transport.h b/transport.h\n> > index a0bc6a1e9e..3bba16f86e 100644\n> > --- a/transport.h\n> > +++ b/transport.h\n> > @@ -30,6 +30,12 @@ struct git_transport_options {\n> > */\n> > unsigned connectivity_checked:1;\n> >\n> > + /*\n> > + * Transport will attempt to pull only object-info. Fallbacks\n> > + * to pulling entire object if object-info is not supported\n> > + */\n> > + unsigned object_info : 1;\n> > +\n> > int depth;\n> > const char *deepen_since;\n> > const struct string_list *deepen_not;\n> > @@ -53,6 +59,9 @@ struct git_transport_options {\n> > * common commits to this oidset instead of fetching any packfiles.\n> > */\n> > struct oidset *acked_commits;\n> > +\n> > + struct oid_array *object_info_oids;\n> > + struct string_list *object_info_options;\n> > };\n> >\n> > enum transport_family {\n> > --\n> > 2.33.0.664.g0785eb7698\n"},{"id":"452904","messageId":"xmqqpmm0ye1j.fsf@gitster.g","threadId":"57388","inReplyTo":"CAFySSZAwXJc6BG8Q9P3h9njyMGz9YsOuCWSwTp+33vafiJ4L_A@mail.gmail.com","subject":"Re: [PATCH v3 3/3] object-info: add option for retrieving object info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-01T16:16:40Z","receivedAt":"2022-04-01T16:37:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n>> But, as a larger question for the list, I'm wondering if it's\n>> best practice to separate client from server in tests, or if it's\n>> an accepted style to just use one repository in cases when it\n>> doesn't really matter?\n> I am as well!\n\nI do not speak for \"the list\", but many fetch-push tests were\nwritten to use the same repository, and many early submodule tests\nimitated them to use the initial test repository and add it as its\nsubmodule, out of pure laziness (and I am probably the most to be\nblamed)---these scenarios are not very useful but when the transfer\nitself is not the focus of the tests it was a handy way. Of course\nit is not encouraged, and those tests that do care about seeing\nobjects getting transferred do make sure that they are as\nindependent as the real world settings that we want the feature\nbeing tested to be working in.\n"},{"id":"454726","messageId":"20220502170904.2770649-1-calvinwan@google.com","threadId":"57388","inReplyTo":"20220328191112.3092139-1-calvinwan@google.com","subject":"[PATCH v4 0/8] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:08:56Z","receivedAt":"2022-05-02T17:09:46Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Sometimes it is useful to get information about an object without having\nto download it completely. The server logic has already been implemented\nas “a2ba162cda (object-info: support for retrieving object info,\n2021-04-20)”. This patch implements the client option for it.\n\nAdd `--object-info` option to `cat-file --batch-command`. This option\nallows the client to make an object-info command request to a server\nthat supports protocol v2. If the server is v2, but does not allow for\nthe object-info command request, the entire object is fetched and the\nrelevant object info is returned.\n\nSummary of changes ==================\n\nPatches 1, 2, 3, and 7 are small changes that setup the main\nimplementation. Patch 4 sets up object-info to be backwards compatible\nfor future patch series that adds additional attributes. Patch 5 adds\ninternal trasnport functions to send and receive object-info command\nrequest packets. Patch 6 adds the fallback if object-info is not\nsupported or fails. Patch 8 adds the cat-file implementation.\n\nChanges since V3 ================\n\n * Object-info is now implemented in cat-file --batch-command rather\n than fetch (new base commit)\n * Removed config option to advertise object-info\n * Added forwards and backwards compability for object-info\n * Split up some patches to better describe and visualize changes\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n\nCalvin Wan (8):\n fetch-pack: refactor packet writing\n fetch-pack: move fetch default settings\n object-store: add function to free object_info contents\n object-info: send attribute packet regardless of object ids\n transport: add client side capability to request object-info\n transport: add object-info fallback to fetch\n cat-file: move parse_cmd and DEFAULT_FORMAT up\n cat-file: add --batch-command remote-object-info command\n\n Documentation/git-cat-file.txt | 16 +-\n builtin/cat-file.c | 225 ++++++++++++++++-----\n fetch-pack.c | 61 ++++--\n fetch-pack.h | 10 +\n object-file.c | 16 ++\n object-store.h | 3 +\n protocol-caps.c | 14 +-\n t/t1006-cat-file.sh | 347 +++++++++++++++++++++++++++++++++\n transport-helper.c | 7 +-\n transport.c | 97 ++++++++-\n transport.h | 11 ++\n 11 files changed, 728 insertions(+), 79 deletions(-)\n\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454727","messageId":"20220502170904.2770649-2-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 1/8] fetch-pack: refactor packet writing","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:08:57Z","receivedAt":"2022-05-02T17:09:55Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A subsequent patch will need to write capabilities for another command,\nso refactor write_fetch_command_and_capabilities() to be used by both\nfetch and future command.\n\n---\n fetch-pack.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 4e1e88eea0..e06125c90a 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1246,13 +1246,13 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \treturn haves_added;\n }\n \n-static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n-\t\t\t\t\t\t const struct string_list *server_options)\n+static void write_command_and_capabilities(struct strbuf *req_buf,\n+\t\t\t\t\t\t const struct string_list *server_options, const char* command)\n {\n \tconst char *hash_name;\n \n-\tif (server_supports_v2(\"fetch\", 1))\n-\t\tpacket_buf_write(req_buf, \"command=fetch\");\n+\tif (server_supports_v2(command, 1))\n+\t\tpacket_buf_write(req_buf, \"command=%s\", command);\n \tif (server_supports_v2(\"agent\", 0))\n \t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n \tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n@@ -1288,7 +1288,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \tint done_sent = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n-\twrite_fetch_command_and_capabilities(&req_buf, args->server_options);\n+\twrite_command_and_capabilities(&req_buf, args->server_options, \"fetch\");\n \n \tif (args->use_thin_pack)\n \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n@@ -2072,7 +2072,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n \t\tint received_ready = 0;\n \n \t\tstrbuf_reset(&req_buf);\n-\t\twrite_fetch_command_and_capabilities(&req_buf, server_options);\n+\t\twrite_command_and_capabilities(&req_buf, server_options, \"fetch\");\n \n \t\tpacket_buf_write(&req_buf, \"wait-for-done\");\n \n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454728","messageId":"20220502170904.2770649-4-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 3/8] object-store: add function to free object_info contents","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:08:59Z","receivedAt":"2022-05-02T17:10:05Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Introduce free_object_info_contents.\n\n---\n object-file.c | 16 ++++++++++++++++\n object-store.h | 3 +++\n 2 files changed, 19 insertions(+)\n\ndiff --git a/object-file.c b/object-file.c\nindex 5ffbf3d4fd..34a6e34adf 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2659,3 +2659,19 @@ int read_loose_object(const char *path,\n \t\tmunmap(map, mapsize);\n \treturn ret;\n }\n+\n+void free_object_info_contents(struct object_info *object_info)\n+{\n+\tif (!object_info)\n+\t\treturn;\n+\tif (object_info->typep)\n+\t\tfree(object_info->typep);\n+\tif (object_info->sizep)\n+\t\tfree(object_info->sizep);\n+\tif (object_info->disk_sizep)\n+\t\tfree(object_info->disk_sizep);\n+\tif (object_info->delta_base_oid)\n+\t\tfree(object_info->delta_base_oid);\n+\tif (object_info->type_name)\n+\t\tfree(object_info->type_name);\n+}\n\\ No newline at end of file\ndiff --git a/object-store.h b/object-store.h\nindex bd2322ed8c..840e04b56f 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -533,4 +533,7 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+/* Free pointers inside of object_info, but not object_info itself */\n+void free_object_info_contents(struct object_info *object_info);\n+\n #endif /* OBJECT_STORE_H */\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454729","messageId":"20220502170904.2770649-5-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 4/8] object-info: send attribute packet regardless of object ids","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:09:00Z","receivedAt":"2022-05-02T17:10:09Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Currently on the server side of object-info, if the client does not send\nany object ids in the request or if the client requests an attribute the\nserver does not support, the server will either not send any packets\nback or error out. Consider the scenario where the client git version is\nahead of the server git version, and the client can request additional\nobject-info besides 'size'. There needs to be a way to tell whether the\nserver can honor all of the client attribute requests before the client\nsends a request with all of the object ids. In a future patch, if the\nclient were to make an initial command request with only attributes, the\nserver would be able to confirm which attributes it could return.\n\n---\n protocol-caps.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex bbde91810a..bc7def0727 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -11,6 +11,7 @@\n \n struct requested_info {\n \tunsigned size : 1;\n+\tunsigned unknown : 1;\n };\n \n /*\n@@ -40,12 +41,12 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \tstruct string_list_item *item;\n \tstruct strbuf send_buffer = STRBUF_INIT;\n \n-\tif (!oid_str_list->nr)\n-\t\treturn;\n-\n \tif (info->size)\n \t\tpacket_writer_write(writer, \"size\");\n \n+\tif (info->unknown || !oid_str_list->nr)\n+\t\tgoto release;\n+\n \tfor_each_string_list_item (item, oid_str_list) {\n \t\tconst char *oid_str = item->string;\n \t\tstruct object_id oid;\n@@ -72,12 +73,13 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n \t\tstrbuf_reset(&send_buffer);\n \t}\n+release:\n \tstrbuf_release(&send_buffer);\n }\n \n int cap_object_info(struct repository *r, struct packet_reader *request)\n {\n-\tstruct requested_info info;\n+\tstruct requested_info info = { 0 };\n \tstruct packet_writer writer;\n \tstruct string_list oid_str_list = STRING_LIST_INIT_DUP;\n \n@@ -92,9 +94,7 @@ int cap_object_info(struct repository *r, struct packet_reader *request)\n \t\tif (parse_oid(request->line, &oid_str_list))\n \t\t\tcontinue;\n \n-\t\tpacket_writer_error(&writer,\n-\t\t\t\t \"object-info: unexpected line: '%s'\",\n-\t\t\t\t request->line);\n+\t\tinfo.unknown = 1;\n \t}\n \n \tif (request->status != PACKET_READ_FLUSH) {\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454730","messageId":"20220502170904.2770649-3-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 2/8] fetch-pack: move fetch default settings","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:08:58Z","receivedAt":"2022-05-02T17:10:11Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"When the state machine in do_fetch_pack_v2() is in FETCH_CHECK_LOCAL, we\nset a few v2-specific defaults. It will be helpful for a future patch to\nhave these defaults set if the initial state is not FETCH_CHECK_LOCAL.\nThis is a safe change since the initial state is currently always\nFETCH_CHECK_LOCAL, so we're guaranteed to hit that code whether it's in\nthe state machine or not.\n\n---\n fetch-pack.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex e06125c90a..45473b4602 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1610,18 +1610,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\treader.me = \"fetch-pack\";\n \t}\n \n+\t/* v2 supports these by default */\n+\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\tuse_sideband = 2;\n+\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n+\t\targs->deepen = 1;\n+\n \twhile (state != FETCH_DONE) {\n \t\tswitch (state) {\n \t\tcase FETCH_CHECK_LOCAL:\n \t\t\tsort_ref_list(&ref, ref_compare_name);\n \t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n \n-\t\t\t/* v2 supports these by default */\n-\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n-\t\t\tuse_sideband = 2;\n-\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n-\t\t\t\targs->deepen = 1;\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454731","messageId":"20220502170904.2770649-6-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 5/8] transport: add client side capability to request object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:09:01Z","receivedAt":"2022-05-02T17:10:15Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Sometimes it is useful to get information about an object without having\nto download it completely. The server logic has already been implemented\nas “a2ba162cda (object-info: support for retrieving object info,\n2021-04-20)”. This patch implements the client option for it. Currently,\nonly 'size' is supported and the server must be v2, however future\npatches can implement additional options.\n\nThe question of version mismatch often comes up with client/server\nrelationships. There are two cases to consider here (assuming either\nclient or server functionality for object-info changes between the\ndifferent versions):\n\n 1. client version > server version\n \t- client can request additional attributes from server\n 2. server version > client version\n \t- server can return additional info to the client\n\nThe second case is a non-issue since the client would never be able to\nrequest additional info from the server. In order to solve the first\ncase, recall an earlier patch where the server sends back the attributes\neven if no object ids are sent by the client. This allows the client to\nfirst check whether the server can accept the requested attributes\nbefore sending the entire request.\n\n---\n fetch-pack.c | 23 +++++++++++++\n fetch-pack.h | 8 +++++\n transport-helper.c | 7 ++--\n transport.c | 81 +++++++++++++++++++++++++++++++++++++++++++++-\n transport.h | 11 +++++++\n 5 files changed, 127 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 45473b4602..506ca28e05 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1278,6 +1278,29 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n \tpacket_buf_delim(req_buf);\n }\n \n+void send_object_info_request(int fd_out, struct object_info_args *args)\n+{\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\tstruct string_list unsorted_object_info_options = *args->object_info_options;\n+\tsize_t i;\n+\n+\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n+\n+\tif (unsorted_string_list_has_string(&unsorted_object_info_options, \"size\"))\n+\t\tpacket_buf_write(&req_buf, \"size\");\n+\n+\tif (args->oids) {\n+\t\tfor (i = 0; i < args->oids->nr; i++)\n+\t\t\tpacket_buf_write(&req_buf, \"oid %s\", oid_to_hex(&args->oids->oid[i]));\n+\t}\n+\n+\tpacket_buf_flush(&req_buf);\n+\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n+\t\tdie_errno(_(\"unable to write request to remote\"));\n+\n+\tstrbuf_release(&req_buf);\n+}\n+\n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 8c7752fc82..c27018d48c 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -69,6 +69,12 @@ struct fetch_pack_args {\n \tunsigned connectivity_checked:1;\n };\n \n+struct object_info_args {\n+\tconst struct string_list *object_info_options;\n+\tconst struct string_list *server_options;\n+\tconst struct oid_array *oids;\n+};\n+\n /*\n * sought represents remote references that should be updated from.\n * On return, the names that were found on the remote will have been\n@@ -102,4 +108,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+void send_object_info_request(int fd_out, struct object_info_args *args);\n+\n #endif\ndiff --git a/transport-helper.c b/transport-helper.c\nindex b4dbbabb0c..093946f7fd 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -686,13 +686,16 @@ static int fetch_refs(struct transport *transport,\n \n \t/*\n \t * If we reach here, then the server, the client, and/or the transport\n-\t * helper does not support protocol v2. --negotiate-only requires\n-\t * protocol v2.\n+\t * helper does not support protocol v2. --negotiate-only and --object-info\n+\t * require protocol v2.\n \t */\n \tif (data->transport_options.acked_commits) {\n \t\twarning(_(\"--negotiate-only requires protocol v2\"));\n \t\treturn -1;\n \t}\n+\tif (transport->smart_options->object_info) {\n+\t\tdie(_(\"--object-info requires protocol v2\"));\n+\t}\n \n \tif (!data->get_refs_list_called)\n \t\tget_refs_list_using_list(transport, 0);\ndiff --git a/transport.c b/transport.c\nindex 3d64a43ab3..08c505e1d0 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -353,6 +353,79 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \treturn refs;\n }\n \n+static int fetch_object_info(struct transport *transport, struct object_info **object_info_data)\n+{\n+\tsize_t i;\n+\tint size_index = -1;\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct object_info_args args;\n+\tstruct packet_reader reader;\n+\tstruct string_list server_attributes = STRING_LIST_INIT_DUP;\n+\n+\tmemset(&args, 0, sizeof(args));\n+\targs.server_options = transport->server_options;\n+\targs.object_info_options = transport->smart_options->object_info_options;\n+\n+\tconnect_setup(transport, 0);\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\tPACKET_READ_CHOMP_NEWLINE |\n+\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n+\tdata->version = discover_version(&reader);\n+\n+\ttransport->hash_algo = reader.hash_algo;\n+\n+\tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tif (!server_supports_v2(\"object-info\", 0))\n+\t\t\treturn -1;\n+\t\t/**\n+\t\t * Send a request with only attributes first. If server can return all\n+\t\t * of the requested attributes, then send request with object ids\n+\t\t */\n+\t\tsend_object_info_request(data->fd[1], &args);\n+\t\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n+\t\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_list_split(&server_attributes, reader.line, ' ', -1);\n+\t\tpacket_reader_read(&reader);\n+\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\t\tif (server_attributes.nr != args.object_info_options->nr)\n+\t\t\treturn -1;\n+\t\tfor (i = 0; i < server_attributes.nr; i++) {\n+\t\t\tif (!strcmp(server_attributes.items[i].string, \"size\"))\n+\t\t\t\tsize_index = i + 1;\n+\t\t}\n+\t\targs.oids = transport->smart_options->object_info_oids;\n+\t\tsend_object_info_request(data->fd[1], &args);\n+\t\tbreak;\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tdie(_(\"wrong protocol version. expected v2\"));\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL)\n+\t\tdie(_(\"error reading object info header\"));\n+\ti = 0;\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\t\tstruct string_list object_info_values = STRING_LIST_INIT_DUP;\n+\n+\t\tstring_list_split(&object_info_values, reader.line, ' ', -1);\n+\t\tif (size_index > 0) {\n+\t\t\tif (!strcmp(object_info_values.items[size_index].string, \"\"))\n+\t\t\t\tdie(\"object-info: not our ref %s\",\n+\t\t\t\t\tobject_info_values.items[0].string);\n+\t\t\t*(*object_info_data)[i].sizep = strtoul(object_info_values.items[size_index].string, NULL, 10);\n+\t\t}\n+\t\ti++;\n+\t}\n+\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\n+\treturn 0;\n+}\n+\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n \t\t\t\t\tstruct transport_ls_refs_options *options)\n {\n@@ -392,8 +465,14 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.server_options = transport->server_options;\n \targs.negotiation_tips = data->options.negotiation_tips;\n \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n+\targs.object_info = transport->smart_options->object_info;\n \n-\tif (!data->got_remote_heads) {\n+\tif (transport->smart_options && transport->smart_options->object_info) {\n+\t\tif (!fetch_object_info(transport, data->options.object_info_data))\n+\t\t\tgoto cleanup;\n+\t\tret = -1;\n+\t\tgoto cleanup;\n+\t} else if (!data->got_remote_heads) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n \t\tfor (i = 0; i < nr_heads; i++) {\ndiff --git a/transport.h b/transport.h\nindex 12bc08fc33..dbf60bb710 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -6,6 +6,7 @@\n #include \"remote.h\"\n #include \"list-objects-filter-options.h\"\n #include \"string-list.h\"\n+#include \"object-store.h\"\n \n struct git_transport_options {\n \tunsigned thin : 1;\n@@ -31,6 +32,12 @@ struct git_transport_options {\n \t */\n \tunsigned connectivity_checked:1;\n \n+\t/*\n+\t * Transport will attempt to pull only object-info. Fallbacks\n+\t * to pulling entire object if object-info is not supported\n+\t */\n+\tunsigned object_info : 1;\n+\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n@@ -54,6 +61,10 @@ struct git_transport_options {\n \t * common commits to this oidset instead of fetching any packfiles.\n \t */\n \tstruct oidset *acked_commits;\n+\n+\tstruct oid_array *object_info_oids;\n+\tstruct object_info **object_info_data;\n+\tstruct string_list *object_info_options;\n };\n \n enum transport_family {\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454732","messageId":"20220502170904.2770649-8-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 7/8] cat-file: move parse_cmd and DEFAULT_FORMAT up","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:09:03Z","receivedAt":"2022-05-02T17:10:33Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"It would be useful for a future patch to use these definitions so moving\nthem higher.\n\n---\n builtin/cat-file.c | 25 ++++++++++++-------------\n 1 file changed, 12 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 50cf38999d..cfe74c36eb 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -34,6 +34,8 @@ struct batch_options {\n \tconst char *format;\n };\n \n+#define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n+\n static const char *force_path;\n \n static int filter_object(const char *path, unsigned mode,\n@@ -556,6 +558,16 @@ static void parse_cmd_info(struct batch_options *opt,\n \tbatch_one_object(line, output, opt, data);\n }\n \n+static const struct parse_cmd {\n+\tconst char *name;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+\t{ \"flush\", NULL, 0},\n+};\n+\n static void dispatch_calls(struct batch_options *opt,\n \t\tstruct strbuf *output,\n \t\tstruct expand_data *data,\n@@ -583,17 +595,6 @@ static void free_cmds(struct queued_cmd *cmd, size_t *nr)\n \t*nr = 0;\n }\n \n-\n-static const struct parse_cmd {\n-\tconst char *name;\n-\tparse_cmd_fn_t fn;\n-\tunsigned takes_args;\n-} commands[] = {\n-\t{ \"contents\", parse_cmd_contents, 1},\n-\t{ \"info\", parse_cmd_info, 1},\n-\t{ \"flush\", NULL, 0},\n-};\n-\n static void batch_objects_command(struct batch_options *opt,\n \t\t\t\t struct strbuf *output,\n \t\t\t\t struct expand_data *data)\n@@ -659,8 +660,6 @@ static void batch_objects_command(struct batch_options *opt,\n \tstrbuf_release(&input);\n }\n \n-#define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n-\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454733","messageId":"20220502170904.2770649-7-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 6/8] transport: add object-info fallback to fetch","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:09:02Z","receivedAt":"2022-05-02T17:10:33Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"If a v2 server does not support object-info or if the client requests\ninformation that isn't supported by object-info, fetch the objects as if\nit were a standard v2 fetch (however without changing any refs).\n\n---\n fetch-pack.c | 14 +++++++++++++-\n fetch-pack.h | 2 ++\n transport.c | 18 +++++++++++++++---\n 3 files changed, 30 insertions(+), 4 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 506ca28e05..938d082b80 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1639,6 +1639,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\targs->deepen = 1;\n \n+\tif (args->object_info)\n+\t\tstate = FETCH_SEND_REQUEST;\n+\n \twhile (state != FETCH_DONE) {\n \t\tswitch (state) {\n \t\tcase FETCH_CHECK_LOCAL:\n@@ -1648,7 +1651,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n-\t\t\tif (!args->refetch && everything_local(args, &ref))\n+\t\t\tif (!args->refetch && !args->object_info && everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n@@ -2035,6 +2038,15 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\targs->connectivity_checked = 1;\n \t}\n \n+\tif (args->object_info) {\n+\t\tstruct ref *ref_cpy_reader = ref_cpy;\n+\t\tint i = 0;\n+\t\twhile (ref_cpy_reader) {\n+\t\t\toid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args->object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE);\n+\t\t\tref_cpy_reader = ref_cpy_reader->next;\n+\t\t\ti++;\n+\t\t}\n+\t}\n \tupdate_shallow(args, sought, nr_sought, &si);\n cleanup:\n \tclear_shallow_info(&si);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex c27018d48c..552fd7bde0 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -17,6 +17,7 @@ struct fetch_pack_args {\n \tconst struct string_list *deepen_not;\n \tstruct list_objects_filter_options filter_options;\n \tconst struct string_list *server_options;\n+\tstruct object_info **object_info_data;\n \n \t/*\n \t * If not NULL, during packfile negotiation, fetch-pack will send \"have\"\n@@ -43,6 +44,7 @@ struct fetch_pack_args {\n \tunsigned reject_shallow_remote:1;\n \tunsigned deepen:1;\n \tunsigned refetch:1;\n+\tunsigned object_info:1;\n \n \t/*\n \t * Indicate that the remote of this request is a promisor remote. The\ndiff --git a/transport.c b/transport.c\nindex 08c505e1d0..b85e52d9a8 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -436,10 +436,12 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n \tint ret = 0;\n+\tsize_t i;\n \tstruct git_transport_data *data = transport->data;\n \tstruct ref *refs = NULL;\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n+\tstruct ref *wanted_refs = xcalloc(1, sizeof (struct ref));\n \n \tmemset(&args, 0, sizeof(args));\n \targs.uploadpack = data->options.uploadpack;\n@@ -468,10 +470,19 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.object_info = transport->smart_options->object_info;\n \n \tif (transport->smart_options && transport->smart_options->object_info) {\n+\t\tstruct ref *ref = wanted_refs;\n+\n \t\tif (!fetch_object_info(transport, data->options.object_info_data))\n \t\t\tgoto cleanup;\n-\t\tret = -1;\n-\t\tgoto cleanup;\n+\t\targs.object_info_data = data->options.object_info_data;\n+\t\tfor (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n+\t\t\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n+\t\t\ttemp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i);\n+\t\t\ttemp_ref->exact_oid = 1;\n+\t\t\tref->next = temp_ref;\n+\t\t\tref = ref->next;\n+\t\t}\n+\t\ttransport->remote_refs = wanted_refs->next;\n \t} else if (!data->got_remote_heads) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n@@ -489,7 +500,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \telse if (data->version <= protocol_v1)\n \t\tdie_if_server_options(transport);\n \n-\tif (data->options.acked_commits) {\n+\tif (data->options.acked_commits && !transport->smart_options->object_info) {\n \t\tif (data->version < protocol_v2) {\n \t\t\twarning(_(\"--negotiate-only requires protocol v2\"));\n \t\t\tret = -1;\n@@ -532,6 +543,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tfree_refs(refs_tmp);\n \tfree_refs(refs);\n+\tfree_refs(wanted_refs);\n \treturn ret;\n }\n \n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454734","messageId":"20220502170904.2770649-9-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-02T17:09:04Z","receivedAt":"2022-05-02T17:10:33Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Since the `info` command in cat-file --batch-command prints object info\nfor a given object, it is natural to add another command in cat-file\n--batch-command to print object info for a given object from a remote.\nAdd `remote-object-info` to cat-file --batch-command.\n\nWhile `info` takes object ids one at a time, this creates overhead when\nmaking requests to a server so `remote-object-info` instead can take\nmultiple object ids at once.\n\ncat-file --batch-command is generally implemented in the following\nmanner:\n\n - Receive and parse input from user\n - Call respective function attached to command\n - Set batch mode state, get object info, print object info\n\nIn --buffer mode, this changes to:\n\n - Receive and parse input from user\n - Store respective function attached to command in a queue\n - After flush, loop through commands in queue\n - Call respective function attached to command\n - Set batch mode state, get object info, print object info\n\nNotice how the getting and printing of object info is accomplished one\nat a time. As described above, this creates a problem for making\nrequests to a server. Therefore, `remote-object-info` is implemented in\nthe following manner:\n\n - Receive and parse input from user\n If command is `remote-object-info`:\n - Get object info from remote\n - Loop through object info\n - Call respective function attached to `info`\n - Set batch mode state, use passed in object info, print object\n info\n Else:\n - Call respective function attached to command\n - Parse input, get object info, print object info\n\nAnd finally for --buffer mode `remote-object-info`:\n - Receive and parse input from user\n - Store respective function attached to command in a queue\n - After flush, loop through commands in queue:\n If command is `remote-object-info`:\n - Get object info from remote\n - Loop through object info\n - Call respective function attached to `info`\n - Set batch mode state, use passed in object info, print\n object info\n Else:\n - Call respective function attached to command\n - Set batch mode state, get object info, print object info\n\nTo summarize, `remote-object-info` gets object info from the remote and\nthen generates multiples `info` commands with the object info passed in.\nIt cannot be implemented similarly to `info` and `content` because of\nthe overhead with remote commenication.\n\n---\n Documentation/git-cat-file.txt | 16 +-\n builtin/cat-file.c | 200 +++++++++++++++----\n t/t1006-cat-file.sh | 347 +++++++++++++++++++++++++++++++++\n 3 files changed, 522 insertions(+), 41 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex 24a811f0ef..0d9e8e6214 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -115,6 +115,10 @@ info ::\n \tPrint object info for object reference ``. This corresponds to the\n \toutput of `--batch-check`.\n \n+remote-object-info []::\n+\tPrint object info for object references `` at specified .\n+\tThis command may only be combined with `--buffer`.\n+\n flush::\n \tUsed with `--buffer` to execute all preceding commands that were issued\n \tsince the beginning or since the last flush was issued. When `--buffer`\n@@ -245,7 +249,8 @@ newline. The available atoms are:\n \tThe full hex representation of the object name.\n \n `objecttype`::\n-\tThe type of the object (the same as `cat-file -t` reports).\n+\tThe type of the object (the same as `cat-file -t` reports). See\n+\t`CAVEATS` below.\n \n `objectsize`::\n \tThe size, in bytes, of the object (the same as `cat-file -s`\n@@ -253,13 +258,14 @@ newline. The available atoms are:\n \n `objectsize:disk`::\n \tThe size, in bytes, that the object takes up on disk. See the\n-\tnote about on-disk sizes in the `CAVEATS` section below.\n+\tnote about on-disk sizes in the `CAVEATS` section below. Not\n+\tsupported by `remote-object-info`.\n \n `deltabase`::\n \tIf the object is stored as a delta on-disk, this expands to the\n \tfull hex representation of the delta base object name.\n \tOtherwise, expands to the null OID (all zeroes). See `CAVEATS`\n-\tbelow.\n+\tbelow. Not supported by `remote-object-info`.\n \n `rest`::\n \tIf this atom is used in the output string, input lines are split\n@@ -346,6 +352,10 @@ directory name.\n CAVEATS\n -------\n \n+Note that since object type is currently not supported by the\n+object-info command request, git fetches the entire object instead to\n+get the object info.\n+\n Note that the sizes of objects on disk are reported accurately, but care\n should be taken in drawing conclusions about which refs or objects are\n responsible for disk usage. The size of a packed non-delta object may be\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex cfe74c36eb..7baa9bed61 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -16,6 +16,9 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n+#include \"alias.h\"\n+#include \"remote.h\"\n+#include \"transport.h\"\n \n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n@@ -32,11 +35,14 @@ struct batch_options {\n \tint unordered;\n \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n+\tint use_remote_info;\n };\n \n #define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n \n static const char *force_path;\n+static struct object_info *remote_object_info;\n+static struct oid_array object_info_oids = OID_ARRAY_INIT;\n \n static int filter_object(const char *path, unsigned mode,\n \t\t\t const struct object_id *oid,\n@@ -425,48 +431,115 @@ static void batch_one_object(const char *obj_name,\n \tint flags = opt->follow_symlinks ? GET_OID_FOLLOW_SYMLINKS : 0;\n \tenum get_oid_result result;\n \n-\tresult = get_oid_with_context(the_repository, obj_name,\n-\t\t\t\t flags, &data->oid, &ctx);\n-\tif (result != FOUND) {\n-\t\tswitch (result) {\n-\t\tcase MISSING_OBJECT:\n-\t\t\tprintf(\"%s missing\\n\", obj_name);\n-\t\t\tbreak;\n-\t\tcase SHORT_NAME_AMBIGUOUS:\n-\t\t\tprintf(\"%s ambiguous\\n\", obj_name);\n-\t\t\tbreak;\n-\t\tcase DANGLING_SYMLINK:\n-\t\t\tprintf(\"dangling %\"PRIuMAX\"\\n%s\\n\",\n-\t\t\t (uintmax_t)strlen(obj_name), obj_name);\n-\t\t\tbreak;\n-\t\tcase SYMLINK_LOOP:\n-\t\t\tprintf(\"loop %\"PRIuMAX\"\\n%s\\n\",\n-\t\t\t (uintmax_t)strlen(obj_name), obj_name);\n-\t\t\tbreak;\n-\t\tcase NOT_DIR:\n-\t\t\tprintf(\"notdir %\"PRIuMAX\"\\n%s\\n\",\n-\t\t\t (uintmax_t)strlen(obj_name), obj_name);\n-\t\t\tbreak;\n-\t\tdefault:\n-\t\t\tBUG(\"unknown get_sha1_with_context result %d\\n\",\n-\t\t\t result);\n-\t\t\tbreak;\n+\tif (!opt->use_remote_info) {\n+\t\tresult = get_oid_with_context(the_repository, obj_name,\n+\t\t\t\t\t\tflags, &data->oid, &ctx);\n+\t\tif (result != FOUND) {\n+\t\t\tswitch (result) {\n+\t\t\tcase MISSING_OBJECT:\n+\t\t\t\tprintf(\"%s missing\\n\", obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase SHORT_NAME_AMBIGUOUS:\n+\t\t\t\tprintf(\"%s ambiguous\\n\", obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase DANGLING_SYMLINK:\n+\t\t\t\tprintf(\"dangling %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t\t(uintmax_t)strlen(obj_name), obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase SYMLINK_LOOP:\n+\t\t\t\tprintf(\"loop %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t\t(uintmax_t)strlen(obj_name), obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase NOT_DIR:\n+\t\t\t\tprintf(\"notdir %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t\t(uintmax_t)strlen(obj_name), obj_name);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\tBUG(\"unknown get_sha1_with_context result %d\\n\",\n+\t\t\t\t\tresult);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tfflush(stdout);\n+\t\t\treturn;\n \t\t}\n-\t\tfflush(stdout);\n-\t\treturn;\n-\t}\n \n-\tif (ctx.mode == 0) {\n-\t\tprintf(\"symlink %\"PRIuMAX\"\\n%s\\n\",\n-\t\t (uintmax_t)ctx.symlink_path.len,\n-\t\t ctx.symlink_path.buf);\n-\t\tfflush(stdout);\n-\t\treturn;\n+\t\tif (ctx.mode == 0) {\n+\t\t\tprintf(\"symlink %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t(uintmax_t)ctx.symlink_path.len,\n+\t\t\t\tctx.symlink_path.buf);\n+\t\t\tfflush(stdout);\n+\t\t\treturn;\n+\t\t}\n \t}\n \n \tbatch_object_write(obj_name, scratch, opt, data, NULL, 0);\n }\n \n+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)\n+{\n+\tint retval = 0;\n+\tsize_t i;\n+\tstruct remote *remote = NULL;\n+\tstruct object_id oid;\n+\tstruct string_list object_info_options = STRING_LIST_INIT_NODUP;\n+\tstatic struct transport *gtransport;\n+\tchar *format = DEFAULT_FORMAT;\n+\n+\tif (opt->format)\n+\t\tformat = xstrdup(opt->format);\n+\n+\tremote = remote_get(argv[0]);\n+\tif (!remote)\n+\t\tdie(_(\"must supply valid remote when using --object-info\"));\n+\toid_array_clear(&object_info_oids);\n+\tfor (i = 1; i < argc; i++) {\n+\t\tif (get_oid(argv[i], &oid))\n+\t\t\tdie(_(\"malformed object id '%s'\"), argv[i]);\n+\t\toid_array_append(&object_info_oids, &oid);\n+\t}\n+\n+\tgtransport = transport_get(remote, NULL);\n+\tif (gtransport->smart_options) {\n+\t\tsize_t j;\n+\t\tint include_size = 0, include_type = 0;\n+\n+\t\tremote_object_info = xcalloc(object_info_oids.nr, sizeof(struct object_info));\n+\t\tgtransport->smart_options->object_info = 1;\n+\t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n+\t\t/**\n+\t\t * 'size' is the only option currently supported.\n+\t\t * Other options that are passed in the format will default to a\n+\t\t * standard fetch request rather than object-info.\n+\t\t */\n+\t\tif (strstr(format, \"%(objectsize)\")) {\n+\t\t\tstring_list_append(&object_info_options, \"size\");\n+\t\t\tinclude_size = 1;\n+\t\t}\n+\t\tif (strstr(format, \"%(objecttype)\")) {\n+\t\t\tstring_list_append(&object_info_options, \"type\");\n+\t\t\tinclude_type = 1;\n+\t\t}\n+\t\tif (strstr(format, \"%(objectsize:disk)\"))\n+\t\t\tdie(_(\"objectsize:disk is currently not supported with remote-object-info\"));\n+\t\tif (strstr(format, \"%(deltabase)\"))\n+\t\t\tdie(_(\"deltabase is currently not supported with remote-object-info\"));\n+\t\tif (object_info_options.nr > 0) {\n+\t\t\tgtransport->smart_options->object_info_options = &object_info_options;\n+\t\t\tfor (j = 0; j < object_info_oids.nr; j++) {\n+\t\t\t\tif (include_size)\n+\t\t\t\t\tremote_object_info[j].sizep = xcalloc(1, sizeof(long));\n+\t\t\t\tif (include_type)\n+\t\t\t\t\tremote_object_info[j].typep = xcalloc(1, sizeof(enum object_type));\n+\t\t\t}\n+\t\t\tgtransport->smart_options->object_info_data = &remote_object_info;\n+\t\t\tretval = transport_fetch_refs(gtransport, NULL);\n+\t\t}\n+\t} else {\n+\t\tretval = -1;\n+\t}\n+\treturn retval;\n+}\n+\n struct object_cb_data {\n \tstruct batch_options *opt;\n \tstruct expand_data *expand;\n@@ -538,6 +611,7 @@ typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n struct queued_cmd {\n \tparse_cmd_fn_t fn;\n \tchar *line;\n+\tconst char *name;\n };\n \n static void parse_cmd_contents(struct batch_options *opt,\n@@ -565,9 +639,49 @@ static const struct parse_cmd {\n } commands[] = {\n \t{ \"contents\", parse_cmd_contents, 1},\n \t{ \"info\", parse_cmd_info, 1},\n+\t{ \"remote-object-info\", parse_cmd_info, 1},\n \t{ \"flush\", NULL, 0},\n };\n \n+static void parse_remote_info(struct batch_options *opt,\n+\t\t\t char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t const struct parse_cmd *p_cmd,\n+\t\t\t struct queued_cmd *q_cmd)\n+{\n+\tint count;\n+\tsize_t i;\n+\tconst char **argv;\n+\n+\tcount = split_cmdline(line, &argv);\n+\tif (get_remote_info(opt, count, argv))\n+\t\tgoto cleanup;\n+\topt->use_remote_info = 1;\n+\tdata->skip_object_info = 1;\n+\tdata->mark_query = 0;\n+\tfor (i = 0; i < object_info_oids.nr; i++) {\n+\t\tif (remote_object_info[i].sizep)\n+\t\t\tdata->size = *remote_object_info[i].sizep;\n+\t\tif (remote_object_info[i].typep)\n+\t\t\tdata->type = *remote_object_info[i].typep;\n+\n+\t\tdata->oid = object_info_oids.oid[i];\n+\t\tif (p_cmd)\n+\t\t\tp_cmd->fn(opt, argv[i+1], output, data);\n+\t\telse\n+\t\t\tq_cmd->fn(opt, argv[i+1], output, data);\n+\t}\n+\topt->use_remote_info = 0;\n+\tdata->skip_object_info = 0;\n+\tdata->mark_query = 1;\n+\n+cleanup:\n+\tfor (i = 0; i < object_info_oids.nr; i++)\n+\t\tfree_object_info_contents(&remote_object_info[i]);\n+\tfree(remote_object_info);\n+}\n+\n static void dispatch_calls(struct batch_options *opt,\n \t\tstruct strbuf *output,\n \t\tstruct expand_data *data,\n@@ -579,8 +693,12 @@ static void dispatch_calls(struct batch_options *opt,\n \tif (!opt->buffer_output)\n \t\tdie(_(\"flush is only for --buffer mode\"));\n \n-\tfor (i = 0; i < nr; i++)\n-\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\tfor (i = 0; i < nr; i++) {\n+\t\tif (!strcmp(cmd[i].name, \"remote-object-info\"))\n+\t\t\tparse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]);\n+\t\telse\n+\t\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t}\n \n \tfflush(stdout);\n }\n@@ -640,11 +758,17 @@ static void batch_objects_command(struct batch_options *opt,\n \t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n \t\t\tfree_cmds(queued_cmd, &nr);\n \t\t} else if (!opt->buffer_output) {\n-\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tif (!strcmp(cmd->name, \"remote-object-info\")) {\n+\t\t\t\tchar *line = xstrdup_or_null(p);\n+\t\t\t\tparse_remote_info(opt, line, output, data, cmd, NULL);\n+\t\t\t} else {\n+\t\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\t}\n \t\t} else {\n \t\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n \t\t\tcall.fn = cmd->fn;\n \t\t\tcall.line = xstrdup_or_null(p);\n+\t\t\tcall.name = cmd->name;\n \t\t\tqueued_cmd[nr++] = call;\n \t\t}\n \t}\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 1b85207694..69ee3f8330 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -2,6 +2,9 @@\n \n test_description='git cat-file'\n \n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n . ./test-lib.sh\n \n test_cmdmode_usage () {\n@@ -1087,4 +1090,348 @@ test_expect_success 'batch-command flush without --buffer' '\n \tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n '\n \n+# This section tests --batch-command with remote-object-info command\n+# If a filter is not set, the filter defaults to \"%(objectname) %(objectsize) %(objecttype)\"\n+# Since \"%(objecttype)\" is currently not supported by the command request, object-info,\n+# the filters are set to \"%(objectname) %(objectsize)\".\n+\n+set_transport_variables () {\n+ tree_sha1=$(git -C \"$1\" write-tree)\n+\tcommit_sha1=$(echo_without_newline \"$commit_message\" | git -C \"$1\" commit-tree $tree_sha1)\n+\ttag_sha1=$(echo_without_newline \"$tag_content\" | git -C \"$1\" hash-object -t tag --stdin -w)\n+\ttag_size=$(strlen \"$tag_content\")\n+}\n+\n+# Test --batch-command remote-object-info with 'git://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\techo_without_newline \"$hello_content\" > $daemon_parent/hello &&\n+\tgit -C \"$daemon_parent\" update-index --add hello\n+'\n+\n+set_transport_variables \"$daemon_parent\"\n+\n+test_expect_success 'batch-command remote-object-info git://' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $tree_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info git:// multiple sha1 per line' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info http:// default filter' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command --buffer remote-object-info git://' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" --buffer >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1 $tag_sha1\n+\t\tflush\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_git_daemon\n+\n+# Test --batch-command remote-object-info with 'http://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\techo_without_newline \"$hello_content\" > $HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" update-index --add hello\n+'\n+set_transport_variables \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\n+test_expect_success 'batch-command remote-object-info http://' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $tree_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $commit_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info http:// one line' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command --buffer remote-object-info http://' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" --buffer >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $commit_sha1 $tag_sha1\n+\t\tflush\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info http:// default filter' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n+\n+\t\tgit cat-file --batch-command >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on unspported filter option (objectsize:disk)' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command=\"%(objectsize:disk)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"objectsize:disk is currently not supported with remote-object-info\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on unspported filter option (deltabase)' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command=\"%(deltabase)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"deltabase is currently not supported with remote-object-info\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on server with legacy protocol' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git -c protocol.version=0 cat-file --batch-command=\"%(objectname) %(objectsize)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"object-info requires protocol v2\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on server with legacy protocol fallback' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git -c protocol.version=0 cat-file --batch-command 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"object-info requires protocol v2\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on malformed OID' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tmalformed_object_id=\"this_id_is_not_valid\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command=\"%(objectname) %(objectsize)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $malformed_object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"malformed object id '$malformed_object_id'\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on malformed OID fallback' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tmalformed_object_id=\"this_id_is_not_valid\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $malformed_object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"malformed object id '$malformed_object_id'\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on missing OID' '\n+\tgit clone \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" missing_oid_repo &&\n+\ttest_commit -C missing_oid_repo message1 c.txt &&\n+\t(\n+\t\tcd missing_oid_repo &&\n+\t\tobject_id=$(git rev-parse message1:c.txt) &&\n+\t\ttest_must_fail git cat-file --batch-command=\"%(objectname) %(objectsize)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"object-info: not our ref $object_id\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on missing OID fallback' '\n+\t(\n+\t\tcd missing_oid_repo &&\n+\t\tobject_id=$(git rev-parse message1:c.txt) &&\n+\t\ttest_must_fail git cat-file --batch-command 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"fatal: remote error: upload-pack: not our ref $object_id\" err\n+\t)\n+'\n+\n+# Test --batch-command remote-object-info with 'file://' transport\n+\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init server &&\n+\tgit -C server config protocol.version 2 &&\n+\techo_without_newline \"$hello_content\" > server/hello &&\n+\tgit -C server update-index --add hello\n+'\n+\n+set_transport_variables \"server\"\n+\n+test_expect_success 'batch-command remote-object-info file://' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $tree_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $commit_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info file:// multiple sha1 per line' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command --buffer remote-object-info file://' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" --buffer >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $commit_sha1 $tag_sha1\n+\t\tflush\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info file:// default filter' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.36.0.rc2.10170.gb555eefa6f\n\n"},{"id":"454754","messageId":"xmqq8rrjv8x5.fsf@gitster.g","threadId":"57388","inReplyTo":"20220502170904.2770649-3-calvinwan@google.com","subject":"Re: [PATCH v4 2/8] fetch-pack: move fetch default settings","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-02T22:58:30Z","receivedAt":"2022-05-02T22:58:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> When the state machine in do_fetch_pack_v2() is in FETCH_CHECK_LOCAL, we\n> set a few v2-specific defaults. It will be helpful for a future patch to\n> have these defaults set if the initial state is not FETCH_CHECK_LOCAL.\n\n\n> This is a safe change since the initial state is currently always\n> FETCH_CHECK_LOCAL, so we're guaranteed to hit that code whether it's in\n> the state machine or not.\n\nWhat does \"it\" (which supposed to be able to be in the state machine\nand also not to be in the state matchine) in the last sentence refer\nto?\n\n>\n> ---\n\nMissing sign-off.\n\nThe patch looks correct and I agree that this is a benign no-op\nbecause the initial value of state is FETCH_CHECK_LOCAL (i.e. the\nblock of code moved here will execute pretty much as the first\nthing, and the relative order between that block and sorting of ref\nlist should not matter). I just didn't understand the explanation\ngiven by the patch why it is safe.\n\nThanks.\n\n> fetch-pack.c | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n>\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index e06125c90a..45473b4602 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1610,18 +1610,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\treader.me = \"fetch-pack\";\n> \t}\n> \n> +\t/* v2 supports these by default */\n> +\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n> +\tuse_sideband = 2;\n> +\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> +\t\targs->deepen = 1;\n> +\n> \twhile (state != FETCH_DONE) {\n> \t\tswitch (state) {\n> \t\tcase FETCH_CHECK_LOCAL:\n> \t\t\tsort_ref_list(&ref, ref_compare_name);\n> \t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n> \n> -\t\t\t/* v2 supports these by default */\n> -\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n> -\t\t\tuse_sideband = 2;\n> -\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> -\t\t\t\targs->deepen = 1;\n> -\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n"},{"id":"454756","messageId":"xmqqzgjztt7s.fsf@gitster.g","threadId":"57388","inReplyTo":"20220502170904.2770649-4-calvinwan@google.com","subject":"Re: [PATCH v4 3/8] object-store: add function to free object_info contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-02T23:23:03Z","receivedAt":"2022-05-02T23:25:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Introduce free_object_info_contents.\n>\n> ---\n> object-file.c | 16 ++++++++++++++++\n> object-store.h | 3 +++\n> 2 files changed, 19 insertions(+)\n>\n> diff --git a/object-file.c b/object-file.c\n> index 5ffbf3d4fd..34a6e34adf 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -2659,3 +2659,19 @@ int read_loose_object(const char *path,\n> \t\tmunmap(map, mapsize);\n> \treturn ret;\n> }\n> +\n> +void free_object_info_contents(struct object_info *object_info)\n> +{\n> +\tif (!object_info)\n> +\t\treturn;\n\nThis is OK, but ...\n\n\n> +\tif (object_info->typep)\n> +\t\tfree(object_info->typep);\n> +\tif (object_info->sizep)\n> +\t\tfree(object_info->sizep);\n> +\tif (object_info->disk_sizep)\n> +\t\tfree(object_info->disk_sizep);\n> +\tif (object_info->delta_base_oid)\n> +\t\tfree(object_info->delta_base_oid);\n> +\tif (object_info->type_name)\n> +\t\tfree(object_info->type_name);\n\n\tif (PTR)\n\t\tfree(PTR);\n\ncan and should be written as\n\n\tfree(PTR);\n\nIf we are reusing object_info after calling this function, we\n_might_ want to use FREE_AND_NULL() instead of free().\n\n> +}\n> \\ No newline at end of file\n\nLet's avoid such incomplete lines.\n\n> diff --git a/object-store.h b/object-store.h\n> index bd2322ed8c..840e04b56f 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -533,4 +533,7 @@ int for_each_object_in_pack(struct packed_git *p,\n> int for_each_packed_object(each_packed_object_fn, void *,\n> \t\t\t enum for_each_object_flags flags);\n> \n> +/* Free pointers inside of object_info, but not object_info itself */\n> +void free_object_info_contents(struct object_info *object_info);\n> +\n> #endif /* OBJECT_STORE_H */\n"},{"id":"454758","messageId":"xmqqee1btr8b.fsf@gitster.g","threadId":"57388","inReplyTo":"20220502170904.2770649-5-calvinwan@google.com","subject":"Re: [PATCH v4 4/8] object-info: send attribute packet regardless of object ids","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-03T00:05:56Z","receivedAt":"2022-05-03T00:06:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Currently on the server side of object-info, if the client does not send\n> any object ids in the request or if the client requests an attribute the\n> server does not support, the server will either not send any packets\n> back or error out.\n\nThere is an early return when oid_str_list->nr (i.e. number of\nobjects we are queried) is zero, and we return without showing the\n\"size\" token in the output (called \"attribute\"). The change in this\npatch changes the behaviour and makes such a request responded with\n\"size\" but without any size information for no objects.\n\nIt is unclear why that is a good change, though.\n\nThe loop that accepts requests in today's code sends an error when\nit sees a line that it does not understand. The patch stops doing\nso. Also, after ignoring such an unknown line, the requests that\nwere understood are responded to by send_info() function. The patch\ninstead refuses to give any output in such a case.\n\nIt is unclear why either of these two makes it a good change,\nthough.\n\n> Consider the scenario where the client git version is\n> ahead of the server git version, and the client can request additional\n> object-info besides 'size'. There needs to be a way to tell whether the\n> server can honor all of the client attribute requests before the client\n> sends a request with all of the object ids.\n\nYes. If we want to learn about \"size\", \"frotz\", and \"nitfol\"\nattributes about these objects, we can send \"size\", \"frotz\",\n\"nitfol\", and then start feeding object names. Then we can observe\nthat \"frotz\" were not liked to learn that this old server does not\nunderstand it, and the same for \"nitfol\". At least we would have\nlearned about size of these objects, without this patch.\n\nNow, an updated server side with this patch would respond with\n\"size\" and no other response. Does it mean it does not understand\n\"frotz\", it does not understand \"nitfol\", or neither? Did we get no\nresponse because we didn't feed objects, or because we asked for\nsomething they don't know about?\n\nHow well does the current client work with today's server side and\nthe server side updated with this patch? I _think_ this change is\nessentially a no-op when there is no version skew (i.e. we do not\nask for anything today's server does not know about and we do not\nnecessarily ask today's server about zero objects).\n\nAm I correct to assume that those who are talking with today's\nserver side are all out-of-tree custom clients?\n\nI am wondering how much damage we are causing them with this change\nin behaviour, especially with the lack of \"You asked for X, but I\ndon't understand X\", which is replaced by no output, which would be\ntotally unexpected by them.\n\nIt totally is possible that this \"object-info\" request is treated as\nan experimental curiosity that is not ready for production and has\nno existing users. If this were in 2006, I would just _declare_\nsuch is a case and move on, breaking the protocol for existing users\nwhose number is zero. But I am afraid that we no longer live in\nsuch a simpler world, so...\n\n> In a future patch, if the\n> client were to make an initial command request with only attributes, the\n> server would be able to confirm which attributes it could return.\n>\n> ---\n> protocol-caps.c | 14 +++++++-------\n> 1 file changed, 7 insertions(+), 7 deletions(-)\n>\n> diff --git a/protocol-caps.c b/protocol-caps.c\n> index bbde91810a..bc7def0727 100644\n> --- a/protocol-caps.c\n> +++ b/protocol-caps.c\n> @@ -11,6 +11,7 @@\n> \n> struct requested_info {\n> \tunsigned size : 1;\n> +\tunsigned unknown : 1;\n> };\n\nOK.\n\n> /*\n> @@ -40,12 +41,12 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n> \tstruct string_list_item *item;\n> \tstruct strbuf send_buffer = STRBUF_INIT;\n> \n> -\tif (!oid_str_list->nr)\n> -\t\treturn;\n> -\n> \tif (info->size)\n> \t\tpacket_writer_write(writer, \"size\");\n> \n> +\tif (info->unknown || !oid_str_list->nr)\n> +\t\tgoto release;\n> +\n> \tfor_each_string_list_item (item, oid_str_list) {\n> \t\tconst char *oid_str = item->string;\n> \t\tstruct object_id oid;\n> @@ -72,12 +73,13 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n> \t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n> \t\tstrbuf_reset(&send_buffer);\n> \t}\n> +release:\n> \tstrbuf_release(&send_buffer);\n> }\n\nOK, except for the bypass for info->unknown case, which I am not\nsure about.\n\n> int cap_object_info(struct repository *r, struct packet_reader *request)\n> {\n> -\tstruct requested_info info;\n> +\tstruct requested_info info = { 0 };\n\nWow. We have been using info uninitialized? Is this a silent\nbugfix?\n\nIf info.size bit was on due to on-stack garbage, we would have given\nour response with \"size\" attribute prefixed, even when the client\nside never requested it.\n\n> @@ -92,9 +94,7 @@ int cap_object_info(struct repository *r, struct packet_reader *request)\n> \t\tif (parse_oid(request->line, &oid_str_list))\n> \t\t\tcontinue;\n> \n> -\t\tpacket_writer_error(&writer,\n> -\t\t\t\t \"object-info: unexpected line: '%s'\",\n> -\t\t\t\t request->line);\n> +\t\tinfo.unknown = 1;\n\n"},{"id":"454761","messageId":"xmqqilqnsaep.fsf@gitster.g","threadId":"57388","inReplyTo":"20220502170904.2770649-6-calvinwan@google.com","subject":"Re: [PATCH v4 5/8] transport: add client side capability to request object-info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-03T00:54:38Z","receivedAt":"2022-05-03T01:16:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> The question of version mismatch often comes up with client/server\n> relationships. There are two cases to consider here (assuming either\n> client or server functionality for object-info changes between the\n> different versions):\n>\n> 1. client version > server version\n> \t- client can request additional attributes from server\n> 2. server version > client version\n> \t- server can return additional info to the client\n>\n> The second case is a non-issue since the client would never be able to\n> request additional info from the server.\n\nThat assumes that we can never remove attributes once we start\nsupporting them, so it is not a non-issue exactly. The way we make\nit a non-issue is to make sure that the server returns no more than\nwhat the client has asked, i.e. the same way we deal with the first\ncase.\n\n> In order to solve the first\n> case, recall an earlier patch where the server sends back the attributes\n> even if no object ids are sent by the client. This allows the client to\n> first check whether the server can accept the requested attributes\n> before sending the entire request.\n\nIt requires two round-trips when the server does not know something\nthe client has asked, one without any object names, and then a real\nrequest with the list of object names?\n\nI think in practice, because the way the previous step is\nstructured, you can send a full request and in the non-error\ncodepath the server side will understand everything and give full\nresponse back without an extra round-trip. Only in the error case,\nthe client will see the set of acceptable attributes and no object\ndata (and implicitly treat that as an indication of an error), but\nthen what would the client do after that happens?\n\n - The client will just give up; the attributes it wanted but the\n server couldn't supply are so fundamentally necessary to perform\n whatever the client wanted to do.\n\n - The client will retry the request by asking a known subset of\n attributes for the same set of objects. This requires an extra\n round-trip. If an earlier step didn't make .unknown case bypass\n the response loop, this extra round-trip would have been totally\n unnecessary.\n\n> index 8c7752fc82..c27018d48c 100644\n> --- a/fetch-pack.h\n> +++ b/fetch-pack.h\n> @@ -69,6 +69,12 @@ struct fetch_pack_args {\n> \tunsigned connectivity_checked:1;\n> };\n> \n> +struct object_info_args {\n> +\tconst struct string_list *object_info_options;\n> +\tconst struct string_list *server_options;\n> +\tconst struct oid_array *oids;\n> +};\n\nThese are const pointers because these lists are not owned by this\nstructure and are borrowed from somebody else?\n\n> diff --git a/transport-helper.c b/transport-helper.c\n> index b4dbbabb0c..093946f7fd 100644\n> --- a/transport-helper.c\n> +++ b/transport-helper.c\n> @@ -686,13 +686,16 @@ static int fetch_refs(struct transport *transport,\n> \n> \t/*\n> \t * If we reach here, then the server, the client, and/or the transport\n> -\t * helper does not support protocol v2. --negotiate-only requires\n> -\t * protocol v2.\n> +\t * helper does not support protocol v2. --negotiate-only and --object-info\n> +\t * require protocol v2.\n> \t */\n\nOK.\n\n> \tif (data->transport_options.acked_commits) {\n> \t\twarning(_(\"--negotiate-only requires protocol v2\"));\n> \t\treturn -1;\n> \t}\n> +\tif (transport->smart_options->object_info) {\n> +\t\tdie(_(\"--object-info requires protocol v2\"));\n> +\t}\n\nOK.\n\n> diff --git a/transport.c b/transport.c\n> index 3d64a43ab3..08c505e1d0 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -353,6 +353,79 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> \treturn refs;\n> }\n> \n> +static int fetch_object_info(struct transport *transport, struct object_info **object_info_data)\n> +{\n> +\tsize_t i;\n> +\tint size_index = -1;\n> +\tstruct git_transport_data *data = transport->data;\n> +\tstruct object_info_args args;\n> +\tstruct packet_reader reader;\n> +\tstruct string_list server_attributes = STRING_LIST_INIT_DUP;\n> +\n> +\tmemset(&args, 0, sizeof(args));\n> +\targs.server_options = transport->server_options;\n> +\targs.object_info_options = transport->smart_options->object_info_options;\n\nIt is unclear who fills smart_options->object_info_options and from\nwhat. Are we referring to something that is not yet used, or worse\nyet, code that is not yet written?\n\nIt seems that the result of applying 1-5/8 does not compile.\n\n> +\tconnect_setup(transport, 0);\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\tPACKET_READ_CHOMP_NEWLINE |\n> +\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n> +\tdata->version = discover_version(&reader);\n> +\n> +\ttransport->hash_algo = reader.hash_algo;\n> +\n> +\tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tif (!server_supports_v2(\"object-info\", 0))\n> +\t\t\treturn -1;\n> +\t\t/**\n> +\t\t * Send a request with only attributes first. If server can return all\n> +\t\t * of the requested attributes, then send request with object ids\n> +\t\t */\n\nDoesn't it force the most pessimistic behaviour, i.e. always do a\nlikely-to-be-useless (when the feature becomes widely avaiable)\nprobe round-trip before making a real request?\n\n> +\t\tsend_object_info_request(data->fd[1], &args);\n> +\t\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> +\t\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> +\t\t\treturn -1;\n> +\t\t}\n\nAll these check_stateless_delimiter() calls in this patch are on\noverly long lines.\n\n> +\t\tstring_list_split(&server_attributes, reader.line, ' ', -1);\n> +\t\tpacket_reader_read(&reader);\n> +\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> +\t\tif (server_attributes.nr != args.object_info_options->nr)\n> +\t\t\treturn -1;\n\nSo, I am guessing (without having the code that prepares the right\nhand side) that we are comparing the number of attributes we asked\n(on the right hand side) and the number of attributes the other side\nsaid it supports in response to our request (on the left hand\nside). \n\nThis does not protect against a server that responds with attribute\nnames duplicated or any somewhat odd third-party servers that are\nnot identical to what you wrote here. I would rather see us to loop\nthrough object_info_options array and see if server_attributes list\nsays all of them are supported more explicitly. That can be done in\nthe loop below, where you'd figure out the \"foo_index\" for each\nattribute \"foo\" you are supporting in the code.\n\n> +\t\tfor (i = 0; i < server_attributes.nr; i++) {\n> +\t\t\tif (!strcmp(server_attributes.items[i].string, \"size\"))\n> +\t\t\t\tsize_index = i + 1;\n> +\t\t}\n\nsize_index was initialized to -1 and I was expecting we can tell the\nattribute is not used by checking (size_index < 0), but this seems\nto make size_index 1 based, not 0 based. Intended?\n\nI think this is to skip the object name that comes as the first item\nin the response, but it would be more \"pure\" to keep foo_index\n0-based and add the offset by whatever constant number of things\nthat come in front (currently, 1 for object name) near a lot closer\nto where we parse and read the data, i.e. in the while() loop below.\n\n> +\t\targs.oids = transport->smart_options->object_info_oids;\n> +\t\tsend_object_info_request(data->fd[1], &args);\n\nAnd this is the second round-trip.\n\n> +\t\tbreak;\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tdie(_(\"wrong protocol version. expected v2\"));\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> +\n> +\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL)\n> +\t\tdie(_(\"error reading object info header\"));\n> +\ti = 0;\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> +\t\tstruct string_list object_info_values = STRING_LIST_INIT_DUP;\n> +\n> +\t\tstring_list_split(&object_info_values, reader.line, ' ', -1);\n> +\t\tif (size_index > 0) {\n> +\t\t\tif (!strcmp(object_info_values.items[size_index].string, \"\"))\n> +\t\t\t\tdie(\"object-info: not our ref %s\",\n> +\t\t\t\t\tobject_info_values.items[0].string);\n> +\t\t\t*(*object_info_data)[i].sizep = strtoul(object_info_values.items[size_index].string, NULL, 10);\n> +\t\t}\n> +\t\ti++;\n> +\t}\n> +\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> +\n> +\treturn 0;\n> +}\n\nAll of the above look written for the best case scenario (i.e. it\nassumes there won't be any unexpected type of response), which makes\na good fuzz target. A malicious server side can throw non-number in\nthe \"size\" slot and strtoul() does not check for a malformatted\nnumber, or it can return more response records than it was\nrequested, and overflow object_info_data[] array the caller of this\nfunction prepared.\n\nsmart_options->object_info_oids may know how many response records\nwe should expect, so at least this while() loop should be able to\nprotect against an object_info_data[] overflow attack using it as\nthe upper limit of i.\n\n> static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n> \t\t\t\t\tstruct transport_ls_refs_options *options)\n> {\n> @@ -392,8 +465,14 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \targs.server_options = transport->server_options;\n> \targs.negotiation_tips = data->options.negotiation_tips;\n> \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n> +\targs.object_info = transport->smart_options->object_info;\n> \n> -\tif (!data->got_remote_heads) {\n> +\tif (transport->smart_options && transport->smart_options->object_info) {\n> +\t\tif (!fetch_object_info(transport, data->options.object_info_data))\n> +\t\t\tgoto cleanup;\n> +\t\tret = -1;\n> +\t\tgoto cleanup;\n> +\t} else if (!data->got_remote_heads) {\n> \t\tint i;\n> \t\tint must_list_refs = 0;\n> \t\tfor (i = 0; i < nr_heads; i++) {\n> diff --git a/transport.h b/transport.h\n> index 12bc08fc33..dbf60bb710 100644\n> --- a/transport.h\n> +++ b/transport.h\n> @@ -6,6 +6,7 @@\n> #include \"remote.h\"\n> #include \"list-objects-filter-options.h\"\n> #include \"string-list.h\"\n> +#include \"object-store.h\"\n> \n> struct git_transport_options {\n> \tunsigned thin : 1;\n> @@ -31,6 +32,12 @@ struct git_transport_options {\n> \t */\n> \tunsigned connectivity_checked:1;\n> \n> +\t/*\n> +\t * Transport will attempt to pull only object-info. Fallbacks\n> +\t * to pulling entire object if object-info is not supported\n> +\t */\n> +\tunsigned object_info : 1;\n> +\n> \tint depth;\n> \tconst char *deepen_since;\n> \tconst struct string_list *deepen_not;\n> @@ -54,6 +61,10 @@ struct git_transport_options {\n> \t * common commits to this oidset instead of fetching any packfiles.\n> \t */\n> \tstruct oidset *acked_commits;\n> +\n> +\tstruct oid_array *object_info_oids;\n> +\tstruct object_info **object_info_data;\n> +\tstruct string_list *object_info_options;\n> };\n> \n> enum transport_family {\n"},{"id":"454798","messageId":"CAFySSZBuxMdO-TNUeXcHijmG8hmrSs6LDvJ=O4ZePiyZAcJaeA@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqilqnsaep.fsf@gitster.g","subject":"Re: [PATCH v4 5/8] transport: add client side capability to request object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-03T18:58:46Z","receivedAt":"2022-05-03T18:59:06Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> It requires two round-trips when the server does not know something\n> the client has asked, one without any object names, and then a real\n> request with the list of object names?\n\n> I think in practice, because the way the previous step is\n> structured, you can send a full request and in the non-error\n> codepath the server side will understand everything and give full\n> response back without an extra round-trip. Only in the error case,\n> the client will see the set of acceptable attributes and no object\n> data (and implicitly treat that as an indication of an error), but\n> then what would the client do after that happens?\n\n> - The client will just give up; the attributes it wanted but the\n> server couldn't supply are so fundamentally necessary to perform\n> whatever the client wanted to do.\n\n> - The client will retry the request by asking a known subset of\n> attributes for the same set of objects. This requires an extra\n> round-trip. If an earlier step didn't make .unknown case bypass\n> the response loop, this extra round-trip would have been totally\n> unnecessary.\n\nI do agree that the two round trips are unnecessary and I can send the\nfull request to begin with. The client (after patch 6) will fallback to\nfetch after this initial request fails.\n\n> These are const pointers because these lists are not owned by this\n> structure and are borrowed from somebody else?\n\nYes git_transport_options holds them.\n\n> It is unclear who fills smart_options->object_info_options and from\n> what. Are we referring to something that is not yet used, or worse\n> yet, code that is not yet written?\n\nThe code that calls this function is in patch 8. I decided to separate\nout the transport side and the new command in cat-file --batch-command\nto make it easier to explain to a reviewer how the individual parts\nwork. Looking holistically at your comments, should I have gone about\nsplitting the patches differently? It seems like I was a little too\naggressive this time, creating confusion by requiring the reviewer to\nlook at the next patch to answer some questions.\n\n> It seems that the result of applying 1-5/8 does not compile.\n\nI'll check why this is the case. I should get in the habit of compiling\nall of my patches individually.\n\n> size_index was initialized to -1 and I was expecting we can tell the\n> attribute is not used by checking (size_index < 0), but this seems\n> to make size_index 1 based, not 0 based. Intended?\n\nYes the server returns the packet as \"object_id SP size\" so the first\nvalue is always the object_id.\n\n> All of the above look written for the best case scenario (i.e. it\n> assumes there won't be any unexpected type of response), which makes\n> a good fuzz target.\n\nI didn't take into account the possibility of a malicious server in my\nimplementation. Will go back through and protect against it.\n\n\nOn Mon, May 2, 2022 at 5:54 PM Junio C Hamano wrote:\n>\n> Calvin Wan writes:\n>\n> > The question of version mismatch often comes up with client/server\n> > relationships. There are two cases to consider here (assuming either\n> > client or server functionality for object-info changes between the\n> > different versions):\n> >\n> > 1. client version > server version\n> > - client can request additional attributes from server\n> > 2. server version > client version\n> > - server can return additional info to the client\n> >\n> > The second case is a non-issue since the client would never be able to\n> > request additional info from the server.\n>\n> That assumes that we can never remove attributes once we start\n> supporting them, so it is not a non-issue exactly. The way we make\n> it a non-issue is to make sure that the server returns no more than\n> what the client has asked, i.e. the same way we deal with the first\n> case.\n>\n> > In order to solve the first\n> > case, recall an earlier patch where the server sends back the attributes\n> > even if no object ids are sent by the client. This allows the client to\n> > first check whether the server can accept the requested attributes\n> > before sending the entire request.\n>\n> It requires two round-trips when the server does not know something\n> the client has asked, one without any object names, and then a real\n> request with the list of object names?\n>\n> I think in practice, because the way the previous step is\n> structured, you can send a full request and in the non-error\n> codepath the server side will understand everything and give full\n> response back without an extra round-trip. Only in the error case,\n> the client will see the set of acceptable attributes and no object\n> data (and implicitly treat that as an indication of an error), but\n> then what would the client do after that happens?\n>\n> - The client will just give up; the attributes it wanted but the\n> server couldn't supply are so fundamentally necessary to perform\n> whatever the client wanted to do.\n>\n> - The client will retry the request by asking a known subset of\n> attributes for the same set of objects. This requires an extra\n> round-trip. If an earlier step didn't make .unknown case bypass\n> the response loop, this extra round-trip would have been totally\n> unnecessary.\n>\n> > index 8c7752fc82..c27018d48c 100644\n> > --- a/fetch-pack.h\n> > +++ b/fetch-pack.h\n> > @@ -69,6 +69,12 @@ struct fetch_pack_args {\n> > unsigned connectivity_checked:1;\n> > };\n> >\n> > +struct object_info_args {\n> > + const struct string_list *object_info_options;\n> > + const struct string_list *server_options;\n> > + const struct oid_array *oids;\n> > +};\n>\n> These are const pointers because these lists are not owned by this\n> structure and are borrowed from somebody else?\n>\n> > diff --git a/transport-helper.c b/transport-helper.c\n> > index b4dbbabb0c..093946f7fd 100644\n> > --- a/transport-helper.c\n> > +++ b/transport-helper.c\n> > @@ -686,13 +686,16 @@ static int fetch_refs(struct transport *transport,\n> >\n> > /*\n> > * If we reach here, then the server, the client, and/or the transport\n> > - * helper does not support protocol v2. --negotiate-only requires\n> > - * protocol v2.\n> > + * helper does not support protocol v2. --negotiate-only and --object-info\n> > + * require protocol v2.\n> > */\n>\n> OK.\n>\n> > if (data->transport_options.acked_commits) {\n> > warning(_(\"--negotiate-only requires protocol v2\"));\n> > return -1;\n> > }\n> > + if (transport->smart_options->object_info) {\n> > + die(_(\"--object-info requires protocol v2\"));\n> > + }\n>\n> OK.\n>\n> > diff --git a/transport.c b/transport.c\n> > index 3d64a43ab3..08c505e1d0 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -353,6 +353,79 @@ static struct ref *handshake(struct transport *transport, int for_push,\n> > return refs;\n> > }\n> >\n> > +static int fetch_object_info(struct transport *transport, struct object_info **object_info_data)\n> > +{\n> > + size_t i;\n> > + int size_index = -1;\n> > + struct git_transport_data *data = transport->data;\n> > + struct object_info_args args;\n> > + struct packet_reader reader;\n> > + struct string_list server_attributes = STRING_LIST_INIT_DUP;\n> > +\n> > + memset(&args, 0, sizeof(args));\n> > + args.server_options = transport->server_options;\n> > + args.object_info_options = transport->smart_options->object_info_options;\n>\n> It is unclear who fills smart_options->object_info_options and from\n> what. Are we referring to something that is not yet used, or worse\n> yet, code that is not yet written?\n>\n> It seems that the result of applying 1-5/8 does not compile.\n>\n> > + connect_setup(transport, 0);\n> > + packet_reader_init(&reader, data->fd[0], NULL, 0,\n> > + PACKET_READ_CHOMP_NEWLINE |\n> > + PACKET_READ_DIE_ON_ERR_PACKET);\n> > + data->version = discover_version(&reader);\n> > +\n> > + transport->hash_algo = reader.hash_algo;\n> > +\n> > + switch (data->version) {\n> > + case protocol_v2:\n> > + if (!server_supports_v2(\"object-info\", 0))\n> > + return -1;\n> > + /**\n> > + * Send a request with only attributes first. If server can return all\n> > + * of the requested attributes, then send request with object ids\n> > + */\n>\n> Doesn't it force the most pessimistic behaviour, i.e. always do a\n> likely-to-be-useless (when the feature becomes widely avaiable)\n> probe round-trip before making a real request?\n>\n> > + send_object_info_request(data->fd[1], &args);\n> > + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> > + check_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> > + return -1;\n> > + }\n>\n> All these check_stateless_delimiter() calls in this patch are on\n> overly long lines.\n>\n> > + string_list_split(&server_attributes, reader.line, ' ', -1);\n> > + packet_reader_read(&reader);\n> > + check_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> > + if (server_attributes.nr != args.object_info_options->nr)\n> > + return -1;\n>\n> So, I am guessing (without having the code that prepares the right\n> hand side) that we are comparing the number of attributes we asked\n> (on the right hand side) and the number of attributes the other side\n> said it supports in response to our request (on the left hand\n> side).\n>\n> This does not protect against a server that responds with attribute\n> names duplicated or any somewhat odd third-party servers that are\n> not identical to what you wrote here. I would rather see us to loop\n> through object_info_options array and see if server_attributes list\n> says all of them are supported more explicitly. That can be done in\n> the loop below, where you'd figure out the \"foo_index\" for each\n> attribute \"foo\" you are supporting in the code.\n>\n> > + for (i = 0; i < server_attributes.nr; i++) {\n> > + if (!strcmp(server_attributes.items[i].string, \"size\"))\n> > + size_index = i + 1;\n> > + }\n>\n> size_index was initialized to -1 and I was expecting we can tell the\n> attribute is not used by checking (size_index < 0), but this seems\n> to make size_index 1 based, not 0 based. Intended?\n>\n> I think this is to skip the object name that comes as the first item\n> in the response, but it would be more \"pure\" to keep foo_index\n> 0-based and add the offset by whatever constant number of things\n> that come in front (currently, 1 for object name) near a lot closer\n> to where we parse and read the data, i.e. in the while() loop below.\n>\n> > + args.oids = transport->smart_options->object_info_oids;\n> > + send_object_info_request(data->fd[1], &args);\n>\n> And this is the second round-trip.\n>\n> > + break;\n> > + case protocol_v1:\n> > + case protocol_v0:\n> > + die(_(\"wrong protocol version. expected v2\"));\n> > + case protocol_unknown_version:\n> > + BUG(\"unknown protocol version\");\n> > + }\n> > +\n> > + if (packet_reader_read(&reader) != PACKET_READ_NORMAL)\n> > + die(_(\"error reading object info header\"));\n> > + i = 0;\n> > + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> > + struct string_list object_info_values = STRING_LIST_INIT_DUP;\n> > +\n> > + string_list_split(&object_info_values, reader.line, ' ', -1);\n> > + if (size_index > 0) {\n> > + if (!strcmp(object_info_values.items[size_index].string, \"\"))\n> > + die(\"object-info: not our ref %s\",\n> > + object_info_values.items[0].string);\n> > + *(*object_info_data)[i].sizep = strtoul(object_info_values.items[size_index].string, NULL, 10);\n> > + }\n> > + i++;\n> > + }\n> > + check_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n> > +\n> > + return 0;\n> > +}\n>\n> All of the above look written for the best case scenario (i.e. it\n> assumes there won't be any unexpected type of response), which makes\n> a good fuzz target. A malicious server side can throw non-number in\n> the \"size\" slot and strtoul() does not check for a malformatted\n> number, or it can return more response records than it was\n> requested, and overflow object_info_data[] array the caller of this\n> function prepared.\n>\n> smart_options->object_info_oids may know how many response records\n> we should expect, so at least this while() loop should be able to\n> protect against an object_info_data[] overflow attack using it as\n> the upper limit of i.\n>\n> > static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n> > struct transport_ls_refs_options *options)\n> > {\n> > @@ -392,8 +465,14 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > args.server_options = transport->server_options;\n> > args.negotiation_tips = data->options.negotiation_tips;\n> > args.reject_shallow_remote = transport->smart_options->reject_shallow;\n> > + args.object_info = transport->smart_options->object_info;\n> >\n> > - if (!data->got_remote_heads) {\n> > + if (transport->smart_options && transport->smart_options->object_info) {\n> > + if (!fetch_object_info(transport, data->options.object_info_data))\n> > + goto cleanup;\n> > + ret = -1;\n> > + goto cleanup;\n> > + } else if (!data->got_remote_heads) {\n> > int i;\n> > int must_list_refs = 0;\n> > for (i = 0; i < nr_heads; i++) {\n> > diff --git a/transport.h b/transport.h\n> > index 12bc08fc33..dbf60bb710 100644\n> > --- a/transport.h\n> > +++ b/transport.h\n> > @@ -6,6 +6,7 @@\n> > #include \"remote.h\"\n> > #include \"list-objects-filter-options.h\"\n> > #include \"string-list.h\"\n> > +#include \"object-store.h\"\n> >\n> > struct git_transport_options {\n> > unsigned thin : 1;\n> > @@ -31,6 +32,12 @@ struct git_transport_options {\n> > */\n> > unsigned connectivity_checked:1;\n> >\n> > + /*\n> > + * Transport will attempt to pull only object-info. Fallbacks\n> > + * to pulling entire object if object-info is not supported\n> > + */\n> > + unsigned object_info : 1;\n> > +\n> > int depth;\n> > const char *deepen_since;\n> > const struct string_list *deepen_not;\n> > @@ -54,6 +61,10 @@ struct git_transport_options {\n> > * common commits to this oidset instead of fetching any packfiles.\n> > */\n> > struct oidset *acked_commits;\n> > +\n> > + struct oid_array *object_info_oids;\n> > + struct object_info **object_info_data;\n> > + struct string_list *object_info_options;\n> > };\n> >\n> > enum transport_family {\n"},{"id":"454799","messageId":"CAFySSZDdhD19Czh31HB2V+FNWYB-Xon68_qZNSmeSNzA8Od=PQ@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqee1btr8b.fsf@gitster.g","subject":"Re: [PATCH v4 4/8] object-info: send attribute packet regardless of object ids","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-03T19:21:01Z","receivedAt":"2022-05-03T19:21:17Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> Now, an updated server side with this patch would respond with\n> \"size\" and no other response. Does it mean it does not understand\n> \"frotz\", it does not understand \"nitfol\", or neither? Did we get no\n> response because we didn't feed objects, or because we asked for\n> something they don't know about?\n\nPatches 4-6 describe the following design idea I had:\n - Send initial request with only attributes\n - If server can fulfill object-info attributes, then go ahead\n - Else fallback to fetch\n\nI responded to your comments on patch 5 regarding not needing\nthat first request, and sending the entire request from the beginning\nso I imagine v5 would look something like this instead:\n - Send full request\n - Server either responds with a fulfilled request or only the attributes\n it can return.\n - Fallback to fetch if request is unfulfilled\n\n> How well does the current client work with today's server side and\n> the server side updated with this patch? I _think_ this change is\n> essentially a no-op when there is no version skew\n\nYou are correct that this change is essentially a no-op when there is\nno version skew, which is intended. There is no current client; it's\nimplemented in patch 5.\n\n> Am I correct to assume that those who are talking with today's\n> server side are all out-of-tree custom clients?\n\n> I am wondering how much damage we are causing them with this change\n> in behaviour, especially with the lack of \"You asked for X, but I\n> don't understand X\", which is replaced by no output, which would be\n> totally unexpected by them.\n\nInternally we don't have anyone using this yet. If we want to be\ntotally safe, I am also OK with having object-info return everything\nit understands about the request rather than nothing. This was\nintended to be an optimization since we would be defaulting back\nto fetch to get object-info rather than only returning what object-info\nknows.\n\n> Wow. We have been using info uninitialized? Is this a silent\n> bugfix?\n\nYes unfortunately.\nOn Mon, May 2, 2022 at 5:06 PM Junio C Hamano wrote:\n>\n> Calvin Wan writes:\n>\n> > Currently on the server side of object-info, if the client does not send\n> > any object ids in the request or if the client requests an attribute the\n> > server does not support, the server will either not send any packets\n> > back or error out.\n>\n> There is an early return when oid_str_list->nr (i.e. number of\n> objects we are queried) is zero, and we return without showing the\n> \"size\" token in the output (called \"attribute\"). The change in this\n> patch changes the behaviour and makes such a request responded with\n> \"size\" but without any size information for no objects.\n>\n> It is unclear why that is a good change, though.\n>\n> The loop that accepts requests in today's code sends an error when\n> it sees a line that it does not understand. The patch stops doing\n> so. Also, after ignoring such an unknown line, the requests that\n> were understood are responded to by send_info() function. The patch\n> instead refuses to give any output in such a case.\n>\n> It is unclear why either of these two makes it a good change,\n> though.\n>\n> > Consider the scenario where the client git version is\n> > ahead of the server git version, and the client can request additional\n> > object-info besides 'size'. There needs to be a way to tell whether the\n> > server can honor all of the client attribute requests before the client\n> > sends a request with all of the object ids.\n>\n> Yes. If we want to learn about \"size\", \"frotz\", and \"nitfol\"\n> attributes about these objects, we can send \"size\", \"frotz\",\n> \"nitfol\", and then start feeding object names. Then we can observe\n> that \"frotz\" were not liked to learn that this old server does not\n> understand it, and the same for \"nitfol\". At least we would have\n> learned about size of these objects, without this patch.\n>\n> Now, an updated server side with this patch would respond with\n> \"size\" and no other response. Does it mean it does not understand\n> \"frotz\", it does not understand \"nitfol\", or neither? Did we get no\n> response because we didn't feed objects, or because we asked for\n> something they don't know about?\n>\n> How well does the current client work with today's server side and\n> the server side updated with this patch? I _think_ this change is\n> essentially a no-op when there is no version skew (i.e. we do not\n> ask for anything today's server does not know about and we do not\n> necessarily ask today's server about zero objects).\n>\n> Am I correct to assume that those who are talking with today's\n> server side are all out-of-tree custom clients?\n>\n> I am wondering how much damage we are causing them with this change\n> in behaviour, especially with the lack of \"You asked for X, but I\n> don't understand X\", which is replaced by no output, which would be\n> totally unexpected by them.\n>\n> It totally is possible that this \"object-info\" request is treated as\n> an experimental curiosity that is not ready for production and has\n> no existing users. If this were in 2006, I would just _declare_\n> such is a case and move on, breaking the protocol for existing users\n> whose number is zero. But I am afraid that we no longer live in\n> such a simpler world, so...\n>\n> > In a future patch, if the\n> > client were to make an initial command request with only attributes, the\n> > server would be able to confirm which attributes it could return.\n> >\n> > ---\n> > protocol-caps.c | 14 +++++++-------\n> > 1 file changed, 7 insertions(+), 7 deletions(-)\n> >\n> > diff --git a/protocol-caps.c b/protocol-caps.c\n> > index bbde91810a..bc7def0727 100644\n> > --- a/protocol-caps.c\n> > +++ b/protocol-caps.c\n> > @@ -11,6 +11,7 @@\n> >\n> > struct requested_info {\n> > unsigned size : 1;\n> > + unsigned unknown : 1;\n> > };\n>\n> OK.\n>\n> > /*\n> > @@ -40,12 +41,12 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n> > struct string_list_item *item;\n> > struct strbuf send_buffer = STRBUF_INIT;\n> >\n> > - if (!oid_str_list->nr)\n> > - return;\n> > -\n> > if (info->size)\n> > packet_writer_write(writer, \"size\");\n> >\n> > + if (info->unknown || !oid_str_list->nr)\n> > + goto release;\n> > +\n> > for_each_string_list_item (item, oid_str_list) {\n> > const char *oid_str = item->string;\n> > struct object_id oid;\n> > @@ -72,12 +73,13 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n> > packet_writer_write(writer, \"%s\", send_buffer.buf);\n> > strbuf_reset(&send_buffer);\n> > }\n> > +release:\n> > strbuf_release(&send_buffer);\n> > }\n>\n> OK, except for the bypass for info->unknown case, which I am not\n> sure about.\n>\n> > int cap_object_info(struct repository *r, struct packet_reader *request)\n> > {\n> > - struct requested_info info;\n> > + struct requested_info info = { 0 };\n>\n> Wow. We have been using info uninitialized? Is this a silent\n> bugfix?\n>\n> If info.size bit was on due to on-stack garbage, we would have given\n> our response with \"size\" attribute prefixed, even when the client\n> side never requested it.\n>\n> > @@ -92,9 +94,7 @@ int cap_object_info(struct repository *r, struct packet_reader *request)\n> > if (parse_oid(request->line, &oid_str_list))\n> > continue;\n> >\n> > - packet_writer_error(&writer,\n> > - \"object-info: unexpected line: '%s'\",\n> > - request->line);\n> > + info.unknown = 1;\n>\n"},{"id":"454801","messageId":"20220503230658.4170414-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"xmqq8rrjv8x5.fsf@gitster.g","subject":"Re: [PATCH v4 2/8] fetch-pack: move fetch default settings","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-03T23:06:57Z","receivedAt":"2022-05-03T23:07:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano writes:\n> Calvin Wan writes:\n> \n> > When the state machine in do_fetch_pack_v2() is in FETCH_CHECK_LOCAL, we\n> > set a few v2-specific defaults. It will be helpful for a future patch to\n> > have these defaults set if the initial state is not FETCH_CHECK_LOCAL.\n> \n> \n> > This is a safe change since the initial state is currently always\n> > FETCH_CHECK_LOCAL, so we're guaranteed to hit that code whether it's in\n> > the state machine or not.\n> \n> What does \"it\" (which supposed to be able to be in the state machine\n> and also not to be in the state matchine) in the last sentence refer\n> to?\n\nI think the \"it\" refers to the initialization code that was moved.\n\n> The patch looks correct and I agree that this is a benign no-op\n> because the initial value of state is FETCH_CHECK_LOCAL (i.e. the\n> block of code moved here will execute pretty much as the first\n> thing, and the relative order between that block and sorting of ref\n> list should not matter). I just didn't understand the explanation\n> given by the patch why it is safe.\n> \n> Thanks.\n\nAgreed - I think the commit message would be better off if it was worded\nsomething like:\n\n There are some initialization steps that need to be done at the start\n of the execution of the do_fetch_pack_v2() state machine. Currently,\n these are done in the FETCH_CHECK_LOCAL state, which is the initial\n state.\n\n However, a subsequent patch will allow for another initial state,\n while still requiring these initialization steps. Therefore, move\n these initialization steps to before the state machine, so that they\n are run regardless of the initial state.\n\nI think this description suffices for a reviewer to see that it is safe,\nbut if you want, you can add:\n\n Note that there is no change in behavior, because we're moving code\n from the beginning of the first state to just before the execution of\n the state machine.\n"},{"id":"454802","messageId":"20220503231139.4171524-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-5-calvinwan@google.com","subject":"Re: [PATCH v4 4/8] object-info: send attribute packet regardless of object ids","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-03T23:11:39Z","receivedAt":"2022-05-03T23:11:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Calvin Wan writes:\n> Currently on the server side of object-info, if the client does not send\n> any object ids in the request or if the client requests an attribute the\n> server does not support, the server will either not send any packets\n> back or error out. Consider the scenario where the client git version is\n> ahead of the server git version, and the client can request additional\n> object-info besides 'size'. There needs to be a way to tell whether the\n> server can honor all of the client attribute requests before the client\n> sends a request with all of the object ids. \n\nThis part explains the problem...\n\n> In a future patch, if the\n> client were to make an initial command request with only attributes, the\n> server would be able to confirm which attributes it could return.\n\n...and this part explains what we want to do in the future, but this\ncommit message doesn't explain what this commit does. (The commit\nsubject has something related, but does not have enough detail. For\nexample, is it an empty attribute packet?)\n\nHaving said that, the way we usually communicate client-server version\ndifferences is through capabilities - is there a reason why we can't use\nthat here?\n"},{"id":"454803","messageId":"20220503231558.4172288-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-6-calvinwan@google.com","subject":"Re: [PATCH v4 5/8] transport: add client side capability to request object-info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-03T23:15:58Z","receivedAt":"2022-05-03T23:16:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Calvin Wan writes:\n> Sometimes it is useful to get information about an object without having\n> to download it completely. The server logic has already been implemented\n> as “a2ba162cda (object-info: support for retrieving object info,\n> 2021-04-20)”. This patch implements the client option for it. Currently,\n> only 'size' is supported and the server must be v2, however future\n> patches can implement additional options.\n> \n> The question of version mismatch often comes up with client/server\n> relationships. There are two cases to consider here (assuming either\n> client or server functionality for object-info changes between the\n> different versions):\n> \n> 1. client version > server version\n> \t- client can request additional attributes from server\n> 2. server version > client version\n> \t- server can return additional info to the client\n> \n> The second case is a non-issue since the client would never be able to\n> request additional info from the server. In order to solve the first\n> case, recall an earlier patch where the server sends back the attributes\n> even if no object ids are sent by the client. This allows the client to\n> first check whether the server can accept the requested attributes\n> before sending the entire request.\n\nFrom this description, it seems like the intention is to send an\nobject-info request, and then if the server responds in a certain way\n(here, sending back the attributes - presumably without sending any\nactual information), then we know that the server doesn't support our\nrequest and we need to fall back. As Junio says [1], this requires one\nRTT more than necessary. We could just determine if the server supports\nthe attributes we want by using capabilities (without needing to send\nthe request to check).\n\n[1] https://lore.kernel.org/git/xmqqilqnsaep.fsf@gitster.g/\n"},{"id":"454804","messageId":"20220503232701.4173948-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-7-calvinwan@google.com","subject":"Re: [PATCH v4 6/8] transport: add object-info fallback to fetch","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-03T23:27:01Z","receivedAt":"2022-05-03T23:27:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Calvin Wan writes:\n> If a v2 server does not support object-info or if the client requests\n> information that isn't supported by object-info, fetch the objects as if\n> it were a standard v2 fetch (however without changing any refs).\n\nWhat do you mean by \"however without changing any refs\"? (I would have\nexpected that no refs would be changed, so this, to me, implies that we\nwould expect some refs to be changed.)\n\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 506ca28e05..938d082b80 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1639,6 +1639,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n> \t\targs->deepen = 1;\n> \n> +\tif (args->object_info)\n> +\t\tstate = FETCH_SEND_REQUEST;\n> +\n> \twhile (state != FETCH_DONE) {\n> \t\tswitch (state) {\n> \t\tcase FETCH_CHECK_LOCAL:\n> @@ -1648,7 +1651,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n> -\t\t\tif (!args->refetch && everything_local(args, &ref))\n> +\t\t\tif (!args->refetch && !args->object_info && everything_local(args, &ref))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n\nI think that the purpose of all these changes is to skip certain steps\nwhen we know that we're fetching as a fallback for object-info, but I\ndon't think they're necessary - it's fine to not fetch certain objects\nif we have them present.\n\n> @@ -2035,6 +2038,15 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> \t\targs->connectivity_checked = 1;\n> \t}\n> \n> +\tif (args->object_info) {\n> +\t\tstruct ref *ref_cpy_reader = ref_cpy;\n> +\t\tint i = 0;\n> +\t\twhile (ref_cpy_reader) {\n> +\t\t\toid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args->object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE);\n> +\t\t\tref_cpy_reader = ref_cpy_reader->next;\n> +\t\t\ti++;\n> +\t\t}\n> +\t}\n\nCould this part be done in the same function that falls back\n(fetch_refs_via_pack(), I believe)? That would make the code easier to\nunderstand - here, we don't know why we would need to copy the OIDs, but\nin the function that falls back, we can look up and see that this is\ndone because we couldn't do something else.\n\n> diff --git a/transport.c b/transport.c\n> index 08c505e1d0..b85e52d9a8 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -436,10 +436,12 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \t\t\t int nr_heads, struct ref **to_fetch)\n> {\n> \tint ret = 0;\n> +\tsize_t i;\n> \tstruct git_transport_data *data = transport->data;\n> \tstruct ref *refs = NULL;\n> \tstruct fetch_pack_args args;\n> \tstruct ref *refs_tmp = NULL;\n> +\tstruct ref *wanted_refs = xcalloc(1, sizeof (struct ref));\n\nIf you only need these new variables in a block, declare them there (and\nfree them at the end of that block).\n\n> @@ -489,7 +500,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \telse if (data->version <= protocol_v1)\n> \t\tdie_if_server_options(transport);\n> \n> -\tif (data->options.acked_commits) {\n> +\tif (data->options.acked_commits && !transport->smart_options->object_info) {\n> \t\tif (data->version < protocol_v2) {\n> \t\t\twarning(_(\"--negotiate-only requires protocol v2\"));\n> \t\t\tret = -1;\n\nIs this addition necessary? --negotiate-only and object-info do not work\ntogether.\n"},{"id":"454825","messageId":"xmqqee19nw30.fsf@gitster.g","threadId":"57388","inReplyTo":"CAFySSZBuxMdO-TNUeXcHijmG8hmrSs6LDvJ=O4ZePiyZAcJaeA@mail.gmail.com","subject":"Re: [PATCH v4 5/8] transport: add client side capability to request object-info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-04T15:42:11Z","receivedAt":"2022-05-04T15:42:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n>> It seems that the result of applying 1-5/8 does not compile.\n>\n> I'll check why this is the case. I should get in the habit of compiling\n> all of my patches individually.\n\nFor a 8-patch series, we should make sure that all 8 states resuting\nfrom applying patches 1-thru-N for each N (1 <= N <= 8) builds and\ntests OK for bisectability.\n\nI often do not check that due to lack of time on the receiving end,\nbut for this series, I didn't understand the sudden appearance of\nthe object_info_options member in the code that didn't even seem to\nbe populated anywhere, and I noticed that the series was organized\nincorrectly. Perhaps a simple rebase misordering?\n\n>> size_index was initialized to -1 and I was expecting we can tell the\n>> attribute is not used by checking (size_index < 0), but this seems\n>> to make size_index 1 based, not 0 based. Intended?\n>\n> Yes the server returns the packet as \"object_id SP size\" so the first\n> value is always the object_id.\n\nI think this is to skip the object name that comes as the first item\nin the response, but it would be more \"pure\" to keep foo_index\n0-based and add the offset by whatever constant number of things\nthat come in front (currently, 1 for object name) near a lot closer\nto where we parse and read the data, i.e. in the while() loop below.\n\nIOW, the code that sets size_index based on the attribute query\nresponse should not have to know how many fixed elements will come\nbefore these attributes on the response lines. That knowledge\nbelongs to the code below:\n\n> + i = 0;\n> + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> + struct string_list object_info_values = STRING_LIST_INIT_DUP;\n> +\n> + string_list_split(&object_info_values, reader.line, ' ', -1);\n> + if (size_index > 0) {\n> + if (!strcmp(object_info_values.items[size_index].string, \"\"))\n\nAnd here the code that uses size_index should be like\n\n\t\tif (0 <= size_index &&\n\t\t !strcmp(object_info_values.items[1 + size_index].string, \"\"))\n\t\t\t...\n\nif we wanted the logic to be more \"pure\" and keep foo_index 0-based.\n"},{"id":"454827","messageId":"xmqq8rrhnvph.fsf@gitster.g","threadId":"57388","inReplyTo":"20220503231558.4172288-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 5/8] transport: add client side capability to request object-info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-04T15:50:18Z","receivedAt":"2022-05-04T15:50:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> RTT more than necessary. We could just determine if the server supports\n> the attributes we want by using capabilities (without needing to send\n> the request to check).\n\nHmph. capabilities may cut in both ways, though.\n\nThose clients that are not interested in object-info at all (which\nare the majority of case where people clone, fetch or ls-remote),\nthey do not even need to know what kind of object attributes the\nobject-info command is prepared to report, and they would appreciate\nnot having to spend any extra byte in the capability-advertisement.\n\nOf course, for object-info clients, having it upfront does make\nthings simpler.\n\nSo...\n\n\n"},{"id":"454837","messageId":"xmqq8rrhm7x2.fsf@gitster.g","threadId":"57388","inReplyTo":"xmqqzgjztt7s.fsf@gitster.g","subject":"Re: [PATCH v4 3/8] object-store: add function to free object_info contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-04T19:09:29Z","receivedAt":"2022-05-04T19:09:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +}\n>> \\ No newline at end of file\n>\n> Let's avoid such incomplete lines.\n\nAs this breaks my build cycle (\"make sparse\" is part of my\npost-integration check), I have added this workaround on the tip of\nthe topic, but please make sure I do not have to redo that when you\nreroll.\n\nThanks.\n\nSubject: [PATCH] SQUASH??? compilation fix\n\n\"make sparse\" rightfully complains that this file ends in an\nincomplete line.\n---\n object-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 34a6e34adf..c6addb6969 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2674,4 +2674,4 @@ void free_object_info_contents(struct object_info *object_info)\n \t\tfree(object_info->delta_base_oid);\n \tif (object_info->type_name)\n \t\tfree(object_info->type_name);\n-}\n\\ No newline at end of file\n+}\n-- \n2.36.0-244-ge81c54a17b\n\n"},{"id":"454839","messageId":"20220504212738.162853-1-jonathantanmy@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-9-calvinwan@google.com","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-04T21:27:38Z","receivedAt":"2022-05-04T21:27:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks for the patch. I'll limit myself to high-level comments for now,\nsince the design may need to change if we want to support mixing\nremote-object-info with other commands in a --batch-command --buffer\ninvocation.\n\nCalvin Wan writes:\n> Since the `info` command in cat-file --batch-command prints object info\n> for a given object, it is natural to add another command in cat-file\n> --batch-command to print object info for a given object from a remote.\n> Add `remote-object-info` to cat-file --batch-command.\n> \n> While `info` takes object ids one at a time, this creates overhead when\n> making requests to a server so `remote-object-info` instead can take\n> multiple object ids at once.\n> \n> cat-file --batch-command is generally implemented in the following\n> manner:\n> \n> - Receive and parse input from user\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n\nYou mention below that this only works in --buffer mode, so I don't\nthink that this is relevant.\n\n> In --buffer mode, this changes to:\n> \n> - Receive and parse input from user\n> - Store respective function attached to command in a queue\n> - After flush, loop through commands in queue\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n> \n> Notice how the getting and printing of object info is accomplished one\n> at a time. As described above, this creates a problem for making\n> requests to a server. Therefore, `remote-object-info` is implemented in\n> the following manner:\n\nIf you do want this command to work with --buffer and without, probably\nclearer to replace \"Notice\" with \"Notice that when cat-file is invoked\nboth without --buffer and with it\". (Having said that, it makes sense to\nme to only make it work with --buffer - making one request per OID would\nprobably not be something that the user would want.)\n\n> - Receive and parse input from user\n> If command is `remote-object-info`:\n> - Get object info from remote\n> - Loop through object info\n> - Call respective function attached to `info`\n> - Set batch mode state, use passed in object info, print object\n> info\n> Else:\n> - Call respective function attached to command\n> - Parse input, get object info, print object info\n> \n> And finally for --buffer mode `remote-object-info`:\n> - Receive and parse input from user\n> - Store respective function attached to command in a queue\n> - After flush, loop through commands in queue:\n> If command is `remote-object-info`:\n> - Get object info from remote\n> - Loop through object info\n> - Call respective function attached to `info`\n> - Set batch mode state, use passed in object info, print\n> object info\n> Else:\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n> \n> To summarize, `remote-object-info` gets object info from the remote and\n> then generates multiples `info` commands with the object info passed in.\n> It cannot be implemented similarly to `info` and `content` because of\n> the overhead with remote commenication.\n\nWhat happens if remote-object-info is mixed with other commands?\n\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index 24a811f0ef..0d9e8e6214 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -115,6 +115,10 @@ info ::\n> \tPrint object info for object reference ``. This corresponds to the\n> \toutput of `--batch-check`.\n> \n> +remote-object-info []::\n> +\tPrint object info for object references `` at specified .\n> +\tThis command may only be combined with `--buffer`.\n\nHere you mention that we need --buffer.\n\n> @@ -253,13 +258,14 @@ newline. The available atoms are:\n> \n> `objectsize:disk`::\n> \tThe size, in bytes, that the object takes up on disk. See the\n> -\tnote about on-disk sizes in the `CAVEATS` section below.\n> +\tnote about on-disk sizes in the `CAVEATS` section below. Not\n> +\tsupported by `remote-object-info`.\n> \n> `deltabase`::\n> \tIf the object is stored as a delta on-disk, this expands to the\n> \tfull hex representation of the delta base object name.\n> \tOtherwise, expands to the null OID (all zeroes). See `CAVEATS`\n> -\tbelow.\n> +\tbelow. Not supported by `remote-object-info`.\n\nOK - makes sense that these are not supported by remote-object-info,\nbecause the remote may not even store objects in delta format.\n\n> @@ -32,11 +35,14 @@ struct batch_options {\n> \tint unordered;\n> \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n> \tconst char *format;\n> +\tint use_remote_info;\n> };\n> \n> #define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n> \n> static const char *force_path;\n> +static struct object_info *remote_object_info;\n> +static struct oid_array object_info_oids = OID_ARRAY_INIT;\n\nHmm...I would have expected the information to appear in the same data\nstructure used to store other command invocations like \"contents\",\n\"info\", and \"flush\", not separately like this. This design doesn't seem\nto support mixing commands.\n\n> @@ -538,6 +611,7 @@ typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n> struct queued_cmd {\n> \tparse_cmd_fn_t fn;\n> \tchar *line;\n> +\tconst char *name;\n> };\n\nRather than store a pointer like this (and thus needing to keep track of\nthe lifetime of the memory pointed to by this pointer), if we only need\nto know whether this queued_cmd is a remote-object-info call, we can\njust use \"unsigned is_remote_object_info : 1\".\n\n> @@ -565,9 +639,49 @@ static const struct parse_cmd {\n> } commands[] = {\n> \t{ \"contents\", parse_cmd_contents, 1},\n> \t{ \"info\", parse_cmd_info, 1},\n> +\t{ \"remote-object-info\", parse_cmd_info, 1},\n> \t{ \"flush\", NULL, 0},\n> };\n\nDouble parse_cmd_info?\n"},{"id":"454843","messageId":"xmqqv8ukkf6o.fsf@gitster.g","threadId":"57388","inReplyTo":"xmqqzgjztt7s.fsf@gitster.g","subject":"Re: [PATCH v4 3/8] object-store: add function to free object_info contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-05T00:15:27Z","receivedAt":"2022-05-05T00:15:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +\tif (object_info->typep)\n>> +\t\tfree(object_info->typep);\n>> +\tif (object_info->sizep)\n>> +\t\tfree(object_info->sizep);\n>> +\tif (object_info->disk_sizep)\n>> +\t\tfree(object_info->disk_sizep);\n>> +\tif (object_info->delta_base_oid)\n>> +\t\tfree(object_info->delta_base_oid);\n>> +\tif (object_info->type_name)\n>> +\t\tfree(object_info->type_name);\n>\n> \tif (PTR)\n> \t\tfree(PTR);\n>\n> can and should be written as\n>\n> \tfree(PTR);\n>\n> If we are reusing object_info after calling this function, we\n> _might_ want to use FREE_AND_NULL() instead of free().\n\n> As this breaks my build cycle (\"make sparse\" is part of my\n> post-integration check), I have added this workaround on the tip of\n> the topic, but please make sure I do not have to redo that when you\n> reroll.\n>\n> Thanks.\n\nAgain, this breaks the build at GitHub CI (static-analysis), I added\nthis workaround on the tip of the topic, merged it to 'seen' and\npushed the result out, but please make sure I do not have to redo\nthat when you reroll.\n\nThanks.\n"},{"id":"454868","messageId":"CAFySSZB72GTaJRvoGXgfkBE3w6+pYz9QVpGgEcsxy_8EDG-xpg@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqv8ukkf6o.fsf@gitster.g","subject":"Re: [PATCH v4 3/8] object-store: add function to free object_info contents","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-05T16:47:05Z","receivedAt":"2022-05-05T16:47:25Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Apologies for the mistakes. Getting a script from Josh Steadmon to\nmake sure these things never happen again :)\n\nOn Wed, May 4, 2022 at 5:15 PM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> >> + if (object_info->typep)\n> >> + free(object_info->typep);\n> >> + if (object_info->sizep)\n> >> + free(object_info->sizep);\n> >> + if (object_info->disk_sizep)\n> >> + free(object_info->disk_sizep);\n> >> + if (object_info->delta_base_oid)\n> >> + free(object_info->delta_base_oid);\n> >> + if (object_info->type_name)\n> >> + free(object_info->type_name);\n> >\n> > if (PTR)\n> > free(PTR);\n> >\n> > can and should be written as\n> >\n> > free(PTR);\n> >\n> > If we are reusing object_info after calling this function, we\n> > _might_ want to use FREE_AND_NULL() instead of free().\n>\n> > As this breaks my build cycle (\"make sparse\" is part of my\n> > post-integration check), I have added this workaround on the tip of\n> > the topic, but please make sure I do not have to redo that when you\n> > reroll.\n> >\n> > Thanks.\n>\n> Again, this breaks the build at GitHub CI (static-analysis), I added\n> this workaround on the tip of the topic, merged it to 'seen' and\n> pushed the result out, but please make sure I do not have to redo\n> that when you reroll.\n>\n> Thanks.\n"},{"id":"454869","messageId":"xmqqee17gbgh.fsf@gitster.g","threadId":"57388","inReplyTo":"CAFySSZB72GTaJRvoGXgfkBE3w6+pYz9QVpGgEcsxy_8EDG-xpg@mail.gmail.com","subject":"Re: [PATCH v4 3/8] object-store: add function to free object_info contents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-05T17:01:50Z","receivedAt":"2022-05-05T17:02:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Apologies for the mistakes. Getting a script from Josh Steadmon to\n> make sure these things never happen again :)\n\nDon't fret, mistakes happen. The static-analysis job is quite picky\nand catch these for us.\n\nSample runs are found here:\n\nhttps://github.com/git/git/runs/6299058086?check_suite_focus=true\nhttps://github.com/git/git/runs/6295167966?check_suite_focus=true\n\n"},{"id":"454870","messageId":"CAFySSZCL-ccVK+ZOU5=X4fB3-eu=Cmu-MGpT1cMW+bqmdRcF9w@mail.gmail.com","threadId":"57388","inReplyTo":"20220504212738.162853-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-05T18:13:19Z","receivedAt":"2022-05-05T18:13:38Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> You mention below that this only works in --buffer mode, so I don't\n> think that this is relevant.\n\nIt works in non-buffer mode as well. I meant that if you wanted to use\n`remote-object-info` with another option, then --filter is the only\none that it works with. I tried to match the documentation for\n--batch-command that says something similar:\n\n--batch-command=\nEnter a command mode that reads commands and arguments from stdin. May\nonly be combined with --buffer, --textconv or --filters\n\n> What happens if remote-object-info is mixed with other commands?\nIt works with other commands. The idea is that you would enter a line like:\n\nremote-object-info ... \n\nrather than\n\nremote-object-info \nremote-object-info \nremote-object-info \n\nsince each invocation of remote-object-info would cause another round\ntrip to the server.\n\n> Hmm...I would have expected the information to appear in the same data\n> structure used to store other command invocations like \"contents\",\n> \"info\", and \"flush\", not separately like this. This design doesn't seem\n> to support mixing commands.\n\nHere's an example of how it works with other commands. Let's say I\ncall git cat-file --batch-command --buffer and input the following:\n\nremote-object-info remote1 oid1 oid2 oid3\ncontents oid4\ninfo oid5\nremote-object-info remote2 oid6 oid7 oid8\nflush\n\nThis would result in the following:\n\n(call transport and store info in separate structure)\ninfo oid1 (using passed in object-info)\ninfo oid2\ninfo oid3\ncontents oid4\ninfo oid5\n(call transport and store info in separate structure)\ninfo oid6\ninfo oid7\ninfo oid8\n\n> Rather than store a pointer like this (and thus needing to keep track of\n> the lifetime of the memory pointed to by this pointer), if we only need\n> to know whether this queued_cmd is a remote-object-info call, we can\n> just use \"unsigned is_remote_object_info : 1\".\n\nOK\n\n>> @@ -565,9 +639,49 @@ static const struct parse_cmd {\n>> } commands[] = {\n>> { \"contents\", parse_cmd_contents, 1},\n>> { \"info\", parse_cmd_info, 1},\n>> + { \"remote-object-info\", parse_cmd_info, 1},\n>> { \"flush\", NULL, 0},\n>> };\n\n> Double parse_cmd_info?\n\nSee above for why remote-object-info becomes info\n\n\n\nOn Wed, May 4, 2022 at 2:27 PM Jonathan Tan wrote:\n>\n> Thanks for the patch. I'll limit myself to high-level comments for now,\n> since the design may need to change if we want to support mixing\n> remote-object-info with other commands in a --batch-command --buffer\n> invocation.\n>\n> Calvin Wan writes:\n> > Since the `info` command in cat-file --batch-command prints object info\n> > for a given object, it is natural to add another command in cat-file\n> > --batch-command to print object info for a given object from a remote.\n> > Add `remote-object-info` to cat-file --batch-command.\n> >\n> > While `info` takes object ids one at a time, this creates overhead when\n> > making requests to a server so `remote-object-info` instead can take\n> > multiple object ids at once.\n> >\n> > cat-file --batch-command is generally implemented in the following\n> > manner:\n> >\n> > - Receive and parse input from user\n> > - Call respective function attached to command\n> > - Set batch mode state, get object info, print object info\n>\n> You mention below that this only works in --buffer mode, so I don't\n> think that this is relevant.\n>\n> > In --buffer mode, this changes to:\n> >\n> > - Receive and parse input from user\n> > - Store respective function attached to command in a queue\n> > - After flush, loop through commands in queue\n> > - Call respective function attached to command\n> > - Set batch mode state, get object info, print object info\n> >\n> > Notice how the getting and printing of object info is accomplished one\n> > at a time. As described above, this creates a problem for making\n> > requests to a server. Therefore, `remote-object-info` is implemented in\n> > the following manner:\n>\n> If you do want this command to work with --buffer and without, probably\n> clearer to replace \"Notice\" with \"Notice that when cat-file is invoked\n> both without --buffer and with it\". (Having said that, it makes sense to\n> me to only make it work with --buffer - making one request per OID would\n> probably not be something that the user would want.)\n>\n> > - Receive and parse input from user\n> > If command is `remote-object-info`:\n> > - Get object info from remote\n> > - Loop through object info\n> > - Call respective function attached to `info`\n> > - Set batch mode state, use passed in object info, print object\n> > info\n> > Else:\n> > - Call respective function attached to command\n> > - Parse input, get object info, print object info\n> >\n> > And finally for --buffer mode `remote-object-info`:\n> > - Receive and parse input from user\n> > - Store respective function attached to command in a queue\n> > - After flush, loop through commands in queue:\n> > If command is `remote-object-info`:\n> > - Get object info from remote\n> > - Loop through object info\n> > - Call respective function attached to `info`\n> > - Set batch mode state, use passed in object info, print\n> > object info\n> > Else:\n> > - Call respective function attached to command\n> > - Set batch mode state, get object info, print object info\n> >\n> > To summarize, `remote-object-info` gets object info from the remote and\n> > then generates multiples `info` commands with the object info passed in.\n> > It cannot be implemented similarly to `info` and `content` because of\n> > the overhead with remote commenication.\n>\n> What happens if remote-object-info is mixed with other commands?\n>\n> > diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> > index 24a811f0ef..0d9e8e6214 100644\n> > --- a/Documentation/git-cat-file.txt\n> > +++ b/Documentation/git-cat-file.txt\n> > @@ -115,6 +115,10 @@ info ::\n> > Print object info for object reference ``. This corresponds to the\n> > output of `--batch-check`.\n> >\n> > +remote-object-info []::\n> > + Print object info for object references `` at specified .\n> > + This command may only be combined with `--buffer`.\n>\n> Here you mention that we need --buffer.\n>\n> > @@ -253,13 +258,14 @@ newline. The available atoms are:\n> >\n> > `objectsize:disk`::\n> > The size, in bytes, that the object takes up on disk. See the\n> > - note about on-disk sizes in the `CAVEATS` section below.\n> > + note about on-disk sizes in the `CAVEATS` section below. Not\n> > + supported by `remote-object-info`.\n> >\n> > `deltabase`::\n> > If the object is stored as a delta on-disk, this expands to the\n> > full hex representation of the delta base object name.\n> > Otherwise, expands to the null OID (all zeroes). See `CAVEATS`\n> > - below.\n> > + below. Not supported by `remote-object-info`.\n>\n> OK - makes sense that these are not supported by remote-object-info,\n> because the remote may not even store objects in delta format.\n>\n> > @@ -32,11 +35,14 @@ struct batch_options {\n> > int unordered;\n> > int transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n> > const char *format;\n> > + int use_remote_info;\n> > };\n> >\n> > #define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n> >\n> > static const char *force_path;\n> > +static struct object_info *remote_object_info;\n> > +static struct oid_array object_info_oids = OID_ARRAY_INIT;\n>\n> Hmm...I would have expected the information to appear in the same data\n> structure used to store other command invocations like \"contents\",\n> \"info\", and \"flush\", not separately like this. This design doesn't seem\n> to support mixing commands.\n>\n> > @@ -538,6 +611,7 @@ typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n> > struct queued_cmd {\n> > parse_cmd_fn_t fn;\n> > char *line;\n> > + const char *name;\n> > };\n>\n> Rather than store a pointer like this (and thus needing to keep track of\n> the lifetime of the memory pointed to by this pointer), if we only need\n> to know whether this queued_cmd is a remote-object-info call, we can\n> just use \"unsigned is_remote_object_info : 1\".\n>\n> > @@ -565,9 +639,49 @@ static const struct parse_cmd {\n> > } commands[] = {\n> > { \"contents\", parse_cmd_contents, 1},\n> > { \"info\", parse_cmd_info, 1},\n> > + { \"remote-object-info\", parse_cmd_info, 1},\n> > { \"flush\", NULL, 0},\n> > };\n>\n> Double parse_cmd_info?\n"},{"id":"454871","messageId":"CAFySSZAdSNpaZ_ghN2Oa6OE4rQ-H387YT7VPPGE_FK0zTXpFCA@mail.gmail.com","threadId":"57388","inReplyTo":"20220503230658.4170414-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 2/8] fetch-pack: move fetch default settings","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-05T18:13:34Z","receivedAt":"2022-05-05T18:13:49Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Done thanks for the suggestions\n\nOn Tue, May 3, 2022 at 4:07 PM Jonathan Tan wrote:\n>\n> Junio C Hamano writes:\n> > Calvin Wan writes:\n> >\n> > > When the state machine in do_fetch_pack_v2() is in FETCH_CHECK_LOCAL, we\n> > > set a few v2-specific defaults. It will be helpful for a future patch to\n> > > have these defaults set if the initial state is not FETCH_CHECK_LOCAL.\n> >\n> >\n> > > This is a safe change since the initial state is currently always\n> > > FETCH_CHECK_LOCAL, so we're guaranteed to hit that code whether it's in\n> > > the state machine or not.\n> >\n> > What does \"it\" (which supposed to be able to be in the state machine\n> > and also not to be in the state matchine) in the last sentence refer\n> > to?\n>\n> I think the \"it\" refers to the initialization code that was moved.\n>\n> > The patch looks correct and I agree that this is a benign no-op\n> > because the initial value of state is FETCH_CHECK_LOCAL (i.e. the\n> > block of code moved here will execute pretty much as the first\n> > thing, and the relative order between that block and sorting of ref\n> > list should not matter). I just didn't understand the explanation\n> > given by the patch why it is safe.\n> >\n> > Thanks.\n>\n> Agreed - I think the commit message would be better off if it was worded\n> something like:\n>\n> There are some initialization steps that need to be done at the start\n> of the execution of the do_fetch_pack_v2() state machine. Currently,\n> these are done in the FETCH_CHECK_LOCAL state, which is the initial\n> state.\n>\n> However, a subsequent patch will allow for another initial state,\n> while still requiring these initialization steps. Therefore, move\n> these initialization steps to before the state machine, so that they\n> are run regardless of the initial state.\n>\n> I think this description suffices for a reviewer to see that it is safe,\n> but if you want, you can add:\n>\n> Note that there is no change in behavior, because we're moving code\n> from the beginning of the first state to just before the execution of\n> the state machine.\n"},{"id":"454873","messageId":"xmqqbkwbes4d.fsf@gitster.g","threadId":"57388","inReplyTo":"CAFySSZCL-ccVK+ZOU5=X4fB3-eu=Cmu-MGpT1cMW+bqmdRcF9w@mail.gmail.com","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-05T18:44:50Z","receivedAt":"2022-05-05T18:46:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> It works with other commands. The idea is that you would enter a line like:\n>\n> remote-object-info ... \n>\n> rather than\n>\n> remote-object-info \n> remote-object-info \n> remote-object-info \n>\n> since each invocation of remote-object-info would cause another round\n> trip to the server.\n\nHmm. It looks unfortunate, but if we do not care about hitting the\npkt-line length limit, that might be OK. I dunno.\n\nI would have expected to see something like\n\n\tstart batch\n\trequest 1\n\trequest 2\n\t...\n\trequest 2000\n\tflush batch\n\nduring which the other side patiently listens to our requests. They\n(meaning the local process that reads the above and talks to a\nremote as needed) can coalesce the requests of the same kind\n(e.g. going to the same remote) while buffering and optimize their\noperation without having the caller of them to worry about it that\nway, no?\n"},{"id":"454875","messageId":"xmqq1qx7eqyk.fsf@gitster.g","threadId":"57388","inReplyTo":"xmqqbkwbes4d.fsf@gitster.g","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-05T19:09:55Z","receivedAt":"2022-05-05T19:10:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I would have expected to see something like\n>\n> \tstart batch\n> \trequest 1\n> \trequest 2\n> \t...\n> \trequest 2000\n> \tflush batch\n>\n> during which the other side patiently listens to our requests. They\n> (meaning the local process that reads the above and talks to a\n> remote as needed) can coalesce the requests of the same kind\n> (e.g. going to the same remote) while buffering and optimize their\n> operation without having the caller of them to worry about it that\n> way, no?\n\nAh, that is in the batch mode, and you were syaing that \"one request\nwith multiple objects\" would allow multiple object-info requests to\nbe hanled efficiently even in non-batch mode. If that was what you\nwere talking about, I think that does make sense. Thanks.\n\n"},{"id":"454876","messageId":"CAFySSZDA-j1m3OQRRmXAj-0EP4UcZmExuoJUCpjhCtTUBsLNyg@mail.gmail.com","threadId":"57388","inReplyTo":"xmqq1qx7eqyk.fsf@gitster.g","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-05-05T19:19:42Z","receivedAt":"2022-05-05T19:19:57Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> Ah, that is in the batch mode, and you were syaing that \"one request\n> with multiple objects\" would allow multiple object-info requests to\n> be hanled efficiently even in non-batch mode. If that was what you\n> were talking about, I think that does make sense. Thanks.\n\nCorrect, also I think you mean \"buffer\" mode rather than \"batch\" mode.\n\nOn Thu, May 5, 2022 at 12:09 PM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > I would have expected to see something like\n> >\n> > start batch\n> > request 1\n> > request 2\n> > ...\n> > request 2000\n> > flush batch\n> >\n> > during which the other side patiently listens to our requests. They\n> > (meaning the local process that reads the above and talks to a\n> > remote as needed) can coalesce the requests of the same kind\n> > (e.g. going to the same remote) while buffering and optimize their\n> > operation without having the caller of them to worry about it that\n> > way, no?\n>\n> Ah, that is in the batch mode, and you were syaing that \"one request\n> with multiple objects\" would allow multiple object-info requests to\n> be hanled efficiently even in non-batch mode. If that was what you\n> were talking about, I think that does make sense. Thanks.\n>\n"},{"id":"460155","messageId":"20220728230210.2952731-1-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 0/6] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:04Z","receivedAt":"2022-07-28T23:02:26Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Sometimes it is useful to get information about an object without having\nto download it completely. The server logic has already been implemented\nas “a2ba162cda (object-info: support for retrieving object info,\n2021-04-20)”. This patch implements the client option for it.\n\nAdd `--object-info` option to `cat-file --batch-command`. This option\nallows the client to make an object-info command request to a server\nthat supports protocol v2. If the server is v2, but does not allow for\nthe object-info command request, the entire object is fetched and the\nrelevant object info is returned.\n\n=== Changes since v4 ===\n - Instead of making 2 rounds trips to determine whether the server can\n handle the client object-info request, the server now advertises the\n features it can send back so the client can immediately fallback to\n fetch if any feature is not supported\n - Fixed nits and clarified some documentation/commit messages\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n\nCalvin Wan (6):\n fetch-pack: refactor packet writing\n fetch-pack: move fetch initialization\n protocol-caps: initialization bug fix\n serve: advertise object-info feature\n transport: add client support for object-info\n cat-file: add remote-object-info to batch-command\n\n Documentation/git-cat-file.txt | 16 +-\n builtin/cat-file.c | 225 ++++++++++++++++-----\n fetch-pack.c | 52 +++--\n fetch-pack.h | 10 +\n object-file.c | 11 ++\n object-store.h | 3 +\n protocol-caps.c | 2 +-\n serve.c | 10 +-\n t/t1006-cat-file.sh | 348 +++++++++++++++++++++++++++++++++\n t/t5555-http-smart-common.sh | 2 +-\n t/t5701-git-serve.sh | 2 +-\n transport-helper.c | 7 +-\n transport.c | 110 ++++++++++-\n transport.h | 11 ++\n 14 files changed, 734 insertions(+), 75 deletions(-)\n\n\nbase-commit: 9dd64cb4d310986dd7b8ca7fff92f9b61e0bd21a\n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460156","messageId":"20220728230210.2952731-2-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 1/6] fetch-pack: refactor packet writing","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:05Z","receivedAt":"2022-07-28T23:02:35Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A subsequent patch will need to write capabilities for another command,\nso refactor write_fetch_command_and_capabilities() to be used by both\nfetch and future command.\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n---\n fetch-pack.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex cb6647d657..5176420b62 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1261,13 +1261,13 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \treturn haves_added;\n }\n \n-static void write_fetch_command_and_capabilities(struct strbuf *req_buf,\n-\t\t\t\t\t\t const struct string_list *server_options)\n+static void write_command_and_capabilities(struct strbuf *req_buf,\n+\t\t\t\t\t\t const struct string_list *server_options, const char* command)\n {\n \tconst char *hash_name;\n \n-\tif (server_supports_v2(\"fetch\", 1))\n-\t\tpacket_buf_write(req_buf, \"command=fetch\");\n+\tif (server_supports_v2(command, 1))\n+\t\tpacket_buf_write(req_buf, \"command=%s\", command);\n \tif (server_supports_v2(\"agent\", 0))\n \t\tpacket_buf_write(req_buf, \"agent=%s\", git_user_agent_sanitized());\n \tif (advertise_sid && server_supports_v2(\"session-id\", 0))\n@@ -1303,7 +1303,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \tint done_sent = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n-\twrite_fetch_command_and_capabilities(&req_buf, args->server_options);\n+\twrite_command_and_capabilities(&req_buf, args->server_options, \"fetch\");\n \n \tif (args->use_thin_pack)\n \t\tpacket_buf_write(&req_buf, \"thin-pack\");\n@@ -2090,7 +2090,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n \t\tint received_ready = 0;\n \n \t\tstrbuf_reset(&req_buf);\n-\t\twrite_fetch_command_and_capabilities(&req_buf, server_options);\n+\t\twrite_command_and_capabilities(&req_buf, server_options, \"fetch\");\n \n \t\tpacket_buf_write(&req_buf, \"wait-for-done\");\n \n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460157","messageId":"20220728230210.2952731-3-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 2/6] fetch-pack: move fetch initialization","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:06Z","receivedAt":"2022-07-28T23:02:37Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"There are some variables that are initialized at the start of the\ndo_fetch_pack_v2() state machine. Currently, these are initialized\nin FETCH_CHECK_LOCAL, which is the initial state set at the beginning\nof the function.\n\nHowever, a subsequent patch will allow for another initial state,\nwhile still requiring these initialized variables. Therefore, move\nthe initialization to before the state machine, so that they are set\nregardless of the initial state.\n\nNote that there is no change in behavior, because we're moving code\nfrom the beginning of the first state to just before the execution of\nthe state machine.\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n---\n fetch-pack.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 5176420b62..8c862b017e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1628,18 +1628,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\treader.me = \"fetch-pack\";\n \t}\n \n+\t/* v2 supports these by default */\n+\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\tuse_sideband = 2;\n+\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n+\t\targs->deepen = 1;\n+\n \twhile (state != FETCH_DONE) {\n \t\tswitch (state) {\n \t\tcase FETCH_CHECK_LOCAL:\n \t\t\tsort_ref_list(&ref, ref_compare_name);\n \t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n \n-\t\t\t/* v2 supports these by default */\n-\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n-\t\t\tuse_sideband = 2;\n-\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n-\t\t\t\targs->deepen = 1;\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460158","messageId":"20220728230210.2952731-4-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 3/6] protocol-caps: initialization bug fix","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:07Z","receivedAt":"2022-07-28T23:02:38Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Initialize info. If info.size bit was on due to on-stack garbage,\nwe would have given our response with \"size\" attribute prefixed,\neven when the client side never requested it.\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n---\n protocol-caps.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex bbde91810a..2ad9f45c59 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -77,7 +77,7 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \n int cap_object_info(struct repository *r, struct packet_reader *request)\n {\n-\tstruct requested_info info;\n+\tstruct requested_info info = { 0 };\n \tstruct packet_writer writer;\n \tstruct string_list oid_str_list = STRING_LIST_INIT_DUP;\n \n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460159","messageId":"20220728230210.2952731-5-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 4/6] serve: advertise object-info feature","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:08Z","receivedAt":"2022-07-28T23:02:46Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"In order for a client to know what object-info components a server can\nprovide, advertise supported object-info features. This will allow a\nclient to decide whether to query the server for object-info or fetch as\na fallback.\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n---\n serve.c | 10 +++++++++-\n t/t5555-http-smart-common.sh | 2 +-\n t/t5701-git-serve.sh | 2 +-\n 3 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/serve.c b/serve.c\nindex 733347f602..1adf9df4a8 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -56,6 +56,14 @@ static int session_id_advertise(struct repository *r, struct strbuf *value)\n \treturn 1;\n }\n \n+static int object_info_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, \"size\");\n+\treturn 1;\n+}\n+\n static void session_id_receive(struct repository *r,\n \t\t\t const char *client_sid)\n {\n@@ -132,7 +140,7 @@ static struct protocol_capability capabilities[] = {\n \t},\n \t{\n \t\t.name = \"object-info\",\n-\t\t.advertise = always_advertise,\n+\t\t.advertise = object_info_advertise,\n \t\t.command = cap_object_info,\n \t},\n };\ndiff --git a/t/t5555-http-smart-common.sh b/t/t5555-http-smart-common.sh\nindex b1cfe8b7db..5a16d4259a 100755\n--- a/t/t5555-http-smart-common.sh\n+++ b/t/t5555-http-smart-common.sh\n@@ -131,7 +131,7 @@ test_expect_success 'git upload-pack --advertise-refs: v2' '\n \tfetch=shallow wait-for-done\n \tserver-option\n \tobject-format=$(test_oid algo)\n-\tobject-info\n+\tobject-info=size\n \t0000\n \tEOF\n \ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb..ebb32644e3 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -20,7 +20,7 @@ test_expect_success 'test capability advertisement' '\n \tfetch=shallow wait-for-done\n \tserver-option\n \tobject-format=$(test_oid algo)\n-\tobject-info\n+\tobject-info=size\n \t0000\n \tEOF\n \n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460160","messageId":"20220728230210.2952731-6-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 5/6] transport: add client support for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:09Z","receivedAt":"2022-07-28T23:02:49Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Sometimes it is useful to get information about an object without having\nto download it completely. The server logic has already been implemented\nas “a2ba162cda (object-info: support for retrieving object info,\n2021-04-20)”. This patch adds client functions to communicate with the\nserver.\n\nThe client currently supports requesting a list of object ids with\nfeatures 'size' and 'type' from a v2 server. If a server does not\nadvertise either of the requested features, then the client falls back\nto making the request through 'fetch'.\n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n---\n fetch-pack.c | 28 ++++++++++++\n fetch-pack.h | 10 +++++\n transport-helper.c | 7 ++-\n transport.c | 110 +++++++++++++++++++++++++++++++++++++++++++--\n transport.h | 11 +++++\n 5 files changed, 161 insertions(+), 5 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 8c862b017e..d373aed775 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1293,6 +1293,31 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n \tpacket_buf_delim(req_buf);\n }\n \n+void send_object_info_request(int fd_out, struct object_info_args *args)\n+{\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\tsize_t i;\n+\n+\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n+\n+\tif (unsorted_string_list_has_string(args->object_info_options, \"size\"))\n+\t\tpacket_buf_write(&req_buf, \"size\");\n+\n+\tif (unsorted_string_list_has_string(args->object_info_options, \"type\"))\n+\t\tpacket_buf_write(&req_buf, \"type\");\n+\n+\tif (args->oids) {\n+\t\tfor (i = 0; i < args->oids->nr; i++)\n+\t\t\tpacket_buf_write(&req_buf, \"oid %s\", oid_to_hex(&args->oids->oid[i]));\n+\t}\n+\n+\tpacket_buf_flush(&req_buf);\n+\tif (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)\n+\t\tdie_errno(_(\"unable to write request to remote\"));\n+\n+\tstrbuf_release(&req_buf);\n+}\n+\n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n@@ -1634,6 +1659,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\targs->deepen = 1;\n \n+\tif (args->object_info)\n+\t\tstate = FETCH_SEND_REQUEST;\n+\n \twhile (state != FETCH_DONE) {\n \t\tswitch (state) {\n \t\tcase FETCH_CHECK_LOCAL:\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 8c7752fc82..11c513f748 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -17,6 +17,7 @@ struct fetch_pack_args {\n \tconst struct string_list *deepen_not;\n \tstruct list_objects_filter_options filter_options;\n \tconst struct string_list *server_options;\n+\tstruct object_info **object_info_data;\n \n \t/*\n \t * If not NULL, during packfile negotiation, fetch-pack will send \"have\"\n@@ -43,6 +44,7 @@ struct fetch_pack_args {\n \tunsigned reject_shallow_remote:1;\n \tunsigned deepen:1;\n \tunsigned refetch:1;\n+\tunsigned object_info:1;\n \n \t/*\n \t * Indicate that the remote of this request is a promisor remote. The\n@@ -69,6 +71,12 @@ struct fetch_pack_args {\n \tunsigned connectivity_checked:1;\n };\n \n+struct object_info_args {\n+\tstruct string_list *object_info_options;\n+\tconst struct string_list *server_options;\n+\tstruct oid_array *oids;\n+};\n+\n /*\n * sought represents remote references that should be updated from.\n * On return, the names that were found on the remote will have been\n@@ -102,4 +110,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+void send_object_info_request(int fd_out, struct object_info_args *args);\n+\n #endif\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 322c722478..48a6680200 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -686,13 +686,16 @@ static int fetch_refs(struct transport *transport,\n \n \t/*\n \t * If we reach here, then the server, the client, and/or the transport\n-\t * helper does not support protocol v2. --negotiate-only requires\n-\t * protocol v2.\n+\t * helper does not support protocol v2. --negotiate-only and --object-info\n+\t * require protocol v2.\n \t */\n \tif (data->transport_options.acked_commits) {\n \t\twarning(_(\"--negotiate-only requires protocol v2\"));\n \t\treturn -1;\n \t}\n+\tif (transport->smart_options->object_info) {\n+\t\tdie(_(\"--object-info requires protocol v2\"));\n+\t}\n \n \tif (!data->get_refs_list_called)\n \t\tget_refs_list_using_list(transport, 0);\ndiff --git a/transport.c b/transport.c\nindex 52db7a3cb0..2d503e2fbd 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -353,6 +353,80 @@ static struct ref *handshake(struct transport *transport, int for_push,\n \treturn refs;\n }\n \n+static int fetch_object_info(struct transport *transport, struct object_info **object_info_data)\n+{\n+\tsize_t i;\n+\tint size_index = -1;\n+\tstruct git_transport_data *data = transport->data;\n+\tstruct object_info_args args;\n+\tstruct packet_reader reader;\n+\n+\tmemset(&args, 0, sizeof(args));\n+\targs.server_options = transport->server_options;\n+\targs.object_info_options = transport->smart_options->object_info_options;\n+\targs.oids = transport->smart_options->object_info_oids;\n+\n+\tconnect_setup(transport, 0);\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\tPACKET_READ_CHOMP_NEWLINE |\n+\t\t\tPACKET_READ_GENTLE_ON_EOF |\n+\t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n+\tdata->version = discover_version(&reader);\n+\n+\ttransport->hash_algo = reader.hash_algo;\n+\n+\tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tif (!server_supports_v2(\"object-info\", 0))\n+\t\t\treturn -1;\n+\t\tif (unsorted_string_list_has_string(args.object_info_options, \"size\")\n+\t\t\t\t&& !server_supports_feature(\"object-info\", \"size\", 0)) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (unsorted_string_list_has_string(args.object_info_options, \"type\")\n+\t\t\t\t&& !server_supports_feature(\"object-info\", \"type\", 0)) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tsend_object_info_request(data->fd[1], &args);\n+\t\tbreak;\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tdie(_(\"wrong protocol version. expected v2\"));\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\tfor (i = 0; i < args.object_info_options->nr; i++) {\n+\t\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n+\t\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (unsorted_string_list_has_string(args.object_info_options, reader.line)) {\n+\t\t\tif (!strcmp(reader.line, \"size\"))\n+\t\t\t\tsize_index = i;\n+\t\t\tcontinue;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\ti = 0;\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL && i < args.oids->nr) {\n+\t\tstruct string_list object_info_values = STRING_LIST_INIT_DUP;\n+\n+\t\tstring_list_split(&object_info_values, reader.line, ' ', -1);\n+\t\tif (0 <= size_index) {\n+\t\t\tif (!strcmp(object_info_values.items[1 + size_index].string, \"\"))\n+\t\t\t\tdie(\"object-info: not our ref %s\",\n+\t\t\t\t\tobject_info_values.items[0].string);\n+\t\t\t*(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10);\n+\t\t}\n+\t\ti++;\n+\t}\n+\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\n+\treturn 0;\n+}\n+\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n \t\t\t\t\tstruct transport_ls_refs_options *options)\n {\n@@ -363,10 +437,12 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t int nr_heads, struct ref **to_fetch)\n {\n \tint ret = 0;\n+\tsize_t i;\n \tstruct git_transport_data *data = transport->data;\n \tstruct ref *refs = NULL;\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n+\tstruct ref *object_info_refs = xcalloc(1, sizeof (struct ref));\n \n \tmemset(&args, 0, sizeof(args));\n \targs.uploadpack = data->options.uploadpack;\n@@ -392,8 +468,25 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.server_options = transport->server_options;\n \targs.negotiation_tips = data->options.negotiation_tips;\n \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n-\n-\tif (!data->got_remote_heads) {\n+\targs.object_info = transport->smart_options->object_info;\n+\n+\tif (transport->smart_options && transport->smart_options->object_info) {\n+\t\tstruct ref *ref = object_info_refs;\n+\n+\t\tif (!fetch_object_info(transport, data->options.object_info_data))\n+\t\t\tgoto cleanup;\n+\t\targs.object_info_data = data->options.object_info_data;\n+\t\targs.quiet = 1;\n+\t\targs.no_progress = 1;\n+\t\tfor (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n+\t\t\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n+\t\t\ttemp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i);\n+\t\t\ttemp_ref->exact_oid = 1;\n+\t\t\tref->next = temp_ref;\n+\t\t\tref = ref->next;\n+\t\t}\n+\t\ttransport->remote_refs = object_info_refs->next;\n+\t} else if (!data->got_remote_heads) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n \t\tfor (i = 0; i < nr_heads; i++) {\n@@ -433,12 +526,22 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfiles, data->version);\n \n+\tif (args.object_info) {\n+\t\tstruct ref *ref_cpy_reader = object_info_refs->next;\n+\t\tint i = 0;\n+\t\twhile (ref_cpy_reader) {\n+\t\t\toid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args.object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE);\n+\t\t\tref_cpy_reader = ref_cpy_reader->next;\n+\t\t\ti++;\n+\t\t}\n+\t}\n+\n \tdata->got_remote_heads = 0;\n \tdata->options.self_contained_and_connected =\n \t\targs.self_contained_and_connected;\n \tdata->options.connectivity_checked = args.connectivity_checked;\n \n-\tif (!refs)\n+\tif (refs == NULL && !args.object_info)\n \t\tret = -1;\n \tif (report_unmatched_refs(to_fetch, nr_heads))\n \t\tret = -1;\n@@ -453,6 +556,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tfree_refs(refs_tmp);\n \tfree_refs(refs);\n+\tfree_refs(object_info_refs);\n \treturn ret;\n }\n \ndiff --git a/transport.h b/transport.h\nindex b5bf7b3e70..5512fdb140 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -6,6 +6,7 @@\n #include \"remote.h\"\n #include \"list-objects-filter-options.h\"\n #include \"string-list.h\"\n+#include \"object-store.h\"\n \n struct git_transport_options {\n \tunsigned thin : 1;\n@@ -31,6 +32,12 @@ struct git_transport_options {\n \t */\n \tunsigned connectivity_checked:1;\n \n+\t/*\n+\t * Transport will attempt to pull only object-info. Fallbacks\n+\t * to pulling entire object if object-info is not supported\n+\t */\n+\tunsigned object_info : 1;\n+\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n@@ -54,6 +61,10 @@ struct git_transport_options {\n \t * common commits to this oidset instead of fetching any packfiles.\n \t */\n \tstruct oidset *acked_commits;\n+\n+\tstruct oid_array *object_info_oids;\n+\tstruct object_info **object_info_data;\n+\tstruct string_list *object_info_options;\n };\n \n enum transport_family {\n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460161","messageId":"20220728230210.2952731-7-calvinwan@google.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"[PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-07-28T23:02:10Z","receivedAt":"2022-07-28T23:02:52Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Since the `info` command in cat-file --batch-command prints object info\nfor a given object, it is natural to add another command in cat-file\n--batch-command to print object info for a given object from a remote.\nAdd `remote-object-info` to cat-file --batch-command.\n\nWhile `info` takes object ids one at a time, this creates overhead when\nmaking requests to a server so `remote-object-info` instead can take\nmultiple object ids at once.\n\ncat-file --batch-command is generally implemented in the following\nmanner:\n\n - Receive and parse input from user\n - Call respective function attached to command\n - Set batch mode state, get object info, print object info\n\nIn --buffer mode, this changes to:\n\n - Receive and parse input from user\n - Store respective function attached to command in a queue\n - After flush, loop through commands in queue\n - Call respective function attached to command\n - Set batch mode state, get object info, print object info\n\nNotice how the getting and printing of object info is accomplished one\nat a time. As described above, this creates a problem for making\nrequests to a server. Therefore, `remote-object-info` is implemented in\nthe following manner:\n\n - Receive and parse input from user\n If command is `remote-object-info`:\n - Get object info from remote\n - Loop through object info\n - Call respective function attached to `info`\n - Set batch mode state, use passed in object info, print object\n info\n Else:\n - Call respective function attached to command\n - Parse input, get object info, print object info\n\nAnd finally for --buffer mode `remote-object-info`:\n - Receive and parse input from user\n - Store respective function attached to command in a queue\n - After flush, loop through commands in queue:\n If command is `remote-object-info`:\n - Get object info from remote\n - Loop through object info\n - Call respective function attached to `info`\n - Set batch mode state, use passed in object info, print\n object info\n Else:\n - Call respective function attached to command\n - Set batch mode state, get object info, print object info\n\nTo summarize, `remote-object-info` gets object info from the remote and\nthen generates multiple `info` commands with the object info passed in.\n\nIn order for remote-object-info to avoid remote communication overhead\nin the non-buffer mode, the objects are passed in as such:\n\nremote-object-info ... \n\nrather than\n\nremote-object-info \nremote-object-info \n...\nremote-object-info \n\nSigned-off-by: Calvin Wan \nHelped-by: Jonathan Tan \n---\n Documentation/git-cat-file.txt | 16 +-\n builtin/cat-file.c | 225 ++++++++++++++++-----\n object-file.c | 11 ++\n object-store.h | 3 +\n t/t1006-cat-file.sh | 348 +++++++++++++++++++++++++++++++++\n 5 files changed, 549 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex 24a811f0ef..c073d5e50d 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -115,6 +115,10 @@ info ::\n \tPrint object info for object reference ``. This corresponds to the\n \toutput of `--batch-check`.\n \n+remote-object-info []::\n+\tPrint object info for object references `` at specified .\n+\tThis command may be combined with `--buffer`.\n+\n flush::\n \tUsed with `--buffer` to execute all preceding commands that were issued\n \tsince the beginning or since the last flush was issued. When `--buffer`\n@@ -245,7 +249,8 @@ newline. The available atoms are:\n \tThe full hex representation of the object name.\n \n `objecttype`::\n-\tThe type of the object (the same as `cat-file -t` reports).\n+\tThe type of the object (the same as `cat-file -t` reports). See\n+\t`CAVEATS` below.\n \n `objectsize`::\n \tThe size, in bytes, of the object (the same as `cat-file -s`\n@@ -253,13 +258,14 @@ newline. The available atoms are:\n \n `objectsize:disk`::\n \tThe size, in bytes, that the object takes up on disk. See the\n-\tnote about on-disk sizes in the `CAVEATS` section below.\n+\tnote about on-disk sizes in the `CAVEATS` section below. Not\n+\tsupported by `remote-object-info`.\n \n `deltabase`::\n \tIf the object is stored as a delta on-disk, this expands to the\n \tfull hex representation of the delta base object name.\n \tOtherwise, expands to the null OID (all zeroes). See `CAVEATS`\n-\tbelow.\n+\tbelow. Not supported by `remote-object-info`.\n \n `rest`::\n \tIf this atom is used in the output string, input lines are split\n@@ -346,6 +352,10 @@ directory name.\n CAVEATS\n -------\n \n+Note that since object type is currently not supported by the\n+object-info command request, git fetches the entire object instead to\n+get the object info.\n+\n Note that the sizes of objects on disk are reported accurately, but care\n should be taken in drawing conclusions about which refs or objects are\n responsible for disk usage. The size of a packed non-delta object may be\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 50cf38999d..68b72377e6 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -16,6 +16,9 @@\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n+#include \"alias.h\"\n+#include \"remote.h\"\n+#include \"transport.h\"\n \n enum batch_mode {\n \tBATCH_MODE_CONTENTS,\n@@ -32,9 +35,14 @@ struct batch_options {\n \tint unordered;\n \tint transform_mode; /* may be 'w' or 'c' for --filters or --textconv */\n \tconst char *format;\n+\tint use_remote_info;\n };\n \n+#define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n+\n static const char *force_path;\n+static struct object_info *remote_object_info;\n+static struct oid_array object_info_oids = OID_ARRAY_INIT;\n \n static int filter_object(const char *path, unsigned mode,\n \t\t\t const struct object_id *oid,\n@@ -423,48 +431,115 @@ static void batch_one_object(const char *obj_name,\n \tint flags = opt->follow_symlinks ? GET_OID_FOLLOW_SYMLINKS : 0;\n \tenum get_oid_result result;\n \n-\tresult = get_oid_with_context(the_repository, obj_name,\n-\t\t\t\t flags, &data->oid, &ctx);\n-\tif (result != FOUND) {\n-\t\tswitch (result) {\n-\t\tcase MISSING_OBJECT:\n-\t\t\tprintf(\"%s missing\\n\", obj_name);\n-\t\t\tbreak;\n-\t\tcase SHORT_NAME_AMBIGUOUS:\n-\t\t\tprintf(\"%s ambiguous\\n\", obj_name);\n-\t\t\tbreak;\n-\t\tcase DANGLING_SYMLINK:\n-\t\t\tprintf(\"dangling %\"PRIuMAX\"\\n%s\\n\",\n-\t\t\t (uintmax_t)strlen(obj_name), obj_name);\n-\t\t\tbreak;\n-\t\tcase SYMLINK_LOOP:\n-\t\t\tprintf(\"loop %\"PRIuMAX\"\\n%s\\n\",\n-\t\t\t (uintmax_t)strlen(obj_name), obj_name);\n-\t\t\tbreak;\n-\t\tcase NOT_DIR:\n-\t\t\tprintf(\"notdir %\"PRIuMAX\"\\n%s\\n\",\n-\t\t\t (uintmax_t)strlen(obj_name), obj_name);\n-\t\t\tbreak;\n-\t\tdefault:\n-\t\t\tBUG(\"unknown get_sha1_with_context result %d\\n\",\n-\t\t\t result);\n-\t\t\tbreak;\n+\tif (!opt->use_remote_info) {\n+\t\tresult = get_oid_with_context(the_repository, obj_name,\n+\t\t\t\t\t\tflags, &data->oid, &ctx);\n+\t\tif (result != FOUND) {\n+\t\t\tswitch (result) {\n+\t\t\tcase MISSING_OBJECT:\n+\t\t\t\tprintf(\"%s missing\\n\", obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase SHORT_NAME_AMBIGUOUS:\n+\t\t\t\tprintf(\"%s ambiguous\\n\", obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase DANGLING_SYMLINK:\n+\t\t\t\tprintf(\"dangling %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t\t(uintmax_t)strlen(obj_name), obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase SYMLINK_LOOP:\n+\t\t\t\tprintf(\"loop %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t\t(uintmax_t)strlen(obj_name), obj_name);\n+\t\t\t\tbreak;\n+\t\t\tcase NOT_DIR:\n+\t\t\t\tprintf(\"notdir %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t\t(uintmax_t)strlen(obj_name), obj_name);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\tBUG(\"unknown get_sha1_with_context result %d\\n\",\n+\t\t\t\t\tresult);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tfflush(stdout);\n+\t\t\treturn;\n \t\t}\n-\t\tfflush(stdout);\n-\t\treturn;\n-\t}\n \n-\tif (ctx.mode == 0) {\n-\t\tprintf(\"symlink %\"PRIuMAX\"\\n%s\\n\",\n-\t\t (uintmax_t)ctx.symlink_path.len,\n-\t\t ctx.symlink_path.buf);\n-\t\tfflush(stdout);\n-\t\treturn;\n+\t\tif (ctx.mode == 0) {\n+\t\t\tprintf(\"symlink %\"PRIuMAX\"\\n%s\\n\",\n+\t\t\t\t(uintmax_t)ctx.symlink_path.len,\n+\t\t\t\tctx.symlink_path.buf);\n+\t\t\tfflush(stdout);\n+\t\t\treturn;\n+\t\t}\n \t}\n \n \tbatch_object_write(obj_name, scratch, opt, data, NULL, 0);\n }\n \n+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)\n+{\n+\tint retval = 0;\n+\tsize_t i;\n+\tstruct remote *remote = NULL;\n+\tstruct object_id oid;\n+\tstruct string_list object_info_options = STRING_LIST_INIT_NODUP;\n+\tstatic struct transport *gtransport;\n+\tchar *format = DEFAULT_FORMAT;\n+\n+\tif (opt->format)\n+\t\tformat = xstrdup(opt->format);\n+\n+\tremote = remote_get(argv[0]);\n+\tif (!remote)\n+\t\tdie(_(\"must supply valid remote when using --object-info\"));\n+\toid_array_clear(&object_info_oids);\n+\tfor (i = 1; i < argc; i++) {\n+\t\tif (get_oid(argv[i], &oid))\n+\t\t\tdie(_(\"malformed object id '%s'\"), argv[i]);\n+\t\toid_array_append(&object_info_oids, &oid);\n+\t}\n+\n+\tgtransport = transport_get(remote, NULL);\n+\tif (gtransport->smart_options) {\n+\t\tsize_t j;\n+\t\tint include_size = 0, include_type = 0;\n+\n+\t\tremote_object_info = xcalloc(object_info_oids.nr, sizeof(struct object_info));\n+\t\tgtransport->smart_options->object_info = 1;\n+\t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n+\t\t/**\n+\t\t * 'size' is the only option currently supported.\n+\t\t * Other options that are passed in the format will fallback to a\n+\t\t * standard fetch request rather than object-info.\n+\t\t */\n+\t\tif (strstr(format, \"%(objectsize)\")) {\n+\t\t\tstring_list_append(&object_info_options, \"size\");\n+\t\t\tinclude_size = 1;\n+\t\t}\n+\t\tif (strstr(format, \"%(objecttype)\")) {\n+\t\t\tstring_list_append(&object_info_options, \"type\");\n+\t\t\tinclude_type = 1;\n+\t\t}\n+\t\tif (strstr(format, \"%(objectsize:disk)\"))\n+\t\t\tdie(_(\"objectsize:disk is currently not supported with remote-object-info\"));\n+\t\tif (strstr(format, \"%(deltabase)\"))\n+\t\t\tdie(_(\"deltabase is currently not supported with remote-object-info\"));\n+\t\tif (object_info_options.nr > 0) {\n+\t\t\tgtransport->smart_options->object_info_options = &object_info_options;\n+\t\t\tfor (j = 0; j < object_info_oids.nr; j++) {\n+\t\t\t\tif (include_size)\n+\t\t\t\t\tremote_object_info[j].sizep = xcalloc(1, sizeof(long));\n+\t\t\t\tif (include_type)\n+\t\t\t\t\tremote_object_info[j].typep = xcalloc(1, sizeof(enum object_type));\n+\t\t\t}\n+\t\t\tgtransport->smart_options->object_info_data = &remote_object_info;\n+\t\t\tretval = transport_fetch_refs(gtransport, NULL);\n+\t\t}\n+\t} else {\n+\t\tretval = -1;\n+\t}\n+\treturn retval;\n+}\n+\n struct object_cb_data {\n \tstruct batch_options *opt;\n \tstruct expand_data *expand;\n@@ -536,6 +611,7 @@ typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n struct queued_cmd {\n \tparse_cmd_fn_t fn;\n \tchar *line;\n+\tconst char *name;\n };\n \n static void parse_cmd_contents(struct batch_options *opt,\n@@ -556,6 +632,56 @@ static void parse_cmd_info(struct batch_options *opt,\n \tbatch_one_object(line, output, opt, data);\n }\n \n+static const struct parse_cmd {\n+\tconst char *name;\n+\tparse_cmd_fn_t fn;\n+\tunsigned takes_args;\n+} commands[] = {\n+\t{ \"contents\", parse_cmd_contents, 1},\n+\t{ \"info\", parse_cmd_info, 1},\n+\t{ \"remote-object-info\", parse_cmd_info, 1},\n+\t{ \"flush\", NULL, 0},\n+};\n+\n+static void parse_remote_info(struct batch_options *opt,\n+\t\t\t char *line,\n+\t\t\t struct strbuf *output,\n+\t\t\t struct expand_data *data,\n+\t\t\t const struct parse_cmd *p_cmd,\n+\t\t\t struct queued_cmd *q_cmd)\n+{\n+\tint count;\n+\tsize_t i;\n+\tconst char **argv;\n+\n+\tcount = split_cmdline(line, &argv);\n+\tif (get_remote_info(opt, count, argv))\n+\t\tgoto cleanup;\n+\topt->use_remote_info = 1;\n+\tdata->skip_object_info = 1;\n+\tdata->mark_query = 0;\n+\tfor (i = 0; i < object_info_oids.nr; i++) {\n+\t\tif (remote_object_info[i].sizep)\n+\t\t\tdata->size = *remote_object_info[i].sizep;\n+\t\tif (remote_object_info[i].typep)\n+\t\t\tdata->type = *remote_object_info[i].typep;\n+\n+\t\tdata->oid = object_info_oids.oid[i];\n+\t\tif (p_cmd)\n+\t\t\tp_cmd->fn(opt, argv[i+1], output, data);\n+\t\telse\n+\t\t\tq_cmd->fn(opt, argv[i+1], output, data);\n+\t}\n+\topt->use_remote_info = 0;\n+\tdata->skip_object_info = 0;\n+\tdata->mark_query = 1;\n+\n+cleanup:\n+\tfor (i = 0; i < object_info_oids.nr; i++)\n+\t\tfree_object_info_contents(&remote_object_info[i]);\n+\tfree(remote_object_info);\n+}\n+\n static void dispatch_calls(struct batch_options *opt,\n \t\tstruct strbuf *output,\n \t\tstruct expand_data *data,\n@@ -567,8 +693,12 @@ static void dispatch_calls(struct batch_options *opt,\n \tif (!opt->buffer_output)\n \t\tdie(_(\"flush is only for --buffer mode\"));\n \n-\tfor (i = 0; i < nr; i++)\n-\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\tfor (i = 0; i < nr; i++) {\n+\t\tif (!strcmp(cmd[i].name, \"remote-object-info\"))\n+\t\t\tparse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]);\n+\t\telse\n+\t\t\tcmd[i].fn(opt, cmd[i].line, output, data);\n+\t}\n \n \tfflush(stdout);\n }\n@@ -583,17 +713,6 @@ static void free_cmds(struct queued_cmd *cmd, size_t *nr)\n \t*nr = 0;\n }\n \n-\n-static const struct parse_cmd {\n-\tconst char *name;\n-\tparse_cmd_fn_t fn;\n-\tunsigned takes_args;\n-} commands[] = {\n-\t{ \"contents\", parse_cmd_contents, 1},\n-\t{ \"info\", parse_cmd_info, 1},\n-\t{ \"flush\", NULL, 0},\n-};\n-\n static void batch_objects_command(struct batch_options *opt,\n \t\t\t\t struct strbuf *output,\n \t\t\t\t struct expand_data *data)\n@@ -639,11 +758,17 @@ static void batch_objects_command(struct batch_options *opt,\n \t\t\tdispatch_calls(opt, output, data, queued_cmd, nr);\n \t\t\tfree_cmds(queued_cmd, &nr);\n \t\t} else if (!opt->buffer_output) {\n-\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\tif (!strcmp(cmd->name, \"remote-object-info\")) {\n+\t\t\t\tchar *line = xstrdup_or_null(p);\n+\t\t\t\tparse_remote_info(opt, line, output, data, cmd, NULL);\n+\t\t\t} else {\n+\t\t\t\tcmd->fn(opt, p, output, data);\n+\t\t\t}\n \t\t} else {\n \t\t\tALLOC_GROW(queued_cmd, nr + 1, alloc);\n \t\t\tcall.fn = cmd->fn;\n \t\t\tcall.line = xstrdup_or_null(p);\n+\t\t\tcall.name = cmd->name;\n \t\t\tqueued_cmd[nr++] = call;\n \t\t}\n \t}\n@@ -659,8 +784,6 @@ static void batch_objects_command(struct batch_options *opt,\n \tstrbuf_release(&input);\n }\n \n-#define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n-\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\ndiff --git a/object-file.c b/object-file.c\nindex 5b270f046d..417c76def3 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2839,3 +2839,14 @@ int read_loose_object(const char *path,\n \t\tmunmap(map, mapsize);\n \treturn ret;\n }\n+\n+void free_object_info_contents(struct object_info *object_info)\n+{\n+\tif (!object_info)\n+\t\treturn;\n+\tfree(object_info->typep);\n+\tfree(object_info->sizep);\n+\tfree(object_info->disk_sizep);\n+\tfree(object_info->delta_base_oid);\n+\tfree(object_info->type_name);\n+}\ndiff --git a/object-store.h b/object-store.h\nindex 5222ee5460..015838eb08 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -547,4 +547,7 @@ int for_each_object_in_pack(struct packed_git *p,\n int for_each_packed_object(each_packed_object_fn, void *,\n \t\t\t enum for_each_object_flags flags);\n \n+/* Free pointers inside of object_info, but not object_info itself */\n+void free_object_info_contents(struct object_info *object_info);\n+\n #endif /* OBJECT_STORE_H */\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex dadf3b1458..55b2a1f06f 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -2,6 +2,9 @@\n \n test_description='git cat-file'\n \n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n . ./test-lib.sh\n \n test_cmdmode_usage () {\n@@ -1093,4 +1096,349 @@ test_expect_success 'batch-command flush without --buffer' '\n \tgrep \"^fatal:.*flush is only for --buffer mode.*\" err\n '\n \n+# This section tests --batch-command with remote-object-info command\n+# If a filter is not set, the filter defaults to \"%(objectname) %(objectsize) %(objecttype)\"\n+# Since \"%(objecttype)\" is currently not supported by the command request, object-info,\n+# the filters are set to \"%(objectname) %(objectsize)\".\n+# Tests with the default filter are used to test the fallback to 'fetch' command\n+\n+set_transport_variables () {\n+ tree_sha1=$(git -C \"$1\" write-tree)\n+\tcommit_sha1=$(echo_without_newline \"$commit_message\" | git -C \"$1\" commit-tree $tree_sha1)\n+\ttag_sha1=$(echo_without_newline \"$tag_content\" | git -C \"$1\" hash-object -t tag --stdin -w)\n+\ttag_size=$(strlen \"$tag_content\")\n+}\n+\n+# Test --batch-command remote-object-info with 'git://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\techo_without_newline \"$hello_content\" > $daemon_parent/hello &&\n+\tgit -C \"$daemon_parent\" update-index --add hello\n+'\n+\n+set_transport_variables \"$daemon_parent\"\n+\n+test_expect_success 'batch-command remote-object-info git://' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $tree_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info git:// multiple sha1 per line' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info http:// default filter' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n+\t\tGIT_TRACE_PACKET=1 git cat-file --batch-command >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command --buffer remote-object-info git://' '\n+\t(\n+\t\tcd \"$daemon_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" --buffer >actual <<-EOF &&\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1 $tag_sha1\n+\t\tflush\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_git_daemon\n+\n+# Test --batch-command remote-object-info with 'http://' transport\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\techo_without_newline \"$hello_content\" > $HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" update-index --add hello\n+'\n+set_transport_variables \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\"\n+\n+test_expect_success 'batch-command remote-object-info http://' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $tree_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $commit_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info http:// one line' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command --buffer remote-object-info http://' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" --buffer >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $commit_sha1 $tag_sha1\n+\t\tflush\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info http:// default filter' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n+\n+\t\tgit cat-file --batch-command >actual <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on unspported filter option (objectsize:disk)' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command=\"%(objectsize:disk)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"objectsize:disk is currently not supported with remote-object-info\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on unspported filter option (deltabase)' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command=\"%(deltabase)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"deltabase is currently not supported with remote-object-info\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on server with legacy protocol' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git -c protocol.version=0 cat-file --batch-command=\"%(objectname) %(objectsize)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"object-info requires protocol v2\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on server with legacy protocol fallback' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\n+\t\ttest_must_fail git -c protocol.version=0 cat-file --batch-command 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n+\t\tEOF\n+\t\ttest_i18ngrep \"object-info requires protocol v2\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on malformed OID' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tmalformed_object_id=\"this_id_is_not_valid\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command=\"%(objectname) %(objectsize)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $malformed_object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"malformed object id '$malformed_object_id'\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on malformed OID fallback' '\n+\t(\n+\t\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\t\tmalformed_object_id=\"this_id_is_not_valid\" &&\n+\n+\t\ttest_must_fail git cat-file --batch-command 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $malformed_object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"malformed object id '$malformed_object_id'\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on missing OID' '\n+\tgit clone \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" missing_oid_repo &&\n+\ttest_commit -C missing_oid_repo message1 c.txt &&\n+\t(\n+\t\tcd missing_oid_repo &&\n+\t\tobject_id=$(git rev-parse message1:c.txt) &&\n+\t\ttest_must_fail git cat-file --batch-command=\"%(objectname) %(objectsize)\" 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"object-info: not our ref $object_id\" err\n+\t)\n+'\n+\n+test_expect_success 'remote-object-info fails on missing OID fallback' '\n+\t(\n+\t\tcd missing_oid_repo &&\n+\t\tobject_id=$(git rev-parse message1:c.txt) &&\n+\t\ttest_must_fail git cat-file --batch-command 2>err <<-EOF &&\n+\t\tremote-object-info \"$HTTPD_URL/smart/http_parent\" $object_id\n+\t\tEOF\n+\t\ttest_i18ngrep \"fatal: remote error: upload-pack: not our ref $object_id\" err\n+\t)\n+'\n+\n+# Test --batch-command remote-object-info with 'file://' transport\n+\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init server &&\n+\tgit -C server config protocol.version 2 &&\n+\techo_without_newline \"$hello_content\" > server/hello &&\n+\tgit -C server update-index --add hello\n+'\n+\n+set_transport_variables \"server\"\n+\n+test_expect_success 'batch-command remote-object-info file://' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $tree_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $commit_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info file:// multiple sha1 per line' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1 $tree_sha1 $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command --buffer remote-object-info file://' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command=\"%(objectname) %(objectsize)\" --buffer >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $commit_sha1 $tag_sha1\n+\t\tflush\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'batch-command remote-object-info file:// default filter' '\n+\t(\n+\t\tcd server &&\n+\n+\t\techo \"$hello_sha1 blob $hello_size\" >expect &&\n+\t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n+\t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n+\t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n+\t\tgit cat-file --batch-command >actual <<-EOF &&\n+\t\tremote-object-info \"file://$(pwd)\" $hello_sha1 $tree_sha1\n+\t\tremote-object-info \"file://$(pwd)\" $commit_sha1 $tag_sha1\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.37.1.455.g008518b4e5-goog\n\n"},{"id":"460165","messageId":"xmqqo7x8aj1b.fsf@gitster.g","threadId":"57388","inReplyTo":"20220728230210.2952731-1-calvinwan@google.com","subject":"Re: [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-28T23:56:16Z","receivedAt":"2022-07-28T23:56:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Sometimes it is useful to get information about an object without having\n> to download it completely. The server logic has already been implemented\n> as “a2ba162cda (object-info: support for retrieving object info,\n> 2021-04-20)”. This patch implements the client option for it.\n>\n> Add `--object-info` option to `cat-file --batch-command`. This option\n> allows the client to make an object-info command request to a server\n> that supports protocol v2. If the server is v2, but does not allow for\n> the object-info command request, the entire object is fetched and the\n> relevant object info is returned.\n\nThanks. Since this has changed significantly enough to render\nrange-diff less useful, I think a fresh round of full review may be\nneeded. I'll use a more recent tip of 'master' as the base when I\nqueue these patches.\n\n"},{"id":"460166","messageId":"xmqqk07waiqw.fsf@gitster.g","threadId":"57388","inReplyTo":"xmqqo7x8aj1b.fsf@gitster.g","subject":"Re: [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-29T00:02:31Z","receivedAt":"2022-07-29T00:02:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Thanks. Since this has changed significantly enough to render\n> range-diff less useful, I think a fresh round of full review may be\n> needed. I'll use a more recent tip of 'master' as the base when I\n> queue these patches.\n\nHaving said that, here is a diff between\n\n a/ is previous round queued with fixups merged to 'master'\n b/ is the latest round queued on top of 'master'.\n\nwhich may help people to see what got changed since the previous\nround.\n\nI notice that use of CALLOC_ARRAY() suggested during the review of\nthe last round was dropped (it should be easy to resurrect).\n\n\n Documentation/git-cat-file.txt | 2 +-\n builtin/cat-file.c | 4 +--\n fetch-pack.c | 17 +++--------\n fetch-pack.h | 4 +--\n object-file.c | 1 -\n protocol-caps.c | 12 ++++----\n serve.c | 10 +++++-\n t/t1006-cat-file.sh | 3 +-\n t/t5555-http-smart-common.sh | 2 +-\n t/t5701-git-serve.sh | 2 +-\n transport.c | 69 +++++++++++++++++++++++++-----------------\n 11 files changed, 70 insertions(+), 56 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\nindex 0d9e8e6214..c073d5e50d 100644\n--- a/Documentation/git-cat-file.txt\n+++ b/Documentation/git-cat-file.txt\n@@ -117,7 +117,7 @@ info ::\n \n remote-object-info []::\n \tPrint object info for object references `` at specified .\n-\tThis command may only be combined with `--buffer`.\n+\tThis command may be combined with `--buffer`.\n \n flush::\n \tUsed with `--buffer` to execute all preceding commands that were issued\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 48bf0ee02f..57c090f249 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -515,12 +515,12 @@ static int get_remote_info(struct batch_options *opt, int argc, const char **arg\n \t\tsize_t j;\n \t\tint include_size = 0, include_type = 0;\n \n-\t\tCALLOC_ARRAY(remote_object_info, object_info_oids.nr);\n+\t\tremote_object_info = xcalloc(object_info_oids.nr, sizeof(struct object_info));\n \t\tgtransport->smart_options->object_info = 1;\n \t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n \t\t/**\n \t\t * 'size' is the only option currently supported.\n-\t\t * Other options that are passed in the format will default to a\n+\t\t * Other options that are passed in the format will fallback to a\n \t\t * standard fetch request rather than object-info.\n \t\t */\n \t\tif (strstr(format, \"%(objectsize)\")) {\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex ca06a7130f..d373aed775 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1296,14 +1296,16 @@ static void write_command_and_capabilities(struct strbuf *req_buf,\n void send_object_info_request(int fd_out, struct object_info_args *args)\n {\n \tstruct strbuf req_buf = STRBUF_INIT;\n-\tstruct string_list unsorted_object_info_options = *args->object_info_options;\n \tsize_t i;\n \n \twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n \n-\tif (unsorted_string_list_has_string(&unsorted_object_info_options, \"size\"))\n+\tif (unsorted_string_list_has_string(args->object_info_options, \"size\"))\n \t\tpacket_buf_write(&req_buf, \"size\");\n \n+\tif (unsorted_string_list_has_string(args->object_info_options, \"type\"))\n+\t\tpacket_buf_write(&req_buf, \"type\");\n+\n \tif (args->oids) {\n \t\tfor (i = 0; i < args->oids->nr; i++)\n \t\t\tpacket_buf_write(&req_buf, \"oid %s\", oid_to_hex(&args->oids->oid[i]));\n@@ -1669,7 +1671,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n-\t\t\tif (!args->refetch && !args->object_info && everything_local(args, &ref))\n+\t\t\tif (!args->refetch && everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n@@ -2056,15 +2058,6 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\targs->connectivity_checked = 1;\n \t}\n \n-\tif (args->object_info) {\n-\t\tstruct ref *ref_cpy_reader = ref_cpy;\n-\t\tint i = 0;\n-\t\twhile (ref_cpy_reader) {\n-\t\t\toid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args->object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE);\n-\t\t\tref_cpy_reader = ref_cpy_reader->next;\n-\t\t\ti++;\n-\t\t}\n-\t}\n \tupdate_shallow(args, sought, nr_sought, &si);\n cleanup:\n \tclear_shallow_info(&si);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 552fd7bde0..11c513f748 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -72,9 +72,9 @@ struct fetch_pack_args {\n };\n \n struct object_info_args {\n-\tconst struct string_list *object_info_options;\n+\tstruct string_list *object_info_options;\n \tconst struct string_list *server_options;\n-\tconst struct oid_array *oids;\n+\tstruct oid_array *oids;\n };\n \n /*\ndiff --git a/object-file.c b/object-file.c\nindex 333fed9f68..417c76def3 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2844,7 +2844,6 @@ void free_object_info_contents(struct object_info *object_info)\n {\n \tif (!object_info)\n \t\treturn;\n-\n \tfree(object_info->typep);\n \tfree(object_info->sizep);\n \tfree(object_info->disk_sizep);\ndiff --git a/protocol-caps.c b/protocol-caps.c\nindex bc7def0727..2ad9f45c59 100644\n--- a/protocol-caps.c\n+++ b/protocol-caps.c\n@@ -11,7 +11,6 @@\n \n struct requested_info {\n \tunsigned size : 1;\n-\tunsigned unknown : 1;\n };\n \n /*\n@@ -41,12 +40,12 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \tstruct string_list_item *item;\n \tstruct strbuf send_buffer = STRBUF_INIT;\n \n+\tif (!oid_str_list->nr)\n+\t\treturn;\n+\n \tif (info->size)\n \t\tpacket_writer_write(writer, \"size\");\n \n-\tif (info->unknown || !oid_str_list->nr)\n-\t\tgoto release;\n-\n \tfor_each_string_list_item (item, oid_str_list) {\n \t\tconst char *oid_str = item->string;\n \t\tstruct object_id oid;\n@@ -73,7 +72,6 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n \t\tpacket_writer_write(writer, \"%s\", send_buffer.buf);\n \t\tstrbuf_reset(&send_buffer);\n \t}\n-release:\n \tstrbuf_release(&send_buffer);\n }\n \n@@ -94,7 +92,9 @@ int cap_object_info(struct repository *r, struct packet_reader *request)\n \t\tif (parse_oid(request->line, &oid_str_list))\n \t\t\tcontinue;\n \n-\t\tinfo.unknown = 1;\n+\t\tpacket_writer_error(&writer,\n+\t\t\t\t \"object-info: unexpected line: '%s'\",\n+\t\t\t\t request->line);\n \t}\n \n \tif (request->status != PACKET_READ_FLUSH) {\ndiff --git a/serve.c b/serve.c\nindex 733347f602..1adf9df4a8 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -56,6 +56,14 @@ static int session_id_advertise(struct repository *r, struct strbuf *value)\n \treturn 1;\n }\n \n+static int object_info_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, \"size\");\n+\treturn 1;\n+}\n+\n static void session_id_receive(struct repository *r,\n \t\t\t const char *client_sid)\n {\n@@ -132,7 +140,7 @@ static struct protocol_capability capabilities[] = {\n \t},\n \t{\n \t\t.name = \"object-info\",\n-\t\t.advertise = always_advertise,\n+\t\t.advertise = object_info_advertise,\n \t\t.command = cap_object_info,\n \t},\n };\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 7f0baea07c..55b2a1f06f 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1100,6 +1100,7 @@ test_expect_success 'batch-command flush without --buffer' '\n # If a filter is not set, the filter defaults to \"%(objectname) %(objectsize) %(objecttype)\"\n # Since \"%(objecttype)\" is currently not supported by the command request, object-info,\n # the filters are set to \"%(objectname) %(objectsize)\".\n+# Tests with the default filter are used to test the fallback to 'fetch' command\n \n set_transport_variables () {\n tree_sha1=$(git -C \"$1\" write-tree)\n@@ -1163,7 +1164,7 @@ test_expect_success 'batch-command remote-object-info http:// default filter' '\n \t\techo \"$tree_sha1 tree $tree_size\" >>expect &&\n \t\techo \"$commit_sha1 commit $commit_size\" >>expect &&\n \t\techo \"$tag_sha1 tag $tag_size\" >>expect &&\n-\t\tgit cat-file --batch-command >actual <<-EOF &&\n+\t\tGIT_TRACE_PACKET=1 git cat-file --batch-command >actual <<-EOF &&\n \t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $hello_sha1 $tree_sha1\n \t\tremote-object-info \"$GIT_DAEMON_URL/parent\" $commit_sha1 $tag_sha1\n \t\tEOF\ndiff --git a/t/t5555-http-smart-common.sh b/t/t5555-http-smart-common.sh\nindex b1cfe8b7db..5a16d4259a 100755\n--- a/t/t5555-http-smart-common.sh\n+++ b/t/t5555-http-smart-common.sh\n@@ -131,7 +131,7 @@ test_expect_success 'git upload-pack --advertise-refs: v2' '\n \tfetch=shallow wait-for-done\n \tserver-option\n \tobject-format=$(test_oid algo)\n-\tobject-info\n+\tobject-info=size\n \t0000\n \tEOF\n \ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 1896f671cb..ebb32644e3 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -20,7 +20,7 @@ test_expect_success 'test capability advertisement' '\n \tfetch=shallow wait-for-done\n \tserver-option\n \tobject-format=$(test_oid algo)\n-\tobject-info\n+\tobject-info=size\n \t0000\n \tEOF\n \ndiff --git a/transport.c b/transport.c\nindex 1baa1a194a..64bcc311ff 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -360,15 +360,16 @@ static int fetch_object_info(struct transport *transport, struct object_info **o\n \tstruct git_transport_data *data = transport->data;\n \tstruct object_info_args args;\n \tstruct packet_reader reader;\n-\tstruct string_list server_attributes = STRING_LIST_INIT_DUP;\n \n \tmemset(&args, 0, sizeof(args));\n \targs.server_options = transport->server_options;\n \targs.object_info_options = transport->smart_options->object_info_options;\n+\targs.oids = transport->smart_options->object_info_oids;\n \n \tconnect_setup(transport, 0);\n \tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n \t\t\tPACKET_READ_CHOMP_NEWLINE |\n+\t\t\tPACKET_READ_GENTLE_ON_EOF |\n \t\t\tPACKET_READ_DIE_ON_ERR_PACKET);\n \tdata->version = discover_version(&reader);\n \n@@ -378,25 +379,14 @@ static int fetch_object_info(struct transport *transport, struct object_info **o\n \tcase protocol_v2:\n \t\tif (!server_supports_v2(\"object-info\", 0))\n \t\t\treturn -1;\n-\t\t/**\n-\t\t * Send a request with only attributes first. If server can return all\n-\t\t * of the requested attributes, then send request with object ids\n-\t\t */\n-\t\tsend_object_info_request(data->fd[1], &args);\n-\t\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n-\t\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\t\tif (unsorted_string_list_has_string(args.object_info_options, \"size\")\n+\t\t\t\t&& !server_supports_feature(\"object-info\", \"size\", 0)) {\n \t\t\treturn -1;\n \t\t}\n-\t\tstring_list_split(&server_attributes, reader.line, ' ', -1);\n-\t\tpacket_reader_read(&reader);\n-\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n-\t\tif (server_attributes.nr != args.object_info_options->nr)\n+\t\tif (unsorted_string_list_has_string(args.object_info_options, \"type\")\n+\t\t\t\t&& !server_supports_feature(\"object-info\", \"type\", 0)) {\n \t\t\treturn -1;\n-\t\tfor (i = 0; i < server_attributes.nr; i++) {\n-\t\t\tif (!strcmp(server_attributes.items[i].string, \"size\"))\n-\t\t\t\tsize_index = i + 1;\n \t\t}\n-\t\targs.oids = transport->smart_options->object_info_oids;\n \t\tsend_object_info_request(data->fd[1], &args);\n \t\tbreak;\n \tcase protocol_v1:\n@@ -406,18 +396,29 @@ static int fetch_object_info(struct transport *transport, struct object_info **o\n \t\tBUG(\"unknown protocol version\");\n \t}\n \n-\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL)\n-\t\tdie(_(\"error reading object info header\"));\n+\tfor (i = 0; i < args.object_info_options->nr; i++) {\n+\t\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n+\t\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (unsorted_string_list_has_string(args.object_info_options, reader.line)) {\n+\t\t\tif (!strcmp(reader.line, \"size\"))\n+\t\t\t\tsize_index = i;\n+\t\t\tcontinue;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n \ti = 0;\n-\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL && i < args.oids->nr) {\n \t\tstruct string_list object_info_values = STRING_LIST_INIT_DUP;\n \n \t\tstring_list_split(&object_info_values, reader.line, ' ', -1);\n-\t\tif (size_index > 0) {\n-\t\t\tif (!strcmp(object_info_values.items[size_index].string, \"\"))\n+\t\tif (0 <= size_index) {\n+\t\t\tif (!strcmp(object_info_values.items[1 + size_index].string, \"\"))\n \t\t\t\tdie(\"object-info: not our ref %s\",\n \t\t\t\t\tobject_info_values.items[0].string);\n-\t\t\t*(*object_info_data)[i].sizep = strtoul(object_info_values.items[size_index].string, NULL, 10);\n+\t\t\t*(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10);\n \t\t}\n \t\ti++;\n \t}\n@@ -441,7 +442,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tstruct ref *refs = NULL;\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n-\tstruct ref *wanted_refs = xcalloc(1, sizeof (struct ref));\n+\tstruct ref *object_info_refs = xcalloc(1, sizeof (struct ref));\n \n \tmemset(&args, 0, sizeof(args));\n \targs.uploadpack = data->options.uploadpack;\n@@ -470,11 +471,13 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.object_info = transport->smart_options->object_info;\n \n \tif (transport->smart_options && transport->smart_options->object_info) {\n-\t\tstruct ref *ref = wanted_refs;\n+\t\tstruct ref *ref = object_info_refs;\n \n \t\tif (!fetch_object_info(transport, data->options.object_info_data))\n \t\t\tgoto cleanup;\n \t\targs.object_info_data = data->options.object_info_data;\n+\t\targs.quiet = 1;\n+\t\targs.no_progress = 1;\n \t\tfor (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n \t\t\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n \t\t\ttemp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i);\n@@ -482,7 +485,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\tref->next = temp_ref;\n \t\t\tref = ref->next;\n \t\t}\n-\t\ttransport->remote_refs = wanted_refs->next;\n+\t\ttransport->remote_refs = object_info_refs->next;\n \t} else if (!data->got_remote_heads) {\n \t\tint i;\n \t\tint must_list_refs = 0;\n@@ -500,7 +503,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \telse if (data->version <= protocol_v1)\n \t\tdie_if_server_options(transport);\n \n-\tif (data->options.acked_commits && !transport->smart_options->object_info) {\n+\tif (data->options.acked_commits) {\n \t\tif (data->version < protocol_v2) {\n \t\t\twarning(_(\"--negotiate-only requires protocol v2\"));\n \t\t\tret = -1;\n@@ -523,12 +526,22 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\t\t to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfiles, data->version);\n \n+\tif (args.object_info) {\n+\t\tstruct ref *ref_cpy_reader = object_info_refs->next;\n+\t\tint i = 0;\n+\t\twhile (ref_cpy_reader) {\n+\t\t\toid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args.object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE);\n+\t\t\tref_cpy_reader = ref_cpy_reader->next;\n+\t\t\ti++;\n+\t\t}\n+\t}\n+\n \tdata->got_remote_heads = 0;\n \tdata->options.self_contained_and_connected =\n \t\targs.self_contained_and_connected;\n \tdata->options.connectivity_checked = args.connectivity_checked;\n \n-\tif (!refs)\n+\tif (refs == NULL && !args.object_info)\n \t\tret = -1;\n \tif (report_unmatched_refs(to_fetch, nr_heads))\n \t\tret = -1;\n@@ -543,7 +556,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tfree_refs(refs_tmp);\n \tfree_refs(refs);\n-\tfree_refs(wanted_refs);\n+\tfree_refs(object_info_refs);\n \treturn ret;\n }\n \n"},{"id":"460170","messageId":"220729.86tu70ig47.gmgdl@evledraar.gmail.com","threadId":"57388","inReplyTo":"20220728230210.2952731-7-calvinwan@google.com","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-29T06:25:17Z","receivedAt":"2022-07-29T06:32:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 28 2022, Calvin Wan wrote:\n\n> Since the `info` command in cat-file --batch-command prints object info\n> for a given object, it is natural to add another command in cat-file\n> --batch-command to print object info for a given object from a remote.\n\nIs it ?:)\n\n> Add `remote-object-info` to cat-file --batch-command.\n\nI realize this bit of implementation changed in v4, i.e. it used to be\nin \"fetch\", and I'm happy to have it moved out of there, we don't need\nto overload it more.\n\nBut I remember thinking (and perhaps commenting on-list, I can't\nremember) that the \"object-info\" server verb was a bit odd at the time\nthat it was implemented. I understand the motivation, but surely it was\nstumbling its way towards being something more generic, i.e. being able\nto just expose cmd_cat_file() in some form.\n\nWhich is one of the goals I've had in mind with working on fixing memory\nleaks in various places, i.e. once you get common commands to clean up\nafter themselves it usually becomes to have a \"command server\".\n\nSo (and I don't mind if this is longer term, just asking), is there a\nreason for why we wouldn't want to do away with object-info and this\n\"cat-file talks to a remote\", in favor of just having support for\ninvoking arbitrary commands from a remote.\n\nOf course that set of allowed RCE commands would be zero by default, but\nif we had some way to define tha \"cat-file\" was allowed to be called,\nand only if you invoked:\n\n\tcat-file --batch=\"%(objectsize)\"\n\nOr whatever, but over the v2 protocol, wouldn't we basically have\nobject-info in a more roundabout way?\n"},{"id":"460240","messageId":"xmqqo7x77qp6.fsf@gitster.g","threadId":"57388","inReplyTo":"20220728230210.2952731-4-calvinwan@google.com","subject":"Re: [PATCH v5 3/6] protocol-caps: initialization bug fix","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-29T17:51:17Z","receivedAt":"2022-07-29T17:51:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Initialize info. If info.size bit was on due to on-stack garbage,\n> we would have given our response with \"size\" attribute prefixed,\n> even when the client side never requested it.\n>\n> Signed-off-by: Calvin Wan \n> Helped-by: Jonathan Tan \n> ---\n> protocol-caps.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n\nOK, this is new in this round and it makes sense.\n\n> diff --git a/protocol-caps.c b/protocol-caps.c\n> index bbde91810a..2ad9f45c59 100644\n> --- a/protocol-caps.c\n> +++ b/protocol-caps.c\n> @@ -77,7 +77,7 @@ static void send_info(struct repository *r, struct packet_writer *writer,\n> \n> int cap_object_info(struct repository *r, struct packet_reader *request)\n> {\n> -\tstruct requested_info info;\n> +\tstruct requested_info info = { 0 };\n> \tstruct packet_writer writer;\n> \tstruct string_list oid_str_list = STRING_LIST_INIT_DUP;\n"},{"id":"460241","messageId":"xmqqk07v7qe4.fsf@gitster.g","threadId":"57388","inReplyTo":"20220728230210.2952731-5-calvinwan@google.com","subject":"Re: [PATCH v5 4/6] serve: advertise object-info feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-29T17:57:55Z","receivedAt":"2022-07-29T17:58:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> In order for a client to know what object-info components a server can\n> provide, advertise supported object-info features. This will allow a\n> client to decide whether to query the server for object-info or fetch as\n> a fallback.\n\nOK. Now we will no longer advertise a bare \"object-info\", but\n\"object-info=size\" (and possibly in the future things other than\n\"size\"). How would this change affect older clients who knows what\nto do with \"object-info\" but not \"object-info=\" yet?\n\n> +static int object_info_advertise(struct repository *r,\n> +\t\t\t\t struct strbuf *value)\n> +{\n> +\tif (value)\n> +\t\tstrbuf_addstr(value, \"size\");\n> +\treturn 1;\n> +}\n> +\n> static void session_id_receive(struct repository *r,\n> \t\t\t const char *client_sid)\n> {\n> @@ -132,7 +140,7 @@ static struct protocol_capability capabilities[] = {\n> \t},\n> \t{\n> \t\t.name = \"object-info\",\n> -\t\t.advertise = always_advertise,\n> +\t\t.advertise = object_info_advertise,\n> \t\t.command = cap_object_info,\n> \t},\n> };\n> diff --git a/t/t5555-http-smart-common.sh b/t/t5555-http-smart-common.sh\n> index b1cfe8b7db..5a16d4259a 100755\n> --- a/t/t5555-http-smart-common.sh\n> +++ b/t/t5555-http-smart-common.sh\n> @@ -131,7 +131,7 @@ test_expect_success 'git upload-pack --advertise-refs: v2' '\n> \tfetch=shallow wait-for-done\n> \tserver-option\n> \tobject-format=$(test_oid algo)\n> -\tobject-info\n> +\tobject-info=size\n> \t0000\n> \tEOF\n> \n> diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\n> index 1896f671cb..ebb32644e3 100755\n> --- a/t/t5701-git-serve.sh\n> +++ b/t/t5701-git-serve.sh\n> @@ -20,7 +20,7 @@ test_expect_success 'test capability advertisement' '\n> \tfetch=shallow wait-for-done\n> \tserver-option\n> \tobject-format=$(test_oid algo)\n> -\tobject-info\n> +\tobject-info=size\n> \t0000\n> \tEOF\n"},{"id":"460243","messageId":"xmqqbkt77q07.fsf@gitster.g","threadId":"57388","inReplyTo":"20220728230210.2952731-6-calvinwan@google.com","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-29T18:06:16Z","receivedAt":"2022-07-29T18:06:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> +void send_object_info_request(int fd_out, struct object_info_args *args)\n> +{\n> +\tstruct strbuf req_buf = STRBUF_INIT;\n> +\tsize_t i;\n> +\n> +\twrite_command_and_capabilities(&req_buf, args->server_options, \"object-info\");\n> +\n> +\tif (unsorted_string_list_has_string(args->object_info_options, \"size\"))\n> +\t\tpacket_buf_write(&req_buf, \"size\");\n> +\n> +\tif (unsorted_string_list_has_string(args->object_info_options, \"type\"))\n> +\t\tpacket_buf_write(&req_buf, \"type\");\n> +\n> +\tif (args->oids) {\n> +\t\tfor (i = 0; i < args->oids->nr; i++)\n> +\t\t\tpacket_buf_write(&req_buf, \"oid %s\", oid_to_hex(&args->oids->oid[i]));\n> +\t}\n\nIf !args->oids then we say \"we want to request object-info to learn\nsize and type for the following objects: oh, there are no objects we\nare interested in\". I wonder if an early return\n\n\tif (!args->oids)\n\t\treturn;\n\nat the beginning of the function that turns it into a benign no-op,\nmay make more sense? Calling \"send_X()\" helper and seeing nothing\ncome out on the wire might make it look awkward, though.\n\n> @@ -363,10 +437,12 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \t\t\t int nr_heads, struct ref **to_fetch)\n> {\n> \tint ret = 0;\n> +\tsize_t i;\n> \tstruct git_transport_data *data = transport->data;\n> \tstruct ref *refs = NULL;\n> \tstruct fetch_pack_args args;\n> \tstruct ref *refs_tmp = NULL;\n> +\tstruct ref *object_info_refs = xcalloc(1, sizeof (struct ref));\n\nStyle: no SP between \"sizeof\" and \"(\".\n"},{"id":"460244","messageId":"xmqq1qu37pbf.fsf@gitster.g","threadId":"57388","inReplyTo":"20220728230210.2952731-7-calvinwan@google.com","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-29T18:21:08Z","receivedAt":"2022-07-29T18:21:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> While `info` takes object ids one at a time, this creates overhead when\n> making requests to a server so `remote-object-info` instead can take\n> multiple object ids at once.\n\nI am not sure if the above is desirable---doesn't it make the\nprotocol (i.e. the requests the invoker of \"cat-file\" needs to\nprepare) unnecessarily complex? If we have \"buffered\" mode, the\nimplementation can collect and batch requests and talk with the\nother side in a single communication with the other end, no?\n\n> In --buffer mode, this changes to:\n>\n> - Receive and parse input from user\n> - Store respective function attached to command in a queue\n> - After flush, loop through commands in queue\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n\nSo, the existing buffered mode is poorly designed, in other words.\nIsn't this something we can fix for performance without affecting\nthe existing callers?\n"},{"id":"460325","messageId":"8a1ff659-2086-f3cf-469a-76f5e2bc60e3@gmail.com","threadId":"57388","inReplyTo":"20220728230210.2952731-1-calvinwan@google.com","subject":"Re: [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-07-31T08:41:38Z","receivedAt":"2022-07-31T08:41:44Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 29/07/2022 00:02, Calvin Wan wrote:\n> Sometimes it is useful to get information about an object without having\n> to download it completely. The server logic has already been implemented\n> as “a2ba162cda (object-info: support for retrieving object info,\n> 2021-04-20)”. This patch implements the client option for it.\n> \n> Add `--object-info` option to `cat-file --batch-command`. \n\nThis part of the cover letter seems to have been left over from a \nprevious iteration. I noticed that in patch 6 there is a reference to \n'--object-info' in an error message as well.\n\nThis might be a daft question but could we teach cat-file to check if an \nobject is available locally and if it is not retrieve just the metadata \nfrom the remote rather than adding a new command to --batch-command? \nThat way it would be transparent to the user and they would not have to \nworry about whether they might be querying a remote object or not.\n\nBest Wishes\n\nPhillip\n\n> This option\n> allows the client to make an object-info command request to a server\n> that supports protocol v2. If the server is v2, but does not allow for\n> the object-info command request, the entire object is fetched and the\n> relevant object info is returned.\n> \n> === Changes since v4 ===\n> - Instead of making 2 rounds trips to determine whether the server can\n> handle the client object-info request, the server now advertises the\n> features it can send back so the client can immediately fallback to\n> fetch if any feature is not supported\n> - Fixed nits and clarified some documentation/commit messages\n> \n> Signed-off-by: Calvin Wan \n> Helped-by: Jonathan Tan \n> \n> Calvin Wan (6):\n> fetch-pack: refactor packet writing\n> fetch-pack: move fetch initialization\n> protocol-caps: initialization bug fix\n> serve: advertise object-info feature\n> transport: add client support for object-info\n> cat-file: add remote-object-info to batch-command\n> \n> Documentation/git-cat-file.txt | 16 +-\n> builtin/cat-file.c | 225 ++++++++++++++++-----\n> fetch-pack.c | 52 +++--\n> fetch-pack.h | 10 +\n> object-file.c | 11 ++\n> object-store.h | 3 +\n> protocol-caps.c | 2 +-\n> serve.c | 10 +-\n> t/t1006-cat-file.sh | 348 +++++++++++++++++++++++++++++++++\n> t/t5555-http-smart-common.sh | 2 +-\n> t/t5701-git-serve.sh | 2 +-\n> transport-helper.c | 7 +-\n> transport.c | 110 ++++++++++-\n> transport.h | 11 ++\n> 14 files changed, 734 insertions(+), 75 deletions(-)\n> \n> \n> base-commit: 9dd64cb4d310986dd7b8ca7fff92f9b61e0bd21a\n"},{"id":"460326","messageId":"CAOLTT8Q-241cExFcbXSUfj42GontKH0uQpX1Voekf4RJ95hAVw@mail.gmail.com","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"Re: [PATCH v4 0/8] cat-file: add --batch-command remote-object-info command","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-07-31T15:02:41Z","receivedAt":"2022-07-31T15:02:56Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hi, Calvin Wan,\n\nCalvin Wan 于2022年5月3日周二 08:14写道：\n>\n> Sometimes it is useful to get information about an object without having\n> to download it completely. The server logic has already been implemented\n> as “a2ba162cda (object-info: support for retrieving object info,\n> 2021-04-20)”. This patch implements the client option for it.\n>\n> Add `--object-info` option to `cat-file --batch-command`. This option\n> allows the client to make an object-info command request to a server\n> that supports protocol v2. If the server is v2, but does not allow for\n> the object-info command request, the entire object is fetched and the\n> relevant object info is returned.\n>\n> Summary of changes ==================\n>\n> Patches 1, 2, 3, and 7 are small changes that setup the main\n> implementation. Patch 4 sets up object-info to be backwards compatible\n> for future patch series that adds additional attributes. Patch 5 adds\n> internal trasnport functions to send and receive object-info command\n> request packets. Patch 6 adds the fallback if object-info is not\n> supported or fails. Patch 8 adds the cat-file implementation.\n>\n\nI have to say I am very curious about this feature. Since the current\ngit partial-clone interface supports only a few filters:\n\nblob:limit\nblob:none\nsparse:oid\ntree:0\n\nThough these filters reduce the number of objects downloaded each time,\nsometimes I just need *only* one blob object, git partial-clone will still\ndownload some additional commits and tree objects from the remote.\n\nThis patch can get the remote-object-info by git cat-file, so can we go\nfurther to get the remote-object-content of the object?\n\nSomething like:\n\n$ git cat-file --batch-command\nremote-object-content origin \n\n> Changes since V3 ================\n>\n> * Object-info is now implemented in cat-file --batch-command rather\n> than fetch (new base commit)\n> * Removed config option to advertise object-info\n> * Added forwards and backwards compability for object-info\n> * Split up some patches to better describe and visualize changes\n>\n> Signed-off-by: Calvin Wan \n> Helped-by: Jonathan Tan \n>\n> Calvin Wan (8):\n> fetch-pack: refactor packet writing\n> fetch-pack: move fetch default settings\n> object-store: add function to free object_info contents\n> object-info: send attribute packet regardless of object ids\n> transport: add client side capability to request object-info\n> transport: add object-info fallback to fetch\n> cat-file: move parse_cmd and DEFAULT_FORMAT up\n> cat-file: add --batch-command remote-object-info command\n>\n> Documentation/git-cat-file.txt | 16 +-\n> builtin/cat-file.c | 225 ++++++++++++++++-----\n> fetch-pack.c | 61 ++++--\n> fetch-pack.h | 10 +\n> object-file.c | 16 ++\n> object-store.h | 3 +\n> protocol-caps.c | 14 +-\n> t/t1006-cat-file.sh | 347 +++++++++++++++++++++++++++++++++\n> transport-helper.c | 7 +-\n> transport.c | 97 ++++++++-\n> transport.h | 11 ++\n> 11 files changed, 728 insertions(+), 79 deletions(-)\n>\n> --\n> 2.36.0.rc2.10170.gb555eefa6f\n>\n\nThanks!\n\nZheNing Hu\n"},{"id":"460327","messageId":"CAOLTT8RyDCokvJBaC1CCoT2+3AE1kOn7evjkTMK_V6KWoaH30g@mail.gmail.com","threadId":"57388","inReplyTo":"20220502170904.2770649-9-calvinwan@google.com","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-07-31T15:24:07Z","receivedAt":"2022-07-31T15:24:24Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Calvin Wan 于2022年5月3日周二 06:33写道：\n>\n> Since the `info` command in cat-file --batch-command prints object info\n> for a given object, it is natural to add another command in cat-file\n> --batch-command to print object info for a given object from a remote.\n> Add `remote-object-info` to cat-file --batch-command.\n>\n> While `info` takes object ids one at a time, this creates overhead when\n> making requests to a server so `remote-object-info` instead can take\n> multiple object ids at once.\n>\n> cat-file --batch-command is generally implemented in the following\n> manner:\n>\n> - Receive and parse input from user\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n>\n> In --buffer mode, this changes to:\n>\n> - Receive and parse input from user\n> - Store respective function attached to command in a queue\n> - After flush, loop through commands in queue\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n>\n> Notice how the getting and printing of object info is accomplished one\n> at a time. As described above, this creates a problem for making\n> requests to a server. Therefore, `remote-object-info` is implemented in\n> the following manner:\n>\n> - Receive and parse input from user\n> If command is `remote-object-info`:\n> - Get object info from remote\n> - Loop through object info\n> - Call respective function attached to `info`\n> - Set batch mode state, use passed in object info, print object\n> info\n> Else:\n> - Call respective function attached to command\n> - Parse input, get object info, print object info\n>\n> And finally for --buffer mode `remote-object-info`:\n> - Receive and parse input from user\n> - Store respective function attached to command in a queue\n> - After flush, loop through commands in queue:\n> If command is `remote-object-info`:\n> - Get object info from remote\n> - Loop through object info\n> - Call respective function attached to `info`\n> - Set batch mode state, use passed in object info, print\n> object info\n> Else:\n> - Call respective function attached to command\n> - Set batch mode state, get object info, print object info\n>\n> To summarize, `remote-object-info` gets object info from the remote and\n> then generates multiples `info` commands with the object info passed in.\n> It cannot be implemented similarly to `info` and `content` because of\n> the overhead with remote commenication.\n>\n> ---\n> Documentation/git-cat-file.txt | 16 +-\n> builtin/cat-file.c | 200 +++++++++++++++----\n> t/t1006-cat-file.sh | 347 +++++++++++++++++++++++++++++++++\n> 3 files changed, 522 insertions(+), 41 deletions(-)\n>\n> diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt\n> index 24a811f0ef..0d9e8e6214 100644\n> --- a/Documentation/git-cat-file.txt\n> +++ b/Documentation/git-cat-file.txt\n> @@ -115,6 +115,10 @@ info ::\n> Print object info for object reference ``. This corresponds to the\n> output of `--batch-check`.\n>\n> +remote-object-info []::\n> + Print object info for object references `` at specified .\n> + This command may only be combined with `--buffer`.\n> +\n> flush::\n> Used with `--buffer` to execute all preceding commands that were issued\n> since the beginning or since the last flush was issued. When `--buffer`\n> @@ -245,7 +249,8 @@ newline. The available atoms are:\n> The full hex representation of the object name.\n>\n> `objecttype`::\n> - The type of the object (the same as `cat-file -t` reports).\n> + The type of the object (the same as `cat-file -t` reports). See\n> + `CAVEATS` below.\n>\n> `objectsize`::\n> The size, in bytes, of the object (the same as `cat-file -s`\n> @@ -253,13 +258,14 @@ newline. The available atoms are:\n>\n> `objectsize:disk`::\n> The size, in bytes, that the object takes up on disk. See the\n> - note about on-disk sizes in the `CAVEATS` section below.\n> + note about on-disk sizes in the `CAVEATS` section below. Not\n> + supported by `remote-object-info`.\n>\n> `deltabase`::\n> If the object is stored as a delta on-disk, this expands to the\n> full hex representation of the delta base object name.\n> Otherwise, expands to the null OID (all zeroes). See `CAVEATS`\n> - below.\n> + below. Not supported by `remote-object-info`.\n>\n> `rest`::\n> If this atom is used in the output string, input lines are split\n\nWhy not forbidden %(rest) here too?\n\n> +static int get_remote_info(struct batch_options *opt, int argc, const char **argv)\n> +{\n> + int retval = 0;\n> + size_t i;\n> + struct remote *remote = NULL;\n> + struct object_id oid;\n> + struct string_list object_info_options = STRING_LIST_INIT_NODUP;\n> + static struct transport *gtransport;\n> + char *format = DEFAULT_FORMAT;\n> +\n> + if (opt->format)\n> + format = xstrdup(opt->format);\n> +\n> + remote = remote_get(argv[0]);\n> + if (!remote)\n> + die(_(\"must supply valid remote when using --object-info\"));\n> + oid_array_clear(&object_info_oids);\n> + for (i = 1; i < argc; i++) {\n> + if (get_oid(argv[i], &oid))\n> + die(_(\"malformed object id '%s'\"), argv[i]);\n> + oid_array_append(&object_info_oids, &oid);\n> + }\n> +\n> + gtransport = transport_get(remote, NULL);\n> + if (gtransport->smart_options) {\n> + size_t j;\n> + int include_size = 0, include_type = 0;\n> +\n> + remote_object_info = xcalloc(object_info_oids.nr, sizeof(struct object_info));\n> + gtransport->smart_options->object_info = 1;\n> + gtransport->smart_options->object_info_oids = &object_info_oids;\n> + /**\n> + * 'size' is the only option currently supported.\n> + * Other options that are passed in the format will default to a\n> + * standard fetch request rather than object-info.\n> + */\n> + if (strstr(format, \"%(objectsize)\")) {\n> + string_list_append(&object_info_options, \"size\");\n> + include_size = 1;\n> + }\n> + if (strstr(format, \"%(objecttype)\")) {\n> + string_list_append(&object_info_options, \"type\");\n> + include_type = 1;\n> + }\n> + if (strstr(format, \"%(objectsize:disk)\"))\n> + die(_(\"objectsize:disk is currently not supported with remote-object-info\"));\n> + if (strstr(format, \"%(deltabase)\"))\n> + die(_(\"deltabase is currently not supported with remote-object-info\"));\n\n%(rest) too?\n\n> + if (object_info_options.nr > 0) {\n> + gtransport->smart_options->object_info_options = &object_info_options;\n> + for (j = 0; j < object_info_oids.nr; j++) {\n> + if (include_size)\n> + remote_object_info[j].sizep = xcalloc(1, sizeof(long));\n> + if (include_type)\n> + remote_object_info[j].typep = xcalloc(1, sizeof(enum object_type));\n> + }\n> + gtransport->smart_options->object_info_data = &remote_object_info;\n> + retval = transport_fetch_refs(gtransport, NULL);\n> + }\n> + } else {\n> + retval = -1;\n> + }\n> + return retval;\n> +}\n\nMaybe such code style will be better?\n\n if (!gtransport->smart_options) {\n return -1;\n }\n ...\n return transport_fetch_refs(gtransport, NULL);\n\n> +\n> struct object_cb_data {\n> struct batch_options *opt;\n> struct expand_data *expand;\n> @@ -538,6 +611,7 @@ typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n> struct queued_cmd {\n> parse_cmd_fn_t fn;\n> char *line;\n> + const char *name;\n> };\n>\n> static void parse_cmd_contents(struct batch_options *opt,\n> @@ -565,9 +639,49 @@ static const struct parse_cmd {\n> } commands[] = {\n> { \"contents\", parse_cmd_contents, 1},\n> { \"info\", parse_cmd_info, 1},\n> + { \"remote-object-info\", parse_cmd_info, 1},\n> { \"flush\", NULL, 0},\n> };\n>\n> +static void parse_remote_info(struct batch_options *opt,\n> + char *line,\n> + struct strbuf *output,\n> + struct expand_data *data,\n> + const struct parse_cmd *p_cmd,\n> + struct queued_cmd *q_cmd)\n> +{\n> + int count;\n> + size_t i;\n> + const char **argv;\n> +\n> + count = split_cmdline(line, &argv);\n> + if (get_remote_info(opt, count, argv))\n> + goto cleanup;\n> + opt->use_remote_info = 1;\n> + data->skip_object_info = 1;\n> + data->mark_query = 0;\n> + for (i = 0; i < object_info_oids.nr; i++) {\n> + if (remote_object_info[i].sizep)\n> + data->size = *remote_object_info[i].sizep;\n> + if (remote_object_info[i].typep)\n> + data->type = *remote_object_info[i].typep;\n> +\n> + data->oid = object_info_oids.oid[i];\n> + if (p_cmd)\n> + p_cmd->fn(opt, argv[i+1], output, data);\n> + else\n> + q_cmd->fn(opt, argv[i+1], output, data);\n> + }\n> + opt->use_remote_info = 0;\n> + data->skip_object_info = 0;\n> + data->mark_query = 1;\n> +\n> +cleanup:\n> + for (i = 0; i < object_info_oids.nr; i++)\n> + free_object_info_contents(&remote_object_info[i]);\n> + free(remote_object_info);\n> +}\n> +\n> static void dispatch_calls(struct batch_options *opt,\n> struct strbuf *output,\n> struct expand_data *data,\n> @@ -579,8 +693,12 @@ static void dispatch_calls(struct batch_options *opt,\n> if (!opt->buffer_output)\n> die(_(\"flush is only for --buffer mode\"));\n>\n> - for (i = 0; i < nr; i++)\n> - cmd[i].fn(opt, cmd[i].line, output, data);\n> + for (i = 0; i < nr; i++) {\n> + if (!strcmp(cmd[i].name, \"remote-object-info\"))\n> + parse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]);\n> + else\n> + cmd[i].fn(opt, cmd[i].line, output, data);\n> + }\n>\n> fflush(stdout);\n> }\n> @@ -640,11 +758,17 @@ static void batch_objects_command(struct batch_options *opt,\n> dispatch_calls(opt, output, data, queued_cmd, nr);\n> free_cmds(queued_cmd, &nr);\n> } else if (!opt->buffer_output) {\n> - cmd->fn(opt, p, output, data);\n> + if (!strcmp(cmd->name, \"remote-object-info\")) {\n> + char *line = xstrdup_or_null(p);\n> + parse_remote_info(opt, line, output, data, cmd, NULL);\n\nmemory leak: \"line\".\n\n> + } else {\n> + cmd->fn(opt, p, output, data);\n> + }\n> } else {\n> ALLOC_GROW(queued_cmd, nr + 1, alloc);\n> call.fn = cmd->fn;\n> call.line = xstrdup_or_null(p);\n> + call.name = cmd->name;\n> queued_cmd[nr++] = call;\n> }\n> }\n> +\n> +test_expect_success 'remote-object-info fails on unspported filter option (objectsize:disk)' '\n> + (\n> + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\n> + test_must_fail git cat-file --batch-command=\"%(objectsize:disk)\" 2>err <<-EOF &&\n> + remote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n> + EOF\n> + test_i18ngrep \"objectsize:disk is currently not supported with remote-object-info\" err\n> + )\n> +'\n> +\n> +test_expect_success 'remote-object-info fails on unspported filter option (deltabase)' '\n> + (\n> + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\n> + test_must_fail git cat-file --batch-command=\"%(deltabase)\" 2>err <<-EOF &&\n> + remote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n> + EOF\n> + test_i18ngrep \"deltabase is currently not supported with remote-object-info\" err\n> + )\n> +'\n> +\n\n%(rest) too?\n\n> +set_transport_variables \"server\"\n> +\n> +test_expect_success 'batch-command remote-object-info file://' '\n> + (\n> + cd server &&\n> +\n> + echo \"$hello_sha1 $hello_size\" >expect &&\n> + echo \"$tree_sha1 $tree_size\" >>expect &&\n> + echo \"$commit_sha1 $commit_size\" >>expect &&\n> + echo \"$tag_sha1 $tag_size\" >>expect &&\n> + git cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n> + remote-object-info \"file://$(pwd)\" $hello_sha1\n> + remote-object-info \"file://$(pwd)\" $tree_sha1\n> + remote-object-info \"file://$(pwd)\" $commit_sha1\n> + remote-object-info \"file://$(pwd)\" $tag_sha1\n> + EOF\n> + test_cmp expect actual\n> + )\n\nCan we support instead of only here?\n\n$ git cat-file --batch-check\nHEAD\n28583b8d8ca72730d7c9e0ea50861ad431a6dea4 commit 3038\nmaster\nab336e8f1c8009c8b1aab8deb592148e69217085 commit 281\norigin/master\n23b219f8e3f2adfb0441e135f0a880e6124f766c commit 282\norigin/master:git.c\ne5d62fa5a92e95e1ede041ebf913d841744c31f8 blob 28398\n\nSo cat-file --batch-check can support it.\n\n$git cat-file --batch-commands\nremote-object-info \"file://$(pwd)\" master:git.c\n\nI guess it cannot work now, right?\n"},{"id":"460347","messageId":"b33ec478-8858-faa7-2678-f02559426539@gmail.com","threadId":"57388","inReplyTo":"20220728230210.2952731-6-calvinwan@google.com","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-08-01T13:30:00Z","receivedAt":"2022-08-01T13:30:09Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 29/07/2022 00:02, Calvin Wan wrote:\n> Sometimes it is useful to get information about an object without having\n> to download it completely. The server logic has already been implemented\n> as “a2ba162cda (object-info: support for retrieving object info,\n> 2021-04-20)”. This patch adds client functions to communicate with the\n> server.\n> \n> The client currently supports requesting a list of object ids with\n> features 'size' and 'type' from a v2 server. If a server does not\n> advertise either of the requested features, then the client falls back\n> to making the request through 'fetch'.\n\nI'm confused by the 'type' support, I might have missed something as I'm \nnot familiar with this code but I couldn't see where we parse the type \nreturned by the server.\n\n> +\tfor (i = 0; i < args.object_info_options->nr; i++) {\n> +\t\tif (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> +\t\t\tcheck_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n\nThis is one of a number of lines in this series that are way over the 80 \ncolumn limit.\n\n> +\t\t\treturn -1;\n> +\t\t}\n> +\t\tif (unsorted_string_list_has_string(args.object_info_options, reader.line)) {\n> +\t\t\tif (!strcmp(reader.line, \"size\"))\n> +\t\t\t\tsize_index = i;\n\nShould we be checking for \"type\" as well? Also does protocol-v2.txt need \nupdating as it only mentions \"size\" as an attribute.\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\treturn -1;\n> +\t}\n> +\n> +\ti = 0;\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL && i < args.oids->nr) {\n> +\t\tstruct string_list object_info_values = STRING_LIST_INIT_DUP;\n> +\n> +\t\tstring_list_split(&object_info_values, reader.line, ' ', -1);\n> +\t\tif (0 <= size_index) {\n\nTo avoid a possible out-of-bounds access we need to check that \nsize_index + 1 < object_info_value.nr in case the server response is \nmalformed\n\n> +\t\t\tif (!strcmp(object_info_values.items[1 + size_index].string, \"\"))\n> +\t\t\t\tdie(\"object-info: not our ref %s\",\n\nI'm a bit confused by this message is it trying to say \"object %s is \nmissing on the server\"?\n\n> +\t\t\t\t\tobject_info_values.items[0].string);\n> +\t\t\t*(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10);\n\nAs Junio pointed out in his comments in v4 there is no error checking \nhere - we should check the server has actually returned a number. Note \nthat strtoul() will happily parse negative numbers so we probably want \nto do something like\n\nconst char *endp\nerrno = 0\nif (!isdigit(*object_info_values.items[1 + size_index].string))\n die(\"...\")\n*(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + \nsize_index].string, &endp, 10);\nif (errno || *endp)\n die(\"...\")\n\nShould be we checking the object id matches what we asked for? (I'm not \nsure if protocol-v2.txt mentions the order in which the objects are \nreturned)\n\nShould we be parsing the object type here as well?\n\n> @@ -392,8 +468,25 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \targs.server_options = transport->server_options;\n> \targs.negotiation_tips = data->options.negotiation_tips;\n> \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n> -\n> -\tif (!data->got_remote_heads) {\n> +\targs.object_info = transport->smart_options->object_info;\n> +\n> +\tif (transport->smart_options && transport->smart_options->object_info) {\n> +\t\tstruct ref *ref = object_info_refs;\n> +\n> +\t\tif (!fetch_object_info(transport, data->options.object_info_data))\n> +\t\t\tgoto cleanup;\n> +\t\targs.object_info_data = data->options.object_info_data;\n> +\t\targs.quiet = 1;\n> +\t\targs.no_progress = 1;\n> +\t\tfor (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n> +\t\t\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n\nUsing CALLOC_ARRAY() or p = xcalloc(1, sizeof(*p)) would be safer here \n(and everywhere else where you have used xcalloc()) as it ensures we \nallocate the correct size.\n\n> diff --git a/transport.h b/transport.h\n> index b5bf7b3e70..5512fdb140 100644\n> --- a/transport.h\n> +++ b/transport.h\n> @@ -31,6 +32,12 @@ struct git_transport_options {\n> \t */\n> \tunsigned connectivity_checked:1;\n> \n> +\t/*\n> +\t * Transport will attempt to pull only object-info. Fallbacks\n> +\t * to pulling entire object if object-info is not supported\n> +\t */\n\nIs it definitely true that we fallback to pulling the entire object? - \nthere is at least one place above where we do\n\n > +\tif (transport->smart_options->object_info) {\n > +\t\tdie(_(\"--object-info requires protocol v2\"));\n > +\t}\n >\n\nBest Wishes\n\nPhillip\n"},{"id":"460354","messageId":"aef199d2-7c89-1e25-4aae-e1e43a71e04f@gmail.com","threadId":"57388","inReplyTo":"20220728230210.2952731-6-calvinwan@google.com","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-08-01T14:26:27Z","receivedAt":"2022-08-01T14:26:43Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 29/07/2022 00:02, Calvin Wan wrote:\n> diff --git a/transport.c b/transport.c\n> index 52db7a3cb0..2d503e2fbd 100644\n> --- a/transport.c\n> +++ b/transport.c\n\n> +\ti = 0;\n> +\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL && i < args.oids->nr) {\n> +\t\tstruct string_list object_info_values = STRING_LIST_INIT_DUP;\n\nI forget to say earlier that this is leaked\n\nBest Wishes\n\nPhillip\n"},{"id":"460367","messageId":"CAFySSZAPzO9YzGhsjhG7YRXv+ctTHM2FXzrwRW2bTW4pUt8q7g@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqk07v7qe4.fsf@gitster.g","subject":"Re: [PATCH v5 4/6] serve: advertise object-info feature","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-01T18:28:37Z","receivedAt":"2022-08-01T18:28:53Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> OK. Now we will no longer advertise a bare \"object-info\", but\n> \"object-info=size\" (and possibly in the future things other than\n> \"size\"). How would this change affect older clients who knows what\n> to do with \"object-info\" but not \"object-info=\" yet?\n\nThis was a tricky tradeoff that I definitely think I should have\ndiscussed more in the commit message. The issue with how object\ninfo is currently implemented is that it is very inflexible for adding\nnew parameters.\n\nThis is how object-info currently parses a client request:\n\nwhile (packet_reader_read(request) == PACKET_READ_NORMAL) {\n if (!strcmp(\"size\", request->line)) {\n info.size = 1;\n continue;\n }\n\n if (parse_oid(request->line, &oid_str_list))\n continue;\n\n packet_writer_error(&writer,\n \"object-info: unexpected line: '%s'\",\n request->line);\n}\n\nObject-info supports \"size\" right now but, let's say I want to add\n\"type\" as a parameter. OK I add another if statement like:\n\nif (!strcmp(\"type\", request->line)) {\n info.type = 1;\n continue;\n}\n\nAnd we update the docs to say \"type\" is now supported by\nobject-info. If a user now attempts to request \"size\" and \"type\"\nfrom a server that has not been updated to support \"type\",\nthen the user gets an error message \"object-info: unexpected\nline: 'type'\", which is another situation that is a bad experience\nfor older clients. The client has no way of knowing that their\nfailure is caused by a server version issue.\n\nEssentially I think at some point we have to bite the bullet and say\nwe need to rework some part of the object-info advertisement (or\nif anyone has a better idea of achieving the same goal) so that we\ncan make future incremental changes to object-info. If the supported\nparameters are posted in the advertisement, then the client doesn't\nhave to first make a request to find out that their requested\nparameter isn't support by the server. While you noted that we can't\nmake the assumption now that nobody is using the current\nobject-info feature, I think the benefit of the change outweighs\nthe cost of affecting the possibly small amount of users of this\nfeature. (A quick search on stack overflow for \"object-info\" tagged\nunder [git] returned no questions about it so that's what I used as\na cursory estimate for how popular this feature is).\n\nCurious to hear what your thoughts on this are Junio, since as\nmuch as I'd like to create a seamless upgrade experience for\nolder clients, I'm out of ideas as to how I would do so.\n"},{"id":"460369","messageId":"220801.86mtcnhkcw.gmgdl@evledraar.gmail.com","threadId":"57388","inReplyTo":"CAFySSZAPzO9YzGhsjhG7YRXv+ctTHM2FXzrwRW2bTW4pUt8q7g@mail.gmail.com","subject":"Re: [PATCH v5 4/6] serve: advertise object-info feature","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-01T18:44:57Z","receivedAt":"2022-08-01T18:47:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 01 2022, Calvin Wan wrote:\n\n>> OK. Now we will no longer advertise a bare \"object-info\", but\n>> \"object-info=size\" (and possibly in the future things other than\n>> \"size\"). How would this change affect older clients who knows what\n>> to do with \"object-info\" but not \"object-info=\" yet?\n>\n> This was a tricky tradeoff that I definitely think I should have\n> discussed more in the commit message. The issue with how object\n> info is currently implemented is that it is very inflexible for adding\n> new parameters.\n>\n> This is how object-info currently parses a client request:\n>\n> while (packet_reader_read(request) == PACKET_READ_NORMAL) {\n> if (!strcmp(\"size\", request->line)) {\n> info.size = 1;\n> continue;\n> }\n>\n> if (parse_oid(request->line, &oid_str_list))\n> continue;\n>\n> packet_writer_error(&writer,\n> \"object-info: unexpected line: '%s'\",\n> request->line);\n> }\n>\n> Object-info supports \"size\" right now but, let's say I want to add\n> \"type\" as a parameter. OK I add another if statement like:\n>\n> if (!strcmp(\"type\", request->line)) {\n> info.type = 1;\n> continue;\n> }\n>\n> And we update the docs to say \"type\" is now supported by\n> object-info. If a user now attempts to request \"size\" and \"type\"\n> from a server that has not been updated to support \"type\",\n> then the user gets an error message \"object-info: unexpected\n> line: 'type'\", which is another situation that is a bad experience\n> for older clients. The client has no way of knowing that their\n> failure is caused by a server version issue.\n>\n> Essentially I think at some point we have to bite the bullet and say\n> we need to rework some part of the object-info advertisement (or\n> if anyone has a better idea of achieving the same goal) so that we\n> can make future incremental changes to object-info. If the supported\n> parameters are posted in the advertisement, then the client doesn't\n> have to first make a request to find out that their requested\n> parameter isn't support by the server. While you noted that we can't\n> make the assumption now that nobody is using the current\n> object-info feature, I think the benefit of the change outweighs\n> the cost of affecting the possibly small amount of users of this\n> feature. (A quick search on stack overflow for \"object-info\" tagged\n> under [git] returned no questions about it so that's what I used as\n> a cursory estimate for how popular this feature is).\n>\n> Curious to hear what your thoughts on this are Junio, since as\n> much as I'd like to create a seamless upgrade experience for\n> older clients, I'm out of ideas as to how I would do so.\n\nI haven't looked deeply into this case, but in general for such protcol\nincompatibilities we could just create an object-info2, and have that\nuse some extensible calling convention we wish we'd have used from day\n1.\n\nThen have new clients understand both (and prefer the new verb), and\nolder clients use object-info without breakage.\n\nOr we could call the new thing \"cat-file\", and have it accept any\narbitrary options it does, and then limit it to some sensible subset for\nnow :)\n"},{"id":"460370","messageId":"xmqqtu6vyf5n.fsf@gitster.g","threadId":"57388","inReplyTo":"CAFySSZAPzO9YzGhsjhG7YRXv+ctTHM2FXzrwRW2bTW4pUt8q7g@mail.gmail.com","subject":"Re: [PATCH v5 4/6] serve: advertise object-info feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-01T18:47:32Z","receivedAt":"2022-08-01T18:47:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> And we update the docs to say \"type\" is now supported by\n> object-info. If a user now attempts to request \"size\" and \"type\"\n> from a server that has not been updated to support \"type\",\n> then the user gets an error message \"object-info: unexpected\n> line: 'type'\", which is another situation that is a bad experience\n> for older clients.\n\nIs it? older clients by definition does not know about \"type\", no?\n\nI am perfectly happy with the capability advertisement to say not\njust \"object-info\", but also what attributes of objects you can be\nqueried, by the way. If clients right now (i.e. without this\nseries) expect that the other side who advertises \"object-info\"\nwithout \"=,,,\" supports only\n\"size\", for example, perhaps treat the bare \"object-info\" capability\njust as a synonym of \"object-info=size\", or something?\n\nI do not deeply care either way, to be honest, as the deployed\nclients with bare \"object-info\" will not survive for a long time\nand will quickly be deprecated anyway. I just wanted to see the\nreasoning behind the decision to ignore them.\n\nThanks.\n\n"},{"id":"460372","messageId":"CAFySSZC7+j-35TD9jy5y7RW8zmQi+mAZEdwdvhuXwN9-XvZQ-w@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqtu6vyf5n.fsf@gitster.g","subject":"Re: [PATCH v5 4/6] serve: advertise object-info feature","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-01T18:58:32Z","receivedAt":"2022-08-01T18:58:46Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> Is it? older clients by definition does not know about \"type\", no?\n\nNewer client, older remote server was the situation I was thinking\nabout\n\n> I am perfectly happy with the capability advertisement to say not\n> just \"object-info\", but also what attributes of objects you can be\n> queried, by the way. If clients right now (i.e. without this\n> series) expect that the other side who advertises \"object-info\"\n> without \"=,,,\" supports only\n> \"size\", for example, perhaps treat the bare \"object-info\" capability\n> just as a synonym of \"object-info=size\", or something?\n\nThis is a good idea! Thanks\n"},{"id":"460662","messageId":"CAFySSZAJMUw30GCs8Etvi1D0hm3Rna2C8K_EMxKQtXea9fqVcg@mail.gmail.com","threadId":"57388","inReplyTo":"xmqqbkt77q07.fsf@gitster.g","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-04T20:28:19Z","receivedAt":"2022-08-04T20:28:34Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> If !args->oids then we say \"we want to request object-info to learn\n> size and type for the following objects: oh, there are no objects we\n> are interested in\". I wonder if an early return\n>\n> if (!args->oids)\n> return;\n>\n> at the beginning of the function that turns it into a benign no-op,\n> may make more sense? Calling \"send_X()\" helper and seeing nothing\n> come out on the wire might make it look awkward, though.\n\nIf I add this change, then I should also add a check to ensure\nargs->object_info_options is also not empty. The question I think we\nshould answer is whether checking for oids and options should\nhappen inside this function or not. I'm leaning towards outside\nthe function for the reason you stated above about nothing\ncoming out of the wire being awkward.\n\n> > + struct ref *object_info_refs = xcalloc(1, sizeof (struct ref));\n>\n> Style: no SP between \"sizeof\" and \"(\".\n\nack\n"},{"id":"460679","messageId":"CAFySSZDvgwbbHCHfyuaqX3tKsr-GjJ9iihygg6rNNe46Ys7_EA@mail.gmail.com","threadId":"57388","inReplyTo":"b33ec478-8858-faa7-2678-f02559426539@gmail.com","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-04T22:20:50Z","receivedAt":"2022-08-04T22:22:34Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> > The client currently supports requesting a list of object ids with\n> > features 'size' and 'type' from a v2 server. If a server does not\n> > advertise either of the requested features, then the client falls back\n> > to making the request through 'fetch'.\n>\n> I'm confused by the 'type' support, I might have missed something as I'm\n> not familiar with this code but I couldn't see where we parse the type\n> returned by the server.\n\nI should clarify that the server does not support 'type', only the client.\nSince the client falls back to 'fetch' to grab the object info if the server\ndoesn't support a requested option (e.g. type), I have 'type' included\nas part of the supported client options.\n\n> > + for (i = 0; i < args.object_info_options->nr; i++) {\n> > + if (packet_reader_read(&reader) != PACKET_READ_NORMAL) {\n> > + check_stateless_delimiter(transport->stateless_rpc, &reader, \"stateless delimiter expected\");\n>\n> This is one of a number of lines in this series that are way over the 80\n> column limit.\n\nack\n\n> > + if (unsorted_string_list_has_string(args.object_info_options, reader.line)) {\n> > + if (!strcmp(reader.line, \"size\"))\n> > + size_index = i;\n>\n> Should we be checking for \"type\" as well? Also does protocol-v2.txt need\n> updating as it only mentions \"size\" as an attribute.\n\nI gave context above -- the server does not accept 'type' so protocol-v2.txt\ndoes not need to be updated.\n\n> To avoid a possible out-of-bounds access we need to check that\n> size_index + 1 < object_info_value.nr in case the server response is\n> malformed\n\nack\n\n> > + if (!strcmp(object_info_values.items[1 + size_index].string, \"\"))\n> > + die(\"object-info: not our ref %s\",\n>\n> I'm a bit confused by this message is it trying to say \"object %s is\n> missing on the server\"?\n\nCorrect. You'll find the same error message in upload-pack.c\n\n> > + object_info_values.items[0].string);\n> > + *(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10);\n>\n> As Junio pointed out in his comments in v4 there is no error checking\n> here - we should check the server has actually returned a number. Note\n> that strtoul() will happily parse negative numbers so we probably want\n> to do something like\n>\n> const char *endp\n> errno = 0\n> if (!isdigit(*object_info_values.items[1 + size_index].string))\n> die(\"...\")\n> *(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 +\n> size_index].string, &endp, 10);\n> if (errno || *endp)\n> die(\"...\")\n\n\n\n> Should be we checking the object id matches what we asked for? (I'm not\n> sure if protocol-v2.txt mentions the order in which the objects are\n> returned)\n\nHmmmm I think I either check for an object id match or update\nprotocol-v2.txt to mention order is consistent.\n\n> Should we be parsing the object type here as well?\n\nWhen the server starts supporting it.\n\n> > + for (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n> > + struct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n>\n> Using CALLOC_ARRAY() or p = xcalloc(1, sizeof(*p)) would be safer here\n> (and everywhere else where you have used xcalloc()) as it ensures we\n> allocate the correct size.\n\nack\n\n> > + /*\n> > + * Transport will attempt to pull only object-info. Fallbacks\n> > + * to pulling entire object if object-info is not supported\n> > + */\n>\n> Is it definitely true that we fallback to pulling the entire object? -\n> there is at least one place above where we do\n\nYes, fetch_refs_via_pack() is where fetch_object_info() is called,\nmaking it easy to fallback if fetch_object_info() fails.\n\n>> + while (packet_reader_read(&reader) == PACKET_READ_NORMAL && i < args.oids->nr) {\n>> + struct string_list object_info_values = STRING_LIST_INIT_DUP;\n>\n> I forget to say earlier that this is leaked\n\nack\n\nThank you for the review!\n"},{"id":"460681","messageId":"CAFySSZDNFAn6jZuby8Xj4EGUq0f7BJHv1gZwBNU0h+5D4+ny1A@mail.gmail.com","threadId":"57388","inReplyTo":"8a1ff659-2086-f3cf-469a-76f5e2bc60e3@gmail.com","subject":"Re: [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-04T22:57:14Z","receivedAt":"2022-08-04T22:59:58Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> This might be a daft question but could we teach cat-file to check if an\n> object is available locally and if it is not retrieve just the metadata\n> from the remote rather than adding a new command to --batch-command?\n> That way it would be transparent to the user and they would not have to\n> worry about whether they might be querying a remote object or not.\n\nDefinitely a good question! If we did this over a large amount of objects and\nwe didn't have those objects locally, then we would have to query for each\nof them, resulting in a lot of transport overhead. Therefore, I needed to use\n--batch-command to allow me to query all at once. This also begs the\nquestion of why I don't check for the objects locally before querying the\nserver. The intention of my patch assumes that the user querying for\nobject-info knows they don't have the object locally and would therefore\nlike to see the size of an object they could potentially fetch. This could\ndefinitely be potential future work (which I should mention in my commit\nmessage!)\n"},{"id":"460783","messageId":"CAOLTT8RrjcqXoFrq540ur5B=cdkLhwteZje23Nn3+wtdZZdZMg@mail.gmail.com","threadId":"57388","inReplyTo":"220729.86tu70ig47.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2022-08-07T05:50:57Z","receivedAt":"2022-08-07T05:51:13Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hi, Ævar,\n\nÆvar Arnfjörð Bjarmason 于2022年7月29日周五 14:33写道：\n>\n>\n> On Thu, Jul 28 2022, Calvin Wan wrote:\n>\n> > Since the `info` command in cat-file --batch-command prints object info\n> > for a given object, it is natural to add another command in cat-file\n> > --batch-command to print object info for a given object from a remote.\n>\n> Is it ?:)\n>\n> > Add `remote-object-info` to cat-file --batch-command.\n>\n> I realize this bit of implementation changed in v4, i.e. it used to be\n> in \"fetch\", and I'm happy to have it moved out of there, we don't need\n> to overload it more.\n>\n> But I remember thinking (and perhaps commenting on-list, I can't\n> remember) that the \"object-info\" server verb was a bit odd at the time\n> that it was implemented. I understand the motivation, but surely it was\n> stumbling its way towards being something more generic, i.e. being able\n> to just expose cmd_cat_file() in some form.\n>\n> Which is one of the goals I've had in mind with working on fixing memory\n> leaks in various places, i.e. once you get common commands to clean up\n> after themselves it usually becomes to have a \"command server\".\n>\n\nNow I'm starting to agree with you on this:\n\nMaybe now git doesn't have a good interface to execute normal git commands\n(except git-upload-pack, git-receive-pack...) on remote git server.\n\nThis patch wants to get remote object-info by the git-upload-pack interface.\nBut this thing can easily work by some RPC server e.g. Gitaly in Gitlab.\n\nI don't know if git itself has the need to reimplement these remote\ncalls In some\nsecure environment?... Is it perhaps possible to get better\nperformance and versatility\nthan Gitaly? I donno.\n\n> So (and I don't mind if this is longer term, just asking), is there a\n> reason for why we wouldn't want to do away with object-info and this\n> \"cat-file talks to a remote\", in favor of just having support for\n> invoking arbitrary commands from a remote.\n>\n> Of course that set of allowed RCE commands would be zero by default, but\n> if we had some way to define tha \"cat-file\" was allowed to be called,\n> and only if you invoked:\n>\n> cat-file --batch=\"%(objectsize)\"\n>\n> Or whatever, but over the v2 protocol, wouldn't we basically have\n> object-info in a more roundabout way?\n\nZheNing Hu\n"},{"id":"460808","messageId":"00a58dbb-32e8-98dd-6c72-235e9c33af3b@gmail.com","threadId":"57388","inReplyTo":"20220728230210.2952731-6-calvinwan@google.com","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-08-08T09:16:13Z","receivedAt":"2022-08-08T09:16:22Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 29/07/2022 00:02, Calvin Wan wrote:\n> diff --git a/transport.c b/transport.c\n> index 52db7a3cb0..2d503e2fbd 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -363,10 +437,12 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \t\t\t int nr_heads, struct ref **to_fetch)\n> {\n> \tint ret = 0;\n> +\tsize_t i;\n> \tstruct git_transport_data *data = transport->data;\n> \tstruct ref *refs = NULL;\n> \tstruct fetch_pack_args args;\n> \tstruct ref *refs_tmp = NULL;\n> +\tstruct ref *object_info_refs = xcalloc(1, sizeof (struct ref));\n\nWhen git is compiled with SANITIZE=address then one of the cat-file \ntests added in patch 6 fails with an out of bounds access. The problem \nis that the last member of struct ref is a flexible array that is \nexpected to contain a NUL terminated string but we don't allocate any \nmemory for the string here. We could just add one to the size of the \nallocation but as there is a dedicated allocation function it would be \nbetter to use alloc_ref(\"\").\n\nI think it would also be worth delaying this allocation until we're sure \nit is going to be needed.\n\n> \tmemset(&args, 0, sizeof(args));\n> \targs.uploadpack = data->options.uploadpack;\n> @@ -392,8 +468,25 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \targs.server_options = transport->server_options;\n> \targs.negotiation_tips = data->options.negotiation_tips;\n> \targs.reject_shallow_remote = transport->smart_options->reject_shallow;\n> -\n> -\tif (!data->got_remote_heads) {\n> +\targs.object_info = transport->smart_options->object_info;\n> +\n> +\tif (transport->smart_options && transport->smart_options->object_info) {\n> +\t\tstruct ref *ref = object_info_refs;\n> +\n> +\t\tif (!fetch_object_info(transport, data->options.object_info_data))\n> +\t\t\tgoto cleanup;\n\nIf we allocate object_info_refs and initialize ref here then we avoid \nallocating it when it is not needed.\n\n> +\t\targs.object_info_data = data->options.object_info_data;\n> +\t\targs.quiet = 1;\n> +\t\targs.no_progress = 1;\n> +\t\tfor (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n> +\t\t\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n\nThis needs to use alloc_ref(\"\") as well\n\n> +\t\t\ttemp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i);\n\nThis would be clearer as ...oids->oid[i] I think\n\nBest Wishes\n\nPhillip\n"},{"id":"460810","messageId":"edc7763d-420f-a1f9-de13-fb2cb4ddfb2a@gmail.com","threadId":"57388","inReplyTo":"CAFySSZDvgwbbHCHfyuaqX3tKsr-GjJ9iihygg6rNNe46Ys7_EA@mail.gmail.com","subject":"Re: [PATCH v5 5/6] transport: add client support for object-info","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-08-08T10:07:15Z","receivedAt":"2022-08-08T10:07:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 04/08/2022 23:20, Calvin Wan wrote:\n>>> The client currently supports requesting a list of object ids with\n>>> features 'size' and 'type' from a v2 server. If a server does not\n>>> advertise either of the requested features, then the client falls back\n>>> to making the request through 'fetch'.\n>>\n>> I'm confused by the 'type' support, I might have missed something as I'm\n>> not familiar with this code but I couldn't see where we parse the type\n>> returned by the server.\n> \n> I should clarify that the server does not support 'type', only the client.\n> Since the client falls back to 'fetch' to grab the object info if the server\n> doesn't support a requested option (e.g. type), I have 'type' included\n> as part of the supported client options.\n\nDoes that mean all the type client code is effectively unused and so \nuntested?\n\n>>> + if (!strcmp(object_info_values.items[1 + size_index].string, \"\"))\n>>> + die(\"object-info: not our ref %s\",\n>>\n>> I'm a bit confused by this message is it trying to say \"object %s is\n>> missing on the server\"?\n> \n> Correct. You'll find the same error message in upload-pack.c\n\nI find the message confusing as we're talking about oids not refs. I \nhave also realized that dying here is incompatible with the normal \ncat-file behavior of printing \" missing\" rather than dying \nwhen a missing object is queried.\n\n>> Should be we checking the object id matches what we asked for? (I'm not\n>> sure if protocol-v2.txt mentions the order in which the objects are\n>> returned)\n> \n> Hmmmm I think I either check for an object id match or update\n> protocol-v2.txt to mention order is consistent.\n\nIf we can then updating the protocol makes sense. Even if we do that a \nbuggy or malicious server could return the objects in a different order. \nA malicious server can also lie about the size so I'm not sure how much \nbenefit there is in checking the oids in terms of validating the \nresponse but I think we should definitely check that the server returns \nthe expected number of sizes.\n\nBest Wishes\n\nPhillip\n"},{"id":"460838","messageId":"CAFySSZCoykakjSrgUq0kcQeP_1WDoPgxBpyLeLYSBakCyuOJVg@mail.gmail.com","threadId":"57388","inReplyTo":"CAOLTT8Q-241cExFcbXSUfj42GontKH0uQpX1Voekf4RJ95hAVw@mail.gmail.com","subject":"Re: [PATCH v4 0/8] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-08T17:32:25Z","receivedAt":"2022-08-08T17:32:41Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> I have to say I am very curious about this feature. Since the current\n> git partial-clone interface supports only a few filters:\n>\n> blob:limit\n> blob:none\n> sparse:oid\n> tree:0\n>\n> Though these filters reduce the number of objects downloaded each time,\n> sometimes I just need *only* one blob object, git partial-clone will still\n> download some additional commits and tree objects from the remote.\n>\n> This patch can get the remote-object-info by git cat-file, so can we go\n> further to get the remote-object-content of the object?\n>\n> Something like:\n>\n> $ git cat-file --batch-command\n> remote-object-content origin \n\nI think this can be potentially future work. Part of my hesitation stems\nfrom the fact that there is no need to print out any object info when\ndownloading an object. Therefore, it seems better suited to first be\nimplemented in git fetch (or added as a filter to partial-clone). And\nthen remote-object-content can be a combination of fetch +\nremote-object-info.\n"},{"id":"460839","messageId":"CAFySSZDMwMa-RFLVeAONnySrmsg53G21mXd-BDjyZEpyAVbMbg@mail.gmail.com","threadId":"57388","inReplyTo":"CAOLTT8RyDCokvJBaC1CCoT2+3AE1kOn7evjkTMK_V6KWoaH30g@mail.gmail.com","subject":"Re: [PATCH v4 8/8] cat-file: add --batch-command remote-object-info command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-08T17:43:23Z","receivedAt":"2022-08-08T17:44:10Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> > `rest`::\n> > If this atom is used in the output string, input lines are split\n>\n> Why not forbidden %(rest) here too?\n\nGood catch. This should definitely be forbidden. I thought at first\nthis was inconsequential, but since I also have remote as part of\nthe input, this would no longer hold true.\n\n> > + if (strstr(format, \"%(objectsize:disk)\"))\n> > + die(_(\"objectsize:disk is currently not supported with remote-object-info\"));\n> > + if (strstr(format, \"%(deltabase)\"))\n> > + die(_(\"deltabase is currently not supported with remote-object-info\"));\n>\n> %(rest) too?\n\nditto\n\n> Maybe such code style will be better?\n>\n> if (!gtransport->smart_options) {\n> return -1;\n> }\n> ...\n> return transport_fetch_refs(gtransport, NULL);\n\nThat does look better!\n\n> > - cmd->fn(opt, p, output, data);\n> > + if (!strcmp(cmd->name, \"remote-object-info\")) {\n> > + char *line = xstrdup_or_null(p);\n> > + parse_remote_info(opt, line, output, data, cmd, NULL);\n>\n> memory leak: \"line\".\n\nack\n\n> > +test_expect_success 'remote-object-info fails on unspported filter option (deltabase)' '\n> > + (\n> > + cd \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> > +\n> > + test_must_fail git cat-file --batch-command=\"%(deltabase)\" 2>err <<-EOF &&\n> > + remote-object-info \"$HTTPD_URL/smart/http_parent\" $hello_sha1\n> > + EOF\n> > + test_i18ngrep \"deltabase is currently not supported with remote-object-info\" err\n> > + )\n> > +'\n> > +\n>\n> %(rest) too?\n\nditto\n\n> > +test_expect_success 'batch-command remote-object-info file://' '\n> > + (\n> > + cd server &&\n> > +\n> > + echo \"$hello_sha1 $hello_size\" >expect &&\n> > + echo \"$tree_sha1 $tree_size\" >>expect &&\n> > + echo \"$commit_sha1 $commit_size\" >>expect &&\n> > + echo \"$tag_sha1 $tag_size\" >>expect &&\n> > + git cat-file --batch-command=\"%(objectname) %(objectsize)\" >actual <<-EOF &&\n> > + remote-object-info \"file://$(pwd)\" $hello_sha1\n> > + remote-object-info \"file://$(pwd)\" $tree_sha1\n> > + remote-object-info \"file://$(pwd)\" $commit_sha1\n> > + remote-object-info \"file://$(pwd)\" $tag_sha1\n> > + EOF\n> > + test_cmp expect actual\n> > + )\n>\n> Can we support instead of only here?\n\nNot at the current moment. The server is unable to handle\nanything besides oids.\n\n>\n> $ git cat-file --batch-check\n> HEAD\n> 28583b8d8ca72730d7c9e0ea50861ad431a6dea4 commit 3038\n> master\n> ab336e8f1c8009c8b1aab8deb592148e69217085 commit 281\n> origin/master\n> 23b219f8e3f2adfb0441e135f0a880e6124f766c commit 282\n> origin/master:git.c\n> e5d62fa5a92e95e1ede041ebf913d841744c31f8 blob 28398\n>\n> So cat-file --batch-check can support it.\n>\n> $git cat-file --batch-commands\n> remote-object-info \"file://$(pwd)\" master:git.c\n>\n> I guess it cannot work now, right?\n\nCorrect.\n"},{"id":"460841","messageId":"CAFySSZBPoJC8je2jr4_nrrzdFG3=HuRcou6qjrJTNOzue1pvgQ@mail.gmail.com","threadId":"57388","inReplyTo":"220729.86tu70ig47.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-08T18:07:01Z","receivedAt":"2022-08-08T18:07:16Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> > Since the `info` command in cat-file --batch-command prints object info\n> > for a given object, it is natural to add another command in cat-file\n> > --batch-command to print object info for a given object from a remote.\n>\n> Is it ?:)\n\nHaha yes this could use a little rewording\n\n> > Add `remote-object-info` to cat-file --batch-command.\n>\n> I realize this bit of implementation changed in v4, i.e. it used to be\n> in \"fetch\", and I'm happy to have it moved out of there, we don't need\n> to overload it more.\n>\n> But I remember thinking (and perhaps commenting on-list, I can't\n> remember) that the \"object-info\" server verb was a bit odd at the time\n> that it was implemented. I understand the motivation, but surely it was\n> stumbling its way towards being something more generic, i.e. being able\n> to just expose cmd_cat_file() in some form.\n>\n> Which is one of the goals I've had in mind with working on fixing memory\n> leaks in various places, i.e. once you get common commands to clean up\n> after themselves it usually becomes to have a \"command server\".\n>\n> So (and I don't mind if this is longer term, just asking), is there a\n> reason for why we wouldn't want to do away with object-info and this\n> \"cat-file talks to a remote\", in favor of just having support for\n> invoking arbitrary commands from a remote.\n>\n> Of course that set of allowed RCE commands would be zero by default, but\n> if we had some way to define tha \"cat-file\" was allowed to be called,\n> and only if you invoked:\n>\n> cat-file --batch=\"%(objectsize)\"\n>\n> Or whatever, but over the v2 protocol, wouldn't we basically have\n> object-info in a more roundabout way?\n\nWhile I do think that if we did have a set of allowed RCE commands, this\nwould be a good candidate to be one of those commands. I am worried\nabout security, maintainability, and server performance risks this change\nwould also carry. Figuring out which commands are secure and would\nnot overload the server, and then maintaining that set seems like a much\nmore worrisome design than having a secure git server.\n"},{"id":"460844","messageId":"CAFySSZAKibH3dcHLFz=1csKwfa3-Ebkou_-soQS3h_+ewoZfUw@mail.gmail.com","threadId":"57388","inReplyTo":"xmqq1qu37pbf.fsf@gitster.g","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2022-08-08T18:37:34Z","receivedAt":"2022-08-08T18:37:48Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> > While `info` takes object ids one at a time, this creates overhead when\n> > making requests to a server so `remote-object-info` instead can take\n> > multiple object ids at once.\n>\n> I am not sure if the above is desirable---doesn't it make the\n> protocol (i.e. the requests the invoker of \"cat-file\" needs to\n> prepare) unnecessarily complex? If we have \"buffered\" mode, the\n> implementation can collect and batch requests and talk with the\n> other side in a single communication with the other end, no?\n\nI'm a little bit confused as to what you are suggesting here, in\nterms of how a user would want to call this command. Are you\nsaying this functionality should only work in \"buffered\" mode?\nOr are you saying that `remote-object-info` should only take\none object id at a time, and in buffer mode, those requests are\nbatched together?\n\n>\n> > In --buffer mode, this changes to:\n> >\n> > - Receive and parse input from user\n> > - Store respective function attached to command in a queue\n> > - After flush, loop through commands in queue\n> > - Call respective function attached to command\n> > - Set batch mode state, get object info, print object info\n>\n> So, the existing buffered mode is poorly designed, in other words.\n> Isn't this something we can fix for performance without affecting\n> the existing callers?\n\nMuch of the existing cat-file code seems to be designed around\ntaking in one object/command at a time, and --buffer can be\nthought of as a wrapper around that preexisting code that simply\nstores a queue of commands rather than running them immediately.\nI think the existing buffered mode is well designed -- this part of the\ncommit message is to explain why at some point I need to preprocess\nwhat is in the buffer\n"},{"id":"461069","messageId":"220811.86a68bqbwb.gmgdl@evledraar.gmail.com","threadId":"57388","inReplyTo":"CAFySSZBPoJC8je2jr4_nrrzdFG3=HuRcou6qjrJTNOzue1pvgQ@mail.gmail.com","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-08-11T10:58:52Z","receivedAt":"2022-08-11T11:04:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 08 2022, Calvin Wan wrote:\n\n>> > Since the `info` command in cat-file --batch-command prints object info\n>> > for a given object, it is natural to add another command in cat-file\n>> > --batch-command to print object info for a given object from a remote.\n>>\n>> Is it ?:)\n>\n> Haha yes this could use a little rewording\n>\n>> > Add `remote-object-info` to cat-file --batch-command.\n>>\n>> I realize this bit of implementation changed in v4, i.e. it used to be\n>> in \"fetch\", and I'm happy to have it moved out of there, we don't need\n>> to overload it more.\n>>\n>> But I remember thinking (and perhaps commenting on-list, I can't\n>> remember) that the \"object-info\" server verb was a bit odd at the time\n>> that it was implemented. I understand the motivation, but surely it was\n>> stumbling its way towards being something more generic, i.e. being able\n>> to just expose cmd_cat_file() in some form.\n>>\n>> Which is one of the goals I've had in mind with working on fixing memory\n>> leaks in various places, i.e. once you get common commands to clean up\n>> after themselves it usually becomes to have a \"command server\".\n>>\n>> So (and I don't mind if this is longer term, just asking), is there a\n>> reason for why we wouldn't want to do away with object-info and this\n>> \"cat-file talks to a remote\", in favor of just having support for\n>> invoking arbitrary commands from a remote.\n>>\n>> Of course that set of allowed RCE commands would be zero by default, but\n>> if we had some way to define tha \"cat-file\" was allowed to be called,\n>> and only if you invoked:\n>>\n>> cat-file --batch=\"%(objectsize)\"\n>>\n>> Or whatever, but over the v2 protocol, wouldn't we basically have\n>> object-info in a more roundabout way?\n>\n> While I do think that if we did have a set of allowed RCE commands, this\n> would be a good candidate to be one of those commands. I am worried\n> about security, maintainability, and server performance risks this change\n> would also carry. Figuring out which commands are secure and would\n> not overload the server, and then maintaining that set seems like a much\n> more worrisome design than having a secure git server.\n\nI'm only suggesting that the interface be extendable enough to allow for\nthat, not that we do it right now, i.e. that the protocol command could\nbe:\n\n\t# With appropriate \\0 delimiting etc.\n\tcmd cat-file -e \n\nAs opposed to something like:\n\n\tobject-exists \n\nBut that we would *not* for now expose some mechanism where the server\noperator could configure which arbitrary commands to allow, we'd just\nintercept that specifically, and dispatch it to the appropriate code in\nbuiltin/cat-file.c (or libify them enough to expose them, and then call\nthat).\n\nOf course one of the points of doing so would be to eventually expose\nthe ability to run a larger set of safe commands remotely, if the server\noperator agrees, but we'd leave that for the future.\n\nRight now we'd get the benefit of not duplicating various parts of\nexisting code, and having a plain one-to-one mapping to existing\ncommands.\n\n"},{"id":"461179","messageId":"xmqqlerr94bb.fsf@gitster.g","threadId":"57388","inReplyTo":"20220502170904.2770649-1-calvinwan@google.com","subject":"Re: [PATCH v4 0/8] cat-file: add --batch-command remote-object-info command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-13T22:17:12Z","receivedAt":"2022-08-13T22:17:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Style and coccinelle fixes; please squash in when you reroll.\n\n * var = xcalloc(count, sizeof(*var)) --> CALLOC_ARRAY(var, count)\n for count != 1\n\n * sizeof (type) --> sizeof(type)\n\nThanks.\n\n builtin/cat-file.c | 2 +-\n transport.c | 4 ++--\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 57c090f249..4afe82322f 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -515,7 +515,7 @@ static int get_remote_info(struct batch_options *opt, int argc, const char **arg\n \t\tsize_t j;\n \t\tint include_size = 0, include_type = 0;\n \n-\t\tremote_object_info = xcalloc(object_info_oids.nr, sizeof(struct object_info));\n+\t\tCALLOC_ARRAY(remote_object_info, object_info_oids.nr);\n \t\tgtransport->smart_options->object_info = 1;\n \t\tgtransport->smart_options->object_info_oids = &object_info_oids;\n \t\t/**\ndiff --git a/transport.c b/transport.c\nindex 64bcc311ff..87197f0ec7 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -442,7 +442,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tstruct ref *refs = NULL;\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n-\tstruct ref *object_info_refs = xcalloc(1, sizeof (struct ref));\n+\tstruct ref *object_info_refs = xcalloc(1, sizeof(struct ref));\n \n \tmemset(&args, 0, sizeof(args));\n \targs.uploadpack = data->options.uploadpack;\n@@ -479,7 +479,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\targs.quiet = 1;\n \t\targs.no_progress = 1;\n \t\tfor (i = 0; i < transport->smart_options->object_info_oids->nr; i++) {\n-\t\t\tstruct ref *temp_ref = xcalloc(1, sizeof (struct ref));\n+\t\t\tstruct ref *temp_ref = xcalloc(1, sizeof(struct ref));\n \t\t\ttemp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i);\n \t\t\ttemp_ref->exact_oid = 1;\n \t\t\tref->next = temp_ref;\n-- \n2.37.2-479-gedbcd27e9d\n\n"},{"id":"463840","messageId":"220928.8635cby73b.gmgdl@evledraar.gmail.com","threadId":"57388","inReplyTo":"20220728230210.2952731-7-calvinwan@google.com","subject":"Re: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-28T13:12:48Z","receivedAt":"2022-09-28T13:14:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 28 2022, Calvin Wan wrote:\n\n> +\t\t\tgtransport->smart_options->object_info_data = &remote_object_info;\n> +\t\t\tretval = transport_fetch_refs(gtransport, NULL);\n\nI see Phillip Wood noted that this fails SANITIZE=address in\nhttps://lore.kernel.org/git/00a58dbb-32e8-98dd-6c72-235e9c33af3b@gmail.com/;\njust a *bump* about this issue. I noticed \"seen\" failed, bisected down\nto 6/6 here, then saw his >month old report...\n"},{"id":"464032","messageId":"xmqqedvs78fl.fsf@gitster.g","threadId":"57388","inReplyTo":"20220728230210.2952731-1-calvinwan@google.com","subject":"Re: [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-30T23:23:58Z","receivedAt":"2022-09-30T23:24:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Sometimes it is useful to get information about an object without having\n> to download it completely. The server logic has already been implemented\n> as “a2ba162cda (object-info: support for retrieving object info,\n> 2021-04-20)”. This patch implements the client option for it.\n>\n> Add `--object-info` option to `cat-file --batch-command`. This option\n> allows the client to make an object-info command request to a server\n> that supports protocol v2. If the server is v2, but does not allow for\n> the object-info command request, the entire object is fetched and the\n> relevant object info is returned.\n>\n> === Changes since v4 ===\n> - Instead of making 2 rounds trips to determine whether the server can\n> handle the client object-info request, the server now advertises the\n> features it can send back so the client can immediately fallback to\n> fetch if any feature is not supported\n> - Fixed nits and clarified some documentation/commit messages\n>\n> Signed-off-by: Calvin Wan \n> Helped-by: Jonathan Tan \n\nFWIW, this one seems to make t1006 fail under SANITIZE=address, i.e.\n\n $ SANITIZE=address make test\n\nI thought it would be better to report this before you send out v6\nafter the release.\n\nThanks.\n"}]}